Back to Articles
Studio Production

AI Stem Separation: What It Does Well and When It Wrecks a Mix

Lucy AilusOctober 27, 20266 min read16 views
AI Stem Separation: What It Does Well and When It Wrecks a Mix

Stem separation used to be a research problem. Now it is a button, and it produces results good enough that people forget what it is doing. The tool does not find the four tracks that were recorded; it estimates them from a mixed file by pattern recognition. Everything that follows is a consequence of that one fact.

That is not a reason to avoid it. It is a reason to know which jobs it is good at before you build a workflow on top of it.

What is actually happening

A mixed recording contains every instrument summed into two channels. Separation models have learned the statistical fingerprints of common instruments and voices, and they estimate how much of the signal each one contributes at every moment, then resynthesize each part.

Nothing in that process has access to the original performance. There is no isolation to recover, only a plausible reconstruction. Where two sources overlap in frequency and time, the model has to choose, and where it chooses wrong it does not fail quietly: it leaves an artifact behind. That is the whole risk model, and almost every practical rule comes out of it.

Where separation is genuinely excellent

Sampling and referencing. Lifting a drum loop, an isolated chord bed or a vocal phrase to study or rework is where these tools earn their keep. The result does not need to be perfect, because it is going to be chopped, pitched and recontextualized anyway.

Live recordings and rehearsal material. Pulling a usable vocal from a phone recording or a board mix to hear what someone actually sang is a huge improvement over listening through a wall of cymbals.

Remix and edit work. Building a shortened edit, a clean intro, or an instrumental version for a client who has lost the multitracks. Separation makes a salvage job possible that would otherwise be impossible.

Analysis and transcription. Isolating a bass line to transcribe it, or a vocal to check a lyric, is far easier than trying to hear it in the mix.

Archival rescue. When a client arrives with only a stereo master and no session, separation is the only path to a rebalanced version.

Where it wrecks the result

Rebuilding a full mix from separated stems. This is the mistake people make first. Four separated stems, remixed and rebalanced, do not sum back to the original and they do not sound like a real multitrack. Each has bled a little of everything else into it, so every fader move drags along a ghost of another instrument. Push the vocal up and the hi-hat gets brighter. Cut the bass and the kick thins out, because some kick lived in the bass stem.

Removing a part from a finished mix. Pulling out a guitar or a keyboard to make room for a replacement sounds worse than the arrangement problem it is trying to solve. The removed instrument leaves a smear behind in the other stems, and the hole it vacates is not silent, it is full of artifacts.

Anything with heavy processing already on it. A mix that has been compressed, saturated and limited has smeared the instruments into each other. Separating it works far worse than separating a dry, lightly processed recording, because the model's fingerprints have been blurred by the bus processing.

Dense, overlapping arrangements. Thick walls of distorted guitars, layered synths, or a loud band in a small room are the hard cases. The instruments share almost everything, so the model guesses more often and the artifacts multiply.

Material with heavy reverb. Reverb tails belong to everything and nothing. Separation tends to distribute them oddly, leaving a stem sounding dry while another carries an inexplicable wash.

How to tell artifacts from the real thing

Listen for these specifically. They are consistent across models.

  • Watery or metallic texture in sustained notes and cymbal wash.
  • Pumping that follows the rhythm of another instrument, because the model is reacting to it.
  • Disappearing ambience when one stem is muted, so the room changes shape rather than just losing an instrument.
  • Hollow attacks on drums or plucked instruments, where the transient has been split between two stems.
  • Sibilance damage in vocals, where consonants have been nibbled at.

The quick test: solo each stem in turn and listen for anything that sounds like it is breathing in time with a part you did not solo. That is leakage, and it is the signature of reconstruction.

Working practice

  1. Keep the source. Never process the only copy. Separation output is a derived asset, not a master.
  2. Separate from the least processed source available. A dry mix separates better than a limited master. If you have both, use the dry one.
  3. Use separation as a source, not as a mix. If you are rebuilding, record or program the parts you can, and treat the separated material as a texture or a placeholder.
  4. Sum back and check. Add the stems together with no processing and compare against the original. Any difference in tone or dynamics is damage you have already accepted.
  5. Re-record when the client is available. Fifteen minutes of a real performance beats an hour of artifact repair, and it always will.
  6. Do not stack separations. Separating a stem again to isolate one more part multiplies the error rather than refining it.

The honest position

Separation is one of the genuinely useful arrivals in audio software. It is also a reconstruction, and reconstructions are for material you cannot get any other way. The professional habit is simple: ask whether the multitrack exists, use it if it does, and reach for separation only when the answer is no.

AI is an assistant here, and a good one. It is not a time machine. When you treat it as a tool for rescuing what would otherwise be lost, it looks like magic. When you treat it as a substitute for having recorded the session properly, it looks like a mix with a hole in it.

Related reading on this site: AI Noise Reduction Versus a Hardware Gate, Top 10 Reasons a DI Track Sounds Wrong Before Mixing, and Mastering the Art of Mixing: A Step-by-Step Guide.

Comments (0)

Leave a Comment

Comments are moderated before appearing.

No comments yet. Be the first to comment!