The Science Behind AI Stem Separation Technology

Separating a mixed song into individual instruments sounds like magic. But it's actually a combination of clever signal processing and pattern-recognition AI trained on millions of examples. Here's how it works.

The Core Problem: Cocktail Party Effect

Humans are remarkably good at focusing on one voice in a noisy room — the "cocktail party effect." Our brains perform unconscious source separation in real time. Getting computers to do the same thing is called the "cocktail party problem" in signal processing, and it's been unsolved for decades.

Audio mixing makes this problem harder: when a producer mixes a song, all the instruments are summed together into a single waveform. The mixing process is inherently information-destroying — you can always add two signals together, but unambiguously separating them is not always possible.

Approach 1: Signal Processing (Filter Banks)

The simplest approach exploits the known frequency characteristics of different instruments:

• Bass guitar and kick drum: dominant below 200 Hz
• Vocals: most energy between 300 Hz and 3 kHz
• Hi-hats and cymbals: above 6 kHz
• Guitar: 80 Hz to 6 kHz

By applying bandpass filters tuned to these ranges, you can extract approximate stems. This is fast, runs in any browser, and requires no training data. The limitation: frequency ranges overlap significantly, so bleed between sources is high.

This is the classic approach to browser-based processing — deterministic, lightweight, and it needs no GPU.

Approach 2: Mid-Side Matrix (for Vocals)

As covered in our Mid-Side Processing article, this exploits stereo positioning. Vocals centred in the mix are captured in the Mid (L+R) channel; panned instruments appear in the Side (L−R) channel.

Simple, elegant, and surprisingly effective on commercial recordings — but fails when instruments share the centre position (bass, kick) or when vocals are multi-tracked and panned.

Approach 3: Spectrogram Masking (The AI Way)

Modern deep learning approaches work on spectrograms — 2D representations of audio where the x-axis is time, y-axis is frequency, and brightness is amplitude. A spectrogram turns audio into an image.

The algorithm:

1. Convert the audio to a spectrogram using Short-Time Fourier Transform (STFT)
2. Feed the spectrogram to a neural network
3. The network predicts a mask — a spectrogram-shaped matrix of values 0–1 indicating "how much of this time-frequency bin belongs to source X"
4. Multiply the original spectrogram by the mask to extract each source
5. Convert back to audio with Inverse STFT

the model: The Architecture Behind Modern Separation

The most successful neural network architecture for audio separation is the the model, originally developed for medical image segmentation. It has an encoder-decoder structure with skip connections:

• Encoder — progressively compresses the spectrogram, learning higher-level patterns
• Bottleneck — captures the most abstract representation of the audio
• Decoder — reconstructs the mask at full resolution
• Skip connections — pass high-resolution details from encoder to decoder, preserving fine-grained frequency information

Deezer's Spleeter and Meta's Demucs both use variants of this architecture, trained on proprietary datasets of songs with their original multitrack stems.

Training Data: The Secret Ingredient

What makes modern neural separation so powerful is training data. Models like Demucs are trained on thousands of songs where the unmixed stems are known — the exact vocal track, drum track, and bass track used in production.

During training, the model sees the mixed audio and tries to predict the individual stems. The difference between prediction and ground truth is the loss, which is minimised through backpropagation over millions of examples.

This is why neural approaches generalise better than filter banks — they've learned the acoustic signatures of real instruments across a huge variety of musical styles and production techniques.

Why Browser-Based Tools Use Signal Processing

Running a the model in the browser is possible in principle (TensorFlow.js exists), but the model file sizes (100MB–1GB) and compute requirements make it impractical for real-time use without a GPU. Signal processing approaches run in milliseconds on any device.

The trade-off: signal processing is faster and private; neural network approaches produce better quality. 7By.in chose browser-first signal processing for the privacy and accessibility benefits — and the downloadable 7By AI Engine is open source so developers can extend it.