[Paper Review] DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio
Speech & Audio
How can we make a better dataset from in-the-wild data for training full-duplex models?
✔️ Background
Training a full-duplex spoken dialogue model requires dialogue recordings where each speaker is on a separate track — but such "stereo" data is scarce and expensive to collect (telephone corpora, controlled recording setups). What's abundant instead is in-the-wild internet audio: monaural mixtures where both speakers' voices are mixed together and the signal is degraded (noise, reverb, compression, clipping). This makes it unusable directly for full-duplex training. DialogueSidon tackles this gap by jointly restoring and separating a degraded monaural two-speaker mixture back into two clean, speaker-wise tracks — turning "found" internet audio into full-duplex training data.
✔️ Paper Info
- Title: DialogueSidon: Recovering Full-Duplex Dialogue Tracks from In-the-Wild Dialogue Audio (https://arxiv.org/abs/2604.09344)
- Authors : Wataru Nakata, Yuki Saito, Kazuki Yamauchi, Emiru Tsunoo, Hiroshi Saruwatari
- Affiliation: The University of Tokyo, National Institute of Advanced Industrial Science and Technology (AIST)
- Conference: SIGDIAL2026 (Long paper)
- Keywords: Speech Conversational dataset
✔️ Main Problem and Key Approach
Limitations of previous works
Recording real conversations (Fisher, CALLHOME) or controlled two-party setups is costly, suffers from channel/latency artifacts, and doesn't scale.
TTS-synthesized dialogues (e.g., Moshi-style pipelines) are driven by turn-based text scripts, so they lack natural overlap, backchannels, and timing variability.
In-the-wild internet audio is plentiful and stylistically diverse, but (1) heavily degraded and (2) only available as a single mixed-down track, not speaker-wise.
Why a simple restoration → separation (or vice versa) cascade doesn't work:
Restoration first: monologue-trained restoration models tend to treat speaker overlap itself as "noise" and suppress or distort it.
Separation first: separation models aren't robust to the heavy, compounded degradations typical of internet audio.
Key idea: extend Sidon — a prior SSL-latent-space restoration model built for single-speaker monologue — into the dialogue setting by adding a speaker-separation capability, so a single degraded mono mixture is mapped to two clean speaker-wise waveforms in one unified model.
✔️ Architecture & Method
DialogueSidon is trained in two stages, both operating in a compact latent space derived from a frozen SSL model (w2v-BERT 2.0).
Stage 1 — SSL-VAE (defines the latent space for clean speech)
A frozen SSL model extracts a high-dimensional (1,536-d) hidden feature from clean speech.
A trainable encoder (2-layer linear + ReLU + variational bottleneck) compresses this into a much smaller latent z (dimension D).
A trainable decoder (based on the Descript Audio Codec / HiFi-GAN-style vocoder with Snake activation) reconstructs the waveform from z.
Trained with reconstruction + adversarial + KL losses. D was swept over {8, 16, 32, 64, 128}.
Stage 2 — Diffusion-based latent predictor (recovers speaker-wise latents from the degraded mix)
A conditioning feature c is extracted from the degraded mixture using a different SSL layer, fine-tuned with LoRA (rank 64) so it adapts to degraded audio while keeping pretrained knowledge.
Auxiliary heads: two linear projections from c give coarse speaker-wise latent estimates. These are matched to ground-truth latents via permutation-invariant training (PIT), which resolves the "who is speaker 1 vs 2" ambiguity and fixes a permutation π* for the rest of the pipeline.
Diffusion refinement: a Diffusion Transformer (DiT, 768-dim, 8 layers, 12 heads, RoPE) takes the stacked two-speaker latents, conditions on c and the auxiliary estimates, and refines them via v-prediction. Training uses a 1,000-step linear noise schedule; inference uses DPM-Solver++ with just 30 steps.
Total loss = auxiliary PIT loss + diffusion v-prediction loss.
At inference, the pipeline is: degraded mix → conditioning feature → coarse auxiliary latents → diffusion refinement → SSL-VAE decoder → two clean speaker waveforms.
✔️ Dataset
CALLHOME (German, English, Japanese, Spanish, Mandarin) + Fisher English — 2,225.9 hours total of telephone conversations, themselves first cleaned with Sidon. To simulate "in-the-wild" degradation, a 7-stage pipeline (reverb via room simulation, background noise from AudioSet/FMA/WHAM!/FSD50K/wind noise, random resampling for band-limiting, clipping, MP3 codec, packet loss, then weighted mixing into mono) was applied 4× per session with different seeds, yielding ~8,902 hours of paired clean/degraded dialogue.
✔️ Results
Latent size ablation (SWB): All DialogueSidon variants massively beat the baselines on WER. D=32 was best at 14.39% WER vs. 33.54% (retrained GENESES), 57.47% (Sidon), and 60.81% (Noisy). DialogueSidon doesn't top NISQA/DNSMOS, but wins on speaker similarity and VAD accuracy. D=128 degrades across the board, so D=32 was used for the rest of the experiments.
Subjective MOS (SWB): DialogueSidon 3.895 > GENESES 3.482 > Sidon 3.289 > Noisy 2.815, all differences significant (p<0.05).
Multilingual (CallFriend, 5 languages): DialogueSidon consistently beats GENESES on p-CER (content preservation) and generally on speaker similarity / VAD accuracy, even though GENESES sometimes scores higher on NISQA/DNSMOS — suggesting GENESES trades content fidelity for smoother-sounding output. The large gap between GENESES (orig) and retrained GENESES across every language reinforces that training on conversational (not monologue) data matters a lot.
In-the-wild (OpenDialog): DialogueSidon's WER (13.86%) is far below GENESES (43.79%) and GENESES (orig) (74.51%). Predicted quality (NISQA/DNSMOS) again favors GENESES, but subjective MOS clearly favors DialogueSidon (3.708 vs 3.131 vs 2.611, all significant) — i.e., human listeners prefer DialogueSidon's content/speaker-consistency tradeoff even when automatic perceptual predictors don't.
✔️ Resources
