[Paper Review] PHOENIX-VAD: STREAMING SEMANTIC ENDPOINT DETECTION FOR FULL-DUPLEX SPEECH INTERACTION
Overview
One of the most critical challenges in building a full-duplex dialogue model is achieving naturalistic turn-taking — the model should know when to respond and when to keep listening. This paper introduces Phoenix-VAD, an LLM-based approach that detects speech endpoints using semantic information, enabling smoother turn transitions. Since LLMs already have strong semantic understanding, the central question becomes: how do we apply that capability in a streaming environment? The authors answer this with a sliding window strategy.
Background & Motivation
Early full-duplex systems relied on acoustic VAD, which only distinguishes speech from silence. This evolved into Semantic VAD (using ASR + a lightweight LLM), and further into models like Moshi (real-time streaming LLM), LSLM (autoregressive generation integrated with turn-taking detection), and Freeze-Omni / MinMo / Mini-Omni2 (which add a linear layer to predict user interruption intent).
Phoenix-VAD positions itself against all of these by claiming four properties simultaneously: ASR-free, plug-and-play, real-time inference, and semantic capability — something none of the prior works fully achieve together.
Method
The approach is conceptually straightforward. Audio is processed chunk-by-chunk using a sliding window during training. The sliding window is defined on 100 Hz Fbank features (stride = 320 ms, window = 2560 ms), which are then passed through a frozen Zipformer encoder (~150M params) that internally downsamples by 4×, producing 25 Hz frame-level representations. These are projected through a lightweight adapter (two linear layers + ReLU) and fed into a fine-tuned LLM (Qwen2.5-0.5B-Instruct with LoRA). The model is prompted as a voice activity detector and trained to predict one of two tokens per chunk: 0 (Continue Speaking) or 1 (Stop Speaking).
Data Construction
Labels are binary (Continue / Stop), which feels simple but seems to work. The dataset was constructed synthetically:
- Text samples were generated using the ChatGPT API combined with internal resources.
- Speech was synthesized using Index-TTS, with speaker diversity ensured by randomly sampling from a timbre library of 1,007 English and 1,010 Chinese prompts from seed-tts-eval.
- Hesitations and interruptions were simulated by inserting silent segments.
- Timestamps were annotated using Paraformer.
Total: ~400,000 samples (~570 hours). Test set: 2,000 complete + 2,000 incomplete samples.
Experiments
The confusion matrix rows represent GT (Ground Truth), the actual label, and the columns represent Est (Estimation), the model's prediction.
For the semantically incomplete dataset (Table 2):
GT=Stop, Est=Stop: 1784 → correctly predicted as end of speech
GT=Stop, Est=Continue: 216 → user finished speaking, but model kept listening
GT=Continue, Est=Stop: 101 → user was still speaking, but model cut off
GT=Continue, Est=Continue: 19662 → correctly predicted as ongoing speech
For the semantically complete dataset (Table 3):
GT=Stop, Est=Stop: 1775 → correctly predicted as end of speech
GT=Stop, Est=Continue: 225 → user finished speaking, but model kept listening
GT=Continue, Est=Stop: 148 → user was still speaking, but model cut off
GT=Continue, Est=Continue: 24826 → correctly predicted as ongoing speech
The ablation study examines two design choices. First, reducing the chunk size from 320 ms to 160 ms hurts performance — the model benefits from having more audio context per prediction step, particularly near speech boundaries where the signal is ambiguous. Second, pretraining the adapter on ASR data alone and freezing it during LLM fine-tuning also degrades performance. An ASR-trained adapter is optimized for "what word is this?" rather than "is the speech finished?", so it passes less useful information to the LLM for boundary detection. Training the adapter jointly with LoRA allows it to learn the temporal cues actually needed for endpoint detection.
Compute
It's refreshing to see resource usage explicitly reported. Training used 32 × NVIDIA A100 80GB GPUs (batch size 64, 1 epoch). Evaluation ran on a single NVIDIA A6000 48GB, with each prediction taking ~50 ms — sufficient for real-time use.
