[Paper Review]ParaS2S

Sujin Koo·2026년 7월 26일

ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction

Speech LLM의 평가 방법을 참고하기 위해 관련 연구를 계속 찾아보고 있다. 오늘 리뷰할 ParaS2S는 S2S 모델이 내용뿐 아니라 감정, 말투, 화자 특성 같은 준언어적(paralinguistic) 정보에도 맞게 응답하는지를 평가하고, 이를 강화학습으로 align하는 논문이다.

이전에 설계하고 진행 중이던 방향과 상당 부분 겹치는 내용이었으며, 아래의 AudioReasoner를 활용한 평가 방식을 참고해서 테스트 해보려 한다.

음성을 평가할 때는 Emotion2Vec 같은 이산적인 감정 라벨보다, AudioReasoner처럼 음성의 톤과 말투를 이해하는 모델을 사용하는 편이 사람 평가와 더 높은 상관관계를 보인다고 주장한다 (표 2 참고).
응답에 감정 태그를 강제로 부여한 경우, 오히려 아무런 출력 스타일 정보도 주지 않은 경우보다 사람 평가와의 상관관계가 낮았다.

리뷰를 영어로 쓸 때도, 한국어로 쓸 때도 있는데 이번에는 영어로 작성했다. 아래는 영어 리뷰다.

TL;DR

Existing S2S models (Qwen2.5-Omni, GLM-4-Voice, GPT-4o Voice Mode, Kimi-Audio) largely fail to adapt their responses to speaking style. They infer emotional state from text content alone and produce "tone-deaf" replies.

ParaS2S is a framework for building speech-to-speech (S2S) models that respond appropriately to paralinguistic cues (emotion, sarcasm, age, gender) not just spoken content. It has two parts:

ParaS2SBench: a benchmark + automatic judge that evaluates whether an S2S model's response (both content and speaking style) naturally fits the input speech.
ParaS2SAlign: an RL pipeline (GRPO) that distills the automatic judge into a fast reward model, then uses it to post-train an S2S model (Kimi-Audio) - beating SFT by ~10%.

1. Motivation

Speech carries content and paralinguistic signal (tone, emotion, speaker traits) that should shape how an assistant responds.
Existing benchmarks (VoiceBench, AIR-Bench, StyleTalk, VoxDialogue, etc.) either ignore paralinguistics entirely or stop at the text level - evaluating the response transcript rather than the response audio.
No prior benchmark directly evaluates output speech for paralinguistic appropriateness. ParaS2SBench is the first to do this end-to-end (audio in → audio out).
Data scarcity is a real constraint: collecting style-annotated, expressive input/output speech pairs is expensive, which motivates using RL instead of large-scale SFT.

2. ParaS2SBench

Design principles

Contrastive speaking styles - the same sentence is spoken in two opposite styles (e.g., "I just bumped into my ex" → surprised vs. sad), forcing the model to rely on voice rather than text.
Scenario-controlled, textually neutral queries - content is paralinguistically relevant, but doesn't give away the speaker's state through words alone.
Speech-to-speech evaluation - judged entirely on the output audio, not a text transcript.

Dataset

Synthetic speech: LLM-authored queries + TTS (gpt-4o-mini-tts, CosyVoice), filtered for neutrality/reasonability/paralinguistic relevance, then human-verified.
Real speech: drawn from IEMOCAP and MELD (emotion-labeled dialogue corpora).
Total size: ~2,690 prompts / ~3,890 utterances across Emotion, Sarcasm, Age, and Gender categories.

Automatic judge pipeline

The judge does not feed raw audio directly into an end-to-end audio-LLM. The authors found that LLMs like gpt-audio tend to hallucinate paralinguistic cues from spoken content rather than actually listening to the voice. Instead, the pipeline separates audio into text-based representations first:

User speech
 ├─ Whisper-V3             → transcription
 └─ Style analyzers         → emotion, sarcasm, age, gender
     (Emotion2Vec-based, fine-tuned with "PolyTone" training)

Response speech
 ├─ Whisper-V3             → transcription
 └─ AudioReasoner (Qwen-Audio 2 + CoT distilled from Gemini) → tone description
                ↓
        Text LLM (GPT-4.1) → 1–5 Continuation Fitness score

PolyTone training: emotion/sarcasm classifiers are fine-tuned on data where "the same sentence is spoken in many different styles", forcing the classifier to key off actual vocal cues rather than inferring style from lexical content (avoiding the same hallucination problem seen in LLMs).
Age/Gender classifiers: off-the-shelf pretrained models (wav2vec2 age/gender, ECAPA-TDNN gender). No custom training needed.
Final scoring: a 1~5 Likert "Continuation Fitness" score from a text LLM, guided by a human-written rubric. It judges whether the combination of response content and tone is an appropriate continuation of the dialogue - not merely whether an isolated target emotion is matched.

Judge validation

The paper runs an ablation across six judge configurations (O1-O6) to isolate exactly which components of the pipeline matter:

What each comparison isolates:

O1 vs. O2 vs. O3 vs. O4 : the effect of how output style is represented (nothing / tone only / emotion tag only / both), with input fixed at ground truth.
O5 vs. O6 : the effect of PolyTone training on the input-side style classifiers, with output representation fixed (tone description only).
O2 vs. O5 : the effect of moving from ground-truth input labels to model-predicted (classifier) input labels.

The standout result (O3 vs. O4):

O3 (emotion tag only, no tone description): 0.731
O4 (emotion tag + tone description): 0.743
O1 (no output style signal at all): 0.757

O1 outperforming O3: forcibly attaching a discrete emotion tag to the response performs worse than giving no output style information at all. This is direct empirical evidence that "emotion" and "tone" are fundamentally different concepts.

3. ParaS2SAlign

Base model: Kimi-Audio (chosen for its open speech tokenizer and strong intelligence). Three stages:

Warm-up SFT: an LLM generates ideal (content, tone) response pairs, synthesized via gpt-4o-mini-tts, used for ~10–100 hours of SFT. This step is necessary because untrained S2S models produce uniformly bad (tone-deaf) samples, giving RL nothing to learn from.
Reward model distillation: sample diverse responses from the warmed-up model, score them with the full judge pipeline (slow but accurate), then distill into a fast reward model (Qwen2.5-Omni + LoRA) for online scoring.
RL post-training with GRPO: optimize the SFT model against the distilled reward model, with a KL penalty (β = 0.2) against the reference policy to prevent catastrophic forgetting of general capabilities.

4. Key Results

Existing S2S models are "tone-deaf": GPT-4o Voice, Qwen2.5-Omni, GLM-4-Voice, and Kimi-Audio all perform close to a naive baseline that ignores speaking style entirely
RL > SFT: GRPO improves ~10% over SFT on ParaS2SBench (both objective judge and human eval, +7.6%)
RL is far more data-efficient: an RL model with only 20 hours of SFT warm-up data beats a pure-SFT model trained on the full 100 hours
Original capabilities preserved: VoiceBench scores stay roughly flat post-RL (thanks to light SFT + KL-regularized GRPO); heavy pure-SFT (>10 epochs) degrades general ability (e.g., collapses to short responses)
GRPO hyperparameters: group size ≥ 8 needed for a usable learning signal; batch size 32 is sufficient; KL weight β = 0.2 is the sweet spot between forgetting and under-exploration

5. Contributions

ParaS2SBench is the first end-to-end (audio-in/audio-out) paralinguistic-aware S2S benchmark.
A pipeline-based automatic judge (separating content and style analysis) that outperforms end-to-end LLM judging.
Shows that RL is more label-efficient than SFT for this problem.
Plan to open-source the data, code, and models.