Digging Into the Moshi Model's Code

Today I took a closer look at how the Moshi model actually works under the hood, code-wise.

Moshi is fundamentally a model designed to listen and speak at the same time. Because of this, unlike typical TTS models, it has a rather unusual setup: the user stream and the Moshi stream are kept separate. Even during training, the backbone LLM — Helium (an RQ-Transformer) — needs to learn natural conversation, so it's trained on stereo audio rather than mono.

A bit of a tangent here: when I talk with AI, sometimes it's just me and the AI, but often it's me, a friend, and the AI together — for example, my friend and I chatting while occasionally throwing a question at the AI. One thing that's bothered me in these situations is that the AI tends to treat the conversation between me and my friend as if it were directed at itself, jumping in and interrupting our conversation or giving unnecessary responses. I suspect multi-speaker conversational models that account for this will eventually emerge. Current models still assume two speakers (largely a dataset limitation, I'd guess), and extending this to multiple speakers — or applying it to conversational robots — still leaves a lot of open problems. Even though the field has moved incredibly fast, it feels like there's still a lot left to solve.

Anyway, back to the main topic. Since Moshi takes in both the user stream and the Moshi stream, both inputs are fed together during training.

I drew this diagram together with a friend while we were trying to wrap our heads around Moshi. As you can see: the user's audio → Mimi tokens, Moshi's audio → Mimi tokens, and on top of that, Moshi has something called an "inner monologue" — essentially a built-in ASR capability. (It lets Moshi track what it's saying directly, without an external ASR, while also helping it generate better responses.) These three components are combined and fed as input into the Temporal Transformer. The output is an intermediate temporal-context latent, which is then used to predict the inner monologue, while the rest is passed as a condition into the Depth Transformer. The Depth Transformer then autoregressively predicts blank → Mimi semantic tokens → Mimi acoustic tokens, and this final output becomes Moshi's next response.

Honestly, the first time I read the Moshi paper, a lot of how it worked was confusing to me. But going through the code and tracing the dimensions of the inputs and outputs really helped the overall flow click into place.

Relevant code repositories:

• Moshi: https://github.com/kyutai-labs/moshi
• Moshi Finetuning: https://github.com/kyutai-labs/moshi-finetune
• Japanese Moshi Finetuning: https://github.com/nu-dialogue/moshi-finetune

One thing worth noting while digging through the code: only the training code for the RQ-Transformer is provided — the training code for the Mimi codec itself is not. That's probably why subsequent papers building on Mimi codec tend to just reuse the existing checkpoint as-is. (You could train your own, sure, but training on large-scale data is a real burden.)

An interesting detail is that in Japanese Moshi, the authors apparently found that reusing the Mimi checkpoint as-is already gave good enough performance, so they didn't bother retraining it for Japanese. From an acoustic standpoint, that makes sense since there's likely little language dependency. But from a semantic standpoint, I would have expected some language dependency that could hurt token-prediction performance when training Moshi later — so this result was a bit surprising. I'd guess this approach probably wouldn't hold up if you wanted to push performance to a truly production-ready level, but it's a useful reference paper to keep in mind when training a Korean version.