Building Low-Latency Voice-to-Text on Mac: Whisper and Cloudflare Workers in Production

Voice-to-text lives or dies on latency. If the transcript shows up half a second after you stop talking, the tool feels instant and you keep using it. If it shows up two seconds later, you go back to the keyboard. Accuracy is the solved part: Whisper-class models already handle it. The engineering problem is shipping that accuracy back to the user fast enough that the wait disappears.

This is a writeup of how Lispr does it on macOS: a free dictation app with a 3.67 MB client and a median end-to-end latency of 346 ms, built in about two months by a small cross-functional team. The short version is a thin native client, an edge proxy on Cloudflare Workers, and a hosted Whisper large-v3-turbo endpoint doing the transcription.

The architecture in one pass

The pipeline has three hops. The Mac client captures audio while you hold a key. It streams that audio to a Cloudflare Worker at the edge, which handles auth, rate limiting, and forwarding. The Worker passes the audio to a hosted Whisper large-v3-turbo endpoint, which returns the transcript. The text travels back the same path and lands at your cursor.

The decision that shapes everything else is keeping the model off the device. Running Whisper locally means a multi-gigabyte model download, an Apple Silicon requirement for usable speed, and a cold start every time the app launches. Moving inference to a hosted endpoint keeps the client at 3.67 MB, lets it run on macOS 11 and older Intel Macs, and turns the user's machine into a thin capture-and-insert layer. The cost is a hard dependency on the network, which is the real tradeoff of this design.

The client

The macOS client is a small native app that lives in the menu bar and the Dock. It does four things: capture audio on a held key, ship it out, receive text, and insert that text at the cursor in whatever app has focus.

Push-to-talk is a deliberate choice over toggle activation. Holding a key means the recording session ends the moment you release, so there is no silence-timeout heuristic to tune and no way to leave dictation running by accident. Insertion uses the macOS accessibility APIs, and the client restores whatever was on the clipboard before it writes, so dictating does not clobber what you copied earlier. The trigger key is configurable, and none of this needs a large binary, which is part of why it stays at 3.67 MB.

The latency budget

End-to-end latency is the number that matters, and it breaks into capture, network round trip, inference, and insertion. The production figures:
The median of 346 ms is what a warm path delivers. The large-v3-turbo variant earns its place here: its pruned decoder runs faster than full large-v3 at close to the same accuracy, and inference is the largest single slice of the budget. The edge Worker keeps the network hop short by terminating close to the user and forwarding over a warm connection to the inference endpoint.

The cold start is the asterisk. After more than a minute idle, the first request runs around 900 ms while the path warms back up, then settles into the median range for everything after. Hiding that spike would mean keeping resources warm at all times, which a free product cannot justify, so the team accepts one slow request after a pause rather than paying to keep the path hot.

Privacy as an architecture constraint

Zero audio retention is a product promise that has to be enforced in the architecture, not bolted on. Audio is encrypted in transit, transcribed, and discarded. Nothing is written to disk on the server side, and there is no account or profile to attach a recording to, because the app never creates one.

This shapes the system in concrete ways. The Worker does not log request bodies. The inference path is stateless, so a transcript exists only long enough to be returned. There is no history feature, because storing history would mean storing voice data.

Build it or use it

If you are building voice-to-text into your own product, the lesson from this stack is that the hard part is the latency budget and the warm path. The transcription itself is the easy input now. The work lives in the plumbing: a thin client, an edge proxy, a warm inference endpoint, and a clear position on cold starts.

If you would rather not build and maintain that yourself, Lispr is a ready Whisper alternative you can install and use for free today, and the full Lispr architecture deep-dive covers these decisions in more detail. On modern infrastructure, a small team can build production voice-to-text without a research lab, and the bar has moved from the model to the experience around it.