James Imbuido | Data Scientist & AI Engineer
← All AI projects
MLFeatured

Real-Time Voice Cloning

A voice-cloning studio in the browser — record a short sample, and the three-model SV2TTS pipeline (GE2E speaker encoder, Tacotron synthesizer, WaveRNN vocoder) converts it to an embedding and speaks any text you type in that voice. Every model runs client-side over WebAssembly, and the recording never leaves the device because there is no upload endpoint to send it to.

Problem

Real-Time Voice Cloning (CorentinJ's SV2TTS) is a well-known PyTorch stack, but it is not a browser stack: the Tacotron decoder is a data-dependent Python loop, the WaveRNN vocoder samples 16,000 times per second from a categorical distribution, and the checkpoints alone are ~440 MB. The problem was to get real voice cloning running entirely client-side — no server inference, no API key, and a privacy property that is architectural rather than promised: microphone audio must never leave the device, verifiable by the fact that no endpoint exists to receive it.

Approach

Port the seams, not the models. The PyTorch graphs were exported once along contract boundaries chosen so that every data-dependent decision (decoder loop, sampling, trimming) stays in typed TypeScript, and every numeric seam (mel, text, partial slicing) got a golden fixture before anything shipped. The feasibility gate ran before the pipeline: a six-graph browser smoke measured load, per-stage latency, and end-to-end time in a real Web Worker before any UI existed, and the FLOP budget predicted for the vocoder was proven 10–30× too pessimistic only by measuring. The chunked vocoder form was designed after re-reading the reference's generate() — per-sample conditioning rows, in-graph gumbel-max sampling with a seeded uniform — so the 64,000-calls-per-second JS loop became 320 runs per four seconds of audio. UI conventions follow the site's chess pattern: models lazy-loaded per stage from a self-hosted wasm runtime, every state machine exported from a plain module, and progress streamed stage-by-stage from the worker.

Models

Three pretrained checkpoints (HuggingFace CorentinJ/SV2TTS) converted to five fp32 ONNX graphs (~111 MB, lazy-loaded per stage): the GE2E speaker encoder (3-layer LSTM over 160-frame mel partials, ~83 ms for a 4-second sample), the Tacotron split into a one-shot text encoder and a single decoder-step cell with the checkpoint's reduction factor r=2 baked in, and the WaveRNN split into a parallel conditioning upsample plus a chunked form that unrolls all 200 WaveRNN steps of one mel frame in-graph with gumbel-max sampling — amortizing the JS↔wasm call overhead 200× and turning a predicted-certain failure into a passing gate. Everything the reference does outside the graphs lives in typed TypeScript mirrors: english_cleaners text front-end, librosa-exact mel spectrograms (both the encoder's power-mel and the synthesizer's dB-normalized path), partial-utterance slicing, the decoder loop with its >0.5 stop token and −3.4 dB trim, mu-law decoding, and the de-emphasis IIR.

Evaluation

Cross-language parity with golden fixtures is the backbone: a Python fixture generator stores 5-significant-digit golden values (text ids, full mel matrices, PyTorch embeddings, chained graph outputs) and a wasm-runtime verifier replays every stage through the exact backend the browser uses — 52/52 checks pass (text exact, mel worst relative error 3.7e-4, embedding cosine 1.000000 against PyTorch, vocoder codes exact on the argmax path). The fixtures earned their keep by catching two real bugs: a zero-padded FFT that silently evaluated the mel at interpolated rather than exact bin frequencies, and two Slaney mel-scale constant/sign errors that textbook descriptions get wrong. Performance was gated before any UI: in real Chrome the vocoder — the make-or-break stage, FLOP-budgeted at 10–30× over its actual speed — runs at 53 ms per mel frame, ~17 s per 4 seconds of audio against a 20 s gate, with the encoder at 83 ms; a full headless-Chrome drive of the shipped page measured 24.3 s end-to-end including the one-time ~110 MB model fetch. Compression was gated too: int8 dynamic quantization scrambles the LSTM encoder (cosine 0.38 vs fp32) and fp16 fails to convert the RNN graphs at all, so the measured decision was fp32 despite the size.

Results

Five fp32 ONNX graphs ship (~111 MB, largest single file 75 MB), loaded lazily only when the visitor records — nothing model-related is on the homepage or the initial page load. The full record → embed → synthesize flow works end-to-end in the browser with stage progress throughout, the result is a downloadable 16 kHz WAV, and re-synthesis is deterministic per (voice, text, seed). The privacy property is structural: audio capture runs through an AudioWorklet into a Web Worker, and the only network request the page makes — a one-time +2 credit reward for trying it — carries an empty body, so there is literally nothing the recording could travel in. Machine-side quality checks pass: output from the shipped browser path shows a natural speech envelope (voiced fraction 0.75, per-frame energy correlation with the requested mel 0.62–0.70, versus 0.47 for the Griffin-Lim fallback floor), verified over real reference-repo speakers as well as a live headless-Chrome drive. Honest edges: whether the output *sounds* right is a human judgment still pending a listening test, first synthesis is slow (~111 MB download + tens of seconds of wasm inference, cached thereafter), and Safari is unmeasured (WebKit wasm SIMD expected to track Chrome's numbers).

Lessons

Measure the budget before believing it. The naive FLOP estimate condemned the vocoder to 30–130 seconds per second of audio and a guaranteed fail; the actual measurement inverted that conclusion, and the real constraint turned out to be JS↔wasm call overhead, not arithmetic — reshaping the export around it bought a 200× amortization. The second lesson is that parity fixtures are where correctness lives in cross-language ports: the two mel bugs the fixtures caught (zero-padded FFT bins, Slaney log-step constants) were both 'obviously fine' implementations that no amount of code review would have caught, because the error was in what the spec meant rather than how it was coded. The third lesson is that fixtures can be vacuous: the shipped vocoder produced a DC ramp instead of speech while the parity gate passed 52/52, because the golden values were generated by the same broken graph — parity against a shared bug proves nothing, so output must also be checked against its physical meaning (a speech-sanity diagnostic over real speakers is what surfaced it). The bug chain itself was instructive: frozen GRU state, a clamp-before-unary-minus NaN that silently degraded argmax to class 0, and — the design error — one uniform per sample instead of per class, since adding a constant to all 512 logits can never change which one wins. And quantization is a measurement, not an optimization: every compression ladder here either destroyed accuracy (int8, cosine 0.38) or failed to convert (fp16), so shipping fp32 was the evidence-based answer even at 111 MB.

Technical Stack

  • PyTorch
  • ONNX
  • ONNX Runtime Web
  • Next.js
  • TypeScript
  • React
  • Web Audio API
  • AudioWorklet
  • Web Workers

Links