TTS audiobook + word-level karaoke (desktop, 100% local)

Parent: key-designs

Desktop-only. Design rationale (desktop/docs/TTS-PLAN.md): fully local, no cloud, no API keys; engine locked to Piper (VITS/ONNX) via sherpa-onnx (WASM — no Python, no native .node addon, to dodge macOS hardened-runtime signing). The engine ships bundled; the only opt-in download is the per-language voice pack (~60 MB) from R2.

Synthesis, process-isolated. main/tts/piper.ts loads one cached OfflineTts per voice (VITS config). The WASM runs in a long-lived utilityProcess child (main/tts/engine-runner.ts), one request at a time — because the same WASM crashes ("memory access out of bounds") inline in the busy main process but is rock-solid in a clean child. Voice download (voice-download.ts) streams to *.part, hashes in-stream via a pass-through Transform (a data listener would corrupt large files), md5-verifies, then atomic-renames; a whole-voice failure wipes the dir (no partial install).

Prebake, incremental. generateAudiobook.ts (renderer) folds a TTS_TEXT_VERSION into the content hash, builds a per-sentence work list + chapter map, and passes already-baked chapters as a resume set. prebake.ts (main) synthesizes one paragraph per engine call (natural prosody) with a 0.45 s inter-paragraph silence, wraps PCM into mono 16-bit WAV, and writes each chapter's row the instant its file lands — so the book is playable ('partial') mid-bake ("烤多少听多少") and cancel/restart resumes.

Word-level karaoke — the interesting part. Piper gives no alignment, so timing is derived from PCM energy: main/tts/word-cues.ts computes per-20 ms RMS voiced flags, trims silence, and distributes the word tokens across the voiced time weighted by word length — so the highlight follows real speech energy and holds through Piper's punctuation pauses, instead of a flat character rate. Critically, the tokens are not re-tokenized in main — they are the renderer's [data-word] span sequence (core tokenize+isWordToken, wordSpans.ts) shipped through IPC, so cues align 1:1 with on-screen spans.

The anchor trick (why it stays in sync). Each VTT cue is stamped with the word's book-absolute raw offset (r<rawOffset> identifier line), computed from each sentence's displayToRaw map. At playback, useReaderAudio.ts walks the page's [data-word] spans, maps each to its raw offset, and paintKaraoke toggles 3-state classes current/read/unread by matching the active cue's rawStart to a span. Because both the cue and the on-screen span are anchored to the same canonical raw offset (raw-display-cursor), they line up even across re-pagination/font changes — the same backbone that makes highlights layout-independent makes karaoke layout-independent. Plus follow-the-word page turning, double-click-to-seek, and auto-pause-on-lookup.

Fallback. TTSProviderRegistry resolves per-language override → default → WebSpeechProvider (window.speechSynthesis, always available) — which has no audio blob and instead builds WordTiming[] from native onboundary word events. That's the karaoke source when Piper isn't installed.