Parent: key-designs
Translation and language-model work for Lucerna, collected in one place: what language something is, what it means, how hard it is, and how it gets translated. This is a group node — each section states the decision and the current state; details live in the linked nodes and files. Split out of high-availability-proposal on 2026-10-28, where the translation record used to live in the nlp-api row of the status table.
M2M translation model (pilot)
Pilot decision (2026-10-13): M2M-100 418M, served as CTranslate2 int8 on CPU; NLLB stays on disk and remains switchable at zero cost. Benchmark (2026-10-10, FLORES-200 devtest, 200 sentences per direction): into English, NLLB wins big; into Chinese, M2M wins; NLLB-distilled-600M truncates early into Chinese. License: M2M-100 is MIT (verified 2026-10-21 against the model card, card text, and fairseq LICENSE; 418M and 1.2B) — a pruned build may enter the paid version.
Packaging — split and combinable
Decision (2026-10-28): translation is packaged per target language. The Chinese package covers all 12 source languages into Chinese; the English package covers all 12 into English. Identity pairs (zh→zh, en→en) need no package. The two packages combine into one: a single shared transformer body plus the union vocabulary — 12 source languages, each able to translate into Chinese or English.
Why combining costs less than the sum: the body is identical in both packages (measured from the shipped file, 2026-10-28: full M2M-100 int8 is 468 MiB, body ≈ 343 MiB, shared embedding ≈ 125 MiB). Two separate packages ship the body twice; the combined package ships it once. The vocabularies overlap almost completely: both packages need all 12 source languages' tokens, and each target's tokens are the other package's source tokens.
Sizes on that measured basis (per-package actuals pending the pruning experiment): one target package ≈ 370–400 MB; two separate ≈ 740 MB; combined ≈ 370–400 MB — about half. Floor: vocab pruning alone cannot go below the body (≈ 343 MiB), so the earlier ~200 MB expectation does not follow from vocab pruning alone; reaching it would need body compression (layer pruning, int4, or distillation).
fastText voice selection
Mechanism (spec v1, 2026-10-26): when the reader taps a word in a cross-language book, the voice must follow the word, not the book — a Japanese sentence quoted in an English book reads in Japanese. The mechanism selects a pronunciation language per tapped word, and must honestly declare don't-know instead of silently falling back. Three layers: hard signals from the text itself (kana, Hangul, Cyrillic); a sentence-level model when the sentence carries enough information (0.5 is only a noise filter, not a don't-know detector); a homograph layer that computes ambiguity from the input itself when there is no sentence context. Gate: CLEAR 292/294, AMBIGUOUS 83/83 declared. A Unihan-scale homograph table (v1: 37 forms) is pending extension. This mechanism is the selection layer, not the engine: the hard gate for any TTS engine is fine-grained control of logical stress and breath pauses (decided 2026-10-09) — an engine that only sounds right, without that control, is out.
Idiom and collocation
Rule (decided 2026-10-26): recognition is rule matching (phrase table); level is a model judgement on the same scale as words. The phrase dictionary stands separate from the word dictionary; recognition runs before lookup. Translations by Doubao, all directions, 5-pass agent proofread per row, no human review. Collection complete 2026-10-28: 12 languages, 2,326 rows. English pilot: trainable 282 rows (226 train / 56 held-out); round 1 verdict OK with reservations (phrase held-out accuracy 0.3214 vs majority baseline 0.2679; word gold no regression — accuracy 0.4286→0.4483, macro F1 0.3495→0.3801; phrase L3 recall 0; 424 rows without a true level are the bottleneck — no fabricated levels). v7 remains the baseline; the 11-language expansion is unlocked.
BERT annotation layer
The wider BERT line, of which grading is one piece: word-sense disambiguation, difficulty labelling (grading), in-book indexing, named-entity extraction, plus a register line. State: grading v7 is the baseline (frozen gold n=1,064: accuracy 0.4286, macro F1 0.3495, within-1 0.8947, MAE 0.700; zh_tw collapse fixed; L5 a dead class, spun off as its own mechanism); register v4 delivered (ONNX). Light models: MobileBERT failed as a v7 replacement (accuracy 0.3383); Distil-mBERT passes the 12-language gate (accuracy 0.3938) but sits below v7. Not advanced, still in scope: word-sense disambiguation, in-book indexing, named-entity extraction. Hard gate for every small model: all 12 languages — de, en, es, fr, it, ja, ko, nl, pt, ru, zh, zh_tw.
Current experiment: 12-language pruning
Running since 2026-10-28. Pruning and packaging run locally; post-pruning correctness evaluation (FLORES BLEU/chrF against the full model) runs on Kaggle GPU. Delivers per-direction results plus the union analysis: target-package sizes and the combined-package size against the sum of separate packages. Results pending.
Transfer: when a new language-model question comes up — a new model, a new selection mechanism, a new evaluation — it lands here first as a section, and graduates to its own node only when it grows its own record. The grouping is by function (translation and language), not by model family: M2M, fastText, and BERT sit together because they answer the same user question — what language is this, and how does it read.