2026-09-23·by Sijie Wang#lucerna#software#architecture

all-forms-underline

All-forms underline (mark one, light up every inflection)

Parent: key-designs

The distinctive reading feature: save one inflection of a word, and every inflected form of its lemma is underlined throughout the book. (Internally "Task #2".)

Form-sets. services/formMatching.ts builds a lemma's match set: entryFormSet = the LLM/dict-harvested forms ∪ the surface word ∪ the lemma; normalizeForms lowercases/trims and drops multi-token surfaces (so "le château" can't match a bare page token, but "château" still does; "ils/elles méritent" can't spuriously match). buildMarkedWordsMap (services/displayUtils.ts) then keys a map by every form, and the reader underlines a page token when markedWords.get(token.toLowerCase()) hits (ClickableText.tsx).

The tokenizer has to agree with everything else. services/tokenizationService.ts is a Unicode-aware splitter (Latin punctuation; CJK / Hiragana / Katakana / Hangul each their own class, per-char for CJK) with a French elision merge pass that re-joins l', d', j', qu' + the following word into one token — matching the tokenization rules baked into the LLM prompt. This matters because token boundaries must be identical across three consumers: the underline matcher, POS coloring, and the LLM word-analysis. A disagreement would underline the wrong span or mis-color.

German separable verbs get special handling here too (services/separableVerbService.ts heuristic on the client), and properly in the Python engine (linguistic-two-engines).

Why it's load-bearing, not cosmetic: the underline is the feedback surface of the whole vocabulary-loop — it's what makes saved words visible while reading, and it depends on form-sets + a tokenizer that agrees with the linguistic layer. Get the tokenizer wrong and the core feature silently mis-paints.

about this entry

One of sijie's wiki entries. The AI on this site is grounded in the same corpus and answers in sijie's voice, with citations back to entries like this one — answering costs sijie money, so it waits behind a code: enter an access code →

all-forms-underline