v4 union split-section dictionary: both full dicts, zero trimming

Format v4 (docs/FORMAT_V4.md): one union CHD MPHF over 226,791 stroke
keys from Plover (147,424) + Lapwing (114,885), sections distributed
across halves. Left blob 387KB (displacements log-class Huffman 59KB,
membership 57KB, fingerprints 113KB, conflicts 9.6KB, value-index
slice 157KB); right blob 515KB (strings front-coded+deflate 193KB,
value-index remainder 324KB, conflicts dup). Shared string table:
73,464 unique translations, Lapwing 93% subset of Plover.

- tools/compile_v4.py: encoder + full verification (all 262,309
  entries byte-exact through cold decode path; hard error if either
  blob exceeds budget — never trims)
- src/dict_v4.c/.h: decoder — canonical Huffman displacement decode,
  conflict binary search, FC block walk, 2-block LRU cache
- src/behavior_steno.c: Plover-style sliding longest-match with
  retrace replaces multi-stroke timeout + prefix checks
- src/split_dict.c/.h: protocol v4 — GET_STRING(string_id) /
  RESOLVE(slot, dict); decisions are 100% left-local, BLE only for
  translation text
- CMake/Kconfig: STENO_DICT_BOTH default, one compiler run emits both
  half blobs; legacy MPHF path kept behind STENO_DICT_V4=n
- Host round-trip test: 42,000 vectors left→right split path,
  0 mismatches (tests/test_dict_v4.c)
This commit is contained in:
afiqzudinhadi 2026-07-03 01:28:31 +08:00
parent dd593c1640
commit 9bf21c9db2
12 changed files with 3338 additions and 666 deletions

74
Kconfig
View file

@ -11,29 +11,43 @@ if STENO_ENGINE
choice STENO_DICT
prompt "Steno dictionary"
default STENO_DICT_PLOVER
default STENO_DICT_BOTH
config STENO_DICT_BOTH
bool "Plover + Lapwing"
help
Compile both full dictionaries into the v4 union format.
The active dictionary is selected at runtime.
config STENO_DICT_PLOVER
bool "Plover main dictionary (MPHF compressed)"
bool "Plover main dictionary"
help
Use Plover main.json via MPHF compression.
Compile only the Plover main dictionary.
config STENO_DICT_LAPWING
bool "Lapwing dictionary (MPHF compressed)"
config STENO_DICT_TEST
bool "Test dictionary (46 entries, simple format)"
bool "Lapwing dictionary"
help
Small built-in test dictionary for development.
Compile only the Lapwing dictionary.
endchoice
config STENO_DICT_MPHF
bool
default y if STENO_DICT_PLOVER || STENO_DICT_LAPWING
config STENO_DICT_V4
bool "v4 union split-section dictionary format"
default y
select ZLIB
help
Use MPHF (minimal perfect hash) dictionary format.
Compile the dictionary into the v4 union split-section format
(docs/FORMAT_V4.md): two half blobs, left (central) holds the
lookup structures, right (peripheral) holds the string table.
Requires STENO_SPLIT_DICT. Disable to fall back to the legacy
single-dictionary MPHF format.
config STENO_DICT_MPHF
bool
default y if !STENO_DICT_V4
select ZLIB
help
Legacy MPHF (minimal perfect hash) dictionary format.
Selects ZLIB for block-compressed string table decompression.
config STENO_CUSTOM_KEYMAP
@ -75,39 +89,43 @@ config STENO_MULTI_STROKE_TIMEOUT_MS
config STENO_DICT_MAX_SIZE
int "Max dictionary binary size (bytes)"
depends on !STENO_DICT_V4
default 473088
help
Max compiled dict size. 473088 = 462KB.
Build fails if dict exceeds this — no trimming.
Max compiled dict size for legacy MPHF single-blob builds.
473088 = 462KB. Build FAILS if the dict exceeds this —
entries are never trimmed.
menuconfig STENO_SPLIT_DICT
bool "Split dictionary across both halves"
default n
select STENO_DICT_MPHF
help
Partition the steno dictionary across both keyboard halves.
Left (central) gets highest-importance entries for zero-latency
local lookup. Right (peripheral) gets remaining entries, queried
over BLE on local miss. Both halves have their own MPHF dict.
Store the steno dictionary across both keyboard halves.
v4 format: left (central) holds the lookup structures
(displacements, membership, fingerprints, conflicts, part of
the value index); right (peripheral) holds the string table,
queried over BLE. Legacy MPHF format: entries partitioned by
importance, right half queried over BLE on local miss.
Build FAILS if either half exceeds its budget — entries are
never trimmed.
if STENO_SPLIT_DICT
config STENO_DICT_LEFT_MAX_SIZE
int "Left half dict budget (bytes)"
default 153600
default 440320
help
Flash budget for left (central) partition. Default 150KB.
Central has more code overhead (behavior, formatter, output,
undo, BLE client) so gets a smaller dict partition.
Highest-importance entries fill this first.
Flash budget for the left (central) half blob. Default
440320 = 430KB. Build FAILS if the compiled blob exceeds
this — entries are never trimmed.
config STENO_DICT_RIGHT_MAX_SIZE
int "Right half dict budget (bytes)"
default 512000
default 527360
help
Flash budget for right (peripheral) partition. Default 500KB.
Peripheral has less code overhead so gets the bulk of the dict.
Remaining entries after left partition is filled.
Flash budget for the right (peripheral) half blob. Default
527360 = 515KB. Build FAILS if the compiled blob exceeds
this — entries are never trimmed.
config STENO_SPLIT_CACHE_SIZE
int "LRU cache entries on central side"