v4 union split-section dictionary: both full dicts, zero trimming

Browse source

Format v4 (docs/FORMAT_V4.md): one union CHD MPHF over 226,791 stroke
keys from Plover (147,424) + Lapwing (114,885), sections distributed
across halves. Left blob 387KB (displacements log-class Huffman 59KB,
membership 57KB, fingerprints 113KB, conflicts 9.6KB, value-index
slice 157KB); right blob 515KB (strings front-coded+deflate 193KB,
value-index remainder 324KB, conflicts dup). Shared string table:
73,464 unique translations, Lapwing 93% subset of Plover.

- tools/compile_v4.py: encoder + full verification (all 262,309
  entries byte-exact through cold decode path; hard error if either
  blob exceeds budget — never trims)
- src/dict_v4.c/.h: decoder — canonical Huffman displacement decode,
  conflict binary search, FC block walk, 2-block LRU cache
- src/behavior_steno.c: Plover-style sliding longest-match with
  retrace replaces multi-stroke timeout + prefix checks
- src/split_dict.c/.h: protocol v4 — GET_STRING(string_id) /
  RESOLVE(slot, dict); decisions are 100% left-local, BLE only for
  translation text
- CMake/Kconfig: STENO_DICT_BOTH default, one compiler run emits both
  half blobs; legacy MPHF path kept behind STENO_DICT_V4=n
- Host round-trip test: 42,000 vectors left→right split path,
  0 mismatches (tests/test_dict_v4.c)

...

This commit is contained in:

afiqzudinhadi 2026-07-03 01:28:31 +08:00

parent dd593c1640

commit 9bf21c9db2

12 changed files with 3338 additions and 666 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download patch file Download diff file Expand all files Collapse all files

1353

tools/compile_v4.py Normal file

View file

File diff suppressed because it is too large Load diff