Commit graph

24 commits

Author SHA1 Message Date
1447b82cce README: expand how-it-works with compression + architecture details 2026-07-04 15:36:12 +08:00
ebabd48def README: move numpy note to dev section, not a user requirement 2026-07-04 14:59:39 +08:00
936de36255 Demote debug lookup logs; rewrite README for v4
README: requirements (2x nRF52840 split, single-MCU unsupported),
usage, flash-budget ceiling, bootloader version check / update /
revert guide with official links, macOS raw-disk flashing notes.
2026-07-04 14:06:20 +08:00
23439a52f7 Parse Plover number strokes: digits = number bar + positional key
parse_stroke silently dropped digit characters, so number strokes
collapsed onto their letter-only bitmasks: "12K" and "K" both parsed
to K → first-wins dedup let "12K"→"12:00" clobber "K"→"can" (and 297
more core entries: -R→are eaten by 12-R→XII, -S→{^s} by 2-S→2s...).
Digits now map per Plover: 1=S- 2=T- 3=P- 4=H- 5=A- 0=O- 6=-F 7=-P
8=-L 9=-T, all implying #. Collisions: 298 → 0; 300 union keys and
161 translations recovered. Verified: all 262,309 entries byte-exact,
42,000-vector host round trip 0 mismatches.
2026-07-03 20:08:45 +08:00
2e7853aad3 Debug: log lookup decisions + transport results 2026-07-03 15:50:58 +08:00
c0756e6428 Peripheral: handle dict requests on a dedicated worker thread
Inflating a 16KB string block inside the BT RX thread overflowed its
stack and froze the peripheral on the first steno stroke. The GATT
write callback now only copies the request (<= 7 bytes) and submits
to a 4KB-stack work queue; the worker resolves and notifies. Single
request in flight, busy requests dropped (central times out and
retries). GATT service + handlers now compile on the peripheral
only. Inflate Huffman tables made static (single-context) to cut
per-call stack use ~2KB. Host round-trip re-verified: 42,000
vectors, 0 mismatches.
2026-07-03 15:27:27 +08:00
728cdaf432 Defer steno dict discovery until split link is encrypted + settled
Immediate discovery in the connected callback interleaved with ZMK's
own split pairing/discovery on the same link and broke half pairing.
Discovery now waits for security_changed (>= L2) plus a 2s settle
delay, and retries on failure. No GATT traffic from the steno client
before the link is ready.
2026-07-03 15:11:53 +08:00
dcbadb29f4 Wire the split-dict BLE client: discovery, subscribe, conn tracking
split_conn was never set and GATT writes targeted handle 0 — every
remote string fetch returned -ENOTCONN, so all strokes fell back to
raw steno. Central now: tracks the peripheral link via connection
callbacks (role filter), discovers the steno service / request
characteristic / CCC, subscribes for notifications, and writes to
the discovered handle. Peripheral characteristic gains
WRITE_WITHOUT_RESP. split_dict self-initializes via SYS_INIT.
2026-07-03 14:49:02 +08:00
65fde6d298 Own DEFLATE decoder replaces zlib dependency
Zephyr bundles no zlib — the CONFIG_ZLIB select was a no-op and the
string path returned errors on device. src/inflate.c: minimal raw
DEFLATE (RFC 1951) decompressor, no allocation; one code path for
host and Zephyr. Host round-trip: 42,000 vectors through the real
compiled blobs, 0 mismatches.
2026-07-03 11:37:19 +08:00
404cd2a334 Debug bisect toggles: STENO_DEBUG_NO_GATT, STENO_DEBUG_NO_EMBED 2026-07-03 11:02:34 +08:00
9bf21c9db2 v4 union split-section dictionary: both full dicts, zero trimming
Format v4 (docs/FORMAT_V4.md): one union CHD MPHF over 226,791 stroke
keys from Plover (147,424) + Lapwing (114,885), sections distributed
across halves. Left blob 387KB (displacements log-class Huffman 59KB,
membership 57KB, fingerprints 113KB, conflicts 9.6KB, value-index
slice 157KB); right blob 515KB (strings front-coded+deflate 193KB,
value-index remainder 324KB, conflicts dup). Shared string table:
73,464 unique translations, Lapwing 93% subset of Plover.

- tools/compile_v4.py: encoder + full verification (all 262,309
  entries byte-exact through cold decode path; hard error if either
  blob exceeds budget — never trims)
- src/dict_v4.c/.h: decoder — canonical Huffman displacement decode,
  conflict binary search, FC block walk, 2-block LRU cache
- src/behavior_steno.c: Plover-style sliding longest-match with
  retrace replaces multi-stroke timeout + prefix checks
- src/split_dict.c/.h: protocol v4 — GET_STRING(string_id) /
  RESOLVE(slot, dict); decisions are 100% left-local, BLE only for
  translation text
- CMake/Kconfig: STENO_DICT_BOTH default, one compiler run emits both
  half blobs; legacy MPHF path kept behind STENO_DICT_V4=n
- Host round-trip test: 42,000 vectors left→right split path,
  0 mismatches (tests/test_dict_v4.c)
2026-07-03 01:28:31 +08:00
dd593c1640 Fix Lapwing dict URL: moved to plover-lapwing-aio repo 2026-07-03 00:40:09 +08:00
1d5bef3129 Revert to v2 format, fix CHD with more displacement tries
Spare slots (1.23x) doubled binary size → reverted to v2 (no spare
slots, entry_count == slot_count). CHD now uses entry_count/4
buckets and 1M displacement tries instead of 65K. No trimming.
2026-07-02 15:47:23 +08:00
ab0e90b84b Fix: restore string table + prefix build removed with trim loop 2026-07-02 15:42:23 +08:00
28e0c1a49f Never trim dictionary — hard error if over budget
Removed all trimming/auto-trim logic. Build fails if compiled dict
exceeds flash budget. CHD now uses spare slots (1.23x load factor)
and fewer buckets (entry_count/5) for reliable construction at any
entry count. Binary format bumped to v3: header stores slot_count
and entry_count separately.
2026-07-02 15:39:52 +08:00
1339ec4525 Fix partition + CHD: empirical bytes/entry, 1:1 bucket ratio
Partition: ratio-based split overloaded right side (113K entries →
875KB estimate → heavy trimming to 58K). Now uses 4.5 bytes/entry
from benchmark data → ~34K left, ~113K right, both fit budgets.

CHD: 1:1 bucket ratio (entry_count buckets) prevents hash collision
failures on large partitions. Most buckets have 1 key → trivial
placement. Displacement array grows but still small.
2026-07-02 15:29:25 +08:00
99bf19180e Fix CHD MPHF: increase bucket ratio for large partitions
bucket_count = entry_count/3 caused CHD construction failure on
~113K entry right partition (bucket too crowded → displacement
search exhausted 65536 tries). entry_count/2 gives more buckets,
trades ~19KB displacement overhead for reliable construction.
2026-07-02 15:23:25 +08:00
0908dc7c85 Right-heavy dict partition: 150KB left, 500KB right
Central has more code overhead (behavior, formatter, output, undo,
BLE client) → smaller dict budget. Peripheral is lean → gets bulk
of dict. Ratio-based partition replaces fixed bytes-per-entry estimate.

Full 147K Plover dict, zero trimming target.
2026-07-02 14:30:12 +08:00
20218fa2ab True split-dict: partition dictionary across both halves
Importance-based partitioning — highest-importance entries on left
(central) for zero-latency local lookup, remainder on right
(peripheral) queried over BLE on miss. Both halves embed their own
MPHF binary. Configurable block size for tighter compression.
2026-07-02 14:27:37 +08:00
9d2d5e4e2d Fix: generate steno_dict_path.h for dict embed 2026-07-02 09:17:05 +08:00
bc2e40caf4 Merge split-storage: BLE GATT dict protocol + split-aware build
Combined optimize-dict (MPHF compression, block-compressed strings,
formatter, undo, Unicode output) with split-storage (BLE GATT service,
LRU cache, peripheral-side dict embed).

Split-dict mode: central queries peripheral over BLE.
Non-split mode: dict embedded on central, local lookup.
2026-07-02 08:57:32 +08:00
2af45eafce Steno engine with BLE split-storage architecture
Full steno engine: chord detection, MPHF/trie dict lookup, Plover
formatter (18 commands), undo (ring buffer), HID + Unicode output.

Split-storage specific:
- BLE GATT service for dict queries (query/prefix/batch)
- LRU cache on central side (CONFIG_STENO_SPLIT_CACHE_SIZE)
- Dict embedded on peripheral, engine runs on central
- 3-way dispatch: split_dict / MPHF / simple trie

Build system: auto-fetch Plover dict, MPHF compiler with block
compression (~56K entries in 420KB), CMake integration.
2026-07-02 02:06:37 +08:00
1ec204b845 Scaffolding: ZMK module structure, Kconfig, DTS, behavior driver
- zephyr/module.yml with dts_root for dt-bindings
- Kconfig: STENO_ENGINE, dict selection (Plover/Lapwing/test),
  MPHF toggle, Unicode modes, history size, multi-stroke timeout
- DTS behavior binding (one_param, steno key index)
- behavior_steno.c: chord accumulation, all-up detection,
  multi-stroke buffering with timeout, star undo, formatter pipeline,
  3-way dict dispatch (split/MPHF/simple trie)
- dict_embed.S: .incbin from generated header path
- steno_keys.h: 23-key layout, bit positions matching compiler
2026-07-02 02:04:28 +08:00
234bc8d731 Initial project: benchmarks, status doc, gitignore 2026-06-24 16:51:44 +08:00