zmk-steno-engine

Clean-room stenography engine for ZMK Firmware. Carries the full Plover main dictionary AND the full Lapwing base dictionary — 262,309 entries, zero trimming — on a pair of nRF52840 keyboard halves using the v4 "union split-section" format

Status: working end to end on real hardware (corne, 2x nice!nano v2). Translations, multi-stroke longest-match with retrace, number strokes, runtime dictionary switching (Plover/Lapwing). Known issues tracked in the issue tracker; latency optimization in progress.

How it works

Neither dictionary fits on a single MCU (~700 KB raw translations alone, ~800 KB flash free per half after ZMK + peripherals). The engine solves this by:

1. Merging both dicts into one structure — a union CHD minimal perfect hash function (MPHF) indexes all 227K stroke keys once. Building two separate indexes would cost 1.8x more flash because 33K identical entries and the string table get duplicated.
2. Aggressive compression — translations are sorted, front-coded (common-prefix elimination), then DEFLATE-compressed in 16 KB blocks. Both dicts share the same string table (Lapwing is 93% a subset of Plover), storing all 73K unique translations in 193 KB (28% of raw). MPHF displacements are Huffman-coded (8 bits avg vs 19 fixed).
3. Splitting the structure across halves — sections of the single index are placed on whichever half has budget, not duplicated:

Half	Holds	Role
Left (central)	MPHF displacements, membership bitmap, fingerprints, conflict table, value-index slice	Every stroke decision is resolved locally — zero BLE latency for hit/miss/which-dict
Right (peripheral)	Compressed string table, value-index remainder, conflict dup	Serves translation text on demand over a custom BLE GATT service

4. Custom BLE protocol — left resolves the stroke to a string ID locally, then requests just the text bytes from the right half (one round trip, LRU-cached). The right half inflates and walks the front-coded block on a dedicated worker thread (16 KB buffer).
5. Own DEFLATE decoder — Zephyr ships no zlib; a minimal RFC 1951 decompressor (~300 lines, no allocation) runs on the peripheral.

Requirements

• A split keyboard with two nRF52840 controllers (e.g. corne with 2x nice!nano v2). The dictionary spans both halves — single-MCU keyboards are not supported yet.
• ZMK v0.3.x build environment (GitHub Actions user-config works)
• Adafruit nRF52 bootloader 0.9.0 or newer on BOTH halves — 0.11.0 recommended. See Bootloader below; old bootloaders silently corrupt large firmware images.

Usage

west.yml:

manifest:
  remotes:
    - name: afiqzudinhadi
      url-base: https://github.com/afiqzudinhadi
  projects:
    - name: zmk-steno-engine
      remote: afiqzudinhadi
      revision: main

.conf:

CONFIG_STENO_ENGINE=y
CONFIG_STENO_DICT_BOTH=y            # or STENO_DICT_PLOVER / STENO_DICT_LAPWING
CONFIG_STENO_SPLIT_DICT=y
CONFIG_STENO_DICT_LEFT_MAX_SIZE=440320
CONFIG_STENO_DICT_RIGHT_MAX_SIZE=519168
CONFIG_STENO_SPLIT_TIMEOUT_MS=250

The build FAILS if either half's blob exceeds its budget — entries are never trimmed.

Budget recommendations (nRF52840, Adafruit bootloader 0.11.0, app ceiling 0xEA000 = 802,816 bytes):

Config	Left	Right	Notes
Both dicts (Plover + Lapwing)	440320 (430 KB)	519168 (507 KB)	Tight — little room for extra modules
Plover only	440320	519168	Comfortable — ~35 KB slack per half
Lapwing only	440320	519168	Comfortable

Left budget is smaller because the central half carries more firmware code (behavior driver, formatter, output, undo, BLE client, cache) — leaving less free flash for dictionary data. The peripheral only runs the GATT server + inflate decoder, so it gets the larger share.

Lower these if you add other flash-heavy modules (large RGB animations, display assets, etc.) and the linker overflows. The compiler shifts the value-index split point to fill the right half first; left gets the remainder. If either budget is too small for the compiled dict, the build errors out with the exact overshoot.

Keymap: include the behavior and bind the 23 steno keys (see include/dt-bindings/zmk/steno_keys.h for all key names):

#include <behaviors/steno_engine.dtsi>
#include <dt-bindings/zmk/steno_keys.h>

// example row: &steno STENO_NUM  &steno STENO_SL  &steno STENO_TL ...

Digits in dictionary strokes (e.g. 12K → 12:00) map to the number bar plus their positional key, exactly like Plover.

Flash budget

The app image on each half must stay below the bootloader's app-region ceiling (0xEA000, i.e. 802,816 bytes with Adafruit 0.11.0). Firmware blocks above the ceiling are silently dropped and the half will not boot. The default budgets above keep both halves under it — check the FLASH line in your build log if you change them.

Bootloader

The nice!nano ships with various versions of the Adafruit nRF52 bootloader. Versions from ~2021 (e.g. 0.6.0) corrupt large UF2 flashes — blocks go missing mid-image and the firmware crash-loops with no display, no typing, no USB. This module's images are large (~1.5 MB UF2 per half), so a current bootloader is mandatory.

Check your version

Double-tap the reset button. The half mounts as a NICENANO USB drive; open INFO_UF2.TXT on it:

UF2 Bootloader 0.6.0 ...   ← too old
UF2 Bootloader 0.11.0 ...  ← good

(On newer macOS the volume may not auto-mount — run diskutil list and diskutil mount diskN.)

Update

1. Download the updater for your board from the official releases — for nice!nano: update-nice_nano_bootloader-<version>_nosd.uf2 (the nosd variant keeps the installed SoftDevice, which ZMK needs).
2. Enter the bootloader (double-tap reset) and copy the updater UF2 to the NICENANO drive. The device installs it and reboots.
3. Re-enter the bootloader and confirm the version in INFO_UF2.TXT.
4. Repeat for the other half.

To revert, flash any older update-nice_nano_bootloader-*.uf2 from the same releases page the same way — the process is not one-way.

macOS flashing note

Recent macOS versions (FSKit FAT driver) break Finder/cp copies to UF2 bootloader drives — writes are silently dropped or fail with error -50. Flash from the terminal against the raw disk instead:

diskutil list                       # find the NICENANO disk number
diskutil unmountDisk disk4          # keep the device attached
sudo dd if=firmware.uf2 of=/dev/rdisk4 bs=512

The bootloader flashes any 512-byte sector carrying UF2 magic, so the filesystem layer is not needed.

Repository layout

src/                 engine, v4 decoder, BLE split protocol, inflate
tools/compile_v4.py  dictionary compiler (both JSONs → two half blobs)
tools/fetch_dict.py  dictionary downloader (Plover main, Lapwing base)
tests/test_dict_v4.c host round-trip test (42,000 vectors)

Development

The dictionary compiler (tools/compile_v4.py) requires Python 3 with numpy. GitHub Actions runners have this preinstalled; for local builds install via pip install numpy.

License

PolyForm Noncommercial 1.0.0 — see LICENSE.