r/programming • u/matryun • 14h ago
Subsize 【 subscalar 】 encoding
gitlab.comIndexing strings by character (Unicode scalar) instead of UTF offsets can be more comfortable for parsers or short manipulations.
The minimum cost for reading a >= U+FE character is:
- Lookup rope table by byte key (holding a high bit position), in goal of the scalar value in a sequence of higher fixed scalars
- Check for collision subtable (is not null => repeat the step 1), specializing the target high bit further
For optimal use, contiguous sequences of characters after Latin-* that turn into reserves into the mask sequence still allow for a few trailing Latin-* (e.g. Katakana can still be mixed with Latin-* whitespace), so those trails turn into reseves as well.
Update (I miswrote a detail in the implementation).
https://gitlab.com/mryun/langopt/-/blob/master/encoding/string.md