A byte that no live mid-token state can consume, with no re-entry into the start state, is provably always a token boundary; splitting there reproduces the serial token stream exactly, and a conservative discarded-token variant recovers newline for C-like and JSON lexers.
Berlinkov, Robert Ferens, Andrew Ryzhikov, and Marek Szyku la
1 Pith paper cite this work, alongside 2 external citations. Polarity classification is still indexing.
1
Pith paper citing it
2
external citations · OpenAlex
citation-role summary
background 1
citation-polarity summary
fields
cs.FL 1years
2026 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Certified Split Points for Parallel Lexing: Exact and Modulo Discarded Tokens
A byte that no live mid-token state can consume, with no re-entry into the start state, is provably always a token boundary; splitting there reproduces the serial token stream exactly, and a conservative discarded-token variant recovers newline for C-like and JSON lexers.