A byte that no live mid-token state can consume, with no re-entry into the start state, is provably always a token boundary; splitting there reproduces the serial token stream exactly, and a conservative discarded-token variant recovers newline for C-like and JSON lexers.
Cameron, Kenneth S
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.FL 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Certified Split Points for Parallel Lexing: Exact and Modulo Discarded Tokens
A byte that no live mid-token state can consume, with no re-entry into the start state, is provably always a token boundary; splitting there reproduces the serial token stream exactly, and a conservative discarded-token variant recovers newline for C-like and JSON lexers.