REVIEW 1 cited by
Lost in Space Marking
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We look at a decision taken early in training a subword tokenizer, namely whether it should be the word-initial token that carries a special mark, or the word-final one. Based on surface-level considerations of efficiency and cohesion, as well as morphological coverage, we find that a Unigram LM tokenizer trained on pre-tokenized English text is better off marking the word-initial token, while one trained on raw text benefits from marking word ends. Our findings generalize across domains.
Forward citations
Cited by 1 Pith paper
-
ByteSpan: Information-Driven Subword Tokenisation
A tokeniser that groups predictable bytes using a byte-level LM's surprisal or entropy achieves higher morphological alignment than BPE without loss of compression.
Discussion (0). Continue with ORCID to comment.