REVIEW 6 cited by
ByT5: Towards a token-free future with pre-trained byte-to-byte models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Most widely-used pre-trained language models operate on sequences of tokens corresponding to word or subword units. By comparison, token-free models that operate directly on raw text (bytes or characters) have many benefits: they can process text in any language out of the box, they are more robust to noise, and they minimize technical debt by removing complex and error-prone text preprocessing pipelines. Since byte or character sequences are longer than token sequences, past work on token-free models has often introduced new model architectures designed to amortize the cost of operating directly on raw text. In this paper, we show that a standard Transformer architecture can be used with minimal modifications to process byte sequences. We characterize the trade-offs in terms of parameter count, training FLOPs, and inference speed, and show that byte-level models are competitive with their token-level counterparts. We also demonstrate that byte-level models are significantly more robust to noise and perform better on tasks that are sensitive to spelling and pronunciation. As part of our contribution, we release a new set of pre-trained byte-level Transformer models based on the T5 architecture, as well as all code and data used in our experiments.
Forward citations
Cited by 6 Pith papers
-
Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalization
Byte-Prefix Marginalization maps a teacher's next-token distribution onto the student's vocabulary through shared byte prefixes plus an explicit residual, giving a mass-preserving target for on-policy distillation acr...
-
SpeLLM: Character-Level Multi-Head Decoding
SpeLLM converts a standard token-based LLM into a character-spelling model with multiple parallel output heads, achieving competitive downstream performance with a 5.1% average decoding speedup.
-
PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts
PiKE adaptively re-weights pretraining data sources by gradient magnitude and variance, exploiting low gradient conflicts to speed up convergence and improve downstream accuracy in LLM pretraining.
-
chDzDT: Word-level morphology-aware language model for Algerian social media text
A character-level transformer trained on isolated words, with a five-language label head, yields compact embeddings that rival or beat much larger pre-trained models on morphological tagging across Arabic, English, an...
-
Token-free Models for Sarcasm Detection
ByT5-small reaches 89.87% and CANINE reaches 72.88% on news-headline and Twitter sarcasm detection, each edging a T5 baseline by less than one accuracy point.
-
Cross-Linguistic Transfer in Multilingual NLP: The Role of Language Families and Morphology
The paper finds, through XLM-R experiments on WikiANN, that within-family and morphologically similar language pairs transfer better, but it mostly restates prior findings.
Discussion (0). Continue with ORCID to comment.