REVIEW 5 cited by
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.
Forward citations
Cited by 5 Pith papers
-
Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
A learned dynamic chunking hierarchy lets byte-level language models match or beat BPE-tokenized Transformers at matched compute, with larger gains on Chinese, code, and DNA.
-
Do LLM Modules Generalize? A Study on Motion Generation for Autonomous Driving
On Waymo Sim Agents, LLM-style tokenization, positional embeddings, pretraining, RL post-training, and test-time search can be adapted to improve motion generation, but not all transfer without domain-specific changes.
-
chDzDT: Word-level morphology-aware language model for Algerian social media text
A character-level transformer trained on isolated words, with a five-language label head, yields compact embeddings that rival or beat much larger pre-trained models on morphological tagging across Arabic, English, an...
-
Return of the Encoder: Maximizing Parameter Efficiency for SLMs
At equal small sizes, encoder-decoder language models are faster on edge hardware and, after distillation from a larger decoder-only teacher, match or beat decoder-only models on several benchmarks.
-
Digital Guardians: Can GPT-4, Perspective API, and Moderation API reliably detect hate speech in reader comments of German online newspapers?
On the German HOCON34k hate speech test set, GPT-4o with one-shot prompting scored highest on the combined F2/MCC metric, about 5 points above the fine-tuned BERT baseline.
Discussion (0). Continue with ORCID to comment.