The reviewed record of science sign in
Pith

arxiv: 2106.12672 · v3 · pith:CUPLFF23 · submitted 2021-06-23 · cs.CL · cs.AI· cs.LG

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

Yi Tay , Vinh Q. Tran , Sebastian Ruder , Jai Gupta , Hyung Won Chung , Dara Bahri , Zhen Qin , Simon Baumgartner
show 2 more authors
Cong Yu Donald Metzler
This is my paper

Reviewed by Pithpith:CUPLFF23open to challenge →

classification cs.CL cs.AIcs.LG
keywords subwordcharformertokenizationgbstlearnsmodelmodelsadditionally
0
0 comments X
read the original abstract

State-of-the-art models in natural language processing rely on separate rigid subword tokenization algorithms, which limit their generalization ability and adaptation to new settings. In this paper, we propose a new model inductive bias that learns a subword tokenization end-to-end as part of the model. To this end, we introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. Concretely, GBST enumerates candidate subword blocks and learns to score them in a position-wise fashion using a block scoring network. We additionally introduce Charformer, a deep Transformer model that integrates GBST and operates on the byte level. Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Kronecker Embeddings replace learned embedding tables with a deterministic byte-level character-position factorization and single projection, reducing parameters over 90% with reported gains in loss and robustness on ...

  2. FLEXITOKENS: Flexible Tokenization for Evolving Language Models

    cs.CL 2025-07 unverdicted novelty 7.0

    FLEXITOKENS replaces rigid subword tokenizers and fixed-compression auxiliary losses with a simplified boundary-prediction objective in byte-level models, yielding lower over-fragmentation and up to 10-point gains on ...

  3. Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

    cs.CL 2026-06 unverdicted novelty 6.0

    IPA-based subword tokenizers trained across 24 languages improve tokenization quality and generalization to unseen languages compared to standard text tokenizers, especially for non-Latin scripts.

  4. ADE: Adaptive Dictionary Embeddings -- Scaling Multi-Anchor Representations to Large Language Models

    cs.CL 2026-04 unverdicted novelty 6.0

    ADE scales multi-anchor word representations to transformers via Vocabulary Projection, Grouped Positional Encoding, and context-aware reweighting, achieving 98.7% fewer trainable parameters than DeBERTa-v3-base while...

  5. Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation

    cs.CL 2023-02 unverdicted novelty 6.0

    Semantic entropy improves uncertainty estimation in natural language generation by incorporating semantic equivalences, outperforming standard entropy baselines on predicting model accuracy for question answering.