Pith. sign in

REVIEW 1 major objections 4 minor 58 references

Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content

T0 review · 1 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that when the same sentences and the same latent capacity are given to all three, no encoding wins across tasks: pixels hold onto written form, bytes keep cross-lingual alignment, and tokens best preserve topic information

desk verdict Controlled comparison of token/byte/pixel encodings that mostly delivers, but the token advantage in topic classification looks confounded by parameter count. read the letter →

arxiv 2607.16117 v1 pith:VLJ5Y4F3 submitted 2026-07-17 cs.CL

Ingo Ziegler , Martin Krebs , Desmond Elliott This is my paper · ORCID
classification cs.CL
keywords textencodingsrate-utilityfrontiersubwordtokensbyteencodingpixellanguagemodelscross-lingualalignmentmultilingualismtopicclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes a controlled way to compare text encodings: identical parallel content, a shared bottleneck whose width is swept, and three utilities (surface form, cross-lingual alignment, topic classification). Its central finding is that the encoding that wins depends on the task and the capacity regime: pixels preserve surface form best, bytes preserve cross-lingual alignment best, and tokens support topic prediction best. The rankings are not explained by how many positions each encoding produces; short inputs can throw away meaning while long inputs can hold structure that compresses well. The practical conclusion is that choosing tokens, bytes, or pixels is not a fixed preference but a rate-utility tradeoff across task, language mix, capacity, and compute.

What carries the argument

The rate-utility frontier itself. A single learned query compresses the encoder output into one vector of width D; sweeping D from 256 down to 1, while content stays parallel and identical across languages, traces how much of each utility survives compression. This separates the number of input positions (source rate), the latent capacity granted downstream, and the task-relevant information that remains.

What would settle it

Train the same three encodings at substantially larger scale and document length on parallel data and evaluate the same three utilities through the same bottleneck. If one encoding then wins on all three tasks at a single bottleneck width, the 'no encoding dominates' claim fails; alternatively, if byte alignment advantages vanish in same-script regimes at scale, the specific ranking does not generalize.

Watch

Extended reading notes

Core claim

Under controlled content and controlled capacity, tokens, bytes, and pixels trace crossing rate-utility frontiers rather than a single order of merit. Pixels reconstruct the surface of a sentence most compactly (near-perfect retrieval down to a latent width of 48 in multilingual regimes); bytes align translations across languages most reliably, especially when languages share a script; tokens dominate topic classification in every regime, aided by their dense meaning-bearing units. The frontiers cross because each encoding stores information in a different way: pixels spend capacity on visual detail, bytes on exact character structure, tokens on lexical identity.

Load-bearing premise

The frontiers are measured with small models trained from scratch on 701 parallel sentences per language; the paper's claim that these rankings reflect how the encodings behave in large pretrained systems rests on an assumption the authors state explicitly in the appendix: scale, data mixture, tokenizer training, and hardware optimality 'may shift the frontiers.'

Editorial extensions

If this is right

  • Encoding choice should be tuned per task rather than treated as a fixed preprocessing step.
  • For multilingual retrieval and sentence alignment, byte encodings deserve stronger consideration, especially within a shared script.
  • For encoder-style classification, subword tokens remain the strongest default among the three tested.
  • Pixels are a compute-cheap, compact input interface for surface-sensitive and dense-script applications, but they spend their capacity on visual detail rather than meaning.
  • Compute accounting changes the practical verdict: the byte advantage in alignment is cheap within a script (1.4–1.6× token FLOPs) but expensive across scripts (4.2×), where it nearly disappears.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If these frontiers hold at scale, token-centric pretraining may be systematically sacrificing cross-lingual alignment; hybrid byte-token or byte-aware architectures are a natural next test.
  • A testable extension: training pixel encoders with an auxiliary objective that ties visual form to token or semantic meaning could lift their near-floor performance on topic classification without losing surface preservation.
  • The Chinese case suggests script properties (morpheme density, character-to-meaning mapping) predict which encoding wins; one could build a script-characteristic-based selector for encoding choice.
  • The 'short inputs can discard useful meaning' result implies that average sequence length or tokenizer fertility is an incomplete proxy for comparing encodings; utility must be measured, not inferred from rate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper proposes a controlled comparison of three input encodings—subword tokens, UTF-8 bytes, and rendered pixel patches—on parallel SIB-200 sentences across five language regimes (monolingual English/Chinese, Latin-5, Cyrillic-5, Multiscript-5). A shared Transformer encoder compresses every encoding through a single-query bottleneck of width D swept from 256 down to 1, so that downstream representational capacity is fixed while content is held constant. The paper measures source rates, form preservation (self-retrieval), cross-lingual retrieval, topic classification, and training FLOPs. The main empirical claim is that no encoding dominates: pixels preserve surface form best, bytes preserve cross-lingual alignment best (especially in same-script regimes), and tokens support topic classification best; these rankings are not explained by source length alone. The paper includes compute accounting, five-seed averaging, validation-based checkpoint selection, and a zero-padding FLOP robustness check.

Significance. The controlled parallel-corpus bottleneck design is a valuable contribution: it separates source rate from post-encoding capacity and provides a template for comparing representations that differ in both sequence length and unit type. The compute accounting is unusually careful, and the paper ships code. If the rankings hold, the practical conclusion—encoding choice should be task-, language-, and capacity-dependent rather than a fixed preprocessing default—is well supported and would be useful to the field. The main caveat is that the topic-classification pillar of the central claim is vulnerable to a parameter-count confound (Table 4), which the paper discloses but does not control for.

major comments (1)
  1. [§4.4, Table 4, §3.3] The topic-classification result that anchors the central claim ('tokens support topic prediction best') may reflect parameter allocation rather than an intrinsic property of subword encoding. At D=256, token models in Latin-5, Cyrillic-5, and Multiscript-5 have 13.8M parameters versus 5.6M (bytes) and 5.9M (pixels); 8.2M of this difference is the 32,000-entry embedding table. Bottleneck width D is held fixed, but total front-end capacity is not. With only 701 training sentences per language, the extra embedding capacity can memorize topic-correlated vocabulary, and the reported token margin (macro-F1 0.46–0.50 vs 0.26–0.30 for bytes at D=256) is exactly where such a capacity advantage would be easiest to exploit. Appendix A acknowledges that the small dataset does not punish large vocabularies, but it does not address this parameter asymmetry as a threat to the topic ranking. Please add
minor comments (4)
  1. [Table 4] The 'Token Vocabulary' column is ambiguous: for English it reports 11,653 and for Chinese 4,265, while 32,000 is the SentencePiece vocabulary size. Please clarify whether these are the numbers of distinct observed tokens in the training data or the effective vocabulary sizes after pruning, and note the discrepancy in the caption.
  2. [§4.2] The text says bytes overtake 'below about D=10' in Latin-5 and Cyrillic-5; at D=8 the byte lead is 0.34 vs 0.30 for pixels. Since this is used to support the 'capacity regime' argument, consider marking the crossing point directly in Figure 3, and report whether the crossing is consistent across seeds.
  3. [Appendix A] The limitation 'the token models are not strongly punished for large vocabularies' is useful, but it should be tied explicitly to the parameter-count asymmetry in Table 4; otherwise the reader may not connect it to the topic-classification confound.
  4. [Abstract / §4.4] The sentence 'Tokens dominate every regime' is stated without the scope qualifier; consider 'dominate in the tested regimes' given the small-data caveats in Appendix A.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical rate-utility comparison against an external benchmark, with no fitted parameter defined in terms of the claimed rankings.

full rationale

The central claims are measured outcomes of models trained and evaluated on SIB-200, an external human-translated benchmark. The bottleneck width D is a controlled variable, and no parameter is fitted to the target rankings: tokenizers are trained on Bible text, checkpoints are selected on validation, and all reported numbers are test-set results. The token/byte/pixel rankings are therefore empirical findings, not algebraic consequences of the setup. Self-citations (Rust et al. 2023 for the patch projection; Kesen et al. 2025 for multilingual pixel pretraining) supply standard methods and prior context; they are not used as a uniqueness theorem or as the sole justification for the conclusions, so they are not load-bearing in a circular sense. The form-preservation probe does operationalize 'surface form' as each encoding's own adapter output, which is a construct-validity limitation rather than a circular reduction: the winner is not analytically forced, and the paper's own account attributes the pixel advantage to distinctive image structure and shorter input sequences. The larger embedding tables of token models are a potential capacity confound for the topic-classification ranking, but the paper discloses parameter counts, and the ranking is not an algebraic transform of those counts; this is a correctness/generalization concern, not circularity. Appendix A explicitly concedes that scale, data mixture, and hardware may shift the frontiers, further confirming that the authors do not present the results as deriving from the control definitions alone.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No fitted constants are defined in terms of the target rankings; listed free parameters are standard hyperparameters and rendering choices. The main axioms are representativeness assumptions about the dataset, the bottleneck control, the task probes, and the encodings. No new physical or conceptual entities are introduced.

free parameters (4)
  • SentencePiece vocabulary size = 32,000
    Chosen per regime; larger token adapters (8.2M params) give token models more total parameters than byte/pixel, a potential confound reported in Table 4.
  • Pixel rendering geometry = 36x32 patches, font size 16, max height 36
    Chosen rendering determines pixel source rates and form/alignment results; no sensitivity sweep is reported.
  • Encoder hidden size and depth = 256 hidden, 6 layers, 4 heads
    Standard capacity choices held constant across encodings, but not justified by scaling theory; affects absolute utility levels.
  • Early stopping patience and minimum epochs = 30 epochs patience, 30 minimum
    Chosen schedule affects convergence epoch counts used in the FLOP accounting and therefore the compute-utility comparisons.
assumptions (5)
  • domain assumption SIB-200 is a faithful human-parallel dataset; each sentence has the same meaning and topic label across languages.
    Section 3.1 uses it to control content; if translations or labels are noisy, the comparisons are not content-controlled.
  • domain assumption A single-learned-query bottleneck of width D fixes downstream representational capacity identically for all encodings.
    Section 3.3; this is the paper's main capacity control. It presumes latent width, not total parameters or sequence length, is the relevant capacity.
  • domain assumption InfoNCE self-retrieval, contrastive cross-lingual retrieval, and linear topic probes measure surface form, alignment, and semantic prediction respectively.
    Sections 4.2-4.4 and C.3; these are operationalizations, and different probes might rank encodings differently.
  • domain assumption Per-regime SentencePiece on parallel Bible text, raw UTF-8 bytes, and grayscale Noto patches are representative token/byte/pixel encodings.
    Section 3.2; tokenizer training corpus, font, and patch geometry are choices; other pipelines may shift the frontiers.
  • domain assumption Models trained from scratch on 701 sentences per language reveal encoding properties that persist at pretraining scale.
    Appendix A explicitly says scale may shift frontiers; the paper's external-validity claim depends on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content." pith.science (2026). https://pith.science/paper/VLJ5Y4F3

@misc{pith2026260716117,
  author       = {Pith},
  title        = {Pith review of: Rate-Utility Frontiers for Language Encodings: Comparing Tokens, Bytes, and Pixels Under Controlled Linguistic Content},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLJ5Y4F3}},
  note         = {Machine review of arXiv:2607.16117}
}
read the original abstract

Language models encode text as subword tokens, raw bytes, or rendered pixels, but these encodings are usually compared under modeling constraints that expose different amounts of linguistic content to models across different languages. We instead ask what each encoding preserves when both the content and the downstream capacity are controlled. Using verified parallel sentences across thirteen languages and five scripts, we compare tokens, bytes, and pixels through a shared bottleneck whose width is swept to trace rate-utility frontiers. This separates three quantities that are often conflated: the number of input positions an encoding creates, the latent capacity available after encoding, and the task-relevant information that survives compression. We evaluate three utilities: surface form preservation, cross-lingual sentence alignment, and topic classification. No encoding dominates across tasks or capacity regimes. Pixels preserve surface form best, bytes preserve cross-lingual alignment best, especially in same-script multilingual settings, and tokens support topic prediction best. These performances are not explained by sequence length alone. Short inputs can discard useful meaning, while long inputs can preserve information that compresses well. Choosing an encoding is therefore not a fixed preference for tokens, bytes, or pixels, but a rate-utility tradeoff that depends on the task, language mix, capacity regime, and compute budget.

Figures

Figures reproduced from arXiv: 2607.16117 by the authors.

Figure 1
Figure 1. The same linguistic content produces different [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Source rate relative to monolingual English, by language and encoding. Each cell [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Form preservation. Test Recall@1 for self-retrieval through a width- [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Cross-lingual retrieval. Test cross-lingual Recall@1 through a width- [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Topic classification. Test macro-F1 for SIB-200 topic classification through a width [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Per-pair cross-lingual Recall@1 at D=256, with source language as rows and target language as columns, for each encoding in Multiscript-5 (averaged over five seeds). Darker cells mark harder retrieval pairs. Chinese rows and columns are hardest for every encoding. Pixe…
Figure 7
Figure 7. Figure 7: Test utility against total training FLOPs at [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Per-pair cross-lingual Recall@1 at D=256 for every multilingual regime and encoding, averaged over five seeds. Rows are regimes (Latin-5, Cyrillic-5, Multiscript-5) and columns are encodings (token, byte, pixel). Within each panel, cell (A, B) is Recall@1 for retrievin…
Figure 9
Figure 9. Figure 9: Recalculation of Figure [PITH_FULL_IMAGE:figures/full_fig_p024_9.png]
Figure 10
Figure 10. Figure 10: Decomposition of the training cost into FLOPs per epoch and epochs to the [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 3 canonical work pages

  1. [1]

    Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee

    David Ifeoluwa Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba O. Alabi, Yanke Mao, Haonan Gao, and En-Shiun Annie Lee. SIB -200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the...

    2024

  2. [2]

    Do all languages cost the same? tokenization in the era of commercial language models

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp.\ 9904--9923, Singap...

    doi:10.18653/v1/2023.emnlp-main.614 2023

  3. [3]

    Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond

    Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7: 0 597--610, 2019. doi:10.1162/tacl_a_00288. URL https://aclanthology.org/Q19-1038/

    doi:10.1162/tacl_a_00288 2019

  4. [4]

    Morphbpe: A morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies, 2025

    Ehsaneddin Asgari, Yassine El Kheir, and Mohammad Ali Sadraei Javaheri. Morphbpe: A morpho-aware tokenizer bridging linguistic complexity for efficient llm training across morphologies, 2025. URL https://arxiv.org/abs/2502.00894

    arXiv 2025

  5. [5]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer, 2020. URL https://arxiv.org/abs/2004.05150

    arXiv 2020

  6. [6]

    A massively parallel corpus: the bible in 100 languages

    Christos Christodouloupoulos and Mark Steedman. A massively parallel corpus: the bible in 100 languages. Lang. Resour. Eval., 49 0 (2): 0 375–395, June 2015. ISSN 1574-020X. doi:10.1007/s10579-014-9287-y. URL https://doi.org/10.1007/s10579-014-9287-y

    work page doi:10.1007/s10579-014-9287-y 2015

  7. [7]

    Clark, Dan Garrette, Iulia Turc, and John Wieting

    Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics, 10: 0 73--91, 2022. doi:10.1162/tacl_a_00448. URL https://aclanthology.org/2022.tacl-1.5/

    doi:10.1162/tacl_a_00448 2022

  8. [8]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

    doi:10.18653/v1/ 2019

Show all 58 references
  1. [9]

    Multiscale byte language models -- a hierarchical architecture for causal million-length sequence modeling

    Eric Egli, Matteo Manica, and Jannis Born. Multiscale byte language models -- a hierarchical architecture for causal million-length sequence modeling. In ICML 2025 Workshop on Long-Context Foundation Models, 2025. URL https://openreview.net/forum?id=2r7YTSdWYD

    2025

  2. [10]

    O mer Veysel C a g atan, Akash Kundu, Martin Bernstorff, Shitao Xiao, Akshita Sukhlecha, Bhavish Pahwa, Rafa Po \'s wiata, Kranthi Kiran GV, Shawon Ashraf, Daniel Auras, Bj \

    Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, M \'a rton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemi \'n ski, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Diganta Misra, Shreeya Dhakal, Jonathan Rystr m,...

    2025

  3. [11]

    Glm-5: from vibe coding to agentic engineering, 2026

    GLM-5-Team, :, Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, Chenzheng Zhu, Congfeng Yin, Cunxiang Wang, Gengzheng Pan, Hao Zeng, Haoke Zhang, Haoran Wang, Huilong Chen, Jiajie Zhang, Jian Jiao, Jiaqi Gu...

    2026 arXiv

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  5. [13]

    Bitext mining using distilled sentence representations for low-resource languages

    Kevin Heffernan, Onur C elebi, and Holger Schwenk. Bitext mining using distilled sentence representations for low-resource languages. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 2101--21...

    2022 doi

  6. [14]

    HTML Bible Index -- Ukrainian

    HTML Bible . HTML Bible Index -- Ukrainian . https://www.htmlbible.com/sacrednamebiblecom/ukrainian/index.htm, 2026. Accessed: 2026-05

    2026

  7. [15]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proce...

    2021

  8. [16]

    Ticls: Tightly coupled language text spotter

    Leeje Jang, Yijun Lin, Yao-Yi Chiang, and Jerod Weinman. Ticls: Tightly coupled language text spotter. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.\ 3730--3740, March 2026

    2026

  9. [17]

    Lotz, Ingo Ziegler, Phillip Rust, and Desmond Elliott

    Ilker Kesen, Jonas F. Lotz, Ingo Ziegler, Phillip Rust, and Desmond Elliott. Multilingual pretraining for pixel language models. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in...

    2025 doi

  10. [18]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processi...

    2020

  11. [19]

    Ocr-free document understanding transformer

    Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23...

    2022 doi

  12. [20]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015

    2015

  13. [21]

    E uroparl: A parallel corpus for statistical machine translation

    Philipp Koehn. E uroparl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pp.\ 79--86, Phuket, Thailand, September 13-15 2005. URL https://aclanthology.org/2005.mtsummit-papers.11/

    2005

  14. [22]

    Subword regularization: Improving neural network translation models with multiple subword candidates

    Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 66-...

    2018 doi

  15. [23]

    S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing

    Taku Kudo and John Richardson. S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System ...

    2018 doi

  16. [24]

    P ix2 S truct: Screenshot parsing as pretraining for visual language understanding

    Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. P ix2 S truct: Screenshot parsing as pretraining for visual language understanding. In Andreas Krause, Emma B...

    2023

  17. [25]

    Trocr: transformer-based optical character recognition with pre-trained models

    Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, and Furu Wei. Trocr: transformer-based optical character recognition with pre-trained models. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and T...

    2023

  18. [26]

    Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages

    Tomasz Limisiewicz, Ji r \'i Balhar, and David Mare c ek. Tokenization impacts multilingual language modeling: Assessing vocabulary allocation and overlap across languages. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computati...

    2023 doi

  19. [27]

    Character-aware models improve visual text rendering

    Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, Rj Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of...

    2023 doi

  20. [28]

    Glyph-byt5: A customized text encoder for accurate visual text rendering

    Zeyu Liu, Weicong Liang, Zhanhao Liang, Chong Luo, Ji Li, Gao Huang, and Yuhui Yuan. Glyph-byt5: A customized text encoder for accurate visual text rendering. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Par...

    2024 doi

  21. [29]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  22. [30]

    Beyond fertility: STRR as a metric for multilingual tokenization evaluation

    Mir Tafseer Nayeem, Sawsan Alqahtani, Md Tahmid Rahman Laskar, Tasnim Mohiuddin, and M Saiful Bari. Beyond fertility: STRR as a metric for multilingual tokenization evaluation. In NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, a...

    2025

  23. [31]

    Hierarchical autoregressive transformers: Combining byte- and word-level processing for robust, adaptable language models

    Pit Neitemeier, Bj \"o rn Deiseroth, Constantin Eichenberg, and Lukas Balles. Hierarchical autoregressive transformers: Combining byte- and word-level processing for robust, adaptable language models. In The Thirteenth International Conference on Learning Representations, 2025...

    2025

  24. [32]

    NLLB Team , Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Pran...

    2022 arXiv

  25. [33]

    Byte latent transformer: Patches scale better than tokens

    Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srini Iyer. Byte latent transformer: Patches scale better than tokens. In Wanx...

    2025

  26. [34]

    Language model tokenizers introduce unfairness between languages

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp.\...

    2023

  27. [35]

    Language Models are Unsupervised Multitask Learners , 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners , 2019. URL https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf

    2019

  28. [36]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and ...

    2021

  29. [37]

    When every token counts: Optimal segmentation for low-resource language models

    Bharath Raj S, Garvit Suri, Vikrant Dewangan, and Raghav Sonavane. When every token counts: Optimal segmentation for low-resource language models. In Hansi Hettiarachchi, Tharindu Ranasinghe, Paul Rayson, Ruslan Mitkov, Mohamed Gaber, Damith Premasiri, Fiona Anting Tan, and La...

    2025

  30. [38]

    Making monolingual sentence embeddings multilingual using knowledge distillation

    Nils Reimers and Iryna Gurevych. Making monolingual sentence embeddings multilingual using knowledge distillation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.\...

    2020 doi

  31. [39]

    How good is your tokenizer? on the monolingual performance of multilingual language models

    Phillip Rust, Jonas Pfeiffer, Ivan Vuli \'c , Sebastian Ruder, and Iryna Gurevych. How good is your tokenizer? on the monolingual performance of multilingual language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meet...

    2021

  32. [40]

    Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott

    Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, and Desmond Elliott. Language modelling with pixels. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=FkSp8VW8RjH

    2023

  33. [41]

    Robust open-vocabulary translation from visual text representations

    Elizabeth Salesky, David Etter, and Matt Post. Robust open-vocabulary translation from visual text representations. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language ...

    2021 doi

  34. [42]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1715--17...

    2016 doi

  35. [43]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15 0 (56): 0 1929--1958, 2014. URL http://jmlr.org/papers/v15/srivastava14a.html

    1929

  36. [44]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput., 568 0 (C), February 2024. ISSN 0925-2312. doi:10.1016/j.neucom.2023.127063. URL https://doi.org/10.1016/j.neucom.2023.127063

    2024

  37. [45]

    Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler

    Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast character transformers via gradient-based subword tokenization. In International Conference on Learning Representations, 20...

    2022

  38. [46]

    Representation learning with contrastive predictive coding, 2019

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. URL https://arxiv.org/abs/1807.03748

    2019 arXiv

  39. [47]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Info...

    2017

  40. [48]

    Error patterns in historical ocr: A comparative analysis of trocr and a vision-language model, 2026

    Ari Vesalainen, Eetu Mäkelä, Laura Ruotsalainen, and Mikko Tolonen. Error patterns in historical ocr: A comparative analysis of trocr and a vision-language model, 2026. URL https://arxiv.org/abs/2602.14524

    2026

  41. [49]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE : A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupa a, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop B...

    2018 doi

  42. [50]

    The Holy Bible international: Text and audio bibles

    Wordproject . The Holy Bible international: Text and audio bibles. https://www.wordproject.org/, 2026. Accessed: 2026-05

    2026

  43. [51]

    On layer normalization in the transformer architecture

    Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on ...

    2020

  44. [52]

    B y T 5: Towards a token-free future with pre-trained byte-to-byte models

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. B y T 5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10: 0 291--306, 2022. do...

    2022 doi

  45. [53]

    Qwen3 technical report, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  46. [54]

    Megabyte: Predicting million-byte sequences with multiscale transformers

    LILI Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Pr...

    2023

  47. [55]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin...

    2020

  48. [56]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alch\' e -Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceed...

    2019

  49. [57]

    MIRACL : A multilingual retrieval dataset covering 18 diverse languages

    Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. MIRACL : A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguisti...

    2023 doi

  50. [58]

    The U nited N ations parallel corpus v1.0

    Micha Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The U nited N ations parallel corpus v1.0. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Pi...

    2016

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.