Pith. sign in

REVIEW 4 major objections 6 minor 35 references

SpeLLM: Character-Level Multi-Head Decoding

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SpeLLM replaces an LLM's token-level output head with k parallel character-level heads, cutting decoding runtime by 5.1% while maintaining competitive downstream performance.

desk verdict A genuinely new character-level multi-head decoding method with a modest measured speedup; the main gap is that the fast and accurate configurations are never measured as one system. read the letter →

arxiv 2507.16323 v1 pith:KGABQLFY submitted 2025-07-22 cs.CL

Amit Ben-Artzy , Roy Schwartz This is my paper
classification cs.CL
keywords SpeLLMcharacter-leveldecodingmulti-headoutputself-distillationprojectionbottleneckinferencespeedupvocabularydecouplinglow-resourcelanguages
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the token-level output head of a pretrained LLM—the projection from the final hidden state onto a vocabulary of 100,000+ tokens—is a removable bottleneck. SpeLLM replaces that single large head with k small linear heads that predict the next token as k characters in one parallel step, so the output space grows combinatorially while the output projection shrinks by more than two orders of magnitude. A self-distillation procedure converts a standard BPE-based LLM into this spelling form using only the teacher's top-5 predictions as labels, and the authors demonstrate this on four LLMs. Their SpeLLM variants stay competitive with the teacher on BoolQ, ARC-Easy, GSM8K, and CNN/Daily Mail while decoding 5.1% faster on average across models. If the claim holds, vocabulary size no longer forces a proportional increase in generation cost, which would make larger input vocabularies and better coverage of underrepresented languages practical.

What carries the argument

The central object is a bank of k linear heads over the final hidden state, each with a small character vocabulary of size s, whose argmax outputs concatenate to spell the next token, with a padding symbol for short tokens. Because s is far smaller than the BPE vocabulary S, the output projection becomes drastically cheaper while the set of representable strings is combinatorially large. The argument is carried by three interacting pieces: the parallel character heads; a self-distillation loss that trains each head against the best-matching of the teacher's top-3 token spellings plus an auxiliary token-level loss that preserves the teacher's token representations; and an auto-correct layer that, for strings that are not valid tokens, filters the BPE vocabulary by position-wise top-3 character constraints and re-scores the survivors with the original token head.

What would settle it

Re-score SpeLLM's output against the original corpus token instead of the teacher's top-5 predictions; if exact-match accuracy falls materially below the reported 92% (or 95% with AutoCorrect), the teacher's top-5 supervision is inflating the apparent spelling quality.

Watch

Extended reading notes

Core claim

The central claim is that an LLM can generate its next token as a fixed-length character string predicted in parallel by k independent linear heads instead of as an index into a huge embedding table. With k=10 heads over a 105-symbol character vocabulary, the output projection uses fewer than 0.85% of the original output-layer parameters, yet covers roughly 91% of Latin-script tokens in the tested vocabularies. Distilled only on the teacher's top-5 predictions, the student spells the teacher's intended token exactly about 92% of the time, rising to about 95% with an auto-correct filter that re-scores candidate tokens; including valid prefixes the match rate reaches 94.89% and 97.57%, respectively. On downstream tasks the SpeLLM variants match or beat the teacher on BoolQ and ARC-Easy, lose some ground on GSM8K and CNN/Daily Mail, and regain part of that ground through an entropy-based fallback to the original token head.

Load-bearing premise

The pipeline assumes the teacher model's top-5 token predictions, with the original gold token ignored, are a sufficiently correct and unbiased set of labels, since both the distillation loss and the intrinsic evaluation score against that same top-5 list.

Editorial extensions

If this is right

  • Decoding speedup: across Llama3.2-3B, Llama3-8B, Gemma2-2B, and Gemma2-9B, SpeLLM reduces end-to-end generation runtime by 5.1% on average, with the isolated output-head time dropping to a small fraction of the token-level head's time while transformer layers dominate remaining latency.
  • Quality retention: SpeLLM is comparable or better than the teacher on BoolQ and ARC-Easy, and within a few points on GSM8K and CNN/Daily Mail, with an entropy-fallback mechanism closing much of the GSM8K gap.
  • Output-layer economy: ten heads of 105 characters total 1,050 vectors, under 0.85% of the 128K-256K token output matrices, so input vocabulary can be enlarged without proportionally enlarging the output projection.
  • Head-count flexibility: increasing k from 5 to 15 raises exact-match spelling from 72% to 93%, while total match rates including partial prefixes stay roughly constant, so the architecture is not tied to one head count.
  • Confidence signal: SpeLLM's mean character entropy is a reliable error signal, with about 99% accuracy in the lowest-entropy bin falling to about 65% in the second bin, which validates the entropy-fallback design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same self-distillation recipe should transfer to non-Latin scripts: because the heads use a small character set with an unknown-character symbol, a SpeLLM trained on multilingual data could spell words its BPE tokenizer represents poorly, turning the paper's stated low-resource-language motivation into a testable accuracy comparison.
  • The reported 5.1% end-to-end speedup is a floor for what the output-head decoupling alone buys; since the authors note transformer layers dominate latency, pairing SpeLLM with larger input vocabularies or multi-token prediction could push the total speedup substantially higher.
  • Because both training and intrinsic evaluation use the teacher's top-5 as ground truth, an independent check against original text would separate spelling quality from teacher confidence; the paper's entropy analysis suggests failures would concentrate where the teacher itself is uncertain.
  • A straightforward variant—predicting one character for each of several future tokens across k heads, rather than k characters of a single token—would compound the savings and is hinted at in the paper's conclusion; it would require a teacher that provides multiple future tokens, which existing multi-token-prediction setups already supply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes SpeLLM, a modification of standard transformer LLMs in which the token-level output head is replaced by k parallel character-level linear heads that predict a fixed-length character string in a single decoding step. A pretrained BPE-based LLM serves as teacher, and the SpeLLM student is trained by self-distillation: the teacher's top-5 token predictions (with the gold token ignored) are used as labels, with a character loss and an auxiliary token-level loss. The paper reports intrinsic accuracy against the teacher's top-5 predictions, downstream performance on BoolQ, ARC-Easy, GSM8K, and CNN/Daily Mail, and a 5.1% average runtime speedup over the teacher, across Llama3.2-3B, Llama3-8B, Gemma2-2B, and Gemma2-9B. Additional analyses examine the number of character heads, accuracy vs. token length, entropy-based fallback, and an AutoCorrect mechanism.

Significance. If the main claims held, SpeLLM would be a simple, practical way to shrink the output projection and modestly accelerate decoding of existing LLMs while maintaining output quality, and the paper's release of code would aid reproducibility. The idea of decoupling input and output vocabularies through character-level multi-head prediction is reasonable and the self-distillation setup is straightforward. However, the present evaluation does not fully support the headline claims: the intrinsic metric is circular, the speedup and the competitive downstream results are measured on different configurations, and the downstream and runtime results lack uncertainty quantification. These issues are addressable, so the paper warrants a major revision rather than rejection.

major comments (4)
  1. [Section 3.3 and Section 2.2] The intrinsic evaluation in Table 1 measures agreement with the teacher's top-5 token predictions, which is exactly the training target defined in Section 2.2 (the teacher's top-5 predictions, with the gold token ignored, are used as distillation labels). Thus the reported 91.75% 'full exact match' and 94.89% total are distillation-fidelity scores, not independent measures of generation quality. The text's phrasing ('SpeLLM generates a full match in almost 92% of the cases') overstates what the metric establishes. Please relabel these results as distillation fidelity or, better, add an external metric such as exact match against gold text, character-level perplexity, or downstream task performance on held-out data.
  2. [Section 4 (Fig. 3) and Section 5.3] The 5.1% speedup is measured only for SpeLLM and SpeLLM+AutoCorrect (Fig. 3), while the downstream results that 'closely approach' the teacher on GSM8K and CNN/Daily Mail are obtained with the entropy-fallback variant (Fig. 2 and Section 5.3). No end-to-end runtime is reported for the entropy-fallback configuration, and the fallback trigger rate is not given. Since entropy fallback routes a fraction of tokens through the full token-level head whenever the mean character-head entropy exceeds 0.22, its latency is not captured by the 5.1% figure. Consequently, the abstract's conjunction—competitive performance and 5.1% faster—is not established for any single deployed variant. Please measure and report end-to-end latency for the entropy-fallback variant, including the trigger rate and the resulting average added cost.
  3. [Section 3.3 and Fig. 2/Fig. 3] The downstream evaluation uses only 500 samples for BoolQ, ARC-Easy, and GSM8K and 100 samples for CNN/Daily Mail, and no confidence intervals or significance tests are reported. Figure 2 shows many differences of only a few points between SpeLLM variants and the teacher, so the claims of 'comparable' performance and improvements 'of up to 6.2%' are not statistically supported. Similarly, the runtime measurements in Fig. 3 have no error bars, leaving unclear whether the 5.1% average difference is significant. Please report confidence intervals or significance tests (or increase sample sizes) for both downstream accuracy and runtime.
  4. [Section 2.2 (training labels)] The student is trained to reproduce the teacher's top-5 predictions while the original gold token is ignored. If the correct token is not among the teacher's top-5 at a given position, the student is trained to spell an incorrect but plausible word, and the intrinsic evaluation—also based on the teacher's top-5—cannot detect this failure. The paper should report the frequency with which the gold token appears in the teacher's top-5 on the training and evaluation data, and ideally evaluate against gold tokens directly, to establish the quality of the distillation signal. This is a load-bearing assumption because the conversion process depends entirely on the teacher's correctness.
minor comments (6)
  1. [Section 4] The word 'neglibgble' in the runtime paragraph is a typo and should be 'negligible'.
  2. [Section 3.3] 'Other benchmarks are used in zero-short settings' should read 'zero-shot settings'.
  3. [Section 2.2] 'We than compute the loss' should read 'We then compute the loss'.
  4. [Section 7 (Limitations)] 'our approach is not as accurate on as the teacher model' is ungrammatical; it should read 'not as accurate as the teacher model'.
  5. [Abstract and Section 3.1] The abstract's claim that the approach 'increase[s] support for underrepresented languages' is not supported by the implementation: the character vocabulary in Section 3.1 contains only Latin characters, digits, punctuation, a padding symbol, and a single unknown symbol, with diacritics stripped, so non-Latin scripts are collapsed to the unknown symbol. The paper should either extend the character vocabulary to cover non-Latin scripts or temper the claim to a potential avenue for future work.
  6. [Section 5.3] The entropy-fallback threshold of 0.22 is stated without justification or sensitivity analysis; a brief study of how results vary with the threshold would strengthen the proposal.

Circularity Check

1 steps flagged · score 6.0 of 10

Intrinsic 'accuracy' is defined against the same teacher top-5 set used as distillation labels, so Table 1 measures fit to the training target; downstream and runtime results remain externally grounded.

  1. fitted input called prediction [Sections 2.2-2.3 and 3.3; Table 1]
    "Our goal is for SpeLLM( M) to match M’s BPE predictions. To do so, we run inference over a dataset and record M’s top-5 predictions, while ignoring the original gold word. ... Correctness is assessed by comparing SpeLLM’s output against the top-5 predictions of the model’s token-level head."

    The training objective is to match the teacher's top-5 token predictions (Sec. 2.2), and Sec. 3.3 scores 'correctness' against exactly that same set. Further, Sec. 2.3 selects as label the precomputed top-3 token with the most position-wise character matches with the student's current char_argmax, so the target is partly chosen to be close to the student's own output. Thus Table 1's 91.75%/94.93% 'full exact match' measures agreement with the distillation target, not with the original gold token; the 10-character and prefix criteria are likewise defined against the teacher's top-5 set. The intrinsic evaluation is, by construction, a fit-to-training-signal metric, not an independent spelling-correctness test. Downstream tasks use external labels, so the circularity is partial.

full rationale

The main quantitative result has two independent parts: downstream quality (Fig. 2) and speedup (Fig. 3). Both are measured against external benchmarks or direct timing and are therefore not circular. The circularity is confined to the intrinsic evaluation (Table 1 and related analyses), where 'correctness' is defined as matching the teacher's top-5 set, which is also the source of the distillation labels. The training procedure even selects among those top-5/top-3 candidates using the student's own current character argmax, so the high full-match percentages largely reflect self-consistency with the training target rather than spelling accuracy against the original gold text. This is a partial circularity: it undermines the internal accuracy claim but does not force the downstream or runtime conclusions. The paper does not report end-to-end latency for the entropy-fallback configuration that yields the best downstream numbers; that is a reporting gap rather than a circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 2 invented entities

The central claim depends on several hand-chosen hyperparameters (k, entropy threshold, character set) and on the assumption that the teacher's top-5 outputs are a reliable training signal. The padding and unknown character symbols are invented to make fixed-length character decoding work, but neither is validated against external benchmarks.

free parameters (4)
  • Number of character heads k = 10 (default; 5 and 15 analyzed)
    Architecture hyperparameter chosen by the authors; performance changes with k (Table 2).
  • Entropy fallback threshold = 0.22
    Hand-set threshold for deciding when to fall back to the token-level head; affects GSM8K and CNN results in Fig. 2.
  • Character vocabulary size = 105 symbols
    Selected to cover Latin letters, digits, punctuation, plus padding and unknown symbols; not all scripts are representable.
  • Top-k teacher predictions used = top-5 for labels, top-3 for character matching
    Choices for distillation; the number of candidates affects training signal and auto-correct candidate set.
assumptions (4)
  • domain assumption Teacher's top-5 predictions, ignoring gold labels, provide a sufficient training signal.
    Section 2.2: loss is computed against teacher's top-5 tokens; if the true word is absent from this set, the student is trained to imitate a wrong answer.
  • ad hoc to paper The BPE token can be segmented into a fixed-length character stream of k=10, with longer tokens continued in the next step.
    Section 2.1 and footnote 4; this mismatch is acknowledged for 3.24% of tokens but the training procedure does not explicitly handle cross-token continuation.
  • domain assumption Fine-tuning only the last five feed-forward layers preserves the information needed for character decoding.
    Section 2.2 states only these layers are trained; if the deeper layers are not adapted, the hidden states may not support character-level decoding.
  • domain assumption A Latin-focused character set with diacritic stripping is adequate for downstream tasks.
    Appendix A: 91% of tokens are Latin-only; non-Latin characters collapse to a single unknown symbol, so the model cannot spell them.
invented entities (2)
  • Padding symbol
    purpose: Fills character positions for tokens shorter than k=10
    Introduced ad hoc to enable fixed-length parallel character output.
  • Unknown character symbol
    purpose: Represents characters outside the 105-symbol set (e.g., non-Latin scripts)
    Allows the model to avoid crashes on unseen characters but prevents actual generation of those scripts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpeLLM: Character-Level Multi-Head Decoding." pith.science (2026). https://pith.science/paper/KGABQLFY

@misc{pith2026250716323,
  author       = {Pith},
  title        = {Pith review of: SpeLLM: Character-Level Multi-Head Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KGABQLFY}},
  note         = {Machine review of arXiv:2507.16323}
}
abstract

Scaling LLM vocabulary is often used to reduce input sequence length and alleviate attention's quadratic cost. Yet, current LLM architectures impose a critical bottleneck to this procedure: the output projection layer scales linearly with vocabulary size, rendering substantial expansion impractical. We propose SpeLLM, a method that decouples input and output vocabularies by predicting character-level strings through multiple output heads. In SpeLLM, each of the $k$ linear heads predicts a single character simultaneously, enabling the model to represent a much larger output space using smaller, independent linear heads. We present a self-distillation approach for converting a standard LLM to a SpeLLM. Our experiments with four pre-trained LLMs show their SpeLLM variants achieve competitive performance on downstream tasks while reducing runtime by 5.1% on average across models. Our approach provides a potential avenue for reducing LLM costs, while increasing support for underrepresented languages and domains.

Figures

Figures reproduced from arXiv: 2507.16323 by the authors.

Figure 1
Figure 1. Left: Using a large vocabulary allows us to represent the same text using fewer tokens, which reduces both space and runtime. However, it also requires projecting the output token onto a large embedding table, increasing computational cost and runtime of each decoding step; Middle: Using a smaller vocabulary reduces the processing time of each token, but requires representing text using more tokens; Right: We propos… view at source ↗
Figure 2
Figure 2. Results of SpeLLM on Downstream Tasks. SpeLLM is comparable or superior to the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Runtime analysis of SpeLLM vs. the teacher LLM across models. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Accuracy vs. length of the nearest token (i.e., the token with the lowest number of mistakes [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: SpeLLM predictions grouped by mean character-level entropy with accuracy indicated by [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Relationship between teacher model entropy and SpeLLM’s alignment with teacher [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Analysis of AutoCorrect performance, including overall accuracy, accuracy when Auto [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Accuracy vs. length of nearest token across models (i.e. token with least number of mistakes [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Mean length of the SpeLLM prefix and the corresponding target tokens. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 15 canonical work pages

  1. [1]

    A new algorithm for data compression

    Philip Gage. A new algorithm for data compression. C Users J., 12(2):23–38, February 1994. ISSN 0898-9788. doi: 10.5555/177910.177914

    work page arXiv 1994

  2. [2]

    Neural Machine Translation of Rare Words with Subword Units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. arXiv, August 2015. doi: 10.48550/arXiv.1508.07909

    arXiv doi:10.48550/arxiv.1508.07909 2015

  3. [3]

    The Llama 3 Herd of Models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, et al. The Llama 3 Herd of Models. arXiv, July 2024. doi: 10.48550/arXiv.2407.21783

    arXiv doi:10.48550/arxiv.2407.21783 2024

  4. [4]

    Gemma 2: Improv- ing Open Language Models at a Practical Size

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, et al. Gemma 2: Improv- ing Open Language Models at a Practical Size. arXiv, July 2024. doi: 10.48550/arXiv.2408. 00118

    doi:10.48550/arxiv.2408 2024

  5. [5]

    Do all languages cost the same? tokenization in the era of commercial language models

    Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904–9923, Sing...

    doi:10.18653/v1/2023 2023

  6. [6]

    Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models, 2023

    Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassi...

    arXiv 2023

  7. [7]

    Language model tokenizers introduce unfairness between languages

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. In Proc. of NeurIPS, 2023

    work page 2023

  8. [8]

    MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

    Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, et al. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. arXiv, May 2023. doi: 10.48550/arXiv. 2305.07185

    arXiv doi:10.48550/arxiv.2305.07185 2023

Show all 35 references
  1. [9]

    Byte Latent Transformer: Patches Scale Better Than Tokens

    Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. Byte Latent Transformer: Patches Scale Better Than Tokens. arXiv, Dec...

    2024 doi arXiv

  2. [10]

    Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens

    Itay Itzhak and Omer Levy. Models In a Spelling Bee: Language Models Implicitly Learn the Character Composition of Tokens. arXiv, August 2021. doi: 10.48550/arXiv.2108.11193

    2021 doi arXiv

  3. [11]

    What do tokens know about their characters and how do they know it? ArXiv, abs/2206.02608, 2022

    Ayush Kaushal and Kyle Mahowald. What do tokens know about their characters and how do they know it? ArXiv, abs/2206.02608, 2022. URL https://api.semanticscholar.org/ CorpusID:249394509

    2022 arXiv

  4. [12]

    Training Verifiers to Solve Math Word Problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems. arXiv, October 2021. doi: 10.48550/arXiv...

    2021 doi arXiv

  5. [13]

    Teaching Machines to Read and Comprehend

    Karl Moritz Hermann, Tomáš Koˇciský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching Machines to Read and Comprehend. arXiv, June 2015. doi: 10.48550/arXiv.1506.03340

    2015 doi arXiv

  6. [14]

    Headless Language Models: Learning without Predicting with Contrastive Weight Tying.arXiv, September 2023

    Nathan Godey, Éric de la Clergerie, and Benoît Sagot. Headless Language Models: Learning without Predicting with Contrastive Weight Tying.arXiv, September 2023. doi: 10.48550/arXiv. 2309.08351. 10

    2023 doi arXiv

  7. [15]

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. arXiv, June 2024. doi: 10.48550/arXiv.2406.17557

    2024 doi arXiv

  8. [16]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv, May 2019. doi: 10.48550/arXiv.1905.10044

    2019 doi arXiv

  9. [17]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv, March 2018. doi: 10.48550/arXiv.1803.05457

    2018 doi arXiv

  10. [18]

    Efficient vocabulary reduction for small language models

    Yuta Nozaki, Dai Nakashima, Ryo Sato, Naoki Asaba, and Shintaro Kawamura. Efficient vocabulary reduction for small language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Kareem Darwish, and Apoorv Agarwal, edit...

    2025

  11. [19]

    T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory- Efficient Embeddings

    Björn Deiseroth, Manuel Brack, Patrick Schramowski, Kristian Kersting, and Samuel Weinbach. T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory- Efficient Embeddings. arXiv, June 2024. doi: 10.48550/arXiv.2406.19223

    2024 doi arXiv

  12. [20]

    FR-Spec: Accelerating Large-V ocabulary Language Models via Frequency-Ranked Speculative Sampling

    Weilin Zhao, Tengyu Pan, Xu Han, Yudi Zhang, Ao Sun, Yuxiang Huang, Kaihuo Zhang, Weilun Zhao, Yuxuan Li, Jianyong Wang, Zhiyuan Liu, and Maosong Sun. FR-Spec: Accelerating Large-V ocabulary Language Models via Frequency-Ranked Speculative Sampling. arXiv, February 2025. doi: ...

    2025 doi arXiv

  13. [21]

    LLM V ocabulary Compression for Low-Compute Environments

    Sreeram Vennam, Anish Joishy, and Ponnurangam Kumaraguru. LLM V ocabulary Compression for Low-Compute Environments. arXiv, November 2024. doi: 10.48550/arXiv.2411.06371

    2024 doi arXiv

  14. [22]

    Fast V ocabulary Transfer for Language Model Compression

    Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. Fast V ocabulary Transfer for Language Model Compression. arXiv, February 2024. doi: 10.18653/v1/2022. emnlp-industry.41

    2024 doi

  15. [23]

    V ocabulary-level Memory Efficiency for Language Model Fine-tuning

    Miles Williams and Nikolaos Aletras. V ocabulary-level Memory Efficiency for Language Model Fine-tuning. arXiv, September 2023. doi: 10.48550/arXiv.2309.08708

    2023 doi arXiv

  16. [24]

    Efficient softmax approximation for GPUs

    Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. Efficient softmax approximation for GPUs. arXiv, September 2016. doi: 10.48550/arXiv.1609.04309

    2016 doi arXiv

  17. [25]

    Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, and Anand Raghu- nathan

    Jacob R. Stevens, Rangharajan Venkatesan, Steve Dai, Brucek Khailany, and Anand Raghu- nathan. Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers. arXiv, March 2021. doi: 10.48550/arXiv.2103.09301

    2021 doi arXiv

  18. [26]

    Baharav, Ryan Kang, Colin Sullivan, Mo Tiwari, Eric Luxenberg, David Tse, and Mert Pilanci

    Tavor Z. Baharav, Ryan Kang, Colin Sullivan, Mo Tiwari, Eric Luxenberg, David Tse, and Mert Pilanci. Adaptive Sampling for Efficient Softmax Approximation. Advances in Neural Information Processing Systems , 37:117580–117613, December

  19. [27]

    Better & Faster Large Language Models via Multi-token Prediction

    Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, et al. Better & Faster Large Language Models via Multi-token Prediction. arXiv, April 2024. doi: 10.48550/arXiv. 2404.19737

    2024 doi arXiv

  20. [28]

    Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, et al. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv, January 2024. doi: 10.48550/ arXiv.2401.10774

    2024 doi arXiv

  21. [29]

    Improving Multilingual Models with Language-Clustered V ocabularies

    Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. Improving Multilingual Models with Language-Clustered V ocabularies. arXiv, October 2020. doi: 10.48550/arXiv. 2010.12777. 11

    2020 doi arXiv

  22. [30]

    ByT5: Towards a token-free future with pre-trained byte-to-byte models

    Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a token-free future with pre-trained byte-to-byte models. arXiv, May 2021. doi: 10.48550/arXiv.2105.13626

    2021 doi arXiv

  23. [31]

    Clark, Dan Garrette, Iulia Turc, and John Wieting

    Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. arXiv, March 2021. doi: 10.1162/tacl_a_00448

    2021 doi

  24. [32]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. arXiv, November

  25. [35]

    All models are trained on a single GPU: smaller models on Nvidia L40S, and larger models on Nvidia A100

    with a learning rate of 5e-5 and a weight decay of 0.01. All models are trained on a single GPU: smaller models on Nvidia L40S, and larger models on Nvidia A100. Token Coverage Llama uses a vocabulary of 128K tokens, while Gemma models use 256K. Our approach employs 10 linear ...

  26. [2017]

    A Appendix Accuracy over various token lengths In Section 5.2, we analyze the accuracy of models with varying number of character-level heads across different token lengths

    doi: 10.48550/arXiv.1711.05101. A Appendix Accuracy over various token lengths In Section 5.2, we analyze the accuracy of models with varying number of character-level heads across different token lengths. Here we create a plot similar to Fig. 4 with additional models (all wit...

    doi arXiv

  27. [2024]

    URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/ d52dbd66219dc4e432e0bd4f9c25c4c3-Abstract-Conference.html

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.