Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Learning to Insert [PAUSE] Tokens for Better Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Inserting [PAUSE] tokens before the tokens a model is least confident about, chosen by log-likelihood during fine-tuning, improves reasoning on math and code tasks across model sizes.

desk verdict A plausible cheap fine-tuning trick that is oversold: the AQUA-RAT leakage concern is probably a misreading, but missing seeds and the 'consistent' overclaim sink the empirical case as written. read the letter →

arxiv 2506.03616 v1 pith:EQJLXUER submitted 2025-06-04 cs.CL

Eunki Kim , Sangryul Kim , James Thorne This is my paper
classification cs.CL
keywords DynamicInsertingTokensTrainingpausetokeninsertionlog-likelihoodconfidencemathematicalreasoningcodefine-tuninglosscalibrationlanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Dynamic Inserting Tokens Training (DIT), a fine-tuning method that inserts a [PAUSE] token immediately before the M tokens in a training target that have the lowest log-likelihood under the model. The authors claim this simple modification consistently beats standard supervised fine-tuning and earlier pause-token insertion strategies on GSM8K, AQUA-RAT, and MBPP, with gains up to 4.7 percentage points on GSM8K, 3.23 percentage points on AQUA-RAT, and 3.4 percentage points in pass@1 on MBPP. They interpret the result as showing that low-likelihood positions mark genuine uncertainty, and that giving the model a pause before those positions raises the training loss on hard tokens and improves the probability of previously low-confidence tokens during inference. If the claim is right, a cheap fine-tuning change can improve reasoning without pre-training on pause tokens or hand-crafted insertion rules.

What carries the argument

The central object is a pause-injected training sequence: given a (prefix, target) pair, a forward pass ranks the target tokens by their next-token log-likelihood, and a hyperparameter $M_{\mathrm{DIT}}$ controls how many of the lowest-ranked positions receive a single [PAUSE] immediately before them. The model is then fine-tuned with the usual next-token prediction loss, but the positions where the next token is [PAUSE] are excluded from the loss, so [PAUSE] acts as a dummy token that never needs to be predicted. The mechanism does its work by making hard tokens harder: inserting an unpredicted token before an already low-likelihood token raises the cross-entropy loss at that position, which is exactly the training signal the paper argues is missing in standard fine-tuning. The empirical footprint of the mechanism is that DIT has higher training loss than SFT on the same data, while the probability of the formerly lowest-probability tokens rises at inference.

What would settle it

A direct falsification: insert [PAUSE] tokens immediately before the M tokens with the highest log-likelihood in the same training sequences, under identical settings. If accuracy matches DIT within noise, then the paper's claim that low-confidence positions are special is false, and the gain is just from adding tokens.

Watch

Extended reading notes

Core claim

Across supervised fine-tuning on three reasoning datasets (GSM8K, AQUA-RAT, MBPP) and three decoder-only models (Phi-2 2.7B, Phi-3 Mini 3.8B, Llama-3-8B), the paper reports that DIT—inserting [PAUSE] tokens immediately before the M tokens with the lowest next-token log-likelihood in each target sequence, and excluding the [PAUSE] tokens from the loss—is the only insertion scheme that consistently beats plain fine-tuning. The best gains are 4.7 percentage points on GSM8K, 3.23 percentage points on AQUA-RAT, and 3.4 percentage points in pass@1 on MBPP. The paper interprets the improvement as a training-signal effect: because [PAUSE] is an unseen token, the loss at the following hard token increases, and the model is forced to learn those tokens more robustly; at inference the long tail of low log-probability tokens is shortened, meaning previously uncertain tokens become more confident. The reported results are not uniformly positive—on Llama-3 with AQUA-RAT, DIT is slightly below SFT—and the paper attributes that to instruction format and hyperparameter choices rather than to the method itself.

Load-bearing premise

The load-bearing premise is that the tokens with the lowest next-token log-likelihood in the training target are exactly where the model's uncertainty hurts reasoning, rather than merely being formatting artifacts such as equation delimiters or sentence boundaries.

Editorial extensions

If this is right

  • DIT shows that pause-token benefits do not require pre-training on pause tokens; a fine-tuning-only insertion scheme suffices to improve results on GSM8K, AQUA-RAT, and MBPP.
  • Insertion position is the active ingredient: random, append-at-end, and after-every-word insertion often hurt, while log-likelihood-selected positions improve performance.
  • Making [PAUSE] a true dummy token (excluded from the loss) and inserting single tokens rather than blocks is important; ablations on GSM8K/Phi-2 put the optimum at five single [PAUSE] tokens per sequence.
  • The gains transfer across model sizes from 2.7B to 8B and across mathematical and code reasoning, indicating the effect is not tied to one architecture or answer format.
  • Per training example, DIT costs roughly one extra forward pass beyond SFT, which the paper argues is a small price relative to the accuracy gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors do not test inference-time insertion; a natural extension is to insert [PAUSE] at low-confidence positions during decoding after DIT training, which could concentrate extra computation at the same hard tokens and trade tokens for accuracy at inference.
  • Because the insertion rule depends only on next-token log-likelihood, the same recipe could be run with other uncertainty signals, such as semantic entropy over sampled rationales, to test whether better uncertainty estimates produce better insertion points.
  • The focal-loss analogy suggests DIT may be part of a broader family of hard-example training techniques; if so, it should transfer to tasks where errors concentrate in a few tokens, such as instruction following or long numerical derivations, rather than being specific to the three datasets tested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dynamic Inserting Tokens Training (DIT), a fine-tuning method that inserts a special [PAUSE] token before the M_DIT tokens with the lowest log-likelihood in each target sequence, based on a forward pass of the model. The loss on [PAUSE] tokens is excluded, so the inserted tokens act as dummy tokens that alter the context for predicting the subsequent low-confidence token. The authors compare DIT against standard SFT and three heuristic insertion baselines (Random, Append, After-All-Words) on GSM8K, AQUA-RAT, and MBPP, using Phi-2, Phi-3 Mini, and Llama 3 8B. They report accuracy/pass@1 gains up to 4.7%p on GSM8K, 3.23%p on AQUA-RAT, and 3.4%p on MBPP, and present analyses (training loss, token log-probability distributions) aimed at explaining why the inserted tokens help.

Significance. If the empirical claims held, the paper would make a useful and inexpensive contribution: a fine-tuning-only token-insertion strategy, requiring no pause-token pre-training and no inference-time changes, that improves reasoning on math and code benchmarks for small open models. The method is simple, reproducible (code and data links are provided), and the authors include a complexity analysis and an honest limitations section. However, the significance is reduced by (i) the invalid AQUA-RAT evaluation due to train/test leakage, (ii) the lack of statistical grounding (no error bars/multiple seeds) for small differences, and (iii) the overstatement of 'consistent' superiority given that several comparisons in Table 1 favor SFT or APPD. The mechanistic interpretation as a 'model-based, dynamic' approach is also not yet distinguished from a structural heuristic.

major comments (4)
  1. [Appendix A / Table 2] The AQUA-RAT evaluation is invalid due to train/test contamination. Appendix A states that the validation and test sets were 'supplemented' by randomly extracting 2,000 samples from the training set, and Table 2 shows that training used 70,104 samples. Since the 2,000 extracted samples are a subset of the training data, the test set overlaps with the training set. Consequently, all AQUA-RAT numbers in Table 1, including the claimed 3.23%p gain for Phi-2, are unreliable and cannot support any comparative claim. The experiment should be re-run on a clean split (e.g., the original withheld test set, possibly using the full 98k training data or a proper sampling scheme).
  2. [§3.1, §3.2, Appendix B] The claimed mechanism is not established. The paper interprets low token log-likelihood as 'model uncertainty' and uses it to justify the placement of [PAUSE] tokens, but Appendix B reports that the low-likelihood positions correspond to 'the start and end positions of equations and the points where sentences change,' and the examples in Appendix E show insertions before formatting tokens such as '=', '»', '####', and after sentence boundaries. This suggests the selection rule is largely capturing structural or format-related surprises rather than content uncertainty about the reasoning itself. The causal story in §5.2–5.4 (that [PAUSE] increases the loss on genuinely difficult tokens and boosts their probability) is therefore not supported by the evidence. The authors should either provide evidence that the selected positions are not merely structural (e.g., comparing against random insertion within the same structural categories), or revise the framing to characterize DIT as a learned structural heuristic.
  3. [Abstract and Table 1] The abstract's claim that DIT 'consistently outperforms traditional fine-tuning and previous token insertion methods' is contradicted by the paper's own results. In Table 1, DIT is worse than SFT on Phi-3 Mini GSM8K (78.62 vs 78.84) and on Llama 3 AQUA-RAT (69.96 vs 70.71), and worse than APPD on Phi-3 Mini GSM8K (78.62 vs 79.45), Phi-3 Mini AQUA-RAT (62.82 vs 63.22), and Llama 3 AQUA-RAT (69.96 vs 72.18). The text in §5.1 partially acknowledges this, but the abstract and contribution bullets should be revised to reflect the mixed results (e.g., 'can improve' instead of 'consistently outperforms').
  4. [§5.1 / Appendix C] The experiments appear to be based on a single run, and no error bars or multiple seeds are reported. Several of the reported improvements are very small (e.g., Phi-3 Mini AQUA-RAT: 62.82 vs 62.46, a 0.36%p difference; Phi-3 Mini MBPP is a tie at 32.2). Without variance estimates or at least two or three seeds, these differences cannot be distinguished from training noise, which further weakens the 'consistent outperformance' claim. The authors should report means and standard deviations across seeds, or at minimum verify that the main conclusions hold under different random seeds.
minor comments (6)
  1. [Abstract and Section 1] The abstract and Section 1 report 'up to 4.7%p' on GSM8K, while the contributions bullet in Section 1 reports 'up to 4.25%p'; these numbers should be reconciled.
  2. [Table 1] In the Phi-3 Mini MBPP row, SFT and DIT both show 32.2; the bold/underline formatting should clarify whether this is a tie and how ties are handled.
  3. [Section 4.1] The abbreviation 'AAW' is introduced as 'AA W' in the bullet list; please standardize the formatting.
  4. [Appendix B / Section 6] The choice to exclude [PAUSE] tokens from the loss is only described in Appendix B, but the comparison with 'DIT(learnable)' in Table 3 shows a large drop (47.15 vs 56.33) that makes this design decision important; it should be motivated in the main text.
  5. [Section 5.1] The text states that for Llama 3 on AQUA-RAT and MBPP, DIT slightly decreases compared to SFT, but Table 1 shows DIT improves on MBPP (14.0 vs 12.6); this should be corrected.
  6. [References] The references for the base models (e.g., 'Microsoft, 2023' as a blog post) are informal; please use the canonical technical reports or model cards.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DIT's position selection and evaluation are independent, and reported gains rest on external benchmarks.

full rationale

The paper's derivation chain is self-contained and empirically grounded. DIT selects insertion positions by computing token log-likelihoods on training targets (Section 3.1, Eqs. 5–6); this selection is not defined in terms of the evaluation metric or the final answer, so the method's effectiveness is not guaranteed by construction. The central claims (Table 1) are comparisons against SFT, RAN, APPD, and AAW on held-out GSM8K, AQUA-RAT, and MBPP splits, so the reported gains are empirical rather than forced. The loss-increase observation (Figure 3) is an expected consequence of the insertion rule and is presented as a mechanism, not as the sole evidence for downstream gains. The token-probability analysis (Figure 4) is post-hoc and may compare different low-probability token sets, but that is a validity/interpretation concern, not circularity. The authors explicitly acknowledge the similarity to focal loss (Section 5.4), and there are no load-bearing self-citations or imported uniqueness theorems. The limitations sections candidly note cases where DIT underperforms and that the mechanism remains partially obscured. In sum, no step in the derivation reduces to its own inputs; the central claim rests on external benchmark evaluations.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central method rests on treating token log-likelihood as a confidence signal (an assumption inherited from prior work) and on the untested hypothesis that inserting [PAUSE] before low-confidence tokens improves learning. The only tuned numeric parameter is M_DIT=5, selected on GSM8K.

free parameters (1)
  • M_DIT = 5
    Number of [PAUSE] tokens inserted per instance, chosen as optimal on GSM8K via ablation (Figure 5); applied to all datasets, but no evidence it is optimal elsewhere.
assumptions (3)
  • domain assumption Token log-likelihood is a reliable indicator of model confidence and of tokens that benefit from extra computation.
    Relies on prior work (Jiang et al., Kadavath et al., 2022) and is the basis for position selection; not independently validated here.
  • ad hoc to paper Inserting a [PAUSE] token before low-likelihood tokens during fine-tuning improves generalization without requiring [PAUSE] at inference.
    Core hypothesis; supported only by the paper's experiments, no theoretical or external verification.
  • domain assumption The pretrained models' tokenizers and architectures can accept a new special token [PAUSE] without breaking pretrained representations.
    Standard practice, but the paper does not describe tokenizer or vocabulary extension details.
invented entities (1)
  • [PAUSE] special token
    purpose: Inserted before low-likelihood tokens during training to increase loss signal on hard tokens and provide extra computation.
    The token is a new artifact whose benefit is demonstrated only within this paper; no external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning to Insert [PAUSE] Tokens for Better Reasoning." pith.science (2026). https://pith.science/paper/EQJLXUER

@misc{pith2026250603616,
  author       = {Pith},
  title        = {Pith review of: Learning to Insert [PAUSE] Tokens for Better Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EQJLXUER}},
  note         = {Machine review of arXiv:2506.03616}
}
read the original abstract

To enhance reasoning capabilities, previous works have explored incorporating special-purpose tokens into the training process. These strategies strengthen the learning mechanism of transformer-based large language models (LLMs). Building on prior research, in which inserting dummy tokens consecutively just before reasoning steps can enhance effectiveness, we introduce a novel approach termed Dynamic Inserting Tokens Training (DIT). Our method identifies positions within sequences where model confidence is lowest according to token log-likelihood. Strategically inserting [PAUSE] tokens on these positions bolsters the model's predictive capabilities for subsequent tokens. Experimental results across diverse datasets and models, from the 2.7B model to the 8B model, demonstrate that DIT consistently outperforms traditional fine-tuning and previous token insertion methods. With this simple yet effective method, we achieve accuracy gains of up to 4.7%p on GSM8K, 3.23%p on AQUA-RAT, and pass@1 improvements of up to 3.4%p on MBPP datasets. Our work shows a model-based, dynamic approach rather than a heuristic one, thereby broadening the scope of research in reasoning.

Figures

Figures reproduced from arXiv: 2506.03616 by the authors.

Figure 1
Figure 1. We propose DIT, a log-likelihood-based [PAUSE] token inserting methodology. We suggest that learning with [PAUSE] token insertion using this method will result in a probability boost for low-probability to￾kens during inference. (NTP) paradigm has enabled remarkable advances in language understanding and generation, with performance scaling predictably as model size and training data increase, by scaling laws (Brown… view at source ↗
Figure 2
Figure 2. We illustrate the DIT training process, which is divided into two steps. In the first step, we perform a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Train Loss Comparison between SFT and DIT (ours) on GSM8K Dataset. The DIT approach results in higher training loss compared to SFT, even when excluding losses associated with [PAUSE] tokens. 3.40%p on MBPP compared to SFT. These result indicate that our approach of inserting [PAUSE] tokens at strategically identified positions signifi￾cantly enhances performance. Similarly, with the Phi-3 Mini model, DIT achieves a… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Impact of varying the number of [PAUSE] tokens on model performance. The graph shows that inserting five [PAUSE] tokens achieves optimal perfor￾mance improvement for the Phi-2 model on the GSM8K dataset. how many low log-likelihood tokens should have [PAUSE] tokens ins…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Implicit Reasoning in Large Language Models: A Comprehensive Survey

    cs.CL 2025-09 conditional novelty 5.0 of 10

    A survey organizing implicit (silent) reasoning in LLMs into three execution paradigms, plus evidence, benchmarks, and challenges.

Reference graph

Works this paper leans on

49 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157

    arXiv 2024

  5. [5]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  6. [6]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    arXiv 2021

  7. [7]

    Gregor Bachmann and Vaishnavh Nagarajan. 2024. The pitfalls of next-token prediction. arXiv preprint arXiv:2403.06963

    arXiv 2024

  8. [8]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

Show all 49 references
  1. [9]

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022. Recurrent memory transformer. Advances in Neural Information Processing Systems, 35:11079--11091

    2022

  2. [10]

    Mikhail S Burtsev, Yuri Kuratov, Anton Peganov, and Grigory V Sapunov. 2020. Memory transformer. arXiv preprint arXiv:2006.11527

    2020 arXiv

  3. [11]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [12]

    Tri Dao. 2024. https://openreview.net/forum?id=mZn2Xyh9Ec Flashattention-2: Faster attention with better parallelism and work partitioning . In The Twelfth International Conference on Learning Representations

    2024

  5. [13]

    Timoth \'e e Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2023. Vision transformers need registers. arXiv preprint arXiv:2309.16588

    2023 arXiv

  6. [14]

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and ukasz Kaiser. 2018. Universal transformers. arXiv preprint arXiv:1807.03819

    2018 arXiv

  7. [15]

    Li Dong, Chris Quirk, and Mirella Lapata. 2018. https://doi.org/10.18653/v1/P18-1069 Confidence modeling for neural semantic parsing . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 743--753, Melbourne,...

    2018 doi

  8. [16]

    Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Sean Welleck, Peter West, Chandra Bhagavatula, Ronan Le Bras, et al. 2024. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36

    2024

  9. [17]

    Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Chukwunyere Osi, Prateek Sharma, Fan Chen, and Lei Jiang. 2023. Llmcarbon: Modeling the end-to-end carbon footprint of large language models. In The Twelfth International Conference on Learning Representations

    2023

  10. [18]

    K Ruwani M Fernando and Chris P Tsokos. 2021. Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33(7):2940--2951

    2021

  11. [19]

    Markus Freitag and Yaser Al-Onaizan. 2017. https://doi.org/10.18653/v1/W17-3207 Beam search strategies for neural machine translation . In Proceedings of the First Workshop on Neural Machine Translation, pages 56--60, Vancouver. Association for Computational Linguistics

    2017 doi

  12. [20]

    Ulrich Germann. 2003. https://aclanthology.org/N03-1010 Greedy decoding for statistical machine translation in almost linear time . In Proceedings of the 2003 Human Language Technology Conference of the North A merican Chapter of the Association for Computational Linguistics ,...

    2003

  13. [21]

    Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. 2024. https://openreview.net/forum?id=ph04CRkPdC Think before you speak: Training language models with pause tokens . In The Twelfth International Conference on Learning Rep...

    2024

  14. [22]

    David Herel and Tomas Mikolov. 2024. Thinking tokens for language modeling. arXiv preprint arXiv:2405.08644

    2024 arXiv

  15. [23]

    Namgyu Ho, Laura Schmid, and Se-Young Yun. 2022. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071

    2022 arXiv

  16. [24]

    Shaojie Jiang, Thomas Wolf, Christof Monz, and Maarten de Rijke. 2020. Tldr: token loss dynamic reweighting for reducing repetitive utterance generation. arXiv preprint arXiv:2003.11963

    2020 arXiv

  17. [25]

    Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.495 Active retrieval augmented generation . In Proceedings of the 2023 Conference on Empirical Methods in...

    2023 doi

  18. [26]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221

    2022 arXiv

  19. [27]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361

    2020 arXiv

  20. [28]

    Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL - A Conditional Transformer Language Model for Controllable Generation . arXiv preprint arXiv:1909.05858

    2019 arXiv

  21. [29]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR

    2023

  22. [30]

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll \'a r. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980--2988

    2017

  23. [31]

    Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. https://doi.org/10.18653/v1/P17-1015 Program induction by rationale generation: Learning to solve and explain algebraic word problems . In Proceedings of the 55th Annual Meeting of the Association for Computational ...

    2017 doi

  24. [32]

    Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. https://doi.org/10.18653/v1/2020.acl-main.769 End-to-end bias mitigation by modelling biases in corpora . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8706...

    2020 doi

  25. [33]

    Microsoft. 2023. https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/ Phi-2: The surprising power of small language models . [Blog post]

    2023

  26. [34]

    Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. 2020. Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems, 33:15288--15299

    2020

  27. [35]

    Khanh Nguyen and Brendan O ' Connor. 2015. https://doi.org/10.18653/v1/D15-1182 Posterior calibration and exploratory analysis for natural language processing models . In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1587--1598, ...

    2015 doi

  28. [36]

    Alec Radford and Karthik Narasimhan. 2018. https://api.semanticscholar.org/CorpusID:49313245 Improving language understanding by generative pre-training

    2018

  29. [37]

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training

    2018

  30. [38]

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. 2021. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937--13949

    2021

  31. [39]

    Siddharth Samsi, Dan Zhao, Joseph McDonald, Baolin Li, Adam Michaleas, Michael Jones, William Bergeron, Jeremy Kepner, Devesh Tiwari, and Vijay Gadepally. 2023. From words to watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance ...

    2023

  32. [40]

    Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling reasoning capabilities into smaller language models. arXiv preprint arXiv:2212.00193

    2022 arXiv

  33. [41]

    Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019. Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470

    2019 arXiv

  34. [42]

    Linwei Tao, Minjing Dong, and Chang Xu. 2023. Dual focal loss for calibration. In International Conference on Machine Learning, pages 33833--33849. PMLR

    2023

  35. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  36. [44]

    Neeraj Varshney, Man Luo, and Chitta Baral. 2022. Can open-domain qa reader utilize external knowledge efficiently like humans? arXiv preprint arXiv:2211.12707

    2022 arXiv

  37. [45]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  38. [46]

    Rose E Wang, Esin Durmus, Noah Goodman, and Tatsunori Hashimoto. 2022. Language modeling via stochastic processes. arXiv preprint arXiv:2203.11370

    2022 arXiv

  39. [47]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  40. [48]

    Kai Wu, Boyuan Jiang, Zhengkai Jiang, Qingdong He, Donghao Luo, Shengzhi Wang, Qingwen Liu, and Chengjie Wang. 2024. Noiseboost: Alleviating hallucination with noise perturbation for multimodal large language models. arXiv preprint arXiv:2405.20081

    2024 arXiv

  41. [49]

    Fuzhao Xue, Valerii Likhosherstov, Anurag Arnab, Neil Houlsby, Mostafa Dehghani, and Yang You. 2023. Adaptive computation with elastic input sequence. In International Conference on Machine Learning, pages 38971--38988. PMLR

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.