Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Further fine-tuning a pre-trained LLM on a tiny dataset until training loss nears zero sharply improves its long, open-ended text generation under greedy decoding.

desk verdict Hyperfitting is a real, reproducible effect that improves open-ended generation, but the paper overstates the anti-memorization evidence. read the letter →

arxiv 2412.04318 v2 pith:X3ZRTUE5 submitted 2024-12-05 cs.CL cs.AI

Fredrik Carlsson , Fangyu Liu , Daniel Ward , Murathan Kurfali , Joakim Nivre This is my paper · ORCID
classification cs.CLcs.AI
keywords hyperfittingopen-endedtextgenerationoverfittinggreedydecodingnucleussamplingtype-tokenratiotop-rankencouragementlanguagemodelfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces 'hyperfitting'—fine-tuning a pre-trained LLM on a small set of 2000 sequences until its training loss is near zero—and reports that this counter-intuitive procedure greatly improves long, open-ended text generation under greedy decoding. The hyperfitted models beat nucleus sampling on both diversity and human preference, and the effect holds across model sizes from 1.1B to 70B parameters and extends to autoregressive image generation. The authors show that hyperfitted models rarely repeat training sequences and that explicitly blocking those sequences does not reduce quality, arguing the gain is not dataset memorization. They propose that low training loss sharpens the model's predicted distributions and encourages desirable tokens into the top ranks, a behavior they call 'top-rank encouragement.' If the claim holds, hyperfitting is a simple, cheap fix for a well-known failure mode of LLMs.

What carries the argument

The mechanism is hyperfitting itself: fine-tuning a pre-trained LLM on 2000 sequences of 256 tokens for 20 epochs with a low learning rate (1e-6) until near-zero training loss. The paper's proposed explanatory mechanism is 'top-rank encouragement'—that achieving a very low training loss teaches the model to place desirable tokens at the top of its predicted distribution, a property distinct from perplexity, which the paper shows does not track generation quality. This sharpened ranking transfers to unseen contexts even when the specific top-ranked token is wrong, and the authors use a citation blocker (suppressing any 5-token subsequence that appears in the training data) as a control to show the effect is not simple repetition of the training set.

What would settle it

Run a paraphrase-aware semantic similarity check between the hyperfitted models' generated texts and the 2000 training sequences; if a large share of high-quality continuations are near-semantic rewrites of training content that the exact-match and BLEU checks miss, the central claim of a generalizable sharpening phenomenon would be false.

Watch

Extended reading notes

Core claim

The central discovery is that deliberately overfitting a pre-trained LLM to near-zero training loss on a small sample set changes its generation behavior qualitatively: greedy decoding becomes less repetitive and more human-preferred over long sequences. Hyperfitted models allocate nearly all probability mass to a single token at each step, a sharpening that persists on unseen contexts and produces high perplexity even when the generated text is judged good. The same 2000 sequences, presented in shuffled order, produce models with roughly 30% different top-1 predictions, suggesting the training data does not deterministically dictate which tokens emerge as favorites. The paper interprets these results as evidence that hyperfitting collapses and sharpens the corpus-average modeling space from pre-training, and states this as the 'top-rank encouragement' hypothesis: the behavior of predicting good tokens in the top ranks is itself learnable.

Load-bearing premise

The conclusion that hyperfitted models generalize rather than memorize rests on the assumption that the citation blocker and overlap metrics, which detect exact 5-token substring matches and BLEU overlap, fully capture training-data reuse; if the models were paraphrasing or semantically recalling the 2000 training sequences, the claimed generalization would be an illusion.

Editorial extensions

If this is right

  • Greedy decoding with hyperfitted models can replace nucleus sampling for long open-ended generation, producing both higher diversity and higher human preference.
  • The phenomenon is reproducible across models from 1.1B to 70B parameters and crosses into autoregressive image generation, so it is not specific to one architecture or text domain.
  • Hyperfitting degrades MMLU and GLUE performance only slightly, so the sharpening does not come at the cost of catastrophic loss of general capabilities.
  • TTR remains above 50 with as few as 16 training samples, making the method computationally cheap and easy to apply.
  • Because hyperfitted models produce extremely sharp distributions, sampling without temperature becomes near-deterministic, which must be considered when combining hyperfitting with other decoding schemes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the sharpening mechanism is general, hyperfitting should also apply to other autoregressive modalities such as audio and video generation, a direct extension the paper does not test.
  • The 30% divergence in top-1 predictions under data shuffling suggests hyperfitting selects one of many equally good modes of the pre-trained distribution; an ensemble of models hyperfitted on differently shuffled data could be a cheap way to increase output diversity.
  • Near-deterministic outputs may make hyperfitted models more susceptible to adversarial or biased generations, since sampling no longer provides a stochastic escape; measuring this risk would be a natural follow-up.
  • If top-rank encouragement is learnable, a training objective that directly optimizes the rank of the correct token, rather than likelihood, might produce the same long-sequence benefit without the perplexity penalty; that is a testable design change.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript reports an empirical phenomenon, termed 'hyperfitting': fine-tuning a pre-trained LLM on 2,000 fixed sequences of 256 tokens for 20 epochs with a small learning rate (1e-6) until training loss approaches zero, then generating with greedy decoding. Across TinyLlama 1.1B, DeepSeek 7B, Llama 3.1 8B/70B, and ImageGPT-Large, the authors report that hyperfitted models receive substantially higher human preference ratings and higher TTR than the original models, that greedy decoding with hyperfitted models outperforms nucleus sampling baselines at 256 tokens, and that a 5-gram citation blocker does not degrade quality. The paper also reports sharply peaked next-token distributions, about 30% shifts in top-1 predictions when training data order is shuffled, and preserved or slightly degraded MMLU/GLUE performance. The authors interpret these results as a generalizable sharpening of pre-trained knowledge, propose a 'top-rank encouragement' hypothesis, and argue that hyperfitting is distinct from grokking and double descent.

Significance. The central observation is simple, unexpected, and potentially useful: an intervention that worsens validation loss can greatly improve long-sequence generation quality, without new data or sampling heuristics. The paper's strengths include a large human evaluation (over 20,000 annotations), multiple LLMs across three domains, an image-generation extension, public skeleton code and models, and an explicit citation-blocking control. The main weakness is that the strongest interpretive claims--that the effect is generalization rather than retrieval and that it is fundamentally distinct from grokking/double descent--are not established by the reported evidence; the paper itself concedes the latter. These gaps are addressable with additional analyses and by softening the claims.

major comments (4)
  1. [Section 7.2 and Abstract] The abstract claims hyperfitting is 'distinctly different' from grokking and double descent, but Section 7.2 explicitly states: 'Admittedly, this entails we cannot track an aligned validation score, preventing us from proving that hyperfitting fundamentally differs from previous discoveries.' The differences listed in Section 7.2 (model scale, task type, time course, absence of weight decay) are observational and do not establish a distinct mechanism. Please either remove the 'distinctly different' wording from the abstract and conclusions, or provide an aligned validation metric (e.g., a generation-quality score computed on held-out contexts during training) that can separate hyperfitting from delayed generalization.
  2. [Section 4.2 and Appendix B.2, Table 5] The evidence that hyperfitted models generalize rather than memorize is insufficient. The citation blocker (Section 3) and the overlap metrics in Section 4.2 detect only exact 5-token substrings, BLEU overlap, and longest common subsequences; they do not detect 4-gram recombinations, sentence-level paraphrase, or semantic recall. Table 5 shows that 85-89% of hyperfitted fiction-conditioned texts contain at least one >5-token overlap with the 2,000 training sequences, so the blocker is frequently active. The fact that quality survives when exact 5-grams are blocked shows only that the model can fall back to other memorized or paraphrased fragments, not that it generalizes. To support the paper's central claim, please add a semantic-similarity or paraphrase-level audit comparing hyperfitted outputs to training documents, or a control condition that clearly removes the possibility of training-set recall; otherwise the conclusion that hyperfitting causes a 'generalizable increase' should be substantially qualified.
  3. [Section 4.1 and Table 1] The human-preference results are reported without confidence intervals, significance tests, or inter-annotator agreement, although three annotations per comparison are collected. This matters because some comparisons are close: for the 128-token scenario, hyperfitted DeepSeek (49.4) and Llama 3.1 (50.1) do not improve over their Top-P baselines (50.0 and 50.5), while the abstract's 'outperform Top-P' claim is supported only at 256 tokens. Please report uncertainty estimates and agreement metrics, and restrict the 'outperform' claim to the length regimes where it is supported.
  4. [Section 3 and Figure 2] The definition of hyperfitting relies on achieving 'near-zero training loss,' but training-loss curves are reported only for TinyLlama (Figure 2) and ImageGPT (Appendix B.1, Figure 10); no final training loss is given for DeepSeek or the Llama models. Without this information, a reader cannot verify that the procedure matched the stated condition. Please report final training loss (or loss curves) for all hyperfitted models and state the numerical threshold used for 'near-zero.'
minor comments (5)
  1. [Abstract and Appendix B.2, Table 5] The phrase 'rarely fall into repeating sequences they were trained on' is hard to reconcile with Table 5, where 85-89% of hyperfitted fiction-conditioned texts contain at least one >5-token overlap; please state the overlap length threshold (e.g., >10 tokens) and reference it in the abstract if the claim is retained.
  2. [Table 1] The column headers '128 Pref', '256 Pref', '128 TTR', and '256 TTR' are ambiguous; use 'Pref (128)', 'Pref (256)', and similar.
  3. [Section 6.2 and Table 4] The text says 'no clear trend emerges,' but the News-hyperfitted model is best on all three datasets; please clarify whether this should be interpreted as a trend or as within-sample variation.
  4. [Section 8] There is a typo in 'they doesn't change predicted probabilities'; it should read 'they do not change predicted probabilities.'
  5. [Section 8] The phrase 'besides all phenomena seemingly contradicting early stopping' is awkward; consider rewording to 'like the other phenomena, hyperfitting appears to contradict early stopping.'

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the main claims are evaluated against external human preferences and held-out generation; the only author-overlapping citation is independently validated in the appendix.

full rationale

The paper's central claim that hyperfitting improves open-ended generation is an empirical intervention measured by external outcomes (over 20,000 human annotations on held-out contexts, Section 4 and Appendix A) and by diversity metrics on generated texts; no fitted parameter is relabeled as a prediction and no equation reduces to its own input. The only author-overlapping citation is the use of Carlsson et al. (2024) in Section 3 to justify TTR as a quality proxy ('the average TTR has been shown to correlate well with human preferences'), but the paper independently validates this proxy in Appendix A.1 using its own annotation consensus data, so the self-citation is not load-bearing. The paper also explicitly owns the key limitation that hyperfitted models show increased exact-overlap rates (Table 5: 85–89% of fiction-conditioned texts triggered the 5-token blocker) and that Appendix B.2 calls this 'evident'; this weakens the generalization inference without making it circular, because the headline preference results do not depend on the overlap metric being perfect. Section 7.2 even concedes it cannot prove hyperfitting differs from grokking or double descent ('preventing us from proving that hyperfitting fundamentally differs from previous discoveries'), an honest limitation rather than a self-supporting derivation. Overall the derivation chain is self-contained against external judgments, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 6 assumptions · 1 invented entities

The central claim depends on the specific hyperfitting recipe (LR, epochs, batch, sample count) and on several domain assumptions about evaluation validity. The invented entity 'top-rank encouragement' is a post-hoc hypothesis rather than a measured quantity.

free parameters (5)
  • learning_rate = 1e-6
    Chosen by hand to preserve pre-trained knowledge while reaching near-zero training loss; sensitivity is not tested, so the claimed effect is specific to this setting.
  • epochs = 20
    Chosen so training loss approaches zero; Figure 2 shows TTR rises with training, so the endpoint of near-zero loss matters for the magnitude of the reported effect.
  • training_samples = 2000 sequences of 256 tokens from Fiction-Stories
    Default hyperfitting set for all text experiments; Section 6.3 shows the effect persists down to 16 samples but weakens at 8, so the 2000-sample choice affects the reported results.
  • batch_size = 8
    Fixed across experiments; the authors note that with 8 training samples all updates become identical, which they use to explain the quality drop at that size.
  • citation_blocker_ngram = 5 tokens
    Threshold for the citation blocker used to test whether blocking training-data subsequences changes output quality; the conclusion about non-memorization depends on this threshold.
assumptions (6)
  • domain assumption Pre-trained LLMs (TinyLlama, DeepSeek, Llama 3.1) are representative of LLMs in general
    The paper generalizes from three model families to the phenomenon; other architectures and training distributions are not tested.
  • domain assumption TTR is a valid automatic proxy for human preference for long open-ended text
    Validated in Appendix A.1 on the paper's own annotations, but it is still a heuristic that ignores content quality.
  • domain assumption Hired Fiverr annotators give reliable and unbiased preference labels
    No inter-annotator agreement is reported; the reliability of the 10 USD/hour freelancer judgments is assumed.
  • domain assumption The 300 evaluation contexts from Wikipedia, Fiction-Stories, and BBC News are representative of open-ended text generation
    The central claim concerns open-ended generation generally, but the evaluation is limited to three English corpora.
  • domain assumption Greedy decoding is the right probe for the underlying model's generative ability
    The authors deliberately avoid sampling, but real usage often involves sampling; the effect may differ under stochastic decoding.
  • ad hoc to paper Exact 5-token overlap detection in the citation blocker is sufficient to detect training-data memorization
    The blocker only catches exact substring matches; paraphrased or semantic recall of the 2000 training sequences would go undetected, which matters for the claim that hyperfitted output is not recycled training data.
invented entities (1)
  • top-rank encouragement
    purpose: Hypothesized mechanism explaining why low training loss yields better top-rank token predictions despite poor perplexity
    Section 7.3 introduces this as a hypothesis; the paper provides correlational evidence (low entropy, sharpened distributions) but no direct test that distinguishes top-rank encouragement from other sharpening effects.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation." pith.science (2026). https://pith.science/paper/X3ZRTUE5

@misc{pith2026241204318,
  author       = {Pith},
  title        = {Pith review of: The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X3ZRTUE5}},
  note         = {Machine review of arXiv:2412.04318}
}
read the original abstract

This paper introduces the counter-intuitive generalization results of overfitting pre-trained large language models (LLMs) on very small datasets. In the setting of open-ended text generation, it is well-documented that LLMs tend to generate repetitive and dull sequences, a phenomenon that is especially apparent when generating using greedy decoding. This issue persists even with state-of-the-art LLMs containing billions of parameters, trained via next-token prediction on large datasets. We find that by further fine-tuning these models to achieve a near-zero training loss on a small set of samples -- a process we refer to as hyperfitting -- the long-sequence generative capabilities are greatly enhanced. Greedy decoding with these Hyperfitted models even outperform Top-P sampling over long-sequences, both in terms of diversity and human preferences. This phenomenon extends to LLMs of various sizes, different domains, and even autoregressive image generation. We further find this phenomena to be distinctly different from that of Grokking and double descent. Surprisingly, our experiments indicate that hyperfitted models rarely fall into repeating sequences they were trained on, and even explicitly blocking these sequences results in high-quality output. All hyperfitted models produce extremely low-entropy predictions, often allocating nearly all probability to a single token.

Figures

Figures reproduced from arXiv: 2412.04318 by the authors.

Figure 1
Figure 1. Example of greedy decoding using Llama 3.1 and its hyperfitted counterpart. Color indi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Training and validation loss for TinyLlama during overfitting, along with the resulting [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Distribution of the longest overlap between 1000 generated texts and the dataset [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: A subsequence from the validation data and the corresponding top-3 predictions. The [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Left: Top-1 rank similarity matrix of Llama 3.1 (8B) hyperfitted on identical, but shuffled, data. Right: The resulting mean TTR of 300 generated texts as the number of training samples vary. 6 DATA INFLUENCE The following experiments aim to investigate the effect and …
Figure 6
Figure 6. Figure 6: Examples of generated images using greedy decoding when passed 25% of an image. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Visualization of how distributions with the same probability for the next token (visualized [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Screenshot of the interface used by the annotators. The example texts are from the 256 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Left: TTR distributions of generated texts where all three annotators agreed on their quality, with 5% of outliers filtered away Right: TTR of the 96 most recent tokens in relation to the currently generated sequence length. A HUMAN ANNOTATION The research reported in …
Figure 10
Figure 10. Figure 10: Training and validation loss curves for hyperfitted ImageGPT model. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: More examples of generated images using the original and hyperfitted ImageGPT (large). [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Re-distilling a model's own RL-trained policy into 1K SFT samples reproduces RL accuracy at a fraction of the compute.

  2. Rethinking Early Stopping: Refine, Then Calibrate

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Stopping training on the loss after temperature scaling (a refinement estimate) instead of the raw validation loss, then applying temperature scaling afterwards, lowers test logloss.

  3. Quality-constrained Entropy Maximization Policy Optimization for LLM Diversity

    cs.CL 2026-02 conditional novelty 4.0 of 10

    Quality-constrained entropy maximization yields simple DPO-like objectives that increase LLM output diversity while preserving or slightly improving quality, with theoretical guarantees under tuned temperature conditions.

Reference graph

Works this paper leans on

36 extracted references · 10 canonical work pages · cited by 3 Pith papers

  1. [1]

    The pitfalls of next-token prediction, 2024

    Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, 2024. URL https://arxiv.org/abs/2403.06963

    arXiv 2024

  2. [2]

    Reconciling modern machine-learning practice and the classical bias–variance trade-off

    Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116 0 (32): 0 15849–15854, July 2019. ISSN 1091-6490. doi:10.1073/pnas.1903070116. URL http://dx.doi.org/10.1073/pnas.1903070116

    doi:10.1073/pnas.1903070116 2019

  3. [3]

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, ...

    arXiv 2024

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, ...

    arXiv 2005

  5. [5]

    Lundberg, Harsha Nori, Hamid Palangi, Marco T \' u lio Ribeiro, and Yi Zhang

    S \' e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco T \' u lio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4 . CoRR, abs/2303.12712, 2023. doi:10.48550/ARXIV.2303.12712. U...

    arXiv doi:10.48550/arxiv.2303.12712 2023

  6. [6]

    Branch- GAN : Improving text generation with (not so) large language models

    Fredrik Carlsson, Johan Broberg, Erik Hillbom, Magnus Sahlgren, and Joakim Nivre. Branch- GAN : Improving text generation with (not so) large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=sHEJJmzBIN

    work page 2024

  7. [7]

    Generative pretraining from pixels

    Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp.\ 1691--1703. PMLR, 2020

    work page 2020

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany...

    arXiv 2024

Show all 36 references
  1. [9]

    Text english code fiction nonfiction dataset, 2024

    Alasdair Forsythe. Text english code fiction nonfiction dataset, 2024. URL https://huggingface.co/datasets/alasdairforsythe/text-english-code-fiction-nonfiction. Accessed: 2024-09-30

    2024

  2. [10]

    A theoretical analysis of the repetition problem in text generation

    Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. In AAAI Conference on Artificial Intelligence, 2020. URL https://api.semanticscholar.org/CorpusID:229923515

    2020

  3. [11]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder: When the large language model meets programming -- the rise of code intelligence, 2024. URL https://arxiv....

    2024 arXiv

  4. [12]

    Identity mappings in deep residual networks

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision -- ECCV 2016, pp.\ 630--645, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46493-0

    2016

  5. [13]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ

    2021

  6. [14]

    The curious case of neural text degeneration

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rygGQyrFvH

    2020

  7. [15]

    Varshney, Caiming Xiong, and Richard Socher

    Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. ArXiv, abs/1909.05858, 2019. URL https://api.semanticscholar.org/CorpusID:202573071

    1909 arXiv

  8. [16]

    Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction

    Yucheng Li, Frank Guerin, and Chenghua Lin. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 18600--18607, 2024

    2024

  9. [17]

    Michaud, and Max Tegmark

    Ziming Liu, Eric J. Michaud, and Max Tegmark. Omnigrok: Grokking beyond algorithmic data, 2023. URL https://arxiv.org/abs/2210.01117

    2023 arXiv

  10. [18]

    Starcoder 2 and the stack v2: The next generation, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Z...

    2024 arXiv

  11. [19]

    Language model evaluation beyond perplexity

    Clara Meister and Ryan Cotterell. Language model evaluation beyond perplexity. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference...

    2021 doi

  12. [20]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openr...

    2017

  13. [21]

    Deep double descent: Where bigger models and more data hurt

    Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=B1g5sA4twr

    2020

  14. [22]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022 arXiv

  15. [23]

    Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022

    Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets, 2022. URL https://arxiv.org/abs/2201.02177

    2022 arXiv

  16. [24]

    Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adri \`a Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, ...

    2023

  17. [25]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. EMNLP 2018, pp.\ 353, 2018

    2018

  18. [26]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. T...

    2022

  19. [27]

    Emergent abilities of large language models

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022 b

    2022

  20. [28]

    Neural text generation with unlikelihood training

    Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. CoRR, abs/1908.04319, 2019. URL http://arxiv.org/abs/1908.04319

    1908 arXiv

  21. [29]

    Learning to break the loop: Analyzing and mitigating repetitions for neural text generation, 2022

    Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation, 2022. URL https://arxiv.org/abs/2206.02369

    2022 arXiv

  22. [30]

    Understanding deep learning requires rethinking generalization, 2017

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017. URL https://arxiv.org/abs/1611.03530

    2017 arXiv

  23. [31]

    Understanding deep learning (still) requires rethinking generalization

    Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64 0 (3): 0 107–115, February 2021. ISSN 0001-0782. doi:10.1145/3446776. URL https://doi.org/10.1145/3446776

    2021 doi

  24. [32]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. CoRR, abs/2401.02385, 2024. URL https://arxiv.org/abs/2401.02385

    2024 arXiv

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  26. [34]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  27. [35]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  28. [36]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.