Pith. sign in

REVIEW 3 major objections 5 minor 32 references

Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Using Elastic Weight Consolidation, this paper continually pretrains all of Gemma2's parameters on Lithuanian text and reports that English fluency and domain knowledge are preserved on every benchmark tested, while Lithuanian improves on…

desk verdict A clean, reproducible empirical data point: EWC lets you add Lithuanian to Gemma2-2B without wrecking English, but the MMLU circularity and single runs make the headline counts softer than they look. read the letter →

arxiv 2505.05946 v2 pith:76AWD4Y2 submitted 2025-05-09 cs.CL cs.AIcs.LG

Vytenis Šliogeris , Povilas Daniušis , Artūras Nakvosas This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords continuallearningelasticweightconsolidationcatastrophicforgettinglargelanguagemodelsLithuanianperplexitydomainknowledgefull-parameterpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This technical report asks whether a large language model can learn a new language fluently without discarding what it already knows, and answers with a qualified yes: autoregressively pretraining all parameters of Gemma2 (2B version) on 10% of the Lithuanian part of the CulturaX dataset, with an Elastic Weight Consolidation (EWC) penalty, preserved English linguistic fluency and English domain knowledge on all seven benchmarks tested. The same regularizer also improved Lithuanian fluency and raised Lithuanian accuracy on five of the seven benchmarks. The authors interpret this as evidence that a simple regularizer, informed only by Fisher information computed on the MMLU benchmark, can add a low-resource language to a general-purpose model without access to the original training data.

What carries the argument

The load-bearing object is the EWC regularizer, $L_B(\theta)+\frac{\lambda}{2}\sum_i F_i(\theta_i-\theta_{A,i})^2$, where $F_i$ is the empirical Fisher information of the model on MMLU data before continual pretraining. This penalty slows the movement of parameters that MMLU gradients mark as carrying domain knowledge, while leaving the rest of the model free to adapt to Lithuanian. The authors compute $F_i$ from gradients of the log-likelihood on MMLU question-answer pairs, so the identity of the protected knowledge is set entirely by what MMLU represents.

What would settle it

Run the identical Lithuanian continual-pretraining protocol but estimate Fisher information on a single held-out benchmark (for example ARC-Easy) instead of MMLU; if the other six English benchmarks still stay flat the choice of MMLU is incidental, but if they drop, the paper's claim that MMLU-based Fisher protects domain knowledge fails for this setup.

Watch

Extended reading notes

Core claim

The central claim is that EWC regularisation, with Fisher information estimated on MMLU, turns what would otherwise be a catastrophic forgetting event into a net gain. Without EWC ($\lambda = 0$), English benchmark accuracy drops substantially after Lithuanian pretraining; with an intermediate regularisation strength (roughly $\lambda \in [10^2, 10^{11}]$), English accuracy is preserved on all seven benchmarks, and Lithuanian accuracy is higher than the untrained model on five of them. The paper states the result plainly: EWC mitigated catastrophic forgetting in the English component in linguistic fluency and domain knowledge in 7 of 7 cases, and improved the Lithuanian component in fluency and domain knowledge on 5 of 7 language-understanding benchmarks.

Load-bearing premise

The argument stands on the assumption that Fisher information computed on MMLU identifies the parameters underlying general domain knowledge, so that freezing them protects competence beyond MMLU; the paper itself notes that this step lacks a theoretical justification.

Editorial extensions

If this is right

  • Adding Lithuanian under EWC preserves English fluency (perplexity) and English accuracy on all seven tested benchmarks, whereas the same pretraining without EWC degrades them.
  • The same run raises Lithuanian accuracy above the untrained baseline on ARC-Easy, GSM8K, HellaSwag, MMLU, and WinoGrande, while keeping Belebele and TruthfulQA stable.
  • Very strong regularisation ($\lambda>10^{11}$) freezes the model so that it neither forgets English nor learns Lithuanian, bracketing an operational range for $\lambda$ between roughly $10^2$ and $10^{11}$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Nothing in the results ties the protected knowledge to a specific language, so a natural extension is to compute Fisher information on a mix of MMLU and Lithuanian text and check whether preservation survives on a held-out domain that MMLU underrepresents, such as code or medicine.
  • If the MMLU-derived Fisher mask generalises across domains, the same mask could be reused when adding other languages or modalities, turning EWC into a one-time audit of which parameters carry a pretrained model's competence.
  • Because the report covers one random seed and one data slice (10%), the 5/7 Lithuanian improvements could partly reflect dataset ordering or benchmark proximity to MMLU; repeating the run on a different slice of the Lithuanian corpus would test the stability of that result.
  • The observed co-occurrence of lower perplexity and higher benchmark accuracy, if causal, would make fluency training a proxy objective for domain knowledge in low-resource languages, a claim that is testable on other language pairs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This preprint reports a continual-pretraining experiment in which the authors take Gemma2-2B-it, autoregressively train on 10% of the Lithuanian portion of CulturaX, and use Elastic Weight Consolidation with Fisher information computed on MMLU to preserve prior knowledge. The authors evaluate English and Lithuanian linguistic fluency via perplexity and domain knowledge via seven standard benchmarks, and report that EWC preserves English fluency and accuracy on all seven benchmarks while improving Lithuanian performance on five of seven. The paper is a technical report with hyperparameters and open-sourced code.

Significance. If the results hold, the paper makes a useful practical contribution: it suggests that a simple regularizer can add a low-resource language to an LLM without original training data and with modest degradation of general competence. The paper is also honest about limitations, including the lack of theoretical justification for the Fisher source. However, the strongest evidence is weakened by using MMLU as both the Fisher source and an evaluation benchmark, and by the absence of uncertainty quantification, so the practical significance hinges on further validation.

major comments (3)
  1. [Section 3, Eq. (4) and Table 1] The claim of '7/7 cases' in the abstract and Section 1 is inflated by circularity: Eq. (4) in Section 3 estimates Fisher information on MMLU, and MMLU is one of the seven benchmarks in Table 1. The English preservation result should be recomputed excluding MMLU (6/6) and, ideally, with the Fisher computed on a different benchmark to demonstrate that the penalty protects general domain knowledge rather than MMLU-specific skills. As written, the paper's own Limitations section acknowledges the need for a theoretical justification, but an empirical de-circularization is absent.
  2. [Section 4, Figures 1-3] All headline counts in Section 4 and Figure 3 come from single-run evaluations without error bars, confidence intervals, or multiple seeds. The differences between λ values that determine '7/7' and '5/7' may be within run-to-run noise, especially on benchmarks like TruthfulQA where the gaps appear small. The paper should report variance estimates at least for the selected λ values, or explicitly frame the results as observations from a single run rather than statistically supported claims.
  3. [Section 4 and Conclusions] The contributions do not specify which value of λ produces the reported improvements. Figure 3 shows that the optimal λ varies by benchmark (e.g., GSM8K appears preserved at high λ while other benchmarks may require lower λ). The authors should define a prespecified selection rule (for example, the largest λ that preserves English fluency within a tolerance) and then report the Lithuanian benchmark counts for that λ, rather than inferring 5/7 from visual inspection of the full sweep.
minor comments (5)
  1. [Section 3, Linguistic fluency] The perplexity computation on TruthfulQA is underspecified; please state whether the perplexity is computed over the answer tokens conditioned on the question, and how the Lithuanian translation is obtained.
  2. [Figure captions] Figure captions refer to 'Fine-tuned' as 'autoregressive pretraining with EWC regularisation,' but the λ=0 curve is also fine-tuned without EWC; the label should be clarified (e.g., 'continually pretrained' versus 'initial model').
  3. [Section 3, Domain knowledge] The Lithuanian benchmarks from references [21,22] are used for evaluation, but their construction, number of instances, and translation methodology are not described; this is important for assessing the validity of the Lithuanian domain-knowledge results.
  4. [Figure 2 and Section 3] Figure 2 caption says 'LT-Llama-13B' while the text uses 'LT-Llama2-13B'; please make the notation consistent.
  5. [Limitations] The Limitations section should explicitly mention the lack of multi-seed runs and the MMLU circularity, since both are central to the strength of the evidence.

Circularity Check

1 steps flagged · score 4.0 of 10

MMLU is both the Fisher-information source and one of the seven evaluation benchmarks, so the '7/7' preservation claim is partly circular by construction.

  1. fitted input called prediction [Section 2.2 (Eq. 4), Section 3 'Domain knowledge', and Section 5 Conclusions]
    "To protect the parameters of Gemma2, which are potentially responsible for domain knowledge, we use MMLU data ... for Fisher's information estimation, estimating it via the empirical Fisher estimator ... Fi = (1/|D_MMLU|) sum_{(x,y) in D_MMLU} (partial / partial theta_i log p_theta(y|x) |_{theta_i = theta_A,i})^2 ... Note that although the English version of MMLU data was used in EWC to estimate Fisher information, we included this dataset in our benchmarks because, in our opinion, its empirical performance still may be interesting."

    The EWC penalty in Eqs. (3)-(4) is constructed from gradients evaluated on D_MMLU, so the regularizer explicitly penalizes parameter movement in directions that alter the conditional log-likelihood on MMLU examples. Counting MMLU accuracy among the seven benchmarks used to claim 'domain knowledge preserved in 7/7 cases' means one of the seven reported successes is not an independent held-out test of the regularizer; it is a near-neighbor of the objective used to define the penalty. The other six benchmarks provide independent evidence, so the circularity is partial, but the 7/7 headline is inflated by this overlap.

full rationale

The paper's central derivation chain is: estimate Fisher information from MMLU (Eq. 4), apply EWC while pretraining on Lithuanian CulturaX, then evaluate domain knowledge on a seven-benchmark suite and claim all seven are preserved. One link in that chain is partly circular: MMLU is simultaneously the data source for the Fisher penalty and one of the seven evaluation benchmarks. This is not a complete collapse, because the EWC penalty operates on log-likelihood while the evaluation uses accuracy, so MMLU accuracy is not strictly forced; moreover, six other English benchmarks are independent of the Fisher source, and the Lithuanian translated benchmarks provide an external check. The self-citations [21,22] to the authors' own Lithuanian benchmark translations are used as evaluation resources rather than as the justification of the main claim, so they are not load-bearing circularity. The Limitations section explicitly acknowledges that 'Our approach to evaluating Fisher information via MMLU also asks for a theoretical justification,' but a missing theoretical justification is a limitation rather than a circular step. The identifiable circularity is limited to the MMLU-in-evaluation overlap, which inflates the 7/7 count but does not by itself determine the other six results.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. Its central claim rests on the choice of perplexity as the fluency metric, the benchmark suite as domain knowledge, the empirical Fisher approximation, the specific data subset, and the reference model for Lithuanian; all are standard assumptions in this literature.

free parameters (1)
  • Regularisation strength lambda (range selection) = 10^2 to 10^11 (post hoc)
    The paper concludes EWC helps based on performance in this range, selected after inspecting the sweep; the authors state optimal lambda selection is open.
assumptions (5)
  • domain assumption Perplexity on TruthfulQA and Lithuanian Q/A datasets measures linguistic fluency.
    Section 3 defines fluency via perplexity; this equates a model property with a specific metric.
  • domain assumption Accuracy on ARC, Belebele, GSM8K, HellaSwag, MMLU, TruthfulQA, and Winogrande measures domain knowledge.
    Section 3 lists these benchmarks as proxies for domain knowledge.
  • standard math The empirical Fisher estimator approximates the true Fisher information.
    They use the empirical Fisher estimator from [17], which is a standard approximation.
  • domain assumption The first 10% of Lithuanian CulturaX is sufficient for fluency improvement.
    The paper uses only this subset and later flags limited data as a limitation.
  • domain assumption LT-Llama2-13B is a valid reference for Lithuanian grammatical accuracy.
    Used as a perplexity reference in Figure 2; relies on external claim [23].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge." pith.science (2026). https://pith.science/paper/76AWD4Y2

@misc{pith2026250505946,
  author       = {Pith},
  title        = {Pith review of: Full-Parameter Continual Pretraining of Gemma2: Insights into Fluency and Domain Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/76AWD4Y2}},
  note         = {Machine review of arXiv:2505.05946}
}
read the original abstract

In this technical report, we empirically investigate the relationship between linguistic fluency and domain knowledge in the context of continual learning with large language models (LLMs). Specifically, we enhance the linguistic fluency of the Gemma2 LLM for the Lithuanian language by autoregressively pretraining its full parameter set on the first 10\% of the Lithuanian language component of the CulturaX dataset. To prevent catastrophic forgetting of the model's existing domain knowledge, we apply Elastic Weight Consolidation (EWC), leveraging Fisher information estimated using data from the Massive Multitask Language Understanding (MMLU) benchmark. In the post-training evaluations, we assess linguistic fluency through perplexity and evaluate domain knowledge using accuracy on a suite of language understanding benchmarks, including ARC-Easy, Belebele, GSM8K, HellaSwag, MMLU, TruthfulQA, and Winogrande, in both English and Lithuanian. The empirical results demonstrate that EWC not only mitigates catastrophic forgetting by preserving the model's performance in terms of both linguistic fluency and domain knowledge but also improves or maintains these capabilities for the newly added Lithuanian language. These findings highlight the potential for more efficient adaptation of general-purpose LLMs to under-represented languages without requiring access to the original training data. The accompanying codebase is openly accessible at https://github.com/Neurotechnology/LLM_EWC.

Figures

Figures reproduced from arXiv: 2505.05946 by the authors.

Figure 1
Figure 1. Comparison of perplexity and average accuracy in domain understanding tasks with varying regularisation [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Perplexity of the answer (evaluated with LT-Llama-13B) versus regularisation strength [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Accuracy of the models versus EWC regularisation strength [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 21 canonical work pages

  1. [1]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al. Attention is all you need. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017

    work page 2017

  2. [2]

    Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansion, 2025

    Mingyang Wang, Alisa Stoll, Lukas Lange, Heike Adel, Hinrich Schütze, and Jannik Strötgen. Bring Your Own Knowledge: A Survey of Methods for LLM Knowledge Expansion, 2025

    work page 2025

  3. [3]

    Continual learning of large language models: A comprehensive survey.arXiv preprint arXiv:2404.16789, 2024

    Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey.arXiv preprint arXiv:2404.16789, 2024

    arXiv 2024

  4. [4]

    The MIT Press, Cambridge, 1965

    Noam Chomsky.Aspects of the Theory of Syntax. The MIT Press, Cambridge, 1965

    work page 1965

  5. [5]

    Overcoming Catastrophic Forgetting in Neural Networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming Catastrophic Forgetting in Neural Networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    work page 2017

  6. [6]

    Breaking Language Barriers: Cross- Lingual Continual Pre-Training at Scale.arXiv preprint arXiv:2407.02118, 2024

    Wenzhen Zheng, Wenbo Pan, Xu Xu, Libo Qin, Li Yue, and Ming Zhou. Breaking Language Barriers: Cross- Lingual Continual Pre-Training at Scale.arXiv preprint arXiv:2407.02118, 2024

    arXiv 2024

  7. [7]

    Fine-tuned language models are continual learn- ers.arXiv preprint arXiv:2205.12393, 2022

    Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learn- ers.arXiv preprint arXiv:2205.12393, 2022

    arXiv 2022

  8. [8]

    Simple and scalable strategies to continually pre-train large language models.arXiv preprint arXiv:2403.08763, 2024

    Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and scalable strategies to continually pre-train large language models.arXiv preprint arXiv:2403.08763, 2024

    arXiv 2024

Show all 32 references
  1. [9]

    LAMOL: LAnguage MOdeling for Lifelong Language Learn- ing.arXiv preprint arXiv:1909.03329, 2019

    Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. LAMOL: LAnguage MOdeling for Lifelong Language Learn- ing.arXiv preprint arXiv:1909.03329, 2019

    1909 arXiv

  2. [10]

    Learning Without Forgetting

    Zhizhong Li and Derek Hoiem. Learning Without Forgetting. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors,Computer Vision – ECCV 2016, pages 614–629, Cham, 2016. Springer International Publishing

    2016

  3. [11]

    Learning to solve NLP tasks in an incremental number of languages

    Giuseppe Castellucci, Simone Filice, Danilo Croce, and Roberto Basili. Learning to solve NLP tasks in an incremental number of languages. InProceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on ...

    2021

  4. [12]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Repre- sentations, 2022. 6

    2022

  5. [13]

    CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation, 2024

    Muhammad Fawi. CURLoRA: Stable LLM Continual Fine-Tuning and Catastrophic Forgetting Mitigation, 2024

    2024

  6. [14]

    Language models meet world models: Embodied experiences enhance language models.Advances in Neural Information Process- ing Systems, 36:75392–75412, 2023

    Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models.Advances in Neural Information Process- ing Systems, 36:75392–75412, 2023

    2023

  7. [15]

    Overcoming catastrophic forgetting in massively multilingual continual learning.arXiv preprint arXiv:2305.16252, 2023

    Genta Indra Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preotiuc-Pietro. Overcoming catastrophic forgetting in massively multilingual continual learning.arXiv preprint arXiv:2305.16252, 2023

    2023 arXiv

  8. [16]

    Unifying Importance Based Regularisation Methods for Continual Learning

    Frederik Benzing. Unifying Importance Based Regularisation Methods for Continual Learning. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors,Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 ofProceedin...

  9. [17]

    van de Ven

    Gido M. van de Ven. On the Computation of the Fisher Information in Continual Learning. InICLR Blogposts 2025, 2025. https://d2jud02ci9yv69.cloudfront.net/2025-04-28-fisher-120/blog/fisher/

    2025

  10. [18]

    Examining forgetting in continual pre-training of aligned large language models

    Chen-An Li and Hung-Yi Lee. Examining forgetting in continual pre-training of aligned large language models. arXiv preprint arXiv:2401.03129, 2024

    2024 arXiv

  11. [19]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  12. [20]

    Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.arXiv preprint arXiv:2309.09400, 2023

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A Rossi, and Thien Huu Nguyen. Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages.arXiv preprint arXiv:2309.09400, 2023

    2023 arXiv

  13. [21]

    Open Llama2 Model for the Lithuanian Language

    Art ¯uras Nakvosas, Povilas Daniušis, and Vytas Mulevi ˇcius. Open Llama2 Model for the Lithuanian Language. arXiv preprint arXiv:2408.12963, 2024

    2024 arXiv

  14. [22]

    Open Llama2 Models for the Lithuanian Language

    Art ¯uras Nakvosas, Povilas Daniušis, and Vytas Muleviˇcius. Open Llama2 Models for the Lithuanian Language. Informatica, pages 1–22, 2025

    2025

  15. [23]

    Localizing AI: Evaluating Open-Weight Lan- guage Models for Languages of Baltic States.arXiv preprint arXiv:2501.03952, 2025

    Jurgita Kapo ˇci¯ut˙e-Dzikien˙e, Toms Bergmanis, and M¯arcis Pinnis. Localizing AI: Evaluating Open-Weight Lan- guage Models for Languages of Baltic States.arXiv preprint arXiv:2501.03952, 2025

    2025 arXiv

  16. [24]

    Align- ing AI With Shared Human Values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Align- ing AI With Shared Human Values.Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  17. [25]

    Measuring Massive Multitask Language Understanding.Proceedings of the International Conference on Learn- ing Representations (ICLR), 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding.Proceedings of the International Conference on Learn- ing Representations (ICLR), 2021

    2021

  18. [26]

    The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

    Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. InProceedings of t...

    2024

  19. [27]

    Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  20. [28]

    HellaSwag: Can a Machine Re- ally Finish Your Sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Re- ally Finish Your Sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  21. [29]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.arXiv:1803.05457v1, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, and Tushar Khot and. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.arXiv:1803.05457v1, 2018

    2018 arXiv

  22. [30]

    TruthfulQA: Measuring How Models Mimic Human Falsehoods, 2021

    Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods, 2021

    2021

  23. [31]

    WinoGrande: An Adversarial Wino- grad Schema Challenge at Scale

    Sakaguchi Keisuke, Le Bras Ronan, Bhagavatula Chandra, and Choi Yejin. WinoGrande: An Adversarial Wino- grad Schema Challenge at Scale. 2019. 7

    2019

  24. [32]

    Fine-tuned

    Hila Gonen, Srini Iyer, Terra Blevins, Noah Smith, and Luke Zettlemoyer. Demystifying prompts in language models via perplexity estimation. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10136–...

    2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.