Pith. sign in

REVIEW 5 major objections 6 minor 54 references

NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Standard scientific benchmarks go flat and noisy for small language models during their first 200 billion tokens, while a completion-style rewrite of MMLU produces smooth, monotonic curves that rank models by size and architecture.

desk verdict A useful competition baseline, but the claim that completion-style prompts fix early-training benchmarks is confounded and under-powered as presented. read the letter →

arxiv 2506.07731 v1 pith:56YEXXLB submitted 2025-06-09 cs.AI

Mouadh Yagoubi , Yasser Dahou , Billel Mokeddem , Younes Belkada , Phuc H. Le-Khac , Basma El Amel Boussaha , Reda Alami , Jingwei Zuo
show 7 more authors
Damiano Marsili Mugariya Farooq Mounia Lalmas Georgia Gkioxari Patrick Gallinari Philip Torr Hakim Hacid
This is my paper
classification cs.AI
keywords EarlytraininganalysisEvaluationbenchmarksScientificknowledgeLow-resourceMLresearchLanguageModelsSmallMMLU-varPromptformat
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that state-of-the-art benchmarks such as MMLU, GPQA, MATH, and MuSR stop being informative during the first 200 billion tokens of small-model training: their learning curves are noisy and non-discriminative, so a 0.5B model can look as strong as a 3B model. It then shows that a simple reformatting of MMLU, presenting the question as a completion prompt and scoring each answer by its log-probability, produces smooth, monotonically increasing curves that rank models by size and also separate architectures of the same size. The paper reads this as evidence that small models do acquire scientific knowledge early, but that multiple-choice decision-making understates it, and it builds a competition around designing evaluation tasks that give strong early-training signal, stable model rankings up to 1 trillion tokens, and measurable sensitivity to scientific knowledge. A sympathetic reader would care because reliable early-training evaluation would let developers make data-mixture, architecture, and hyperparameter decisions from small checkpoints instead of waiting for full-scale runs.

What carries the argument

The carrying mechanism is MMLU-var, a completion-style variant of MMLU: each question is presented without answer choices and is followed by the token "Answer:", and each candidate answer is evaluated separately by the log-probability it receives as a continuation of that prompt, with the highest-likelihood candidate chosen. This replaces the multiple-choice decision procedure, which the paper argues small models master only late in training, with a cloze-style likelihood comparison that can reflect knowledge the model has not yet learned to deploy in a forced-choice setting. The same design philosophy extends to the competition's scoring apparatus: Signal Quality rewards learning curves that are monotone and autocorrelated, Ranking Consistency rewards evaluations whose architecture rankings at 200 billion tokens persist up to 1 trillion tokens, and Scientific Compliance rewards tasks whose scores separate a knowledge-mixture-trained model from a web-only-trained model.

What would settle it

Run the same MMLU questions and the same checkpoints through both formats with identical scoring: if the completion-style curves are no smoother, or the multiple-choice curves become smooth, the format explanation is falsified. A quantitative version would compare the monotonicity and autocorrelation scores of the two formats across all six checkpoints; if the differences disappear under controlled scoring, the central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that the format of an evaluation can determine whether small models show measurable progress early in training. On the standard MMLU multiple-choice benchmark, the six 0.5B-3B checkpoints produce noisy curves that do not consistently separate model sizes or architectures. On MMLU-var, where each candidate answer is scored by its log-probability as a continuation after "Answer:", scores rise smoothly and monotonicity is strong, models rank by size, and architectural variants within a size become distinguishable. The paper also reports that a 1B model trained on a curated scientific data mixture matches a web-only model on HellaSwag yet clearly outperforms it on MMLU-var, which it takes as evidence that scientific knowledge is being acquired and can be detected early when the probe matches the model's emerging behavior.

Load-bearing premise

The load-bearing premise is that changing MMLU from a multiple-choice question to a completion-style prompt is what produces the cleaner early-training signal; the paper never runs an ablation that varies only the prompt format while holding the scoring function, question set, and checkpoints fixed, so the improvement could in principle come from another part of the change.

Editorial extensions

If this is right

  • If the central claim is right, evaluation of small models should treat prompt format as a first-class design choice: multiple-choice decision procedures can hide knowledge that completion-style scoring reveals.
  • The clean separation between a 1B model trained on a curated scientific mixture and one trained on web-only data, on MMLU-var but not HellaSwag, gives developers a checkpoint-level way to compare data mixtures long before a full training run ends.
  • Grade-school-level scientific benchmarks such as ARC-Easy and SciQ join MMLU-var as usable early-training signals, with SciQ needing leakage filtering because the correct answer appears verbatim in 82% of its prompts.
  • The proposed scoring recipe, signal quality, ranking consistency, and scientific-knowledge compliance, lets organizers rank candidate tasks numerically; in their table MMLU-var achieves the top total score (71.7%), ahead of ARC-Easy (65.5%) and SciQ (62.7%).
  • Early-stage benchmark design, if credible, would make foundational LLM research more benchmark-informed: model developers could use small checkpoints to guide architecture and data decisions instead of waiting for 1-trillion-token runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper never isolates prompt format from scoring and question selection, so a controlled format-only comparison is the natural next experiment; if it fails, the cleaner MMLU-var curves may come from a different component of the evaluation change.
  • Editorial inference: if the format effect is causal, benchmark difficulty is not a fixed property of a question set but a function of the model's output distribution, so early-training evaluation may need formats that adapt as models develop.
  • Editorial inference: the same completion-style reformulation could be applied to other multiple-choice science benchmarks, such as GPQA, MMLU-Pro, and ARC, to check whether the effect transfers beyond MMLU.
  • Editorial inference: the competition's ranking-consistency metric compares only two architectures per size as a binary arch1-versus-arch2 decision, so its rankings are coarse; adding more architectures or using continuous rank measures would give a finer test of early-training signal.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. This paper, framed as a NeurIPS 2025 competition proposal, presents baseline experiments on three small language models (0.5B, 1B, and 3B) with two architecture variants each, using checkpoints up to 200B tokens. The authors report that standard benchmarks such as MMLU, GPQA, MATH, and MuSR produce noisy and non-discriminative learning curves in early training, while a completion-style variant of MMLU (MMLU-var) gives smoother curves that rank models by size and architecture. They further report that a 1B model trained on a knowledge-rich data mixture outperforms a web-only model on MMLU-var but not on HellaSwag. The paper defines a composite scoring metric (Signal Quality, Ranking Consistency, and Scientific-Knowledge Compliance) intended to rank participant submissions, along with compliance and leakage checks. The contribution is primarily organizational: it provides models, checkpoints, a starting kit, and a baseline evaluation protocol for a community challenge.

Significance. If its empirical claims held, the paper would address a real gap: standard multiple-choice benchmarks are poorly calibrated for early SLM training, and a simple continuation-format variant may provide more informative learning curves. The public release of models, checkpoints, and a reproducible starting kit is a genuine strength, as is the inclusion of raw curves in Appendices A and E. The proposed competition is also feasible on free-tier GPU resources, broadening participation. However, the central comparison currently rests on a confounded prompt/scoring change, only four checkpoints per model, and internally inconsistent metric definitions. With additional controlled experiments and corrected scoring, the framework could become a useful baseline for the community.

major comments (5)
  1. [Appendix B / Section 1.4] Appendix B shows that the standard MMLU prompt includes the list of A-D choices and is scored as a selection among letter choices, while MMLU-var removes that list and instead scores each full answer string by its continuation log-likelihood. Thus the comparison in Figure 2 and Table 1 changes the prompt and the scoring rule simultaneously. Section 1.4 attributes the improvement to early SLMs' difficulty in 'explicitly compare and reason over multiple-choices in the input space,' but no ablation isolates the presence of the choice list while holding the question set, answer candidates, and scoring protocol fixed. The headline claim that simple prompt modifications fix benchmark informativeness is therefore not yet supported; an additional experiment varying only the prompt format is needed.
  2. [Section 1.5, Eq. (4); Figures 1-3] With only four checkpoints per model (50, 100, 150, 200 BT), the autocorrelation in Eq. (4) is computed at a single lag L = floor(n/4) = 1, and the monotonicity score in Eq. (2) is a Spearman correlation on four points. Such small samples cannot distinguish a genuinely smooth curve from a noisy one, and no error bars, repeated seeds, or significance tests are reported anywhere in the paper. The Signal Quality column in Table 1, which dominates the total score through α1 = 0.5, is therefore not statistically grounded. The authors should either evaluate more checkpoints, report bootstrap or permutation intervals, or weaken the 'smooth vs noisy' contrast accordingly.
  3. [Section 1.5, Eq. (8); Table 1] ScoreCS in Eq. (8) is defined as the normalized gap between two organizer-trained 1B models, one trained on the scientific-knowledge mixture and one on web-only data. Using this same ScoreCS to rank benchmarks in Table 1 and Appendix G, and then concluding that scientific-knowledge benchmarks 'consistently outperform' general-language ones, is at least partly circular: the metric is constructed to reward benchmarks that separate these two particular checkpoints, so HellaSwag's zero reflects the fact that the two chosen models tie on HellaSwag (Figure 3), not an independent property of commonsense tasks. Validation on held-out model pairs or against an external criterion would make the compliance ranking informative rather than self-fulfilling.
  4. [Section E.4 vs Section 1.5] The global scoring weights are inconsistent across the paper. Section 1.5 defines Score = 0.5*ScoreSQ + 0.1*ScoreRC + 0.4*ScoreCS, while Section E.4 states 'setting αSQ = 0.4, αRC = 0.2, and αCS = 0.4' and then calculates the total score using 0.5, 0.1, 0.4. Because the leaderboard total is the competition's official ranking criterion, this contradiction must be resolved and all table values recomputed from a single authoritative weight vector.
  5. [Figure 3 caption vs Section 1.4] Figure 3's caption says the comparison is 'from a 0.5B parameter model,' but Section 1.4 says 'we trained a 1B model,' and Appendix E.3 describes the compliance calculation with two 1B models. The data-mixture conclusion depends on which size was actually used; this inconsistency must be corrected before the result can be interpreted.
minor comments (6)
  1. [Throughout] Many typos and formatting errors should be cleaned up: 'bilion' and 'trilion' in Section 1.5, 'netbooks' in Section 1.7, 'close-style' for 'cloze-style' in Section 1.1, 'expect for' in the Figure 2 caption, and 'datamixture' in several places.
  2. [Section 1.5 and Table 6] Notation is inconsistent: 'Score SQ', 'ScoreSQ', and 'Score_SQ' are used interchangeably; the scientific-compliance component is called ScoreCS in Section 1.5 but 'SC' in Table 6; and 'BT', 'billion tokens', '1TT', and '1 trillion tokens' are all used without being defined as synonyms.
  3. [Figure 4 / Appendix A] Figure 4 and Appendix A use '500M' for the 0.5B models, while the main text and Table 2 use '0.5B'; the model names in the figure legends should be brought into agreement.
  4. [Equation (2)] Equation (2) states that Spearman's rank correlation uses rank differences d_j between iteration indices and scores, but d_j is not explicitly defined as the difference of the two rank sequences; please spell out the definition or cite a standard reference.
  5. [Appendix E.1, Figure 5 caption] The caption of Figure 5 refers to 'SC' as ScoreMonotonicity, which conflicts with the use of 'SC' for Scientific Compliance elsewhere; rename to avoid ambiguity.
  6. [Section 1.4] Section 1.4 mentions that HellaSwag and WinoGrande provide meaningful signals, but the quantitative support appears only in Appendix A; a pointer to Figure 4 would help the reader.

Circularity Check

1 steps flagged · score 3.0 of 10

ScoreCS defines 'scientific compliance' as a data-mixture gap, making part of the benchmark ranking definitional; raw curves keep the central MMLU-vs-MMLU-var comparison independent.

  1. self definitional [Section 1.5, Eq. (8); 'Metric Scores Analysis' and Table 1]
    "To quantify domain compliance, we compare the average performance of two 1B models across training steps: a model trained on the scientific knowledge-focused datamixture ( xSciKW-DS) and a model trained on web-only data (xWeb-DS). The score is define as the normalized performance gap: ScoreCS = max(0, 1/n Σ (xSciKW-DS_i − xWeb-DS_i)) ... HellaSwag receives a zero on the Compliance metric, which evaluates whether the benchmark is classified as scientific knowledge (central to this competition)."

    By Eq. (8), ScoreCS is literally the average score gap between the two organizer-trained 1B models (scientific-mix vs web-only). Calling this gap 'Compliance to Reasoning and Knowledge Domains' and then using it as 40% of the total score to conclude that 'Benchmarks targeting scientific knowledge consistently outperform HellaSwag' makes a substantial part of the ranking a restatement of the definition: a benchmark earns scientific-compliance credit exactly when it separates the two self-trained models. The statement that HellaSwag 'receives a zero on the Compliance metric, which evaluates whether the benchmark is classified as scientific knowledge' is also definition-driven, since the metric does not classify benchmark content but measures a score gap.

full rationale

The paper's principal empirical observation—that MMLU, GPQA, MATH, and MuSR give noisy/non-discriminative curves during early training while MMLU-var is smoother and more monotonic—is supported by raw checkpoint curves in Figures 1-3 and does not reduce to the scoring equations. No self-citation chain is load-bearing: MMLU-var is attributed to the external OlMoE work [15], and no uniqueness theorem or ansatz is imported from the authors' prior work. The competition's SQ and RC metrics are transparently defined statistical summaries, and RC even uses hidden post-200BT checkpoints, so those components are not fitted inputs. The only definitional short-circuit is ScoreCS: Eq. (8) equates 'scientific knowledge compliance' with the performance gap between two organizer-trained data-mixture models, and this constructed gap is then used to rank benchmarks and to declare scientific benchmarks superior to HellaSwag. Because raw curves and SQ/RC provide independent content, the paper is only partially circular rather than wholly so. The separate MMLU-vs-MMLU-var confound (prompt format and scoring protocol changing together, only four checkpoints) is a validity threat, not a circularity, and is therefore noted but not scored as a circular step.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims and the competition's scoring system depend on several hand-set constants and assumptions about what makes a benchmark scientifically informative. These are design choices, not derived results.

free parameters (4)
  • Global score weights α1, α2, α3 = 0.5, 0.1, 0.4
    Set by organizers in Section 1.5; directly determine final rankings in Table 6.
  • Signal quality weights β1, β2 = 0.5, 0.5
    Set in Section 1.5; combine monotonicity and autocorrelation subscores.
  • Compliance pre-check threshold = 80% Accept
    Section F.1: benchmark is auto-compliant if over 80% of questions are classified Accept by gpt-4o.
  • Data mixture proportions = 50% FineWeb-edu, 21.6% The Stack, 18.9% InfiMM, 9.5% TxT360
    Section 1.3; determines the knowledge-rich training set used to define ScoreCS.
assumptions (5)
  • domain assumption The completion-style prompt format (MMLU-var) is the cause of the improved learning curves, not other factors such as question selection or scoring method.
    Section 1.4 and Figure 2 compare MMLU and MMLU-var, but no controlled ablation is provided.
  • domain assumption The 1B model pair (knowledge-mix vs web-only) provides a valid probe for scientific knowledge sensitivity (ScoreCS, Eq. 8).
    The entire Compliance subscore depends on this pair; its representativeness is assumed.
  • ad hoc to paper Weights α1=0.5, α2=0.1, α3=0.4 and β1=β2=0.5 are reasonable for combining the scores.
    Chosen by organizers without sensitivity analysis (Section 1.5).
  • domain assumption A 4-point learning curve (checkpoints at 50, 100, 150, 200B tokens) is sufficient to characterize early training dynamics.
    All figures use only four checkpoint evaluations per model.
  • domain assumption The gpt-4o classifier's Accept/Reject labels correctly identify scientific knowledge benchmarks.
    Used for the compliance pre-check (Section F.1), calibrated on a handful of benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models." pith.science (2026). https://pith.science/paper/56YEXXLB

@misc{pith2026250607731,
  author       = {Pith},
  title        = {Pith review of: NeurIPS 2025 E2LM Competition : Early Training Evaluation of Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/56YEXXLB}},
  note         = {Machine review of arXiv:2506.07731}
}
read the original abstract

Existing benchmarks have proven effective for assessing the performance of fully trained large language models. However, we find striking differences in the early training stages of small models, where benchmarks often fail to provide meaningful or discriminative signals. To explore how these differences arise, this competition tackles the challenge of designing scientific knowledge evaluation tasks specifically tailored for measuring early training progress of language models. Participants are invited to develop novel evaluation methodologies or adapt existing benchmarks to better capture performance differences among language models. To support this effort, we provide three pre-trained small models (0.5B, 1B, and 3B parameters), along with intermediate checkpoints sampled during training up to 200B tokens. All experiments and development work can be run on widely available free cloud-based GPU platforms, making participation accessible to researchers with limited computational resources. Submissions will be evaluated based on three criteria: the quality of the performance signal they produce, the consistency of model rankings at 1 trillion tokens of training, and their relevance to the scientific knowledge domain. By promoting the design of tailored evaluation strategies for early training, this competition aims to attract a broad range of participants from various disciplines, including those who may not be machine learning experts or have access to dedicated GPU resources. Ultimately, this initiative seeks to make foundational LLM research more systematic and benchmark-informed from the earliest phases of model development.

Figures

Figures reproduced from arXiv: 2506.07731 by the authors.

Figure 1
Figure 1. Noisy results obtained with state-of-the-art benchmarks with different models sizes. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. MMLU-var (left) vs MMLU (right) comparison on different configurations and sizes of models. Different col￾ors for each size of the model (0.5B, 1B and 3B) and markers distinguish experiments within the same category. MMLU-var al￾lows a clear comparison between variants while MMLU is giving noise (expect for 3B arch2). We observe that results on MMLU do not consistently differentiate between models of varying sizes a… view at source ↗
Figure 3
Figure 3. Comparison of results from a 0.5B parameter model trained on two datasets — Web-Only and Knowledge Data￾Mixture. While both model con￾figurations achieve similar perfor￾mance on the HellaSwag bench￾mark, the model trained on the Knowledge Data-Mixture signifi￾cantly outperforms the Web-Only model on MMLU-var. 1.5 Metrics We propose to evaluate submitted solutions based on three main criteria, which will be combined … view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Results of state-of-the-art benchmarks with different models sizes. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: Signal quality metric calculated for different benchmarks using 1B model. (SC corresponds [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Ranking Consistency metric calculated for different benchmarks using 0.5B model. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Compliance to scientific knowledge domains metric calculated for different benchmarks [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]
Figure 8
Figure 8. Figure 8: Validation loss across different datasets for two model variants: Scientific-DataMix (blue) [PITH_FULL_IMAGE:figures/full_fig_p027_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 25 canonical work pages

  1. [1]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    arXiv 2024

  2. [2]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    arXiv 2024

  3. [3]

    Falcon2-11b technical report.arXiv preprint arXiv:2407.14885, 2024

    Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campesan, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, et al. Falcon2-11b technical report.arXiv preprint arXiv:2407.14885, 2024

    arXiv 2024

  4. [4]

    A survey of large language models

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2), 2023

    arXiv 2023

  5. [5]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    arXiv 1905

  6. [6]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021

    2021

  7. [7]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  8. [8]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

    2024

Show all 54 references
  1. [9]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024

    2024

  2. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  3. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021

    2021 arXiv

  4. [12]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  5. [13]

    What is the role of small models in the llm era: A survey

    Lihu Chen and Gaël Varoquaux. What is the role of small models in the llm era: A survey. arXiv preprint arXiv:2409.06857, 2024. 9

    2024 arXiv

  6. [14]

    Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer

    Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022

    2022 arXiv

  7. [15]

    Olmoe: Open mixture-of-experts language models

    Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024

    2024 arXiv

  8. [16]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018

    2018 arXiv

  9. [17]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 94–106, Copenhagen, Denmark, September 2017....

    2017

  10. [18]

    Toward an evaluation science for generative ai systems

    Laura Weidinger, Deb Raji, Hanna Wallach, Margaret Mitchell, Angelina Wang, Olawale Salaudeen, Rishi Bommasani, Sayash Kapoor, Deep Ganguli, Sanmi Koyejo, et al. Toward an evaluation science for generative ai systems. arXiv preprint arXiv:2503.05336, 2025

    2025 arXiv

  11. [19]

    Efficient large language models: A survey

    Zhongwei Wan, Xin Wang, Che Liu, Samiul Alam, Yu Zheng, Jiachen Liu, Zhongnan Qu, Shen Yan, Yi Zhu, Quanlu Zhang, et al. Efficient large language models: A survey. arXiv preprint arXiv:2312.03863, 2023

    2023 arXiv

  12. [20]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  13. [21]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  14. [22]

    A survey on large language models: Applications, challenges, limitations, and practical usage

    Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 3, 2023

    2023

  15. [23]

    Llm merging: Building llms efficiently through merging

    Derek Tam, Margaret Li, Prateek Yadav, Rickard Brüel Gabrielsson, Jiacheng Zhu, Kristjan Greenewald, Mikhail Yurochkin, Mohit Bansal, Colin Raffel, and Leshem Choshen. Llm merging: Building llms efficiently through merging. In NeurIPS 2024 Competition Track, 2024

    2024

  16. [24]

    Edge-llms: Edge-device large language model competition

    Shiwei Liu, Kai Han, Adriana Fernandez-Lopez, AJAY KUMAR JAISW AL, Zahra Atashgahi, Boqian Wu, Edoardo Ponti, Callie Hao, Rebekka Burkholz, Olga Saukh, et al. Edge-llms: Edge-device large language model competition. In NeurIPS 2024 Competition Track, 2024

    2024

  17. [25]

    https://llm-efficiency- challenge.github.io/, 2024

    Neurips large language model efficiency challenge: 1 llm + 1gpu + 1day. https://llm-efficiency- challenge.github.io/, 2024

    2024

  18. [26]

    Stages and individual differences in cognitive development

    Kurt W Fischer and Louise Silvern. Stages and individual differences in cognitive development. Annual review of psychology, 36(1):613–648, 1985

    1985

  19. [27]

    The new taxonomy of educational objectives

    Robert J Marzano and John S Kendall. The new taxonomy of educational objectives. Corwin Press, 2006

    2006

  20. [28]

    Taxonomy ofeducational objectives the clas- sification ofeducational goals

    David R Krathwohl, B Bloom, and B Masia. Taxonomy ofeducational objectives the clas- sification ofeducational goals. Handbook II: Affective Domain.(New York: McKay, 1964.), 1964

    1964

  21. [29]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro V on Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale. In The Thirty-eight Conference on Neural Information Processing ...

    2024

  22. [30]

    Fineweb-edu: the finest collection of educational content, 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024

    2024

  23. [31]

    The stack: 3 tb of permissively licensed source code

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Fer- randis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. The stack: 3 tb of permissively licensed source code. Preprint, 2022

    2022

  24. [32]

    Starcoder 2 and the stack v2: The next generation, 2024

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul,...

    2024

  25. [33]

    Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning, 2024

    Xiaotian Han, Yiren Jian, Xuefeng Hu, Haogeng Liu, Yiqi Wang, Qihang Fan, Yuang Ai, Huaibo Huang, Ran He, Zhenheng Yang, and Quanzeng You. Infimm-webmath-40b: Advancing multimodal pre-training for enhanced mathematical reasoning, 2024

    2024

  26. [34]

    Liping Tang, Nikhil Ranjan, Omkar Pangarkar, Xuezhi Liang, Zhen Wang, Li An, Bhaskar Rao, Linghao Jin, Huijuan Wang, Zhoujun Cheng, Suqi Sun, Cun Mu, Victor Miller, Xuezhe Ma, Yue Peng, Zhengzhong Liu, and Eric P. Xing. Txt360: A top-quality llm pre-training dataset requires t...

    2024

  27. [35]

    Physics of language models: Part 3.1, knowledge storage and extraction

    Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.1, knowledge storage and extraction. arXiv preprint arXiv:2309.14316, 2023

    2023 arXiv

  28. [36]

    M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81–93, 1938

    1938

  29. [37]

    Finetasks: Finding signal in a haystack of 200+ multilingual tasks

    Hynek Kydlíˇcek, Guilherme Penedo, Clémentine Fourier, Nathan Habib, and Thomas Wolf. Finetasks: Finding signal in a haystack of 200+ multilingual tasks

  30. [38]

    Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform

    Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard, Wei-Wei Tu, Quanming Yao, Huan Zhao, and Isabelle Guyon. Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform. Patterns, 3(7), 2022

    2022

  31. [39]

    The LAMBADA dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of t...

    2016

  32. [40]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020

    2020

  33. [41]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023

    2023

  34. [42]

    RACE: Large-scale ReAding comprehension dataset from examinations

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785–794, Copenhagen, Denmark, September 2017. A...

    2017

  35. [43]

    Semantic parsing on Freebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October 2013. Association ...

    2013

  36. [44]

    Challenging big-bench tasks and whether chain-of-thought can solve them

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022

    arXiv

  37. [45]

    DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019

    2019

  38. [46]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019

    2019

  39. [47]

    Adversarial nli: A new benchmark for natural language understanding

    Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational L...

    2020

  40. [48]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  41. [49]

    Mask R-CNN

    Zayne Rea Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In The Twelfth International Conference on Learning Representations, 2024. 3 Resources 3.1 Organizing team The team proposing ...

    2021

  42. [51]

    26 Figure 8: Validation loss across different datasets for two model variants: Scientific-DataMix (blue) and webOnly-Data (red)

    Domain transfer gap: While FineWebEdu (a subset of the training data for Web-Only) shows a small gap between variants, all other datasets exhibit substantially larger gaps, often exceeding the total reduction in loss throughout training, indicating significant out-of- distribu...

  43. [52]

    Lower loss on one dataset does not necessarily translate to higher capability on corresponding benchmarks

    Cross-dataset comparison challenges: Each dataset produces distinct numerical loss ranges due to inherent characteristics, making direct comparison across domains difficult. Lower loss on one dataset does not necessarily translate to higher capability on corresponding benchmarks

  44. [53]

    Loss interpretation limitations: The small numerical differences in loss values throughout training are difficult to translate into meaningful capability improvements, particularly when averaging across datasets with different baseline values

  45. [54]

    Early training ambiguity: Despite Scientific-Mix eventually showing lower loss on FineWebEdu compared to WebOnly, both variants exhibit similar loss trajectories in early training stages, making it challenging to distinguish performance differences during this critical period....

  46. [2013]

    and vice director for 6 years before, He also acted as vice director of the scientific committee of the faculty of engineering at UPMC (2010 to 2021). Philip Torr (Scientific Advisor) Professor Philip Torr did his PhD (DPhil) at the Robotics Research Group of the University of...

    2010

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.