Pith. sign in

REVIEW 3 major objections 4 minor 4 cited by

Establishing Task Scaling Laws via Compute-Efficient Model Ladders

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Task accuracy of large overtrained language models can be predicted from a small fixed set of 'ladder' models via a two-step parametric pipeline, landing within 2 absolute points on four multiple-choice benchmarks for under 1% of the…

desk verdict A solid empirical paper that does something genuinely new—predicting individual task accuracy from a 1%-compute ladder—but the headline 'within 2 points' rests on a sigmoid extrapolation that is thinner than the paper's tone suggests. read the letter →

arxiv 2412.04403 v2 pith:JROKZP6P submitted 2024-12-05 cs.CL cs.AI

Akshita Bhagia , Jiacheng Liu , Alexander Wettig , David Heineman , Oyvind Tafjord , Ananya Harsh Jha , Luca Soldaini , Noah A. Smith
show 4 more authors
Dirk Groeneveld Pang Wei Koh Jesse Dodge Hannaneh Hajishirzi
This is my paper · ORCID
classification cs.CLcs.AI
keywords taskscalinglawsmodelladdersovertrainedlanguagemodelslawextrapolationaccuracypredictionrankedclassificationtwo-stepcompute-efficientevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that the downstream task accuracy of a large, overtrained language model — the score a 7B or 13B model would get on a benchmark like MMLU or HellaSwag — can be predicted before the model is trained, using only a small 'ladder' of cheap models. The method trains 16 small models (190M to 1.3B parameters, each trained on 1x to 10x its Chinchilla-optimal data budget) and fits two curves: a power law that maps parameter count $N$ and token budget $D$ to a task-specific loss, and a sigmoid that maps that loss to task accuracy. Predicting for two held-out targets — a 7B model trained on roughly 4T tokens and a 13B model on 5T tokens — the chained curves land within 2 absolute accuracy points on four of eight tasks (MMLU, HellaSwag, PIQA, SocialIQA) and within about 4 points on average, at a cost of about 1% of the target compute. The reason to care: if the claim holds, pretraining decisions that currently require a full expensive run — how much data, what mixture, what model size — become cheap, testable predictions on the same benchmarks used to judge the models.

What carries the argument

The machinery is the model ladder plus the two fitted curves it feeds. The ladder is a fixed grid of 16 models — four sizes ($N \in \{190\text{M}, 370\text{M}, 760\text{M}, 1.3\text{B}\}$ non-embedding parameters) by four data budgets ($D \in \{1\text{x}, 2\text{x}, 5\text{x}, 10\text{x}\}$ of the Chinchilla-optimal amount) — sharing the target models' architecture, data mixture, and learning-rate schedule, so that scale, not recipe, is the only difference being extrapolated. Step 1 fits the power law $L(N, D) = A/N^\alpha + B/D^\beta + E$ to the final-checkpoint task losses, and step 2 fits the sigmoid $\mathrm{Acc}(L) = a/(1 + e^{-k(L-L_0)}) + b$ to the loss-accuracy pairs of roughly 1,400 ladder checkpoints. The load-bearing move is that both curves are fitted on small models and then evaluated by extrapolation along $N$ and $D$; the ladder stays cheap — about 1% of target compute — while the power law carries the prediction from the small-model regime up to the target scale.

What would settle it

Train a fresh ladder and one or two intermediate-size models from the same family and data mixture (for instance a 2B-3.5B model at roughly 20x-28x the Chinchilla-optimal budget), fit both curves using ladder points only, and compare predicted versus measured accuracy at the intermediate sizes; if the error grows systematically with the ladder-to-target gap, or if the fitted exponents shift noticeably when the intermediate runs are added to the fit, the extrapolation is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a chained two-step parametric pipeline predicts the accuracy of overtrained language models on individual multiple-choice tasks, when the problems are posed in ranked-classification form. Step 1 fits the power law $L(N, D) = A/N^\alpha + B/D^\beta + E$ to the task loss of the ladder models — task loss being the bits-per-byte of the correct answer — with a separate fit per task. Step 2 fits a sigmoid $\mathrm{Acc}(L) = a/(1 + e^{-k(L-L_0)}) + b$ to roughly 1,400 (loss, accuracy) points gathered from both final and intermediate checkpoints of the same ladder. Chaining the two fitted functions predicts the target models' accuracy; on MMLU, HellaSwag, PIQA, and SocialIQA the prediction is within 2 absolute points for both the 7B-4T and 13B-5T targets, and the average absolute error across both targets and all eight OLMES tasks is roughly 4 points. The paper further claims that checkpoint-to-checkpoint variance of the largest ladder model predicts which tasks will be hard to forecast (Pearson $r = 0.821$, $p = 0.004$ for 7B-4T), and that task-specific loss is the more broadly reliable intermediate feature, with C4 language-modeling loss helping only some tasks and TaskCE amplifying small step-1 errors.

Load-bearing premise

The formulas fitted on small ladder models (up to 1.3B parameters, trained on up to ten times the data-optimal budget) are assumed to keep their shape and fitted values when extended to models of 7B to 13B parameters trained on far more data, so that a prediction is just an extension of the curves; if the curves bend differently at large scale, every predicted accuracy is biased.

Editorial extensions

If this is right

  • A pretraining team can estimate whether a planned run will hit a target task score (MMLU, HellaSwag, PIQA, SocialIQA) before spending the training compute: for these four tasks the paper's chained prediction is within 2 absolute points for both the 7B-4T and 13B-5T targets.
  • The ladder transfers beyond the sizes it was built for: predictions for a 32B-6T model of the same family, made at 0.45% compute, keep the same trend, with the low-variance tasks (HellaSwag, PIQA, SocialIQA) within 3 absolute points.
  • Task predictability can be assessed before training the target by measuring the checkpoint-to-checkpoint variance (SD10) of the largest ladder model; tasks with high variance (ARC-Challenge, ARC-Easy, OpenBookQA) are the ones whose predictions should be distrusted.
  • Design guidance follows from the comparisons: a task-specific loss is the more broadly reliable intermediate feature, including model size $N$ in the input features matters more than extending training duration, and a single-step fit straight from $(N, D)$ to accuracy is less robust than the two-step chain.
  • For the multiple-choice (MC) format, the ladder alone cannot fit the sigmoid because small models sit at random accuracy, but using early checkpoints of the target run the paper predicts MMLU MC accuracy within 3% relative error; the same two-step idea therefore extends to MC with an added cost of intermediate target checkpoints.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the ladder is mixture-matched to the target, the same two-curve setup should be able to compare candidate data mixtures cheaply — for example, which mixture yields a higher predicted MMLU score for a fixed $(N, D)$ — a use the paper motivates but does not run.
  • The claimed within-2-point accuracy is demonstrated for one model family, one data mixture, and four RC-formatted tasks; the honest scope is 'same family, same mixture, ranked-classification tasks', and generality to other recipes or generation-style evaluations is untested.
  • The chained errors compound through the sigmoid's slope: on tasks where accuracy rises steeply over a narrow loss range, tiny step-1 errors in the predicted loss become large accuracy errors, so reporting fitted-parameter uncertainty propagated through both curves would give practitioners honest intervals; the paper shows step-2 intervals but does not propagate step-1 uncertainty.
  • The SD10 variance diagnostic could be turned into a pre-registration criterion: a practitioner could set a variance threshold on the largest ladder model before committing to a task set or ladder design, since the paper shows variance correlates with prediction error ($r \approx 0.82$-$0.86$).
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a two-step pipeline for predicting the ranked-classification (RC) accuracy of overtrained language models. Step 1 fits the power law L(N,D)=A/N^alpha+B/D^beta+E to task loss using 16 ladder models (190M-1.3B parameters, 1x-10x Chinchilla data). Step 2 fits a sigmoid mapping task loss to accuracy using approximately 1,400 ladder checkpoints plus one synthetic anchor at (L=0, Acc=1). The chained predictor is applied to two held-out target models, 7B-4T and 13B-5T, on eight OLMES tasks. The paper reports absolute errors below 2 points on MMLU, HellaSwag, PiQA, and SocialIQA, an average absolute error near 4 points across all eight tasks, and a further validation on 32B-6T in Appendix D.

Significance. If the result holds, the paper provides a practical and comparatively cheap way to estimate downstream task accuracy of same-family overtrained models from a fixed ladder of small models, which is directly useful for pretraining decisions. The paper is transparent in reporting all task-level results, gives precise fitting details, genuinely holds out the target models from the ladder fits, and includes a welcome 32B-6T stress test. The variance analysis (SD10) is a useful diagnostic for saying in advance which tasks are likely to be hard to predict. The main caveat is that the headline claim is narrower than the abstract suggests, and the step-2 extrapolation is more fragile than the reported fitting errors indicate.

major comments (3)
  1. [Section 3.2, Eq. (2), Figure 4] The step-2 sigmoid is effectively unconstrained in the loss region that matters for the target predictions. The ladder checkpoints used for fitting lie almost entirely above the target models' predicted task losses, and the only data point below that region is the synthetic anchor (L=0, Acc=1) inserted to stabilize the fit. For MMLU, for example, the observed ladder points end near L≈0.94 while the chained predictions for 7B-4T and 13B-5T are near L≈0.75–0.76, so the sigmoid branch that maps loss to the reported 48–85% accuracies is extrapolated from a functional-form assumption plus one anchor. The reported step-2 fitting errors (0.4–2.6%) are in-sample diagnostics over the observed loss range and cannot detect misspecification in the extrapolation region. The 32B-6T check in Appendix D uses the same ladder and the same extrapolation region, so it does not independently validate the upper branch. Please add a sensitivity analysis (for example, vary or remove the anchor, or compare against an alternative bounded curve) and clearly label the extrapolated part of the curve in Figure 4.
  2. [Section 2.3 and Figure 2] The 'within 2 points' headline covers four of eight remaining tasks, and two additional tasks (BoolQ, Winogrande) are excluded from the main task set because their metrics are described as 'noisier.' Since the four successful tasks are identified after the predictions were made, the claim should be explicitly scoped as a demonstration on a selected subset rather than as a general capability. The full table shows absolute errors of 8.0–11.1 points on ARC-Challenge and ARC-Easy, so the abstract's statement that 'we can predict the accuracy of both target models within 2 points' is too strong as written. The SD10 analysis in Section 5 is a useful post hoc explanation, but it does not replace a pre-specified selection rule. I also note that BoolQ, excluded in Section 2.3, has a relative chained error of only 1.8% in Table 4, which suggests that the exclusion criterion is not based solely on the reported noise measure.
  3. [Section 3.1, Eq. (1)] The paper validates the power-law form for task loss only in-sample on the 16 ladder points, and the extrapolation from N≤1.3B and D≤10xC to N=7–13B and D≈28xC is load-bearing for the entire chained prediction. I request an internal extrapolation check inside the ladder: for example, fit Eq. (1) without the 1.3B models or without the 10xC runs and predict the held-out ladder losses. This would show whether the fitted form extrapolates along each axis before being used for the targets. Figures 9–10 are cumulative ablations rather than true held-out extrapolations, so they do not fully address this concern. Without such a check, the two target models and the 32B-6T result are the only evidence for the extrapolation, and the 32B result is subject to the same limitation.
minor comments (4)
  1. [Abstract and Section 2.1] The '1% of the compute used for the target models' claim is ambiguous: the ladder costs 3.2% of 7B-4T, 1.3% of 13B-5T, and less than 1.0% of both combined. Please state the comparison explicitly.
  2. [Section C.5, Eq. (8)] There is a typo in the text describing Eq. (8): 'D/D^β' should be 'B/D^β'.
  3. [Figure 4 caption] The caption refers to 'prediction intervals' for the fitted sigmoid, but the text does not define how these intervals are computed or whether they are propagated through the chained prediction.
  4. [Table 3] Table 3 is dense and hard to read because two target models and multiple design choices are combined in one grid; splitting by target model or adding a clear visual separation would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: target predictions are held out and the two-step fits use only ladder data.

full rationale

The derivation chain in the RC setup is self-contained: Eq. (1) is fitted to the 16 ladder final-checkpoint (N, D, L) points and Eq. (2) to roughly 1,400 ladder-checkpoint (L, Acc) pairs plus an explicitly declared synthetic anchor (L=0, Acc=1); the target 7B-4T and 13B-5T losses and accuracies are never used in either fit (Secs. 3.1-3.2). The reported target errors are therefore genuine held-out checks against measured OLMo 2 models, and the 32B-6T check in Sec. D is a further held-out extrapolation. No equation is defined in terms of its own prediction target and no fitted parameter is renamed as a prediction: the sigmoid upper branch is constrained by functional form plus the synthetic anchor, which is an extrapolation risk, not a circular reduction. The post-hoc choice of task loss as the recommended intermediate feature and the four-task 'within 2 points' summary are selection effects rather than derivation steps. The MC case study in Sec. B.2 fits on early checkpoints of the target model itself and is accordingly a self-extrapolation, but it is presented separately and does not support the abstract's ladder-based RC claim. Self-citations to OLMo/OLMES define the target models and tasks and are not load-bearing for the scaling-law fit.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central prediction is an extrapolation built from fitted parametric curves. Neither the power-law form for task loss nor the sigmoid for accuracy is derived from first principles; both are assumed and calibrated on the 16-ladder-model dataset. All numeric constants in the prediction pipeline are fitted values, and the validity of the extrapolation to 7B-13B is assumed, not proven.

free parameters (3)
  • Step 1 power law constants (A, B, alpha, beta, E) per task = 8 tasks x 5 constants; values in Table 5
    Fitted by Huber loss on 16 ladder final checkpoints per task (Eq. 1); these determine the predicted task loss for the targets.
  • Step 2 sigmoid constants (a, b, k, L0) per task = 8 tasks x 4 constants; values in Table 5
    Fitted by L2 loss on ~1400 smoothed checkpoint points plus an artificial (0,1) anchor (Eq. 2); these map loss to accuracy.
  • Ladder grid (N and D points) = 4 sizes x 4 data multipliers
    Hand-chosen to span the compute budget; no optimization over grid. The choice of 1xC = 20N follows Chinchilla.
assumptions (4)
  • domain assumption Task loss follows the power law L(N,D)=A/N^alpha + B/D^beta + E
    Postulated in §3.1 following Hoffmann et al.; validated only by goodness-of-fit on the same ladder data, not by independent derivation.
  • domain assumption Task accuracy follows a sigmoid in task loss
    Postulated in §3.2 following Dubey et al.; fitted on ladder checkpoints, with an added artificial anchor point to stabilize the fit.
  • domain assumption Ladder models and target models are comparable in architecture, data mixture, and training recipe
    Assumed in §2.1; ladder and targets are the same OLMo 2 family, but no cross-family or cross-mixture validation is given.
  • domain assumption The Chinchilla-optimal ratio D=20N defines 1xC for ladder models
    Used to construct the data grid in §2.1; taken from Hoffmann et al., not re-derived for these tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Establishing Task Scaling Laws via Compute-Efficient Model Ladders." pith.science (2026). https://pith.science/paper/JROKZP6P

@misc{pith2026241204403,
  author       = {Pith},
  title        = {Pith review of: Establishing Task Scaling Laws via Compute-Efficient Model Ladders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JROKZP6P}},
  note         = {Machine review of arXiv:2412.04403}
}
read the original abstract

We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task performance. Therefore, we leverage a two-step prediction approach: (1) use model and data size to predict an intermediate loss, then (2) use it to predict task performance. We train a set of small-scale "ladder" models, collect data points to fit the parameterized functions of the two prediction steps, and make predictions for two target models: a 7B model trained to 4T tokens and a 13B model trained to 5T tokens. Training the ladder models only costs 1% of the compute used for the target models. On four multiple-choice tasks formatted as ranked classification, we can predict the accuracy of both target models within 2 points of absolute error. We find that tasks with higher prediction error also have higher variance in the metrics over model checkpoints. We also contrast multiple design choices for predicting accuracy, and present recommendations for extending our method to new models and tasks.

Figures

Figures reproduced from arXiv: 2412.04403 by the authors.

Figure 1
Figure 1. Predicting MMLU accuracy with our method. We use model size [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Task accuracy prediction for the target models using task loss as the intermediate [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Task loss vs training scale (N, D), with fitting on the power function in Equation 1. ■ = 1xC; ✚ = 2xC; = 5xC; ⋆ = 10xC. We report the average relative fitting error in parentheses after the task name, and prediction error next to the target model point. 3 Method We break down task accuracy prediction into two steps: 1) predicting the intermediate feature (we use task loss to illustrate this), and 2) using it to pre… view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Task RC accuracy vs task loss, with fitting on the sigmoid function in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Relative SD over the final 10 checkpoints (SD [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Intermediate checkpoints and standard deviation over the final 10 checkpoints [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: Upper: MC accuracy curves during training of 7B-4T and 13B-5T. Lower left: Task MC accuracy vs task loss, with data points from all intermediate checkpoints of 7B-4T and 13B-5T. A sigmoidal function cannot fit the data points. Lower right: Task MC accuracy vs task loss…
Figure 8
Figure 8. Figure 8: Prediction error on the 7B-4T target model as a function of the total compute [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Prediction error on the 7B-4T target model when including [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Prediction error on the 7B-4T target model when including models trained [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Step 1 using compute-flops C instead of (N, D). ■ = 1xC; ✚ = 2xC; = 5xC; ⋆ = 10xC. We report the average relative fitting error in parentheses following the task name, and prediction error in the plot next to the target model point. C.2 Task cross-entropy as the inter…
Figure 12
Figure 12. Figure 12: Predicting final task cross-entropy (Equation 6) from model parameters and token budget in step 1. Results [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Predicting the task metric from the task cross-entropy (Equation 6) in step 2 [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Chaining predictions from step 1 ( [PITH_FULL_IMAGE:figures/full_fig_p021_14.png]
Figure 15
Figure 15. Figure 15: Using language modeling loss on C4-en validation as the intermediate feature. [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Comparison of absolute and relative prediction errors for all three intermediate [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Task RC accuracy vs training scale (N, D), with fitting on the single-step function in Equation 8. 32B-6T Task Pred Actual Error %Error HellaSwag 88.0 85.3 2.7 3.07% ARC-Challenge 53.6 66.8 13.2 24.63% ARC-Easy 77.8 88.8 11.0 14.14% PIQA 82.9 83.6 0.7 0.84% Commonsens…
Figure 18
Figure 18. Figure 18: Task accuracy prediction for the 32B model using task loss as the intermediate [PITH_FULL_IMAGE:figures/full_fig_p024_18.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

    cs.CL 2026-07 conditional novelty 7.5 of 10

    Across ~38,000 hours on 134 ultra-long real-world tasks, aggregate agent performance follows a log-sigmoid of interaction time, and measured learning speed doubles about every three months.

  2. A Latent Variable Framework for Scaling Laws in Large Language Models

    stat.AP 2025-12 conditional novelty 6.0 of 10

    A latent-variable scaling model with consistency guarantees predicts multi-benchmark LLM performance per family and derives skill-specific compute allocations.

  3. Fantastic Pretraining Optimizers and Where to Find Them

    cs.LG 2025-09 conditional novelty 6.0 of 10

    After careful tuning, alternative optimizers speed up LLM pretraining by only 1.1x to 1.4x over AdamW, and the advantage shrinks with model scale.

  4. Language Models Improve When Pretraining Data Matches Target Tasks

    cs.CL 2025-07 conditional novelty 6.0 of 10

    Ranking pretraining documents by similarity to benchmark training examples (BETR) yields consistent benchmark gains and a 2.1x compute multiplier over DCLM-Baseline.

Reference graph

Works this paper leans on

29 extracted references · 6 canonical work pages · cited by 4 Pith papers

  1. [1]

    Olmo 2: The best fully open language model to date, 2024

    Ai2. Olmo 2: The best fully open language model to date, 2024. URL https://allenai.org/blog/olmo2

    work page 2024

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenhang Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, K. Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, ...

    arXiv 2023

  3. [3]

    PIQA : Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. PIQA : Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (05): 0 7432--7439, Apr. 2020. doi:10.1609/aaai.v34i05.6239. URL https://ojs.aaai.org/index.php/AAAI/article/view/6239

    doi:10.1609/aaai.v34i05.6239 2020

  4. [4]

    Scaling laws for predicting downstream performance in llms

    Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in llms. 2024. URL https://api.semanticscholar.org/CorpusID:273323177

    work page 2024

  5. [5]

    B ool Q : Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. B ool Q : Exploring the surprising difficulty of natural yes/no questions. pp.\ 2924--2936, Minneapolis, Minnesota, June 2019. doi:10.18653/v1/N19-1300. URL N19-1300

    doi:10.18653/v1/n19-1300 2019

  6. [6]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, 2018. URL http://arxiv.org/abs/1803.05457

    arXiv 2018

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi \`e re, Bet...

    arXiv 2024

  8. [8]

    Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, and Ludwig Schmidt

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean-Pierre Mercat, Alex Fang, Jeffrey Li, Sedrick Scott Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff...

    arXiv 2024

Show all 29 references
  1. [9]

    Olmes: A standard for language model evaluations, 2024

    Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. Olmes: A standard for language model evaluations, 2024

    2024

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  3. [11]

    Rae, Oriol Vinyals, and L

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022 arXiv

  4. [12]

    Predicting emergent abilities with infinite resolution evaluation

    Shengding Hu, Xin Liu, Xu Han, Xinrong Zhang, Chaoqun He, Weilin Zhao, Yankai Lin, Ning Ding, Zebin Ou, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun. Predicting emergent abilities with infinite resolution evaluation. In International Conference on Learning Representations, 2023....

    2023

  5. [13]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361, 2020. URL https://api.semanticscholar.org/CorpusID:210861095

    2001 arXiv

  6. [14]

    Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke S

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Kumar Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean-Pierre Mercat, Mayee Chen, Suchin Gururangan, Mitchell Worts...

    2024 arXiv

  7. [15]

    Can a suit of armor conduct electricity? a new dataset for open book question answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. pp.\ 2381--2391, Brussels, Belgium, October-November 2018. doi:10.18653/v1/D18-1260. URL D18-1260

    2018 doi

  8. [16]

    Llm foundry, 2024

    MosaicML. Llm foundry, 2024. URL https://github.com/mosaicml/llm-foundry. Accessed: 2024-12-03

    2024

  9. [17]

    Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel

    Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. ArXiv, abs/2305.16264, 2023. URL https://api.semanticscholar.org/CorpusID:258888192

    2023 arXiv

  10. [18]

    Resolving discrepancies in compute-optimal scaling of language models

    Tomer Porian, Mitchell Wortsman, Jenia Jitsev, Ludwig Schmidt, and Yair Carmon. Resolving discrepancies in compute-optimal scaling of language models. ArXiv, abs/2406.19146, 2024. URL https://api.semanticscholar.org/CorpusID:270764838

    2024 arXiv

  11. [19]

    Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J

    Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21: 0 140:1--140:67, 2019. URL https://api....

    2019

  12. [20]

    Wino G rande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Wino G rande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (05): 0 8732--8740, Apr. 2020. doi:10.1609/aaai.v34i05.6399. URL https://...

    2020 doi

  13. [21]

    Social IQ a: Commonsense reasoning about social interactions

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQ a: Commonsense reasoning about social interactions. pp.\ 4463--4473, Hong Kong, China, November 2019. doi:10.18653/v1/D19-1454. URL D19-1454

    2019 doi

  14. [22]

    Why has predicting downstream capabilities of frontier AI models with scale remained elusive? In Trustworthy Multi-modal Foundation Models and AI Agents (TiFA), 2024

    Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier AI models with scale remained elusive? In Trustworthy Multi-modal Foundation...

    2024

  15. [23]

    C ommonsense QA : A question answering challenge targeting commonsense knowledge

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. C ommonsense QA : A question answering challenge targeting commonsense knowledge. pp.\ 4149--4158, Minneapolis, Minnesota, June 2019. doi:10.18653/v1/N19-1421. URL N19-1421

    2019 doi

  16. [24]

    H ella S wag: Can a machine really finish your sentence? pp.\ 4791--4800, Florence, Italy, July 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. H ella S wag: Can a machine really finish your sentence? pp.\ 4791--4800, Florence, Italy, July 2019. doi:10.18653/v1/P19-1472. URL P19-1472

    2019 doi

  17. [25]

    Map-neo: Highly capable and transparent bilingual large language model series

    Ge Zhang, Scott Qu, Jiaheng Liu, Chenchen Zhang, Chenghua Lin, Chou Leuang Yu, Danny Pan, Esther Cheng, Jie Liu, Qunshu Lin, Raven Yuan, Tuney Zheng, Wei Pang, Xinrun Du, Yiming Liang, Yi Ma, Yizhi Li, Ziyang Ma, Bill Yuchen Lin, Emmanouil Benetos, Huan Yang, Junting Zhou, Kai...

    2024 arXiv

  18. [26]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  19. [27]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  20. [28]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  21. [29]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.