{"id":"4f5d80f1-82f0-4d07-8778-81d874d0e1a7","arxiv_id":"2412.04403","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-step parametric model trained on small 'ladder' language models predicts individual task accuracy of 7B and 13B models to within 2 points on four of eight benchmarks, using about 1% of target compute.","lead":"This paper shows how to predict what a large language model will score on standard reasoning tests by first training small 'ladder' models and fitting two curves that connect model size, training data, loss, and accuracy. The method uses about 1% of the training compute of the big models and predicts four of eight tasks within 2 accuracy points.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Step 2's sigmoid is fit on ladder checkpoints whose losses all lie above the target region; the only sub-target data point is a synthetic (L=0, Acc=1) anchor, so the headline 2-point predictions rest on an unconstrained parametric extrapolation.","rationale":"The reader's weakest assumption correctly identifies the extrapolation of Eq. 1 and the transfer of Eq. 2 as the fragile bridge. I agree with that diagnosis, but the more specific and actionable weak point is inside Eq. 2: the sigmoid's upper branch is fit with no real data in the target loss region. The ladder models' final checkpoints, and all their intermediate checkpoints, have task losses above the target models' predicted losses, so the only information below the target region is the manually inserted (0,1) anchor. This makes the 'within 2 points' claim contingent on the sigmoid's parametric form in a data-free regime. The paper deserves credit for genuine held-out evaluation on 7B-4T and 13B-5T, for the variance analysis in Sec. 5, and for the 32B-6T extrapolation in Sec. D; these are real checks and they do provide partial support. But the 32B check exercises the same unconstrained upper branch rather than independently validating it. The proposed ablation of the artificial anchor, or a nonparametric link, would directly test whether the headline result is an artifact of the fitting procedure. I do not think this changes the reader's conditional verdict—the paper remains a useful empirical contribution with a clear, stated extrapolation assumption—but the conditional should explicitly name the anchor sensitivity as a required robustness check. One additional minor inconsistency: the abstract and Sec. 4 state HellaSwag is predicted within 2 points on both targets, while Fig. 2 shows 2.1 points absolute error for the 13B-5T model; this should be corrected for precision, though it does not change the overall verdict.","tokens_in":20958,"tokens_out":8540,"duration_ms":78770,"concrete_test":"Re-fit Eq. 2 on the same ladder checkpoint data with the artificial (0,1) anchor removed, and, as a robustness variant, with a flexible nonparametric link such as isotonic regression or a Gaussian process. If the chained predictions on MMLU, HellaSwag, PIQA, and SocialIQA for the 7B-4T and 13B-5T models remain within 2 points of observed accuracies, the anchor and sigmoid form are not load-bearing. If they move by more than roughly 1 point, the headline result depends on a synthetic data point and an unverified parametric extrapolation, and the paper should report that dependence and temper the abstract accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The chained prediction's second step (Eq. 2, Sec. 3.2) fits a sigmoid to roughly 1,400 (loss, accuracy) pairs from ladder-model checkpoints plus one manually inserted anchor at (L=0, Acc=1). For the four headline tasks, the fitted ladder losses do not extend down to the target models' predicted losses: e.g., the 1B-10xC final MMLU point sits near L≈0.94 while the 7B/13B predictions are near L≈0.75–0.76. Thus the entire upper branch of the sigmoid—the branch that maps target losses to the reported 48–85% accuracies—is constrained only by the synthetic anchor and the assumed functional form, not by observed data. The reported step-2 fitting error (0.4–2.6%) measures fit quality in the ladder's observed loss range and cannot detect misspecification in the extrapolation region. If the true loss-accuracy relation bends differently at low loss (earlier saturation, different curvature, or a second transition), all four 'within 2 points' results would shift by an amount invisible to the fitting diagnostics. The 32B-6T validation (Sec. D) partially reassures for low-variance tasks, but it uses the same ladder and the same extrapolation region, so it cannot independently confirm the sigmoid's upper branch.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-step pipeline for predicting the ranked-classification (RC) accuracy of overtrained language models. Step 1 fits the power law L(N,D)=A/N^alpha+B/D^beta+E to task loss using 16 ladder models (190M-1.3B parameters, 1x-10x Chinchilla data). Step 2 fits a sigmoid mapping task loss to accuracy using approximately 1,400 ladder checkpoints plus one synthetic anchor at (L=0, Acc=1). The chained predictor is applied to two held-out target models, 7B-4T and 13B-5T, on eight OLMES tasks. The paper reports absolute errors below 2 points on MMLU, HellaSwag, PiQA, and SocialIQA, an average absolute error near 4 points across all eight tasks, and a further validation on 32B-6T in Appendix D.","tokens_in":21284,"tokens_out":9279,"duration_ms":99050,"significance":"If the result holds, the paper provides a practical and comparatively cheap way to estimate downstream task accuracy of same-family overtrained models from a fixed ladder of small models, which is directly useful for pretraining decisions. The paper is transparent in reporting all task-level results, gives precise fitting details, genuinely holds out the target models from the ladder fits, and includes a welcome 32B-6T stress test. The variance analysis (SD10) is a useful diagnostic for saying in advance which tasks are likely to be hard to predict. The main caveat is that the headline claim is narrower than the abstract suggests, and the step-2 extrapolation is more fragile than the reported fitting errors indicate.","major_comments":[{"comment":"The step-2 sigmoid is effectively unconstrained in the loss region that matters for the target predictions. The ladder checkpoints used for fitting lie almost entirely above the target models' predicted task losses, and the only data point below that region is the synthetic anchor (L=0, Acc=1) inserted to stabilize the fit. For MMLU, for example, the observed ladder points end near L≈0.94 while the chained predictions for 7B-4T and 13B-5T are near L≈0.75–0.76, so the sigmoid branch that maps loss to the reported 48–85% accuracies is extrapolated from a functional-form assumption plus one anchor. The reported step-2 fitting errors (0.4–2.6%) are in-sample diagnostics over the observed loss range and cannot detect misspecification in the extrapolation region. The 32B-6T check in Appendix D uses the same ladder and the same extrapolation region, so it does not independently validate the upper branch. Please add a sensitivity analysis (for example, vary or remove the anchor, or compare against an alternative bounded curve) and clearly label the extrapolated part of the curve in Figure 4.","section":"Section 3.2, Eq. (2), Figure 4"},{"comment":"The 'within 2 points' headline covers four of eight remaining tasks, and two additional tasks (BoolQ, Winogrande) are excluded from the main task set because their metrics are described as 'noisier.' Since the four successful tasks are identified after the predictions were made, the claim should be explicitly scoped as a demonstration on a selected subset rather than as a general capability. The full table shows absolute errors of 8.0–11.1 points on ARC-Challenge and ARC-Easy, so the abstract's statement that 'we can predict the accuracy of both target models within 2 points' is too strong as written. The SD10 analysis in Section 5 is a useful post hoc explanation, but it does not replace a pre-specified selection rule. I also note that BoolQ, excluded in Section 2.3, has a relative chained error of only 1.8% in Table 4, which suggests that the exclusion criterion is not based solely on the reported noise measure.","section":"Section 2.3 and Figure 2"},{"comment":"The paper validates the power-law form for task loss only in-sample on the 16 ladder points, and the extrapolation from N≤1.3B and D≤10xC to N=7–13B and D≈28xC is load-bearing for the entire chained prediction. I request an internal extrapolation check inside the ladder: for example, fit Eq. (1) without the 1.3B models or without the 10xC runs and predict the held-out ladder losses. This would show whether the fitted form extrapolates along each axis before being used for the targets. Figures 9–10 are cumulative ablations rather than true held-out extrapolations, so they do not fully address this concern. Without such a check, the two target models and the 32B-6T result are the only evidence for the extrapolation, and the 32B result is subject to the same limitation.","section":"Section 3.1, Eq. (1)"}],"minor_comments":[{"comment":"The '1% of the compute used for the target models' claim is ambiguous: the ladder costs 3.2% of 7B-4T, 1.3% of 13B-5T, and less than 1.0% of both combined. Please state the comparison explicitly.","section":"Abstract and Section 2.1"},{"comment":"There is a typo in the text describing Eq. (8): 'D/D^β' should be 'B/D^β'.","section":"Section C.5, Eq. (8)"},{"comment":"The caption refers to 'prediction intervals' for the fitted sigmoid, but the text does not define how these intervals are computed or whether they are propagated through the chained prediction.","section":"Figure 4 caption"},{"comment":"Table 3 is dense and hard to read because two target models and multiple design choices are combined in one grid; splitting by target model or adding a clear visual separation would improve readability.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical scaling-law study with a useful diagnostic and honest reporting of all tasks. The main weakness is the step-2 extrapolation supported by a synthetic anchor, which is central to the headline 2-point claim. I would be willing to accept after the authors add a sensitivity analysis for the anchor/functional form and reframe the claim to match the subset of tasks it actually covers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The new thing here is the fixed ladder: four model sizes times four data budgets, no compute-optimal search, then a two-step chain from (N,D) to task loss with a Chinchilla-style power law, and task loss to accuracy with a sigmoid. The targets (7B-4T, 13B-5T) are genuinely held out, and the \"within 2 points\" claim holds on four low-variance tasks. The paper also does honest housekeeping: it reports all eight tasks, gives average errors (3.8 and 4.2 points), compares alternative intermediate features, and includes a variance analysis (SD10) that predicts which tasks will be hard. That variance analysis is a useful practical artifact. The 32B-6T check is a good-faith robustness test, and low-variance tasks staying predictable there is real evidence. The citation pattern is fine; they distinguish themselves from Gadre, Dubey, and Chen without overclaiming.\n\nThe main soft spot is the sigmoid's upper branch. Step 2 fits roughly 1,400 (loss, accuracy) points from ladder checkpoints, but those losses sit above the target region; the only sub-target point is a synthetic (L=0, Acc=1) anchor. So the 48–85% accuracy predictions on MMLU, HellaSwag, PIQA, and SocialIQA are produced by a curve whose steep part is constrained by exactly one synthetic point plus the assumed functional form. The reported step-2 fitting error (0.4–2.6%) measures fit in the observed range; it cannot detect misspecification in the extrapolation region. That does not make the result false—the empirical match on two targets and one larger target is encouraging—but it does mean the headline claim is less certain than the numbers suggest. If the true loss-accuracy relation bends even slightly differently at low loss, the four \"within 2\" results could shift by more than 2 points.\n\nMinor issues: BoolQ and Winogrande are excluded post hoc as noisy; task loss is chosen as the primary intermediate feature after seeing target results; and the \"within 2 points\" is a cherry-picked subset of the eight tasks. These are disclosed, but they should temper how the result is advertised.\n\nWho is this for? Pretraining teams deciding whether a data or architecture change will move a benchmark before committing a large run, and scaling-law researchers. It deserves a serious referee; I would send it out with a request for uncertainty quantification on the extrapolation, an explicit ablation of the anchor, and ideally validation on one non-OLMo model family.\n\nRecommendation: engage with it. The core idea is useful and the execution is careful. The reviewer should push on the sigmoid extrapolation rather than dismiss the paper.","headline":"A solid empirical paper that does something genuinely new—predicting individual task accuracy from a 1%-compute ladder—but the headline 'within 2 points' rests on a sigmoid extrapolation that is thinner than the paper's tone suggests.","tokens_in":21873,"tokens_out":4008,"would_cite":true,"duration_ms":101624,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Task accuracy of large overtrained language models can be predicted from a small fixed set of 'ladder' models via a two-step parametric pipeline, landing within 2 absolute points on four multiple-choice benchmarks for under 1% of the…","keywords":["task scaling laws","model ladders","overtrained language models","scaling law extrapolation","task accuracy prediction","ranked classification","two-step prediction","compute-efficient evaluation"],"falsifier":"Train a fresh ladder and one or two intermediate-size models from the same family and data mixture (for instance a 2B-3.5B model at roughly 20x-28x the Chinchilla-optimal budget), fit both curves using ladder points only, and compare predicted versus measured accuracy at the intermediate sizes; if the error grows systematically with the ladder-to-target gap, or if the fitted exponents shift noticeably when the intermediate runs are added to the fit, the extrapolation is falsified.","tokens_in":20767,"feed_emoji":"📈","tokens_out":10730,"duration_ms":82749,"temperature":0.7,"pith_summary":"This paper is trying to establish that the downstream task accuracy of a large, overtrained language model — the score a 7B or 13B model would get on a benchmark like MMLU or HellaSwag — can be predicted before the model is trained, using only a small 'ladder' of cheap models. The method trains 16 small models (190M to 1.3B parameters, each trained on 1x to 10x its Chinchilla-optimal data budget) and fits two curves: a power law that maps parameter count $N$ and token budget $D$ to a task-specific loss, and a sigmoid that maps that loss to task accuracy. Predicting for two held-out targets — a 7B model trained on roughly 4T tokens and a 13B model on 5T tokens — the chained curves land within 2 absolute accuracy points on four of eight tasks (MMLU, HellaSwag, PIQA, SocialIQA) and within about 4 points on average, at a cost of about 1% of the target compute. The reason to care: if the claim holds, pretraining decisions that currently require a full expensive run — how much data, what mixture, what model size — become cheap, testable predictions on the same benchmarks used to judge the models.","feed_headline":"Small models predict big-model accuracy within 2 points","feed_subtitle":"Two fitted curves link model size and data count to task accuracy for 1% of the training budget.","key_machinery":"The machinery is the model ladder plus the two fitted curves it feeds. The ladder is a fixed grid of 16 models — four sizes ($N \\in \\{190\\text{M}, 370\\text{M}, 760\\text{M}, 1.3\\text{B}\\}$ non-embedding parameters) by four data budgets ($D \\in \\{1\\text{x}, 2\\text{x}, 5\\text{x}, 10\\text{x}\\}$ of the Chinchilla-optimal amount) — sharing the target models' architecture, data mixture, and learning-rate schedule, so that scale, not recipe, is the only difference being extrapolated. Step 1 fits the power law $L(N, D) = A/N^\\alpha + B/D^\\beta + E$ to the final-checkpoint task losses, and step 2 fits the sigmoid $\\mathrm{Acc}(L) = a/(1 + e^{-k(L-L_0)}) + b$ to the loss-accuracy pairs of roughly 1,400 ladder checkpoints. The load-bearing move is that both curves are fitted on small models and then evaluated by extrapolation along $N$ and $D$; the ladder stays cheap — about 1% of target compute — while the power law carries the prediction from the small-model regime up to the target scale.","core_discovery":"The central claim is that a chained two-step parametric pipeline predicts the accuracy of overtrained language models on individual multiple-choice tasks, when the problems are posed in ranked-classification form. Step 1 fits the power law $L(N, D) = A/N^\\alpha + B/D^\\beta + E$ to the task loss of the ladder models — task loss being the bits-per-byte of the correct answer — with a separate fit per task. Step 2 fits a sigmoid $\\mathrm{Acc}(L) = a/(1 + e^{-k(L-L_0)}) + b$ to roughly 1,400 (loss, accuracy) points gathered from both final and intermediate checkpoints of the same ladder. Chaining the two fitted functions predicts the target models' accuracy; on MMLU, HellaSwag, PIQA, and SocialIQA the prediction is within 2 absolute points for both the 7B-4T and 13B-5T targets, and the average absolute error across both targets and all eight OLMES tasks is roughly 4 points. The paper further claims that checkpoint-to-checkpoint variance of the largest ladder model predicts which tasks will be hard to forecast (Pearson $r = 0.821$, $p = 0.004$ for 7B-4T), and that task-specific loss is the more broadly reliable intermediate feature, with C4 language-modeling loss helping only some tasks and TaskCE amplifying small step-1 errors.","pith_inferences":["Because the ladder is mixture-matched to the target, the same two-curve setup should be able to compare candidate data mixtures cheaply — for example, which mixture yields a higher predicted MMLU score for a fixed $(N, D)$ — a use the paper motivates but does not run.","The claimed within-2-point accuracy is demonstrated for one model family, one data mixture, and four RC-formatted tasks; the honest scope is 'same family, same mixture, ranked-classification tasks', and generality to other recipes or generation-style evaluations is untested.","The chained errors compound through the sigmoid's slope: on tasks where accuracy rises steeply over a narrow loss range, tiny step-1 errors in the predicted loss become large accuracy errors, so reporting fitted-parameter uncertainty propagated through both curves would give practitioners honest intervals; the paper shows step-2 intervals but does not propagate step-1 uncertainty.","The SD10 variance diagnostic could be turned into a pre-registration criterion: a practitioner could set a variance threshold on the largest ladder model before committing to a task set or ladder design, since the paper shows variance correlates with prediction error ($r \\approx 0.82$-$0.86$)."],"forward_implications":["A pretraining team can estimate whether a planned run will hit a target task score (MMLU, HellaSwag, PIQA, SocialIQA) before spending the training compute: for these four tasks the paper's chained prediction is within 2 absolute points for both the 7B-4T and 13B-5T targets.","The ladder transfers beyond the sizes it was built for: predictions for a 32B-6T model of the same family, made at 0.45% compute, keep the same trend, with the low-variance tasks (HellaSwag, PIQA, SocialIQA) within 3 absolute points.","Task predictability can be assessed before training the target by measuring the checkpoint-to-checkpoint variance (SD10) of the largest ladder model; tasks with high variance (ARC-Challenge, ARC-Easy, OpenBookQA) are the ones whose predictions should be distrusted.","Design guidance follows from the comparisons: a task-specific loss is the more broadly reliable intermediate feature, including model size $N$ in the input features matters more than extending training duration, and a single-step fit straight from $(N, D)$ to accuracy is less robust than the two-step chain.","For the multiple-choice (MC) format, the ladder alone cannot fit the sigmoid because small models sit at random accuracy, but using early checkpoints of the target run the paper predicts MMLU MC accuracy within 3% relative error; the same two-step idea therefore extends to MC with an added cost of intermediate target checkpoints."],"supporting_citations":[{"why":"Supplies the Chinchilla power-law form $L(N,D) = A/N^\\alpha + B/D^\\beta + E$ and the '1xC' compute-optimal data budget that defines the ladder's training grid.","marker":"Hoffmann et al., 2022"},{"why":"Supplies the two-step prediction pattern and the sigmoidal loss-to-accuracy mapping, plus the per-character normalized NLL idea behind the paper's task-loss metric.","marker":"Dubey et al., 2024"},{"why":"Supplies the relative-error evaluation metric and the prior baseline that predicts average top-1 error from C4 loss, which the paper extends to individual tasks.","marker":"Gadre et al., 2024"},{"why":"Supplies the compute estimate $C \\approx 6ND$ used to report the ladder's 1% compute cost.","marker":"Kaplan et al., 2020"},{"why":"Supplies the method for extrapolating peak LR, batch size, and warmup steps from the 7B-4T configuration to the ladder models.","marker":"Porian et al., 2024"},{"why":"Defines the target models (7B-4T and 13B-5T), their data mixture, and the stage-1 pretraining setup the ladder must match.","marker":"OLMo et al., 2024"},{"why":"Supplies the OLMES evaluation suite, the eight tasks, the 5-shot setting, and the ranked-classification versus multiple-choice distinction the method relies on.","marker":"Gu et al., 2024"},{"why":"Cited as the motivation that accuracy depends on the losses of both correct and incorrect choices, which motivates the TaskCE alternative intermediate feature.","marker":"Schaeffer et al., 2024"}],"fun_headline_variants":["Model ladders predict large-model accuracy at 1% compute","Predict big-model accuracy with 100x less compute","Task scaling laws from 1% compute predict accuracy within 2 points","Two-step fit predicts task accuracy within 2 points","Ladder models forecast big-model accuracy at 1% training cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The formulas fitted on small ladder models (up to 1.3B parameters, trained on up to ten times the data-optimal budget) are assumed to keep their shape and fitted values when extended to models of 7B to 13B parameters trained on far more data, so that a prediction is just an extension of the curves; if the curves bend differently at large scale, every predicted accuracy is biased.","fun_headline_variants_meta":{"raw":{"variants":["Model ladders predict large-model accuracy at 1% compute","Predict big-model accuracy with 100x less compute","Task scaling laws from 1% compute predict accuracy within 2 points","Two-step fit predicts task accuracy within 2 points","Ladder models forecast big-model accuracy at 1% training cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3994,"prompt_tokens":1042,"completion_tokens":2952,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":2867}},"tokens_in":658,"tokens_out":2952,"duration_ms":20068,"temperature":1.0,"reasoning_tokens":2867,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:28.781246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a fresh ladder and one or two intermediate-size models from the same family and data mixture (for instance a 2B-3.5B model at roughly 20x-28x the Chinchilla-optimal budget), fit both curves using ladder points only, and compare predicted versus measured accuracy at the intermediate sizes; if the error grows systematically with the ladder-to-target gap, or if the fitted exponents shift noticeably when the intermediate runs are added to the fit, the extrapolation is falsified.","supporting_citations":[{"cited_title":"Why has predicting downstream capabilities of frontier AI models with scale remained elusive? In Trustworthy Multi-modal Foundation Models and AI Agents (TiFA), 2024","cited_arxiv_id":null,"evidence_quote":"Cited as the motivation that accuracy depends on the losses of both correct and incorrect choices, which motivates the TaskCE alternative intermediate feature."}],"review_version":1}