Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Post-training accuracy at a given compute budget follows a stable sigmoid curve—except on math reasoning, where the attainable ceiling keeps rising.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:52 UTC pith:R4TLXHSY

load-bearing objection A serious observational study whose temporal-split validation of high-quantile sigmoid boundaries is the real contribution, but the prescriptive 'attainable' language outruns the evidence: the boundaries are quantiles of a selected population, not proven capability ceilings. the 4 major comments →

arxiv 2602.15327 v2 pith:R4TLXHSY submitted 2026-02-17 cs.LG cs.AIcs.CLstat.ML

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Hanlin Zhang , Jikai Jin , Vasilis Syrgkanis , Sham Kakade This is my paper
classification cs.LG cs.AIcs.CLstat.ML MSC 68T0762G0862K05
keywords prescriptive scalingcapability boundariesquantile regressionscaling lawslanguage modelspost-trainingleaderboard evaluationI-optimal design
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish a prescriptive scaling law: given a pre-training compute budget, what downstream accuracy can a practitioner reliably attain after post-training? It claims that the attainable upper envelope of post-trained performance is a simple monotone sigmoid function of log-compute, estimated as a 0.98 conditional quantile via smoothed quantile regression. It then claims this boundary is temporally stable for most benchmarks—fitting on earlier model generations predicts the next generation's top scores within about 2% coverage error on four of six tasks—while math reasoning shows a boundary that advances over time. If true, practitioners can translate compute budgets into realistic performance expectations, monitor when capability shifts, and cut evaluation costs by roughly 80% via a balanced I-optimal sampling design. The paper positions capability boundaries, rather than average scaling laws, as the decision-oriented object for budgeting model development.

Core claim

The central claim is that the highest reliably attainable post-training performance—modeled as the 98th conditional quantile of benchmark scores given log pre-training compute—is well approximated by the sigmoid qτ(z) = y0 + L σ(a + βz), and that this boundary transfers to later model generations on most tasks. The exceptions are informative: math reasoning (MATH Lvl 5) shows a consistently advancing boundary, indicating algorithmic progress moves the attainable envelope, while knowledge-heavy tasks like MMLU-Pro remain scale-limited, with small models near a stable ceiling. The paper further shows that this boundary is a better predictor of potential than raw pretrained accuracies, which ar

What carries the argument

The capability boundary estimator: a smoothed pinball-loss quantile regression at τ=0.98 with a monotone, saturating sigmoid parameterization in log-compute, validated by rolling chronological train/test splits. The pinball loss asymmetry makes the fitted curve track the upper envelope of the model population rather than the mean. The same boundary machinery is extended to a size–time saturation model, a PCA-based latent-factor analysis, a logit-on-logit contamination shift test, and a balanced I-optimal sampling algorithm that selects which models to evaluate under a parameter-count-weighted budget.

Load-bearing premise

The load-bearing assumption is that the public leaderboard population fairly represents contemporary post-training practice, so the fitted 0.98-quantile is a truthful 'attainable' ceiling rather than an artifact of which models happen to be reported.

What would settle it

Evaluate all post-trained open-weight models released in the six months after the leaderboard cutoff with known pre-training compute on a stable benchmark such as GPQA; if more than ~2% of them land above the predicted 0.98-quantile band per compute bin, temporal stability and the sigmoid boundary are refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • At a fixed pre-training budget, the 0.98-quantile sigmoid supplies a concrete performance expectation—e.g., 0.83 on IFEval and 0.54 on MATH Lvl 5 at 10^24 FLOPs—so compute can be budgeted against a target accuracy.
  • The boundary fit on earlier generations covers the next generation with under ~2% coverage error on BBH, GPQA, MMLU-Pro, and MUSR, making the compute-to-attainable-accuracy map nearly deterministic for contemporary practice on those tasks.
  • Math reasoning (MATH Lvl 5) is the consistent exception: its boundary advances over time, concentrated at high compute, so math expectations must be updated as recipes evolve.
  • The pretrain–post-train gap is task-dependent: post-training adds little on knowledge benchmarks like MMLU-Pro where larger models retain a persistent advantage, but lifts small models close to the top on math (estimated 13B attainable boundary ≈0.94 late-period), clarifying where 'saturation' claims hold.
  • A balanced I-optimal design recovers near-full-data boundaries with roughly 20% of the evaluation budget—as low as 5% on GPQA and MUSR—while a cross-benchmark contamination test finds no statistically significant AIME-2025 inflation relative to MATH-500 (p ≈ 0.15).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the sigmoid boundary holds as a near-law-like map, the practical definition of 'attainable' becomes the upper envelope of public leaderboards; that makes selection bias in what gets published a first-order threat to any prescriptive use, since a systematically withheld strong recipe would quietly raise the true boundary.
  • The PCA observation that only the first latent factor scales with compute suggests capability progress is largely one-dimensional; a testable extension is to replace six benchmark regressions with a single factor-based boundary and check temporal stability on the same rolling splits.
  • The cross-benchmark contamination test is a template: the same logit-on-logit regression against a release-gated benchmark could be run on newly released datasets to detect train-on-test leakage earlier than manual inspection.
  • The temporal-stability claim is established on open-weight, leaderboard-reported populations; a sharper stress test would fit the sigmoid on closed frontier models with known compute and validate on later frontier releases, since the paper's frontier GPQA fit is cross-sectional.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes 'prescriptive scaling laws': high conditional quantiles (τ=0.98) of post-trained benchmark accuracy as a function of log pre-training FLOPs, estimated by smoothed quantile regression with a monotone saturating sigmoid. It uses observational data from the Open LLM Leaderboard v1/v2, frontier-model leaderboards, and 2.4k newly evaluated PROTEUS-2K checkpoints. The central empirical claims are that the post-training capability boundary is approximately sigmoid in log-compute, that this boundary is temporally stable for four of six studied benchmarks under chronological train/validation splits, and that math reasoning (and to a lesser extent instruction following) shows a shifting boundary. The paper also presents task-dependent saturation diagnostics, a contamination shift test for AIME-2025, and a budget-aware balanced I-optimal sampling algorithm that recovers near-full-data boundaries at ~20% (sometimes ~5%) of the evaluation budget. Table 1 gives headline point predictions at 10^24 FLOPs (e.g., 0.83 IFEval, 0.54 MATH Lvl 5).

Significance. If the central claims hold, the paper makes a valuable contribution: it moves scaling-law analysis from mean trends to a calibrated, high-quantile capability boundary, validates that boundary out-of-sample in time for a majority of tasks, and releases a new evaluation dataset and a practical active-sampling procedure. The quantile-regression formulation is principled, the chronological splits are a genuine OOD check, and the saturation and contamination case studies are useful applications. However, the paper's headline 'attainable accuracy' interpretation depends on an unverified representativeness assumption about public leaderboard submissions, and the Table 1 point estimates are in-sample fitted values with no uncertainty quantification. These issues do not invalidate the sigmoid-shape or temporal-stability findings for the observed population, but they require substantive reframing before the prescriptive claims can be accepted.

major comments (4)
  1. [§2.1, Appendix A.2, Table 1] The central prescriptive interpretation—that Table 1 gives accuracies 'attainable with contemporary post-training practice'—is not supported by the data. Section 2.1 explicitly defines qτ(z) as a conditional quantile of the observed leaderboard population and Appendix A.2 states that substantial selection effects could shift the implied boundary. The paper does not close this gap: leaderboard submission is itself a performance filter, and the PROTEUS-2K collection (Appendix F) is curated by Hugging Face likes plus manual selection. Temporal stability cannot correct for this because selection may be roughly stable over time. The authors should either (i) consistently use 'observed-population upper envelope' language throughout, including the abstract and Table 1, or (ii) provide evidence that the sample is representative of contemporary practice, e.g., by including non-submitted checkpoin
  2. [§3.1 and Table 1] The headline predictions at 10^24 FLOPs are computed from the no-split 0.98-quantile sigmoid fit, not from the chronological validation protocol, and no confidence intervals are reported. The rolling-split analysis (Table 2, Figure 2) is the paper's strongest evidence, but Table 1 extrapolates the in-sample fit to the upper end of the compute range without quantifying uncertainty or checking whether the split-fitted curves agree there. For IFEval and MATH Lvl 5, where the paper documents non-stationarity, Table 1's single-number predictions are especially fragile. Please report interval estimates (e.g., bootstrap or split-based) and task-specific caveats for all headline numbers.
  3. [§3.1.2, Figures 12 and 14] The paper's 'mostly stable' claim is supported for four tasks, but the deviations on MATH Lvl 5 and IFEval are concentrated in specific compute bins and are large: e.g., Figure 12 shows OOD coverage error of −15.5% for MATH Lvl 5 in a high-compute bin at t=3, and IFEval shows −11.5% and −8.0% at t=1. The abstract and conclusion single out math reasoning as the exception, but IFEval also shows substantial under-coverage. While the aggregate metrics in Figure 2 are useful, the bin-wise results suggest that the boundary is not uniformly reliable across the compute range for these tasks. Please report which compute regimes and which tasks support the coverage-error <2% claim, and avoid implying uniform stability for the remaining tasks without a per-bin breakdown.
  4. [§2.1, Appendix D] All analysis conditions on estimated pre-training FLOPs, but the provenance and accuracy of these compute values are not examined. The paper relies on third-party estimates (Epoch AI, LifeArchitect, leaderboard metadata) with no discussion of systematic errors (e.g., different FLOP-accounting conventions across families, uncertainty in training tokens, or MoE vs dense equivalences). Since the x-axis is the sole predictor and the fitted sigmoid depends on log-compute, errors in FLOPs could shift the boundary and affect both the shape and the temporal comparisons. Please document the sources and validation of the compute estimates and ideally perform a sensitivity analysis with plausible perturbations.
minor comments (4)
  1. [Throughout] Benchmark name is spelled inconsistently: 'MATH Lvl 5' in most places, 'MA TH Lvl 5' in the heatmap in Appendix D.2, and 'MATH LVL5' in Figure captions. Please standardize.
  2. [Table 2 vs Appendix D.1] Table 2 reports absolute pinball loss and absolute calibration error, but Appendix D.1 says 'In Table 2, we report percent changes relative to the constant baseline.' This is contradictory and should be corrected in one place.
  3. [Appendix H.1] The text refers to 'Mock ATME'; this appears to be a typo for 'AIME'. Please correct.
  4. [Appendix F] The PROTEUS-2K selection procedure (Hugging Face likes plus manually picked official releases) is itself a selection filter and could bias the external-validity check in Section 3.3. This should be discussed explicitly alongside the selection-effects caveat, not only in Appendix F.

Circularity Check

0 steps flagged

No circular derivation: the temporal-stability claim is tested out-of-sample on later periods, and the sigmoid-shape claim is a model comparison against more flexible I-splines.

full rationale

The central derivation is not circular by the paper's own equations. The capability boundary is defined as a conditional 0.98-quantile of observed scores (Sec. 2.1), estimated by smoothed pinball regression; the sigmoid conclusion (Finding 1) is supported by comparison with a strictly more general I-spline class (Table 2), so the sigmoid shape is not assumed into existence. The strongest claim, temporal stability, is genuinely out-of-sample: boundaries are fit on P_t and evaluated on P_{t+1} with coverage error and pinball loss (Sec. 3.1.2, Figs. 2, 12-15); nothing in that procedure forces low OOD error. Table 1's 10^24-FLOP numbers are in-sample evaluations of a no-split fit and are labeled 'estimated,' so they are not presented as independent validation. The acknowledged population caveat (Sec. 2.1: 'this interpretation is relative to the population of reported models we observe'; Appendix A.2) is an external-validity limitation, not a circular reduction, because the paper explicitly does not claim the observed quantile equals an unobserved true attainable frontier. Self-citations (Jin et al. 2025) motivate variance and latent-factor analyses but are not load-bearing; the PCA result is recomputed on the paper's own data (Remark 2, App. H.3). No step reduces by construction to its input.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The capability boundary is a statistical functional (conditional quantile), not a new entity; the balanced I-optimal design is an algorithm, and Proteus-2k is a dataset. No new forces, dimensions, or conserved quantities are postulated.

free parameters (5)
  • quantile level τ = 0.98
    Chosen by hand as 'high but not extreme'; sensitivity to κ/λ checked, but τ itself not varied.
  • smoothing κ and ridge λ = κ=50, λ=1e-3
    Numerical hyperparameters; appendix G sweeps them and shows stability except λ=1e-1.
  • sigmoid boundary parameters (y0, L, a, β) = per-task, per-period
    Fitted by minimizing smoothed pinball loss; these are the estimated boundary itself, not ad hoc constants.
  • size–time model parameters (α, β, φ, δ, θ) = per task, e.g., MATH β̂=0.25, β̂+θ̂=0.76 in Table 3
    Fitted in §5.1 to quantify saturation; used for the task-dependent ceiling narrative.
  • contamination shift γ = positive, p=0.15
    Fitted in §5.2.2 regression logit(AIME)=α+β logit(MATH-500)+γ·1{post-AIME}; not significant.
axioms (5)
  • standard math Smoothed pinball loss minimization recovers the conditional τ-quantile
    Standard quantile regression theory (Koenker & Bassett 1978; Steinwart & Christmann 2011), invoked in §2.1.
  • domain assumption Post-training benchmark scores have a conditional distribution whose τ-quantile is well approximated by a sigmoid in log-compute
    The paper's Finding 1 claims this empirically; used as the estimation model in §2.2.
  • domain assumption Chronological periods P1..P4 are exchangeable enough that Pt+1 serves as validation for Pt
    Temporal validation protocol in §3.1; the MATH boundary shift shows this can fail.
  • ad hoc to paper Evaluation cost is linear in parameter count
    Budget model in §4.1: 'we assume evaluation cost grows roughly linearly with model size'; drives the I-optimal design.
  • ad hoc to paper AIME-2025 and MATH-500 logits are linearly related under the null of no contamination
    Required for the cross-benchmark shift test in §5.2.2; authors state the sigmoid-sigmoid implication.

pith-pipeline@v1.3.0-alltime-deepseek · 33262 in / 11965 out tokens · 118654 ms · 2026-08-02T22:52:19.428028+00:00 · methodology

0 comments
read the original abstract

Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.

Figures

Figures reproduced from arXiv: 2602.15327 by Hanlin Zhang, Jikai Jin, Sham Kakade, Vasilis Syrgkanis.

Figure 1
Figure 1. Figure 1: Sigmoid capability boundaries across time. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Temporal drift and the stability of knowledge-intensive capabilities. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Pre-training vs. post-training scaling laws. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: MATH LVL 5: evaluation on newly released open-weight models. (a) and (b): fitted sigmoid capability boundaries on leaderboard models (red) and newly evaluated models (blue) in periods Pt for t ∈ {3, 4}. (c) and (d): on PROTEUS-2K, fitted capability boundary on leaderboard models in period P4 (red) and on models released after the retirement of the Open LLM Leaderboard. (c) contains models from old base mod… view at source ↗
Figure 5
Figure 5. Figure 5: Performance of balanced I-optimal design as a function of budget parameter α, averaged over t = 1, 2, 3. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Task-dependent saturation on Open LLM Leaderboard v2. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Scaling laws for frontier models. 6 Related Works Scaling laws and downstream predictability. Classical scaling laws relate model size, data, and compute to pretraining loss under controlled settings (Hoffmann et al., 2022; Kaplan et al., 2020). Translating these forecasts into actionable guidance for future model development (Hernandez et al., 2021; Kaplan et al., 2020; McCandlish et al., 2018; Zhang et a… view at source ↗
Figure 8
Figure 8. Figure 8: Visualizing the pinball loss and its effect on boundary fitting. (a) The true pinball loss and its smoothed approximation for several quantile levels τ ; smoothing (softplus) affects only a narrow band around r = 0. (b) The corresponding gradients, showing that positive residuals (boundary below data) produce much larger gradients than negative residuals. (c) On synthetic data, the pinball boundary closely… view at source ↗
Figure 9
Figure 9. Figure 9: Summary diagnostics connecting pretraining compute to post-training capability. [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Pretrained vs. post-trained overlays across all tasks. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Sigmoid capability boundaries over time. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Bin-wise coverage across periods (supplement to Figure [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Bin-wise pinball loss across periods (supplement to Figure [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: In-distribution and out-of-distribution coverage error across log-compute bins for train (Pt) and validation (Pt+1), t = 1, 2, 3 across six Open LLM Leaderboard tasks. 3.48 × 10 1 8.09 × 10 1 1.56 × 10 2 7.23 × 10 2 7.78 × 10 3 BBH GPQA IFEval MATH Lvl 5 MMLU-PRO MUSR 3.5 2.9 4.1 3.8 3.8 2.2 2.1 2.3 2.5 2.6 6.1 4.5 7.5 6.5 6.5 2.6 2.7 3.6 4.2 5.1 3.7 2.7 4.0 3.4 3.8 3.0 2.4 2.8 2.4 2.9 t = 1 | ID 3.48 × 1… view at source ↗
Figure 15
Figure 15. Figure 15: In-distribution and out-of-distribution pinball loss across log-compute bins for train (Pt) and validation (Pt+1), t = 1, 2, 3 across six Open LLM Leaderboard tasks. the remaining ones all have mild ID and OOD errors across different compute bins, implying that the scaling remains stable on these benchmarks. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: compares the scaling of pretrained and post-trained models. The findings are largely similar with the pretraining-compute-based counterparts. One novel finding is that on MUSR, while larger pretrained models do not show a clear benefit over smaller ones, post-trained models have a much clearer scaling in terms of model size. This suggests that for multi-step reasoning tasks, larger models could have large… view at source ↗
Figure 17
Figure 17. Figure 17: Comparison of sigmoid performance boundaries for (Pt) and (Pt+1), 1 ≤ t ≤ 3 across different tasks. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Comparison of sigmoid performance boundaries within each Pt for old and held-out new models. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Comparison of sigmoid performance boundaries within each Pt for old and held-out new models 31 [PITH_FULL_IMAGE:figures/full_fig_p031_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: In-period tuning has negligible effect on cross-temporal generalization. [PITH_FULL_IMAGE:figures/full_fig_p033_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: OOD pinball loss under a (κ, λ) sweep. Lower is better. Results are shown for two tasks and two representative period splits (t=1 and t=3). 33 [PITH_FULL_IMAGE:figures/full_fig_p033_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: OOD absolute coverage error |τˆ − τ | under a (κ, λ) sweep. Lower is better. Coverage is most sensitive to overly large λ, while κ has a weaker effect once it is in a moderate range. H Additional Results H.1 Public Leaderboards of Frontier Models In this subsection, we apply the same methodology to fit a sigmoid scaling law using data from Epoch AI. Compared with the Open LLM Leaderboard, Epoch AI include… view at source ↗
Figure 23
Figure 23. Figure 23: Sigmoid scaling law for frontier models. Evaluation data is publicly available from Artificial Analysis and lifearchitect.ai. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Comparison of sigmoid performance boundaries across different time periods Pt Open LLM Leaderboard v1. H.2 Results for Open LLM Leaderboard v1 In [PITH_FULL_IMAGE:figures/full_fig_p035_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: The scaling of different principal components. [PITH_FULL_IMAGE:figures/full_fig_p036_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Open LLM Leaderboard v2: saturation diagnostics by task. [PITH_FULL_IMAGE:figures/full_fig_p037_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Open LLM Leaderboard v1: saturation diagnostics by task. [PITH_FULL_IMAGE:figures/full_fig_p037_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: In-sample and out-of-sample coverage calibration error on period t + 1 as a function of budget parameter α when the boundary is estimated using balanced I-optimal design on period t. Curves correspond to different evaluation tasks. 5 10 20 50 100 ® (% of evaluation budget) 10 −2 Pinball loss t = 1 In-distribution (ID) 5 10 20 50 100 ® (% of evaluation budget) t = 1 Out-of-distribution (OOD) BBH GPQA IFEva… view at source ↗
Figure 29
Figure 29. Figure 29: In-sample and out-of-sample pinball loss on period t + 1 as a function of budget parameter α when the boundary is estimated using balanced I-optimal design on period t. Curves correspond to different evaluation tasks. 40 [PITH_FULL_IMAGE:figures/full_fig_p040_29.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

    cs.CL 2026-07 conditional novelty 7.5

    Across ~38,000 hours on 134 ultra-long real-world tasks, aggregate agent performance follows a log-sigmoid of interaction time, and measured learning speed doubles about every three months.

Reference graph

Works this paper leans on

37 extracted references · 20 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [3]

    Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning

    Aaron Blakeman, Aaron Grattafiori, Aarti Basant, Abhibha Gupta, Abhinav Khattar, Adi Renduch- intala, Aditya Vavre, Akanksha Shukla, Akhiad Bercovich, Aleksander Ficek, et al. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba-transformer model for agentic reasoning. arXiv preprint arXiv:2512.20848,

    arXiv

  3. [5]

    net/forum?id=sckjveqlCZ

    URL https://openreview. net/forum?id=sckjveqlCZ. 14 Yangyi Chen, Binxuan Huang, Yifan Gao, Zhengyang Wang, Jingfeng Yang, and Heng Ji. Scaling laws for predicting downstream performance in llms.arXiv preprint arXiv:2410.08527,

    Pith/arXiv arXiv

  4. [7]

    Language models scale reliably with over-training and on downstream tasks.arXiv preprint arXiv:2403.08540,

    Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, et al. Language models scale reliably with over-training and on downstream tasks.arXiv preprint arXiv:2403.08540,

    Pith/arXiv arXiv

  5. [11]

    Discovering hierarchical latent capabilities of language models via causal representation learning.arXiv preprint arXiv:2506.10378,

    15 Jikai Jin, Vasilis Syrgkanis, Sham Kakade, and Hanlin Zhang. Discovering hierarchical latent capabilities of language models via causal representation learning.arXiv preprint arXiv:2506.10378,

    Pith/arXiv arXiv

  6. [12]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361,

    Pith/arXiv arXiv 2001

  7. [14]

    Scaling laws are unreliable for downstream tasks: A reality check.arXiv preprint arXiv:2507.00885,

    Nicholas Lourie, Michael Y Hu, and Kyunghyun Cho. Scaling laws are unreliable for downstream tasks: A reality check.arXiv preprint arXiv:2507.00885,

    arXiv

  8. [15]

    An empirical model of large-batch training.arXiv preprint arXiv:1812.06162,

    Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training.arXiv preprint arXiv:1812.06162,

    Pith/arXiv arXiv

  9. [17]

    Olmo 3.arXiv preprint arXiv:2512.13961,

    Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3.arXiv preprint arXiv:2512.13961,

    Pith/arXiv arXiv

  10. [19]

    Why has predicting downstream capabilities of frontier ai models with scale remained elusive?arXiv preprint arXiv:2406.04391,

    Rylan Schaeffer, Hailey Schoelkopf, Brando Miranda, Gabriel Mukobi, Varun Madan, Adam Ibrahim, Herbie Bradley, Stella Biderman, and Sanmi Koyejo. Why has predicting downstream capabilities of frontier ai models with scale remained elusive?arXiv preprint arXiv:2406.04391,

    Pith/arXiv arXiv

  11. [21]

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems.arXiv preprint arXiv:1905.00537,

    Pith/arXiv arXiv 1905

  12. [23]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574,

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark.arXiv preprint arXiv:2406.01574,

    Pith/arXiv arXiv

  13. [24]

    Livebench: A challenging, contamination- limited llm benchmark.arXiv preprint arXiv:2406.19314,

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz- Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. Livebench: A challenging, contamination- limited llm benchmark.arXiv preprint arXiv:2406.19314,

    Pith/arXiv arXiv

  14. [25]

    Knowledge or reasoning? a close look at how llms think across domains.arXiv preprint arXiv:2506.02126,

    Juncheng Wu, Sheng Liu, Haoqin Tu, Hang Yu, Xiaoke Huang, James Zou, Cihang Xie, and Yuyin Zhou. Knowledge or reasoning? a close look at how llms think across domains.arXiv preprint arXiv:2506.02126,

    Pith/arXiv arXiv

  15. [26]

    Unveiling downstream perfor- mance scaling of llms: A clustering-based perspective.arXiv preprint arXiv:2502.17262,

    17 Chengyin Xu, Kaiyuan Chen, Xiao Li, Ke Shen, and Chenggang Li. Unveiling downstream perfor- mance scaling of llms: A clustering-based perspective.arXiv preprint arXiv:2502.17262,

    arXiv

  16. [27]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,

    Pith/arXiv arXiv

  17. [28]

    Train-before-test harmonizes language model rankings.arXiv preprint arXiv:2507.05195, 2025a

    Guanhua Zhang, Ricardo Dominguez-Olmedo, and Moritz Hardt. Train-before-test harmonizes language model rankings.arXiv preprint arXiv:2507.05195, 2025a. Guanhua Zhang, Florian E Dorner, and Moritz Hardt. How benchmark prediction from fewer data misses the mark.arXiv preprint arXiv:2506.07673, 2025b. Hanlin Zhang, Depen Morwani, Nikhil Vyas, Jingfeng Wu, Di...

    arXiv 1909

  18. [29]

    For a residual r=y−q τ (z;θ), the true pinball loss at quantile levelτ∈(0,1)is ρτ (r) = max τ r,(τ−1)r , which is piecewise linear with a kink at r= 0

    that explicitly targets the upper envelope of the data. For a residual r=y−q τ (z;θ), the true pinball loss at quantile levelτ∈(0,1)is ρτ (r) = max τ r,(τ−1)r , which is piecewise linear with a kink at r= 0 . Minimizing its expected value recovers the τ-quantile of the (conditional) response distribution (Koenker and Bassett, 1978). In practice we use a s...

    1978

  19. [30]

    capability boundary,

    The key property of the pinball loss is itsasymmetry. As shown in Figure 8b, the gradient gτ (r) = ∂˜ρτ (r) ∂r is approximately τ−1 for r <0 and τ for r >0. For high quantiles (e.g., τ= 0.98 ), this means that points lyingabovethe boundary (positive residuals, y > qτ ) incur a much larger gradient magnitude than those below it. Intuitively, the optimizer ...

    1988

  20. [31]

    To avoid extrapolating beyond observed compute, OOD evaluation is restricted to the overlap of the training and validation ranges inz= log 10 C

    Rolling train/validation protocol and overlap restriction.For each temporal split t∈ {1,2,3}, we fit each boundary estimator on Pt and evaluate out-of-distribution (OOD) on Pt+1. To avoid extrapolating beyond observed compute, OOD evaluation is restricted to the overlap of the training and validation ranges inz= log 10 C. The table below provides detailed...

    2024

  21. [32]

    This includes Allen AI’s OLMo-3 (Olmo et al., 2025), NVIDIA’s Nemotron nano (Blake- man et al.,

    • The most recent models officially released by well- known industry labs near the end of 2025, which we manually picked. This includes Allen AI’s OLMo-3 (Olmo et al., 2025), NVIDIA’s Nemotron nano (Blake- man et al.,

    2025

  22. [33]

    We release the evaluation results at https: //huggingface.co/datasets/hlzhang109/ proteus-2k

    and cascade (Wang et al., 2025). We release the evaluation results at https: //huggingface.co/datasets/hlzhang109/ proteus-2k. The most up-to-date part of proteus-2k, namely those built on new base models that do not appear on the Open LLM Leaderboard, are pro- vided at https://huggingface.co/datasets/ hlzhang109/proteus-selected. The results for models r...

    2025

  23. [34]

    GPQA diamond, on the other hand, indicates a clear scaling in the FLOPs

    We can see that MATH Lvl 5 and Mock ATME show no pattern of performance gain from increasing the FLOPs. GPQA diamond, on the other hand, indicates a clear scaling in the FLOPs. 1022 1023 1024 1025 Pretraining Compute (FLOPs) 0.0 0.2 0.4 0.6 0.8Score Sigmoid (loss = 0.0087) I-spline (loss = 0.0088) (a)AIME 2025 ( https: //artificialanalysis.ai/ evaluations...

    2025

  24. [35]

    slow death of scaling

    The scaling of different principal components. I Saturation Analysis across Open LLM Leaderboard Versions and Tasks This appendix provides the complete set of plots used to discuss saturation effects and the “slow death of scaling” narrative. We reproduce the core logic of Hooker (2025, Figure

    2025

  25. [36]

    parameter scaling law

    on the Open LLM Leaderboard v1 and v2. These plots are observational: they reflect submitted models, training recipes, post-training, and benchmark targeting over time. They should not be read as a controlled “parameter scaling law”; rather, they summarize how easily larger models translate into higher leaderboard scores for a given task. Open LLM Leaderb...

    2025

  26. [37]

    Open LLM Leaderboard v2: saturation diagnostics by task. Sep Nov 2024 Mar May Jul Date 20 40 60 80ARC (%) large models > 13B small models < 13B dominated large (a)ARC Sep Nov 2024 Mar May Jul Date 20 40 60 80HellaSwag (%) large models > 13B small models < 13B dominated large (b)HELLASWAG Sep Nov 2024 Mar May Jul Date 20 40 60 80MMLU (%) large models > 13B...

    2024

  27. [1978]

    Jakub Krajewski, Amitis Shidani, Dan Busbridge, Sam Wiseman, and Jason Ramapuram

    doi: 10.2307/1913643. Jakub Krajewski, Amitis Shidani, Dan Busbridge, Sam Wiseman, and Jason Ramapuram. Revisiting the scaling properties of downstream metrics in large language model training.arXiv preprint arXiv:2512.08894,

    arXiv

  28. [1995]

    Training on the test task con- founds evaluation and emergence.arXiv preprint arXiv:2407.07890,

    Ricardo Dominguez-Olmedo, Florian E Dorner, and Moritz Hardt. Training on the test task con- founds evaluation and emergence.arXiv preprint arXiv:2407.07890,

    Pith/arXiv arXiv

  29. [2011]

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report.arXiv preprint arXiv:2503.19786,

    Pith/arXiv arXiv

  30. [2016]

    Scaling laws for transfer.arXiv preprint arXiv:2102.01293,

    Danny Hernandez, Tom Brown, Tom Conerly, et al. Scaling laws for transfer.arXiv preprint arXiv:2102.01293,

    Pith/arXiv arXiv

  31. [2018]

    Language models improve when pretraining data matches target tasks.arXiv preprint arXiv:2507.12466,

    David Mizrahi, Anders Boesen Lindbo Larsen, Jesse Allardice, Suzie Petryk, Yuri Gorokhov, Jeffrey Li, Alex Fang, Josh Gardner, Tom Gunter, and Afshin Dehghan. Language models improve when pretraining data matches target tasks.arXiv preprint arXiv:2507.12466,

    Pith/arXiv arXiv

  32. [2019]

    Nemotron-cascade: Scaling cascaded reinforcement learning for general-purpose reasoning models.arXiv preprint arXiv:2512.13607,

    Boxin Wang, Chankyu Lee, Nayeon Lee, Sheng-Chieh Lin, Wenliang Dai, Yang Chen, Yangyi Chen, Zhuolin Yang, Zihan Liu, Mohammad Shoeybi, et al. Nemotron-cascade: Scaling cascaded reinforcement learning for general-purpose reasoning models.arXiv preprint arXiv:2512.13607,

    arXiv

  33. [2021]

    Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

    Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically.arXiv preprint arXiv:1712.00409,

    Pith/arXiv arXiv

  34. [2022]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249,

    Pith/arXiv arXiv

  35. [2023]

    gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925,

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925,

    Pith/arXiv arXiv

  36. [2024]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901,

    1901

  37. [2025]

    Artificial hivemind: The open-ended homogeneity of language models (and beyond).arXiv preprint arXiv:2510.22954,

    Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, and Yejin Choi. Artificial hivemind: The open-ended homogeneity of language models (and beyond).arXiv preprint arXiv:2510.22954,

    arXiv