Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

When AI agents learn from real-world environments, average performance rises with interaction time along a precise log-sigmoid curve.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 07:55 UTC pith:KZD2SRA4

load-bearing objection Strong empirical regularity on day-long agent learning curves, plus a real benchmark; the “single intrinsic log-sigmoid law” packaging is a bit tighter than the uniqueness evidence. the 4 major comments →

arxiv 2607.05155 v1 pith:KZD2SRA4 submitted 2026-07-06 cs.CL cs.LG

EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

Deyao Zhu , Xin Zhou , Shengling Qin , Xuekai Zhu , Hangliang Ding , Shu Zhong , Zixin Wen , Zhonglin Xie
show 39 more authors
Chenhui Gou Linxuan Ren Yueyang Wang Junfeng Zhong Rui Liu Tian Gao Yangguang Lin Jingyuan Zhang Maojia Song Xuan Qi Jinhong Wu Chenyang Zhang Yinzhu Piao Ziru Niu Hongbin Lin Lingxiang Meng Peng Tang Chengyao Tang Shanyu Wu Huanyu Zheng Yu Liu Liya Zhu He Wang Ming Ding Ziyu Wan Hao Liu Sibo Wang Haotian Zhu Xintian Zhang Nan Chai Yipeng Liu Panhao Lai Sihang Yuan Zixin Su Ge Zhang Wangchunshu Zhou Yantao Du Wenhao Huang Guang Shi
This is my paper
classification cs.CL cs.LG
keywords environment learningscaling lawslog-sigmoidAI agentslong-horizon benchmarksinteraction timefrontier expansionEdgeBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Pretraining scaling laws tell us how models improve with data and compute, but what happens after deployment—when an agent must improve by interacting with a messy, feedback-rich environment—has been much less clear. This paper builds EdgeBench, a suite of 134 day-scale real-world tasks spanning science, software, optimization, professional work, formal math, and games, and runs frontier agents for roughly 38,000 hours of environment interaction. Averaging best-so-far performance over those tasks, it reports that overall progress follows a simple three-parameter log-sigmoid in interaction time with very high fit quality, and that this form holds across task families, longer horizons, and early-to-late forecasts. It also reports that measured learning speed of frontier agents has been roughly doubling every three months. The point is not just a new leaderboard: the paper treats environment learning itself as a scalable, measurable object, and argues that the regular curve is what you should expect when many tasks behave like frontier expansion on latent score graphs.

Core claim

Across 134 diverse real-world tasks and about 38,000 hours of agent–environment interaction, overall (task-averaged) best-so-far performance during environment learning follows a log-sigmoid scaling law of interaction time, S(t) = Smax / (1 + (tmid/t)^β), with mean R² around 0.998. The same functional form appears by task family, under 28- and 72-hour windows, and when early trajectories forecast later performance; separately, agent learning speed on a fixed slice roughly doubles every three months across recent model releases.

What carries the argument

The log-sigmoid S(t) = Smax / (1 + (tmid/t)^β), where t is interaction time, Smax is the attainable score ceiling, tmid is the half-ceiling time, and β is the steepness in log time. The paper derives this as the many-task limit of a frontier-expansion process on latent task graphs: unlocked score mass helps unlock remaining mass, the expected growth rate is proportional to x(1−x) in a logarithmic time coordinate induced by self-similar search geometry, and averaging washes out per-task jaggedness when midpoints and speeds concentrate.

Load-bearing premise

The clean law only becomes precise after averaging many tasks whose midpoints and learning speeds line up, and whose underlying task graphs mix influence across the unlocked–locked frontier without strong bottlenecks.

What would settle it

Fit the same three-parameter log-sigmoid to a larger, deliberately heterogeneous mix of tasks (or to single families kept separate) over horizons well past 12 hours; a persistent failure of R², systematic residual structure, or a better fit by a non-log-time S-curve or multi-inflection mixture would undercut the claimed law.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Environment learning can be treated as a scaling object with its own predictable curve, not only as a pile of idiosyncratic task outcomes.
  • Early trajectory segments can be used to forecast later performance under a fixed interaction budget.
  • Progress across model generations can be tracked by learning speed (gain per fixed hours) rather than only by final score.
  • Long-horizon evaluation should report time-aligned improvement trajectories, not only end-state success.
  • Stateful continuous runs that reuse feedback should outperform equal-budget independent restarts if the frontier mechanism is right.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the law is real, post-deployment interaction budgets may become a first-class training and product dial, analogous to how pretraining compute is planned today.
  • Harness design that preserves reusable state (workspace, memory, longer context) may move tmid and β as much as base model upgrades do.
  • Tasks engineered with strong single bottlenecks or non-scale-free feedback cycles would be natural stress tests for where the population-level log-sigmoid breaks.
  • A practical monitoring tool could watch live agent runs for deviation from an early-fit log-sigmoid as a signal of stuck exploration or serving failure.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces EdgeBench, a suite of 134 ultra-long-horizon real-world agent tasks spanning six capability families, and uses roughly 38,000 hours of frontier-agent interaction to study post-deployment environment learning. The central empirical claim is that task-averaged best-so-far performance follows a three-parameter log-sigmoid in interaction time, S(t)=Smax/(1+(tmid/t)^β), with mean R²≈0.998 on the full 134-task average, similarly tight family-level and longer-horizon (28h/72h) fits, and accurate forecasts of later hours from early windows. A frontier-expansion theory on latent task graphs is offered as a sufficient mechanism; a secondary claim is that agent learning speed on a fixed 18-task slice roughly doubles every three months across recent model releases. Supporting analyses include experience-versus-restart ablations, context-length comparisons, submission-efficiency statistics, and a detailed gravitational-wave case study.

Significance. If the aggregate regularity holds under broader scrutiny, this is a substantial contribution: it elevates post-deployment environment learning from anecdotal trajectories to a measurable scaling object, analogous to pretraining and test-time scaling laws, and supplies a large, carefully engineered dual-loop benchmark (work/judge isolation, multilevel feedback, day-scale horizons) that the field currently lacks. Strengths include the scale of the evaluation, public release of 51 tasks plus the full harness, explicit comparison against alternative S-curves (Table 1), predictive early-window tests (Figure 7), and the experience-versus-independent-restart design (Section 5.2), which cleanly separates stateful learning from repeated sampling. The theoretical appendix is unusually thorough for an empirical agent paper and states its failure modes. Even if the unique log-sigmoid interpretation is moderated, the empirical finding of smooth, saturating, log-time-predictable aggregate learning curves across heterogeneous real tasks would remain important.

major comments (4)
  1. Table 1 and Appendix E: the manuscript claims a specific log-sigmoid scaling law, but full-window RMSE for log-probit (0.398), log-Gompertz (0.402), and Weibull (0.404) is within ~3% of log-sigmoid (0.390), and Appendix E explicitly states that the form cannot be chosen by fit. The abstract, title, and Section 3.2 therefore overstate uniqueness. Please reframe the central claim as high-precision evidence for smooth, saturating, log-time S-shaped aggregate curves (with log-sigmoid preferred on mechanistic grounds), report confidence intervals or bootstrap uncertainty on relative RMSE, and avoid language that implies a uniquely identified functional law from the data alone.
  2. Figure 5 versus Assumptions D.1–D.2 (Appendix D.3): family-level fits show extreme tmid dispersion (e.g., ~0.4h to 240h across families/models) and heterogeneous β. The aggregate theorem requires residual midpoint alignment and speed concentration for the average of task frontiers to collapse to a single log-sigmoid rather than a convolution of shifted/heterogeneous sigmoids. The paper does not empirically test these concentration conditions (e.g., distribution of per-task tmid/β, residual mixture diagnostics, or whether fitted aggregate parameters are window-stable beyond the reported forecasts). Without that, R²=0.998 on the 134-task mean strongly supports a smooth population average but only weakly supports a single intrinsic frontier law. Add these diagnostics or qualify the theory-to-data link accordingly.
  3. Section 4 and Figure 9: the ~3-month doubling claim rests on a hand-selected 18-task slice chosen for similar first-attempt performance, a rolling top-2 frontier fit, and a short calendar window of releases. Report sensitivity to slice composition, to using all models rather than rolling top-2, and to alternative learning-speed definitions (e.g., 4h or 6h gains; tmid-based speed). As written, the doubling timescale is presented as a robust generational law while remaining a free empirical summary of a small, selected panel.
  4. Section 3.1 / fit protocol: aggregate curves are fit with three free parameters to a modest number of time checkpoints (roughly hourly over 12h, plus longer-horizon subsets). High R² is expected for smooth monotone averages under a flexible S-curve. Please report degrees of freedom, parameter standard errors, and leave-one-family-out or leave-one-model-out stability of (Smax, β, tmid), and clarify that Smax is an effective ceiling over the fitted regime (as Appendix D.5 notes) rather than an absolute performance bound.
minor comments (6)
  1. Only 51 of 134 tasks are publicly released. State clearly which families and difficulty strata are held back and how external groups can reproduce the aggregate scaling fits without the full suite.
  2. Figure 1 and related plots: report whether scores are mean of best-so-far across three seeds or max, and whether error bands are available; several per-task tables mark * for <3 valid runs (especially GPT-5.4), which should be reflected in aggregate uncertainty.
  3. Appendix B serving incidents for GPT-5.4 are important; consider a short main-text caveat near Figure 7 so forecast deviation is not read as pure model failure.
  4. Notation: u = log t − log tmid is introduced cleanly in Appendix D but used earlier in Section 3.3; a one-line pointer would help non-theory readers.
  5. Related work (Section 6 / Appendix F) is thorough; a compact table column for “primary reported quantity is trajectory vs endpoint” already exists—ensure AutoLab and FrontierSWE runtime comparisons are consistent with their public numbers.
  6. Minor polish: “DeepSeek V4 Pro (preview)” naming, occasional future-dated system cards, and a few figure captions that restate R² without defining the score scale (0–100) could be tightened.

Circularity Check

0 steps flagged

No significant circularity: free log-sigmoid parameters are standard scaling-law practice; forecasting and theory are not forced by construction.

full rationale

The central claim is that task-averaged best-so-far performance follows S(t)=Smax/(1+(tmid/t)^β). Smax, β, and tmid are free empirical descriptors fitted to trajectories (Eq. 1; Fig. 1), not quantities derived from independent first principles and then re-presented as predictions. That is ordinary scaling-law methodology, not self-definitional circularity. Predictive content is tested by fitting only the first 6.5 h and evaluating held-out 6.5–12 h trajectories (Fig. 7; R²≥0.997, RMSE<1), which would fail if the form did not extrapolate. Alternative S-curves are compared honestly (Table 1; Appendix E), and the authors state that fit alone cannot select among near-tied families. The Appendix D theory is a conditional derivation: under weighted cut-mixing, small score units, midpoint alignment, speed concentration, and self-similar graph geometry, the aggregate converges to a log-sigmoid. Those are stated assumptions with explicit failure modes (D.5), not inputs smuggled in as outputs. There is no load-bearing self-citation uniqueness theorem, no ansatz imported from overlapping prior work as external fact, and no renaming of a known environment-learning law as a new derivation. Dispersion of family-level tmid (Fig. 5) and near-tied S-curve RMSE affect uniqueness and strength of the mechanistic claim, not circularity of the derivation chain.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The empirical claim rests on fitted three-parameter log-sigmoids and a selected task slice for learning speed. The theoretical claim rests on modeling tasks as latent score-unit graphs with mean-field cut mixing, concentrated midpoints/speeds, and self-similar search geometry that makes effective progress logarithmic in raw time. These are modeling assumptions plus free fit parameters, not parameter-free theorems.

free parameters (5)
  • Smax (per model / fit window)
    Attainable score ceiling in the fitted regime; free parameter of Eq. (1), not predicted a priori.
  • β (per model / family / horizon)
    Log-time steepness / effective frontier speed; fitted, then interpreted as ηκ or γ/h.
  • tmid (per model / family / horizon)
    Time to half of Smax; free location parameter of the log-sigmoid.
  • Learning-speed doubling timescale (~3 months)
    Obtained from log-linear fit to rolling top-2 leaders’ two-hour gains on the fixed 18-task slice; not a universal constant.
  • 18-task slice selection for learning-speed study
    Tasks chosen so first-attempt scores are comparable (mean 6.87±0.97); selection affects the measured trend.
axioms (6)
  • ad hoc to paper Environment learning is a frontier expansion process on a latent task graph of score units with irreversible unlocks.
    Central modeling premise of §3.3 and Appendix D; not independently measured graph structure.
  • ad hoc to paper Weighted cut-mixing: aggregate unlocked→locked influence ≈ product of unlocked and locked score mass (Condition D.1/D.3).
    Required to get dx/du = β x(1−x); paper states it is weaker than entrywise equality but still an assumption.
  • domain assumption Task midpoints and frontier speeds concentrate across the benchmark so the average of task sigmoids is one sigmoid (Assumptions D.1–D.2).
    Needed for many-task limit Theorem D.2; empirical smoothness is offered as support.
  • ad hoc to paper Self-similar task-graph edge growth makes exposed difficulty scale as log t (Assumptions D.3–D.4).
    Justifies log-time coordinate rather than raw-time logistic/Weibull dynamics.
  • domain assumption Best-so-far performance under dual-loop feedback is a valid readout of environment learning (not only repeated sampling or prior knowledge).
    Addressed partly by experience-vs-restart ablation (§5.2) and matched initial performance on the 18-task slice.
  • standard math Standard continuous-time Markov unlock hazards and mean-field/Doob–Meyer arguments for jump processes.
    Used throughout Appendix D proofs.
invented entities (2)
  • Latent task graph of score units with influence matrix K no independent evidence
    purpose: Provide a mechanistic substrate whose frontier cut yields logistic growth in an effective coordinate.
    Postulated structure; not recovered from data as an explicit graph for each task.
  • EdgeBench dual-loop work/judge environments as a general measure of self-evolution independent evidence
    purpose: Operationalize long-horizon environment learning with multilevel feedback.
    New benchmark construct; independent use depends on public tasks and harness.

pith-pipeline@v1.1.0-grok45 · 61041 in / 3816 out tokens · 37831 ms · 2026-07-11T07:55:56.899746+00:00 · methodology

0 comments
read the original abstract

Pretraining scaling laws reveal that model capability improves predictably with data and compute. But learning from real world environments after deployment remains far less understood. Analyzing roughly 38,000 hours of agent interaction with the environment across 134 real world tasks, we find, to the best of our knowledge, the first evidence that overall performance during environment learning follows a log-sigmoid scaling law with remarkably high precision, reaching R^2 = 0.998. Across model generations, we also find that agent learning speed roughly doubles every three months. This discovery stems from EdgeBench, a suite of 134 real world tasks with ultra-long horizons, spanning scientific discovery, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games. Each task sustains at least 12 hours of continuous agent operation under rich, multilevel feedback, and is built through substantial expert effort. We publicly release 51 tasks and our full evaluation framework to accelerate the study of how agents learn from real world experience.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

    cs.CL 2026-07 conditional novelty 6.0

    In a controllable skill-graph gym, world-model CoT, a little long-horizon data, and high-quality trajectories acquire planning; OPD shapes patterns more reliably than GRPO; MOPD integrates only shared compatible patterns.

Reference graph

Works this paper leans on

87 extracted references · 38 linked inside Pith · cited by 1 Pith paper

  1. [1]

    B. P. Abbott et al. Observation of gravitational waves from a binary black hole merger.Physical Review Letters, 116(6):061102, 2016. doi: 10.1103/PhysRevLett.116.061102

    doi:10.1103/physrevlett.116.061102 2016

  2. [2]

    The Claude Model Family

    Anthropic. The Claude Model Family. https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.5_Addendum.pdf, 2024

    2024

  3. [3]

    Claude Opus 4.8 System Card

    Anthropic. Claude Opus 4.8 System Card. https://www-cdn.anthropic.com/ 0b4915911bb0d19eca5b5ee635c80fef830a37ea.pdf, May 2026. System card. Released May 28, 2026; updated June 3, 2026

    2026

  4. [4]

    Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, et al

    Parth Asawa, Christopher M. Glaze, Gabriel Orlanski, Ramya Ramakrishnan, Benji Xu, et al. Continual learning bench: Evaluating frontier AI systems in real-world stateful environments.arXiv preprint arXiv:2606.05661, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    Self-organized criticality: An explanation of 1/f noise.Physical Review Letters, 59(4):381–384, 1987

    Per Bak, Chao Tang, and Kurt Wiesenfeld. Self-organized criticality: An explanation of 1/f noise.Physical Review Letters, 59(4):381–384, 1987

    1987

  6. [6]

    Application of the logistic function to bio-assay.Journal of the American Statistical Association, 39(227):357–365, 1944

    Joseph Berkson. Application of the logistic function to bio-assay.Journal of the American Statistical Association, 39(227):357–365, 1944. doi: 10.1080/01621459.1944.10500699

    doi:10.1080/01621459.1944.10500699 1944

  7. [7]

    Smith, Dirk Groeneveld, Pang Wei Koh, Jesse Dodge, and Hannaneh Hajishirzi

    Akshita Bhagia, Jiacheng Liu, Alexander Wettig, David Heineman, Oyvind Tafjord, Ananya Harsh Jha, Luca Soldaini, Noah A. Smith, Dirk Groeneveld, Pang Wei Koh, Jesse Dodge, and Hannaneh Hajishirzi. Establishing task scaling laws via compute-efficient model ladders.arXiv preprint arXiv:2412.04403, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Chester I. Bliss. The method of probits.Science, 79(2037):38–39, 1934

    2037

  9. [9]

    Le, Christopher Ré, and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling.arXiv preprint arXiv:2407.21787, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    MLE-bench: Evaluating machine learning agents on machine learning engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. MLE-bench: Evaluating machine learning agents on machine learning engineering. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  11. [11]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery

    Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, et al. ScienceAgentBench: Toward rigorous assessment of language agents for data-driven scientific discovery. InInternational Conference on Learning Representations, 2025

    2025

  13. [13]

    LLF-bench: Benchmark for interactive learning from language feedback.arXiv preprint arXiv:2312.06853, 2023

    Ching-An Cheng, Andrey Kolobov, Dipendra Misra, Allen Nie, and Adith Swaminathan. LLF-bench: Benchmark for interactive learning from language feedback.arXiv preprint arXiv:2312.06853, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Frontier-Eng: Benchmarking self-evolving agents on real-world engineering tasks with generative optimization.arXiv preprint arXiv:2604.12290, 2026

    Yizhe Chi, Deyao Hong, Dapeng Jiang, Tianwei Luo, Kaisen Yang, Boshi Zhang, et al. Frontier-Eng: Benchmarking self-evolving agents on real-world engineering tasks with generative optimization.arXiv preprint arXiv:2604.12290, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    FrontierSWE: Benchmarking coding agents at the limits of human abilities.https://www.frontierswe.com/blog, 2026

    Evan Chu, Rajan Agarwal, Abishek Thangamuthu, Brendan Graham, and Justus Mattern. FrontierSWE: Benchmarking coding agents at the limits of human abilities.https://www.frontierswe.com/blog, 2026

    2026

  16. [16]

    DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Pith/arXiv arXiv 2025

  17. [17]

    DeepSeek-V4: Towards highly efficient million-token context intelligence.https://huggingface

    DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence.https://huggingface. co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf, April 2026. Technical report

    2026

  18. [18]

    NL2Repo-Bench: Towards long-horizon repository generation evaluation of coding agents.arXiv preprint arXiv:2512.12730, 2025

    Jingzhe Ding, Shengda Long, Changxin Pu, et al. NL2Repo-Bench: Towards long-horizon repository generation evaluation of coding agents.arXiv preprint arXiv:2512.12730, 2025

    arXiv 2025

  19. [19]

    EvaLearn: Quantifying the learning capability and efficiency of LLMs via sequential problem solving.arXiv preprint arXiv:2506.02672, 2025

    Shihan Dou, Ming Zhang, Chenhao Huang, Jiayi Chen, Feng Chen, Shichun Liu, Yan Liu, Chenxiao Liu, Cheng Zhong, Zongzhang Zhang, et al. EvaLearn: Quantifying the learning capability and efficiency of LLMs via sequential problem solving.arXiv preprint arXiv:2506.02672, 2025. 19

    arXiv 2025

  20. [20]

    CL-bench Life: Can language models learn from real-life context?arXiv preprint arXiv:2604.27043, 2026

    Shihan Dou, Yujiong Shen, Chenhao Huang, Junjie Ye, Jiayi Chen, Junzhe Wang, Qianyu He, Shichun Liu, Changze Lv, Jiahang Lin, et al. CL-bench Life: Can language models learn from real-life context?arXiv preprint arXiv:2604.27043, 2026

    Pith/arXiv arXiv 2026

  21. [21]

    CL-bench: A benchmark for context learning.arXiv preprint arXiv:2602.03587, 2026

    Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, et al. CL-bench: A benchmark for context learning.arXiv preprint arXiv:2602.03587, 2026

    arXiv 2026

  22. [22]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Faroughy, Sofia Palacios Schweitzer, Ian Pang, Siddharth Mishra-Sharma, and David Shih

    Darius A. Faroughy, Sofia Palacios Schweitzer, Ian Pang, Siddharth Mishra-Sharma, and David Shih. Collider- Bench: Benchmarking AI agents with particle physics analysis reproduction.arXiv preprint arXiv:2605.13950, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    GLM-5: from vibe coding to agentic engineering.https://arxiv.org/abs/2602.15763, 2026

    GLM-5 Team. GLM-5: from vibe coding to agentic engineering.https://arxiv.org/abs/2602.15763, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Benjamin Gompertz. On the nature of the function expressive of the law of human mortality, and on a new mode of determining the value of life contingencies.Philosophical Transactions of the Royal Society of London, 115: 513–583, 1825

  28. [28]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  29. [29]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InAdvances in Neural Information Processing Systems, 2021

    2021

  30. [30]

    Scaling laws for single-agent reinforcement learning.arXiv preprint arXiv:2301.13442, 2023

    Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning.arXiv preprint arXiv:2301.13442, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  32. [32]

    Everything Is a Ralph Loop.https://ghuntley.com/loop/, January 2026

    Geoffrey Huntley. Everything Is a Ralph Loop.https://ghuntley.com/loop/, January 2026. Blog post, January 17, 2026

    2026

  33. [33]

    ALE-bench: A benchmark for long-horizon objective-driven algorithm engineering.arXiv preprint arXiv:2506.09050, 2025

    Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba. ALE-bench: A benchmark for long-horizon objective-driven algorithm engineering.arXiv preprint arXiv:2506.09050, 2025

    arXiv 2025

  34. [34]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InThe Twelfth International Conference on Learning Representations, 2024

    2024

  35. [35]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  36. [36]

    Dhillon, David Brandfonbrener, and Rishabh Agarwal

    Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Sai Surya Duvvuri, Manzil Zaheer, Inderjit S. Dhillon, David Brandfonbrener, and Rishabh Agarwal. The art of scaling reinforcement learning compute for LLMs.arXiv preprint arXiv:2510.13786, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Measuring AI ability to complete long tasks.arXiv preprint arXiv:2503.14499, 2025

    Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, et al. Measuring AI ability to complete long tasks.arXiv preprint arXiv:2503.14499, 2025. 20

    arXiv 2025

  38. [38]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushm...

    Pith/arXiv arXiv 2022

  39. [39]

    Introducing FrontierCode.https://cognition.ai/blog/frontier-code, 2026

    Eric Lu, Ben Pan, Deniz Birlikci, Sam Lee, Ray Wang, Rohan Choudhury, Fermi Ma, TC Qin, Carlo Baronio, Silas Alberti, et al. Introducing FrontierCode.https://cognition.ai/blog/frontier-code, 2026

    2026

  40. [40]

    MLS-Bench: A holistic and rigorous assessment of AI systems on building better AI.arXiv preprint arXiv:2605.08678, 2026

    Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, et al. MLS-Bench: A holistic and rigorous assessment of AI systems on building better AI.arXiv preprint arXiv:2605.08678, 2026

    Pith/arXiv arXiv 2026

  41. [41]

    FrontierCS: Evolving challenges for evolving intelligence.arXiv preprint arXiv:2512.15699, 2025

    Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du, Hanchen Li, Shu Liu, Edwin Chen, Yichuan Wang, et al. FrontierCS: Evolving challenges for evolving intelligence.arXiv preprint arXiv:2512.15699, 2025

    arXiv 2025

  42. [42]

    Humans still beat AI in the long horizon: Revisiting test-time scaling in the agent era.https://joyemang33.github.io/blog/ 2026/humans-dont-just-sample/, 2026

    Qiuyang Mang, Kaiyuan Liu, Bo Peng, Shreyas Pimpalgaonkar, Alex Dimakis, and Alvin Cheung. Humans still beat AI in the long horizon: Revisiting test-time scaling in the agent era.https://joyemang33.github.io/blog/ 2026/humans-dont-just-sample/, 2026

    2026

  43. [43]

    MAA invitational competitions: American invitational mathematics examination (AIME).https://maa.org/maa-invitational-competitions/, 2026

    Mathematical Association of America. MAA invitational competitions: American invitational mathematics examination (AIME).https://maa.org/maa-invitational-competitions/, 2026

    2026

  44. [44]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, et al. Terminal- Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. InThe Fourteenth International Conference on Learning Representations, 2026

    2026

  45. [45]

    Mark E. J. Newman. Power laws, pareto distributions and zipf’s law.Contemporary Physics, 46(5):323–351, 2005

    2005

  46. [46]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Learning to reason with LLMs.https://openai.com/index/learning-to-reason-with-llms/, 2024

    OpenAI. Learning to reason with LLMs.https://openai.com/index/learning-to-reason-with-llms/, 2024

    2024

  48. [48]

    Introducing SWE-bench verified

    OpenAI. Introducing SWE-bench verified. https://openai.com/index/introducing-swe-bench-verified/, 2024

    2024

  49. [49]

    Computer-using agent.https://openai.com/index/computer-using-agent/, 2025

    OpenAI. Computer-using agent.https://openai.com/index/computer-using-agent/, 2025

    2025

  50. [50]

    GPT-4.5 System Card.https://cdn.openai.com/gpt-4-5-system-card-2272025.pdf, 2025

    OpenAI. GPT-4.5 System Card.https://cdn.openai.com/gpt-4-5-system-card-2272025.pdf, 2025

    2025

  51. [51]

    Update to GPT-5 System Card: GPT-5.2

    OpenAI. Update to GPT-5 System Card: GPT-5.2. https://cdn.openai.com/pdf/ 3a4153c8-c748-4b71-8e31-aecbde944f8d/oai_5_2_system-card.pdf, 2025

    2025

  52. [52]

    GPT-5 System Card.arXiv preprint arXiv:2601.03267, 2025

    OpenAI. GPT-5 System Card.arXiv preprint arXiv:2601.03267, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    Follow a Goal

    OpenAI. Follow a Goal. https://developers.openai.com/codex/use-cases/follow-goals, 2026. Codex documentation. Accessed June 18, 2026

    2026

  54. [54]

    GPT-5.4 Thinking System Card

    OpenAI. GPT-5.4 Thinking System Card. https://deploymentsafety.openai.com/gpt-5-4-thinking/ gpt-5-4-thinking.pdf, March 2026. System card. Published March 5, 2026

    2026

  55. [55]

    GPT-5.5 System Card

    OpenAI. GPT-5.5 System Card. https://deploymentsafety.openai.com/gpt-5-5/gpt-5-5.pdf, April 2026. System card. Published April 23, 2026

    2026

  56. [56]

    How predictable is language model benchmark performance?arXiv preprint arXiv:2401.04757, 2024

    David Owen. How predictable is language model benchmark performance?arXiv preprint arXiv:2401.04757, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    GDPval: Evaluating AI model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374, 2025

    Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simon Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, et al. GDPval: Evaluating AI model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374, 2025

    Pith/arXiv arXiv 2025

  58. [58]

    PRBench: End-to-end paper reproduction in physics research.arXiv preprint arXiv:2603.27646, 2026

    Shi Qiu, Junyi Deng, Yiwei Deng, Haoran Dong, Jieyu Fu, Mao Li, Zeyu Li, Zhaolong Zhang, Huiwen Zheng, Leidong Bao, et al. PRBench: End-to-end paper reproduction in physics research.arXiv preprint arXiv:2603.27646, 2026. 21

    arXiv 2026

  59. [59]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark.arXiv preprint arXiv:2311.12022, 2023

    Pith/arXiv arXiv 2023

  60. [60]

    HCAST: Human-calibrated autonomy software tasks.arXiv preprint arXiv:2503.17354, 2025

    David Rein, Joel Becker, Amy Deng, Seraphina Nix, Chris Canal, Daniel O’Connel, Pip Arnott, Ryan Bloom, Thomas Broadley, et al. HCAST: Human-calibrated autonomy software tasks.arXiv preprint arXiv:2503.17354, 2025

    Pith/arXiv arXiv 2025

  61. [61]

    Maddison, and Tatsunori Hashimoto

    Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance.arXiv preprint arXiv:2405.10938, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Thinking vs

    Junhong Shen, Hao Bai, Lunjun Zhang, Yifei Zhou, Amrith Setlur, Shengbang Tong, Diego Caples, Nan Jiang, Tong Zhang, Ameet Talwalkar, and Aviral Kumar. Thinking vs. doing: Agents that reason by scaling test-time interaction.arXiv preprint arXiv:2506.07976, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan

    Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark.arXiv preprint arXiv:2409.11363, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters.arXiv preprint arXiv:2408.03314, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    PaperBench: Evaluating AI’s ability to replicate AI research

    Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. PaperBench: Evaluating AI’s ability to replicate AI research. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedin...

    2025

  66. [66]

    Agents’ last exam.arXiv preprint arXiv:2606.05405, 2026

    Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, et al. Agents’ last exam.arXiv preprint arXiv:2606.05405, 2026

    arXiv 2026

  67. [67]

    Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. SWE-EVO: Benchmarking coding agents in long-horizon software evolution scenarios.arXiv preprint arXiv:2512.18470, 2025

    Pith/arXiv arXiv 2025

  68. [68]

    BrowseComp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Pith/arXiv arXiv 2025

  69. [69]

    Chi, et al

    Tianxin Wei, Noveen Sachdeva, Benjamin Coleman, Zhankui He, Yuanchen Bei, Xuying Ning, Mengting Ai, Yunzhe Li, Jingrui He, Ed H. Chi, et al. Evo-Memory: Benchmarking LLM agent test-time learning with self-evolving memory.arXiv preprint arXiv:2511.20857, 2025

    Pith/arXiv arXiv 2025

  70. [70]

    A statistical distribution function of wide applicability.Journal of Applied Mechanics, 18(3): 293–297, 1951

    Waloddi Weibull. A statistical distribution function of wide applicability.Journal of Applied Mechanics, 18(3): 293–297, 1951

    1951

  71. [71]

    RE-bench: Evaluating frontier AI R&D capabilities of language model agents against human experts

    Hjalmar Wijk, Tao Lin, Joel Becker, Sami Jawhar, Neev Parikh, Thomas Broadley, Lawrence Chan, Michael Chen, Josh Clymer, Jai Dhyani, et al. RE-bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. InProceedings of the 42nd International Conference on Machine Learning, 2025

    2025

  72. [72]

    Sigmoid function.https://en.wikipedia.org/wiki/Sigmoid_function, 2025

    Wikipedia contributors. Sigmoid function.https://en.wikipedia.org/wiki/Sigmoid_function, 2025

    2025

  73. [73]

    StreamBench: Towards benchmarking continuous improvement of language agents

    Cheng-Kuang Wu, Zhi Rui Tam, Chieh-Yen Lin, Yun-Nung Chen, and Hung yi Lee. StreamBench: Towards benchmarking continuous improvement of language agents. InAdvances in Neural Information Processing Systems, 2024

    2024

  74. [74]

    Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models.arXiv preprint arXiv:2408.00724, 2024

    Pith/arXiv arXiv 2024

  75. [75]

    RoadmapBench: Evaluating long-horizon agentic software development across version upgrades.arXiv preprint arXiv:2605.15846, 2026

    Xinbo Xu, Ruihan Yang, Haiyang Shen, et al. RoadmapBench: Evaluating long-horizon agentic software development across version upgrades.arXiv preprint arXiv:2605.15846, 2026

    Pith/arXiv arXiv 2026

  76. [76]

    AutoLab: Can frontier models solve long-horizon auto research and engineering tasks?arXiv preprint arXiv:2606.05080, 2026

    Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, et al. AutoLab: Can frontier models solve long-horizon auto research and engineering tasks?arXiv preprint arXiv:2606.05080, 2026. 22

    Pith/arXiv arXiv 2026

  77. [77]

    ProgramBench: Can language models rebuild programs from scratch?arXiv preprint arXiv:2605.03546, 2026

    John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, and Ofir Press. ProgramBench: Can language models rebuild programs from scratch?arXiv preprint arXiv:2605.03546, 2026

    Pith/arXiv arXiv 2026

  78. [78]

    Christine Ye, Sihan Yuan, Suchetha Cooray, Steven Dillmann, Ian L. V. Roque, Dalya Baron, Philipp Frank, Sergio Martin-Alvarez, Nolan Koblischke, Frank J. Qu, et al. ReplicationBench: Can AI agents replicate astrophysics research papers?arXiv preprint arXiv:2510.24591, 2025

    arXiv 2025

  79. [79]

    GLM-5.1: Towards long-horizon tasks.https://z.ai/blog/glm-5.1, 2026

    Z.ai. GLM-5.1: Towards long-horizon tasks.https://z.ai/blog/glm-5.1, 2026. Model release blog and model card

    2026

  80. [80]

    Prescriptive scaling reveals the evolution of language model capabilities.arXiv preprint arXiv:2602.15327, 2026

    Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, and Sham Kakade. Prescriptive scaling reveals the evolution of language model capabilities.arXiv preprint arXiv:2602.15327, 2026

    Pith/arXiv arXiv 2026

Showing first 80 references.