Pith. sign in

REVIEW 2 major objections 5 minor 15 references

Frontier AI has rapidly crossed human expert baselines on bounded tasks, yet the frontier stays jagged and the human role must shift from production to specification, verification, and oversight.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 07:31 UTC pith:LHKAGFFX

load-bearing objection Careful, well-scoped synthesis of rapid expert-baseline crossings on a jagged frontier; no new data, but the caveats are load-bearing and the role-redesign claim is honestly limited to experiments plus early adopters. the 2 major comments →

arxiv 2607.12125 v1 pith:LHKAGFFX submitted 2026-07-13 cs.HC

Faster AI, Uneven Frontier: Rapid Crossings, a Jagged Frontier, and the Repositioning of Human Judgment

Ancuta Margondai , Julie Rader , Emma Rader , Sara Willox , Mustapha Mouloua This is my paper
classification cs.HC
keywords artificial intelligenceexpert baselinesextended cognitioncognitive offloadinghuman-AI complementarityhuman oversightjagged frontier
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Between 2023 and 2026, frontier AI systems crossed documented human expert baselines on a growing set of bounded, well-specified cognitive tasks, while the length of tasks they can finish at 50% reliability roughly doubled every seven months. The paper argues these crossings are real and fast but uneven: humans retain decisive advantages in long-horizon reliability, genuine novelty, knowing their own limits, sample-efficient learning, and embodied action, and benchmark scores overstate deployable skill. People already use these systems as cognitive extensions; early signals of skill costs from offloading exist but sit against contrary evidence from earlier tools, so whether generative AI is different remains open. Experiments show that simply combining humans with a stronger AI often underperforms the AI alone, which means the surviving human contribution is judgment—specifying goals, verifying output, and providing oversight—rather than throughput. That redesign is visible in experiments but barely visible yet in labor-market data, so the paper calls for rebuilding evaluation, workflows, training, and the person in the loop instead of defending the old division of labor.

Core claim

Frontier AI has rapidly and broadly crossed documented human expert baselines on bounded, well-specified, evaluable cognitive tasks, while the capability frontier remains jagged, with lasting human advantages in long-horizon reliability, genuine novelty, calibrated self-knowledge, sample-efficient learning, and embodiment. Because naive human-plus-AI combinations often underperform the stronger partner, the human contribution must be redesigned toward specification, verification, and oversight rather than defended as production.

What carries the argument

The jagged frontier: the documented pattern that expert baselines fall quickly on bounded evaluable tasks while humans keep real advantages on long-horizon coherence, novelty, self-knowledge, learning efficiency, and embodiment, which, together with the complementarity result that unstructured human-AI teams often destroy value when the AI is stronger, forces the human role to be redesigned rather than preserved as throughput.

Load-bearing premise

That pre-frontier collaboration experiments and early, contested offloading findings will keep describing how people work with current and future generative systems as those systems improve and spread.

What would settle it

A multi-year field study in which unstructured human-plus-frontier-AI teams reliably beat the AI alone on economically realistic tasks, or controlled longitudinal evidence that generative-AI use leaves unaided verification skill intact relative to matched non-users.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation should measure reliability curves over task length and favor audited, novel-by-construction tests rather than static, contamination-prone benchmarks.
  • Individuals should use AI freely inside its demonstrated frontier while scheduling unassisted practice so verification skill does not atrophy.
  • Organizations should put humans on specification, exception handling, and accountability, not unstructured review of stronger systems, and audit rubber-stamping by seeding known errors.
  • Education should teach verification competence explicitly and separate assessment of the unaided human from assessment of the coupled system.
  • Policy should require measurable oversight performance (detection and intervention rates, maintained competence) rather than mere presence of a human in the loop.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If junior tasks are automated first and verification skill is erodible, the pipeline that produces senior judgment will thin unless deliberate practice is rebuilt.
  • The mismatch between classical automatic endorsement and the need to distrust overconfident models predicts systematic over- or under-reliance unless interfaces adapt trust calibration to the person.
  • Aggregate labor-market nulls can persist for years while occupation-internal task bundles restructure, so waiting for wage data will miss the cheap redesign window.
  • The same jagged advantages that leave humans ahead on novelty and long-horizon reliability also bound how far automated oversight can scale without reintroducing human judgment bottlenecks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper advances a three-part position synthesis: (1) between 2023 and 2026 frontier AI systems rapidly crossed documented human expert baselines on bounded, well-specified, evaluable cognitive tasks (GPQA, IMO, SWE-bench, structured diagnosis), with METR-style task-length horizons doubling roughly every seven months at 50% reliability; (2) the capability frontier remains jagged, with lasting human advantages in long-horizon reliability, genuine novelty (ARC-AGI-2, GAIA), calibration, sample-efficient learning, and embodiment, while benchmarks overstate deployed capability for documented reasons (contamination, construct validity, vendor self-evaluation, reliability gap); (3) humans increasingly treat these systems as cognitive extensions, early offloading/deskilling signals are mixed and contested, and experimental human-AI collaboration (Vaccaro meta-analysis and frontier clinical studies) shows naive combination often underperforms the stronger partner, so the human role must be redesigned toward specification, verification, and oversight rather than defended as throughput. The manuscript carefully scopes vendor results, reliability horizons (Ord), pre-frontier meta-analyses, and open empirical questions, then draws theoretical and practical implications for evaluation, workflows, education, and policy.

Significance. If the synthesis holds, it supplies a usable, evidence-weighted frame for HCI, cognitive science, and labor economics that avoids both 'gap is widening' overclaim and pure reassurance. Strengths include explicit discounting of producer-reported crossings, integration of reliability-curve analysis, balanced presentation of deskilling contestation (Budzyń et al. vs. Ahmad and secondary outcomes), and a clear distinction between constitutive and adaptive extension that generates a testable prediction about composition-offloading. The practical recommendations (scheduled unassisted practice, measurable oversight metrics rather than staffing mandates, deliberate junior pipelines) are actionable and grounded in the cited experimental record. The paper is a high-quality position synthesis rather than a new measurement contribution; its value lies in disciplined scoping and in making the role-redesign claim falsifiable against field labor-market data.

major comments (2)
  1. Section 5 and Figure 2: the Vaccaro et al. (2024) meta-analysis (Hedges g = −0.23 vs. best partner; moderator g = +0.46 / −0.54 by relative capability) is load-bearing for the claim that naive combination often underperforms the stronger partner and therefore that the human role must be redesigned. The manuscript correctly flags that the synthesis pools mostly pre-frontier classifier systems with I² = 97.7%. Because the redesign recommendation is the paper's practical payoff, the authors should either (a) add a short sensitivity subsection that isolates the subset of studies closest to generative/LLM settings (or the Goh et al. 2024 and Dell'Acqua et al. 2023 frontier-adjacent results) and shows the same qualitative pattern, or (b) further qualify the recommendation as provisional on pre-frontier evidence and state what would falsify it. Without that, the leap from the meta-analysis to c
  2. Sections 4.2–4.3 and 7.1: the constitutive vs. adaptive extension distinction is presented as a needed refinement of Clark & Chalmers and is used to predict that generative AI may break the historical 'technological reserve' pattern. The distinction is conceptually useful, but the manuscript currently treats it as both a theoretical implication and an explanatory mechanism for the mixed offloading evidence. Either supply a sharper operational criterion (what behavioral or neural signature would distinguish adaptive from constitutive extension in a longitudinal design) or demote the distinction to a framing hypothesis rather than a load-bearing theoretical claim. As written, the prediction is interesting but not yet tightly enough specified to guide the longitudinal studies the authors themselves call for.
minor comments (5)
  1. Figure 1 note: scores are correctly stated as non-comparable across rows, but the visual layout still invites cross-row comparison. Consider a small-multiples or explicitly normalized presentation, or move the figure to an appendix if space is constrained.
  2. Section 2.2: the Nori et al. (2025) NEJM diagnostic result is appropriately labeled a vendor preprint with generalist physicians denied search; a single additional sentence quantifying how much of the 80–85.5% vs. 20% gap survives when physicians are given search/colleagues would strengthen the discount.
  3. Section 6: the Danish payroll null (Humlum & Vestergaard 2025) and Acemoglu (2024) bound are well used as a reality check; a brief note on the two-year horizon relative to historical GPT diffusion timescales would help readers calibrate expectations.
  4. References: a few arXiv preprints (e.g., Kosmyna et al. 2025, Liang et al. 2025) are central to the discounting arguments; ensure final DOIs or version pins are supplied at production if available.
  5. Abstract and Section 1: the phrase 'the length of tasks such systems can complete at 50% reliability doubled roughly every seven months' is accurate but dense; a parenthetical pointer to Ord's reliability-curve qualification earlier in the abstract would further reduce misquotation risk.

Circularity Check

1 steps flagged

No significant circularity: the three-part position is a synthesis of external benchmarks and meta-analyses; one self-citation is present but not load-bearing for the central claims.

specific steps
  1. self citation load bearing [Section 5 (Complementarity Problem), paragraph on calibration interventions]
    "Interventions that improve reliance calibration are, moreover, not themselves neutral: in a three-study program combining agent-based simulation with data from 557 organizational decision-makers, transparency features improved trust calibration across the board while imposing perceived-autonomy costs that were personality-dependent, benefiting high-openness users and penalizing high-extraversion users (Margondai et al., 2026)."

    Authors cite their own concurrent HCIBGO 2026 paper for the claim that transparency aids impose personality-dependent autonomy costs. This is ordinary self-citation of related work, not a load-bearing premise: the central complementarity finding (Vaccaro meta-analysis; Goh et al. GPT-4 diagnosis trial) and the role-repositioning thesis do not depend on it. Flagged only for completeness; it does not force any of the three main claims by construction.

full rationale

This is a position/synthesis paper in cs.HC, not a derivation paper with equations or fitted parameters. Its three claims (rapid expert-baseline crossings on bounded tasks; a jagged frontier of remaining human advantages; human role repositioned toward specification/verification/oversight because naive combination often underperforms the stronger partner) rest on external sources: METR time-horizon doubling, Stanford AI Index, GPQA, IMO/AlphaProof/Gemini Deep Think, ARC-AGI-2, TheAgentCompany, Vaccaro et al. 2024 complementarity meta-analysis, Goh et al. 2024, Brynjolfsson et al., Dell'Acqua et al., Budzyń et al., Benge & Scullin, etc. The paper repeatedly scopes its own caveats (50% reliability vs economic reliability; contamination/construct validity; Vaccaro pre-frontier and I²=97.7%; generative-AI deskilling as open and contested). The only self-citation is Margondai et al. 2026 (transparency/autonomy personality effects), used as a design-parameter illustration in Section 5; it is not required for the rapid-crossings, jagged-frontier, or complementarity conclusions. No self-definitional loop, no fitted input called prediction, no uniqueness theorem imported from the authors, and no renaming of a known result as a new derivation. Score 1 for the single non-load-bearing self-citation; the derivation chain is otherwise external and self-contained against the cited record.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 1 invented entities

This is a position/synthesis paper. Load-bearing content is drawn from external empirical literature rather than free parameters or newly postulated entities. The few domain assumptions concern what counts as a valid expert-baseline crossing and how far pre-frontier collaboration results may be extrapolated; these are stated with caveats rather than hidden.

axioms (3)
  • domain assumption Documented crossings of externally graded or audited expert baselines on bounded tasks constitute evidence of rapid capability progress even after discounting contamination and construct-validity problems.
    Invoked throughout Section 2; the paper applies qualitative discounts but treats the direction and speed of crossings as robust.
  • domain assumption The METR 50%-reliability task-length doubling rate is an informative (if incomplete) quantitative measure of progress on software/research-engineering tasks.
    Section 2.1; qualified by Ord’s reliability-horizon analysis yet still used as the headline rate.
  • domain assumption Pre-frontier human-AI complementarity experiments and early generative-AI offloading studies supply directional evidence relevant to current frontier systems.
    Sections 4–5; the paper itself flags the scope limitation and treats generalization as provisional.
invented entities (1)
  • constitutive vs. adaptive extension distinction no independent evidence
    purpose: Refine the extended-mind framework so that capable, non-automatically-endorsable AI systems can be analyzed without forcing classical trust-and-glue conditions.
    Section 4.1; builds directly on Hernández-Orallo & Vold (2019) and Smart et al. (2025) rather than introducing a wholly new ontological object. Independent evidence is conceptual, not empirical.

pith-pipeline@v1.1.0-grok45 · 20205 in / 2589 out tokens · 34890 ms · 2026-07-15T07:31:39.095377+00:00 · methodology

0 comments
read the original abstract

Between 2023 and 2026, frontier AI systems crossed documented human expert baselines on a growing set of bounded, well-specified, evaluable cognitive tasks, including graduate-level science questions, competition mathematics, software-engineering benchmarks, and structured diagnostic reasoning, while the length of tasks such systems can complete at 50% reliability doubled roughly every seven months. These crossings are rapid and broad, but the frontier is jagged: humans retain decisive advantages in long-horizon reliability, genuinely novel problems, calibrated self-knowledge, sample-efficient learning, and embodied action, and benchmark results overstate deployed capability for reasons that are themselves now documented, namely contamination, construct validity, vendor self-evaluation, and the gap between 50% reliability and the reliability that economic work requires. Concurrently, humans increasingly use these systems as cognitive extensions. The offloading literature predicts costs to unaided skill, and early field evidence is consistent with such costs, though the largest meta-analytic evidence on prior technologies points the other way, and the question of whether generative AI differs is open. Finally, the experimental record on human-AI collaboration shows that naive combination often underperforms the stronger partner, implying that the human contribution must be repositioned toward specification, verification, and oversight, a shift visible in experiments but, so far, barely visible in field labor-market data. This paper states the resulting position, rapid crossings on a jagged frontier with a human role that must be redesigned rather than defended, and draws out its theoretical and practical implications.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages

  1. [1]

    Acemoglu, D. (2024). The simple macroeconomics of AI. NBER Working Paper 32487 / Economic Policy. https://www.nber.org/papers/w32487 Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023). Combining human expertise with artificial intelligence: Experimental evidence from radiology. NBER Working Paper 31422. https://www.nber.org/papers/w31422 Agrawal,...

    Pith/arXiv arXiv doi:10.1016/s2468-1253(25)00164-5 2024

  2. [2]

    Casner, S

    Nature Human Behaviour, 2, 637–644. Casner, S. M., Geven, R. W., Recker, M. P., & Schooler, J. W. (2014). The retention of manual flying skills in the automated cockpit. Human Factors, 56(8), 1506–1516. FASTER AI, UNEVEN FRONTIER 25 Chollet, F., Knoop, M., Kamradt, G., Landers, B., & Pinkard, H. (2025). ARC-AGI-2: A new challenge for frontier AI reasoning...

    Pith/arXiv arXiv 2014

  3. [3]

    https://doi.org/10.1038/s41598-020-62877-0 Dell'Acqua, F. (2022). Falling asleep at the wheel: Human/AI collaboration in a field experiment on HR recruiters [Working paper]. Harvard Business School. Dell'Acqua, F., McFowland, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. (2023). Navigating the jag...

    doi:10.1038/s41598-020-62877-0 2022

  4. [4]

    https://arxiv.org/abs/2406.04127 Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1),

    Pith/arXiv arXiv 2025

  5. [5]

    https://doi.org/10.3390/soc15010006 (see also Correction, Societies, 15(9),

    doi:10.3390/soc15010006

  6. [6]

    GitClear. (2025). AI Copilot code quality: 2025 research. https://www.gitclear.com/ai_assistant_code_quality_2025_research Goh, E., et al. (2024). Large language model influence on diagnostic reasoning: A randomized clinical trial. JAMA Network Open, 7(10), e2440969. https://doi.org/10.1001/jamanetworkopen.2024.40969 Google DeepMind. (2025, July 21). Adva...

    doi:10.1001/jamanetworkopen.2024.40969 2025

  7. [7]

    Hernández-Orallo, J. (2025). Enhancement and assessment in the AI age: An extended mind perspective. Journal of Pacific Rim Psychology. https://doi.org/10.1177/18344909241309376 Hernández-Orallo, J., & Vold, K. (2019). AI extenders: The ethical and societal implications of humans cognitively extended by AI. Proceedings of the AAAI/ACM Conference on AI, Et...

    Pith/arXiv arXiv doi:10.1177/18344909241309376 2025

  8. [8]

    https://arxiv.org/abs/2407.04622 Khan, A., Hughes, J., Valentine, D., et al. (2024). Debating with more persuasive LLMs leads to more truthful answers. ICML

    Pith/arXiv arXiv 2024

  9. [9]

    Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task

    https://arxiv.org/abs/2402.06782 Kohrs, F., et al. (2026). Comment on "Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task." arXiv. https://arxiv.org/abs/2601.00856 FASTER AI, UNEVEN FRONTIER 28 Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X.-H., Beresnitzky, A. V., Braunstein, I., & Maes, P....

    Pith/arXiv arXiv 2026

  10. [10]

    The illusion of thinking

    https://arxiv.org/abs/2503.14499 Ladabaum, U., et al. (2023). Computer-aided detection of polyps does not improve colonoscopist performance in a pragmatic implementation trial. Gastroenterology, 164(3), 481–483. Lawsen, A. (2025). Comment on "The illusion of thinking." arXiv. https://arxiv.org/abs/2506.09250 Lee, H.-P., Sarkar, A., Tankelevitch, L., Droso...

    Pith/arXiv arXiv doi:10.1145/3706598.3713778 2023

  11. [11]

    https://arxiv.org/abs/2506.21521 Margondai, A., Willox, S., Acevedo Diaz, A., Rader, J., Hani, S., Ezcurra, V., & Mouloua, M. (2026). The transparency dilemma: Reconciling trust calibration with perceived autonomy through personality-adaptive AI systems. In F. F.-H. Nah & K. L. Siau (Eds.), FASTER AI, UNEVEN FRONTIER 29 HCI in Business, Government and Org...

    Pith/arXiv arXiv doi:10.1007/978-3-032-30044-7_7 2026

  12. [12]

    https://doi.org/10.3389/frobt.2018.00015 Shi, H., Xu, Z., Wang, H., et al. (2024). Continual learning of large language models: A comprehensive survey. ACM Computing Surveys. https://arxiv.org/abs/2404.16789 Shojaee, P., Mirzadeh, I., et al. (2025). The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of p...

    doi:10.3389/frobt.2018.00015 2018

  13. [13]

    https://arxiv.org/abs/2404.07972 Xiong, M., Hu, Z., Lu, X., et al. (2024). Can LLMs express their uncertainty? An empirical evaluation of confidence elicitation in LLMs. ICLR

    Pith/arXiv arXiv 2024

  14. [14]

    F., Song, Y., Li, B., et al

    https://arxiv.org/abs/2306.13063 Xu, F. F., Song, Y., Li, B., et al. (2024). TheAgentCompany: Benchmarking LLM agents on consequential real world tasks. arXiv. https://arxiv.org/abs/2412.14161 Yu, F., Moehring, A., Banerjee, O., Salz, T., Agarwal, N., & Rajpurkar, P. (2024). Heterogeneity and predictors of the effects of AI assistance on radiologists. Nat...

    Pith/arXiv arXiv 2024

  15. [15]

    https://doi.org/10.1038/s41591-024-02850-w

    work page doi:10.1038/s41591-024-02850-w