Pith. sign in

REVIEW 3 major objections 4 minor 22 references

This paper proves the nonuniformity principle for human-AI coworking: with a fixed number of human reviews in a long AI task, the optimal schedule puts checks early and keeps the gaps between checks non-decreasing.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:41 UTC pith:GHGBFKEA

load-bearing objection A clean scheduling theorem with a genuinely missing assumption; the empirical support is thinner than the theory. the 3 major comments →

arxiv 2607.16530 v1 pith:GHGBFKEA submitted 2026-07-17 cs.AI stat.ME

Nonuniformity Principle in Human-AI Coworking

An Luo , Jie Ding This is my paper
classification cs.AI stat.ME MSC 90B35
keywords human-AI coworkingoversight schedulingnonuniformity principleagentic AIscalable oversightalignment qualitydiscrete convex optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Human oversight in multi-step AI work is scarce, and this paper asks where to spend it. It claims that for a fixed number of check-ins across a long task, the optimal schedule puts oversight early and then lets the gaps between checks grow: the gap sequence is non-decreasing. The claim is proven as a theorem in a model where the AI's uncertainty about the user's true intent grows since the last check, and where reviewing a longer draft costs more. The authors also give an exact algorithm for the case of linearly growing uncertainty and cost, and they report two agent-task experiments — writing a related-work section and constructing an HTML landing page — whose quality-cost winners both follow the non-decreasing-gap rule. If the principle holds generally, it gives concrete, testable guidance for where to place human checkpoints in agentic AI workflows.

Core claim

The paper establishes the nonuniformity principle: every schedule that minimizes the sum of expected alignment error and human oversight cost must place oversight stages with non-decreasing gaps (Theorem 1). Concretely, if d⋆j is the number of production stages between oversight j and oversight j+1, then d⋆0 ≤ d⋆1 ≤ ... ≤ d⋆K−1. This holds under three assumptions: the expected squared error at any point between two checks is a fixed fraction κ of the error that would accumulate without a check; that un-checked error ρ(r) strictly increases with the lag r; and the cost of reviewing at stage s strictly increases with s. The same machinery yields a second result: if review cost grows fast enoug

What carries the argument

The key device is the reduction of the scheduling problem to a separable discrete convex program in the gap variables. The total loss L(S) splits into a sum of per-gap functions Φ(dj) for gaps ending in a check, a terminal function Ψ(dK), and oversight costs c(sk) that grow with the stage index. Because Φ and Ψ are strictly increasing and strictly discrete convex (Lemma 2), and because the cost term makes later checks cheaper per unit of gap, shifting a check earlier in the schedule strictly lowers the objective — yielding the non-decreasing gap ordering of Theorem 1. Algorithm 1 is the exact minimizer for the random-walk/linear-cost special case.

Load-bearing premise

The theorem rests on Assumption 3: after any human check, the expected error at every later stage until the next check is exactly κ times the no-check error, with the same κ for every check regardless of when it happens; if oversight helps more or less at different stages, the loss no longer separates and the non-decreasing-gap conclusion can fail.

What would settle it

Measure the actual error-reduction factor of oversight at different stages of a real long-horizon agent task (e.g., related-work writing) by comparing drafts with and without a check at each stage. If the reduction factor varies with stage or with the schedule, recompute the optimal schedule by brute force over all possible checkpoints; finding an optimal schedule whose gaps decrease would falsify the nonuniformity principle. A simpler simulation-level falsifier: modify the paper's own random-walk model so κ depends on the oversight index, and search for a counterexample to monotone gaps.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For any long-horizon agent with a fixed human-review budget, reviews should be clustered at the start and spaced ever more widely later.
  • If review burden grows steeply enough, the theory predicts a sharp regime: all reviews collapse to the earliest stages.
  • The provided algorithm turns the principle into a plug-in schedule once the user estimates the uncertainty growth rate and the review-cost ratio.
  • The two experimental Pareto frontiers are dominated by schedules whose gaps increase, matching the theorem's prediction.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The constant-κ assumption is doing heavy lifting; if oversight benefit decays with stage, the non-decreasing-gap conclusion could reverse, and the experiments do not measure κ to rule this out.
  • The same separable-convex formulation applies to other scarce-checkpoint placement problems, such as human sign-offs in scientific discovery pipelines or medical data workflows, whenever uncertainty and review load both grow.
  • An adaptive variant that also chooses the number of checks K would be a natural next step and could change how aggressively oversight is front-loaded.
  • A direct test of the principle in a regime with heterogeneous oversight quality — e.g., one very good late reviewer — would clarify its limits.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper formalizes the problem of placing a fixed number K of human oversight stages within a T-stage AI agent workflow. It models a latent requirement process Z_t generated from a hidden human specification, an agent producing drafts via a Bayes decision rule, and a revision operator at each oversight stage. Under assumptions that the no-oversight error at lag r depends only on r (Assumption 2) and that the final error of every reviewed component is a constant fraction κ of the no-oversight error (Assumption 3), the expected alignment loss decomposes into a sum of per-gap convex functions. The main result, Theorem 1, states that any minimizer of the resulting objective (9) has non-decreasing gaps between consecutive oversight stages. The paper also proposes Algorithm 1, an exact greedy solver for the random-walk/linear-cost special case, and reports two empirical studies (writing related-work sections and constructing HTML pages) in which schedules with non-decreasing gaps lie on the quality-cost Pareto hull.

Significance. If the derivation is accepted, the nonuniformity principle is a crisp, actionable design rule: place human oversight densely early and let gaps between oversight stages grow as the workflow progresses. The paper contributes a clean mathematical formulation of oversight scheduling, a nontrivial theorem with a valid exchange argument under the stated assumptions, an exact algorithm with a proof of global optimality (Proposition 2), and a reproducible experimental pipeline using LLM agents. These are genuine strengths: the reduction to a separable convex program is elegant, and the empirical demonstrations are suggestive. However, the central claim is conditional on Assumption 3, a constant-revision-factor premise that is neither derived from the Bayes-rule model nor measured in the experiments. The empirical validation also uses small samples, overlapping standard errors, and parameter values selected after observing the outcomes. The paper is therefore best viewed as a promising theoretical framework with preliminary empirical support, not as an established principle.

major comments (3)
  1. [§3.1–3.2, Theorem 1] Theorem 1 is stated under Assumptions 4 and 5, but its proof requires Assumption 3 (the constant κ). The decomposition L(S)=Σ_{j=0}^{K-1}Φ(d_j)+Ψ(d_K) leading to objective (9) is valid only if, for every s_k<t≤s_{k+1}, E‖w̃_t(S)−Z_t‖²=κρ(t−s_k). Yet by Eq. (3), w̃_t(S) for t≤s_K is the component after the final revision at s_K, and it is revised at every later oversight stage s_{k+1},...,s_K. Its squared error should depend on the number and placement of those later stages; Assumption 3 asserts that dependence is nil. This is not derived from the Bayes rule (2) or from the revision operator R_s, and it is not measured in Section 4. Without Assumption 3, the separable objective (9), Algorithm 1, and Theorem 1 are not established. This is load-bearing: the non-decreasing-gap conclusion is the paper's central claim. The theorem's advertised conditions omit a hidden structural premise.
  2. [§4.4, Eqs. (14)–(15)] The validation of Algorithm 1 is post-hoc. The parameter pairs (κ=0.2, η=0.4) for Spread and (κ=0.7, η=1.3) for Tilt-Early are chosen because they make Algorithm 1 return the schedules already observed to win in Studies 1 and 2. Since κ and η are free parameters that are not estimated from the experimental data or measured independently (e.g., by probing the agent's error with and without oversight), the match in (14)–(15) is a curve-fit rather than a predictive test. To support the claim of validating Proposition 2, the authors should set κ and η before the schedules are run, or at minimum report the full region of (κ,η) that yields each schedule together with an uncertainty quantification for the observed point.
  3. [§4.2–4.3, Tables 2 and 3] The empirical evidence for the nonuniformity principle is statistically thin. In Study 1 (N=40), the best-quality schedule Spread (5.06±0.18) is within one standard error of Uniform (5.01±0.13) and Burst-Late (5.05±0.12); no pairwise test or effect size is reported. In Study 2 (N=10), Tilt-Early (7.74±0.21) and Uniform (7.56±0.20) again overlap. The Pareto-hull analysis treats point estimates as exact, but with these standard errors the ordering could easily change. Moreover, the quality scores come from LLM judges, not human raters, and the cost proxies are schedule-dependent constructs whose relation to the theoretical c(s) is not formally established. The statement in §4.4 that 'the experimental results align with the nonuniformity principle' is stronger than the data support; at best the results are consistent with the principle, with large uncertainty.
minor comments (4)
  1. [§3.2, after Theorem 1] Theorem 1 does not constrain d_K, the terminal gap. The empirical schedules in §4.4 satisfy d0≤d1≤d2, but the discussion implicitly restricts the principle to gaps ending at an oversight stage; this should be stated explicitly in the interpretation of the empirical results.
  2. [Table 3 caption] The caption says 'Bold denotes the highest quality scores,' but no entries in Table 3 are bold. Either add the bold formatting or remove that phrase.
  3. [Figure 4 caption] The caption uses 'intercept-removed' cost without defining the intercept or the removal procedure. The main text says cost is total draft length; clarify how the intercept was computed and why it is removed.
  4. [§3.1, Assumption 3 paragraph] The sentence 'By Lemma 1 and Assumption 2, we have E‖w⁻_{s+r}−Z_{s+r}‖²=ρ_s(r)=ρ(r)' uses Assumption 2 but the notation ρ_s(r) is defined as an expectation of a conditional variance; it may be clearer to write ρ_s(r)=E[Var(Z_{s+r}|H_s)] explicitly before the chain of equalities.

Circularity Check

1 steps flagged

Theorem 1 is a genuine derivation from stated assumptions, but §4.4 'validates' Algorithm 1 by choosing κ and η after seeing the empirical winners so the algorithm reproduces them — a fit by construction, not a prediction.

specific steps
  1. fitted input called prediction [Section 4.4 (validation of Algorithm 1), equations (14)–(15)]
    "We also validate Algorithm 1 and Proposition 2 by deriving that under certain values of κ and η, Algorithm 1 will return Sspread = (1, 4, 9) or Stilt = (1, 3, 6) as a minimizer. ... For example, κ = 0.2 and η = 0.4 satisfy (14), and Algorithm 1 returns d̂ = (1,3,5,1) and Ŝ = (1,4,9). ... For example, κ = 0.7 and η = 1.3 satisfy (15), and Algorithm 1 returns d̂ = (1, 2, 3, 4), Ŝ = (1, 3, 6)."

    The schedules Algorithm 1 is made to return — Spread=(1,4,9) in Study 1 and Tilt-Early=(1,3,6) in Study 2 — are exactly the schedules already identified as the empirical winners in Tables 2 and 3. The parameters κ and η are not estimated from data or fixed a priori; they are chosen (κ=0.2, η=0.4; κ=0.7, η=1.3) specifically to satisfy the derived regions (14)–(15), i.e., so that the algorithm's minimizer equals the observed winner. Since for any schedule in the feasible region a parameter region can be exhibited that makes it the minimizer, the 'validation' reduces to a construction: it confirms the model can reproduce the data it was shown, not that it predicts it. The match is therefore forced by parameter selection rather than by the theory.

full rationale

The central theoretical claim (Theorem 1) is not circular: it is a genuine discrete-convexity argument. Under Assumptions 1–3 the loss L(S) is reduced to the separable objective (9), and Lemma 2 plus Assumption 5 (strictly increasing oversight cost) imply the non-decreasing-gap conclusion for minimizers of (9). This conclusion does not appear in any assumption and is a real mathematical derivation, and the Pareto-hull observation in §4.2–4.3 (the empirically optimal schedules have non-decreasing gaps) is an independent, theory-free check. The genuinely circular element is confined to §4.4, where the specific schedules produced by Algorithm 1 are reproduced by selecting κ and η after the fact from the data; that part is fitted, not predicted, and supports only a partial-circularity finding. Two other concerns are noted but do not constitute circularity under the review rules. First, Theorem 1 is stated 'under Assumptions 4 and 5' even though the reduction to (9) requires the asserted Assumption 3 (constant κ error fraction depending only on lag t−s_k), which is never derived from the Bayes rule (2) or measured; this is an unverified structural premise and an omitted hypothesis in the theorem statement, a rigor gap rather than an equation-level reduction. Second, the self-citations (Luo et al. 2025a,b, 2026) are contextual and not load-bearing for the nonuniformity principle. Overall: the main derivation has independent mathematical content and the qualitative empirical finding is authentic, but one supporting 'prediction' reduces by construction, so a partial-circularity score of 5 is appropriate.

Axiom & Free-Parameter Ledger

3 free parameters · 7 axioms · 1 invented entities

The central theorem is derived from five explicit assumptions (Assumptions 1-5). The most fragile is Assumption 3, which postulates a constant fraction κ of error remaining after each oversight; this is not derived from the agent's Bayes decision rule and is not measured in the experiments. Assumptions 1, 2, and 4 constrain the latent requirement process so that error accumulates in a stationary, convex-in-lag way, and Assumption 5 (increasing cost) is what pushes optimal oversights earlier. For the practical algorithm, the model is specialized to a random walk with linear cost, whose variance σ² is absorbed into the fitted ratio η = λ/σ²; both κ and η are selected post hoc in Section 4.4 to reproduce the empirically preferred schedules.

free parameters (3)
  • κ (revision factor) = 0.2 and 0.7 in §4.4
    Chosen post hoc so Algorithm 1 returns the empirically winning schedules S_spread=(1,4,9) and S_tilt=(1,3,6); no independent estimate of oversight effectiveness.
  • η = λ/σ² (cost-uncertainty ratio) = 0.4 and 1.3 in §4.4
    Same post hoc choice as κ; η is a free ratio that determines Algorithm 1's output and is not estimated from the experiments before validation.
  • λ (cost weight in empirical loss L_λ) = thresholds ≈3.3e-4 (Study 1) and ≈0.38 (Study 2)
    Derived from the empirical Pareto frontier to describe which schedule wins; used as an analysis parameter, not an input to the theory.
axioms (7)
  • domain assumption Assumption 1: Z_{s+r} is conditionally independent of recent drafts given H_s
    Decouples the latent requirement process from the agent's intermediate outputs; needed for Lemma 1.
  • domain assumption Assumption 2: ρ_s(r)=ρ(r) for every oversight stage s
    Stationarity of predictive variance across oversight positions; holds for random-walk and AR(1) examples given in Remark 1.
  • ad hoc to paper Assumption 3: E[||˜w_t(S)-Z_t||^2] = κρ(t-s_k) with constant κ∈(0,1)
    Load-bearing and unmeasured: asserts oversight reduces alignment error by a constant multiplicative factor at every stage and for every schedule; converts the objective into a separable convex program.
  • domain assumption Assumption 4: ρ(r) strictly increasing in r
    Misalignment grows with time since last oversight; used for discrete convexity of Φ and Ψ (Lemma 2).
  • domain assumption Assumption 5: oversight cost c(s) strictly increasing in s
    Later reviews are more costly; this is the premise that pushes oversight earlier and yields non-decreasing gaps.
  • domain assumption Bayes decision rule (2): w^-_t minimizes posterior expected loss
    Assumes the agent is a Bayesian optimizer; under squared loss the draft equals the posterior mean, so Lemma 1's variance interpretation holds.
  • domain assumption Random-walk model ρ(r)=σ²r and linear cost c(s)=λs (§3.3)
    Specialization used by Algorithm 1 and the validation in §4.4; σ² is absorbed into the fitted ratio η.
invented entities (1)
  • Latent requirement process Z_t and specification θ no independent evidence
    purpose: Formalize the human's hidden intent that the AI cannot observe and define the alignment loss E||˜w_t - Z_t||².
    Unobservable by construction; the experiments never measure Z_t directly, using an LLM judge's 1-10 score as a proxy. It has no falsifiable handle outside the model.

pith-pipeline@v1.3.0-alltime-deepseek · 16850 in / 21369 out tokens · 205321 ms · 2026-08-01T20:41:56.510999+00:00 · methodology

0 comments
read the original abstract

As generative AI is increasingly applied to automate multi-step and high-stake workflows, human judgment and involvement remain essential for ensuring the quality of AI-generated outputs. In practice, while it is desirable for human experts to provide oversight on AI regularly, often by reviewing intermediate outputs, giving feedback, making corrections, and steering subsequent steps, such oversight is constrained by the time and resources that humans can afford. This creates a tension between the need for human oversight and AI's efficiency in delivering more output with less intervention. An important but underexplored question, then, is how to optimally engage humans in human-AI coworking. This work was originally motivated by our empirical observation that in long AI workflows, human oversight often improves user satisfaction while reducing unnecessary rework and token consumption. From there, we formulate the problem of where to place oversight stages in human-AI coworking. Under reasonable assumptions, we then develop the nonuniformity principle, which states that the optimal schedule places oversight stages with non-decreasing gaps along the workflow. We empirically validate this principle in two common AI agent workflows: writing literature reviews and constructing websites.

Figures

Figures reproduced from arXiv: 2607.16530 by An Luo, Jie Ding.

Figure 1
Figure 1. Figure 1: Illustration of the nonuniformity principle. (a) Different oversight schedules with [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of main concepts in human-AI coworking. The AI agent constructs a [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison of one Study 1 example, showing the final related work [PITH_FULL_IMAGE:figures/full_fig_p020_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Quality-cost Pareto frontier of study 1: writing related work. Summarized on [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison of one example from Study 2, showing the final HTML [PITH_FULL_IMAGE:figures/full_fig_p024_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Quality-cost Pareto frontier of study 2: HTML landing-page construction over [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 3 canonical work pages

  1. [4]

    Benjamin DeMeo, Charlotte Nesbitt, Samuel A

    doi: 10.1038/s41586-024- 08173-7. Benjamin DeMeo, Charlotte Nesbitt, Samuel A. Miller, Daniel B. Burkhardt, et al. Active learning framework leveraging transcriptomics identifies modulators of disease phenotypes. Science, 390:eadi8577,

    doi:10.1038/s41586-024-

  2. [6]

    Katy Ilonka Gero, Vivian Liu, and Lydia B Chilton

    URL https://arxiv.org/abs/2312.11805. Katy Ilonka Gero, Vivian Liu, and Lydia B Chilton. Sparks: Inspiration for science writing using language models. InProceedings of the 2022 ACM Designing Interactive Systems Conference, pages 1002–1019. ACM,

    Pith/arXiv arXiv 2022

  3. [8]

    doi: 10.18653/v1/2024.acl-long.371. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta programming for a multi-agent collaborative framework. InThe Twelfth International Confere...

    doi:10.18653/v1/2024.acl-long.371 2024

  4. [9]

    Agents’ room: Narrative generation through multi- step collaboration

    Fantine Huot, Reinald Kim Amplayo, Jennimaria Palomaki, Alice Shoshana Jakobovits, Elizabeth Clark, and Mirella Lapata. Agents’ room: Narrative generation through multi- step collaboration. InProceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025),

    2025

  5. [10]

    ICLR 2026 Conference

    International Conference on Learning Representations. ICLR 2026 Conference. https: //openreview.net/group?id=ICLR.cc/2026/Conference,

    2026

  6. [12]

    Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, 30 Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A McFarland, and James Zou

    doi: 10.1126/science.adi1407. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, 30 Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A McFarland, and James Zou. Can large language models provide useful feedback on research papers? A large-scale empirical analysis.NEJM AI, 1(8),

    doi:10.1126/science.adi1407

  7. [13]

    Yi-Cheng Lin, Kang-Chieh Chen, Zhe-Yan Li, Tzu-Heng Wu, Tzu-Hsuan Wu, Kuan-Yu Chen, Hung yi Lee, and Yun-Nung Chen

    doi: 10.1056/AIoa2400196. Yi-Cheng Lin, Kang-Chieh Chen, Zhe-Yan Li, Tzu-Heng Wu, Tzu-Hsuan Wu, Kuan-Yu Chen, Hung yi Lee, and Yun-Nung Chen. Creativity in LLM-based multi-agent systems: A survey. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025),

    doi:10.1056/aioa2400196 2025

  8. [14]

    An Luo, Jin Du, Fangqiao Tian, Xun Xian, Robert Specht, Ganghua Wang, Xuan Bi, Charles Fleming, Jayanth Srinivasa, Ashish Kundu, Mingyi Hong, and Jie Ding. Can agentic AI match the performance of human data scientists? InIEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), pages 206–210, 2025a. An Luo, Xun Xi...

    Pith/arXiv arXiv 2025

  9. [15]

    31 OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al

    doi: 10.1162/99608f92.1f068331. 31 OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  10. [16]

    doi: 10.48550/arXiv.2303. 08774. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to foll...

    doi:10.48550/arxiv.2303

  11. [17]

    Nathan J

    doi: 10.1038/s41598-022-18751-2. Nathan J. Szymanski, Bernardus Rendy, Yuxing Fei, Rishi E. Kumar, et al. An autonomous laboratory for the accelerated synthesis of inorganic materials.Nature, 624:86–91,

    work page doi:10.1038/s41598-022-18751-2

  12. [18]

    32 Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou

    doi: 10.1038/s41586-023-06734-w. 32 Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. A large-scale randomized study of large language model feedback in peer review.Nature Machine Intelligence, 8(3):326–336,

    doi:10.1038/s41586-023-06734-w

  13. [19]

    doi: 10.1038/s42256-026-01188-x. Fangqiao Tian, An Luo, Jin Du, Xun Xian, Robert Specht, Ganghua Wang, Xuan Bi, Jiawei Zhou, Ashish Kundu, Jayanth Srinivasa, Charles Fleming, Rui Zhang, Zirui Liu, Mingyi Hong, and Jie Ding. An outlook on the opportunities and challenges of multi-agent AI systems.arXiv preprint arXiv:2505.18397,

    Pith/arXiv arXiv

  14. [20]

    Guoyong Wang, Kaijun Zhang, Jiyue Jiang, Chaonan Wang, Hui Bi, Haojun Liang, Zuoliang Qi, Ying Huang, Yu Li, and Xiaonan Yang

    doi: 10.1038/s41562-024-02024-1. Guoyong Wang, Kaijun Zhang, Jiyue Jiang, Chaonan Wang, Hui Bi, Haojun Liang, Zuoliang Qi, Ying Huang, Yu Li, and Xiaonan Yang. Human–large language model collaboration in clinical medicine: a systematic review and meta-analysis.npj Digital Medicine, 9(1):195,

    doi:10.1038/s41562-024-02024-1

  15. [21]

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen

    doi: 10.1038/s41746-026-02382-2. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents.Frontiers of Computer Science, 18(6):186345,

    work page doi:10.1038/s41746-026-02382-2

  16. [22]

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F

    doi: 10.1007/s11704-024-40231-1. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Heng Peng, Heng Ji, and Graham Neubig. OpenH...

    doi:10.1007/s11704-024-40231-1

  17. [2019]

    Daniil A

    doi: 10.1145/3290605.3300233. Daniil A. Boiko, Robert MacKnight, Ben Kline, and Gabe Gomes. Autonomous chemical research with large language models.Nature, 624(7992):570–578,

    arXiv

  18. [2022]

    29 Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu

    doi: 10.1145/3532106.3533533. 29 Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 4852–4868,

    arXiv

  19. [2023]

    Tianning Dai, S

    doi: 10.1038/s41586- 023-06792-0. Tianning Dai, S. Vijayakrishnan, Filip T. Szczypiński, et al. Autonomous mobile robots for exploratory synthetic chemistry.Nature, 635:890–897,

    doi:10.1038/s41586-

  20. [2024]

    Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N

    doi: 10.1038/s41586-024-07487-w. Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N. Bennett, Kori Inkpen, Jaime Teevan, Ruth Kikin-Gil, and Eric Horvitz. Guidelines for human–AI interaction. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, pag...

    doi:10.1038/s41586-024-07487-w 2019

  21. [2025]

    Gemini Team Google

    doi: 10.1126/science.adi8577. Gemini Team Google. Gemini: A family of highly capable multimodal models,

    work page doi:10.1126/science.adi8577

  22. [2026]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan

    Accessed: 2026-06-26. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InThe Twelfth International Conference on Learning Representations (ICLR),

    2026