Pith. sign in

REVIEW 2 major objections 6 minor 19 references

Letting coding agents hear each other mid-task nearly doubles success on hard codebase questions.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 07:35 UTC pith:75NK2J3Y

load-bearing objection Strong systems result on a hard coding benchmark; the +10.5 passive step is real but slightly oversold as a pure single-bit isolation. the 2 major comments →

arxiv 2607.28430 v1 pith:75NK2J3Y submitted 2026-07-30 cs.MA

AgentRadio: Passive Awareness for Long-Horizon Multi-Agent Collaboration

Xinxing Ren , Qianbo Zang , Ziyan Wang , Caelum Forder , Suman Deb , Peter Carroll , Zekun Guo This is my paper
classification cs.MA
keywords multi-agent systemscoding agentspassive awarenessasynchronous messaginglong-horizon taskscodebase understandingSWE-Atlas QnAdivision of labor
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long codebase questions force an agent to build software, trace runs across files, and synthesize evidence over many minutes, and a single agent’s context wears out: one strong coding agent solves only about a third of SWE-Atlas QnA tasks. Splitting the work across agents with clean contexts helps, but the pieces still depend on one another—one finding can rewrite another agent’s job—so coordination has to happen while people are still working, not only at handoff points. Most multi-agent setups keep communication and work mutually exclusive: a discovery made mid-run waits until the next phase boundary. This paper introduces AgentRadio, a thin asynchronous messaging layer with threads, non-blocking sends, and a background “wait for mentions” so each agent stays passively aware of teammates and can fold new evidence into ongoing work. Under a fixed five-phase divide-and-negotiate protocol, four agents with that passive channel reach 62.1% task accuracy, beating both a lone agent and a newer single-agent model, with the passive step alone adding over ten points and larger gains on harder tasks.

Core claim

On long-horizon codebase QnA, four coding agents linked by AgentRadio’s passive-awareness channel resolve 62.1% of tasks versus 32.3% for one agent of the same model, and beat a stronger single-agent baseline (57.2%). The decisive isolated step is switching from blocking receive to background wait-for-mention during execution: that change alone adds about 10.5 points, with rubric gains growing as tasks get harder, which the authors read as mid-course correction rather than extra compute.

What carries the argument

AgentRadio’s three primitives—create_thread, send_message, and wait_for_mention—especially the last run as an OS-level background watcher so mentions surface between work steps without interrupting foreground commands, turning messaging into passive awareness under a five-phase explore/divide/execute/review/submit protocol.

Load-bearing premise

The jump from blocking to passive messaging is treated as isolating pure mid-run awareness, assuming the background watcher and the small Phase-3 prompt tweak do not themselves drive the gain through extra tokens, harness quirks, or different behavior beyond when agents can hear.

What would settle it

Rerun the paired L2 versus L3 ladder on the same 124 tasks with an identical Phase-3 prompt and message budget caps, and check whether the ~10-point task lift and the difficulty-stratified rubric gains disappear when passive delivery is disabled or when messages are delayed until phase boundaries.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For interdependent long-horizon coding work, structure that lets agents hear each other during execution can beat spending the same budget on more independent single-agent samples.
  • A small passive communication change can outweigh moving to a newer model generation on this benchmark.
  • Gains should concentrate on cross-cutting tasks (architecture, onboarding) where one agent’s discovery rewrites another’s subtask.
  • Systems that only hand off at phase boundaries will keep missing corrections that only appear mid-execution.
  • The same three primitives can be dropped onto existing coding harnesses without rewriting the harness, as long as background shell tasks are supported.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Passive lateral channels may matter most wherever subgoals are discovered rather than fully known at planning time—debugging, incident response, and multi-repo audits, not only QnA.
  • If message noise can pull agents off good lines of evidence, future work will need filters or priority on mentions so awareness does not become distraction.
  • The result suggests evaluating multi-agent systems by when information can arrive, not only by how many agents or how much total spend.
  • Harness vendors that expose cheap background listeners could unlock collaboration gains without new model training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper introduces AgentRadio, an asynchronous message-passing layer for coding-agent harnesses with three primitives (create_thread, send_message, wait_for_mention). The distinctive claim is that running wait_for_mention as an OS-level background task yields passive awareness: agents can receive teammates’ natural-language messages between work steps without interrupting foreground tool use, removing the mutual exclusion between communication and work. Evaluated under a fixed five-phase division-and-negotiation protocol on SWE-Atlas QnA (124 tasks, 1,306 rubrics), four Claude Code agents with AgentRadio reach 62.1% task accuracy with Opus 4.6 (vs 32.3% single-agent; vs 57.2% single Opus 4.8) and 50.8% with DeepSeek V4 Pro (vs 29.0%). A configuration ladder (single agent → division → blocking negotiation → passive awareness), cost-matched best-of-6 sampling, paired McNemar tests on the L2→L3 step, rubric waterfalls, difficulty stratification, and two case studies are used to argue that mid-course correction is the operative mechanism.

Significance. If the result holds, this is a concrete and timely systems contribution for long-horizon multi-agent coding. The paper targets a documented gap—lateral, mid-execution communication among concurrently working agents—and shows large gains on a hard external benchmark where single agents remain far from saturation. Strengths that should count in the paper’s favor include: a full paired ladder on two model families; a compute-matched best-of-6 baseline that rules out pure budget explanations; rubric-level attribution and difficulty stratification consistent with mid-course correction; qualitative cases that both illustrate the mechanism (MinIO) and bound it (Grafana); public code; and a harness-minimal design (shell scripts + background process) that does not fork the agent runtime. Even under a more conservative reading that attributes the L2→L3 lift to “live mid-P3 messaging under this protocol” rather than solely to the background watcher, the empirical finding that async lateral exchange during execution substantially improves interdependent code-comprehension work remains valuable for cs.MA and agent systems.

major comments (2)
  1. [Contributions; Communication Primitives; Algorithm 1; Experiments (L2 vs L3); Table 2] Causal isolation of L2→L3 is overstated relative to the design actually compared. Contributions and Experiments state that the passive step changes “only the communication primitives and one phase prompt” / “only the communication mode,” and the abstract/conclusion treat the background watcher as the single degree of freedom “worth more than a model generation.” But Algorithm 1 and the P3 description show a bundled contrast: under L2, live P3 sharing “disappears” and discoveries are not posted until P4; under L3, agents are instructed to post on discovery and fold incoming messages at the next step. Send is non-blocking in both modes, so the missing cell is not pure background-vs-foreground receive with identical mid-P3 norms—it is “mid-P3 lateral channel enabled and used” vs “mid-P3 channel effectively unused.” Cost also rises from $15.59 to $19.45/task (Table 2), consistent with redire
  2. [Figure 1; Table 2; Compute and Model-Generation Baselines] The claim that four Opus 4.6 agents under AgentRadio surpass single Opus 4.8 (62.1% vs 57.2%) is used as evidence of “structure over compute/model generation,” but Opus 4.8 is only reported as a single-agent public-leaderboard number (Table 2), not re-run under the authors’ harness, judge settings, and one-run protocol. Without a same-harness Opus 4.8 B0 (and ideally L2/L3), the cross-generation comparison is not paired and should be labeled as an external reference point rather than a controlled structural win. This does not undermine the within-model ladder, which is the load-bearing evidence.
minor comments (6)
  1. [Table 3; Run-to-Run Variance] Table 3’s 30-task variance study is helpful but underspecified: how the subset was sampled (stratified vs convenience), whether the same Docker/task pins were used, and whether McNemar or bootstrap CIs on the full 124-task set could be reported for L1→L2 as well as L2→L3 would strengthen reproducibility claims.
  2. [Implementation; Table 2] Implementation asserts the watcher adds “no LLM calls” and that agents pay only for surfaced messages, yet Table 2’s L2→L3 dollar gap is material. A short breakdown (tokens in surfaced messages vs tokens in extended tool traces after redirection) would make the “economical passive step” claim precise.
  3. [Related Work; Introduction patterns (a)–(c)] Related Work correctly notes asynchrony is rare, but a tighter comparison table (channel type: lateral vs planner→worker; receive: blocking/passive/shared-memory; mid-task steerability) for Anthropic’s research system, MPAS, HyperAgent, HANDRAISER, CodeCRDT, and CAID would help readers place the three primitives quickly.
  4. [Case Studies; Figures 6–7] Figure 2’s illustrative worklog message is fine, but the MinIO/Grafana case figures (Figures 6–7) would be easier to audit if they cited task IDs and rubric IDs from SWE-Atlas QnA so readers can re-grade against the public rubrics.
  5. [The Five-Phase Protocol; Conclusion] Free parameters fixed without sensitivity analysis—agent count (4), assembler gating by agent-1, and wait_for_mention timeout/step-boundary semantics—should be listed explicitly as limitations. Even a small 2- vs 4-agent or timeout ablation on the 30-task subset would reduce the concern that results are protocol-specific.
  6. [Communication Primitives] Minor clarity: define M(t) and step boundaries more formally when contrasting blocking vs passive receive (the paragraph is good but one displayed definition of “step boundary” relative to harness tool calls would prevent ambiguity about when mentions surface).

Circularity Check

0 steps flagged

No circularity: empirical systems result on an external benchmark, not a derivation that reduces to its inputs.

full rationale

AgentRadio is an empirical multi-agent systems paper. Its load-bearing claims are measured task/rubric accuracies on SWE-Atlas QnA (124 held-out expert tasks, 1,306 rubrics, fixed external LLM judge), not closed-form predictions derived from fitted parameters or self-defined quantities. The B0→L1→L2→L3 ladder reports observed lifts under controlled harness/model settings; L3’s 62.1% and the L2→L3 +10.5 pt McNemar contrast are experimental outcomes, not quantities forced by construction from the protocol definition. Self-references point to the system under test and its code release, not to a prior uniqueness theorem or ansatz that forbids alternatives and thereby manufactures the accuracy numbers. Methodological questions about whether L2→L3 fully isolates the background watcher versus mid-P3 messaging norms and prompt wording are causal-identification concerns, not circularity: they do not make the reported accuracies equal the inputs by definition. No self-definitional loop, fitted-input-as-prediction, load-bearing self-citation chain, imported uniqueness, smuggled ansatz, or renaming of a known result is present. Score 0 is the appropriate honest finding.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

Load-bearing content is experimental and engineering, not axiomatic math. The claim rests on benchmark rules, harness capabilities, protocol design, judge reliability, and the operational meaning of background wait_for_mention—not on fitted physical constants or newly postulated particles.

free parameters (3)
  • Number of agents (fixed at 4) = 4
    Team size is a design choice held fixed across L1–L3; headline gains are not shown to be invariant to 2, 3, or 8 agents.
  • Five-phase protocol structure and phase-transition gating by agent-1 = P1–P5 as in Algorithm 1
    Phase boundaries, approval gating, and assembler role are hand-designed; results are for this protocol plus AgentRadio, not for the primitives in isolation across arbitrary workflows.
  • wait_for_mention timeout and step-boundary delivery semantics
    When messages surface depends on harness step boundaries and timeout behavior; not swept in the paper.
axioms (4)
  • domain assumption SWE-Atlas QnA grading is a valid external measure: task resolved iff all rubrics pass under the fixed Claude Opus 4.5 LLM judge; modifying source fails the task.
    All accuracy numbers inherit benchmark and judge assumptions stated in Related Work / Experiments.
  • domain assumption Mainstream coding harnesses can run a shell watcher as a true background OS task so listening is not an LLM step.
    Implementation section: passive mode adds no LLM calls beyond surfaced message tokens.
  • ad hoc to paper L2 vs L3 differs only in communication mode and one phase prompt, holding model, harness, and protocol fixed.
    This is the identifying assumption for attributing +10.5/+11.3 points to passive awareness (Experiments Setup; Communication Primitives).
  • domain assumption Agents instructed by prompt will post on-discovery and fold incoming mentions into ongoing subtasks rather than ignore or thrash.
    Behavioral premise of Phase 3 passive protocol; failure modes appear in Grafana case (no conception formed) and in rubric losses from distracting messages.
invented entities (1)
  • AgentRadio message layer (create_thread, send_message, wait_for_mention) independent evidence
    purpose: Provide lateral async natural-language coordination so working agents remain passively aware of peers.
    Core artifact; implemented as server plus shell scripts, not a physical entity. Independent evidence is the public code repo and the L2/L3 empirical delta.

pith-pipeline@v1.2.0-daily-grok45 · 18484 in / 3477 out tokens · 78624 ms · 2026-07-31T07:35:27.453140+00:00 · methodology

0 comments
read the original abstract

Understanding large codebases is a long-horizon task for Large Language Model (LLM) agents: answering a single question can require building and running the software, tracing execution across files, and synthesizing evidence over tens of minutes. On SWE-Atlas QnA, a benchmark of long-horizon questions over production repositories, a single Claude Code agent (Opus 4.6) resolves only 32.3% of tasks. Dividing the work among agents with clean contexts mitigates this limitation. However, the subtasks of code comprehension are interdependent. One agent's findings can rewrite another's task, so agents must coordinate during execution, not only at phase boundaries. Existing multi-agent systems support such exchange only between phases, through staged handoffs or synchronized rounds. Communication and work remain mutually exclusive. A discovery made mid-execution cannot be shared until the next boundary. We present AgentRadio, an asynchronous message-passing layer that equips coding-agent harnesses with three primitives: threads, messages, and waiting for mentions. The last runs as a background task, surfacing teammates' messages without interrupting foreground work, so each agent remains passively aware of its peers and folds new findings into its ongoing task. Under a five-phase protocol of division of labor and negotiation, four agents organized by AgentRadio resolve 62.1% of tasks, 29.8 points above a single agent and above Claude Code with the newer Opus 4.8 (57.2%). Rubric-level analysis shows the gain growing with task difficulty, consistent with mid-course correction as the underlying mechanism. Our code is available at https://github.com/Coral-Protocol/AgentRadio.

Figures

Figures reproduced from arXiv: 2607.28430 by Caelum Forder, Peter Carroll, Qianbo Zang, Suman Deb, Xinxing Ren, Zekun Guo, Ziyan Wang.

Figure 1
Figure 1. Figure 1: Task accuracy on SWE-Atlas QnA (124 tasks). [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: AgentRadio and the five-phase protocol. Left: each Claude Code agent runs its harness work in the foreground while [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The task mix of SWE-Atlas QnA: 124 questions [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Rubric-level attribution of the Opus 4.6 ladder, as [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Rubrics passed on the 60 tasks that L2 leaves unre [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: The Grafana case with Opus 4.6. The two runs [PITH_FULL_IMAGE:figures/full_fig_p007_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

19 extracted references · 12 linked inside Pith

  1. [2]

    Why Do Multi-Agent LLM Systems Fail? arXiv:2503.13657. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pondé, H.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin,P.;Chan,B.;Gray,S.;Ryder,N.;Pavlov,M.;Power, A.;Kaiser,L.;Bavarian,M.;Winter,C.;Tillet,P.;Such,F.P.; Cummings, D...

    Pith/arXiv arXiv

  2. [4]

    arXiv:2412.21033

    Plan- craft: an evaluation dataset for planning with LLM agents. arXiv:2412.21033. Deng,X.;Da,J.;Pan,E.;He,Y.Y.;Ide,C.;Garg,K.;Lauffer, N.;Park,A.;Pasari,N.;Rane,C.;Sampath,K.;Krishnan,M.; Kundurthy,S.;Hendryx,S.;Wang,Z.;Bharadwaj,V.;Holm, J.;Aluri,R.;Zhang,C.B.C.;Jacobson,N.;Liu,B.;andKen- stler,B.2025. SWE-BenchPro:CanAIAgentsSolveLong- Horizon Software...

    Pith/arXiv arXiv 2025

  3. [6]

    arXiv:2404.02183

    Self-Organized Agents: A LLM Multi-Agent Framework toward Ul- tra Large-Scale Code Generation and Optimization. arXiv:2404.02183. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K

    Pith/arXiv arXiv

  4. [7]

    junyouli;Zhang,Q.;Yu,Y.;FU,Q.;andYe,D.2024

    SWE-bench: Can Language Models Resolve Real-world Github Issues? InInternational Conference on Learning Representations (ICLR). junyouli;Zhang,Q.;Yu,Y.;FU,Q.;andYe,D.2024. More AgentsIsAllYouNeed.TransactionsonMachineLearning Research. Kim,Y.;Gu,K.;Park,C.;Park,C.;Schmidgall,S.;Heydari, A.A.;Yan,Y.;Zhang,Z.;Zhuang,Y.;Liu,Y.;Malhotra,M.; Liang, P. P.; Park...

    2024

  5. [9]

    arXiv:2503.14499

    Measuring AI Ability to Complete Long Software Tasks. arXiv:2503.14499. La Malfa, E.; La Malfa, G.; Marro, S.; Zhang, J.; Black, E.; Luck, M.; Torr, P.; and Wooldridge, M

    Pith/arXiv arXiv

  6. [10]

    InAdvances in Neural Information Processing Systems

    Large Language Models Miss the Multi-agent Mark. InAdvances in Neural Information Processing Systems. Laban,P.;Hayashi,H.;Zhou,Y.;andNeville,J.2026. LLMs Get Lost In Multi-Turn Conversation. InInternational Con- ference on Learning Representations (ICLR). Levy, M.; Jacoby, A.; and Goldberg, Y

    2026

  7. [11]

    https://github.com/openai/ codex

    Codex CLI. https://github.com/openai/ codex. OpenCodecontributors.2025. OpenCode. https://opencode. ai. Phan,H.N.;Nguyen,T.N.;Nguyen,P.X.;andBui,N.D.Q

    2025

  8. [12]

    arXiv:2409.16299

    HyperAgent:GeneralistSoftwareEngineeringAgents to Solve Coding Tasks at Scale. arXiv:2409.16299. Pugachev, S

    Pith/arXiv arXiv

  9. [13]

    arXiv:2510.18893

    CodeCRDT: Observation-Driven Coordination for Multi-Agent LLM Code Generation. arXiv:2510.18893. Qian, C.; Liu, W.; Liu, H.; Chen, N.; Dang, Y.; Li, J.; Yang, C.; Chen, W.; Su, Y.; Cong, X.; Xu, J.; Li, D.; Liu, Z.; and Sun, M

    arXiv

  10. [14]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    ChatDev: Communicative Agents for Software Development. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Qu, A.; Zheng, H.; Zhou, Z.; Yan, Y.; Tang, Y.; Ong, S. Y.; Hong, F.; Zhou, K.; Jiang, C.; Kong, M.; Zhu, J.; Jiang, X.; Li,S.;Wu,C.;Low,B.K.H.;Zhao,J.;andLiang,P.P.2026. CORAL: Towards ...

    Pith/arXiv arXiv 2026

  11. [15]

    In Proceedings of the 42nd International Conference on Ma- chine Learning

    Paper- Bench:EvaluatingAI’sAbilitytoReplicateAIResearch. In Proceedings of the 42nd International Conference on Ma- chine Learning. Wang, D.; Yin, D.; Desai, R.; Li, L.; Celikyilmaz, A.; and Ni,A.2026. LearningtoInterruptinLanguage-basedMulti- agent Communication. arXiv:2604.06452. Wang, H.; Ni, Z.; Zhang, S.; Lu, S.; Hu, S.; He, Z.; Hu, C.; Lin, J.; Guo,...

    Pith/arXiv arXiv 2026

  12. [16]

    arXiv:2505.21577

    RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving. arXiv:2505.21577. Xie, T.; Zhang, D.; Chen, J.; Li, X.; Zhao, S.; Cao, R.; Hua, T. J.; Cheng, Z.; Shin, D.; Lei, F.; Liu, Y.; Xu, Y.; Zhou, S.; Savarese, S.; Xiong, C.; Zhong, V.; and Yu, T

    Pith/arXiv arXiv

  13. [17]

    InAdvancesinNeu- ral Information Processing Systems

    OS- World: Benchmarking Multimodal Agents for Open-Ended TasksinRealComputerEnvironments. InAdvancesinNeu- ral Information Processing Systems. Xu,F.F.;Song,Y.;Li,B.;Tang,Y.;Jain,K.;Bao,M.;Wang, Z.;Zhou,X.;Guo,Z.;Cao,M.;Yang,M.;Lu,H.Y.;Martin, A.; Su, Z.; Maben, L.; Mehta, R.; Chi, W.; Jang, L.; Xie, Y.; Zhou,S.;andNeubig,G.2025. TheAgentCompany:Bench- mar...

    2025

  14. [18]

    InAdvancesinNeuralInformationProcessingSystems

    SWE-agent: Agent- Computer Interfaces Enable Automated Software Engineer- ing. InAdvancesinNeuralInformationProcessingSystems. Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.R.;andCao,Y.2023. ReAct:SynergizingReasoningand ActinginLanguageModels. InInternationalConferenceon Learning Representations (ICLR). Yu, J.; Ding, Y.; and Sato, H

    2023

  15. [19]

    GPTSwarm:LanguageAgentsas OptimizableGraphs.InProceedingsofthe41stInternational Conference on Machine Learning, Proceedings of Machine Learning Research

    Zhuge,M.;Wang,W.;Kirsch,L.;Faccio,F.;Khizbullin,D.; andSchmidhuber,J.2024. GPTSwarm:LanguageAgentsas OptimizableGraphs.InProceedingsofthe41stInternational Conference on Machine Learning, Proceedings of Machine Learning Research

    2024

  16. [2021]

    Chen, Z.; Tang, R.; Deng, G.; Wu, F.; Wu, J.; Jiang, Z.; Prasanna, V.; Cohan, A.; and Wang, X

    EvaluatingLargeLanguageModelsTrainedonCode.ArXiv, abs/2107.03374. Chen, Z.; Tang, R.; Deng, G.; Wu, F.; Wu, J.; Jiang, Z.; Prasanna, V.; Cohan, A.; and Wang, X

    Pith/arXiv arXiv

  17. [2024]

    InProceedings of the 41st International Conference on Machine Learning

    Improving Factuality and Reasoning in Language Models through Multiagent Debate. InProceedings of the 41st International Conference on Machine Learning. Geng,J.;andNeubig,G.2026. EffectiveStrategiesforAsyn- chronous Software Engineering Agents. arXiv:2603.21489. Hong,S.;Zhuge,M.;Chen,J.;Zheng,X.;Cheng,Y.;Wang, J.; Zhang, C.; wang, z.; Yau, S.; Lin, Z.; Zh...

    Pith/arXiv arXiv 2026

  18. [2025]

    arXiv:2508.00828

    Finance Agent Benchmark: Benchmarking LLMs on Real- world Financial Research Tasks. arXiv:2508.00828. Cemri, M.; Pan, M. Z.; Yang, S.; Agrawal, L. A.; Chopra, B.; Tiwari, R.; Keutzer, K.; Parameswaran, A.; Klein, D.; Ramchandran, K.; Zaharia, M.; Gonzalez, J. E.; and Sto- ica, I

    Pith/arXiv arXiv

  19. [2026]

    arXiv:2512.08296

    Towards a Science of Scaling Agent Systems. arXiv:2512.08296. Kwa, T.; West, B.; Becker, J.; Deng, A.; Garcia, K.; Hasin, M.;Jawhar,S.;Kinniment,M.;Rush,N.;Arx,S.V.;Bloom, R.; Broadley, T.; Du, H.; Goodrich, B.; Jurkovic, N.; Miles, L.H.;Nix,S.;Lin,T.;Painter,C.;Parikh,N.;Rein,D.;Sato, L. J. K.; Wijk, H.; Ziegler, D. M.; Barnes, E.; and Chan, L

    Pith/arXiv arXiv