Pith. sign in

REVIEW 3 major objections 6 minor 12 references

Frontier coding agents get at most one in four realistic oncall root-cause tasks right—and that is already a lower bound on how unready they are.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 04:04 UTC pith:7F7Z2WXU

load-bearing objection Solid new oncall-agent benchmark with real telemetry+code stack; ~25%/10% medium/hard RCA accuracy is credible within-bench, with the main soft spot being LLM-assisted labels outside a 40-task verified slice. the 3 major comments →

arxiv 2607.28545 v1 pith:7F7Z2WXU submitted 2026-07-30 cs.CL cs.AIcs.SE

ORCA-bench: How Ready Are Language Model Agents for Oncall?

Albert Gong , Kyuseong Choi , Abhineet Agarwal , Jason Schechner , Ryan Huang , Raj Agrawal , Anish Agarwal , Raaz Dwivedi This is my paper
classification cs.CL cs.AIcs.SE
keywords root cause analysissite reliability engineeringLLM agentsobservabilitymicroservicesOpenTelemetrybenchmarkhallucination
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Writing and patching code is not the same job as oncall root cause analysis. When production breaks, an engineer starts from a vague user complaint, often hours late, and has to reason over live metrics, logs, traces, and source code. ORCA-bench puts general-purpose coding agents in that setting: a live OpenTelemetry microservice system with six days of real telemetry behind Grafana, full source access, and 1,079 tasks that vary how vague the report is, how long detection is delayed, and how many faults overlap. On the realistic Medium setting the best agent names every plausible root cause only 25.3% of the time; on Hard it falls to 10%. Weaker models invent an implausible cause in up to 40% of reports, and stripping source code hurts every score. Because the testbed is a public, fixed, fifty-gigabyte system with tasks run in isolation, the authors argue these numbers understate the gap to real production reliability work.

Core claim

Across five frontier coding agents, the best RCA Accuracy is 25.3% on Medium-difficulty (realistic-input) tasks and 10.0% on Hard. Hallucination of implausible root causes reaches 40% for the weakest model, source-code access is load-bearing for every metric, and even a stronger model on a verified subset does not close the gap. On a curated public six-day testbed with isolated investigations, this already shows agents are not ready for oncall; real production is larger, more dynamic, and more idiosyncratic, so the reported gap is a lower bound on the engineering still required.

What carries the argument

ORCA-bench: a live Astronomy Shop microservice environment with Prometheus, Jaeger, and OpenSearch via Grafana plus source access, paired with 1,079 RCA tasks that jointly vary issue specificity (Easy/Medium/Hard), time-to-detection, and five co-occurring fault scenario types, scored by symptom-level multi-cause rubrics and an LLM judge validated at Cohen’s κ_w = 0.90 against humans.

Load-bearing premise

The main risk is that AI-built user reports and multi-cause answer sets, with full human sign-off on only a forty-task slice, are clean enough that low accuracy on the full suite is not mostly label noise—especially on Hard tasks that average more than four concurrent causes.

What would settle it

If independent human SREs, blind to model outputs, re-label a large stratified sample of Hard and Medium tasks and agents then clear a large majority of those re-labeled tasks—or if the same agents, given only the paper’s interfaces, routinely nail all plausible causes on a private production-scale system—the readiness gap claim would fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Production oncall cannot yet be safely handed to frontier coding agents on ambiguous user reports alone.
  • Benchmarks that omit source code, realistic telemetry UIs, delayed detection, or vague reports will overstate SRE readiness.
  • Source-code access is not optional: removing it drops accuracy and spikes hallucination across every model tested.
  • Hard multi-fault, low-context incidents remain the binding failure mode even when agents can partially cite telemetry.
  • Closing the gap will need more than bigger models: scale, private system priors, memory across incidents, and eventually action feedback.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Hybrid stacks that combine agents with classical causal or graph RCA may be necessary before full autonomy, because pure agent trajectories still miss concurrent causes and lock onto loud noise.
  • Persistent memory and cross-incident learning are natural next levers: the paper’s cold-start isolation likely hides both human advantage today and a path where agents could eventually surpass humans.
  • A public demo shop in pretraining may inflate scores; private idiosyncratic stacks could show a still steeper drop than the code-ablation already reports.
  • Read-only diagnosis without a mitigate-and-watch loop removes the strongest natural feedback signal agents would need to refine hypotheses in the wild.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces ORCA-bench, a production-fidelity benchmark for language-model agents performing oncall root-cause analysis. It pairs a live OpenTelemetry Astronomy Shop deployment (six days of metrics/logs/traces via Grafana/Prometheus/Jaeger/OpenSearch, plus source code) with 1,079 RCA tasks that vary issue specificity (Easy/Medium/Hard), time-to-detection, and co-occurring fault scenarios induced by feature flags. Ground-truth symptoms are SRE-curated; tasks are scored against sets of plausible root causes via a GPT-5.4 judge validated against human re-scores on a 40-task Verified subset (κ_w=0.90). Across five frontier agents under Terminus-2, best RCA Accuracy is 25.3% on Medium and 10.0% on Hard, with substantial hallucination for weaker models and uniform degradation when source code is removed. The authors argue these numbers are a lower bound on the engineering gap before agents can be trusted with real production reliability, and they release the public set.

Significance. If the measurements hold, this is a substantial contribution to SRE/agent evaluation. Prior RCA benchmarks strip out at least one of telemetry interfaces, raw live-load signals, or source code, and none systematically vary report specificity and TTD jointly with multi-fault structure (Table 1). The paper ships a concrete, reusable environment, human-validated scoring agreement, code-access and difficulty ablations, behavioral analyses of agent command graphs and telemetry failure modes, and an explicitly scoped lower-bound claim rather than an overclaim of production readiness. That combination is rare and useful for both the agent and reliability communities. Public release of the dataset further strengthens the contribution.

major comments (3)
  1. [§3 B4–B5; §5 Finding #2; Table I.3] §3 stages B4–B5 and §5 Finding #2: User issues and plausible root-cause answer sets are produced with GPT-5.4, with full human sign-off only on the stratified 40-task Verified subset (symptoms receive broader SRE validation in B6). On Hard tasks the mean answer-set size is 4.41±0.11 (Table I.3), and RCA Accuracy requires naming every listed flag. Over-inclusive multi-cause labels would mechanically suppress exact-match accuracy and inflate the headline gap. The Verified results and Fable-5 subset partially bound this risk, but the manuscript should report (i) human audit rates/disagreement on a larger sample of B5 verdicts stratified by Hard/multi-flag tasks, and/or (ii) sensitivity of RCA Accuracy to answer-set pruning (e.g., majority-plausible or top-k causes). Without that, the 10.0% Hard figure is harder to interpret as pure agent failure versus label strictness.
  2. [§4 E3; Fig. 1; App. I] §4 E3 (M1) and App. I: RCA Accuracy is defined as naming every plausible flag. Given concurrent scenarios (Day-6 six-flag peak; App. C.2) and Hard’s large hypothesis space, this is a very strict primary metric. RCA depth already provides partial credit and is more informative (best ~49% depth vs ~25–31% accuracy). The central readiness claim would be clearer if the paper (a) elevates depth and per-rubric mechanism/incident-time rates alongside accuracy in the abstract/Fig. 1, and (b) reports a relaxed “any-correct-cause” or “fraction of GT causes named” metric so readers can separate “missed the whole set” from “found nothing useful.” The current framing risks overstating unreadiness relative to what oncall practice actually requires (identify a actionable cause, not necessarily every co-active flag).
  3. [§5; §6 Methods we did not study; Fig. 8] §5 and §6: All main results use a single harness (Terminus-2, tmux-only, medium effort, temperature 1). Finding #4 shows 26–40% of telemetry calls empty or erroring, which may partly reflect tool-use skill under this harness rather than RCA reasoning per se. The lower-bound rhetoric in §6 is otherwise careful, but the manuscript should either evaluate at least one alternative workflow (e.g., structured Grafana/PromQL scaffolding or multi-agent planner) on the Verified subset, or more sharply qualify that the measured gap is “under generic coding-agent harnesses,” not an upper bound on what specialized SRE agents could achieve. This is load-bearing for the claim that frontier coding agents are “not yet ready for oncall.”
minor comments (6)
  1. [Abstract; Fig. 1; §5] Abstract and Fig. 1 cite best Medium accuracy 25.3% (Sonnet 4.6) while §5 text sometimes highlights different models for depth vs accuracy; keep the lead numbers and model names consistent across abstract, Fig. 1, and §5.
  2. [Abstract; §3] Task count is stated as 1,079 in the abstract and 1076 in §3 (“ORCA-bench comprises 1076…”); reconcile the off-by-three discrepancy.
  3. [Table 1] Table 1 is helpful; a short column or footnote clarifying what “human verification” covers for each prior benchmark (symptoms vs root causes vs scores) would make the comparison sharper.
  4. [§5 Finding #1; Fig. 4] Fig. 4 / App. J case study is excellent; consider pointing readers there earlier in §5 Finding #1 so the failure mode (distraction by load-generator noise) is concrete before aggregate tables.
  5. [§5; Fig. K.1] Model name “Claude Fable 5” appears without a citation or availability note; clarify whether this is a preview/internal model and how the Verified-only evaluation was scoped.
  6. [Title; §5] Minor typos/style: “HowReadyAre” spacing in the title block; “isuse” in §5; “quantity the severity” → “quantify”; ensure arXiv ID/date consistency in the preprint header.

Circularity Check

0 steps flagged

No significant circularity: ORCA-bench is an empirical benchmark/measurement paper whose agent scores are not forced by construction from its inputs.

full rationale

The paper’s central claims are measured outcomes (RCA Accuracy, depth, hallucination, code-ablation deltas) from frontier agents run in a fixed harness against curated tasks—not first-principles derivations or fitted-then-predicted quantities. Ground-truth symptoms are SRE-validated; the LLM-as-judge is independently human-rescored on ORCA-bench Verified (κ_w=0.90); and agent trajectories are separate from label construction. GPT-5.4 assists stages B4–B5 (issue rephrasing and plausible-cause sets) and also serves as judge, which is a mild methodological coupling, but it does not make reported agent accuracy equal the construction inputs by definition: agents can still succeed or fail independently, Verified human scores corroborate the judge, and the lower-bound rhetoric is scoped to a public isolated testbed rather than smuggled uniqueness or self-citation theorems. No self-definitional identity, fitted-input-as-prediction, load-bearing self-citation uniqueness chain, or renamed known law is present. Score 1 only for the dual GPT-5.4 construction/judge role as a non-load-bearing caveat, not a circular derivation.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

Load-bearing commitments are methodological domain assumptions about what counts as production-fidelity RCA and how labels are defined, plus engineering choices in the demo system—not free physical constants. Invented entities are benchmark constructs (task ladder, scenario typology, scoring rubrics), not new physical objects. The central numerical claims rest on those constructs plus standard LLM-agent evaluation practice.

free parameters (4)
  • TTD discrete grid {15m, 1h, 8h, 24h} and report-time style coupling = {15min, 1h, 8h, 24h}
    Hand-chosen temporal discretization and coupling of longer TTDs to broader report styles; shapes difficulty mix though not fitted to maximize a gap.
  • Per-rubric 0–3 scoring rubric thresholds = 0–3 ordinal + N-1 penalty
    Ordinal depth scale and material-wrongness penalty are author-defined operationalizations of “progress toward root cause.”
  • Quiet-window margin (±10 min) and incident-time match window (±10 min) = ±10 minutes
    Tolerance hyperparameters in control and time scoring that affect pass/fail near boundaries.
  • Agent harness settings (temperature 1, medium effort, max tokens 16384, Terminus-2) = Terminus-2 defaults + medium effort
    Single harness/prompt template; performance is conditional on these un-swept choices (authors note this limitation).
axioms (5)
  • domain assumption Feature-flag faults in OpenTelemetry Astronomy Shop with frontend-visible symptoms are adequate proxies for user-facing production incidents for benchmarking RCA.
    Core environment choice in §3 B1–B2; real incidents include novel code bugs, bad deploys, and dependency failures beyond preset flags.
  • ad hoc to paper The set of plausible root causes is well-defined by flags active in the report-day window before detection, after dropping no-frontend-symptom events and GPT-5.4 plausibility filtering.
    Answer-set construction in §3 B5; multi-label exact-set accuracy depends on this definition.
  • domain assumption LLM-as-judge grades with the stated rubric are valid when human agreement on the Verified subset reaches κ_w≈0.90.
    §4 evaluation design; standard in agent benches but still an assumption when extrapolating to all 1,079 tasks.
  • domain assumption Performance on a public, pretraining-exposed demo with isolated cold-start tasks lower-bounds difficulty of private, evolving production systems.
    Explicitly argued in abstract and §6; directionally plausible but not experimentally proven on private fleets.
  • standard math Standard statistical reporting of means ± SE across tasks is appropriate for comparing agents on this benchmark.
    Used throughout §5 and appendix tables; tasks are not fully i.i.d. (shared schedule/days) but conventional for leaderboards.
invented entities (3)
  • ORCA-bench task distribution (specificity × TTD × scenario types over a 6-day flag schedule) independent evidence
    purpose: Define a controlled, multi-axis RCA evaluation suite with 1,079 tasks.
    Benchmark construct; value is instrumental, validated by SRE process rather than external natural law.
  • Symptom-level per-flag rubrics (metrics/logs/traces/frontend clusters and signal layers) independent evidence
    purpose: Ground-truth evidence checklist for grading depth and citations beyond a single cause name.
    Author-curated with SRE sign-off; falsifiable against the released telemetry for those flags.
  • RCA accuracy / RCA depth / hallucination rate metric triple independent evidence
    purpose: Summarize exact multi-cause identification, partial credit, and implausible causes.
    Evaluation invention; not a physical entity. Independent check via human re-score on Verified.

pith-pipeline@v1.2.0-daily-grok45 · 37966 in / 4126 out tokens · 86849 ms · 2026-07-31T04:04:23.687433+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of ORCA-bench: How Ready Are Language Model Agents for Oncall?." pith.science (2026). https://pith.science/paper/7F7Z2WXU

@misc{pith2026260728545,
  author       = {Pith},
  title        = {Pith review of: ORCA-bench: How Ready Are Language Model Agents for Oncall?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7F7Z2WXU}},
  note         = {Machine review of arXiv:2607.28545}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $\kappa_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/ORCA-bench.

Figures

Figures reproduced from arXiv: 2607.28545 by Abhineet Agarwal, Albert Gong, Anish Agarwal, Jason Schechner, Kyuseong Choi, Raaz Dwivedi, Raj Agrawal, Ryan Huang.

Figure 1
Figure 1. Figure 1: RCA accuracy measures whether agents identified [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Construction pipeline. We construct ORCA-bench in six stages. and manually verify a subset of 40 tasks to obtain ORCA-bench Verified. See Sec. 3 for details. Incident start time [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Task definition. The SRE agent is provided a user-facing issue at the report time (an offset after the incident start time) and begins investigation at the detection time (a time-to-detection after the incident start time). B1. Environment. Our starting point is the OpenTelemetry Astronomy Shop, a microservice-based distributed system intended to illustrate the implementation of OpenTelemetry in a near rea… view at source ↗
Figure 4
Figure 4. Figure 4: Excerpts from a low-scoring and high-scoring incident RCA report on the same task. See App. J for the full, unabridged reports. 0% 50% 100% Control Control Control Control Control Easy Easy Easy Easy Easy Medium Medium Medium Medium Medium Hard Hard Hard Hard Hard RCA Accuracy Easy Easy Easy Easy Easy Medium Medium Medium Medium Medium Hard Hard Hard Hard Hard RCA Depth Opus 4.7 Sonnet 4.6 GPT-5.5 GLM-5 De… view at source ↗
Figure 5
Figure 5. Figure 5: RCA quality by issue specificity. We use GPT-5.4 as the LLM judge and report mean ± 1 standard error across the 195 control, 288 easy, 316 medium, 280 hard tasks. Two failure modes are agents get distracted by background noise and cannot exhaustively find all root causes when issues are concurrent. In [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: RCA quality with and without code access. We use GPT-5.4 as the LLM judge using the metrics defined in Sec. 4 and report mean ± 1 standard error across all N = 884 incident tasks. (a) Opus 4.7 2% 16% 70% 12% 10% 58% 22% 34% 10% 61% 87% 56% 22% (b) GLM-5 13% 20% 57% 10% 27% 23% 43% 11% 61% 81% 63% 37% Telemetry discovery Telemetry query Source code Other [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Agent behavior during investigation. Node area is proportional to the marginal share of commands in that category. Directed edges are transition probabilities P(next | prev) between con￾secutive commands. For readability, we omit edges whose transition probability P(next | prev) ≤ 0.1. GPT-5.5, GLM-5, and DeepSeek-V4-Pro are shown in Fig. I.1. Finding #3: removing source code drops RCA accuracy by 9–16 per… view at source ↗
Figure 8
Figure 8. Figure 8: Telemetry retrieval efficiency and accuracy. We report mean ± 1 standard error across all N = 1079 tasks, all telemetry commands, and all tasks with ground-truth metrics/logs/traces for (a), (b), and (c), respectively. Empty means the call ran successfully but returned no data and error means the call failed. 6 CONCLUSION We introduced ORCA-bench, the first SRE benchmark that exposes language model agents … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 3 linked inside Pith

  1. [1]

    Long code arena: a set of benchmarks for long-context code models

    Egor Bogomolov, Aleksandra Eliseeva, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, Maria Tigina, Yaroslav Golubev, Alexander Kovrigin, Arie Van Deursen, Maliheh Izadi, et al. Long code arena: a set of benchmarks for long-context code models. arXiv preprint arXiv:2406.11612, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    Sregym: A live benchmark for ai sre agents with high-fidelity failure scenarios

    Jackson Clark, Yiming Su, Saad Mohammad Rafid Pial, Yifang Tian, Lily Gniedziejko, Hans-Arno Jacobsen, Yinfang Chen, and Tianyin Xu. Sregym: A live benchmark for ai sre agents with high-fidelity failure scenarios. arXiv preprint arXiv:2605.07161, 2026

    Pith/arXiv arXiv 2026

  3. [3]

    Itbench: Evaluating ai agents across diverse real-world it automation tasks

    Saurabh Jha, Rohan R Arora, Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, et al. Itbench: Evaluating ai agents across diverse real-world it automation tasks. In International Conference on Machine Learning, pages 27134--27197. PMLR, 2025

    2025

  4. [4]

    Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107--54157, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pages 54107--54157, 2024

    2024

  5. [5]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, An...

    Pith/arXiv arXiv 2026

  6. [6]

    Rcaeval: A benchmark for root cause analysis of microservice systems with telemetry data

    Luan Pham, Hongyu Zhang, Huong Ha, Flora Salim, and Xiuzhen Zhang. Rcaeval: A benchmark for root cause analysis of microservice systems with telemetry data. In Companion Proceedings of the ACM on Web Conference 2025, pages 777--780, 2025

    2025

  7. [7]

    Building ai agents for autonomous clouds: Challenges and design principles

    Manish Shetty, Yinfang Chen, Gagan Somashekar, Minghua Ma, Yogesh Simmhan, Xuchao Zhang, Jonathan Mace, Dax Vandevoorde, Pedro Las-Casas, Shachee Mishra Gupta, et al. Building ai agents for autonomous clouds: Challenges and design principles. In Proceedings of the 2024 ACM Symposium on Cloud Computing, pages 99--110, 2024

    2024

  8. [8]

    Openrca: Can large language models locate the root cause of software failures? In The Thirteenth International Conference on Learning Representations, 2025

    Junjielong Xu, Qinan Zhang, Zhiqing Zhong, Shilin He, Chaoyun Zhang, Qingwei Lin, Dan Pei, Pinjia He, Dongmei Zhang, and Qi Zhang. Openrca: Can large language models locate the root cause of software failures? In The Thirteenth International Conference on Learning Representations, 2025

    2025

  9. [9]

    Swe-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025

    John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, et al. Swe-bench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025

    2025

  10. [10]

    Swe-smith: Scaling data for software engineering agents

    John Yang, Kilian Lieret, Carlos Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. Advances in Neural Information Processing Systems, 38, 2026

    2026

  11. [11]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36: 0 46595--46623, 2023

    2023

  12. [12]

    Graders should cheat: privileged information enables expert-level automated evaluations

    Jin Peng Zhou, S \'e b Arnold, Nan Ding, Kilian Q Weinberger, Nan Hua, and Fei Sha. Graders should cheat: privileged information enables expert-level automated evaluations. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 16583--16601, 2025

    2025