Pith. sign in

REVIEW 4 major objections 6 minor 20 references

This paper argues that AI agents should not be released based on capability alone, and proposes the ProofAgent Index (PAI), a four-part governance readiness score that combines Evaluation, Context, Compliance, and Governance, with hard-bloc

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-01 03:27 UTC pith:NCFH5PHA

load-bearing objection Practical governance index with a real open-source harness, but the validation is self-referential – the outcome and predictor come from the same trap taxonomy, so the AUC doesn't yet support the production-readiness claim. the 4 major comments →

arxiv 2607.27677 v1 pith:NCFH5PHA submitted 2026-07-30 cs.MA cs.AI

Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness

Fouad Bousetouane This is my paper
classification cs.MA cs.AI
keywords AI agentsproduction readinessgovernancecompliancecontext engineeringadversarial evaluationreadiness indexProofAgent Index
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that production readiness for AI agents is a governance problem, not a capability problem. A capable agent can still be unsafe to deploy if its operating context is weak, its compliance evidence is incomplete, or if no one can monitor and control it. It introduces the ProofAgent Index, a single decomposable score built from behavioral evaluation, context quality, compliance evidence, and governance control, with hard-block rules that cap the score when any non-negotiable condition fails. Validation across healthcare and finance configurations reports that PAI orders held-out failure risk with AUC 0.98, and that context engineering reduces trap-conditional defects from 65.7% to 17.8% while capability gains saturate after the mid-tier. The practical claim is that release decisions should be made on this kind of readiness evidence, not on demos or benchmark scores.

Core claim

The central discovery is that behavior under test (capability) and the preconditions for safe operation (readiness) diverge: the same capable model can be reliably deployable when its context and governance are strong, and dangerous when they are weak. PAI operationalizes readiness as a weighted geometric mean of four independently measured dimensions — Evaluation, Context, Compliance, Governance — followed by an admissibility gate that caps the final score below the release threshold whenever a hard block fires (prohibited use, safety failure, tool breach, missing compliance evidence, unresolved governance finding, or insufficient capability for the risk tier). In a fully crossed grid of tw

What carries the argument

The central object is the ProofAgent Index (PAI), a composite readiness score with two layers. The measurement layer is a weighted geometric mean of four [0,100] dimension scores — Evaluation (observed behavior), Context (operating-environment quality), Compliance (rule alignment with missing evidence treated as failure), and Governance (ownership, approval, monitoring, rollback) — chosen because geometric aggregation gives limited compensation, so a weak dimension drags the score down more than an arithmetic mean would. The admissibility gate then applies hard-block rules: if a blocking condition fires, the final PAI is capped below the review threshold, preventing a strong aggregate from m

Load-bearing premise

The entire index rests on the assumption that failures caught by the harness's adversarial traps — and the LLM-judge defect scores derived from them — are a valid proxy for what will actually go wrong when the agent operates in a real regulated environment, since both the readiness predictor and the held-out outcome come from the same instrument.

What would settle it

A reader could collect post-deployment operational and compliance incident data for a set of agents that PAI rated Ready and compare incident rates against what PAI predicted; if Ready agents show the same or higher defect rates than Not-ready agents, or if trap-defined failures do not track real-world failures, the index's predictive claim is falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Release decisions for agents can be made on decomposable, auditable evidence instead of capability scores or demos, with a hard-block layer that prevents averaging away critical defects.
  • Context engineering becomes a first-class readiness lever: the validation shows it reduced trap-conditional defect rates from 65.7% to 17.8%, a 72.9% relative reduction.
  • Capability benchmarks alone are insufficient for deployment; the study found mid and strong agents nearly identical in aggregate defect rate (25.16% vs 24.91%), while weak agents remained risky even with strong context (52.13% defect rate).
  • A minimum capability floor may be justified for high-risk deployments, since governance and context improvements cannot fully rescue an insufficiently capable backbone.
  • The four-dimension evidence bundle makes readiness auditable: each PAI report preserves dimension scores, hard-block status, and failure traces, so the score explains why a configuration is ready, blocked, or risky.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The geometric-mean-plus-hard-block structure generalizes: any composite risk score that must respect non-negotiable constraints, such as safety floors in other AI or software release processes, could adopt the same two-stage design instead of a single weighted average.
  • The reported AUC of 0.98 should be read with caution because both the predictor (PAI) and the held-out outcome (PASS/FAIL on traps) are generated by the same harness; an independent evaluation using real-world post-deployment incidents would test whether the readiness signal transfers outside the instrument.
  • A testable extension: track a cohort of agents that PAI rated 'Ready with caveats' and compare their post-deployment incident rates to 'Ready' agents; if the bands do not order real-world incidents, the readiness bands would need recalibration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces the ProofAgent Index (PAI), a governance readiness index for AI agents that combines four dimensions — Evaluation (E), Context (Q), Compliance (C), and Governance (G) — into a weighted geometric mean, then applies a hard-block cap to produce a final readiness score, readiness bands, and release decisions. PAI is implemented in an open-source harness (ProofAgent Harness). The empirical validation uses a configuration grid over two regulated domains (healthcare, finance), three capability tiers, and two context conditions, yielding twelve configurations. The main result is a held-out AUC of 0.98 in predicting PASS/FAIL outcomes on a disjoint exam pack, plus a 10,000-turn validation showing strong context effects and a weak-to-mid capability improvement that saturates. The paper claims that capability is not production readiness and that PAI provides a decomposable, auditable readiness signal.

Significance. If the framework is interpreted as a proposal for structured release governance and the tooling is auditable, the paper makes a useful contribution. The open-source implementation, the reproducible configuration-level table (Table 8), and the explicit statement that defect rates are trap-conditional (Section 4.7) are genuine assets. The paper also correctly emphasizes that composite scores can average away non-negotiable defects, and the hard-block mechanism is a reasonable design choice. However, the central empirical claim that PAI carries 'held out readiness signal' for production readiness is not established by the current validation, because the PASS/FAIL outcome is generated by the same ProofAgent Harness trap taxonomy and LLM-as-judge that produce the Evaluation dimension and hard-block conditions. The AUC therefore largely measures internal consistency of the instrument rather than readiness in any externally anchored sense. The small sample (n=12) further weakens the empirical support. The framework itself is defensible, but the validation must be reframed or supplemented before the title's claim is supported.

major comments (4)
  1. [Section 4.3, Table 7] The PASS/FAIL outcome is defined as the presence of a 'release blocking failure' or 'failure tolerance' violation on the held-out exam pack, both of which are detected by the same ProofAgent Harness trap rules and LLM-as-judge scoring that generate the Evaluation dimension and hard-block conditions in the development pack. The development/exam split separates trap instances, but not the measurement instrument. Thus the AUC=0.98 primarily shows that configurations exhibiting hard-block-like failures in 25 development turns also exhibit them in 10 exam turns. This is an internal-consistency check, not a demonstration that PAI tracks production readiness in a real regulated environment. To support the abstract's claim, the paper needs an external outcome anchor (e.g., human expert release decisions, incident reports, regulatory findings) or must explicitly reframe the claim as 'readiness wi
  2. [Section 4.4, Table 7] The AUC is computed from twelve configurations (eight failures, four passes). With such a small sample, a single tied boundary case changes the AUC, and no confidence interval, bootstrap, or permutation test is reported. The statement in Section 4.13 that the result 'avoids overstating robustness' is not supported; the estimate is extremely fragile. The paper should report uncertainty intervals and a significance test against chance, or substantially temper the strength of conclusion G1.
  3. [Sections 3.2 and 3.3, Table 8] The PAI score depends on several free parameters (epsilon, tau_review, tau_ready, delta, equal weights, compliance passes, failure tolerance) and on qualitative hard-block conditions such as 'critical safety failure' and 'tool use breach,' which are not defined with operational thresholds in the paper. The reported AUC uses a single default parameter set. Without a sensitivity analysis or a precise, implementation-level specification of the hard-block rules and thresholds, it is unclear whether the observed separation reflects a general property of the index or a tuned choice. The paper should provide a formal definition of each hard-block condition and test the stability of the AUC to parameter variation.
  4. [Section 4.2, Table 8] The Compliance (C) and Governance (G) scores are supplied by the configuration through governance profiles and compliance checks, rather than being measured from the agent's behavior. Since the governance profile and compliance evidence also influence the outcome definition (risk tier, failure tolerance, responsibility for hard blocks), the design conflates the 'governance evidence' that feeds the predictor with the criterion that determines the outcome. The paper should clarify this dependency and discuss how it limits the claim that PAI integrates independent evidence from the four dimensions. As it stands, a configuration with a richer governance profile is more likely to pass both because PAI rewards the profile and because the profile relaxes failure tolerance.
minor comments (6)
  1. [Abstract and Section 1.2] The typography 'P AI' in the abstract is inconsistent with 'PAI' used elsewhere; use a single consistent notation throughout.
  2. [Section 4.2] The phrase 'three scoring personas with Debate consensus' is not defined. Please specify the personas and how the debate consensus is implemented; otherwise the LLM-judge reliability cannot be assessed.
  3. [Section 4.3] The statement 'This separation prevents direct circular validation' is too strong given the instrument-level circularity raised above. Rephrase to acknowledge the distinction between trap-instance separation and instrument separation.
  4. [Table 3] The strong tier is described as 'a newer mixture of experts model' without a precise model identifier or version. This is not reproducible from the paper; provide the exact model endpoint or release version.
  5. [Section 4.7] The 10,000-turn defect rates are clearly stated as trap-conditional, which is a strength. However, the phrase 'deployment relevant failures' in Table 10 should be qualified as 'trap-relevant failures' to avoid implying operational production failure rates.
  6. [Section 5.2, Table 16] The mapping of CLI inputs to PAI evidence is clear and useful. A small point: the table entry for '--assess-context' says it 'Produces the Context axis and context findings,' but the context score is also influenced by the context directory; clarify the relationship.

Circularity Check

1 steps flagged

Held-out AUC validates the harness against its own trap taxonomy: the FAIL label and PAI's hard-block cap are defined by the same release-blocking defect construct.

specific steps
  1. self definitional [§4.3 Held out validation strategy; §3.3 Hard block rules]
    "In this experiment, a configuration is marked FAIL if the held out exam pack contains a release blocking failure or violates the failure tolerance defined for the deployment risk tier. ... Hard block conditions include prohibited use, critical safety failure, hallucination resistance below a required floor, tool use breach, critical technical finding, missing mandatory compliance evidence, unresolved governance finding, or insufficient capability for the deployment risk tier."

    The outcome label is not an external anchor: 'release blocking failure' is the same family of defects that triggers PAI's hard-block cap (Eq. 3). The development/exam split is a split of trap instances, not of the measurement construct. Thus AUC=0.98 measures whether configurations whose dev-pack evidence would trigger hard-block-type findings also produce release-blocking-type findings in the exam pack—an internal consistency check of the harness's own taxonomy, not validation against an independent production-readiness criterion.

full rationale

The formal PAI construction (geometric mean plus hard-block cap) is internally coherent, and the dev/exam trap split is real, so the paper is not fully tautological. However, the central validation claim—that PAI 'carries held out readiness signal' (§4.13)—is qualified by the fact that the outcome is defined by the same release-blocking defect taxonomy that PAI's cap is designed to detect, and both sides are scored by the same harness LLM. The paper's defense in §4.13 that the validation 'is not circular with respect to trap exposure' addresses instance leakage but not construct overlap. The self-citations to the author's prior ProofAgent work ([1]–[3]) are descriptive rather than load-bearing, so they do not by themselves raise the score. The 10,000-turn analysis is transparently trap-conditional (§4.7) and does not anchor to production failure rates. Overall this is partial circularity: the AUC is a self-consistency result of the ProofAgent instrument, not an independent demonstration of production readiness.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 3 invented entities

The index depends on hand-set thresholds and on the validity of the author's own harness as both predictor and ground truth.

free parameters (7)
  • epsilon (log-safe floor) = 1
    Log-safe floor in Eq. (2); chosen, not derived.
  • tau_review (review threshold) = 65
    Review-level threshold for bands and cap; experimental default.
  • tau_ready (full readiness threshold) = 85
    Full readiness threshold; experimental default.
  • delta (margin below review threshold) = 1
    Margin below review threshold in Eq. (3).
  • equal weights w_a = 1/4 each
    Default weights for E, Q, C, G in Eq. (1); 'unless otherwise specified'.
  • compliance passes = 3
    Majority-vote threshold for compliance checks (§4.2).
  • failure tolerance per risk tier = unspecified
    Used to define PASS/FAIL on held-out exam pack (§4.3); values not given.
axioms (5)
  • domain assumption LLM-as-judge scores from the harness (primary gemma-3-27b-it, fallback qwen3) are valid measures of agent defects.
    All E and outcome labels rely on these scores; no human-label validation reported.
  • domain assumption ProofAgent Harness adversarial traps are a valid proxy for production deployment risk.
    The paper explicitly says defect rates are trap-conditional, not expected production rates (§4.7), yet uses them as readiness outcomes.
  • ad hoc to paper The four dimensions E, Q, C, G are complete for production readiness.
    Proposed in §3.1 without an independent derivation of completeness.
  • ad hoc to paper The listed hard-block conditions are sufficient and correctly weighted.
    Defined as a list in §3.3 with no formal link to regulatory requirements.
  • ad hoc to paper The PASS/FAIL rule on the exam pack reliably identifies release-blocking failures.
    Threshold and tolerance for the deployment risk tier are not fully specified (§4.3).
invented entities (3)
  • PAI (ProofAgent Index) no independent evidence
    purpose: Composite readiness score for gating agent releases
    No external validation; defined and tested within the author's own harness.
  • Readiness bands (Blocked/Not ready/Ready with caveats/Ready) no independent evidence
    purpose: Interpretation thresholds for release decisions
    Arbitrary thresholds; not validated against production outcomes.
  • Hard block conditions no independent evidence
    purpose: Non-negotiables that cap the score
    Hand-authored list with no independent grounding.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness." pith.science (2026). https://pith.science/paper/NCFH5PHA

@misc{pith2026260727677,
  author       = {Pith},
  title        = {Pith review of: Stop Shipping AI Agents on Faith: Capability Is Not Production Readiness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NCFH5PHA}},
  note         = {Machine review of arXiv:2607.27677}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

AI agents are moving into production workflows where they retrieve information, call tools, maintain state, and act on behalf of users or organizations, but many release decisions still rely on capability signals, demos, or behavioral tests that do not show whether an agent is ready to operate under production constraints. Capability is therefore not production readiness. This paper introduces the ProofAgent Index (PAI), a governance readiness index for AI agents. PAI combines four dimensions of deployment evidence: Evaluation, Context, Compliance, and Governance. Evaluation measures observed behavior, Context measures the operating environment that shapes that behavior, Compliance measures alignment with applicable rules and controls, and Governance measures whether the organization can authorize, monitor, audit, and control the agent during operation. PAI is implemented inside ProofAgent Harness, an open source infrastructure for auditable AI agent evaluation and governance. Validation across two heavily regulated domains, healthcare and finance, shows that PAI carries held out readiness signal and separates higher risk from lower risk configurations. The results show that context engineering strongly changes reliability, capability improves behavior but does not determine readiness, and governance evidence must remain visible rather than averaged away. PAI reframes agent release from a faith based deployment decision into an auditable readiness decision.

Figures

Figures reproduced from arXiv: 2607.27677 by Fouad Bousetouane.

Figure 1
Figure 1. Figure 1: illustrates the readiness stack used in this paper. The lower layers generate technical evidence through agent infrastructure and ProofAgent Harness evaluation. The upper layers translate that evidence into compliance assurance and governance decisions [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Configuration level ROC for PAI. PAI is computed from development trap evidence, while failure is measured on unseen ProofAgent Harness traps. Lower PAI is interpreted as higher readiness risk. Across the twelve configurations, PAI achieves AUC = 0.98, with one tied boundary case at the score boundary. 4.5 Configuration level PAI values [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Human-on-the-bridge: Scalable evaluation for ai agents, 2026

    Fouad Bousetouane. Human-on-the-bridge: Scalable evaluation for ai agents, 2026. URL https://arxiv.org/abs/2606.16871

    arXiv 2026

  2. [2]

    Ai agents do not fail alone: The context fails first.arXiv preprint,

    Fouad Bousetouane. Ai agents do not fail alone: The context fails first.arXiv preprint,

  3. [3]

    ProofAgent Harness: Open Infrastructure for Adversarial Evaluation of AI Agents

    Fouad Bousetouane. Proofagent harness: Open infrastructure for adversarial evaluation of ai agents.arXiv preprint arXiv:2605.24134, 2026. doi: 10.48550/arXiv.2605.24134. URL https://arxiv.org/abs/2605.24134

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.24134 2026

  4. [4]

    Regulation (eu) 2016/679 (general data protection regulation), 2016

    European Union. Regulation (eu) 2016/679 (general data protection regulation), 2016. URL https://eur-lex.europa.eu/eli/reg/2016/679/oj. Official Journal of the European Union

    2016

  5. [5]

    Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence, 2024

    European Union. Regulation (eu) 2024/1689 laying down harmonised rules on artificial intelligence, 2024. URL https://eur-lex.europa.eu/eli/reg/2024/1689/oj. Official Journal of the European Union

    2024

  6. [6]

    Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Samuel R. Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Sco...

    Pith/arXiv arXiv doi:10.48550/arxiv.2209.07858 2022

  7. [7]

    Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection.arXiv preprint arXiv:2302.12173, 2023

    Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection.arXiv preprint arXiv:2302.12173, 2023. doi: 10.48550/arXiv.2302.12173. URL https://arxiv.org/abs/2302.12173. 22

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.12173 2023

  8. [8]

    Iso/iec 42001:2023: Information technology—artificial intelligence— management system, 2023

    ISO/IEC. Iso/iec 42001:2023: Information technology—artificial intelligence— management system, 2023. URL https://www.iso.org/standard/42001. International Organization for Standardization and International Electrotechnical Commission

    2023

  9. [9]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K¨ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨ aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. InAdvances in Neural Information Processing Systems, 2020. doi: 10.48550/a rXiv.2005.11401. URL h...

    Pith/arXiv arXiv doi:10.48550/arxiv.2005.11401 2020

  10. [10]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi: 10.1162/tacl a 00638. URL https://aclanthology.org/2024.tacl-1.9/

    doi:10.1162/tacl 2024

  11. [11]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating llms as agents. InInternational Conference on Learning Representatio...

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.03688 2024

  12. [12]

    G-eval: Nlg evaluation using gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634, 2023

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment.arXiv preprint arXiv:2303.16634, 2023. doi: 1 0 . 4 8 5 5 0 / a r X i v . 2 3 0 3 . 1 6 6 3 4. URL https://arxiv.org/abs/2303.16634

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.16634 2023

  13. [13]

    Toolformer: Lan- guage models can teach themselves to use tools.arXiv preprint arXiv:2302.04761, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess ` ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools.arXiv preprint arXiv:2302.04761, 2023. doi: 10.48550/arXiv.2302.04761. URL https://arxiv.org/abs/2302.04761

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.04761 2023

  14. [14]

    Artificial intelligence risk management framework (ai rmf 1.0)

    Elham Tabassi. Artificial intelligence risk management framework (ai rmf 1.0). Technical Report NIST AI 100-1, National Institute of Standards and Technology, 2023. URL https://doi.org/10.6028/NIST.AI.100-1

    doi:10.6028/nist.ai.100-1 2023

  15. [15]

    Congress

    U.S. Congress. Health insurance portability and accountability act of 1996, 1996. URL https://www.congress.gov/bill/104th-congress/house-bill/3103. Public Law 104-191

    1996

  16. [16]

    Congress

    U.S. Congress. Gramm-leach-bliley act, 1999. URL https://www.congress.gov/bill/10 6th-congress/senate-bill/900. Public Law 106-102

    1999

  17. [17]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023. doi: 10.48550/arXiv.2210.03629. URL https://arxiv.org/abs/2210.03629. 23

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.03629 2023

  18. [18]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InAdvances in Neural Information Processing Systems Datasets and Benchmarks Track, 2023. doi: 10.48550/arXiv.2306.05685. ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.05685 2023

  19. [19]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xi- anyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neu- big

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xi- anyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neu- big. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023. doi: 1 0 . 4 8 5 5 0 / a r X i v . 2 3 0 7 . 1 3 8 5 4. URL https://arxiv.org/abs/2307....

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.13854 2023

  20. [2026]

    URL https://arxiv.org/abs/2607.14275

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.