Pith. sign in

REVIEW 3 major objections 3 minor 41 references

Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that the validity of any agentic benchmark score is bounded by the product of its three pipeline layers, so reported scores without per-stage validity evidence cannot be read as capability measurements.

desk verdict A useful, honest synthesis of evaluation validity that deserves serious review; the central compounding idea holds up, but the headline 5% claim is built on one kappa value and needs direct benchmark-level validation before it is quoted as a floor. read the letter →

arxiv 2608.00794 v3 pith:FJVMDCI5 submitted 2026-08-01 cs.AI

William Caban This is my paper · ORCID
classification cs.AI
keywords agenticAIevaluationconstructvalidityinter-raterreliabilitybenchmarkLLM-as-a-judgesimulationcalibrationcompoundingerrorpipelines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the validity of an agentic AI benchmark score is bounded by the product of the validity of its three pipeline stages: task generation, human-simulator calibration, and automated judgment. Using published measurements, it argues that a pipeline retaining 70% validity at each stage keeps at most 34% of valid signal against the intended construct. A structured scan of 55 agentic evaluation papers finds about 82% use missing, mismatched, or structurally invalid inter-rater reliability metrics, consistent with systematic collapse at the judgment layer. The paper claims that benchmarks without human IRR validation retain less than 5% of valid signal because the automated-grader agreement floor is near chance, so reported scores cannot be interpreted as capability measurements unless each stage is validated. A sympathetic reader would take the paper as a call to report validity evidence, not as a claim that all agentic evaluation is useless.

What carries the argument

The carrying object is the multiplicative validity bound $V_{\mathrm{total}} \le V_1 \times V_2 \times V_3$, defined over an evaluation pipeline's three layers: task generation validity $V_1$ (one minus total variation distance between generated and ideal task distributions), simulation calibration validity $V_2$ (the ICC(A,1) agreement between simulated and real user outcomes, estimated per demographic group), and judgment validity $V_3$ (the correct inter-rater reliability metric for the rating design). The bound does the work of converting an intuition about compounding imperfection into a quantitative diagnostic: it shows that moderate losses at each stage multiply into severe total loss, that the judgment layer is typically the binding constraint, and that reporting the bound alongside a score disciplines what the score can mean.

What would settle it

Run a direct grader-versus-human agreement study on a deterministic benchmark such as SWE-bench or OSWorld, comparing automated grading against two fixed human annotators on a matched sample; if the resulting inter-rater reliability is substantially above 0.05, the Table 3 conclusion that such benchmarks retain under 5% of valid signal would not hold, though the multiplicative model could still survive with a higher measured $V_3$.

Watch

Extended reading notes

Core claim

The central discovery claim is the three-layer compounding validity model $V_{\mathrm{total}} \le V_1 \times V_2 \times V_3$, where $V_1$ is task generation validity, $V_2$ is simulation calibration validity measured as ICC(A,1) between simulated and real outcomes, and $V_3$ is judgment validity measured by the structurally correct IRR metric. Applying published estimates, the paper derives that major benchmarks without human IRR fall below $V_{\mathrm{total}} = 0.05$, and that correcting $V_3$ alone via rigorous human annotation raises the bound roughly sixteen-fold, as in the WebArena Verified case. The paper frames this bound as a conceptual model rather than a proved theorem, but argues that the multiplicative degradation and its qualitative consequences are robust.

Load-bearing premise

The load-bearing premise is that the empirical $V_3$ floor of $\kappa = 0.049$ from a single study of substring-based automated grading generalizes to all benchmarks without human IRR, including deterministic graders such as SWE-bench test-suite execution; the paper itself marks this generalization as an open empirical question in Section 10.4.

Editorial extensions

If this is right

  • Any benchmark score reported without per-stage validity evidence cannot be interpreted as a capability measurement; the burden shifts to evaluating the evaluator.
  • The judgment layer is the binding constraint: investing in rigorous human IRR yields roughly a sixteen-fold increase in valid signal compared with tightening task generation or simulation alone, per the WebArena Verified comparison.
  • Automated grading without human validation should be presumed near-chance in validity until measured, not presumed valid.
  • Simulation-based evaluations need calibration validation stratified by user demographics; otherwise they can be high-reliability measurements of a distorted signal.
  • Reporting $V_{\mathrm{total}}$ alongside every benchmark result provides an interpretability threshold: below 0.50 for consequential decisions, below 0.30 for pipeline redesign.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same multiplicative argument extends to any multi-stage AI measurement pipeline, such as LLM-based hiring screens or medical triage tools, where a single rater-validation floor can dominate the product; this is testable by applying the same equation to those domains.
  • The 82 percent misuse prevalence comes from a purposive 55-paper sample; a random-sample replication would sharpen or soften that claim, and the paper's published coding table makes such a replication feasible.
  • If deterministic graders are later shown to agree with humans at moderate levels, the paper's quantitative collapse becomes milder, but its qualitative conclusion that validity must be reported stage by stage would survive.
  • Regulatory pressure to require validity evidence for safety claims would likely convert the paper's prescriptions from best practice into auditable reporting fields, though the paper itself only gestures at this consequence.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes a three-layer compounding validity model for agentic AI evaluation, V_total <= V_1 x V_2 x V_3, where V_1 measures task-generation validity, V_2 measures simulation calibration, and V_3 measures judgment validity. The authors report a structured survey of 55 published agentic evaluation papers, finding that roughly 82% use structurally mismatched, incomplete, or absent IRR metrics, and they apply the model to six benchmarks to argue that all automated-graded benchmarks without human IRR retain less than 5% of valid signal. The paper derives eight prescriptions, including IRR metric selection rules and domain-stratified reliability thresholds, and it presents the model as a conceptual framework rather than a proved theorem, with a sensitivity analysis and an explicit threats-to-validity section.

Significance. The paper is a useful and timely contribution to the methodology of agentic AI evaluation. Its central qualitative insight — that validity losses at different pipeline stages compound multiplicatively rather than additively — is important, and the paper connects a long-standing psychometric literature to an evaluation practice that has largely ignored construct validity. The openly deposited data and code for the second-rater IRR validation, the transparent coding instrument in Appendix A, and the explicit sensitivity analysis in Table 4 are concrete strengths that make the empirical parts independently checkable. The model is also honestly framed in places as a conceptual model, not a theorem, which is refreshing in a literature that often overclaims formal status. If the quantitative headline claims are brought into line with the actual evidentiary base, the framework and prescriptions would be a valuable resource for benchmark authors and practitioners.

major comments (3)
  1. [Section 7.2, Table 3, and Section 10.4] The central quantitative conclusion — that every benchmark without human IRR has V_total below 0.05 regardless of V_1 and V_2 — rests entirely on assigning V_3 = 0.05 to all such benchmarks, based on a single study (Gurram [25]) of substring-based automated grading on one task type. The paper itself states in Section 10.4 that whether this generalizes to deterministic graders such as SWE-bench test-suite execution is an open empirical question. Because test-suite execution is categorically different from LLM substring matching, the floor cannot be treated as a universal value. Table 4 partially mitigates this by showing that V_total remains below 0.30 for two benchmarks under V_3 = 0.50, but the table excludes plausible V_3 values above 0.50 for deterministic graders; with V_3 = 0.8, SWE-bench would produce V_total about 0.47, which is above the paper's own 0.30 redesign threshold. Please reframe Table 3 as a lower-bound scenario, add benchmark-specific V_3 estimates or an explicit 'unknown' category for deterministic graders, and qualify the Section 7.2 finding accordingly.
  2. [Section 7.2, Table 3, vs. Appendix B.1–B.2] The V_1 entries for WebArena (V_1 = 0.97, attributed to LLM judge misestimation), OSWorld (V_1 = 0.72, attributed to a stale-CSS environment effect), and SWE-bench (V_1 = 0.59, attributed to augmented test cases changing rankings) appear to be drawn from outcome-validity or environment-fidelity findings in Zhu et al. [15], not from total variation distance between generated and ideal task distributions as V_1 is defined in Appendix B.1–B.2. This layer conflation affects the specific numerical values in Table 3. The qualitative multiplicative argument is not destroyed, but the numbers should either be re-derived from task-generation-validity evidence or explicitly relabeled as approximate layer estimates.
  3. [Section 7.1 and Appendix B.3] Equation (1) is introduced as 'bounded above by' and used as the basis for Section 9's interpretation thresholds and Table 3's numeric bounds, but Appendix B.3 states that the inequality is a conceptual model, not a proved mathematical proposition. These statements are in tension at the point of use. Please state clearly at the introduction of Eq. (1) that this is a modeling assumption rather than a proven theorem, so that readers do not treat 'V_total <= V_1 V_2 V_3' as a mathematically guaranteed bound. The paper's qualitative conclusions survive this clarification, but the epistemic status of the bound should be explicit where the bound is first applied.
minor comments (3)
  1. [Table A.5 and Appendix A.3] Table A.5 lists category I as having zero correct papers, but the footnote and Appendix A.3 say Jafari et al. [35] used the correct metrics and reported catastrophically low values (ICC = 0.087–0.295, alpha = -0.203). This row is internally contradictory; either count it as 'correct metric with failing reliability' or split the category to avoid the ambiguity.
  2. [Section 10.4 and Table 4] The sensitivity analysis in Table 4 stops at V_3 = 0.50, which is presented as the optimistic scenario. For deterministic graders such as test-suite execution, V_3 values above 0.50 are plausible, and the robustness claim would be stronger if the table included a V_3 = 0.8 row.
  3. [Section 6.2 and abstract] The abstract and Section 6.2 report 'approximately 82%' IRR misuse, but the coding was performed by a single author with a 20-paper four-rater validation. The sample is also explicitly purposive. Given these caveats, presenting the percentage in the abstract without also saying 'in a purposive sample of 55 papers' slightly overstates the precision of the estimate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the model's inputs are external psychometric measurements, and the one self-referential element is explicitly scoped as an instrument-clarity check.

full rationale

The paper's derivation chain is self-contained against external evidence rather than circular. Equation (1), V_total <= V1 x V2 x V3, is defined in Section 7.1 from three independently specified constructs, and Appendix B.3 explicitly labels it a conceptual model rather than a proved theorem, so it does not smuggle in its conclusion. The values in Table 3 are taken from external published measurements (Zhu et al. for V1, Seshadri et al. for V2, Gurram for the V3 floor, El Hattami et al. for WebArena Verified), not fitted to the paper's own outputs or to the target claims. The V3=0.05 floor is a single external study, and Section 10.4 explicitly states that whether it generalizes to deterministic automated graders is an open empirical question; Table 4 then bounds the sensitivity of the qualitative conclusion. This is an external-validity limitation, not a circular step. The only self-referential element is the use of LLM raters to validate the coding of a paper critical of LLM raters, but Section 10.2 addresses this directly by limiting the claim to instrument clarity (whether independent agents can apply the same decision rules) rather than rater validity, and the four-way Krippendorff alpha is reported as a consistency check, not as evidence for the paper's substantive conclusions. No load-bearing self-citation or fitted-parameter-renamed-as-prediction is present. The central claims therefore do not reduce to their own inputs by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 3 invented entities

The model rests on the multiplicative bound, the independence assumption, and several empirical approximations applied across benchmarks. The paper is transparent about most of these, which is a strength, but the V3 floor and the reliability-validity conflation are load-bearing and under-justified.

free parameters (4)
  • V3 automated grader floor = 0.05 (kappa = 0.049)
    Taken from Gurram (2026) tool-use study and applied to SWE-bench, OSWorld, WebArena and tau-bench automated grading. The paper flags generalization as an open empirical question (Section 10.4).
  • V2 ECE conversion factor = V2 ~ 1 - ECE/100
    Maps Expected Calibration Error to a validity coefficient. The paper notes the equivalence holds only when ECE measures uniform miscalibration (Section 10.1).
  • Reliability thresholds alpha >= 0.70 and >= 0.80 = 0.70, 0.80
    Proposed thresholds for safety and deployment-gating evaluations. The paper labels these as 'we propose' (Section 8, Prescription 5), with no empirical calibration.
  • V_total interpretation thresholds = 0.30, 0.50
    Chosen cutoffs for pipeline redesign and deployment decisions (Section 9); no external basis provided.
assumptions (5)
  • ad hoc to paper V_total <= V1 x V2 x V3
    Central bound stated as a conceptual model, explicitly not a proved theorem (Section 7.1, Appendix B.3).
  • domain assumption Layer failures are independent
    The bound holds under independence; positive correlation lowers V_total (Section 7.1, B.4). The paper notes this is optimistic.
  • domain assumption The V3 floor generalizes to all automated graders
    A single kappa = 0.049 measurement on one task type is applied to SWE-bench test-suite execution, OSWorld, and WebArena (Section 7.2; acknowledged in Section 10.4).
  • domain assumption Purposive sample reflects prevalence of IRR misuse
    The 55-paper scan is purposive, not systematic; selection bias is acknowledged (Sections 6.2, 10.2).
  • domain assumption LLM raters can reliably apply the coding instrument
    Used to validate the survey coding; the paper argues alpha = 0.89 shows instrument clarity rather than rater validity (Section 10.2).
invented entities (3)
  • Ideal construct distribution D_C*
    purpose: Reference distribution for defining V1 as the complement of total variation distance
    Described as a theoretical object that is not directly observable (Section 7.1).
  • Ideal scoring function J*
    purpose: Reference for mapping agent behavior to true capability, underpinning V3
    Introduced in Appendix B.1 as a theoretical benchmark; no operational definition outside the model.
  • V_total and V1, V2, V3 latent validity constructs
    purpose: Decompose pipeline validity into three layers for multiplicative combination
    New measurement constructs; each is estimated via proxies (TV distance, ICC, IRR) rather than directly observed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation." pith.science (2026). https://pith.science/paper/FJVMDCI5

@misc{pith2026260800794,
  author       = {Pith},
  title        = {Pith review of: Measurement Without Validity: The Compounding Reliability Problem in Agentic AI Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FJVMDCI5}},
  note         = {Machine review of arXiv:2608.00794}
}
read the original abstract

Agentic AI evaluation pipelines produce benchmark scores that justify deployment decisions, safety certifications, and regulatory compliance claims. No formal framework has yet characterized how validity degrades across the stages of these pipelines. We present a three-layer compounding validity model, V_total <= V_1 x V_2 x V_3, that captures multiplicative degradation across task generation (V_1), human-simulator calibration (V_2), and automated judgment (V_3). Under empirically grounded estimates, a pipeline retaining 70% validity at each stage is at most 34% valid against the intended construct (range 0.22--0.54). We validate the model against a structured survey of 55 published agentic evaluation papers, finding that approximately 82% apply structurally mismatched, incomplete, or absent inter-rater reliability (IRR) metrics---a pattern consistent with systematic V_3 collapse. We further identify empirical evidence of V_1 failures (task validity flaws in 7 of 10 popular benchmarks) and V_2 miscalibration (up to 9 percentage points inter-simulator variance, with systematic demographic disparities for non-Standard American English speakers). We derive eight prescriptions grounded in psychometric science and domain-stratified reliability thresholds (ICC>=0.70; alpha >= 0.67/0.70/0.80 by consequence level) that practitioners and benchmark authors can apply immediately. The framework provides a tractable knowledge-based tool for diagnosing and correcting evaluation pipeline validity before deployment decisions are made.

Figures

Figures reproduced from arXiv: 2608.00794 by the authors.

Figure 1
Figure 1. IRR metric selection. Left: decision tree based on rater design, matrix com [PITH_FULL_IMAGE:figures/full_fig_p018_1.png] view at source ↗
Figure 1
Figure 1. IRR metric selection. Left: decision tree based on rater design, matrix com [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Three-layer agentic evaluation pipeline with validity estimates. Each layer [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗
Figures from the paper (1 more)
Figure 2
Figure 2. Figure 2: Three-layer agentic evaluation pipeline with validity estimates. Each layer [PITH_FULL_IMAGE:figures/full_fig_p014_2.png]

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 3 canonical work pages

  1. [25]

    Gurram, Evaluating tool-using language agents: Judge reliability, propagation cascades, and runtime mitigation in AgentProp-Bench, arXiv preprint arXiv:2604.16706, 2026

    B. Gurram, Evaluating tool-using language agents: Judge reliability, propagation cascades, and runtime mitigation in AgentProp-Bench, arXiv preprint arXiv:2604.16706, 2026. Under review

    arXiv 2026

  2. [15]

    Y. Zhu, T. Jin, Y. Pruksachatkun, A. Zhang, S. Liu, S. Cui, S. Kapoor, S. Longpre, K. Meng, R. Weiss, et al., Establishing best practices for building rigorous agentic benchmarks, arXiv preprint arXiv:2507.02825, 2025

    arXiv 2025

  3. [1]

    L. J. Cronbach, P. E. Meehl, Construct validity in psychological tests, Psychological Bulletin 52 (1955) 281–302. doi:10.1037/h0041570

    doi:10.1037/h0041570 1955

  4. [2]

    Y. L. Liu, S. L. Blodgett, J. C. K. Cheung, Q. V. Liao, A. Olteanu, Z. Xiao, ECBD: Evidence-centered benchmark design for NLP, in: Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), 2024, pp. 16349–16365. doi:10.18653/v1/2024.acl-long.861

    doi:10.18653/v1/2024.acl-long.861 2024

  5. [3]

    K. J. Meimandi, G. Aránguiz-Dias, G. R. Kim, L. Saadeddin, A. Grif- fith, M. J. Kochenderfer, The measurement imbalance in agentic AI evaluation undermines industry productivity claims, arXiv preprint arXiv:2506.02064, 2025

    arXiv 2025

  6. [4]

    Prandi, V

    M. Prandi, V. Suriani, F. Pierucci, M. Galisai, D. Nardi, P. Bisconti, Bench-2-CoP: Can we trust benchmarking for EU AI compliance?, arXiv preprint arXiv:2508.05464, 2025

    arXiv 2025

  7. [5]

    Cohen, A coefficient of agreement for nominal scales, Educa- tional and Psychological Measurement 20 (1960) 37–46

    J. Cohen, A coefficient of agreement for nominal scales, Educa- tional and Psychological Measurement 20 (1960) 37–46. doi:10.1177/ 001316446002000104

    1960

  8. [6]

    J. L. Fleiss, Measuring nominal scale agreement among many raters, Psychological Bulletin 76 (1971) 378–382. doi:10.1037/h0031619

    doi:10.1037/h0031619 1971

Show all 41 references
  1. [7]

    Krippendorff, Content Analysis: An Introduction to Its Methodology, 2nd ed., Sage Publications, Thousand Oaks, CA, 2004

    K. Krippendorff, Content Analysis: An Introduction to Its Methodology, 2nd ed., Sage Publications, Thousand Oaks, CA, 2004

    2004

  2. [8]

    J. R. Landis, G. G. Koch, The measurement of observer agreement for categorical data, Biometrics 33 (1977) 159–174. doi:10.2307/2529310

    1977 doi

  3. [9]

    Mitchell, S

    M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, T. Gebru, Model cards for model reporting, in: Proceedings of the Conference on Fairness, Accountability, and Transparency, 2019, pp. 220–229. doi:10.1145/3287560.3287596

    2019

  4. [10]

    R. Dhar, D. S. Villegas, A. Karamolegkou, A. Schiavone, Y. Yuan, X. Chen, J. Li, S. Frank, L. De Grazia, M. Swain, et al., EvalCards: A framework for standardized evaluation reporting, arXiv preprint arXiv:2511.21695, 2025. 26

    2025

  5. [11]

    Hayes-Roth, D

    F. Hayes-Roth, D. A. Waterman, D. B. Lenat, Building Expert Systems, Addison-Wesley, Reading, MA, 1983

    1983

  6. [12]

    B. G. Buchanan, E. H. Shortliffe (Eds.), Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project, Addison-Wesley, Reading, MA, 1984

    1984

  7. [13]

    A. Z. Jacobs, H. Wallach, Measurement and fairness, in: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021, pp. 375–385. doi:10.1145/3442188.3445901

    2021

  8. [14]

    James, Counting on consensus: Selecting the right inter-annotator agreement metric for NLP annotation and evaluation, arXiv preprint arXiv:2603.06865, 2026

    J. James, Counting on consensus: Selecting the right inter-annotator agreement metric for NLP annotation and evaluation, arXiv preprint arXiv:2603.06865, 2026

    2026

  9. [16]

    Seshadri, S

    P. Seshadri, S. Cahyawijaya, A. Odumakinde, S. Singh, S. Goldfarb- Tarrant, Lost in simulation: LLM-simulated users are unreliable proxies for human users in agentic evaluations, in: The Fourteenth International Conference on Learning Representations, 2026. ArXiv:2601.17087

    2026

  10. [17]

    Zhuge, C

    M. Zhuge, C. Zhao, D. R. Ashley, W. Wang, D. Khizbullin, Y. Xiong, Z. Liu, E. Chang, R. Krishnamoorthi, Y. Tian, Y. Shi, V. Chandra, J. Schmidhuber, Agent-as-a-judge: Evaluate agents with agents, in: Proceedings of the 42nd International Conference on Machine Learning,

  11. [18]

    Models used: NVIDIA Nemotron-Ultra- 550B, Google Gemma-4-31B, Alibaba Qwen3-80B-A3B

    OpenRouter, Openrouter: A unified api for large language models, https://openrouter.ai, 2026. Models used: NVIDIA Nemotron-Ultra- 550B, Google Gemma-4-31B, Alibaba Qwen3-80B-A3B

    2026

  12. [19]

    Yao, et al.,τ-bench: A benchmark for tool-agent-user interaction in real-world domains, arXiv preprint arXiv:2406.12045, 2024

    S. Yao, et al.,τ-bench: A benchmark for tool-agent-user interaction in real-world domains, arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  13. [20]

    Xie, et al., OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments, arXiv preprint arXiv:2404.07972, 2024

    T. Xie, et al., OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments, arXiv preprint arXiv:2404.07972, 2024. 27

    2024 arXiv

  14. [21]

    C. E. Jiménez, et al., SWE-bench: Can language models resolve real- world GitHub issues?, arXiv preprint arXiv:2310.06770, 2024

    2024 arXiv

  15. [22]

    Mialon, et al., GAIA: A benchmark for general AI assistants, arXiv preprint arXiv:2311.12983, 2024

    G. Mialon, et al., GAIA: A benchmark for general AI assistants, arXiv preprint arXiv:2311.12983, 2024

    2024 arXiv

  16. [23]

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, Y. Bisk, D. Fried, U. Alon, G. Neubig, WebArena: A realistic web environment for building autonomous agents, in: The Twelfth International Conference on Learning Representations, 2024. URL:https://openreview.net/...

    2024

  17. [24]

    Barres, H

    V. Barres, H. Dong, S. Ray, X. Si, K. Narasimhan,τ 2-bench: Evaluating conversational agents in a dual-control environment, arXiv preprint arXiv:2506.07982, 2025

    2025 arXiv

  18. [26]

    El Hattami, M

    A. El Hattami, M. Thakkar, N. Chapados, C. Pal, WebArena Ver- ified: Reliable evaluation for web agents, in: Scaling Environments for Agents Workshop, NeurIPS 2025, 2025. URL:https://openreview. net/forum?id=94tlGxmqkN

    2025

  19. [27]

    S. Fan, X. Ye, Y. Huo, Z.-Y. Chen, Y. Guo, S. Yang, W. Yang, S. Ye, J. Chen, H. Chen, X. Cong, Y. Lin, AgentProcessBench: Diagnosing step- levelprocessqualityintool-usingagents, arXivpreprintarXiv:2603.14465,

    arXiv

  20. [28]

    S. Han, G. Titericz Junior, T. Balough, W. Zhou, Judge’s verdict: A com- prehensive analysis of LLM judge capability through human agreement, arXiv preprint arXiv:2510.09738, 2025

    2025

  21. [29]

    Movva, P

    R. Movva, P. W. Koh, E. Pierson, Annotation alignment: Comparing LLM and human annotations of conversational safety, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. ArXiv:2406.06369

    2024 arXiv

  22. [30]

    Chiang, L

    W.-L. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, I. Stoica, Chatbot arena: 28 An open platform for evaluating LLMs by human preference, arXiv preprint arXiv:2403.04132, 2024

    2024 arXiv

  23. [31]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., Training language models to follow instructions with human feedback, in: Advances in Neural Information Processing Systems, volume 35, 2022, pp. 27730–27744. ArXi...

    2022 arXiv

  24. [32]

    Faisal, M

    F. Faisal, M. M. Rahman, A. Anastasopoulos, Dialectal toxicity detection: Evaluating LLM-as-a-judge consistency across language varieties, in: Findings of the Association for Computational Linguistics: EMNLP 2025, 2025. ArXiv:2411.10954; citation key reflects arXiv preprint da...

    2025 arXiv

  25. [33]

    Spiliopoulou, R

    E. Spiliopoulou, R. Fogliato, H. Burnsky, T. Soliman, J. Ma, G. Horwood, M. Ballesteros, Play favorites: A statistical method to measure self-bias in LLM-as-a-judge, arXiv preprint arXiv:2508.06709, 2025

    2025 arXiv

  26. [34]

    Wataoka, T

    K. Wataoka, T. Takahashi, R. Ri, Self-preference bias in LLM-as-a-judge, in: NeurIPS 2024 Safe Generative AI Workshop, 2024. ArXiv:2410.21819

    2024 arXiv

  27. [35]

    Jafari, P

    K. Jafari, P. U. N. Rust, D. Eddy, R. Fraser, N. Vasan, D. Djordjevic, A. Dadlani, M. Lamparth, E. Kim, M. J. Kochenderfer, Expert evaluation and the limits of human feedback in mental health AI safety testing, arXiv preprint arXiv:2601.18061, 2026. Under review

    2026 arXiv

  28. [36]

    Wohlin, P

    C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, A. Wesslén, Experimentation in Software Engineering, Springer, 2012. doi:10.1007/ 978-3-642-29044-2

    2012

  29. [37]

    Jedlitschka, M

    A. Jedlitschka, M. Ciolkowski, D. Pfahl, Reporting experiments in software engineering, in: F. Shull, J. Singer, D. I. Sjøberg (Eds.), Guide to Advanced Empirical Software Engineering, Springer, 2008, pp. 201–228. doi:10.1007/978-1-84800-044-5_8. Appendix A. Literature Scan: C...

    2008 doi

  30. [38]

    IRR metric reported: Cohen’sκ, Fleiss’ κ, Krippendorff’s α, ICC (form specified or unspecified), percentage agreement only, or no metric reported

  31. [39]

    Rater design: number of raters; whether rater identity was fixed across items or varied (rotating pool, crowdsourced, or multiple independent runs of the same model)

  32. [40]

    Measurement scale: binary, nominal (unordered categories), ordinal or ternary (ordered but discrete), or continuous

  33. [41]

    Correct” = structurally valid metric with explicit rationale. “Failure mode

    Structural validity: whether the reported metric’s structural assump- tions were satisfied by the rater design and measurement scale. Coding applied the following rules: Cohen’sκ for > 2raters or rotating identity → mismatch; Fleiss’ κ for ordinal or continuous scale→ partial ...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.