Pith.

Artificial Intelligence · arXiv:2611.13336 v1 · submitted 9 Nov 2026 · 85 references

Veritas: A Verifier Generation Harness for Agent Tasks

Daoan Zhang, Qinbo Bai, Dehai Min, Xingyu Mu, Ziyi Chen, Jianing Yin

The paper says

“The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round.”

more of the abstract

However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically. Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. On the two pairwise benchmarks they are at or near the best. They also agree better than those of the vendor's own harness for two of three other model families. Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 11 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Veritas is a careful, transparent study of automatic verifier generation for agent tasks. The central claim is real but the two largest Table 2 rows are scored on data that shaped the harness, which caps what they can establish. The held-out terminal benchmarks and the mechanism-transfer results are the strongest evidence. This line is wrong

5 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 85 references say.

The paper says, at §4.1, Table 2, §3.4

The review did not quote this passage.

Pith says: Objection 1 of 5 · major

The two largest rows of Table 2 (τ²-bench with 58 tasks, SWE-bench with 138 tasks) are scored on the fit+validation tasks whose official labels were used to accept the final self-improvement edits (Section 3.4, Appendix A.3). The held-out test splits (27 τ² tasks, 60 SWE tasks) are never reported in Tables 2–5.

more of objection 1

This means the headline comparison for these two benchmarks is not independent evidence; the measured agreement may be optimistically biased by selection. The authors acknowledge this limitation transparently (Section 4.1: 'both terminal benchmarks were added after the mechanisms ... were frozen and were never used to tune them, so their rows are the closest we have to held-out results'), but the central claim 'on all four execution-scored benchmarks' rests on only two genuinely held-out benchmarks (Terminal-Bench, TB-Science) for independent support. The paper should either report the held-out test results for τ² and SWE, or explicitly qualify the claim to reflect that the two largest benchmarks are not held out.

The paper says, at §4.1, Table 2, Appendix C.3

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.73):

“Each configuration is run once (Appendix C.3 reports repeats), and the benchmarks scored by execution have 33 to 138 tasks.”

Pith says: Objection 2 of 5 · major

No confidence intervals or measures of uncertainty are reported for the κ* values in Table 2. Given the small number of tasks (33–138) and the fact that each configuration is run only once (Limitations, [b216]), the reported differences (e.g., 0.720 vs 0.693 on SWE-bench, margin 0.027) may not be statistically significant.

more of objection 2

Appendix C.3 reports that two runs of the same configuration on τ²-bench differ by 0.040 (Veritas) and 0.083 (plain Claude Code), indicating substantial run-to-run variation. The paper should provide confidence intervals or statistical tests for the main comparisons, or at least discuss the impact of run-to-run variability on the reported margins.

The paper says, at §4.4, Table 5

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.86):

“For other model families we compare only with the vendor’s own harness.”

Pith says: Objection 3 of 5 · major

The comparison with other model families (GPT-5.6-sol, Qwen3.7-Plus, Gemini 3.5 Flash) uses only the vendor's own harness as the baseline, not all seven plain harnesses. For GPT-5.6-sol, Veritas is worse on all three reported execution benchmarks (SWE, TB, TB-Sci), and τ²-bench is not reported because 'GPT-5.6-sol delivered too few verifiers on τ²-bench to report'.

more of objection 3

This weakens the generality claim 'Veritas still outperforms the vendor's own harness for two of three other model families' (Section 1), because the one family where it does not outperform is the most capable model (GPT-5.6-sol). The paper should discuss this limitation more prominently and consider whether the advantage of Veritas is model-dependent.

The paper says, at §4.3, Table 4

The review did not quote this passage.

Pith says: Objection 4 of 5 · major

The ablation study removes one mechanism at a time, but the results are reported on only four of the six benchmarks (τ², SWE, LLMBar, JudgeBench), and for SWE-bench the -R condition is identical to Full because 'R is never activated' (evidence does not include trajectory). The ablation does not include Terminal-Bench or TB-Science, which are the only held-out benchmarks. The paper should explain why these benchmarks were omitted from the ablation, and ideally include them to strengthen the analysis.

The paper says, at §4.5, Appendix C.1

The review did not quote this passage.

Pith says: Objection 5 of 5 · major

The analysis of errors by candidate kind (Table 9) shows that Veritas accepts only 0.45 of reference solutions on Terminal-Bench (true positive rate). This is a very low recall for correct candidates. While the paper notes that 'false rejection, not false acceptance, is the dominant error', this level of false rejection would be problematic if the verifier were used in a self-improvement loop, because many correct attempts would be rejected.

more of objection 5

The paper should discuss whether this level of false rejection is acceptable for practical use, and whether the mechanisms could be adjusted to improve recall without sacrificing precision.

8 minor objections

The paper says, at §1, Figure 1

Pith says: minor objection 1 of 8

Figure 1 caption states 'κ* is Cohen's κ against the official labels, with a missing verdict counted as an error.' This definition of κ* should be stated explicitly in the main text (it is defined in §4.1 but the figure appears earlier). Consider adding a forward reference.

The paper says, at §3.3

Pith says: minor objection 2 of 8

The description of the domain rule library (R) states it has '19,603 entries' from six domains. It would be helpful to clarify whether these entries are all used in the evaluation, or only a subset relevant to the benchmarks. The paper notes that 'only the rules it looks up reach the verifier', but the total size may raise concerns about the library being a source of task-specific information.

show 6 more

The paper says, at §4.1, Metrics

Pith says: minor objection 3 of 8

The definition of κ* uses 'every missing verdict ... replaced by the verdict opposite to the label'. This is a reasonable choice but penalizes non-delivery heavily. The paper should note that this choice affects the ranking, and report the judged-subset κ as a secondary metric more consistently (it is mentioned but not always reported).

The paper says, at §4.2, Table 2

Pith says: minor objection 4 of 8

The table shows that on LLMBar, Veritas (0.920) ties Codex CLI (0.920) and is below Terminus 2 (0.933). The text says 'at or near the best' which is accurate, but the claim in the abstract 'on the two pairwise benchmarks they are at or near the best' could be more precise.

The paper says, at Appendix A.3, Table 6

Pith says: minor objection 5 of 8

The self-improvement loop description is detailed, but it would be helpful to state explicitly how many of the 32 rounds were proposed by the model vs. the researchers, and whether any of the accepted edits (rounds 3, 4, 15, 25) were proposed by the model. The text says 'rounds 3 and 4 added the weakest-comparison and output self-check principles to H under the rule of the first phase' but does not state who proposed them.

The paper says, at Appendix B.2, Table 7

Pith says: minor objection 6 of 8

The table shows that many 'cut' and 'linedrop' candidates are accepted by the official tests because they are 'identical to the oracle in effect'. This is a useful caveat, but the paper should clarify whether these candidates are still used in the evaluation (they are, as certified candidates with their official label).

The paper says, at General

Pith says: minor objection 7 of 8

The paper uses the term 'κ*' without explaining the asterisk. Consider using a more descriptive notation or defining it explicitly.

The paper says, at §5, Conclusion

Pith says: minor objection 8 of 8

The conclusion states 'delivery can matter more than judgment' and 'more information can hurt'. These are interesting findings but are not quantified in the conclusion. Consider adding a brief summary of the evidence.

What would settle it

Recompute the τ²-bench and SWE-bench rows of Table 2 on the held-out test splits defined in Appendix A.3 (27 τ² tasks, 60 SWE tasks), which are already gated and certified with candidate suites under Appendix B. Run Veritas and all seven plain harnesses (GLM-5.2, same contracts and κ* metric), repeating the leading configuration three times, and compare each leading margin with the Appendix C.3 noise band (0.040–0.083 on τ²; ≈0.021 between the two SWE Veritas runs). If Veritas still leads on both test splits by margins outside that band, the selection concern is settled and the 'all four' claim stands on independent data. If the margins fall inside the band or invert, the claim should be narrowed to the frozen-terminal benchmarks and the two largest Table 2 rows relabelled as tuning-set evidence.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 11 Nov 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 59 s · 32,551 tokens read, 18,699 written
  3. Editor wrote the verdict above 10 s · 33,693 tokens read, 666 written
  4. Referee major revision: 5 major objections, 8 minor 26 s · 32,975 tokens read, 2,077 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper is worth your time. It asks a concrete question: how much does the harness, rather than the model, determine the quality of automatically generated verifiers for agent tasks? The answer is a clear "a lot" — plain harnesses range from κ* 0.16 to 0.73 on τ²-bench with the same model. The three mechanisms (evidence header, domain rule library, authoring principles) are well-motivated and the ablation shows each matters where it targets. The transfer results (Table 3) are genuinely informative: adding the mechanisms unchanged to other harnesses helps most where they are weakest.

The paper is unusually transparent. The limitations section is honest, the appendices are thorough, and the certification gates for labels are a real strength. The held-out terminal benchmarks (Terminal-Bench, TB-Science) were added after the mechanisms were frozen, and Veritas leads there by 0.078 and 0.074 in κ*. Those rows are the cleanest evidence.

The soft spot is the one the stress-test flags. The two largest rows of Table 2 (τ²-bench, 58 tasks; SWE-bench, 138 tasks) are scored on the same fit+validation tasks whose official labels were used by the self-improvement loop to accept edits to the authoring principles (Section 3.4). The held-out test splits (27 τ² tasks, 60 SWE tasks) never appear in Tables 2–5. The authors state this plainly (Section 4.1 calls the terminal rows "the closest we have to held-out results"), so it is a transparently scoped limitation, but it means the "all four execution-scored benchmarks" claim overweights two in-sample rows covering 196 of 268 tasks. The run-to-run variation (Appendix C.3: 0.040–0.083 spread on τ²) is the same order as the leading margins (0.061, 0.027), so even the ranking is not shown to be stable.

These issues are addressable: rerun on test splits, add repeats, release artifacts. The paper does not ship code or the rule library, which limits reproducibility.

Who this is for: anyone working on automatic evaluation, verifier generation, or self-improvement loops for agents. It deserves a serious referee — the question is important, the design is thoughtful, and the limitations are stated rather than hidden. I would accept a peer-review invitation.

Recommendation: engage with it, but treat the τ² and SWE rows as suggestive rather than conclusive until test-split results appear.

The strongest objection

The central claim — 'agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks' [b5] — is carried by Table 2, whose two largest rows are scored on the data that shaped the harness. Section 3.4 splits τ²-bench and SWE-bench into fit/validation/test; the loop ran 32 rounds on fit+validation and accepted rounds 3–4, adding the weakest-comparison and output self-check principles to H because validation agreement improved, under a phase-1 rule that had 'no significance test and no confirmation run' (Appendix A.3). Section 4.1 then reports Veritas only on those 58 fit+validation τ² tasks and 138 SWE tasks. The 27 τ² and 60 SWE test tasks are gated and certified (Appendix B) but never appear in Tables 2–5. Selection over candidate edits makes validation agreement an optimistically biased estimate of the final harness's performance, and the bias acts precisely on the Veritas-versus-labels comparison; the seven plain harnesses received no such selection. The authors state this plainly (Section 4.1: test tasks are 'held back'; the terminal rows are 'the closest we have to held-out results'), so it is a transparently scoped limitation, but it caps what the two largest rows can establish. Compounding: every configuration runs once (Limitations, [b216]), and Appendix C.3 reports two runs of the same τ² configuration differing by 0.040 (Veritas) and 0.083 (plain Claude Code), while two Veritas SWE runs differ by 0.021 (0.720 in Table 2 vs 0.741 for Full in Table 4). The leading margins of 0.061 (τ²) and 0.027 (SWE) are the same order as that spread, so even the ranking of rows 1–2 is not shown to be stable. The genuinely held-out terminal rows (0.078 and 0.074 margins, mechanisms frozen before those benchmarks existed) and the transfer rows of Table 3 remain real evidence; the concern is that the 'all four execution-scored benchmarks' claim overweights two in-sample rows covering 196 of 268 tasks.

The central claim, as the reader quotes it, is a four-benchmark ranking claim. For it to stand in its strongest reading, each row of Table 2 must be independent evidence that harness design, not tuning, produces the ordering. The weakest link is exactly the two largest rows: the τ² and SWE fit+validation sets were the selection data for the only accepted H-edits under the lenient phase-1 rule, and the same candidates and official labels (166 and 435 candidates, Section 4.1) then score the headline. The paper's own design — mechanical lint against task-naming, per-task splits, confirmation runs in later phases — shows the authors understand selection, which is why the unused 27-task and 60-task test splits are the sharpest gap rather than the acknowledged scope of the limitation. I agree with the reader's weakest_assumption and add one quantitative refinement: the exposure is larger than selection bias alone because the leading margins themselves sit inside the measured single-run spread. With a 0.021 swing between two Veritas SWE runs and a 0.083 swing for plain Claude Code on τ², a single repeat could plausibly reorder rows 1–2 even without any selection bias, and the paper itself reports no confidence intervals. Credit where due: the terminal rows are genuine frozen-mechanism held-out data with clean margins (0.078, 0.074); the certification gates (G1–G5), executed counterexamples, hand adjudication, generation/scoring code separation, and the worked case studies with verifier excerpts are real independent support for the mechanism story. The two accepted H-principles transfer out-of-sample to the terminal benchmarks and lift weaker harnesses (Claude Code +0.348 on Terminal-Bench), so the selection concern bounds the strongest headline reading; it does not impeach the transfer and ablation evidence. The acknowledged label-imperfection limitation [b224]-[b225] is symmetric across harnesses and therefore less threatening to a ranking claim. Verdict: UNCHANGED. The reader's CONDITIONAL is the right disposition: the two largest Table 2 rows should not be read as independent evidence for 'all four', and the missing test-split computation is a bounded, already-scaffolded experiment that would settle it. No cited published result is attacked; the concern is about the paper's own evaluation design.

The referee's report

If the results hold, the paper makes a useful contribution by demonstrating that harness design significantly affects the quality of automatically generated verifiers, and that the proposed mechanisms (E, R, H) are effective and transferable. The problem of generating reliable verifiers for agent tasks is relevant to self-improving systems and benchmark evaluation. The paper provides a systematic comparison across multiple harnesses and benchmarks, and includes a certification pipeline for labels.

Pith's reading of the paper (a machine's summary, not the authors')

This paper introduces Veritas, a harness designed to automatically generate verifiers for agent tasks, aiming to address the bottleneck of manually writing verifiers for self-improving systems. The central claim is that Veritas produces verifiers that agree better with official benchmark verdicts than those generated by seven other harnesses, across four execution-scored benchmarks, when the underlying model is held fixed. The key insight is that the harness design, not just the model, determines verifier quality, and Veritas achieves this through three mechanisms: an evidence header, a domain rule library, and authoring principles.

What carries the argument. The central mechanism is the Veritas harness itself, which adds three components to a coding agent: an evidence header that lists what inputs the verifier will receive and enforces that checks derive only from that evidence; a domain rule library containing 19,603 entries from six domains that supplies requirements the task text does not state; and a set of nine authoring principles that guide how to write a correct verifier in the absence of test candidates or labels.

Core claim. The paper demonstrates that a harness specifically engineered for verifier generation, incorporating mechanisms to address what evidence the verifier will observe, domain rules tasks leave unstated, and principles for writing correct verifiers, produces more reliable verifiers than general-purpose agent harnesses. With the model fixed to GLM-5.2, Veritas achieves higher Cohen's kappa agreement with official labels on all four execution-scored benchmarks (tau2-bench, SWE-bench Verified, Terminal-Bench, TB-Science) compared to seven plain harnesses, with margins of 0.03 to 0.08 in kappa. The mechanisms transfer to other harnesses and improve performance in 9 of 12 harness-benchmark pairs.

The premise it rests on. The official verdicts on certified candidates can be treated as ground truth labels, even though the certification process uses constructed candidates and may miss defects in the official verifier that would be exposed by real agent attempts.

If the paper is right (Pith's inferences, not the authors')

If Veritas's claim holds, self-improvement loops can be run on tasks that lack a hand-written verifier, scaling recursive self-improvement to real-world tasks collected without associated tests.

The finding that harness design substantially determines verifier quality (kappa ranges from 0.16 to 0.73 on tau2-bench with the same model) suggests that future work on agent systems should treat the harness as a first-class object of study and optimization.

Veritas's mechanisms being transferable to other harnesses implies that existing agent frameworks can be upgraded to produce better verifiers without replacing the underlying agent infrastructure.

The observation that generated verifiers tend to be too strict (rejecting correct solutions) while official verifiers tend to be too lenient suggests that combining generated and official verifiers could yield more balanced evaluation.

The paper's reliance on constructed candidates (reference solutions, no-ops, corrupted outputs) rather than real agent attempts means the reported agreement rates may not fully reflect performance on diverse, realistic agent behaviors; testing on actual agent trajectories would be a natural next step.

The self-improvement loop that refined Veritas's authoring principles used the same benchmarks' validation tasks, so the held-out test results on Terminal-Bench and TB-Science are the strongest evidence; the tau2-bench and SWE-bench results on fit+validation tasks may be partially inflated by selection during the loop's design phase.

The finding that plain harnesses lose mainly by failing to deliver verifiers on heavy environments (Terminal-Bench) suggests that delivery reliability, not just judgment accuracy, is a critical axis for verifier generation harnesses that is currently underappreciated.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. We propose Veritas, a harness that generates verifiers automatically. Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. On the two pairwise benchmarks they are at or near the best. They also agree better than those of the vendor's own harness for two of three other model families. Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-11

The authors' main claim, as the classifier picked it out 0.97

“With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.17 · human participants 0.05 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.97 · compares with prior results 0.98 · mentions code or data availability 0.06

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
The effectiveness of a self-improving system relies heavily on the quality of its verifier, which judges the outcome of each improvement round. background 1.00 0.06 0.04 asserted 1.00
However, the verifiers of most tasks are written by hand, which is expensive and does not scale to tasks that come without one. gap 0.75 0.06 0.02 asserted 1.00
We propose Veritas, a harness that generates verifiers automatically. contribution 0.99 0.84 0.15 asserted 0.99
Unlike general-purpose agent harnesses, Veritas adds three mechanisms for verifier generation: an evidence header stating what the verifier will observe, a library of domain rules that tasks leave unstated, and authoring principles, partly refined by a self-improvement loop. method 0.81 0.82 0.06 asserted 1.00
We evaluate on six benchmarks: four scored by execution, whose official verifiers we first certify, and two that compare pairs of responses. method 0.97 0.90 0.03 asserted 0.91
With the model fixed, the verifiers Veritas generates agree better with the official verdicts than those of seven other harnesses on all four execution-scored benchmarks. result 1.00 0.95 0.42 asserted 1.00
On the two pairwise benchmarks they are at or near the best. result 1.00 0.93 0.13 asserted 0.98
They also agree better than those of the vendor's own harness for two of three other model families. result 1.00 0.91 0.08 asserted 1.00
Its mechanisms also improve 9 of 12 harness-benchmark pairs scored by execution across three harnesses. result 1.00 0.92 0.13 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-11: 1 request, 6,812 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (1)
Figure 1 of the paper
Figure 1. Overview. κ ∗ is Cohen’s κ against the official labels, with a missing verdict counted as an error. Plain harnesses lack E, R and H; icons in (b), in reading order: Codex CLI, OpenCode, pi, Terminus 2, Hermes, Claude Code, Gemini CLI. All harnesses use GLM-5.2; values from

Figures are the authors', reproduced from the paper.

References (85)
  1. [1] Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. Otter: Generating tests from issues to validate SWE patches. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofPMLR, pages 752–771, 2025. URLhttps... on Pith
  2. [2] Toufique Ahmed, Jatin Ganhotra, Avraham Shinnar, and Martin Hirzel. Heterogeneous prompting and execution feedback for SWE issue test generation and selection. InProceedings of the 48th International Conference on Software Engineering, 2026. URLhttps://arxiv.org/abs/2508.06365 on PitharXiv
  3. [3] Afra Feyza Akyürek et al. PRBench: Large-scale expert rubrics for evaluating high-stakes professional reasoning. InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 42297–42325, 2026. doi: 10.18653/v1/2026.ac... on Pithdoi:10.18653/v1/2026.acl-long.1958
  4. [4] Anomaly. OpenCode: The open source coding agent.https://github.com/anomalyco/opencode, 2026. Version 1.18.3 on Pith
  5. [5] Anthropic. Claude Code.https://github.com/anthropics/claude-code, 2026. Version 2.1.278 on Pith
  6. [6] Chaithanya Bandi, Razvan-Gabriel Dumitru, Ben Hertzberg, et al. MCP-Atlas: A large-scale benchmark for tool-use competency with real MCP servers.arXiv preprint arXiv:2602.00933, 2026. URLhttps://arxiv.org/ abs/2602.00933 on PithPith's reviewarXiv
  7. [7] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan.τ 2-bench: Evaluating conversational agents in a dual-control environment. InProceedings of the 43rd International Conference on Machine Learning, volume 306 ofPMLR, pages 6736–6782, 2026. URLhttps://proc... on Pith
  8. [8] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT: Code generation with generated tests. InInternational Conference on Learning Representations, 2023. URL https://arxiv.org/abs/2207.10397 on PithPith's reviewarXiv
  9. [9] Jacob Cohen. A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20(1): 37–46, 1960. doi: 10.1177/001316446002000104 on Pithdoi:10.1177/001316446002000104
  10. [10] Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. TICKing all the boxes: Generated checklists improve LLM evaluation and generation.arXiv preprint arXiv:2410.03608, 2024. URL https://arxiv.org/abs/2410.03608 on PithPith's reviewarXiv
  11. [11] Alejandro Cuadron, Pengfei Yu, Yang Liu, and Arpit Gupta. SABER: Small actions, big errors — safeguarding mutating steps in LLM agents. InICLR 2026 Workshop on Memory for LLM-Based Agentic Systems, 2026. URL https://arxiv.org/abs/2512.07850. arXiv:2512.07850 on PitharXiv
  12. [12] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025. URLhttps://arxiv.org/abs/2501.12948. Journal version: Nature 645, 633–638 (2025) on PithPith's reviewarXiv
  13. [13] Xiang Deng, Jeff Da, Edwin Pan, et al. SWE-bench Pro: Can AI agents solve long-horizon software engineering tasks? InProceedings of the 43rd International Conference on Machine Learning, volume 306 ofPMLR, pages 23917–23934, 2026. URLhttps://proceedings.mlr.press/v306/deng26f.html on Pith
  14. [14] Kaustubh D. Dhole, Charles L. A. Clarke, and Eugene Agichtein. ExecRubrics: Executable tool-augmented rubrics for verifiable and efficient long-form evaluation. InFindings of the Association for Computational Linguistics: EMNLP 2026, 2026. URLhttps://arxiv.org/abs/2608.22559 on PithPith's reviewarXiv
  15. [15] Aleksander Ficek, Somshubra Majumdar, Vahid Noroozi, and Boris Ginsburg. Scoring verifiers: Evaluating synthetic verification for code and reasoning. InConference on Language Modeling, 2025. URLhttps://arxiv. org/abs/2502.13820 on PithPith's reviewarXiv
  16. [16] GLM-5-Team. GLM-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763, 2026. URL https://arxiv.org/abs/2602.15763 on PithPith's reviewarXiv
  17. [17] Google. Gemini CLI.https://github.com/google-gemini/gemini-cli, 2026. Version 0.60.0 on Pith
  18. [18] Google DeepMind. AlphaEvolve: How our Gemini-powered coding agent is scaling impact across fields. Blog post, May 2026. URLhttps://deepmind.google/blog/alphaevolve-impact/. 10 on Pith
  19. [19] Google DeepMind. Gemini 3.5 Flash model card. https://deepmind.google/models/model-cards/ gemini-3-5-flash/, 2026. Accessed through the API, October 2026 on Pith
  20. [20] Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. InInternational Conference on Learning Representa- tions, 2026. URLhttps://arxiv.org/abs/2507.17746 on PithPith's reviewarXiv
show 65 more
  1. [21] Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments.https://github.com/harbor-framework/harbor, 2026. Version 0.21.0 on Pith
  2. [22] Harbor Framework Team. Terminus 2.https://github.com/harbor-framework/harbor, 2026. Version 2.0.0, in Harbor 0.21.0 on Pith
  3. [23] Zhanbo Hua et al. CLI-Universe: Towards verifiable task synthesis engine for terminal agents.arXiv preprint arXiv:2606.22883, 2026. URLhttps://arxiv.org/abs/2606.22883 on PithPith's reviewarXiv
  4. [24] Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. DeepSWE: Measuring frontier coding agents on original, long-horizon engineering tasks.arXiv preprint arXiv:2607.07946, 2026. URL https://arxiv.org/abs/2607. 07946 on PithPith's reviewarXiv
  5. [25] Naman Jain, Jaskirat Singh, Manish Shetty, Tianjun Zhang, Liang Zheng, Koushik Sen, and Ion Stoica. R2E-Gym: Procedural environment generation and hybrid verifiers for scaling open-weights SWE agents. InConference on Language Modeling, 2025. URLhttps://openreview.net/forum?id=... on Pith
  6. [26] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? InInternational Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/2310.06770 on PithPith's reviewarXiv
  7. [27] Lara Khatib, Noble Saji Mathews, and Meiyappan Nagappan. AssertFlip: Reproducing bugs via inversion of LLM-generated passing tests. InProceedings of the 48th International Conference on Software Engineering, 2026. URLhttps://arxiv.org/abs/2507.17542 on PitharXiv
  8. [28] Rayan Khoury, Shih-Yao Lin, and Pratyush Mishra. SAGE: State-grounded, abstention-aware evaluation of task-oriented dialogue agents.arXiv preprint arXiv:2609.00434, 2026. URL https://arxiv.org/abs/2609.00434 on PithPith's reviewarXiv
  9. [29] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, et al. Tülu 3: Pushing frontiers in open language model post-training. InConference on Language Modeling, 2025. URL https://arxiv.org/abs/2411.15124 on PithPith's reviewarXiv
  10. [30] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. InFindings of the Association f... on Pith
  11. [31] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026. URL https://arxiv.org/ abs/2603.28052 on PithPith's reviewarXiv
  12. [32] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, 2024. URLhttps://arxiv.org/abs/2305.20050 on PithPith's reviewarXiv
  13. [33] Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, and Yu-Gang Jiang. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026. ... on PithPith's reviewarXiv
  14. [34] Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, and Siva Reddy. AgentRewardBench: Evaluating automatic evaluations of web agent trajectories. InConference on Language Modeli... on PitharXiv
  15. [35] Zihan Ma, Taolin Zhang, Maosong Cao, Junnan Liu, Wenwei Zhang, Minnan Luo, Songyang Zhang, and Kai Chen. Rethinking verification for LLM code generation: From generation to testing. InAdvances in Neural Information Processing Systems, 2025. URLhttps://arxiv.org/abs/2507.06920 on PithPith's reviewarXiv
  16. [36] Ryan Marten. Terminal-Bench 4.0.https://www.tbench.ai/news/terminal-bench-4-0, August 2026. 11 on Pith
  17. [37] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. InInternational Conference on Learning Representations, on Pith
  18. [38] Niels Mündler, Mark Niklas Müller, Jingxuan He, and Martin Vechev. SWT-Bench: Testing and validating real-world bug-fixes with code agents. InAdvances in Neural Information Processing Systems, 2024. URL https://arxiv.org/abs/2406.12952 on PithPith's reviewarXiv
  19. [39] Aarati Andrea Noronha, Kavya Ravikumar, and Carly Xiaoyu Lin. AutoGym: Blueprint-first generation of verifiable agent gyms.arXiv preprint arXiv:2609.22592, 2026. URLhttps://arxiv.org/abs/2609.22592 on PithPith's reviewarXiv
  20. [40] Nous Research. Hermes Agent.https://github.com/NousResearch/hermes-agent, 2026. Versions 0.21.2 and 0.21.4 on Pith
  21. [41] Alexander Novikov, Ngân V˜ u, Marvin Eisenberger, et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery.arXiv preprint arXiv:2506.13131, 2025. URLhttps://arxiv.org/abs/2506.13131 on PithPith's reviewarXiv
  22. [42] OpenAI. Introducing SWE-bench Verified. Blog post, August 2024. URL https://openai.com/index/ introducing-swe-bench-verified/ on Pith
  23. [43] OpenAI. Codex CLI.https://github.com/openai/codex, 2026. Version 0.154.0 on Pith
  24. [44] OpenAI. GPT-5.6: Frontier intelligence that scales with your ambition.https://openai.com/index/gpt-5-6/, July 2026. Modelgpt-5.6-sol, accessed through the API, September–October 2026 on Pith
  25. [45] OpenAI. Separating signal from noise in coding evaluations. Blog post, July 2026. URLhttps://openai.com/ index/separating-signal-from-noise-coding-evaluations/ on Pith
  26. [46] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and... on PithPith's reviewarXiv
  27. [47] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. InAdvances in Neural Information Processing Systems, 2024. URLhttps://arxiv.org/abs/2404.13076 on PithPith's reviewarXiv
  28. [48] Qwen Team. Qwen Code.https://github.com/QwenLM/qwen-code, 2026. Version 0.25.0 on Pith
  29. [49] Qwen Team. Qwen3.7-Plus: Multimodal agent intelligence. https://qwen.ai/blog?id=qwen3.7-plus, 2026. Accessed through the API, October 2026 on Pith
  30. [51] Maxime Robeyns, Martin Szummer, and Laurence Aitchison. A self-improving coding agent.arXiv preprint arXiv:2504.15228, 2025. URLhttps://arxiv.org/abs/2504.15228 on PithPith's reviewarXiv
  31. [52] URLhttps://arxiv.org/abs/2606.26300 on PithPith's reviewarXiv
  32. [53] Scale AI. SWE-bench Pro V2 release notes.https://github.com/scaleapi/SWE-bench_Pro-os/blob/main/v2/ README.md, September 2026 on Pith
  33. [54] Swastik Roy, Rajkumar Pujari, Tharindu Kumarage, Charith Peris, Rahul Gupta, Anna Rumshisky, Pradeep Natarajan, and Venkatesh Saligrama. PReMISE: Policy rubrics as measurement specifications for LLM judges. arXiv preprint arXiv:2605.30803, 2026. URLhttps://arxiv.org/abs/2605.30803 on PithPith's reviewarXiv
  34. [55] Sierra Research. τ-bench 1.0.0: Voice, knowledge, task quality. https://github.com/sierra-research/ tau2-bench/releases/tag/v1.0.0, March 2026 on Pith
  35. [56] Shreya Shankar, J.D. Zamfirescu-Pereira, Björn Hartmann, Aditya G. Parameswaran, and Ian Arawjo. Who validates the validators? Aligning LLM-assisted evaluation of LLM outputs with human preferences. InProceedings of the 37th Annual ACM Symposium on User Interface Software and ... on PithPith's reviewarXiv
  36. [57] Yuxuan Sun, Yuze Zhao, Yufeng Wang, Yao Du, Zhiyuan Ma, Jinbo Wang, Mengdi Zhang, Kai Zhang, and Zhenya Huang. SWE-Mutation: Can LLMs generate reliable test suites in software engineering? InFindings of the Association for Computational Linguistics: ACL 2026, pages 39651–39674... on Pithdoi:10.18653/v1/2026
  37. [58] Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, et al. Agents’ Last Exam.arXiv preprint arXiv:2606.05405, 2026. URLhttps://arxiv.org/abs/2606.05405v2. Version 2, 11 June 2026. 12 on PitharXiv
  38. [59] Zihan Tan, Leixin Sun, Zitong Shi, Yitao Liu, Jiajun Wu, et al. MetaRSI / RSI2: A meta-recursive self- improving system for recursive self-improving systems themselves.arXiv preprint arXiv:2609.06396, 2026. URL https://arxiv.org/abs/2609.06396 on PithPith's reviewarXiv
  39. [60] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. JudgeBench: A benchmark for evaluating LLM-based judges. InInternational Conference on Learning Representations, 2025. URLhttps://arxiv.org/abs/2410.12784 on PithPith's reviewarXiv
  40. [61] Terminal-Bench Team. Terminal-Bench 2.1.https://www.tbench.ai/news/terminal-bench-2-1, May 2026 on Pith
  41. [62] Terminal-Bench-Science Team. Terminal-Bench-Science: Evaluating AI agents on research workflows across scientific domains.https://github.com/harbor-framework/terminal-bench-science, 2026. Version 0.1.0 on Pith
  42. [63] Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models. InAdvances in Neural Information Processing Systems, 2025. URLhttps://arxiv.org/abs/2507.18624 on PitharXiv
  43. [64] Xinming Tu, Tianze Wang, Yingzhou Lu, Kexin Huang, Yuanhao Qu, and Sara Mostafavi. Who guards the benchmarks? Automated auditing of LLM agent benchmarks. InConference on Language Modeling, 2026. URL https://openreview.net/forum?id=3x89U8nvoE on Pith
  44. [65] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024. UR... on Pith
  45. [67] URLhttps://arxiv.org/abs/2605.12673 on PithPith's reviewarXiv
  46. [68] Siwei Wu et al. ModularRSI: Modular and generalizable recursive harness self-improvement.arXiv preprint arXiv:2609.14857, 2026. URLhttps://arxiv.org/abs/2609.14857 on PithPith's reviewarXiv
  47. [69] You Wang, Michael Pradel, and Zhongxin Liu. Are “solved issues” in SWE-bench really solved correctly? An empirical study. InProceedings of the 48th International Conference on Software Engineering, 2026. doi: 10.1145/3744916.3764576. URLhttps://arxiv.org/abs/2503.15223 on PithPith's reviewarXiv
  48. [70] Chen Henry Wu and Aditi Raghunathan. Self-trained verification for training- and test-time self-improvement. arXiv preprint arXiv:2605.30290, 2026. URLhttps://arxiv.org/abs/2605.30290 on PithPith's reviewarXiv
  49. [71] Z.ai. GLM-5.2. https://huggingface.co/zai-org/GLM-5.2, 2026. Accessed through an API, August–October 2026 on Pith
  50. [72] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.τ-bench: A benchmark for tool-agent- user interaction in real-world domains. InInternational Conference on Learning Representations, 2025. URL https://arxiv.org/abs/2406.12045 on PithPith's reviewarXiv
  51. [73] Boxi Yu, Yuxuan Zhu, Pinjia He, and Daniel Kang. UTBoost: Rigorous evaluation of coding agents on SWE-bench. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3762–3774, 2025. doi: 10.18653/v1/2025.acl-long... on Pithdoi:10.18653/v1/2025.acl-long.189
  52. [74] Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. InInternational Conference on Learning Representations, 2024. URL https://arxiv.org/abs/2310.07641. 13 on PithPith's reviewarXiv
  53. [75] Z.ai. Terminal-Bench 2.0 Verified: Instruction & environment fix version. Hugging Face dataset, February 2026. URLhttps://huggingface.co/datasets/zai-org/terminal-bench-2-verified on Pith
  54. [76] Mario Zechner. Pi: A minimal terminal coding harness.https://github.com/earendil-works/pi, 2026. Version 0.85.1 on Pith
  55. [77] Yukun Zhang, Kemu Xu, and Yishen Chen. How do agent harnesses create value? Planning information and release control in stateful LLM agents.arXiv preprint arXiv:2609.20474, 2026. URLhttps://arxiv.org/abs/2609.20474 on PithPith's reviewarXiv
  56. [78] Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel Machine: Open-ended evolution of self-improving agents. InInternational Conference on Learning Representations, 2026. URL https://arxiv.org/abs/2505.22954 on PithPith's reviewarXiv
  57. [79] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. InInternational Conference on Learning Representations, on Pith
  58. [80] Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-Judge: Evaluate agents with agents. InInternational ... on Pith
  59. [82] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. InAdvances in Neural Information Process... on Pith
  60. [83] Yuxuan Zhu et al. Establishing best practices in building rigorous agentic benchmarks. InAdvances in Neural Information Processing Systems 38 (Datasets and Benchmarks Track), 2025. URL https://proceedings. neurips.cc/paper_files/paper/2025/hash/f316275b44ee2de533102913828a8107... on Pith
  61. [85] URLhttps://arxiv.org/abs/2410.10934. A Harness Implementation Every harness runs with a limit of 7,200 seconds per generation and with the default decoding settings of the model and the harness. A.1 Probe, repair and delivery This subsection details stages three and four of th... on PithPith's reviewarXiv
  62. [86] No fabrication:never write a verifier that is always true or always false; never usetry/except to swallow a failed judgment; the verifier must at least separate “correct candidates” from “wrong candidates”—pass the correct ones and fail the wrong ones. 6.No cheating / no leaka... on Pith
  63. [87] Output check:after writing the verifier artifact and before finishing, run one final self-check: (a) the file is written to the path this benchmark’s contract specifies; (b) the artifact can be imported/loaded without syntax or import errors; (c) the entry function the contrac... on Pith
  64. [2025] URLhttps://arxiv.org/abs/2408.15240 on PithPith's reviewarXiv
  65. [2026] URLhttps://arxiv.org/abs/2601.11868 on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment