Pith. sign in

REVIEW 2 major objections 1 cited by

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

RubricRefine generates task-specific rubrics to verify and repair tool-use code for inter-tool contract violations before any execution occurs.

2026-05-20 22:08 UTC pith:24LDJL2T

load-bearing objection RubricRefine gives a practical pre-execution rubric check that lifts multi-tool reliability to 0.86 without runtime costs, though the LLM judge step needs tighter validation. the 2 major comments →

arxiv 2605.09730 v3 pith:24LDJL2T submitted 2026-05-10 cs.LG cs.SE

RubricRefine: Improving Tool-Use Agent Reliability with Training-Free Pre-Execution Refinement

Will LeVine , Brendan Evers , Sam Saltwick , Abhay Venkatesh This is my paper
classification cs.LG cs.SE
keywords tool-use agentspre-execution refinementsemantic contract verificationinference-time methodsLLM code repairmulti-step tool callingagent reliability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that many tool-use agent failures stem from contract violations between tools, such as mismatched output shapes or broken argument flows, which run to completion without raising errors and thus evade standard runtime feedback. RubricRefine creates explicit rubrics from the task description and tool registry, scores candidate code against those checks, and iterates repairs in advance of any tool calls. This yields an average success rate of 0.86 across seven models on a multi-step benchmark while requiring zero executions and lower latency than execution-based baselines. Gains stay flat on single-step tasks, matching the method's design for catching inter-tool structure issues.

Core claim

RubricRefine is a training-free pre-execution method that produces task- and registry-specific rubrics, scores candidate code against explicit semantic contract checks for violations such as wrong output shape or incorrect tool routing, and iteratively repairs detected failures before execution, reaching 0.86 average success on M3ToolEval across seven models with zero execution attempts and up to 2.6 times lower latency than prior inference-time baselines.

What carries the argument

RubricRefine, the process of generating task-specific rubrics from the query and tool registry then scoring and repairing code against explicit contract checks before any execution.

Load-bearing premise

The dominant failures in tool-use agents are inter-tool contract violations that complete without raising runtime errors, and LLM-generated rubrics can reliably detect and repair these before execution.

What would settle it

Running the same models and tasks but replacing the main failure mode with runtime errors that do raise exceptions during execution, then checking whether RubricRefine still outperforms execution-feedback baselines.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Tool-use agents reach higher reliability without consuming any execution attempts or runtime feedback signals.
  • Latency drops by up to 2.6 times relative to methods that rely on actual tool runs for revision.
  • Performance gains concentrate on multi-step tasks that involve multiple interdependent tool calls.
  • Rubric-category ablations show that explicit checks for output shape and argument provenance drive most of the improvement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar rubric generation could be added to agent frameworks to catch contract issues in planning or multi-agent coordination where hidden violations also occur.
  • The flat performance on single-step tasks suggests the method would benefit from hybrid use with execution feedback when tasks lack inter-tool structure.
  • Applying the approach to new tool registries would test whether rubric generation scales when the set of available functions changes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper presents RubricRefine, a training-free method for pre-execution refinement of tool-use agent code. It generates task-specific rubrics to score and repair inter-tool contract violations before execution, achieving an average of 0.86 on M3ToolEval across seven models with zero executions and reduced latency compared to baselines. It shows limited improvement on single-step tasks and includes ablations on rubric categories and calibration.

Significance. This approach addresses a key limitation in iterative self-refinement for tool-use by focusing on semantic contracts without runtime feedback. If validated, the gains in reliability and efficiency could advance the development of more robust LLM agents for complex multi-tool tasks. The empirical focus on when the method works (multi vs single step) is a strength.

major comments (2)
  1. §5.3 Calibration Analysis: The calibration analysis provides internal consistency checks but lacks external validation against human-annotated ground truth for detected contract violations. This is critical as the method depends on the LLM judge reliably identifying actual inter-tool violations (wrong shapes, routing, provenance) rather than surface issues; without this, the 0.86 performance and generalization claims are at risk.
  2. Table 1 (Results on M3ToolEval): The reported improvements (0.86 vs. 0.65-0.75) and latency reductions do not specify the exact implementation details of the prior inference-time baselines, such as whether they used the same LLM for refinement or equivalent iteration counts, which is necessary to substantiate the 2.6× latency improvement claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, providing clarifications and indicating revisions where the concerns are valid and can be addressed without misrepresenting our results.

read point-by-point responses

  1. Referee: §5.3 Calibration Analysis: The calibration analysis provides internal consistency checks but lacks external validation against human-annotated ground truth for detected contract violations. This is critical as the method depends on the LLM judge reliably identifying actual inter-tool violations (wrong shapes, routing, provenance) rather than surface issues; without this, the 0.86 performance and generalization claims are at risk.

    Authors: We acknowledge that the calibration analysis in §5.3 is based on internal consistency metrics, such as agreement across rubric categories and correlation with downstream task success, rather than direct comparison to human-annotated ground truth for contract violations. This is a fair observation, as external validation would more definitively rule out the possibility that the LLM judge is detecting surface-level issues instead of true semantic contract violations. We have revised the manuscript to explicitly discuss this as a limitation in §5.3 and the conclusion, while noting that the multi-model empirical gains on M3ToolEval and the rubric ablation provide supporting evidence for the judge's utility. A full human study is beyond the current scope but could be pursued in follow-up work. revision: partial

  2. Referee: Table 1 (Results on M3ToolEval): The reported improvements (0.86 vs. 0.65-0.75) and latency reductions do not specify the exact implementation details of the prior inference-time baselines, such as whether they used the same LLM for refinement or equivalent iteration counts, which is necessary to substantiate the 2.6× latency improvement claim.

    Authors: We agree that additional implementation details for the baselines are needed to make the latency comparison fully transparent. The original experimental setup referenced prior inference-time methods but did not explicitly restate that the same base LLMs were used for refinement steps and that iteration budgets were aligned for fairness. We have revised Table 1's caption and added a dedicated paragraph in the experimental details section to specify these points, confirming equivalent LLM usage and iteration matching. This directly supports the reported 2.6× latency advantage of RubricRefine, which stems from its pre-execution nature requiring no runtime attempts. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical method with independent benchmark validation

full rationale

The paper defines RubricRefine explicitly as a training-free process that generates task- and registry-specific rubrics, applies LLM scoring for inter-tool contract checks, and performs iterative repair prior to execution. Central performance numbers (0.86 average on M3ToolEval, flat results on API-Bank) are obtained via direct comparison against baselines on fixed external benchmarks, accompanied by rubric-category ablation and calibration analysis. No equations, parameter fits, or self-citations are present that would reduce any claimed result to its own inputs by construction. The method's logic and evaluation remain self-contained against external test sets rather than relying on self-referential definitions or load-bearing prior author work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that LLMs can produce effective rubrics and that inter-tool contract violations dominate failures; no free parameters or new entities are introduced in the abstract.

axioms (1)
  • domain assumption Large language models can generate accurate task- and registry-specific rubrics that capture semantic contract violations in tool-use code.
    Method depends on the quality of LLM-generated rubrics for scoring and repair.

pith-pipeline@v0.9.0 · 5739 in / 1300 out tokens · 44235 ms · 2026-05-20T22:08:27.045816+00:00 · methodology

0 comments
read the original abstract

Iterative self-refinement is a popular inference-time reliability technique, but its effectiveness in code-mode tool use depends heavily on the structure of the feedback signal: unstructured critique helps inconsistently across models, and even revision with real execution feedback improves only modestly ($0.75$ vs. $0.65$ baseline). The dominant failures are inter-tool contract violations (wrong output shape, incorrect tool routing, broken argument provenance) that run to completion without raising errors, making runtime feedback insufficient. We introduce RubricRefine, a training-free method for pre-execution semantic contract verification that generates task- and registry-specific rubrics, scores candidate code against explicit contract checks, and iteratively repairs failures before any execution occurs. RubricRefine reaches $0.86$, averaged across seven models, on M3ToolEval with zero execution attempts, improving over prior inference-time baselines with up to $2.6\times$ lower latency. Performance remains flat on the predominantly single-step API-Bank, consistent with the method's reliance on inter-tool contract structure. A rubric-category ablation and calibration analysis further characterize when and why the method works.

Figures

Figures reproduced from arXiv: 2605.09730 by Abhay Venkatesh, Brendan Evers, Sam Saltwick, Will LeVine.

Figure 1
Figure 1. Figure 1: RubricRefine overview. Setup (top row): the task instruction and tool documentation (○1 ) are passed to the rubric generator VR (○2 ), which produces a task-specific rubric R (○3 ) of itemized contract checks. Refinement loop (bottom row): the generator G (○4 ) produces a candidate cr each round; the candidate flows through the verifier V (○6 ), which scores it against R and emits that round’s score, item-… view at source ↗
Figure 1
Figure 1. Figure 1: RubricRefine overview. Setup (top row): the task instruction and tool documentation (○1 ) are passed to the rubric generator VR (○2 ), which produces a task-specific rubric R (○3 ) of itemized contract checks. Refinement loop (bottom row): the generator G (○4 ) produces a candidate cr each round; the candidate flows through the verifier V (○6 ), which scores it against R and emits that round’s score, item-… view at source ↗
Figure 2
Figure 2. Figure 2: Reliability diagrams for normalized rubric scores on M3ToolEval. Left: GPT-4.1-mini (ECE = 0.063), well-calibrated across all bins. Right: Gemma-4-26B (ECE = 0.165), poorly cal￾ibrated in the middle bins but retaining meaningful top-bin separation (accuracy 0.87 at score = 10, n = 329). RubricRefine’s early stopping depends only on the top bin, so the method remains effec￾tive on Gemma despite the verifier… view at source ↗
Figure 3
Figure 3. Figure 3: Reliability diagram for GPT-4.1 on M3ToolEval (ECE = 0.090). cant difference between the estimated and true accuracy. Expected Calibration Error (ECE) We quantify miscalibration with the Expected Calibration Error (ECE) (Naeini et al., 2015). ECE, aimed at summarizing the mis￾calibration visualized in reliability diagrams, is calculated as ECE = X M m=1 |Bm| |D| [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: provides the appendix wall-clock com￾parison with the full method set and any ad￾ditional budget sweeps. We use this plot to check that the main-paper comparison between RubricRefine and Self-Refine is not driven by a single reporting point. If RubricRefine traces a stronger success–latency frontier across mul￾tiple operating points, then the improvement reflects a better use of inference time rather than … view at source ↗
Figure 5
Figure 5. Figure 5: reports success against total LM calls. This controls for a different notion of budget than wall-clock latency: call count abstracts away from serving noise and asks how effec￾tively each method turns model invocations into successful trajectories [PITH_FULL_IMAGE:figures/full_fig_p025_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Success rate vs. total to￾kens on M3ToolEval (GPT-4.1; same method_eval_fixed_story run as the main tables). model compute with network overhead. To check that RubricRefine’s efficiency advantage transfers to a different serving regime, we also report the same three views for Gemma-4-26B served locally via vLLM ( [PITH_FULL_IMAGE:figures/full_fig_p026_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Inference-cost tradeoffs on M3ToolEval for Gemma-4-26B (served locally via vLLM). Top: success vs. wall-clock latency per task. Middle: success vs. LM calls per task. Bottom: success vs. total tokens per task. RubricRefine achieves the highest success rate while consuming strictly less of each inference-cost axis than Best-of-N+rubric, matching the qualitative pattern seen on the frontier API models. front… view at source ↗
Figure 8
Figure 8. Figure 8: Reliability diagram for Gemma-4-26B’s normalized rubric scores on BoN+rubric trajecto￾ries on M3ToolEval (10 trials; n = 2,160). Com￾pare with [PITH_FULL_IMAGE:figures/full_fig_p027_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: reports the results. Panel (a) shows RubricRefine success rate as a function of the maximum refinement round R. Nearly all gains materialize in the first refinement round (R = 1 → R = 2): averaged across models, suc￾cess jumps by +0.17 absolute in a single round and then plateaus. This confirms that the early￾stopping mechanism (Section 4.5) captures most of the available improvement and that increasing R … view at source ↗
Figure 9
Figure 9. Figure 9: Budget-parameter scaling on M3ToolEval. (a) RubricRefine success rate by maximum refine￾ment round R. (b) Best-of-N+rubric success rate by number of candidates N. Shaded regions show ±1 SE across available trials. Dotted lines indicate per-model CodeAct baselines [PITH_FULL_IMAGE:figures/full_fig_p030_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Per-task-family breakdown of RubricRe￾fine success rate by maximum refinement round R (o3-mini). Travel planning and message decoder saturate at R = 2. Trade calculator does not benefit from additional rounds and degrades beyond R = 2, consistent with rubric-guided contract checks being less effective for arithmetic-heavy tasks. Error bars show ±1 SE. N. In contrast to the sharp saturation of it￾erative r… view at source ↗
Figure 10
Figure 10. Figure 10: Per-task-family breakdown of RubricRe￾fine success rate by maximum refinement round R (o3-mini). Travel planning and message decoder saturate at R = 2. Trade calculator does not benefit from additional rounds and degrades beyond R = 2, consistent with rubric-guided contract checks being less effective for arithmetic-heavy tasks. Error bars show ±1 SE. Model RubricRefine Avg. LM Calls BoN+rubric Avg. LM Ca… view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    cs.AI 2026-07 conditional novelty 6.0

    A survey of 1,250 papers organizes AI self-improvement along two axes—what is improved and loop closure—finding that demonstrated self-improvement strength tracks a verification hierarchy from formal verifiers down to...

Reference graph

Works this paper leans on

52 extracted references · 52 canonical work pages · cited by 1 Pith paper · 9 internal anchors

  1. [1]

    Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

    Yinghui He, Simran Kaur, Adithya Bhaskar, Yongjin Yang, Jiarui Liu, Narutatsu Ri, Liam Fowl, Abhishek Panigrahi, Danqi Chen, and Sanjeev Arora. Self-distillation zero: Self-revision turns binary rewards into dense supervision. arXiv:2604.12002, 2026. URL: https://arxiv.org/abs/2604.12002

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Proceedings of ICML, 2024. arXiv:2402.01030. URL: https://proceedings.mlr.press/v235/wang24h.html

    work page arXiv 2024

  3. [3]

    Api-bank: A comprehensive benchmark for tool-augmented llms

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-Bank : A comprehensive benchmark for tool-augmented LLMs . In Proceedings of EMNLP, 2023. DOI: https://doi.org/10.18653/v1/2023.emnlp-main.187. URL: https://aclanthology.org/2023.emnlp-main.187/

    work page doi:10.18653/v1/2023.emnlp-main.187 2023

  4. [4]

    Self-refine: Iterative refinement with self-feedback

    Aman Madaan et al. Self-refine: Iterative refinement with self-feedback. In Proceedings of NeurIPS, 2023. URL: https://openreview.net/forum?id=S37hOerQLB

    work page 2023

  5. [5]

    Language Models (Mostly) Know What They Know

    Saurav Kadavath et al. Language models (mostly) know what they know. arXiv:2207.05221, 2022. URL: https://arxiv.org/abs/2207.05221

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  6. [6]

    and Caruana, R

    Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of ICML, 2005. DOI: https://doi.org/10.1145/1102351.1102430

    work page doi:10.1145/1102351.1102430 2005

  7. [7]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of ICML, 2017. URL: https://proceedings.mlr.press/v70/guo17a.html

    work page 2017

  8. [8]

    Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers

    Meelis Kull, Telmo Silva Filho, and Peter Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Proceedings of AISTATS, 2017. URL: https://proceedings.mlr.press/v54/kull17a.html

    work page 2017

  9. [9]

    smolagents: a smol library to build great agentic systems

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunism\"aki. smolagents: a smol library to build great agentic systems. GitHub repository, 2025. URL: https://github.com/huggingface/smolagents

    work page 2025

  10. [10]

    Code Mode: the better way to use MCP

    Kenton Varda and Sunil Pai. Code Mode: the better way to use MCP . Cloudflare blog, 2025. URL: https://blog.cloudflare.com/code-mode/

    work page 2025

  11. [11]

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling model parameters. arXiv:2408.03314, 2024. URL: https://arxiv.org/abs/2408.03314

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Let's verify step by step

    Hunter Lightman et al. Let's verify step by step. In Proceedings of ICLR, 2024. URL: https://openreview.net/forum?id=v8L0pN6EOi

    work page 2024

  13. [13]

    When Can

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can LLMs actually correct their own mistakes? A critical survey of self-correction of LLMs . TACL, 2024. DOI: https://doi.org/10.1162/tacl_a_00713. URL: https://aclanthology.org/2024.tacl-1.78/

    work page doi:10.1162/tacl_a_00713 2024

  14. [14]

    PreFlect : From retrospective to prospective reflection in large language model agents

    Hanyu Wang, Yuanpu Cao, Lu Lin, and Jinghui Chen. PreFlect : From retrospective to prospective reflection in large language model agents. arXiv:2602.07187, 2026. URL: https://arxiv.org/abs/2602.07187

    work page arXiv 2026

  15. [15]

    CRITIC : Large language models can self-correct with tool-interactive critiquing

    Zhibin Gou et al. CRITIC : Large language models can self-correct with tool-interactive critiquing. In Proceedings of ICLR, 2024. URL: https://openreview.net/forum?id=Sx038qxjek

    work page 2024

  16. [16]

    arXiv preprint arXiv:2409.00920 , year=

    Weiwen Liu et al. ToolACE : Winning the points of LLM function calling. arXiv:2409.00920, 2024. URL: https://arxiv.org/abs/2409.00920

    work page arXiv 2024

  17. [17]

    Facilitating multi-turn function calling for LLMs via compositional instruction tuning

    Mingyang Chen et al. Facilitating multi-turn function calling for LLMs via compositional instruction tuning. In Proceedings of ICLR, 2025. URL: https://openreview.net/forum?id=owP2mymrTD

    work page 2025

  18. [18]

    Advancing tool-augmented large language models via meta-verification and reflection learning

    Zhiyuan Ma et al. Advancing tool-augmented large language models via meta-verification and reflection learning. In Proceedings of KDD, 2025. DOI: https://doi.org/10.1145/3711896.3736835

    work page doi:10.1145/3711896.3736835 2025

  19. [19]

    FunReason : Enhancing large language models' function calling via self-refinement multiscale loss and automated data refinement

    Bingguang Hao et al. FunReason : Enhancing large language models' function calling via self-refinement multiscale loss and automated data refinement. arXiv:2505.20192, 2025. URL: https://arxiv.org/abs/2505.20192

    work page arXiv 2025

  20. [20]

    Proceduralization

    Shaokun Zhang et al. Nemotron-Research-Tool-N1: Exploring tool-using language models with reinforced reasoning. arXiv:2505.00024, 2025. URL: https://arxiv.org/abs/2505.00024

    work page arXiv 2025

  21. [21]

    ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

    Jiazhan Feng et al. ReTool : Reinforcement learning for strategic tool use in LLMs . arXiv:2504.11536, 2025. URL: https://arxiv.org/abs/2504.11536

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  22. [22]

    GEAR : Augmenting language models with generalizable and efficient tool resolution

    Yining Lu, Haoping Yu, and Daniel Khashabi. GEAR : Augmenting language models with generalizable and efficient tool resolution. In Proceedings of EACL, 2024. URL: https://aclanthology.org/2024.eacl-long.7/

    work page 2024

  23. [23]

    Chain-of-Tools: Utilizing massive unseen tools in the CoT reasoning of frozen language models

    Mengsong Wu et al. Chain-of-Tools: Utilizing massive unseen tools in the CoT reasoning of frozen language models. arXiv:2503.16779, 2025. URL: https://arxiv.org/abs/2503.16779

    work page arXiv 2025

  24. [24]

    Graph rag-tool fusion,

    Elias Lumer et al. Graph RAG -Tool Fusion. arXiv:2502.07223, 2025. URL: https://arxiv.org/abs/2502.07223

    work page arXiv 2025

  25. [25]

    ToolLLM: Facilitating large language models to master 16000+ real-world APIs

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. ToolLLM: Facilitating large language models to master 16000+ real-world APIs . In Proceedings of ICLR, 2024. URL: https://openre...

    work page 2024

  26. [26]

    Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley function calling leaderboard ( BFCL ): From tool use to agentic evaluation of large language models. In Proceedings of ICML, 2025. URL: https://proceedings.mlr.press/v267/patil25a.html

    work page 2025

  27. [27]

    Constitutional AI: Harmlessness from AI Feedback

    Yuntao Bai et al. Constitutional AI : Harmlessness from AI feedback. arXiv:2212.08073, 2022. URL: https://arxiv.org/abs/2212.08073

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  28. [28]

    Judging LLM -as-a-judge with MT-Bench and Chatbot Arena

    Lianmin Zheng et al. Judging LLM -as-a-judge with MT-Bench and Chatbot Arena . In Proceedings of NeurIPS, 2023. URL: https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf

    work page 2023

  29. [29]

    Prometheus: Inducing fine-grained evaluation capability in language models

    Seungone Kim et al. Prometheus: Inducing fine-grained evaluation capability in language models. In Proceedings of ICLR, 2024. URL: https://openreview.net/forum?id=8euJaTveKw

    work page 2024

  30. [30]

    Balwani, Denis Peskoff, Marcos Ayestaran, Sean M

    Manasi Sharma et al. ResearchRubrics : A benchmark of prompts and rubrics for evaluating deep research agents. arXiv:2511.07685, 2025. URL: https://arxiv.org/abs/2511.07685

    work page arXiv 2025

  31. [31]

    Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

    Anisha Gunjal et al. Rubrics as Rewards: Reinforcement learning beyond verifiable domains. arXiv:2507.17746, 2025. URL: https://arxiv.org/abs/2507.17746

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  32. [32]

    arXiv preprint arXiv:2601.04171

    Mohit Raghavendra et al. Agentic Rubrics as contextual verifiers for SWE agents. arXiv:2601.04171, 2026. URL: https://arxiv.org/abs/2601.04171

    work page arXiv 2026

  33. [33]

    DeGroot and Stephen E

    Morris H. DeGroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. Journal of the Royal Statistical Society: Series D (The Statistician), 32(1-2):12--22, 1983. DOI: https://doi.org/10.2307/2987588

    work page doi:10.2307/2987588 1983

  34. [34]

    Cooper, and Milos Hauskrecht

    Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of AAAI, 2015. URL: https://ojs.aaai.org/index.php/AAAI/article/view/9602

    work page 2015

  35. [35]

    Enabling calibration in the zero-shot inference of large vision-language models

    Will LeVine, Benjamin Pikus, Pranav Raja, and Fernando Amat Gil. Enabling calibration in the zero-shot inference of large vision-language models. In Proceedings of ICLR (Tiny Papers), 2023. arXiv:2303.12748. URL: https://openreview.net/forum?id=na1T7ZGYb4

    work page arXiv 2023

  36. [36]

    Accurate layerwise interpretable competence estimation

    Vickram Rajendran and William LeVine. Accurate layerwise interpretable competence estimation. Advances in Neural Information Processing Systems, 32, 2019. URL: https://proceedings.neurips.cc/paper_files/paper/2019/file/a11da6bd58b95b334f8cd49f00918f16-Paper.pdf

    work page 2019

  37. [37]

    Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration

    Meelis Kull, Miquel Perello Nieto, Markus K\"angsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. Advances in Neural Information Processing Systems, 32, 2019. URL: https://proceedings.neurips.cc/paper_files/paper/2019/file/8ca01ea920679a0fe3728441...

    work page 2019

  38. [38]

    Revisiting the calibration of modern neural networks

    Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems, 34:15682--15694, 2021. URL: https://proceedings.neurips.cc/paper_files/paper/2021/file/8420d359404024567b5aefda1231af24-Paper.pdf

    work page 2021

  39. [39]

    Teaching Large Language Models to Self-Debug

    Xinyun Chen, Maxwell Lin, Nathanael Sch\"arli, and Denny Zhou. Teaching large language models to self-debug. In Proceedings of ICLR, 2024. arXiv:2304.05128. URL: https://openreview.net/forum?id=KuPixIqPiq

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  40. [40]

    CodeT: Code Generation with Generated Tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. CodeT : Code generation with generated tests. arXiv preprint, 2022. arXiv:2207.10397. URL: https://arxiv.org/abs/2207.10397

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [41]

    Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with AlphaCodium : From prompt engineering to flow engineering. arXiv preprint, 2024. arXiv:2401.08500. URL: https://arxiv.org/abs/2401.08500

    work page Pith review arXiv 2024

  42. [42]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Proceedings of NeurIPS, 2023. URL: https://proceedings.neurips.cc/paper_files/paper/2023/file/1b44b878bb782e6954cd888628510e90-Paper-Conference.pdf

    work page 2023

  43. [43]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Proceedings of NeurIPS, 2023. URL: https://proceedings.neurips.cc/paper_files/paper/2023/file/271db9922b8d1f4dd7aaef84ed5ac703-Paper-Conference.pdf

    work page 2023

  44. [44]

    Google DeepMind. Gemma 4. 2026. URL: https://deepmind.google/models/gemma/gemma-4/

    work page 2026

  45. [45]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In Proceedings of ICLR, 2024. URL: https://openreview.net/forum?id=IkmD3fKBPQ

    work page 2024

  46. [46]

    Chasing the tail: Effective rubric-based reward modeling for large language model post-training

    Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin. Chasing the tail: Effective rubric-based reward modeling for large language model post-training. In Proceedings of ICLR, 2026. arXiv:2509.21500. URL: https://arxiv.org/abs/2509.21500

    work page arXiv 2026

  47. [47]

    LLM -as-a-Verifier: A general-purpose verification framework

    Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Marco Pavone, Ion Stoica, and Azalia Mirhoseini. LLM -as-a-Verifier: A general-purpose verification framework. Project page, 2026. URL: https://llm-as-a-verifier.notion.site/

    work page 2026

  48. [48]

    Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C.H. Hoi. CodeRL : Mastering code generation through pretrained models and deep reinforcement learning. In Proceedings of NeurIPS, 2022. URL: https://proceedings.neurips.cc/paper_files/paper/2022/file/8636419dea1aa9fbd25fc4248e702da4-Paper-Conference.pdf

    work page arXiv 2022

  49. [49]

    Wang, and Xi Victoria Lin

    Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida I. Wang, and Xi Victoria Lin. LEVER : Learning to verify language-to-code generation with execution. In Proceedings of ICML, 2023. URL: https://proceedings.mlr.press/v202/ni23b.html

    work page 2023

  50. [50]

    Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

    Yujia Li et al. Competition-level code generation with AlphaCode . Science, 378(6624):1092--1097, 2022. DOI: https://doi.org/10.1126/science.abq1158

    work page doi:10.1126/science.abq1158 2022

  51. [51]

    Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. arXiv:2310.04406, 2023. URL: https://arxiv.org/abs/2310.04406

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  52. [52]

    Relevance isn't all you need: Scaling RAG systems with inference-time compute via multi-criteria reranking

    Will LeVine and Bijan Varjavand. Relevance isn't all you need: Scaling RAG systems with inference-time compute via multi-criteria reranking. arXiv:2504.07104, 2025. URL: https://arxiv.org/abs/2504.07104

    work page arXiv 2025