Pith. sign in

REVIEW 3 major objections 4 minor 35 references

Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Tool-augmented LLM judges can outperform plain judges on factual and coding response pairs.

desk verdict Solid empirical study of tool-augmented judges; the headline improvement is real but tied to constructed labels with significant noise, so the central claim should be scoped carefully. read the letter →

arxiv 2507.17015 v1 pith:VDRCM53R submitted 2025-07-22 cs.CL cs.AI

Arduin Findeis , Floris Weers , Guoli Yin , Ke Ye , Ruoming Pang , Tom Gunter This is my paper
classification cs.CLcs.AI
keywords LLM-as-a-judgepairwisepreferenceannotationexternalvalidationtoolsweb-searchfact-checkingcodeexecutionAIannotatorevaluationrewardbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether giving LLM-as-a-judge annotators external validation tools — web search for facts, code execution for code and math — improves their pairwise preference judgments on response domains where plain judges struggle. It argues yes for long-form factual and advanced coding responses, with mixed results for math, and only about 2% agreement regression on out-of-domain chat and safety tasks. To test this, the authors build an agentic evaluation framework that decides when tools are useful, runs them per response, and feeds the outputs into a final preference decision, reverting to a baseline annotator when no tool applies. They introduce three new pairwise datasets (long-form factuality, APPS competition coding, hard GSM8k math) alongside RewardBench evaluations. The central claim is that grounding judgments in external validation, rather than the LLM's internal knowledge, can improve annotation quality in targeted domains.

What carries the argument

The load-bearing mechanism is the evaluation agent: an LLM-driven pipeline with (1) an initial domain assessment that decides per response whether fact-check, code-execution, or math-checking tools are likely to help, (2) tool execution — fact-checking adapted from the SAFE algorithm, and code/math checks run through a code interpreter — and (3) a final assessment that combines tool outputs with the original prompt and responses into a preference judgment with chain-of-thought reasoning. A critical design choice is the revert rule: when no tool is deemed useful, the agent falls back to the baseline annotator, which keeps out-of-domain regression near 2% and limits unnecessary cost. The tools supply external evidence independent of the LLM's internal knowledge, which is what lets the agent override stylistic or self-preferential biases.

What would settle it

Re-run the comparison on a LongFact-style pairwise set whose labels are independently verified by expert fact-checkers rather than constructed by the authors; if the tool-augmented judge does not beat its baseline on those expert labels (or if, on APPS pairs rewritten so correct and incorrect solutions are stylistically matched, the code-execution agent's gain vanishes), the central claim that external tools improve annotation quality would be falsified.

Watch

Extended reading notes

Core claim

The discovery is that frozen LLM annotators, when wrapped in scaffolding that lets them fact-check via web search and verify via code execution, agree with ground-truth preferences substantially more often on long-form factual responses (e.g., 63% to 81% for a GPT-4o pick-best baseline) and on advanced APPS coding responses (e.g., 31% to 71% for GPT-3.5-Turbo, 38% to 72% for ArenaHard), while causing less than 2% regression on RewardBench chat, chat-hard, and safety categories. The paper also finds that plain baseline annotators perform below random on APPS competition pairs, preferring GPT-4-generated incorrect solutions over correct ground-truth solutions, while the code-execution agent does not show that bias. On hard GSM8k math the tool agent beats some baselines but not the strongest one (ArenaHard), though on the harder RewardMATH benchmark the agent outperforms both tested baselines.

Load-bearing premise

The load-bearing assumption is that the constructed pairwise ground-truth labels reliably reflect which response is better, even though human annotators agree with only 76.83% of the LongFact pairwise labels on non-tie judgments and the APPS pairs may carry style differences between correct and incorrect solutions.

Editorial extensions

If this is right

  • Long-form factual annotation can be delegated to tool-augmented AI judges with higher agreement than the same frozen judge without tools, including on pairs where construction-based labels are hard for non-expert humans.
  • Code correctness preferences in competition-level APPS are currently misjudged by leading plain annotators, and code-execution grounding corrects this bias; this suggests reward modeling and evaluation pipelines for code should include execution feedback.
  • Because the agent reverts to baseline when tools do not apply, adding targeted tools need not degrade general chat/safety annotation by more than about 2%.
  • The sensitivity of plain judges to prompt configuration (63% to 78% for GPT-4o on long-form factuality) means improvements from tools must be measured against strong baselines, not just simple pick-best prompts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If these results hold, a practical recipe emerges: keep a strong frozen judge as the default, add narrow, high-precision tool triggers for fact-dense and code domains, and reserve human annotation for tie-heavy or safety-critical pairs; the paper's revert design already embodies this but does not spell it out as a deployment rule.
  • The APPS below-random baseline suggests that current AI-preference datasets in code may carry self-enhancement bias; one testable extension is to audit existing code preference datasets for whether the 'preferred' response is systematically the judge-model's own style, and to re-score them with execution-based judges.
  • The mixed math results hint that code-execution tools help most when the judge must verify a concrete answer, and less when reasoning chains differ stylistically; a testable extension is to condition math-tool use on whether responses contain explicit derivations versus final answers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an agentic framework ('Evaluation Agent') that augments standard LLM-as-a-Judge baselines with external validation tools: a web-search-based fact checker, a code-execution tool, and a math checker. The system first assesses whether a response falls into a supported domain, runs the selected tools, and then merges the tool outputs into a final pairwise preference decision, reverting to the baseline annotator when no tool is selected. The authors evaluate the framework on three newly constructed pairwise datasets (LongFact pairwise, APPS competition pairwise, GSM8k hard pairwise) as well as on RewardBench, RewardMath, and TruthfulQA, reporting mean agreement over five seeds. The central observations are that external tools improve agreement on long-form factual responses (e.g., 63% to 81% for GPT-4o pick-best) and on advanced coding responses (e.g., 31% to 71% for GPT-3.5-Turbo), while causing only modest out-of-domain regression (at most ~2%), with mixed results on math tasks.

Significance. If the claims are confirmed, the paper makes a useful and timely contribution: it provides an open-source, extensible framework for grounding LLM-as-a-Judge annotations in external validation, together with an unusually thorough evaluation that includes multiple baselines, five seeds, out-of-domain checks, a tool-enabled API ablation (Appendix G), and full prompt disclosure. The finding that simple scaffolding matters (Appendix G) and that existing annotator benchmarks are saturated (Appendix C) is valuable for the community. However, the reliability of the newly constructed ground-truth labels is the pivotal unresolved issue: with LongFact human agreement of only 76.83% on non-tie judgments and an acknowledged style confound in APPS, the reported accuracy numbers are not yet a clean measure of annotation quality. The quantitative conclusions should therefore be treated as indicative rather than definitive until the ground-truth labels are independently adjudicated.

major comments (3)
  1. [Section 4.1 / Appendix K] The LongFact pairwise ground-truth labels are not reliable enough to support the paper's central claim. The authors report that human annotators agree with only 76.83% of the constructed labels on non-tie judgments and mark 18% of responses as ties. Since the evaluation metric is agreement with these constructed labels, a tool that specifically detects the injected errors (wrong dates, names, or numbers) can inflate reported accuracy without improving general preference annotation. Observation 3 compares the agent directly against the same human annotators, so the comparison inherits the label noise: lower human agreement may reflect label ambiguity or error rather than agent superiority. I ask for an independent expert adjudication of a held-out sample, or at least a detailed error analysis demonstrating that the constructed labels are correct in cases where humans disagree with the agent.
  2. [Section 4.3.3 / Observation 6] The APPS pairwise construction has a style confound that undermines the code-annotation claim. The paper states that "there may be slight style differences between correct and incorrect responses," and the baseline annotators indeed score only 26-42%, apparently preferring GPT-4-style generated code. Because the code-execution tool keys on actual correctness, the agent's 71-72% accuracy may reflect detection of the construction artifact rather than improved annotation quality on realistic coding-response pairs, where style and correctness are not so cleanly separated. To support the claim that external validation improves code annotation, please control for style (e.g., generate incorrect solutions that closely match the style of correct solutions) or add human/expert adjudicated labels on a sample of pairs and report the agent's performance on those pairs.
  3. [Section 3 / Appendix N] The method contains several free parameters that are neither swept nor justified: the confidence threshold for tool activation, the 50% tie-case revert probability when only one text is deemed tool-worthy, the maximum of three web searches per fact, and the separate math checker selected after unreported preliminary tests. The random 50% revert rule is particularly concerning because it introduces stochasticity into the final annotation path and could materially affect the seed-averaged means. Without a sensitivity analysis or clear criteria for these choices, it is difficult to separate the method's core benefit from implementation tuning; the current reporting makes the central conclusion dependent on several unreported design decisions.
minor comments (4)
  1. [Appendix M, Listing 4] The code snippets in the APPS example contain truncated expressions (e.g., "x[: n %" and "b[i %"), which make the example unverifiable; please ensure the listings are syntactically complete.
  2. [Section 4.2 / Appendix H] The main text says results are based on 5 seeds unless otherwise specified, but the RewardMath results in Appendix H use a single seed for the agent. Please state this explicitly wherever the RewardMath table is discussed.
  3. [Figure 9 caption] The caption says "a drop of up to 9% points"; the correct term is "9 percentage points."
  4. [Appendix N.1] There is a misspelling of "assessment" as "asessment" in the prompt listing; please check for similar typos throughout the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular step: all headline claims compare agent predictions against independently constructed ground-truth labels; no parameter fitted to those labels is later called a prediction.

full rationale

The paper's central claims (Observations 1, 4, 5, and 7) are empirical comparisons of agreement with ground-truth labels. For LongFact pairwise, the labels come from manually injected factual errors and the fact-check tool uses web search, not the labels. For APPS pairwise, the labels come from unit-test-verified ground-truth solutions and the code-execution tool runs and tests code, not the labels. For GSM8k hard, the labels come from gold answers versus known-wrong GPT-4o outputs, and the math checker executes code. In none of these cases is the agent given the ground-truth label as input, so the measured agreement is not a tautology. The only notable design choices, the 50% tie-case revert probability and the creation of a separate math checker, are fixed engineering decisions rather than parameters fitted to the evaluation labels; they are not renamed as predictions. The paper contains no load-bearing self-citation and imports no uniqueness theorem from the authors' prior work; its external anchors (SAFE, LongFact, APPS, GSM8k) are third-party resources. The Limitations section and Appendix J acknowledge label-noise and style-confound concerns, but these are data-quality/validity issues, not circular derivation steps. The evaluation is therefore self-contained against external benchmarks, and no circularity is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities or forced parameters in a derivational sense; the central claim rests on the reliability of constructed datasets and the validity of external tool signals, plus several hand-chosen thresholds and design choices.

free parameters (4)
  • Domain assessment confidence threshold = Not specified in the paper (presumably a manual threshold on a 0-5 confidence scale)
    The initial domain assessment asks the LLM to rate whether web search will help (0-5), and the tool is triggered for values above some threshold. The threshold is never given, indicating hand-tuning that affects tool activation and results.
  • Tie-case revert probability = 0.5
    When only one text is deemed suitable for tools, the framework reverts to the baseline annotator with 50% probability and otherwise uses the agent. This stochastic mixing is arbitrary and not justified by data.
  • Max web searches per fact = 3 (default)
    The fact-checking tool limits web searches per atomic fact to 3, affecting recall of evidence. This is a hand-chosen cap.
  • Selection of separate math checker = ad hoc
    A dedicated math checker was created after preliminary tests showed the standard code interpreter transferred poorly to math. This design decision was made based on performance on math evaluation data, likely the same datasets later reported.
assumptions (4)
  • domain assumption Agreement with ground-truth annotations is a valid proxy for annotation quality.
    All reported metrics are accuracy against constructed or existing labels. If labels are noisy or biased (e.g., LongFact human agreement is only 76.83%), the metric may mislead. This is invoked throughout Section 4.
  • domain assumption Web search results and code interpreter outputs provide reliable external validation signal.
    The tools ground the final decision, but the fact-checking tool ultimately relies on an LLM to judge the search results, and code execution may have false results. Invoked in Section 3, Tool A and B.
  • domain assumption The newly created datasets (LongFact pairwise, APPS competition pairwise, GSM8k hard pairwise) have correct ground-truth preferences.
    Labels are based on manual error insertion or known correctness; however, human annotators disagree with LongFact labels 23.17% and APPS pairs have style differences. Introduced in Section 4.1.
  • domain assumption The initial domain assessment correctly identifies when tools are useful.
    The agent relies on an LLM to decide tool activation; manual inspection found wrong tool selection in 9 of 30 failure cases. Invoked in Step 1 of Section 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?." pith.science (2026). https://pith.science/paper/VDRCM53R

@misc{pith2026250717015,
  author       = {Pith},
  title        = {Pith review of: Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VDRCM53R}},
  note         = {Machine review of arXiv:2507.17015}
}
read the original abstract

Pairwise preferences over model responses are widely collected to evaluate and provide feedback to large language models (LLMs). Given two alternative model responses to the same input, a human or AI annotator selects the "better" response. This approach can provide feedback for domains where other hard-coded metrics are difficult to obtain (e.g., chat response quality), thereby helping model evaluation or training. However, for some domains high-quality pairwise comparisons can be tricky to obtain - from AI and humans. For example, for responses with many factual statements, annotators may disproportionately weigh writing quality rather than underlying facts. In this work, we explore augmenting standard AI annotator systems with additional tools to improve performance on three challenging response domains: long-form factual, math and code tasks. We propose a tool-using agentic system to provide higher quality feedback on these domains. Our system uses web-search and code execution to ground itself based on external validation, independent of the LLM's internal knowledge and biases. We provide extensive experimental results evaluating our method across the three targeted response domains as well as general annotation tasks, using RewardBench (incl. AlpacaEval and LLMBar), RewardMath, as well as three new datasets for domains with saturated pre-existing datasets. Our results indicate that external tools can indeed improve performance in many, but not all, cases. More generally, our experiments highlight the sensitivity of performance to simple parameters (e.g., prompt) and the need for improved (non-saturated) annotator benchmarks. We share our code at https://github.com/apple/ml-agent-evaluator.

Figures

Figures reproduced from arXiv: 2507.17015 by the authors.

Figure 1
Figure 1. Summary of our approach and results: We extend standard LLM-as-a-Judge baselines with external validation tools based on web-search and code execution. We observe that the resulting system is often, but not always, able to improve performance (measured as agreement with ground-truth annotations) across a range of response domains that are typically challenging for LLM-as-a-Judge systems: (1) long-form factual, (2) a… view at source ↗
Figure 2
Figure 2. Overview of our tool-using AI annotator architecture, referred to as Evaluation Agent. In the (1) initial domain assessment the appropriate tools are selected for each response (e.g., for a wiki-style text the fact check tool); then, in (2) tool usage, each selected tool is run and the tool outputs are combined into a single prompt to make a (3) final decision. If none of the tools are selected (i.e., no tool deemed… view at source ↗
Figure 3
Figure 3. Detailed overview of our evaluation agent: the model responses are first processed by the (1) initial domain assessment, where an LLM is prompted to answer questions about the response text. In (2) tool usage, each tool that is deemed useful in Step (1) is run. Initially, available tools include fact check, code exec and math exec. The first tool is based on web-search, the latter two tools on a code interpreter. Fi… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Long-form fact checking results on Long￾Fact pairwise data. We augment multiple baseline an￾notators (light green) with our evaluation agent frame￾work (dark green) and observe that our agents have higher average agreement with ground-truth annotations across baselines…
Figure 5
Figure 5. Figure 5: Results annotating responses on our pair￾wise set of mathematical tasks based on GSM8k. We observe that our method improves performance over some baselines, but the overall level of agreement re￾mains relatively low (around 56%). Further work is needed to improve the m…
Figure 7
Figure 7. Figure 7: General out-of-domain annotation capabil￾ities result based on RewardBench (Lambert et al., 2024). We observe that our agent achieves similar per￾formance to the baseline annotator across these tasks — at worst seeing a reduction of ∼2% in agreement. Future work may be…
Figure 8
Figure 8. Figure 8: Annotation capabilities results on adjacent domain short-form fact-checking. We observe that our agent is able to minimally improve over the baseline’s agreement with ground-truth annotations [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Average results on RewardBench’s code task subsets based on HumanEval in different programming languages. We see a drop of up to 9% points across baselines. The noise or variability added by the code interpreter pipeline may be partially to blame for the decrease in ag…
Figure 10
Figure 10. Figure 10: Results on RewardBench’s math tasks. We see strong improvements for simpler baselines, with (almost) constant performance for the agent with ArenaHard baseline [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Annotation results of standard GPT-4o with tools enabled on our pairwise LongFact dataset. We also include the other results shown in the paper alongside the new baselines [PITH_FULL_IMAGE:figures/full_fig_p017_11.png]
Figure 12
Figure 12. Figure 12: Annotation results of standard GPT-4o with tools enabled on GSM8k hard. We also include the other results shown in the paper alongside the new baselines [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Annotation results of standard GPT-4o with tools enabled on APPS coding tasks. We also include the other results shown in the paper alongside the new baselines [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Annotation results of standard GPT-4o with tools enabled on Rewardbench out-of-domain tasks. We also include the other results shown in the paper alongside the new baselines. Fact ( [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 6 canonical work pages

  1. [1]

    Bowman, Zac Hatfield-Dodds , Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson , Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, ...

    arXiv doi:10.48550/arxiv.2212.08073 2022

  2. [2]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andr \'e F. T. Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K. Surikuchi, Ece Takmaz, and Alberto Testoni. 20...

    arXiv doi:10.48550/arxiv.2406.18403 2024

  3. [3]

    Nitay Calderon, Roi Reichart, and Rotem Dror. 2025. https://doi.org/10.48550/arXiv.2501.10970 The Alternative Annotator Test for LLM-as-a-Judge : How to Statistically Justify Replacing Human Annotators with LLMs . Preprint, arXiv:2501.10970

    arXiv doi:10.48550/arxiv.2501.10970 2025

  4. [4]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. https://doi.org/10.48550/arXiv.2308.07201 ChatEval : Towards Better LLM-based Evaluators through Multi-Agent Debate . Preprint, arXiv:2308.07201

    arXiv doi:10.48550/arxiv.2308.07201 2023

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv doi:10.48550/arxiv.2107.03374 2021

  6. [6]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. https://doi.org/10.48550/arXiv.2403.04132 Chatbot Arena : An Open Platform for Evaluating LLMs by Human Preference . Preprint, arXiv:2403.04132

    arXiv doi:10.48550/arxiv.2403.04132 2024

  7. [8]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021 b . https://arxiv.org/abs/2110.14168 Training Verifiers to Solve Math Word Problems . Preprint, arXiv:2110.14168

    arXiv 2021

  8. [9]

    Hashimoto

    Yann Dubois, Bal \'a zs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. https://doi.org/10.48550/arXiv.2404.04475 Length- Controlled AlpacaEval : A Simple Way to Debias Automatic Evaluators . Preprint, arXiv:2404.04475

    arXiv doi:10.48550/arxiv.2404.04475 2024

Show all 35 references
  1. [10]

    Liang, and Tatsunori B

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S. Liang, and Tatsunori B. Hashimoto. 2023. AlpacaFarm : A Simulation Framework for Methods that Learn from Human Feedback . Advances in Neural Information Processing Sy...

    2023

  2. [11]

    Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu

    Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. 2023. https://doi.org/10.48550/arXiv.2210.08726 RARR : Researching and Revising What Language Models Say , Using Langu...

    2023 doi arXiv

  3. [12]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. CRITIC : Large Language Models Can Self-Correct with Tool-Interactive Critiquing . In The Twelfth International Conference on Learning Representations

    2023

  4. [13]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2105.09938 Measuring Coding Challenge Competence With APPS . Preprint, arXiv:2105.09938

    2021 arXiv

  5. [14]

    Tom Hosking, Phil Blunsom, and Max Bartolo. 2024. https://doi.org/10.48550/arXiv.2309.16349 Human Feedback is not Gold Standard . Preprint, arXiv:2309.16349

    2024 doi arXiv

  6. [15]

    Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, and Jinyoung Yeo. 2024. https://doi.org/10.48550/arXiv.2410.01729 Evaluating Robustness of Reward Models for Mathematical Reasoning

    2024 doi arXiv

  7. [16]

    Hannah Rose Kirk, Alexander Whitefield, Paul R \"o ttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, Bertie Vidgen, and Scott A. Hale. 2024. https://arxiv.org/abs/2404.16019 The PRISM Alignment Project : What Participatory ...

    2024 arXiv

  8. [17]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. https://doi.org/10.48550/arXiv.2403.13787 RewardBench : Evaluating Reward Models fo...

    2024 doi arXiv

  9. [18]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. 2024. https://doi.org/10.48550/arXiv.2309.00267 RLAIF vs. RLHF : Scaling Reinforcement Learning from Human Fe...

    2024 doi arXiv

  10. [19]

    Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. 2024 a . https://doi.org/10.48550/arXiv.2310.01045 Tool- Augmented Reward Modeling . Preprint, arXiv:2310.01045

    2024 doi arXiv

  11. [20]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024 b . https://doi.org/10.48550/arXiv.2406.11939 From Crowdsourced Data to High-Quality Benchmarks : Arena-Hard and BenchBuilder Pipeline . Preprint, arXiv:2406.11939

    2024 doi arXiv

  12. [21]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's Verify Step by Step . Preprint, arXiv:2305.20050

    2023 arXiv

  13. [22]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.48550/arXiv.2109.07958 TruthfulQA : Measuring How Models Mimic Human Falsehoods . Preprint, arXiv:2109.07958

    2022 doi arXiv

  14. [23]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://doi.org/10.48550/arXiv.2303.16634 G- Eval : NLG Evaluation using GPT-4 with Better Human Alignment . Preprint, arXiv:2303.16634

    2023 doi arXiv

  15. [24]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2305.14251 FActScore : Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation . Preprint,...

    2023 arXiv

  16. [25]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  17. [26]

    Bowman, and Shi Feng

    Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. https://doi.org/10.48550/arXiv.2404.13076 LLM Evaluators Recognize and Favor Their Own Generations . arXiv preprint

    2024 doi arXiv

  18. [27]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. https://doi.org/10.48550/arXiv.2305.18290 Direct Preference Optimization : Your Language Model is Secretly a Reward Model . arXiv preprint

    2023 doi arXiv

  19. [28]

    Christiano

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

    2020

  20. [29]

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large Language Models are Inconsistent and Biased Evaluators . arXiv preprint

    2024

  21. [30]

    Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis. 2024. https://doi.org/10.48550/arXiv.2404.18796 Replacing Judges with Juries : Evaluating LLM Generations with a Panel of Diverse ...

    2024 doi arXiv

  22. [31]

    Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. 2024. https://doi.org/10.48550/arXiv.2311.090...

    2024 doi arXiv

  23. [32]

    Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024. https://doi.org/10.48550/arXiv.2403.18802 Long-form factuality in large language models . Preprint, arXiv:2403.18802

    2024 doi arXiv

  24. [33]

    Zhen Yang, Ming Ding, Qingsong Lv, Zhihuan Jiang, Zehai He, Yuyi Guo, Jinfeng Bai, and Jie Tang. 2023. https://arxiv.org/abs/2309.03241 Gpt can solve mathematical problems without a calculator . Preprint, arXiv:2309.03241

    2023 arXiv

  25. [34]

    Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. https://doi.org/10.48550/arXiv.2310.07641 Evaluating Large Language Models at Evaluating Instruction Following . Preprint, arXiv:2310.07641

    2024 doi arXiv

  26. [35]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://doi.org/10.48550/arXiv.2306.05685 Judging LLM-as-a-Judge with MT-Bench and Chatbot ...

    2023 doi arXiv

  27. [36]

    Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and J \"u rgen Schmidhuber. 2024. https://doi.org/10.48550/arXiv.2410.10934 Agent-as-a-...

    2024 doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.