Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper introduces a 5,352-pair benchmark for LLM code judging and finds thinking models outperform non-thinking ones, with all judges sensitive to response order and prompting format.

desk verdict A solid, execution-grounded benchmark for LLM-as-a-Judge in coding; the main thinking-model result holds up, but the preprocessing and difficulty-split analyses need more rigor. read the letter →

arxiv 2507.10535 v2 pith:IEWBA3OE submitted 2025-07-14 cs.CL cs.AIcs.SE

Hongchao Jiang , Yiming Chen , Yushi Cao , Hung-yi Lee , Robby T. Tan This is my paper
classification cs.CLcs.AIcs.SE
keywords LLM-as-a-judgecodegenerationrepairunittestthinkingmodelspairwisecomparisonpositionbiasbenchmarkconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether LLMs can reliably judge code, and builds a new benchmark to test it: 5,352 curated pairs across code generation, code repair, and unit test generation. Every pair contains one good and one bad response, with good/bad labels set by running against unit tests from a large competition-problem collection. Across 26 judges, the paper finds that thinking models—models that spend extra tokens reasoning before answering—clearly beat non-thinking models, with even an 8B thinking model outscoring judge-tuned models up to 70B. It also finds that judge decisions are fragile: swapping the order of the two responses changes accuracy by up to 14 percentage points. The practical upshot is that execution-free, pairwise judging with full raw responses is the most accurate prompting style, but no evaluated model is stable enough to be trusted without safeguards.

What carries the argument

The central object is CodeJudgeBench itself: each instance is a triplet of a coding instruction, a good response, and a bad response, where the good/bad label comes from execution against LiveCodeBench's unit tests for code generation and code repair, and from direct comparison with expected outputs for unit test generation. A judge is scored by whether it picks the good response, with each pair evaluated twice in both orders. This construction turns 'can an LLM judge code?' into a binary pairwise accuracy measurement, and the same triplet structure is reused for code generation, code repair, and unit test generation so the three tasks differ only in what the judge is comparing.

What would settle it

Have expert programmers independently judge a random sample of CodeJudgeBench pairs; if experts disagree with the unit-test-derived good/bad labels on more than a few percent of pairs, then the reported accuracies measure label noise, not judging ability.

Watch

Extended reading notes

Core claim

On CodeJudgeBench, thinking models substantially outperform non-thinking models on all three coding tasks, with closed-source frontier models like Gemini-2.5-Pro and Claude-4-Sonnet reaching the highest accuracy around 80-82 percent average, while most non-thinking models sit near the 50 percent random-guess baseline. The benchmark's difficulty ordering is consistent: unit-test generation is hardest to judge, code repair is easiest, and all models drop sharply on the hard split. Every model shows sensitivity to response order, up to 14 percentage points, and most also vary depending on which coding model produced the candidate code. The paper argues pairwise comparison beats pointwise scoring—largely because independent pointwise scores tie on roughly half of code pairs—and that keeping comments and reasoning in the raw response helps judges more than stripping code to its bare form.

Load-bearing premise

The benchmark's good/bad labels are whatever LiveCodeBench's unit tests (and expected test outputs) say, so if those tests are wrong, incomplete, or overfit to the candidate programs, every pair in the benchmark is mislabeled and every reported accuracy is wrong.

Editorial extensions

If this is right

  • Small thinking models such as Qwen3-8B can outperform judge-tuned models up to 70B on code judging, so reasoning ability may matter more than scale or preference fine-tuning for this task.
  • Because order swaps shift accuracy by up to 14 percentage points, any pairwise evaluation that reports one order hides a material error bar; order-averaged reporting is necessary.
  • Pairwise prompting should replace pointwise scoring for code, since pointwise judgments tie on roughly half of the pairs and lose the comparison signal.
  • Feeding judges the full raw model response, including comments and reasoning, improves accuracy over code-only or comment-stripped inputs.
  • Judge accuracy varies with which coding model wrote the candidate code, implying judges are influenced by style or formatting, not just correctness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: if position bias is inherent to these judges, Best-of-N pipelines that require multiple pairwise comparisons should calibrate per-model position bias before trusting their selection.
  • The hard split, where all models fall toward chance, may be more a measure of whether the judge recognizes a specific bug pattern than of general code understanding; a human-audited hard subset would separate those two explanations.
  • The paper's execution-free judging results suggest ranking candidates may be viable, but certification of a single code response likely still needs execution, since a judge that flips on order cannot safely bless a final answer.
  • An implicit design principle for future judge training: treat code evaluation as binary pairwise discrimination rather than scalar scoring, and train on raw model outputs with commentary intact.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CodeJudgeBench, a benchmark of 5,352 curated good/bad response pairs for evaluating LLM-as-a-Judge models on three coding tasks: code generation, code repair, and unit test generation. The responses are generated by strong LLMs and labeled as good or bad using LiveCodeBench unit tests and ground-truth outputs. The authors benchmark 26 LLM judge models and report that thinking models substantially outperform non-thinking models, that judges are sensitive to response order (swaps changing accuracy by up to 14 percentage points), that pairwise judging outperforms pointwise judging, and that using full raw responses improves judge accuracy. The dataset and code are publicly released.

Significance. If the claims hold, CodeJudgeBench is a useful and timely resource: it is larger than prior coding-judge benchmarks, covers three distinct tasks, and uses an external unit-test oracle rather than the judged models themselves to define labels, which avoids the most obvious form of circularity. The thinking-versus-non-thinking gap and the position-bias findings are practically important and actionable for anyone building LLM-based code evaluation pipelines. The main limitations are that oracle-label quality is not independently validated and that the paper's 'significant' claims are not backed by confidence intervals or significance tests; these issues are addressable without changing the benchmark's core design.

major comments (4)
  1. [§3.2–§3.4] The correctness labels are defined exclusively by LiveCodeBench unit tests (a response is good iff it passes all tests and bad iff it fails at least one, §3.2; TestGen outputs are checked against dataset ground truth, §3.4). The paper reports no human or independent verification of any sample of these labels. Because LiveCodeBench test suites can be incomplete or overfit, incorrect labels would propagate to every accuracy number, to the order-sensitivity deltas in §5.2, and to the difficulty splits in §3.5. Please provide at least a human spot-check of sampled labels with agreement statistics, or a robustness analysis that recomputes the main comparisons under plausible label-noise rates.
  2. [§5.1, Table 3, Figure 4] The abstract and §5.1 use 'significantly' to describe thinking-vs-non-thinking gaps and order-sensitivity differences, but all reported results are point estimates with no confidence intervals, bootstrap, or significance tests. Per-split sample sizes are sometimes small (e.g., TestGen easy has 184 samples in Table 1), and each pair is judged only twice with results averaged (§4.3), so the reported deltas, including the up-to-14-point order-swap effect, could be within noise. Please report confidence intervals or significance tests for the central comparisons, at minimum the thinking-vs-non-thinking gap, the order-swap deltas, and the pairwise-vs-pointwise contrast.
  3. [§3.5, Table 3] The easy/medium/hard splits are defined by the proportion of LLMs that correctly judge each sample. Table 3 then reports accuracy on these splits, and the text interprets cross-split differences as meaningful task-difficulty effects. This is partly circular: a sample is 'hard' because judges answered it incorrectly, and judges are then shown to perform worse on 'hard' samples. The difficulty labels should be grounded in an external signal (e.g., problem ratings or pass@1 rates from LiveCodeBench), or the paper should explicitly frame the splits as judge-perceived difficulty and avoid causal claims about intrinsic task difficulty.
  4. [§5.3, Table 4] The pairwise-vs-pointwise conclusion is supported only by pointwise results on the CodeGen task, and the corresponding pairwise accuracies for the same models and same sample subset are not shown. Table 4 reports correct/wrong/tie percentages, but without paired pairwise numbers it is unclear how large the claimed advantage is and how ties are scored. Please present a paired comparison on the identical sample set, including the tie-handling rule, so the two prompting schemes are compared on equal footing.
minor comments (4)
  1. [§4.3] The heading contains a typo: 'Impelmentation' should be 'Implementation'.
  2. [Table 5] The column headers of Table 5 are garbled: the FC/NC/RR labels repeat across the Gemini, Claude, and Flash splits and do not align clearly with the reported numbers, making it difficult to verify the preprocessing comparison.
  3. [Table 3 caption] The table caption says 'We highlight the best performance with bold, and the best open-source performance with underline,' but the table body contains no visible bold or underline formatting in the manuscript text, so the highlights cannot be identified.
  4. [§5.2] The Z-score normalization in Figure 5 is described in the text, but the normalization is applied within each model split; please state explicitly what the mean and standard deviation are taken over so the reader can interpret the reported Z-scores.

Circularity Check

1 steps flagged · score 1.0 of 10

Mild self-definitional difficulty split; core benchmark measurements are independent of the judges being tested.

  1. self definitional [Section 3.5 (Data Statistics), Table 1, and Table 3 difficulty columns]
    "Following [17, 54], we categorize the samples in each task into three difficulty levels: easy, medium, and hard based on the proportion of LLMs that correctly judge each sample. As pairwise judging is a binary task susceptible to random guessing, we only use top performing LLMs from both open-source and close-source for the assessment."

    The easy/medium/hard labels are defined directly from judge-model accuracy on each sample. Table 3 then reports accuracy separately on those same splits, so the monotonic decrease from easy to hard accuracy is enforced by construction rather than discovered empirically. This makes the per-difficulty columns taxonomic rather than predictive. However, the paper's headline claims—thinking models outperforming non-thinking models, response-order sensitivity, pairwise beating pointwise, and BoN results—are computed from overall paired labels and do not depend on this difficulty stratification, so the circularity is minor and not load-bearing.

full rationale

CodeJudgeBench's ground-truth labels are produced by executing generated code against LiveCodeBench's unit tests and by directly comparing generated test outputs with the dataset's expected outputs (Sections 3.2-3.4). These oracles are external to the 26 LLM-as-a-Judge models being measured, so the central accuracy numbers, the thinking-versus-non-thinking gap, response-order sensitivity, and prompting comparisons are not self-referential. No fitted parameter is relabeled as a prediction, and no load-bearing self-citation chain is used. The only circular element is the difficulty stratification in Section 3.5: because easy/medium/hard is defined as the proportion of judges that answer a sample correctly, Table 3's easy-to-hard accuracy gradient holds by definition. This affects the descriptive difficulty narrative and the per-split columns, but not the paper's main comparative findings, which remain independently meaningful.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The benchmark relies on external oracles, not on fitted parameters or on the judges' own preferences, which keeps the main measurement independent. The main assumptions are the correctness of LiveCodeBench as ground truth and the stability of single-sample judge calls. No new entities are introduced.

assumptions (3)
  • domain assumption LiveCodeBench-v6 unit tests and ground-truth outputs are correct and comprehensive oracles for code and test-case correctness.
    Section 3.2 states responses passing all unit tests are labeled good while those failing any are labeled bad; Section 3.4 states test outputs are verified by direct comparison with dataset ground truth. If these oracles are wrong or incomplete, every label in the benchmark is unreliable and the headline accuracy numbers would shift.
  • domain assumption Candidate responses from frontier LLMs are representative of realistic coding outputs and do not leak into the test set.
    Section 3.1 relies on LiveCodeBench's contamination mitigation by continual collection of new problems. The benchmark's usefulness as a progress tracker depends on this.
  • domain assumption Averaging two position-swapped evaluations per pair yields a stable estimate of judge accuracy.
    Section 4.3 says each sample is evaluated twice, with the good response first and second, and results are averaged. If judge outputs are highly stochastic, two samples may not be enough to estimate accuracy reliably.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks." pith.science (2026). https://pith.science/paper/IEWBA3OE

@misc{pith2026250710535,
  author       = {Pith},
  title        = {Pith review of: CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IEWBA3OE}},
  note         = {Machine review of arXiv:2507.10535}
}
read the original abstract

Large Language Models (LLMs) have significantly advanced the state-of-the-art in various coding tasks. Beyond directly answering user queries, LLMs can also serve as judges, assessing and comparing the quality of responses generated by other models. Such an evaluation capability is crucial both for benchmarking different LLMs and for improving response quality through response ranking. However, despite the growing adoption of the LLM-as-a-Judge paradigm, its effectiveness in coding scenarios remains underexplored due to the absence of dedicated benchmarks. To address this gap, we introduce CodeJudgeBench, a benchmark explicitly designed to evaluate the performance of LLM-as-a-Judge models across three critical coding tasks: code generation, code repair, and unit test generation. Through comprehensive benchmarking of 26 LLM-as-a-Judge models, we find that recent thinking models significantly outperform non-thinking models on our carefully designed code judging tasks. Notably, even relatively small thinking models, such as Qwen3-8B, can outperform specially trained LLM-as-a-Judge models up to 70B in size. Nevertheless, all models still exhibit significant randomness in their judgment of coding tasks. For pairwise judging tasks, simply changing the order in which responses are presented can substantially impact accuracy. In addition, when judging code and unit tests written by different LLMs, LLM-as-a-Judge models also show variance in performance. This sensitivity raises concerns about the reliability and consistency of LLM-as-a-Judge in coding scenarios. Lastly, we study optimal prompting strategies for LLM-as-a-Judge. We find that using pair-wise comparison outperforms scalar point-wise judging. Furthermore, retaining comments and reasoning in the full, unprocessed LLM response leads to improved judge performance.

Figures

Figures reproduced from arXiv: 2507.10535 by the authors.

Figure 1
Figure 1. Overview of CodeJudgeBench and comparison of LLM-as-a-Judge performance on CodeGen Task [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. We benchmark three variants of LLM-as-a [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed CodeJudgeBench. The left side illustrates the data curation process of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The performance of LLM-as-a-Judge when the correct response is presented in either position A [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: The performance of LLM-as-a-Judge on responses generated by Gemini-2.5-Pro (Gemini), Gemini [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Illustration of response after different pre [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpsLLM: Construction of Large Language Model for Software Operations with Multi-stage Learning

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    OpsLLM's pipeline (HITL data curation, SFT, GRPO RL with a domain process reward model) improves LLM accuracy on software-operations QA and RCA, especially on in-distribution root-cause-analysis tasks.

  2. LLM-as-a-Verifier: A General-Purpose Verification Framework

    cs.AI 2026-07 conditional novelty 5.0 of 10

    Expecting over scoring-token logits yields continuous, scalable verification that improves agent trajectory selection and dense RL rewards across coding, robotics, and medical benchmarks.

  3. Automatic Failure Attribution and Critical Step Prediction Method for Multi-Agent Systems Based on Causal Inference

    cs.AI 2025-09 reject novelty 5.0 of 10

    A hierarchical causal attribution framework (performance causal inversion, Shapley values, CDC-MAS step discovery) localizes failure agents and steps in LLM multi-agent systems, reporting up to 36.2% step accuracy.

Reference graph

Works this paper leans on

61 extracted references · 35 canonical work pages · cited by 3 Pith papers

  1. [1]

    Phi-4 technical report

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S´ ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024

    arXiv 2024

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Automated unit test improvement using large language models at meta

    Nadia Alshahwan, Jubin Chheda, Anastasia Finogenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. Automated unit test improvement using large language models at meta. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, page 185–196, New Yor...

    work page 2024

  4. [4]

    Claude 3.7

    Anthropic. Claude 3.7. https://www.anthropic.com/news/claude-3-7-sonnet , 2025. Accessed: 2025-5-15

    work page 2025

  5. [5]

    Claude 4

    Anthropic. Claude 4. https://www.anthropic.com/news/claude-4, 2025. Accessed: 2025-5-25

    work page 2025

  6. [6]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv: 2108.07732, 2021

    arXiv 2021

  7. [7]

    Codet: Code generation with generated tests

    Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. In ICLR, 2023

    2023

  8. [8]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

Show all 61 references
  1. [9]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Sch¨ arli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations. OpenReview.net, 2024

    2024

  2. [10]

    Rm-r1: Reward modeling as reasoning

    Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru Wang, Yu Zhang, Denghui Zhang, Tong Zhang, et al. Rm-r1: Reward modeling as reasoning. arXiv preprint arXiv:2505.02387, 2025

    2025

  3. [11]

    Acereason-nemotron: Advancing math and code reasoning through reinforcement learning

    Yang Chen, Zhuolin Yang, Zihan Liu, Chankyu Lee, Peng Xu, Mohammad Shoeybi, Bryan Catan- zaro, and Wei Ping. Acereason-nemotron: Advancing math and code reasoning through reinforcement learning. arXiv preprint arXiv: 2505.16400, 2025

    2025 arXiv

  4. [12]

    Google DeepMind. Gemini. https://deepmind.google/models/gemini/, 2025. Accessed: 2025-5-20

    2025

  5. [13]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv: 2501.12948, 2025

    2025 arXiv

  6. [14]

    Pentestgpt: An llm-empowered automatic penetration testing tool

    Gelei Deng, Yi Liu, V ´ ıctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. Pentestgpt: An llm-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782, 2023

    2023 arXiv

  7. [15]

    Codemonkeys: Scaling test-time compute for software engineering

    Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher R´ e, and Azalia Mirhoseini. Codemonkeys: Scaling test-time compute for software engineering. arXiv preprint arXiv: 2501.14723, 2025

    2025 arXiv

  8. [16]

    Scoring verifiers: Eval- uating synthetic verification for code and reasoning

    Aleksander Ficek, Somshubra Majumdar, Vahid Noroozi, and Boris Ginsburg. Scoring verifiers: Eval- uating synthetic verification for code and reasoning. arXiv preprint arXiv: 2502.13820, 2025. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks 16

    2025 arXiv

  9. [17]

    Gonzalez, and Ion Stoica

    Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. How to evaluate reward models for RLHF. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  10. [18]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  11. [19]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. arXiv preprint arXiv: 2411.15594, 2024

    2024 arXiv

  12. [20]

    From code to courtroom: Llms as the new software judges

    Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv: 2503.02246, 2025

    2025 arXiv

  13. [21]

    An empirical study on fine-tuning large language models of code for automated program repair

    Kai Huang, Xiangxin Meng, Jian Zhang, Yang Liu, Wenjie Wang, Shuhao Li, and Yuqing Zhang. An empirical study on fine-tuning large language models of code for automated program repair. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 1162–

    2023

  14. [22]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learn...

    2025

  15. [23]

    Self-planning code generation with large language models

    Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models. ACM Trans. Softw. Eng. Methodol., 33(7), September 2024

    2024

  16. [24]

    Critiquellm: Towards an informative critique generation model for evaluation of large language model generation

    Pei Ke, Bosi Wen, Andrew Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Sheng-Ping Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. Critiquellm: Towards an informative critique generation model for evaluation of large language model generation. Annual Meetin...

    2024

  17. [25]

    Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, S. Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. Conference on Empirical Methods in Natura...

    2024

  18. [26]

    Overfitting in semantics-based automated program repair

    Xuan-Bach D Le, Ferdian Thung, David Lo, and Claire Le Goues. Overfitting in semantics-based automated program repair. In Proceedings of the 40th international conference on software engineering, pages 163–163, 2018

    2018

  19. [27]

    Generative judge for evaluating alignment

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, hai zhao, and Pengfei Liu. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Representations, 2024

    2024

  20. [28]

    Competition-level code generation with alphacode

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R´ emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022

    2022

  21. [29]

    Llms for relational reasoning: How far are we? In Proceedings of the 1st International Workshop on Large Language Models for Code, pages 119–126, 2024

    Zhiming Li, Yushi Cao, Xiufeng Xu, Junzhe Jiang, Xu Liu, Yon Shin Teo, Shang-Wei Lin, and Yang Liu. Llms for relational reasoning: How far are we? In Proceedings of the 1st International Workshop on Large Language Models for Code, pages 119–126, 2024

    2024

  22. [30]

    RM-bench: Benchmarking reward models of language models with subtlety and style

    Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmarking reward models of language models with subtlety and style. In The Thirteenth International Conference on Learning Representations, 2025. CodeJudgeBench: Benchmarking LLM-as-a-Judge for Cod...

    2025

  23. [31]

    Deepcoder: A fully open-source 14b coder at o3-mini level

    Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepcoder: A fully open-source 14b coder at o3-mini level. https://pretty-radio-b75.notion.si...

  24. [32]

    Llm critics help catch llm bugs

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. Llm critics help catch llm bugs. arXiv preprint arXiv: 2407.00215, 2024

    2024 arXiv

  25. [33]

    Swt-bench: Testing and validating real- world bug-fixes with code agents

    Niels M¨ undler, Mark M¨ uller, Jingxuan He, and Martin Vechev. Swt-bench: Testing and validating real- world bug-fixes with code agents. Advances in Neural Information Processing Systems, 37:81857–81887, 2024

    2024

  26. [34]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, P. Welinder, P. Christiano, J. Leike, and Ry...

    2022

  27. [35]

    M-prometheus: A suite of open multilingual llm judges

    Jos´ e Pombal, Dongkeun Yoon, Patrick Fernandes, Ian Wu, Seungone Kim, Ricardo Rei, Graham Neubig, and Andr´ e FT Martins. M-prometheus: A suite of open multilingual llm judges. arXiv preprint arXiv:2504.04953, 2025

    2025

  28. [36]

    Codebleu: a method for automatic evaluation of code synthesis

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. arXiv preprint arXiv: 2009.10297, 2020

    2009 arXiv

  29. [37]

    Skywork critic model series

    Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. Skywork critic model series. https://huggingface.co/Skywork, September 2024

    2024

  30. [38]

    Scaling llm test-time compute optimally can be more effective than scaling model parameters

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv: 2408.03314, 2024

    2024 arXiv

  31. [39]

    Judgebench: A benchmark for evaluating LLM-based judges

    Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  32. [40]

    Code repair with LLMs gives an exploration-exploitation tradeoff

    Hao Tang, Keya Hu, Jin Peng Zhou, Si Cheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. Code repair with LLMs gives an exploration-exploitation tradeoff. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  33. [41]

    Qwq-32b: Embracing the power of reinforcement learning, March 2025

    Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025

    2025

  34. [42]

    Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering. Proc. ACM Softw. Eng., 2(ISSTA), June 2025

    2025

  35. [43]

    Self-taught evaluators

    Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint arXiv:2408.02666, 2024

    2024 arXiv

  36. [44]

    PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. PandaLM: An automatic evaluation benchmark for LLM instruction tuning optimization. In The Twelfth Inter...

    2024

  37. [45]

    Chi, Tatsunori Hashimoto, O

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. Trans. Mach. L...

    2022

  38. [46]

    Weyssow, Aton Kamanda, Xin Zhou, and H

    M. Weyssow, Aton Kamanda, Xin Zhou, and H. Sahraoui. Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences. ACM Transactions on Software Engineering and Methodology, 2024

    2024

  39. [47]

    Bloomberggpt: A large language model for finance

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023

    2023 arXiv

  40. [48]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  41. [49]

    Qwen2.5 technical report

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  42. [50]

    Code-diting: A reasoning-based metric for functional alignment in code evaluation

    Guang Yang, Yu Zhou, Xiang Chen, Wei Zheng, Xing Hu, Xin Zhou, David Lo, and Taolue Chen. Code-diting: A reasoning-based metric for functional alignment in code evaluation. arXiv preprint arXiv: 2505.19502, 2025

    2025 arXiv

  43. [51]

    Fingpt: Open-source financial large language models

    Hongyang Yang, Xiao-Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023

    2023

  44. [52]

    Demystifying long chain-of- thought reasoning in llms, 2025

    Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of- thought reasoning in llms, 2025

    2025

  45. [53]

    Acecoder: Acing coder rl via automated test-case synthesis

    Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. Acecoder: Acing coder rl via automated test-case synthesis. ArXiv, 2502.01718, 2025

    2025 arXiv

  46. [54]

    Codecriticbench: A holistic code critique benchmark for large language models, 2025

    Chenchen Zhang, Jinxiang Xia, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, Yuanxing Zhang, Zhexu Wang, Weixun Wang, Yancheng He, Ken Deng, Wangchunshu Zhou, Wenhao Huang, and Zhaoxiang Zhang. Codecriticbench: A holistic cod...

    2025

  47. [55]

    Yuwei Zhao, Ziyang Luo, Yuchen Tian, Hongzhan Lin, Weixiang Yan, Annan Li, and Jing Ma. CodeJudge-eval: Can large language models be good judges in code understanding? In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert, ...

    2025

  48. [56]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  49. [57]

    RMB: Compre- hensively benchmarking reward models in LLM alignment

    Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. RMB: Compre- hensively benchmarking reward models in LLM alignment. In The Thirteenth International Confe...

    2025

  50. [58]

    Leveraging large language model for automatic patch correctness assessment

    Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Hung Huu Nguyen, Thanh Le-Cong, Junda He, Bach Le, and David Lo. Leveraging large language model for automatic patch correctness assessment. IEEE Transactions on Software Engineering, 2024

    2024

  51. [59]

    Evaluating judges as evaluators: The JETTS benchmark of LLM-as-judges as test-time scaling evaluators

    Yilun Zhou, Austin Xu, PeiFeng Wang, Caiming Xiong, and Shafiq Joty. Evaluating judges as evaluators: The JETTS benchmark of LLM-as-judges as test-time scaling evaluators. In Forty-second International Conference on Machine Learning, 2025

    2025

  52. [60]

    JudgeLM: Fine-tuned large language models are scalable judges

    Lianghui Zhu, Xinggang Wang, and Xinlong Wang. JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  53. [1174]

    IEEE Computer Society, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.