Pith. sign in

REVIEW 4 major objections 4 minor 6 cited by

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-as-a-judge methods can match human evaluators on code translation and code generation, but not on code summarization.

desk verdict A genuinely useful first meta-evaluation of LLM judges on hard SE tasks, but the 'near-human' claim compares the wrong bars and needs a major statistical rewrite. read the letter →

arxiv 2502.06193 v3 pith:N7HQU4CC submitted 2025-02-10 cs.SE cs.AI

Ruiqi Wang , Jiyu Guo , Cuiyun Gao , Guodong Fan , Chun Yong Chong , Xin Xia This is my paper · ORCID
classification cs.SEcs.AI
keywords LLM-as-a-judgehumanevaluationcodetranslationgenerationsummarizationmeta-evaluationempiricalstudysoftwareengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether LLM-as-a-judge methods can replace human evaluators for software engineering outputs. It builds a meta-evaluation dataset of 450 code responses across translation, generation, and summarization with two-aspect human scores, then measures how well nine LLM-judge variants and five conventional metrics correlate with those human judgments. The central finding is that output-based judges powered by large LLMs reach Pearson correlations of 81.32 and 68.51 with human scores on code translation and code generation, close to the human-human agreement of 85.86 and 79.70, and far above the best conventional metric. On code summarization, however, every studied method fails to reach even moderate alignment, so the paper concludes that LLM judges can potentially replace humans in certain SE tasks but not across the board.

What carries the argument

The instrument that carries the study is a meta-evaluation protocol: 450 (instruction, response) triplets, 150 per task, each scored by two human evaluators on two aspects plus an overall 1-5 score, with human-human agreement (Pearson 85.86, 79.70, 73.74 across tasks) serving as the ceiling. Nine judge methods are grouped into three families by the LLM feature they consume, namely embedding similarities (BERTScore, MoverScore), token log-probabilities (GPTScore, FFLM), and prompted output judgments (Vanilla, G-Eval, BatchEval, plus the fine-tuned Auto-J and Prometheus 2), and are compared against five conventional metrics. Alignment is measured by Pearson, Spearman, and Kendall correlations between machine scores and human scores, and the decisive contrast is between reference-free output-based judges, which can exploit the LLM's own code knowledge, and reference-based similarity metrics, which are hurt when response and reference differ in implementation style.

What would settle it

Rerun the code-generation meta-evaluation on instructions with more than five dependencies or without GPT-4o-generated dependency descriptions; if output-based judges' Pearson correlation with human scores drops below the conventional metric, the near-human claim fails on the original benchmark. A second decisive check would reuse the same judges on a fresh translation dataset whose responses are not near-lexical copies of the references, since the paper's own analysis attributes translation success to the near-copy relationship.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that output-based LLM-as-a-judge methods, prompts that ask a large language model to directly output a score with evaluation aspects and criteria, achieve near-human alignment with human judgments when scoring code translation and code generation responses, clearly surpassing conventional lexical metrics. BatchEval reaches a Pearson correlation of 81.32 on translation and 68.51 on generation, while the best conventional metric, ChrF++, reaches only 34.23 and 64.92. These methods also produce score distributions with means and variances that resemble human score patterns, unlike embedding-based and probability-based methods, and unlike fine-tuned judge LLMs such as Auto-J and Prometheus 2, which remain below large general-purpose LLMs. The paper further finds that pairwise comparison prompting is unreliable: accuracy hovers near chance on generation and summarization, and judgments flip when response order is swapped. The conclusion is therefore task-dependent: LLM judges can stand in for humans on translation and generation, but not on summarization, where conventional metrics are still superior.

Load-bearing premise

The results rest on the assumption that the filtered and augmented instruction sets used for the meta-evaluation represent realistic SE evaluation workloads; the authors discarded short summaries, limited code-generation tasks to at most five dependencies, and had GPT-4o generate dependency descriptions that were manually checked for leaks, so any shift these edits cause in task difficulty or content distribution would stop the measured human-alignment numbers from generalizing to the original benchmarks.

Editorial extensions

If this is right

  • For code translation and code generation, output-based LLM judges can serve as standalone evaluators, delivering near-human alignment even without reference answers, saving the cost of unit tests or human annotation.
  • For code summarization, LLM-as-a-judge methods should not replace conventional metrics; ChrF++-style lexical measures remain the better option there.
  • Greedy decoding with a large LLM costs less than inference strategies like G-Eval and BatchEval while staying close to their alignment, making it the practical default for individual scoring.
  • Pairwise comparison prompting is not production-ready: accuracy is near chance on generation and summarization, and order-swap inconsistency is severe, so individual scoring should be preferred.
  • Fine-tuning small judge LLMs on NLP preference data does not close the gap to large general-purpose LLMs on SE tasks, pointing to the need for SE-specific preference training sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension: apply output-based judges to code-editing or bug-fix tasks where the reference differs structurally from the response; the paper's own summarization results suggest alignment will drop sharply once near-copy relationships disappear.
  • The GPT-4o-generated dependency descriptions were created from reference answers, and although manually checked, this pipeline may leak reference-derived content into prompts; whether this inflates judges' apparent alignment on code generation is an open question the paper's design cannot exclude.
  • The 'replace human evaluators' conclusion is bounded to single-response scoring; if pairwise ranking is required, current LLM judges are worse than a simple lexical metric plus human tie-breaking would be, so the practical recommendation is to use LLM judges as first-pass scorers and reserve humans for comparisons.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper empirically evaluates whether LLM-as-a-judge methods can align with human judgments on three software engineering tasks: code translation, code generation, and code summarization. The authors collect 50 instructions per task, generate three responses per instruction using a random subset of 12 code LLMs, obtain human scores from two evaluators, and then compare nine LLM-as-a-judge variants plus conventional metrics against these human scores. They report that output-based methods using large LLMs achieve the highest correlations with human scores (Pearson R = 81.32 for translation with BatchEval, R = 68.51 for generation with DeepSeek-V2.5), which they describe as 'near-human' relative to human-human agreement (R = 85.86 and 79.70). They also analyze score distributions and pairwise comparisons. The paper concludes that state-of-the-art LLM-as-a-judge methods can potentially replace human evaluators in certain SE tasks.

Significance. This is a valuable and timely empirical study: it is the first meta-evaluation of LLM-as-a-judge methods specifically on realistic SE tasks, it includes a broad set of methods and conventional metrics, and it provides a publicly available replication package with 450 human-scored responses. The finding that output-based judges with large LLMs substantially outperform lexical metrics in code translation is useful for practitioners. However, the central 'near-human' claim is currently not established: the comparison uses an apples-to-oranges benchmark, the statistical analysis ignores clustering and reports no confidence intervals or equivalence tests, and the datasets are modified after preliminary experiments in ways that may limit external validity. The paper's empirical material is strong enough to be revised, but its headline conclusion needs re-analysis and re-framing.

major comments (4)
  1. [§4.1.3, §5.1, Table 3, footnote 10] The 'near-human' claim is based on an apples-to-oranges comparison. Human scores used as the gold standard are averages of two evaluators (§4.1.3), while the human-human reference values of R=85.86 and R=79.70 in footnote 10 are correlations between the two individual raters. An independent human rater correlated with the averaged gold standard would be expected to achieve roughly r_HH * sqrt(2/(1+r_HH)), i.e., 89.06 and 84.09 for translation and generation under a parallel-measures model, not 85.86 and 79.70. BatchEval's R=81.32 and DeepSeek-V2.5's R=68.51 therefore fall 7.74 and 15.58 points below the corrected human-level benchmark, which contradicts the abstract's 'near-human evaluation' wording. The authors should either compare LLM scores against a single human rater's scores, compare against the average-score reliability, or substantially soften the replaceability claim.
  2. [§4.1.2–§4.1.3, §4.3, Table 3] The statistical evidence for the central claim is incomplete. The 150 responses per task are clustered in 50 instructions with three responses each, but the reported correlations treat responses as independent, and Section 4.3 equates a significant p-value with a 95% confidence interval. No confidence intervals, cluster-robust standard errors, or equivalence tests are reported, so the point estimates in Table 3 cannot support a claim of equivalence with human-level performance. For instance, at the effective sample size of 50 instructions, the difference between R=81.32 and the corrected human benchmark of 89.06 is plausibly within sampling error. The paper should provide bootstrap or cluster-robust confidence intervals for every correlation in Table 3 and an explicit equivalence or non-inferiority test for the 'near-human' comparisons.
  3. [§4.1.1–§4.1.2] The datasets used are not the original benchmarks. After preliminary experiments, the authors removed code-summarization reference summaries shorter than 15 tokens and re-selected instructions, restricted code-generation instructions to at most five dependencies, and augmented dependency information with GPT-4o-generated descriptions. The paper does not report how many instructions were discarded at each step or how these changes alter task difficulty and content, and the GPT-4o-generated descriptions may advantage GPT-4o-based judges in code generation. As a result, the external validity of the conclusion that LLM-as-a-judge 'can potentially replace human evaluators in certain SE tasks' is limited to this filtered and augmented sample. Please report the full filtering flow, compare characteristics before and after filtering, and discuss or control for the confound between the judge model and the description generator.
  4. [§4.1.3, §5.3, footnotes 11 and 14] The pairwise-comparison tie thresholds are fitted to the evaluation data: the human threshold of 0.5 and the G-Eval threshold of 0.7 are selected so that ties occur for about one-third of pairs. Consequently, the Accuracy and Agreement numbers in Table 6 are not unbiased estimates of how these methods would perform with a pre-specified decision rule. The paper should report sensitivity analyses across a range of thresholds or pre-register the thresholds, and should temper the RQ3 conclusions until this is done.
minor comments (4)
  1. [Table 3 caption] The caption marks coefficients with p > 0.05 in red, but no correction for multiple comparisons is applied across the many methods and three correlation types; with 19 rows and three columns, some significant results are expected by chance.
  2. [§4.1.2] The random selection of three response generators per instruction should be seeded or the exact assignment released in the replication package so that the generated response set is reproducible.
  3. [Figure 2] The kernel density estimates omit the bandwidth and kernel choice; please report these details so that the distributions can be compared reliably.
  4. [§4.3] The statement that a p-value below 0.05 'ensures a 95% confidence interval' is a misunderstanding; a p-value is not an interval estimate and this wording should be revised.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RQ1 correlations are direct empirical measurements of LLM judgments against human scores, with no fitted parameter or self-citation chain driving the central claim.

full rationale

The paper's central claim is an empirical meta-evaluation: 450 (instruction, response, human-score) triplets are scored by multiple LLM-as-a-judge methods and conventional metrics, and the reported RQ1 numbers are Pearson/Spearman/Kendall correlations between those independently produced score sets and human scores. No parameter is fitted to the human scores and then renamed as a prediction; the judge methods are applied with fixed prompts and published inference strategies. Comparing the RQ1 correlations to human-human agreement is statistically questionable because the human gold standard is the average of two raters while the human-human correlation is between individual raters, but that is a benchmark-validity concern, not circularity. The ComplexCodeEval dataset and other benchmarks are external artifacts; the authors' overlap with ComplexCodeEval does not make those benchmark results equivalent to the present paper's conclusions. The data-dependent design choices are disclosed preprocessing decisions (reference summaries at least 15 tokens, at most five dependencies, GPT-4o-generated dependency descriptions) and the RQ3 tie thresholds, explicitly chosen to produce about one-third ties; these affect generalizability and secondary pairwise results but are not fitted parameters smuggled in as predictions of the central claim. The core correlation results are self-contained empirical measurements, not reductions to their inputs.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central RQ1 result is a direct empirical comparison and uses no fitted parameters. Two tie thresholds in the pairwise experiments are chosen to hit a target tie rate, and several domain assumptions about the human gold standard, instruction representativeness, and statistical independence underpin the analysis.

free parameters (2)
  • Human tie threshold for pairwise comparison = 0.5
    Chosen so ties occur for about a third of the response pairs per task (Section 4.1.3); affects RQ3 accuracy, not RQ1 correlations.
  • G-Eval pairwise tie threshold = 0.7
    Chosen so ties occur for about a third of the response pairs (Section 4.2.4); affects RQ3 accuracy.
assumptions (4)
  • domain assumption Average of two human evaluator overall scores is the ground-truth quality
    Section 4.1.3 treats the mean of two evaluators' scores as the human gold standard; inter-evaluator agreement is high but not perfect, and no adjudication or third evaluator is used.
  • domain assumption Post-hoc filtered and augmented instructions are representative of real-world SE tasks
    Section 4.1.2 filters reference summaries shorter than 15 tokens, limits dependencies to five, and augments dependency info with GPT-4o, changing the task distribution relative to the original benchmarks.
  • domain assumption Responses to the same instruction are independent observations for correlation analysis
    Table 3 pools 150 responses per task (three per instruction) without clustering or bootstrapping by instruction, which can inflate significance and underestimate uncertainty.
  • domain assumption GPT-4o-augmented dependency descriptions do not leak reference-answer information
    Section 4.1.2 uses GPT-4o to generate dependency descriptions from reference answers and manually examines them, but the process cannot guarantee full information isolation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering." pith.science (2026). https://pith.science/paper/N7HQU4CC

@misc{pith2026250206193,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N7HQU4CC}},
  note         = {Machine review of arXiv:2502.06193}
}
read the original abstract

Recently, large language models (LLMs) have been deployed to tackle various software engineering (SE) tasks like code generation, significantly advancing the automation of SE tasks. However, assessing the quality of these LLM-generated code and text remains challenging. The commonly used Pass@k metric necessitates extensive unit tests and configured environments, demands a high labor cost, and is not suitable for evaluating LLM-generated text. Conventional metrics like BLEU, which measure only lexical rather than semantic similarity, have also come under scrutiny. In response, a new trend has emerged to employ LLMs for automated evaluation, known as LLM-as-a-judge. These LLM-as-a-judge methods are claimed to better mimic human assessment than conventional metrics without relying on high-quality reference answers. Nevertheless, their exact human alignment in SE tasks remains unexplored. In this paper, we empirically explore LLM-as-a-judge methods for evaluating SE tasks, focusing on their alignment with human judgments. We select seven LLM-as-a-judge methods that utilize general-purpose LLMs, alongside two LLMs specifically fine-tuned for evaluation. After generating and manually scoring LLM responses on three recent SE datasets of code translation, code generation, and code summarization, we then prompt these methods to evaluate each response. Finally, we compare the scores generated by these methods with human evaluation. The results indicate that output-based methods reach the highest Pearson correlation of 81.32 and 68.51 with human scores in code translation and generation, achieving near-human evaluation, noticeably outperforming ChrF++, one of the best conventional metrics, at 34.23 and 64.92. Such output-based methods prompt LLMs to output judgments directly, and exhibit more balanced score distributions that resemble human score patterns. Finally, we provide...

Figures

Figures reproduced from arXiv: 2502.06193 by the authors.

Figure 1
Figure 1. Overview of different LLM-as-a-judge methods. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Score distributions of selected metrics. [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
Figure 3
Figure 3. Case study. The successful case from code translation is on the left while the failing case from code [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency

    cs.SE 2025-06 conditional novelty 6.0 of 10

    A two-stage detector-plus-LLM-fixer trained on a new, LLM-filtered dataset reports state-of-the-art code-comment inconsistency detection (F1 89.54%) and 18.84% relative GLEU gain in repair.

  2. Evaluating the Use of LLMs for Documentation to Code Traceability

    cs.SE 2025-06 conditional novelty 6.0 of 10

    LLMs identify documentation-to-code trace links with F1 up to 80.4%, outperforming TF-IDF, BM25, and CodeBERT, but their explanations and chain reconstructions need human oversight.

  3. Larger Is Not Always Better: Exploring Small Open-source Language Models in Logging Statement Generation

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A fine-tuned 14B small open-source model with LoRA and RAG outperforms larger proprietary LLMs on automated Java logging statement generation in AL-Bench point estimates.

  4. Spiritual-LLM : Gita Inspired Mental Health Therapy In the Era of LLMs

    cs.AI 2025-06 conditional novelty 5.0 of 10

    A Gita-based mental-health dialogue dataset helps small LLMs score higher on spirituality-oriented metrics, but the evaluation loop is largely self-referential.

  5. Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches

    cs.SE 2025-06 conditional novelty 5.0 of 10

    A ReAct agent that reads a 13,572-line Ecore model via file tools matched direct full-context prompting on accuracy for the best models while using roughly 180 times fewer prompt tokens.

  6. CRScore++: Reinforcement Learning with Verifiable Tool and AI Feedback for Code Review

    cs.SE 2025-05 conditional novelty 5.0 of 10

    A two-stage SFT+DPO pipeline using linter and code-smell tool outputs plus an LLM judge improves generated code review comments and appears to transfer from Python to Java and JavaScript, though the judge-based evalua...

Reference graph

Works this paper leans on

65 extracted references · 39 canonical work pages · cited by 6 Pith papers

  1. [1]

    Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J

    Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA086. Publication date: July 2025. ISSTA086:20 Ruiqi Wang, Jiyu Guo, Cuiyun Gao, G...

    arXiv 2021

  2. [2]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005 . Association for Computational Linguistics, 65–72

    work page 2005

  3. [3]

    Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    work page 2024

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    work page

  5. [5]

    DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao...

    arXiv 2024

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Vo...

    work page 2019

  7. [7]

    Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. ClassEval: A Manually-Crafted Benchmark for Evaluating LLMs on Class-level Code Generation. CoRR abs/2308.01861 (2023)

    arXiv 2023

  8. [8]

    Aryaz Eghbali and Michael Pradel. 2022. CrystalBLEU: Precisely and Efficiently Measuring the Similarity of Code. In 37th IEEE/ACM International Conference on Automated Software Engineering, ASE 2022, Rochester, MI, USA, October 10-14, 2022. ACM, 28:1–28:12

    work page 2022

Show all 65 references
  1. [9]

    Jia Feng, Jiachen Liu, Cuiyun Gao, Chun Yong Chong, Chaozheng Wang, Shan Gao, and Xin Xia. 2024. ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code. CoRR abs/2409.10280 (2024)

    2024 arXiv

  2. [10]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP ...

    2020

  3. [11]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. GPTScore: Evaluate as You Desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 20...

    2024

  4. [12]

    Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2024. LLM-based NLG Evaluation: Current Status and Challenges. CoRR abs/2402.01383 (2024). https://doi.org/10.48550/arXiv.2402.01383

    2024 doi arXiv

  5. [13]

    Alex Gu, Baptiste Rozière, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024

    2024

  6. [14]

    Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, ...

    2022

  7. [15]

    Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming - The Rise of Code Intelligence. CoRR abs/2401.14196 (2024)

    2024 arXiv

  8. [16]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA086. Pub...

    2021

  9. [17]

    Grundy, and Haoyu Wang

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John C. Grundy, and Haoyu Wang. 2023. Large Language Models for Software Engineering: A Systematic Literature Review. CoRR abs/2308.10620 (2023)

    2023 arXiv

  10. [18]

    Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, et al. 2024. Qwen2.5-coder technical report. CoRR abs/2409.12186 (2024)

    2024 arXiv

  11. [19]

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet Challenge: Evaluating the State of Semantic Code Search. CoRR abs/1909.09436 (2019)

    2019 arXiv

  12. [20]

    Qi Jia, Siyu Ren, Yizhu Liu, and Kenny Q. Zhu. 2023. Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 ....

    2023

  13. [21]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

  14. [22]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. CoRR abs/2405.01535 (2024)

    2024 arXiv

  15. [23]

    Kusner, Yu Sun, Nicholas I

    Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From Word Embeddings To Document Distances. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015 (JMLR Workshop and Conference Proceedings, ...

    2015

  16. [24]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  17. [25]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58t...

    2020

  18. [26]

    Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2024. Generative Judge for Evaluating Alignment. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    2024

  19. [27]

    Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svy- atkovskiy, Shengyu Fu, and Neel Sundaresan. 2022. Automating code review activities by large-scale pre-training. In Proceedings of the 30th ACM Joint European Software ...

    2022

  20. [28]

    Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu. 2023. Split and Merge: Aligning Position Biases in Large Language Model based Evaluators. CoRR abs/2310.01432 (2023)

    2023 arXiv

  21. [29]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  22. [30]

    Minqian Liu, Ying Shen, Zhiyang Xu, Yixin Cao, Eunah Cho, Vaibhav Kumar, Reza Ghanadan, and Lifu Huang. 2024. X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects. In Proceedings of the 2024 Conference of the Nor...

    2024

  23. [31]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, ...

    2023

  24. [32]

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and S...

    2021

  25. [33]

    OpenAI. 2023. GPT-4 Technical Report. CoRR abs/2303.08774 (2023). Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA086. Publication date: July 2025. ISSTA086:22 Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, Xin Xia

    2023 arXiv

  26. [34]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA . ACL, 311–318

    2002

  27. [35]

    Maja Popovic. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017. Association for Computational Linguistics, 612–618

    2017

  28. [36]

    Alec Radford. 2018. Improving language understanding by generative pre-training. (2018)

    2018

  29. [37]

    Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia...

    2024 arXiv

  30. [38]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. CoRR abs/2009.10297 (2020)

    2020 arXiv

  31. [39]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défoss...

    2023 arXiv

  32. [40]

    Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity Bias in Preference Labeling by Large Language Models. CoRR abs/2310.10076 (2023)

    2023 arXiv

  33. [41]

    Manav Singhal, Tushar Aggarwal, Abhijeet Awasthi, Nagarajan Natarajan, and Aditya Kanade. 2024. NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness. CoRR abs/2401.15963 (2024)

    2024 arXiv

  34. [42]

    Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. CoRR abs/2408.03314 (2024)

    2024 arXiv

  35. [43]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  36. [44]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 20...

    2017

  37. [45]

    Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2023. Learning Personalized Story Evaluation. CoRR abs/2310.03304 (2023)

    2023 arXiv

  38. [46]

    Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

    Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2024. Replication package for paper "Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering". https://github.com/BackOnTruck/llm-judge-empirical

    2024

  39. [47]

    Joty, and Steven C

    Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, ...

    2021

  40. [48]

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2024. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. In The Twelfth...

    2024

  41. [49]

    Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent Abilities of Large Language Mod...

    2022

  42. [50]

    Association for Computational Linguistics, 8696–8708

  43. [51]

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2024. Magicoder: Empowering Code Generation with OSS-Instruct. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024

    2024

  44. [52]

    Sahraoui

    Martin Weyssow, Aton Kamanda, and Houari A. Sahraoui. 2024. CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences. CoRR abs/2403.09032 (2024)

    2024 arXiv

  45. [53]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neur...

    2022

  46. [54]

    Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. IN- STRUCTSCORE: Towards Explainable Text Generation Evaluation with Automatic Feedback. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processi...

    2023

  47. [55]

    Weixiang Yan, Yuchen Tian, Yunzhe Li, Qian Chen, and Wen Wang. 2023. CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023 . Association for Computat...

    2023

  48. [56]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. CoRR abs/1910.03771 (2019)

    2019 arXiv

  49. [57]

    Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. BARTScore: Evaluating Generated Text as Text Generation. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual . 27...

    2021

  50. [58]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020

    2020

  51. [59]

    Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Boyuan Pan, Heda Wang, Yao Hu, and Kan Li. 2024. BatchEval: Towards Human-like Text Evaluation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Ba...

    2024

  52. [60]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information P...

    2023

  53. [61]

    Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X. In Proceedings of the 29th AC...

    2023

  54. [62]

    Meyer, and Steffen Eger

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing an...

    2019

  55. [65]

    Shuyan Zhou, Uri Alon, Sumit Agarwal, and Graham Neubig. 2023. CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023 . Asso...

    2023

  56. [2021]

    CoRR abs/2107.03374 (2021)

    Evaluating Large Language Models Trained on Code. CoRR abs/2107.03374 (2021)

    2021 arXiv

  57. [2024]

    CoRR abs/2401.04088 (2024)

    Mixtral of Experts. CoRR abs/2401.04088 (2024)

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.