{"id":"cb441adb-40e0-4bd9-b095-383a1a0cd91d","arxiv_id":"2502.06193","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Output-based LLM-as-a-judge methods using large LLMs achieve near-human correlation with human scores for code translation and generation, but not for code summarization or pairwise comparisons.","lead":"This paper tests whether LLM judges can grade code translation, generation, and summarization as well as human evaluators do. It finds that large output-based LLM judges match humans closely on translation and generation, but fail on summarization and on pairwise comparisons.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'near-human' claim uses an invalid benchmark: LLM scores are correlated with the average of two human raters but compared to single-rater human-human correlation, which is a lower ceiling; the implied single-rater LLM correlation is well below human-human agreement.","rationale":"The study's central claim is the 'near-human' alignment of output-based LLM judges. The strongest quantitative support is Table 3, where BatchEval/DeepSeek-V2.5 reach R = 81.32 and 68.51 versus human-human R = 85.86 and 79.70. Read in good faith, the human-labeled dataset, multi-method comparison, and public replication package are real contributions. However, the near-human inference is not supported by the reported statistics. The LLM correlations are computed against the average of the two human evaluators, while the human-human correlations are between the two individual evaluators. Averaging raters raises the reliability of the gold standard; under the classical true-score model, an independent judge with human-level reliability would be expected to correlate with the averaged gold standard at rho * sqrt(2/(1+rho)), which is 0.891 for translation and 0.841 for generation, not 0.859 and 0.797. The observed LLM correlations fall short of these corrected bars. Equivalently, the implied LLM correlation with a single human rater is about 0.784 and 0.649. Without confidence intervals or equivalence tests, the claim that these are 'near-human' is not statistically grounded. This is load-bearing because the conclusion that LLMs can potentially replace human evaluators in certain SE tasks is drawn directly from this comparison. The reader's weakest assumption about post-hoc filtering and representativeness is also valid, but the statistical comparison issue is more immediate and internal. I recommend keeping the conditional verdict: the paper needs to correct the benchmark, report uncertainty such as cluster bootstrap by instruction, and soften the conclusion, but the empirical material still merits publication after revision.","tokens_in":25046,"tokens_out":10184,"duration_ms":89224,"concrete_test":"Using the public replication package, recompute Table 3 with each individual human evaluator's raw scores as the gold standard, not their average. Report Pearson R for BatchEval/DeepSeek-V2.5 against Evaluator 1 and Evaluator 2 separately, together with the human-human R between the two evaluators and bootstrap or Fisher-z confidence intervals. If the LLM-to-single-human correlations (or their upper CIs) remain below the human-human correlation, the 'near-human' claim fails; if they overlap, the original comparison may still be salvageable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim (abstract, §5.1, Table 3) is that output-based judges reach Pearson R = 81.32 (translation) and 68.51 (generation), 'near-human' relative to human-human correlations of 85.86 and 79.70 (footnote 10). This comparison is apples-to-oranges. Per §4.1.3, the human gold standard is the average of two evaluators, while the human-human correlation is between the two individual evaluators. Averaging two noisy raters produces a more reliable target than either rater alone, so the relevant bar for an independent human-level judge is not rho_HH but rho_HH * sqrt(2/(1+rho_HH)). For translation this is 0.8586 * sqrt(2/1.8586) = 0.891; for generation, 0.7970 * sqrt(2/1.797) = 0.841. The best LLM values (0.813, 0.685) fall below these corrected benchmarks by about 7.8 and 15.6 correlation points. Equivalently, under a classical true-score model the reported LLM-to-average correlations imply LLM-to-single-human correlations of roughly 0.784 and 0.649, far below the human-human 0.8586 and 0.7970. No confidence intervals, cluster-robust standard errors, or equivalence tests are reported (n = 50 instructions, 3 responses each), so 'near-human' and 'can replace human evaluators' are not statistically established even on the filtered sample. This is load-bearing because the conclusion of replaceability rests on this near-human comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically evaluates whether LLM-as-a-judge methods can align with human judgments on three software engineering tasks: code translation, code generation, and code summarization. The authors collect 50 instructions per task, generate three responses per instruction using a random subset of 12 code LLMs, obtain human scores from two evaluators, and then compare nine LLM-as-a-judge variants plus conventional metrics against these human scores. They report that output-based methods using large LLMs achieve the highest correlations with human scores (Pearson R = 81.32 for translation with BatchEval, R = 68.51 for generation with DeepSeek-V2.5), which they describe as 'near-human' relative to human-human agreement (R = 85.86 and 79.70). They also analyze score distributions and pairwise comparisons. The paper concludes that state-of-the-art LLM-as-a-judge methods can potentially replace human evaluators in certain SE tasks.","tokens_in":25359,"tokens_out":8537,"duration_ms":71809,"significance":"This is a valuable and timely empirical study: it is the first meta-evaluation of LLM-as-a-judge methods specifically on realistic SE tasks, it includes a broad set of methods and conventional metrics, and it provides a publicly available replication package with 450 human-scored responses. The finding that output-based judges with large LLMs substantially outperform lexical metrics in code translation is useful for practitioners. However, the central 'near-human' claim is currently not established: the comparison uses an apples-to-oranges benchmark, the statistical analysis ignores clustering and reports no confidence intervals or equivalence tests, and the datasets are modified after preliminary experiments in ways that may limit external validity. The paper's empirical material is strong enough to be revised, but its headline conclusion needs re-analysis and re-framing.","major_comments":[{"comment":"The 'near-human' claim is based on an apples-to-oranges comparison. Human scores used as the gold standard are averages of two evaluators (§4.1.3), while the human-human reference values of R=85.86 and R=79.70 in footnote 10 are correlations between the two individual raters. An independent human rater correlated with the averaged gold standard would be expected to achieve roughly r_HH * sqrt(2/(1+r_HH)), i.e., 89.06 and 84.09 for translation and generation under a parallel-measures model, not 85.86 and 79.70. BatchEval's R=81.32 and DeepSeek-V2.5's R=68.51 therefore fall 7.74 and 15.58 points below the corrected human-level benchmark, which contradicts the abstract's 'near-human evaluation' wording. The authors should either compare LLM scores against a single human rater's scores, compare against the average-score reliability, or substantially soften the replaceability claim.","section":"§4.1.3, §5.1, Table 3, footnote 10"},{"comment":"The statistical evidence for the central claim is incomplete. The 150 responses per task are clustered in 50 instructions with three responses each, but the reported correlations treat responses as independent, and Section 4.3 equates a significant p-value with a 95% confidence interval. No confidence intervals, cluster-robust standard errors, or equivalence tests are reported, so the point estimates in Table 3 cannot support a claim of equivalence with human-level performance. For instance, at the effective sample size of 50 instructions, the difference between R=81.32 and the corrected human benchmark of 89.06 is plausibly within sampling error. The paper should provide bootstrap or cluster-robust confidence intervals for every correlation in Table 3 and an explicit equivalence or non-inferiority test for the 'near-human' comparisons.","section":"§4.1.2–§4.1.3, §4.3, Table 3"},{"comment":"The datasets used are not the original benchmarks. After preliminary experiments, the authors removed code-summarization reference summaries shorter than 15 tokens and re-selected instructions, restricted code-generation instructions to at most five dependencies, and augmented dependency information with GPT-4o-generated descriptions. The paper does not report how many instructions were discarded at each step or how these changes alter task difficulty and content, and the GPT-4o-generated descriptions may advantage GPT-4o-based judges in code generation. As a result, the external validity of the conclusion that LLM-as-a-judge 'can potentially replace human evaluators in certain SE tasks' is limited to this filtered and augmented sample. Please report the full filtering flow, compare characteristics before and after filtering, and discuss or control for the confound between the judge model and the description generator.","section":"§4.1.1–§4.1.2"},{"comment":"The pairwise-comparison tie thresholds are fitted to the evaluation data: the human threshold of 0.5 and the G-Eval threshold of 0.7 are selected so that ties occur for about one-third of pairs. Consequently, the Accuracy and Agreement numbers in Table 6 are not unbiased estimates of how these methods would perform with a pre-specified decision rule. The paper should report sensitivity analyses across a range of thresholds or pre-register the thresholds, and should temper the RQ3 conclusions until this is done.","section":"§4.1.3, §5.3, footnotes 11 and 14"}],"minor_comments":[{"comment":"The caption marks coefficients with p > 0.05 in red, but no correction for multiple comparisons is applied across the many methods and three correlation types; with 19 rows and three columns, some significant results are expected by chance.","section":"Table 3 caption"},{"comment":"The random selection of three response generators per instruction should be seeded or the exact assignment released in the replication package so that the generated response set is reproducible.","section":"§4.1.2"},{"comment":"The kernel density estimates omit the bandwidth and kernel choice; please report these details so that the distributions can be compared reliably.","section":"Figure 2"},{"comment":"The statement that a p-value below 0.05 'ensures a 95% confidence interval' is a misunderstanding; a p-value is not an interval estimate and this wording should be revised.","section":"§4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the artifact is a clear strength. The main risk is that the headline conclusion rests on a statistical comparison that the authors have not actually performed; the revision should either provide the corrected benchmark comparison and cluster-robust intervals, or the conclusion should be downgraded to 'promising but not yet human-level.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is the first serious empirical study of LLM-as-a-judge methods on genuinely hard SE tasks, and the dataset is a real contribution. But the marquee claim—'near-human evaluation'—doesn't survive a careful look at what is being correlated with what.\n\nWhat is new and good: 450 human-scored responses across code translation, code generation, and code summarization, with 12 response LLMs and nine judge methods, plus conventional metrics, and a public replication package. The design separates embedding-, probability-, output-based, and fine-tuned judges, and the finding that output-based large-LLM judges (BatchEval, G-Eval, GPT-4o) clearly beat both lexical metrics and smaller fine-tuned judges on translation and generation is solid. The RQ3 result—pairwise comparisons are order-inconsistent—is an honest and important negative. Task-dependence is clearly shown; code summarization defeats all judges.\n\nThe soft spots, in order of severity. First, the 'near-human' comparison is invalid as reported. In Section 4.1.3, human gold scores are the average of two raters, but the human-human correlations in footnote 10 are between the two individual raters. Averaging two noisy raters raises the expected correlation of any single judge with that average to about 0.96 for translation and 0.95 for generation, not 0.8586 and 0.7970. So an LLM score of 0.813 with the average is well below a new human rater's expected 0.96, and the paper's direct comparison understates the gap. Even the less direct correction—implied LLM-to-single-human correlations around 0.78 and 0.65, versus 0.86 and 0.80—shows the central claim is not established. Second, there are no confidence intervals or cluster-robust errors; responses are nested three per instruction, with only 50 instructions per task. Third, Section 4.1.2 describes post-hoc filtering and GPT-4o-generated dependency descriptions, so the resulting instructions may not represent the original benchmarks or real use. The RQ3 instability is a known caveat, not a load-bearing flaw.\n\nFor anyone building or using LLM-as-a-judge metrics for code evaluation, this paper is a natural starting point, and the negative findings are as valuable as the positive ones. It needs revision: either re-frame the conclusion as 'correlates promisingly with averaged human scores' and add uncertainty quantification, or drop the replaceability language. I would send it to review, expecting major revision.","headline":"A genuinely useful first meta-evaluation of LLM judges on hard SE tasks, but the 'near-human' claim compares the wrong bars and needs a major statistical rewrite.","tokens_in":25932,"tokens_out":5240,"would_cite":true,"duration_ms":43425,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-as-a-judge methods can match human evaluators on code translation and code generation, but not on code summarization.","keywords":["LLM-as-a-judge","human evaluation","code translation","code generation","code summarization","meta-evaluation","empirical study","software engineering"],"falsifier":"Rerun the code-generation meta-evaluation on instructions with more than five dependencies or without GPT-4o-generated dependency descriptions; if output-based judges' Pearson correlation with human scores drops below the conventional metric, the near-human claim fails on the original benchmark. A second decisive check would reuse the same judges on a fresh translation dataset whose responses are not near-lexical copies of the references, since the paper's own analysis attributes translation success to the near-copy relationship.","tokens_in":24825,"feed_emoji":"🤖","tokens_out":5946,"duration_ms":46716,"temperature":0.7,"pith_summary":"The paper asks whether LLM-as-a-judge methods can replace human evaluators for software engineering outputs. It builds a meta-evaluation dataset of 450 code responses across translation, generation, and summarization with two-aspect human scores, then measures how well nine LLM-judge variants and five conventional metrics correlate with those human judgments. The central finding is that output-based judges powered by large LLMs reach Pearson correlations of 81.32 and 68.51 with human scores on code translation and code generation, close to the human-human agreement of 85.86 and 79.70, and far above the best conventional metric. On code summarization, however, every studied method fails to reach even moderate alignment, so the paper concludes that LLM judges can potentially replace humans in certain SE tasks but not across the board.","feed_headline":"LLM judges reach near-human scores on two code tasks","feed_subtitle":"Output-based judges score 81.32 and 68.51 Pearson correlation with human raters, beating conventional metrics.","key_machinery":"The instrument that carries the study is a meta-evaluation protocol: 450 (instruction, response) triplets, 150 per task, each scored by two human evaluators on two aspects plus an overall 1-5 score, with human-human agreement (Pearson 85.86, 79.70, 73.74 across tasks) serving as the ceiling. Nine judge methods are grouped into three families by the LLM feature they consume, namely embedding similarities (BERTScore, MoverScore), token log-probabilities (GPTScore, FFLM), and prompted output judgments (Vanilla, G-Eval, BatchEval, plus the fine-tuned Auto-J and Prometheus 2), and are compared against five conventional metrics. Alignment is measured by Pearson, Spearman, and Kendall correlations between machine scores and human scores, and the decisive contrast is between reference-free output-based judges, which can exploit the LLM's own code knowledge, and reference-based similarity metrics, which are hurt when response and reference differ in implementation style.","core_discovery":"On the paper's own terms, the discovery is that output-based LLM-as-a-judge methods, prompts that ask a large language model to directly output a score with evaluation aspects and criteria, achieve near-human alignment with human judgments when scoring code translation and code generation responses, clearly surpassing conventional lexical metrics. BatchEval reaches a Pearson correlation of 81.32 on translation and 68.51 on generation, while the best conventional metric, ChrF++, reaches only 34.23 and 64.92. These methods also produce score distributions with means and variances that resemble human score patterns, unlike embedding-based and probability-based methods, and unlike fine-tuned judge LLMs such as Auto-J and Prometheus 2, which remain below large general-purpose LLMs. The paper further finds that pairwise comparison prompting is unreliable: accuracy hovers near chance on generation and summarization, and judgments flip when response order is swapped. The conclusion is therefore task-dependent: LLM judges can stand in for humans on translation and generation, but not on summarization, where conventional metrics are still superior.","pith_inferences":["A testable extension: apply output-based judges to code-editing or bug-fix tasks where the reference differs structurally from the response; the paper's own summarization results suggest alignment will drop sharply once near-copy relationships disappear.","The GPT-4o-generated dependency descriptions were created from reference answers, and although manually checked, this pipeline may leak reference-derived content into prompts; whether this inflates judges' apparent alignment on code generation is an open question the paper's design cannot exclude.","The 'replace human evaluators' conclusion is bounded to single-response scoring; if pairwise ranking is required, current LLM judges are worse than a simple lexical metric plus human tie-breaking would be, so the practical recommendation is to use LLM judges as first-pass scorers and reserve humans for comparisons."],"forward_implications":["For code translation and code generation, output-based LLM judges can serve as standalone evaluators, delivering near-human alignment even without reference answers, saving the cost of unit tests or human annotation.","For code summarization, LLM-as-a-judge methods should not replace conventional metrics; ChrF++-style lexical measures remain the better option there.","Greedy decoding with a large LLM costs less than inference strategies like G-Eval and BatchEval while staying close to their alignment, making it the practical default for individual scoring.","Pairwise comparison prompting is not production-ready: accuracy is near chance on generation and summarization, and order-swap inconsistency is severe, so individual scoring should be preferred.","Fine-tuning small judge LLMs on NLP preference data does not close the gap to large general-purpose LLMs on SE tasks, pointing to the need for SE-specific preference training sets."],"supporting_citations":[{"why":"Defines the G-Eval output-based judge with chain-of-thought and repeated sampling, one of the two inference-strategy methods evaluated.","marker":"[31]"},{"why":"Defines BatchEval, the multi-round batched scoring approach that achieves the highest human alignment on code translation.","marker":"[56]"},{"why":"Supplies the CodeTransOcean code-translation instructions and reference answers used to build the translation meta-evaluation set.","marker":"[55]"},{"why":"Supplies the ComplexCodeEval code-generation instructions whose dependencies the authors filter and augment.","marker":"[9]"},{"why":"Supplies the CodeXGLUE code-summarization instructions; the paper's summarization failure result is measured on this dataset.","marker":"[32]"},{"why":"Provides Auto-J, one of the two fine-tuned judge LLMs whose underperformance supports the claim that NLP-trained judges lag behind large general-purpose LLMs.","marker":"[26]"},{"why":"Provides Prometheus 2, the other fine-tuned judge LLM, giving the second SFT data point on SE tasks.","marker":"[22]"},{"why":"Introduces the LLM-as-a-judge concept and its human-alignment validation paradigm, which this study transfers to software engineering.","marker":"[60]"}],"fun_headline_variants":["Output-based LLM judges match human scores on code tasks","LLM judges near-human on translation and generation","LLM-as-a-judge shines on code tasks, flops on summarization","Output-based judges beat conventional metrics on code eval","LLM judges align with humans on code translation and generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that the filtered and augmented instruction sets used for the meta-evaluation represent realistic SE evaluation workloads; the authors discarded short summaries, limited code-generation tasks to at most five dependencies, and had GPT-4o generate dependency descriptions that were manually checked for leaks, so any shift these edits cause in task difficulty or content distribution would stop the measured human-alignment numbers from generalizing to the original benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["Output-based LLM judges match human scores on code tasks","LLM judges near-human on translation and generation","LLM-as-a-judge shines on code tasks, flops on summarization","Output-based judges beat conventional metrics on code eval","LLM judges align with humans on code translation and generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1514,"prompt_tokens":1100,"completion_tokens":414,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":716,"completion_tokens_details":{"reasoning_tokens":332}},"tokens_in":716,"tokens_out":414,"duration_ms":3808,"temperature":1.0,"reasoning_tokens":332,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:26:37.357670+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the code-generation meta-evaluation on instructions with more than five dependencies or without GPT-4o-generated dependency descriptions; if output-based judges' Pearson correlation with human scores drops below the conventional metric, the near-human claim fails on the original benchmark. A second decisive check would reuse the same judges on a fresh translation dataset whose responses are not near-lexical copies of the references, since the paper's own analysis attributes translation success to the near-copy relationship.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the G-Eval output-based judge with chain-of-thought and repeated sampling, one of the two inference-strategy methods evaluated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CodeTransOcean code-translation instructions and reference answers used to build the translation meta-evaluation set."},{"cited_title":"ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code","cited_arxiv_id":"2409.10280","evidence_quote":"Supplies the ComplexCodeEval code-generation instructions whose dependencies the authors filter and augment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CodeXGLUE code-summarization instructions; the paper's summarization failure result is measured on this dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Auto-J, one of the two fine-tuned judge LLMs whose underperformance supports the claim that NLP-trained judges lag behind large general-purpose LLMs."},{"cited_title":"Xing, Hao Zhang, Joseph E","cited_arxiv_id":null,"evidence_quote":"Introduces the LLM-as-a-judge concept and its human-alignment validation paradigm, which this study transfers to software engineering."}],"review_version":1}