{"id":"a58412e7-5c55-48c1-8e9a-28484da56c13","arxiv_id":"2507.16587","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Even the best tested LLM judge, GPT-4-turbo, frequently misjudges code correctness (e.g., 50% of wrong Java functions judged correct) but agrees moderately with humans when judging code summary quality.","lead":"This study asks whether large language models can reliably judge the correctness of generated code and the quality of code summaries, comparing eight LLMs against test execution and human ratings. It finds that GPT-4-turbo is the best judge but still frequently misjudges code correctness, while agreeing moderately with humans on summary quality.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Code-generation accuracy figures hinge on CoderEval tests being a correct oracle; the paper filters only trivial cases and admits unquantified unreliable test outcomes, so the headline false-positive rates may be miscalibrated.","rationale":"The reader's CONDITIONAL verdict is appropriate. The test-suite oracle is the weakest link in the code-generation part, and it is explicitly acknowledged in Section 4; my concern is a sharper version of the reader's assumption, adding the unquantified 'unreliable test outcomes' admission in Section 3.1.2 as direct evidence that the oracle is imperfect. The summarization half is less threatened: human agreement is reported (Krippendorff's alpha 0.81 for Java and 0.69 for Python content adequacy), and the LLM-human comparison uses the median of three human ratings, a defensible though imperfect choice. Other possible concerns—prompt selection, unrepresentative long-function selection for summarization, non-independence of judgments—are secondary because the authors show prompt stability and do not rest the headline on significance tests. The proposed re-annotation is feasible with the replication package. If test labels largely agree with human judgment, the quantitative claims stand and conditional can later be lifted. If they disagree substantially, the exact false-positive rates should be revised. Either way, the qualitative conclusion that even the best tested judge is unreliable for code correctness may survive, so this is a calibration concern rather than a reason to reject the paper. No verdict change is warranted relative to the reader's CONDITIONAL.","tokens_in":27353,"tokens_out":9013,"duration_ms":97566,"concrete_test":"Stratified re-annotation: draw 50 items from each of the four GPT-4 automated-CoT cells in Fig. 1 for Java and similarly for Python (true positives, true negatives, false positives, false negatives). Give two independent human annotators with software engineering backgrounds only the CoderEval natural-language description, the function signature, and the candidate code, blind to both test outcomes and GPT-4 judgments; ask for a binary correct/wrong judgment against the description. Measure agreement between test labels and the human majority across the sampled items. If disagreement exceeds roughly 5%, recompute GPT-4's confusion matrix and Table 2 Kappa using human labels as the oracle; check whether the 50%/35% false-positive misjudgment rates and the 'GPT-4 is best' ranking survive. This quantifies the unquantified 'unreliable test outcomes' admission in Section 3.1.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central code-generation result—that GPT-4-turbo misjudges 50% of wrong Java implementations and 35% of wrong Python implementations as correct (Fig. 1)—is computed against pass/fail on the filtered CoderEval test suites. That oracle is the load-bearing condition for every accuracy and Kappa figure in Section 3.1. The quality-assurance step in Section 2.2.1 removes only three detectable failure modes: target functions that themselves fail; empty bodies that pass; and dummy returns that pass. It does not establish that all remaining pass/fail labels agree with a human reading of whether the candidate satisfies the natural-language description. The paper concedes this in Section 4 ('Using tests as a proxy for code correctness is a limitation of our study'), and Section 3.1.2 contains direct evidence that the oracle is imperfect: 'a few unreliable test outcomes led to false negatives, i.e., the implementation was actually wrong (so the LLM was right), but it was passing the CoderEval tests.' No rate is given. The same paragraph also attributes 27% of false positives to 'ambiguous requirements text,' a category in which tests and LLM can disagree because the specification is underspecified. If even a moderate fraction of test labels disagree with human judgment of correctness against the description, GPT-4's false-positive percentages and the Kappa ordering in Table 2 are miscalibrated, and the headline claim about how often the best judge 'misjudges' code would need to be re-expressed. Because the summarization result is separately grounded in human judgments, this concern is specific to the code-generation half of the paper, but that half is where the strongest negative claim lives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an empirical study of eight LLMs used as judges for two code-related tasks: code generation (judging whether a candidate implementation is correct) and code summarization (judging content adequacy, conciseness, and fluency & understandability). For code generation, the authors use 1,405 Java and 1,281 Python candidate implementations drawn from CoderEval and compare each LLM's binary correctness judgment against pass/fail on the benchmark's test suites. For code summarization, five LLMs judge 1,163 summaries (human-written and LLM-generated) for 198 Java/Python functions, and the judgments are compared to median human ratings from nine judges. The authors find that GPT-4-turbo is the best judge in both tasks, that smaller open models frequently fail to produce valid judgments or produce near-constant ratings, and that even GPT-4-turbo misjudges 50% of failing Java implementations and 35% of failing Python implementations as correct. For content adequacy in summarization, GPT-4-turbo reaches Krippendorff's alpha of 0.58 (Java) and 0.63 (Python), the strongest agreement observed. The paper also analyzes self-bias, conducts a manual qualitative analysis of misjudgments, and probes code understanding through mutation and semantically-equivalent code transformation experiments.","tokens_in":27598,"tokens_out":5680,"duration_ms":59809,"significance":"The study is carefully designed in several important respects: every extracted judgment was manually checked by two authors, four prompt variants were compared for both tasks, multiple LLM families and parameter scales were considered, and the self-bias analysis is statistically grounded. The mutation and semantically-equivalent code experiment is a welcome direct probe of whether true-positive judgments reflect code understanding rather than guessing. The public replication package and the newly constructed human-judgment dataset for code summaries are valuable community assets. Provided the test-suite oracle concern can be addressed, the paper delivers an important, largely negative result on LLM-as-a-judge for code correctness and a cautiously positive result for summary content adequacy, with clear implications for automated code review and evaluation practice.","major_comments":[{"comment":"The accuracy and Kappa figures for code generation treat the filtered CoderEval test suites as a binary correctness oracle. The quality-assurance step removes only three detectable failure modes: target functions that fail their own tests, empty bodies that pass, and dummy returns that pass. It does not establish that the remaining pass/fail labels agree with a human reading of whether the candidate satisfies the natural-language description. Section 3.1.2 itself reports \"a few unreliable test outcomes\" and attributes 27% of false positives to \"ambiguous requirements text,\" a category in which the test outcome and the LLM's judgment can legitimately disagree. Because the headline claim is the false-positive rate (50% for Java, 35% for Python for GPT-4-turbo), the paper should quantify how many pass/fail labels would change under a gold human correctness judgment, and/or re-present Fig. 1 and Table 2 after excluding cases labeled as ambiguous or unreliable. Without this, the absolute misjudgment rates are not well calibrated.","section":"Section 2.2.1 and Section 3.1 (Fig. 1, Table 2)"},{"comment":"The manual analysis of false positives and false negatives is based on a sample of 15 false positives and 15 false negatives per LLM, yet the text presents the resulting category percentages (37% uncaught wrong behavior, 32% coding context, 27% ambiguous requirements) as if they were exact proportions of the full confusion matrix. The paper should clarify that these percentages come from a small qualitative sample and should provide the confusion matrices restricted to the cases whose test outcome and requirements text are both deemed reliable, so the reader can see how much the headline rates change. This is particularly important because the authors themselves state that ambiguous-requirements cases are \"not really failures of the judging LLMs.\"","section":"Section 3.1.2"},{"comment":"The mutation and semantically-equivalent code experiment samples only true-positive instances that both GPT-4 and GPT-3.5 judged correctly. This selection conditions on the property being tested: those are the instances on which both judges already agreed, and they may be systematically easier than the full set of judged implementations. The reported rates (e.g., 79% of semantically equivalent Java code still judged correct by GPT-4) are therefore not population estimates for all true positives. The paper should either sample from all true positives or explicitly present these numbers as an upper-bound consistency check, and it should avoid generalizing them into overall code-understanding capabilities.","section":"Section 3.1.2 (Fig. 2)"}],"minor_comments":[{"comment":"There is a typo: \"190 Pythjon\" should be \"190 Python.\"","section":"Section 2.3.1"},{"comment":"There is a typo: \"The fist, named zero shot\" should read \"The first, named zero shot.\"","section":"Section 2.3.2"},{"comment":"The model name \"CodeLlalma\" appears twice and should be \"CodeLlama.\"","section":"Section 3.2 and Section 3.3"},{"comment":"For conciseness and fluency & understandability, human inter-rater agreement is only moderate (alpha between 0.56 and 0.62), so the median-human oracle is noisier for those criteria. The text could add one sentence noting that agreement with the LLM is therefore likely underestimated for those dimensions, which would contextualize the lower alpha values without changing the conclusions.","section":"Section 2.2.2 and Table 5"},{"comment":"Kappa and Krippendorff alpha values are reported without confidence intervals. Since the number of judgments differs across models because of invalid outputs, including confidence intervals would make the ranking of near-adjacent models (e.g., GPT-3.5 versus GPT-4 on Python code generation) easier to interpret.","section":"Table 2 and Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and addresses a timely topic. The core empirical apparatus is strong, and the main weakness is the unquantified dependence of the code-generation results on the test-suite oracle. I would not reject the paper on this basis, because the authors already acknowledge the limitation and partially mitigate it, but the revision should make the sensitivity of the headline rates explicit. The summarization contribution and the public dataset are likely to be of independent interest to the community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline finding is solid: GPT-4-turbo, the best judge tested, still misclassifies about half of wrong Java implementations and a third of wrong Python implementations as correct. The paper gives the most thorough evidence yet that LLM-as-a-judge is not yet trustworthy for code correctness, and the scope alone makes it worth knowing about. What's genuinely new: CoderEval with QA filtering, a fresh human-judged summarization dataset, self-bias analysis, and a mutation-based sanity check that strengthens the code-generation result.\n\nThe methodology is careful. They filter out CoderEval problems with obviously broken tests, manually verify every extracted judgment, try four prompts per task, and report the full prompt comparison. The summarization dataset is a real contribution: 1,163 summaries with three judgments each, and the human agreement is properly measured. The self-bias analysis is clean. The mutation experiment is a nice touch — GPT-4 flips its verdict on 21–36% of semantically equivalent code, which independently confirms that its correctness judgments are unstable.\n\nThe main soft spot is the one the authors admit: test suites as the oracle for code correctness. The QA step removes only trivial cases (empty bodies, dummy returns), and the paper itself notes that some unreliable tests caused false negatives. No rate is given for how often the oracle disagrees with human reading of the requirements. That means the specific false-positive percentages in Figure 1 could be miscalibrated. But note: the mutation result and the low Kappa values would survive even if some labels were flipped. The central claim is not load-bearing on oracle perfection. I'd want the authors to estimate the oracle error rate in a follow-up, but I wouldn't block the paper on it. The lack of confidence intervals on the agreement scores is a minor issue; the sample sizes are large enough that the model ranking is unlikely to change.\n\nThis is a paper for anyone building evaluation pipelines with LLM judges, and for researchers studying code generation metrics. It deserves a serious referee. I'd send it to review, ask for the oracle-error estimate and CIs, and otherwise let it through with minor revisions.","headline":"Well-executed large empirical study showing GPT-4 is a mediocre code-correctness judge and a moderate summary judge; the oracle concern is real but doesn't overturn the central finding.","tokens_in":28214,"tokens_out":1149,"would_cite":true,"duration_ms":14751,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An eight-model study finds LLM judges unreliable for code correctness but moderately aligned with humans on summaries","keywords":["LLM-as-a-judge","code generation","code summarization","code correctness","human evaluation","inter-rater agreement","empirical software engineering"],"falsifier":"Re-run the code-generation protocol on a set of candidates whose correctness is independently verified by formal specification or multiple human experts, and compare GPT-4-turbo's verdicts with that verified oracle. If agreement rises well above the reported kappa values (0.21 for Java, 0.10 for Python), the bottleneck is the test-suite oracle rather than the judge; if it stays low, the judge itself lacks the asserted reliability.","tokens_in":27149,"feed_emoji":"⚖️","tokens_out":7541,"duration_ms":74394,"temperature":0.7,"pith_summary":"This paper tests whether large language models can take over the evaluation of two generative software-engineering tasks where quantitative metrics are known to be weak proxies: judging whether generated code is correct, and judging the quality of code summaries. It asks eight LLMs to assess 2,686 code-generation candidates and five LLMs to rate 1,163 summaries, comparing their verdicts against test-suite execution and against the median rating of nine human judges. The paper finds that GPT-4-turbo is the strongest judge on both tasks, yet for code generation its agreement with the test oracle is only fair for Java and weak for Python, and it misjudges 50% of wrong Java implementations and 35% of wrong Python implementations as correct. For summarization, GPT-4-turbo reaches moderate agreement with human judges on content adequacy (Krippendorff's alpha 0.58 for Java, 0.63 for Python), while smaller models stay at chance level or fail to produce judgments. The practical point is that LLM-as-a-judge cannot yet replace tests or humans for correctness, but may already be a useful complement for summary-quality assessment.","feed_headline":"GPT-4 judge marks half of wrong Java code as correct","feed_subtitle":"Eight-model study: the best LLM judge is unreliable on code correctness but only moderately matches humans on summary quality.","key_machinery":"For code generation, the load-bearing mechanism is a pairwise verdict protocol: the judge sees only the natural-language problem description, the required function signature, and one candidate implementation, and must answer correct/wrong with a rationale; the paper then aligns that binary verdict with the pass/fail outcome of the benchmark's test suite using confusion matrices and Cohen's kappa. For summarization, the judge rates each summary on content adequacy, conciseness, and fluency/understandability on a 1-5 scale, and the ratings are compared to the median of three independent human ratings via Krippendorff's alpha. To test whether correct verdicts reflect real code understanding rather than guessing, the authors mutate 332 implementations with a mutation tool and also create semantically equivalent variants, then check whether the judges reject the mutants and still accept the equivalent code.","core_discovery":"The central claim is a measurement: among the eight models tested, the best judge is GPT-4-turbo, and even it is far from reliable in the correctness task. With the best performing prompt (automated chain-of-thought), Cohen's kappa between GPT-4-turbo's pass/fail judgment and actual test results is 0.21 for Java and 0.10 for Python; its confusion matrices show 72% of correct Java implementations accepted but only 50% of wrong ones rejected, and for Python 46% of correct implementations accepted while 35% of wrong ones are accepted. That is, the model errs in opposite directions across languages. For code summarization the same model's zero-shot ratings agree with the median human rating at Krippendorff's alpha 0.58 (Java) and 0.63 (Python) for content adequacy, with fair-to-moderate agreement on conciseness and fluency; this is the strongest agreement any tested judge achieves, and it occurs on the same quality dimension on which humans agree most among themselves. The paper also reports that smaller models (DeepSeek Coder up to 33B, CodeLlama up to 34B) either fail to understand the judging task or produce near-chance agreement, and that GPT-4 shows only negligible self-bias in code generation while all judges systematically underestimate human-written code.","pith_inferences":["The asymmetry—accepting wrong Java code while rejecting correct Python code—suggests a language-dependent response bias that a per-language calibration of the judge's decision threshold could partially correct; the paper does not test such calibration.","Because 27% of false positives were attributed to ambiguous requirements text, some judged 'misjudgments' are really specification failures; making descriptions precise might raise judge accuracy beyond what model improvements alone would yield.","One could extend the mutation analysis into a prompting strategy: since GPT-4-turbo already detects injected mutants in 88-97% of cases, pairing candidate code with a mutated version in the prompt might convert an unreliable absolute judge into a more reliable relative judge.","Another untested extension is to apply the same protocol to bug-fixing or code review tasks, where the judge's false-positive tendency would have different consequences than in generation scoring."],"forward_implications":["Automated code review and grading systems that rely on the best current LLM judge would accept about half of wrong Java functions as correct, so correctness verdicts still need tests or human inspection.","Code-generation evaluations that cannot run tests (e.g., no executable suite) are not yet safe to delegate to LLM judges; the reported kappa values are too low.","For code summarization, content adequacy scores from GPT-4-turbo align with human median scores to a moderate degree, making it a plausible complement to metrics like BLEU and ROUGE that are known to correlate weakly with human judgment.","Model size is a practical gate: judges in the 1B-34B range either fail to produce judgments or agree with humans at chance level, so the cost-saving rationale for small local judges does not hold for these tasks.","The three-human-per-summary protocol with explicit rating guidelines is what makes the summarization oracle trustworthy; replacing it with a single rating would likely lower the apparent agreement of any judge."],"supporting_citations":[{"why":"Supplies the CoderEval benchmark: 460 Java/Python problems, descriptions, signatures, target implementations, and the test suites used as the correctness oracle.","marker":"[45]"},{"why":"CodeJudge, a prior GPT-3.5-as-a-judge approach whose slow-thinking prompt is adopted as one of the four judging prompts and whose accuracy results serve as comparison.","marker":"[35]"},{"why":"ICE-Score, prior work using GPT-3.5-turbo to judge code usefulness and correctness on HumanEval-X; it provides the weak-to-moderate correlation baseline this study extends.","marker":"[49]"},{"why":"Source of the zero-shot judging prompt that asks for a rationale; the prompt family is adapted to both tasks.","marker":"[39]"},{"why":"Roy et al.'s human-evaluation dataset for code summaries; the paper rejects it as oracle due to low human agreement and outdated summaries, motivating a new dataset.","marker":"[33]"},{"why":"Cohen's kappa, the inter-rater agreement statistic used to compare judge verdicts with test outcomes.","marker":"[6]"},{"why":"Krippendorff's alpha, the agreement metric used for summarization judgments and for measuring human judge reliability.","marker":"[25]"},{"why":"Universalmutator, the mutation tool used to create buggy versions of correct implementations in the code-understanding check.","marker":"[12]"},{"why":"ProgramTransformer, whose transformation operators generate semantically equivalent programs used to test whether judges accept equivalent code.","marker":"[32]"}],"fun_headline_variants":["Even the best LLM judge accepts half of wrong Java code","GPT-4-turbo: best LLM judge but accepts 50% of wrong Java code","Best LLM judge unreliable: accepts half of wrong Java code","Even top LLM judge accepts half of wrong Java methods","GPT-4-turbo judges code poorly: accepts 50% of wrong Java"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire correctness measurement rests on treating the filtered executable test suites as the ground truth for whether a function is correct, so if those tests are incomplete or misleading the reported judge accuracy is off by an unknown amount.","fun_headline_variants_meta":{"raw":{"variants":["Even the best LLM judge accepts half of wrong Java code","GPT-4-turbo: best LLM judge but accepts 50% of wrong Java code","Best LLM judge unreliable: accepts half of wrong Java code","Even top LLM judge accepts half of wrong Java methods","GPT-4-turbo judges code poorly: accepts 50% of wrong Java"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000904,"raw_usage":{"total_tokens":4012,"prompt_tokens":1191,"completion_tokens":2821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":807,"completion_tokens_details":{"reasoning_tokens":2723}},"tokens_in":807,"tokens_out":2821,"duration_ms":20931,"temperature":1.0,"reasoning_tokens":2723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:05:39.463244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the code-generation protocol on a set of candidates whose correctness is independently verified by formal specification or multiple human experts, and compare GPT-4-turbo's verdicts with that verified oracle. If agreement rises well above the reported kappa values (0.21 for Java, 0.10 for Python), the bottleneck is the test-suite oracle rather than the judge; if it stays low, the judge itself lacks the asserted reliability.","supporting_citations":[{"cited_title":"CodeJudge: Evaluating code generation with large language models,","cited_arxiv_id":null,"evidence_quote":"CodeJudge, a prior GPT-3.5-as-a-judge approach whose slow-thinking prompt is adopted as one of the four judging prompts and whose accuracy results serve as comparison."},{"cited_title":"ICE-score: Instructing large language models to evaluate code,","cited_arxiv_id":null,"evidence_quote":"ICE-Score, prior work using GPT-3.5-turbo to judge code usefulness and correctness on HumanEval-X; it provides the weak-to-moderate correlation baseline this study extends."},{"cited_title":"Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences,","cited_arxiv_id":null,"evidence_quote":"Source of the zero-shot judging prompt that asks for a rationale; the prompt family is adapted to both tasks."},{"cited_title":"Reassessing automatic evaluation metrics for code summarization tasks,","cited_arxiv_id":null,"evidence_quote":"Roy et al.'s human-evaluation dataset for code summaries; the paper rejects it as oracle due to low human agreement and outdated summaries, motivating a new dataset."},{"cited_title":"A coefficient of agreement for nominal scales,","cited_arxiv_id":null,"evidence_quote":"Cohen's kappa, the inter-rater agreement statistic used to compare judge verdicts with test outcomes."},{"cited_title":"Reliability in content analysis: Some common misconceptions and recommendations,","cited_arxiv_id":null,"evidence_quote":"Krippendorff's alpha, the agreement metric used for summarization judgments and for measuring human judge reliability."},{"cited_title":"An ex- tensible, regular-expression-based tool for multi-language mutant generation,","cited_arxiv_id":null,"evidence_quote":"Universalmutator, the mutation tool used to create buggy versions of correct implementations in the code-understanding check."},{"cited_title":"Programtransformer: A tool for generating semantically equivalent transformed programs,","cited_arxiv_id":null,"evidence_quote":"ProgramTransformer, whose transformation operators generate semantically equivalent programs used to test whether judges accept equivalent code."}],"review_version":1}