{"id":"8f0cb939-ab36-4da0-8791-7b024c265de1","arxiv_id":"2505.20206","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4o and Gemini 2.0 Flash correctly judged code correctness in about 64-68% of cases and corrected 54-68% of faulty code, with better results when given problem descriptions.","lead":"This paper tested two AI models, GPT-4o and Gemini 2.0 Flash, as code reviewers on hundreds of Python code blocks with unit tests. The models were right about whether code worked about two-thirds of the time, and they fixed broken code about half to two-thirds of the time, which suggests AI review needs human oversight.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline numbers are measured against HumanEval's small unit-test suites that define 'correct'; if those tests under-specify intended behavior, the 68.5% accuracy and 67.8% correction ratio do not transfer to code review and need a held-out-test audit.","rationale":"Agreement: the reader's weakest assumption is the same premise I flag - unit-test-based correctness and HumanEval representativeness. I would phrase the risk as test-suite under-specification with a concrete audit, and I add the Ground Truth observation (GPT-4o at 42.07% on canonical correct code) as evidence that the setup is not a clean probe of code-review correctness. Why this is most load-bearing: every metric in Section IV is computed against this oracle, so the entire headline result inherits its weakness. It matters more than the missing human baseline or the 5.16% formatting-error handling, because those affect the magnitude of the numbers but not the interpretation of the construct. Credit: the Zenodo replication package is a genuine strength and makes the proposed audit feasible. The paper itself discloses the unit-test limitation (Section VI-D), and the reader's CONDITIONAL verdict already conditions on it, so I do not move the verdict. I would keep the condition explicit: before the 68.50%/67.83% figures are used to guide practice, a small extended-test or fresh-problem check should be reported.","tokens_in":11342,"tokens_out":11997,"duration_ms":126085,"concrete_test":"Take a random sample of 60 code blocks from the Mixed dataset and 60 from the Ground Truth dataset. For each HumanEval problem, write 5-10 additional edge-case tests (empty input, boundary values, large inputs, malformed but acceptable input) that are not in the original suite. Re-run the stored original labels and the stored LLM corrected code blocks against the extended suite. If more than ~15% of blocks previously labeled Correct or counted as successful corrections fail the extended tests, the reported accuracy and correction/regression ratios are artifacts of test-suite incompleteness and the paper should report bounds or revised numbers. If the extended-test failure rate is near zero, the unit-test oracle is adequate and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline number depends on the Section III oracle: 'Code blocks that pass all unit tests are deemed Correct.' HumanEval provides short, interview-level problems with a small number of visible tests, so both the labels on the 492 Mixed blocks and the success of an LLM-suggested fix are determined by a weak test suite. A corrected block that passes the shipped tests can still fail on inputs those tests do not cover, and an 'Incorrect' label can hinge on one edge case in the suite. The Ground Truth result is a red flag for this construct: GPT-4o, a model that is very likely trained on HumanEval solutions, classifies only 42.07% of the 164 canonical correct solutions as Correct. That is hard to read as a clean measurement of 'approval correctness'; it looks like the task is also capturing the model's propensity to propose edits and its response-format behavior. Sections VI-C and VI-D acknowledge limited representativeness and unit-test coverage, but the abstract and RQ answers still report the raw percentages as LLM code-review performance. This is the load-bearing step: if the unit-test oracle under-specifies correctness, then 68.50%, 67.83%, and the regression ratios are not estimates of the constructs the paper claims to measure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a controlled evaluation of two LLMs (GPT-4o and Gemini 2.0 Flash) on a code-review task. For each of 492 AI-generated code blocks from the Mixed dataset and 164 canonical HumanEval solutions, the models are prompted to classify the code as Correct or Incorrect and to supply a corrected code block when they judge it Incorrect. The authors define correctness by all unit tests passing, and measure Correctness Accuracy, Correction Ratio, and Regression Ratio under prompts with and without problem descriptions. They find that GPT-4o reaches 68.50% correctness accuracy and a 67.83% correction ratio on the Mixed dataset with problem descriptions, that removing problem descriptions degrades performance, and that model ranking reverses on the Ground Truth dataset. They conclude that LLMs are not yet reliable for fully automated code review and propose a human-in-the-loop process.","tokens_in":11585,"tokens_out":6197,"duration_ms":62510,"significance":"If the measurements are valid, this is a useful empirical data point with a practical process proposal and a complete replication package, which is a strength. However, the validity of the headline numbers depends critically on the unit-test oracle and on the treatment of YAML/indentation errors; the Ground Truth results suggest that the task is partly measuring format compliance and edit propensity rather than code correctness judgment. These issues do not invalidate the paper's descriptive claims about the tested configurations, but they substantially weaken the generalizability to real code review. The paper is transparent about several threats in Section VI, and the reproducibility package is commendable.","major_comments":[{"comment":"The unit-test oracle defines correctness as passing all tests in HumanEval's small, interview-level suites. These tests under-specify intended behavior, so the measured accuracy, correction ratio, and regression ratio are estimates of performance on those visible tests, not on the intended functionality. This is acknowledged in Sections VI-C and VI-D, but the abstract and Section V still report the raw percentages as LLM code-review performance. I recommend adding a held-out test audit (e.g., additional tests, mutation testing, or a manually annotated subset) or explicitly rephrasing the claims as 'passing the provided tests.'","section":"Section III-A, III-C (Eqs. 1-3)"},{"comment":"The decision to classify any YAML or indentation failure as an incorrect code suggestion conflates semantic correctness with response-format compliance. Since 4.08% of outputs had indentation errors and 1.08% had YAML format errors, the regression and correction ratios in Section IV include these failures. To support the claim in Section V-B that 'up to 24.80% of correct code blocks received incorrect code suggestions,' report the metrics after auto-fixing or removing format-only failures, or at least provide a separate analysis of these cases.","section":"Section VI-A"},{"comment":"GPT-4o's 42.07% correctness accuracy on the 164 canonical correct HumanEval solutions is a red flag: these code blocks are known to pass the unit tests, and HumanEval is likely part of GPT-4o's training data. Without a breakdown of these errors into 'model judged the code Incorrect and provided changes,' 'model produced a format error,' and 'model's classification matched a failing edge case,' this metric cannot be interpreted as a pure measure of correctness judgment. This is a load-bearing point because the Ground Truth experiment was introduced as a control group.","section":"Section IV-B, Fig 8"},{"comment":"The RQs ask how accurately and how effectively the LLMs are; without a human reviewer baseline or a comparison to existing code-review tools, percentages like 68.50% are hard to interpret. The related work survey in Section II-C covers prior automation but does not provide a numeric reference point. Adding even a small-scale human agreement study or a state-of-the-art tool comparison would make the contribution significantly stronger.","section":"Section II-C, IV"}],"minor_comments":[{"comment":"The prompt template in Figure 2 and the text in Section VI-A contain the typo 'Y AML' instead of 'YAML'.","section":"Figure 2 and Section VI-A"},{"comment":"Sections V-B and V-C report regression ratios of 'up to 24.80%' and 'up to 23.79%' respectively; clarify which configuration each figure refers to and ensure they are consistent.","section":"Sections V-B and V-C"},{"comment":"The chi-square test described in Section IV is not appropriate for comparing variance across three runs; the standard deviations are already informative, and the test appears misapplied or at least inadequately explained.","section":"Section IV"},{"comment":"The paper does not discuss inference cost or latency, which are relevant for the proposed human-in-the-loop process.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The Mixed dataset originates from Yetistiren et al. [55], which shares an author (Eray Tüzün) with this manuscript. The paper does not explicitly disclose this overlap; the editor may wish to confirm that reuse of the dataset is properly authorized and that any self-citation is appropriately framed. Also, the empirical contribution is modest but the replication package is a strong asset; the manuscript may be a better fit for an empirical software engineering venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a solid, transparent empirical benchmark, and the headline result—LLMs are moderately accurate but not reliable enough for fully automated code review—holds up. The specific percentages (68.50% accuracy, 67.83% correction ratio for GPT-4o with problem descriptions) are new numbers for these models, and the authors share their data and code. That alone makes it worth a serious referee.\n\nWhat the paper does well: the setup is clear (a prompt, two datasets, three metrics), the claims are modest, and the threats-to-validity section is unusually honest. The human-in-the-loop process they propose is sensible and not oversold. They also consistently find that adding problem descriptions helps, which is a practically useful result.\n\nThe soft spots are in the construct, not the execution. The correctness oracle is \"passes all unit tests\" on HumanEval-derived programs. Those tests are short and under-specifying, so both the labels and the model's fixes are measured against a weak standard. The red flag is the Ground Truth result: GPT-4o classifies only 42.07% of canonical correct solutions as Correct. That is hard to read as \"approval accuracy\"; it looks like the model is also being scored on its willingness to edit correct code and its YAML formatting. The authors treat formatting errors (indentation, YAML) as \"Incorrect,\" which is defensible but inflates false positives. And there is no human or existing-tool baseline, so we don't know if 68.5% is good or poor.\n\nThese are not fatal. The paper acknowledges most of them in Section VI. The problem is that the abstract and RQ answers still report the raw percentages as \"code review performance.\" A revision should either add a held-out test audit or temper the language so the numbers are clearly task-specific, not \"LLM code review accuracy.\"\n\nWho is this for? SE researchers working on LLM-assisted review, and practitioners who want to run their own benchmark on their own code. The authors explicitly support that. The citation pattern is fine; the Mixed dataset comes from prior work by one of the authors, but the measurements are new and computed against unit tests, so the circularity burden is low.\n\nRecommendation: accept for peer review. With a solid revision that addresses the oracle and baseline issues, this becomes a worthwhile reference point.","headline":"A transparent, modest benchmark that delivers a plausible result (LLMs are not yet reliable reviewers) but needs a held-out test audit before the headline percentages are taken as general code-review accuracy.","tokens_in":12119,"tokens_out":2718,"would_cite":true,"duration_ms":26645,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM code review is moderately accurate but not reliable enough for full automation, with the strongest configurations reaching 68.50% correctness classification and 67.83% correction ratio, and it proposes a…","keywords":["code review","large language models","GPT-4o","Gemini 2.0 Flash","code correctness","human-in-the-loop","automated code review","HumanEval"],"falsifier":"Apply the same protocol to human-written pull requests whose defects are confirmed by human reviewers and guarded by independent test suites; if the best LLM configuration matches human verdicts no better than a baseline that always answers Correct, the claim of moderate reliability would be refuted for real-world review.","tokens_in":11146,"feed_emoji":"🤖","tokens_out":7514,"duration_ms":77103,"temperature":0.7,"pith_summary":"This paper sets out to determine whether large language models can execute code review, deciding whether a proposed code block is correct and, if not, producing a corrected block, well enough to automate the task. The authors asked GPT-4o and Gemini 2.0 Flash to review 492 AI-generated code blocks of mixed correctness and 164 canonical HumanEval solutions, scoring verdicts and suggested fixes against unit tests. The best configuration, GPT-4o with the problem description provided, reached $68.50\\%$ correctness classification and $67.83\\%$ correction of incorrect blocks, while regressions (suggestions that break correct code) reached $24.80\\%$ in the worst configuration. Performance consistently dropped when the problem description was omitted, and the better-performing model was different on the two datasets. The paper concludes that LLM reviews are moderately useful but not reliable enough for full automation and proposes a human-in-the-loop process to keep a person accountable for merge decisions.","feed_headline":"Best LLM code review tops out at 68.5% accuracy","feed_subtitle":"With regressions up to 24.8%, the paper argues for a human-in-the-loop review process.","key_machinery":"The mechanism is an evaluation protocol built from three metrics and eight configurations. Each LLM is prompted in chain-of-thought style to classify a code block as Correct or Incorrect and to emit a revised block in YAML when it is Incorrect; the verdict is compared with unit-test results, the revised block is re-run against the same unit tests, and correct blocks that receive breaking suggestions are counted as regressions. The core definitions are Correctness Accuracy $= \\frac{\\#\\text{accurate assessments}}{\\#\\text{all blocks}}$, Correction Ratio $= \\frac{\\#\\text{correct suggestions}}{\\#\\text{incorrect blocks}}$, and Regression Ratio $= \\frac{\\#\\text{incorrect suggestions}}{\\#\\text{correct blocks}}$. The two prompt variants (with and without the problem description), two datasets, and two models form eight configurations, each run three times to check stability.","core_discovery":"On the paper's own terms, the central finding is a quantified reliability profile for two state-of-the-art LLMs as code reviewers. With problem descriptions on the 492-block mixed dataset, GPT-4o classified correctness correctly $68.50\\%$ of the time and Gemini 2.0 Flash $63.89\\%$; correction ratios were $67.83\\%$ and $54.26\\%$, respectively. Removing the description hurt both models, in some metrics by more than 20 percentage points, which the paper reads as evidence that code comments and pull-request descriptions are important for automated review. On the 164 ground-truth HumanEval blocks, Gemini outperformed GPT-4o on correctness accuracy ($66.67\\%$ versus $42.07\\%$), showing that model rankings depend on code type. Because regressions reached $24.80\\%$ in the worst configuration, the paper argues that fully automated LLM review is currently risky and proposes a human-in-the-loop workflow.","pith_inferences":["Because passing unit tests is a noisy proxy for correctness, real-world accuracy is likely lower than the reported numbers; equally, real reviews judge style and maintainability, which are outside this study, so the paper's moderate verdict does not close the question.","The consistent benefit of problem descriptions points to a cheap, testable intervention: enriching review prompts with linked issue text, failing-test output, or diff context may improve correctness classification more than model substitution alone.","API-hosted models change over time, so these numbers are a snapshot; a versioned, longitudinal rerun of the same eight configurations would reveal whether LLM code-review reliability is improving."],"forward_implications":["Providing a problem description consistently improved correctness classification and correction ratios, so teams adopting LLM review should encourage rich pull-request descriptions and code comments.","At the measured accuracy and regression levels, handing merge decisions to an LLM unattended is unsafe; a human-in-the-loop step remains necessary.","The best model differs across datasets (GPT-4o wins on mixed AI-generated code, Gemini wins on canonical HumanEval solutions), so model choice should be validated on the target codebase before adoption.","The protocol itself, verdict plus revised code scored by unit tests, can be reused by practitioners to benchmark models and prompts on their own code."],"supporting_citations":[{"why":"Supplies the HumanEval benchmark, its canonical solutions, and the unit tests used as the objective correctness standard.","marker":"[15]"},{"why":"Supplies the 492 AI-generated code blocks (ChatGPT, CodeWhisperer, Copilot) with varying correctness that form the main mixed dataset.","marker":"[55]"},{"why":"Frames modern code review and its expectations, defining the approval-and-suggestion task the prompts emulate.","marker":"[10]"},{"why":"Documents the time pressure and knowledge-sharing goals of code review, motivating both automation and the human-in-the-loop proposal.","marker":"[31]"},{"why":"Background on the GPT-4 family, the basis for the GPT-4o model evaluated.","marker":"[7]"},{"why":"Background on the Gemini 2.0 family, the basis for the Gemini 2.0 Flash model evaluated.","marker":"[4]"}],"fun_headline_variants":["LLM code review hits 68.5% accuracy, needs human oversight","Code review AI tops out at 68.5% accuracy, study finds","Best LLM code reviewer: 68.5% accurate, then a human","AI code review: best accuracy 68.5% but rankings vary by code","LLM code review accuracy maxes at 68.5%, human loop needed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that passing all unit tests fully defines code correctness and that the HumanEval-derived blocks are representative of code review; if test coverage is thin or the code differs from real pull requests, the reported accuracy and correction ratios will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["LLM code review hits 68.5% accuracy, needs human oversight","Code review AI tops out at 68.5% accuracy, study finds","Best LLM code reviewer: 68.5% accurate, then a human","AI code review: best accuracy 68.5% but rankings vary by code","LLM code review accuracy maxes at 68.5%, human loop needed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1566,"prompt_tokens":1004,"completion_tokens":562,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":458}},"tokens_in":620,"tokens_out":562,"duration_ms":5391,"temperature":1.0,"reasoning_tokens":458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:56:37.984547+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the same protocol to human-written pull requests whose defects are confirmed by human reviewers and guarded by independent test suites; if the best LLM configuration matches human verdicts no better than a baseline that always answers Correct, the claim of moderate reliability would be refuted for real-world review.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HumanEval benchmark, its canonical solutions, and the unit tests used as the objective correctness standard."},{"cited_title":"Evalu- ating the code quality of ai-assisted code generation tools: An empirical study on github copilot, amazon codewhisperer, and chatgpt, 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the 492 AI-generated code blocks (ChatGPT, CodeWhisperer, Copilot) with varying correctness that form the main mixed dataset."},{"cited_title":"Expectations, outcomes, and chal- lenges of modern code review","cited_arxiv_id":null,"evidence_quote":"Frames modern code review and its expectations, defining the approval-and-suggestion task the prompts emulate."},{"cited_title":"Code reviewing in the trenches: Challenges and best practices","cited_arxiv_id":null,"evidence_quote":"Documents the time pressure and knowledge-sharing goals of code review, motivating both automation and the human-in-the-loop proposal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Background on the Gemini 2.0 family, the basis for the Gemini 2.0 Flash model evaluated."}],"review_version":1}