{"id":"2ba8b782-04ef-469a-b0da-52413b108941","arxiv_id":"2502.07399","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CodeQUEST, a GPT-4o-based evaluator-optimizer loop, reports a 52.6% mean relative improvement in code quality, but the improvement is measured by the same model that enforces monotonic score increases.","lead":"This paper introduces CodeQUEST, an LLM-based system that scores code on ten quality dimensions and iteratively rewrites it based on its own feedback. The authors report a 52.6% mean relative improvement, but the headline result is weakened because the improvement is measured by the same model that decides what counts as improved.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 52.6% improvement claim is produced by a self-scoring acceptance rule; the external proxy validation in Section 4.3 is too weak to establish that the GPT-4o score is a valid quality measure.","rationale":"I read the paper as an engineering report whose central quantitative claim is that CodeQUEST improves code quality by a mean RPI of 52.6%. For that claim to hold, the Evaluator's GPT-4o-derived score must be a valid measure of quality, at least in the direction of change. That condition is least secure exactly where the Reader placed it: Section 2.2.3's acceptance rule filters on the same score used to measure the outcome, so improvement is partly built into the protocol. The paper does attempt external validation, and I credit that: the proxy correlations in Table 3 and the SecurityEval examples show the evaluator is not pure noise. But the validation is not strong enough to support the headline. Spearman correlation of 0.23 between delta proxy and delta CodeQUEST is weak, the observations are clustered by example and hence not independent, only Python is covered, and the 34 non-MBPP examples have no functional test. None of this is an ad hominem point: the framework may be a useful prompt recipe and the released code is a positive. The issue is that the evidence presented does not establish robust code-quality improvement, so I agree with the Reader's REJECT and find no reason to change it.","tokens_in":24044,"tokens_out":4606,"duration_ms":43621,"concrete_test":"Run a blind before/after evaluation on all 42 examples: present original and final accepted versions in randomized order to two or more independent human raters, have them score both on the same ten CodeQUEST dimensions and flag any behavior change, and compare the human-rated improvement with the GPT-4o-rated 52.6% RPI. If human-rated improvement is not substantially positive and in the same direction, the acceptance rule in Section 2.2.3, rather than genuine quality gain, is the source of the headline claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that CodeQUEST effectively and robustly improves code quality depends on treating GPT-4o's aggregate score as a valid measure of quality. In Section 2.2.3 the Optimizer rejects every candidate rewrite whose CodeQUEST score does not increase relative to the last accepted version. Consequently the 41/42 improved examples and the 52.6% mean RPI in Section 4.2 are guaranteed by the selection rule unless the evaluator score tracks true quality. The only external validation, Section 4.3, is too weak to break this circularity: it covers only the 28 Python examples, it uses 138 non-independent delta observations, the strongest relationship is Pearson rp=0.53 and Spearman rs=0.23 between CodeQUEST and proxy deltas, and there is no human evaluation. Semantic behavior was checked only for the 8 MBPP examples; for the other 34 examples nothing ensures that the accepted rewrites preserve functionality. The SecurityEval examples in Appendix E are anecdotal evidence that the evaluator can detect vulnerabilities, not evidence that the 50-item aggregate score is accurate enough to support the reported RPI.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces CodeQUEST, a GPT-4o-based framework for evaluating and iteratively improving code quality across ten dimensions (e.g., readability, maintainability, efficiency, security). The Evaluator produces per-dimension quantitative scores and qualitative feedback; the Optimizer uses that feedback to rewrite code, validates compilation, and accepts only rewrites whose CodeQUEST score increases relative to the previous accepted version. On a hand-curated dataset of 42 examples (28 Python, 14 JavaScript), the authors report that 41 of 42 examples improved under their framework, with a mean relative percentage improvement (RPI) of 52.6%. They also compare CodeQUEST's scores against proxy metrics (Pylint, Radon, Bandit), reporting correlations between per-iteration changes in the CodeQUEST score and the proxy score, and they provide qualitative examples, including a security-focused case study.","tokens_in":24404,"tokens_out":7089,"duration_ms":61213,"significance":"If the claims were supported, CodeQUEST would be a practically useful contribution: an open-source, configurable LLM pipeline for automated code-quality assessment and iterative improvement. The paper's strengths include the detailed description of the ten-dimension evaluation rubric, the explicit optimization loop with code validation, the release of code, and the SecurityEval examples showing that CodeQUEST detects hard-coded secrets that Bandit misses. However, the empirical evidence as presented does not establish the central claims. The improvement result is to a large degree a consequence of the acceptance criterion, and the external validation is based on non-independent observations with modest correlations and no human evaluation or functional testing for most examples. The potential of the approach is real, but the current manuscript does not convincingly demonstrate that CodeQUEST reliably improves actual code quality or that its aggregate score is a valid measure of code quality.","major_comments":[{"comment":"The reported improvement (41/42 examples, mean RPI 52.6%) is a direct consequence of the Optimizer's acceptance rule, which only accepts a new version if its CodeQUEST score, produced by the same GPT-4o evaluator that generated the feedback, is higher than the previous accepted version. The paper acknowledges this in Section 4.2 (\"by design, ensures a monotonic improvement\"), but this monotonicity is a property of the selection rule, not evidence about actual code quality. To support the headline claim, the authors would need to compare initial and final code versions against external quality measures (e.g., proxy scores, test outcomes, human judges) rather than against the same model's score.","section":"Section 2.2.3 and Section 4.2"},{"comment":"The proxy validation is too weak to break the circularity around the improvement claim. The correlations are computed over 138 incremental observations that are not independent, since they come from only 28 Python examples across up to five iterations each. The strongest relationship is Pearson rp=0.53, while the Spearman rank correlation is only rs=0.23, and the analysis excludes two invalid attempts (footnote 6). No confidence intervals or clustered/mixed-effects analysis are provided. This does not establish that the CodeQUEST aggregate score is a valid measure of code quality, and it offers no direct evidence for the JavaScript portion of the dataset. A more informative test would be to report whether the proxy scores themselves improve from the original to the final version.","section":"Section 4.3, Table 3"},{"comment":"Semantic preservation is verified only for the 8 MBPP examples, which have test cases; for the other 34 examples, only compilation (or syntactic validity) is checked. Given that the Optimizer is instructed to make \"meaningful changes\" across all ten quality dimensions, the accepted rewrites may improve the GPT-4o score while changing or breaking intended behavior. The paper's own Section 5 acknowledges the hallucination risk, but no mitigation is reported for the non-MBPP examples. The authors should either run functional tests or provide a manual semantic check for all 42 examples before claiming that code quality improved.","section":"Section 2.2.2 and Section 4.2"},{"comment":"The claim that CodeQUEST \"can effectively and robustly evaluate code quality\" is supported mainly by qualitative comparison on a single example (Table 2) and a figure showing score distributions. No quantitative accuracy measure against ground-truth quality labels is provided. The proxy correlations in Section 4.3 are the only quantitative evidence, and they are modest. The authors should consider a more direct validation, such as agreement with human quality ratings on a larger sample.","section":"Section 4.1"}],"minor_comments":[{"comment":"The description of the optional self-consistency mechanism is placed in a footnote that interrupts the discussion of the prompt; move it to the main text or an appendix for readability.","section":"Section 2.1, footnote 2"},{"comment":"The column heading \"T est cases\" contains an erroneous space; additionally, the use of \"True\" and \"False\" in that column is unconventional and should be replaced with \"Yes\"/\"No\" or with a count of test cases.","section":"Table 1"},{"comment":"Report 95% confidence intervals for the Pearson and Spearman correlations and state whether the p-values are one- or two-sided.","section":"Section 4.3, Table 3"},{"comment":"The code listings contain numerous OCR-like artifacts (e.g., \"__ in it_ _\", \"C o n s t r u c t s\", \"m a x _ h e i g h t\"); these should be cleaned to match the actual source code.","section":"Appendices A and D"},{"comment":"The baseline prompt asks for a score on the same −5 to 5 scale as a single dimension, while CodeQUEST's overall score is an average over ten dimensions; the comparison of absolute scores between baseline and CodeQUEST may be misleading and should be clarified.","section":"Section 3.2"}],"recommendation":"reject","confidential_remarks":"For the editor: the paper's central claims are overstated relative to the evidence. The improvement result is largely an artifact of the Optimizer's acceptance rule, and the proxy validation is based on non-independent data with modest correlations and no human evaluation. I do not see a path to acceptance without a redesign of the evaluation protocol (e.g., an independent evaluator or external quality metrics on final versions) and substantially weakened claims. As submitted, the manuscript does not meet the evidentiary bar for the claims in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a well-structured engineering report, but the central quantitative claim—52.6% mean relative improvement—is largely an artifact of the optimizer’s acceptance rule. In Section 2.2.3, any candidate whose CodeQUEST score does not increase is rejected, and the final score is produced by the same GPT-4o model that generated the feedback. So the 41/42 improved examples and the RPI are what the selection rule mechanically produces, unless the evaluator score tracks real quality. The paper does not establish that it does.\n\nWhat is genuinely new and useful: the ten-dimension rubric with five true/false statements per dimension is a concrete, reusable prompt design; the framework includes a code-validation step (compile and optional tests), which many self-refine schemes omit; and the qualitative outputs are noticeably more detailed than the baseline CoT. The two SecurityEval examples show the evaluator catching hardcoded API keys that Bandit misses, which is a real point in its favor. The authors also released code and are candid in the threats-to-validity section.\n\nThe soft spots are not minor. External validation in Section 4.3 covers only the 28 Python examples, uses 138 non-independent delta observations, and the strongest correlation is Pearson 0.53 with Spearman 0.23. There is no human evaluation, and semantic preservation is only checked for the 8 MBPP examples. For the other 34 examples, nothing rules out the accepted rewrite breaking intended behavior. The paper’s own Section 5 acknowledges the need for human validation and larger datasets, but that does not fix the circularity in the current numbers.\n\nThe proper reading of this paper is as a prompt-engineering recipe plus a cautionary tale about LLM self-evaluation, not as evidence that CodeQUEST robustly improves code quality. For an SE venue, a serious referee could push the authors to add a human study, run functional tests on all examples, and compare against an optimizer not conditioned on the same model’s score. That would be a worthwhile revision. As it stands, the headline claim is not supported.\n\nI'd bring this to a reading group for the methodological discussion, and I’d send it to peer review—not to accept as-is, but because the framework is relevant and the circularity is instructive. The authors have done enough honest work here to merit a chance to fix the evaluation.","headline":"A cleanly written LLM code-quality loop whose headline improvement number is mostly a self-scoring artifact; the framework and qualitative results are worth a look, but the 52.6% claim doesn't survive contact with its own acceptance rule.","tokens_in":24797,"tokens_out":3986,"would_cite":false,"duration_ms":29992,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM that scores code on ten quality dimensions and only accepts rewrites that score higher can evaluate and improve code, yielding a mean relative improvement of 52.6% on 42 Python/JavaScript examples.","keywords":["code quality evaluation","large language models","GPT-4o","iterative code improvement","static analysis validation","Python","JavaScript","software maintainability"],"falsifier":"Have expert programmers blindly rate the original and final versions of the 138 improved Python outputs; if experts do not prefer the final versions, or if a control loop that replaces CodeQUEST's score with a random score produces a similar 52.6% 'improvement', then the reported gain is an artifact of the acceptance rule rather than real quality improvement.","tokens_in":23847,"feed_emoji":"🧑‍💻","tokens_out":7666,"duration_ms":64719,"temperature":0.7,"pith_summary":"The paper tries to establish that an LLM can be turned into a dependable code-quality referee and improver if the judgment is structured rather than left open-ended. CodeQUEST scores code on ten named quality dimensions, each probed by five forced-choice statements, and then runs an optimizer that rewrites the code using the qualitative feedback and accepts only rewrites whose overall score does not drop. On 42 hand-curated Python and JavaScript examples, this loop improved 41 of them, with a mean relative percentage improvement of 52.6% (median 57.1%) over at most five iterations. The claim matters because it offers a concrete, configurable recipe for automating a large share of code review and refactoring, with the score acting as a built-in quality floor.","feed_headline":"GPT-4o judge-and-fix loop lifts code quality 52.6%","feed_subtitle":"It scores code on ten dimensions, rejects rewrites that score lower, and finds flaws static analyzers miss.","key_machinery":"The load-bearing mechanism is the Evaluator-Optimizer loop with its acceptance rule. The Evaluator decomposes code quality into ten named dimensions—Readability, Maintainability, Testability, Efficiency, Robustness, Security, Documentation, Modularity, Scalability, Portability—and for each dimension asks five questions answerable only by 'True', 'False', or 'Not Applicable'; mapping those to $+1$, $-1$, and $0$ and averaging yields a code-level score in $[-5,5]$. The Optimizer feeds the qualitative dimension feedback back to GPT-4o, rewrites the code, rejects rewrites that fail compilation or supplied tests, and keeps a rewrite only if its overall CodeQUEST score does not drop relative to the last accepted version. That acceptance rule is what makes improvement monotone by construction; the external proxy metrics validate the scores but do not participate in the loop.","core_discovery":"On the paper's own terms, the central discovery is that LLM-based code-quality assessment is prompt-sensitive in a fixable way: a dimension-decomposed, forced-choice evaluation produces more comprehensive qualitative feedback and better-calibrated scores than a simple chain-of-thought rating, and those scores can drive a monotonic improvement loop. The framework's Evaluator produces a code-level score as the average of ten dimension scores, each in $[-5,5]$; the Optimizer rewrites code from the dimension feedback, validates that the rewrite compiles and passes available tests, and rejects any version whose overall score falls. Across the curated dataset, 41 of 42 examples improved, the mean relative percentage improvement was $52.6\\% \\pm 17.9\\%$, and the per-iteration change in CodeQUEST scores correlated with per-iteration changes in Pylint, Radon, and Bandit proxies more strongly than the baseline did. The paper also reports that CodeQUEST caught hardcoded API keys in SecurityEval examples that Bandit reported as clean, indicating the LLM evaluator can complement static analyzers.","pith_inferences":["A natural next experiment, not run here, would swap the Evaluator's score for a random score while keeping the same acceptance rule; if the reported improvement persists, the acceptance rule—not genuine quality tracking—would be doing the work.","The same judge-and-accept loop could be repurposed for other quality targets, such as performance budgets or accessibility, by replacing the ten dimensions; the paper's generality claim would then be testable without new machinery.","The paper's own threat-to-validity section concedes hallucination, stochasticity, small dataset size, and dependence on GPT-4o's training coverage; those caveats mean the 52.6% figure is better read as evidence for the recipe than as a guarantee for arbitrary codebases."],"forward_implications":["Most of the quality gain arrives in the first iteration cycle (about 2 code-quality units on average), with later iterations adding 0.52, 0.27, and then below 0.1 units, so a one- or two-iteration budget captures most of the benefit at lower API cost.","Because the Optimizer rejects any rewrite whose overall CodeQUEST score falls, the final delivered version is guaranteed by construction to score at least as high as the original on the framework's own scale.","The framework's per-iteration score changes correlate with changes in Pylint, Radon Maintainability Index, and Bandit proxies (Pearson $r = 0.53$ for CodeQUEST versus $0.27$ for the baseline), suggesting the evaluator tracks externally measurable quality shifts better than a generic chain-of-thought prompt does.","On the SecurityEval examples, CodeQUEST flagged hardcoded API keys that Bandit missed, so an LLM-based evaluator can complement rather than replace static analysis security tools."],"supporting_citations":[{"why":"GPT-4 technical report: the model family that powers both the Evaluator and the Optimizer.","marker":"[1]"},{"why":"Chain-of-thought prompting: the zero-shot CoT template used in the Evaluator's prompt.","marker":"[35]"},{"why":"Self-consistency: the optional retry-based mechanism for reducing variance in dimension scores.","marker":"[34]"},{"why":"Pylint: one of the three proxy metrics, standing in for Readability and Documentation.","marker":"[31]"},{"why":"Radon Maintainability Index: proxy for Maintainability, scaled to 0–10.","marker":"[19]"},{"why":"Bandit: proxy for Security, with a heuristic mapping of output logs to a 0–10 score.","marker":"[7]"},{"why":"MBPP: source of 8 Python examples with test cases, used for functional validation of rewrites.","marker":"[4]"},{"why":"SecurityEval: source of vulnerable Python examples where CodeQUEST detects issues Bandit misses.","marker":"[29]"},{"why":"Prior use of Radon Maintainability Index as a maintainability proxy, informing the validation setup.","marker":"[23]"}],"fun_headline_variants":["GPT-4o rewrites code, rejects low scores, gains 52.6%","Dimension-by-dimension GPT-4o code review beats static analyzers","LLM evaluator catches hardcoded API keys that Bandit misses","52.6% code quality boost from iterative GPT-4o feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the GPT-4o-generated numerical score genuinely tracks code quality, so a higher score after rewriting means better code and not merely a version the model prefers.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o rewrites code, rejects low scores, gains 52.6%","Dimension-by-dimension GPT-4o code review beats static analyzers","LLM evaluator catches hardcoded API keys that Bandit misses","52.6% code quality boost from iterative GPT-4o feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3055,"prompt_tokens":978,"completion_tokens":2077,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":1994}},"tokens_in":594,"tokens_out":2077,"duration_ms":13859,"temperature":1.0,"reasoning_tokens":1994,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:53:42.550411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have expert programmers blindly rate the original and final versions of the 138 improved Python outputs; if experts do not prefer the final versions, or if a control loop that replaces CodeQUEST's score with a random score produces a similar 52.6% 'improvement', then the reported gain is an artifact of the acceptance rule rather than real quality improvement.","supporting_citations":[{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting: the zero-shot CoT template used in the Evaluator's prompt."},{"cited_title":"Chi, Sha- ran Narang, Aakanksha Chowdhery, and Denny Zhou","cited_arxiv_id":null,"evidence_quote":"Self-consistency: the optional retry-based mechanism for reducing variance in dimension scores."},{"cited_title":"Pylint: https://github.com/pylint-dev/pylint","cited_arxiv_id":null,"evidence_quote":"Pylint: one of the three proxy metrics, standing in for Readability and Documentation."},{"cited_title":"Radon mi https://radon.readthedocs.io/en/latest/intro.html","cited_arxiv_id":null,"evidence_quote":"Radon Maintainability Index: proxy for Maintainability, scaled to 0–10."},{"cited_title":"https://github.com/pycqa/bandit","cited_arxiv_id":null,"evidence_quote":"Bandit: proxy for Security, with a heuristic mapping of output logs to a 0–10 score."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SecurityEval: source of vulnerable Python examples where CodeQUEST detects issues Bandit misses."}],"review_version":1}