{"id":"5f92bd87-3ce5-4043-b34d-db2d9ff2465f","arxiv_id":"2509.09918","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"WALL combines SonarQube with GPT-3.5 Turbo and GPT-4o to automatically fix detected code issues, reporting 71.6% and 85.5% revision rates plus cost reductions from a hybrid workflow.","lead":"WALL is a web application that uses SonarQube to detect code issues and GPT-3.5/4o to auto-generate fixes. The paper claims high revision rates and up to 40% cost savings from a hybrid cheap-then-advanced model strategy, but the evaluation is weakened by missing ground truth.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 'success rates' only measure whether the LLM changed lines, not whether SonarQube issues were resolved; the central quality and cost claims rest on this unvalidated metric.","rationale":"The reader's weakest assumption is exactly the load-bearing concern: 'revised' is defined as any line change, not as a validated fix. This is not a matter of disagreement with broader consensus about LLM capabilities; it is a correctness risk internal to the paper's measurement. Even if the arithmetic in Tables II and III were corrected, the fundamental metric would still be unable to distinguish a correct repair from an arbitrary edit. The internal inconsistencies (Section V totals vs. Table II sums, and the 40% vs. ~20% cost saving) are additional symptoms of the same underlying problem: the evaluation has no ground truth tied to actual issue resolution. Because the strongest claim in the paper is precisely these success rates and cost savings, the reported evidence does not support the conclusion. The paper may describe a useful engineering prototype, but the central quantitative claims need independent validation before acceptance. This supports the reader's REJECT verdict, so no verdict change is needed.","tokens_in":6910,"tokens_out":4676,"duration_ms":52813,"concrete_test":"Run WALL on the open-source 'open-instruct' repository using the released code and the same prompt, then rescan each revised file with SonarQube using the same rule set and compare issue instances before/after. Compute the fraction of original issues (matched by rule and location) that are actually absent after revision and the number of new SonarQube issues introduced per file. If the rescan-confirmed resolution rate is materially lower than the reported 'revised' rates, or if new-issue density is high, the central claim that WALL achieves high-quality revisions is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—high revision success rates, especially with the hybrid GPT-3.5/GPT-4o approach optimizing performance and cost—depends entirely on the definition of 'success' in Section V: 'success rate, defined as the percentage of issues resolved by each GPT model.' But the evaluation pipeline never verifies resolution. Section IV-C states that metrics are computed from line-level differences between original and revised files, with the ground truth defined as 'the percentage of updated, removed, or changed lines relative to the original files.' A high metric value therefore only means the revised file closely mirrors the original text; it does not mean the SonarQube rule violation is gone, the defect is fixed, or no new defect was introduced. The paper explicitly notes the absence of test cases and acknowledges human intervention is required, yet the reported 71.6% and 85.5% rates are presented as 'revision success' and the conclusion claims 'high-quality revisions.' Internally, the numbers also do not hold together: Section V reports 5,441 GPT-3.5 revisions, but Table II sums to 3,894; GPT-4o's 6,495 conflicts with the table sum of 6,232. The stated 'up to 40%' cost saving for the hybrid also does not follow from Table II, where the GPT-4o-only cost is $39.78 and the GPT-3.5+GPT-4o cost is $31.96—a reduction of about 20%, not 40%. The most load-bearing defect is the metric: without rescanning with SonarQube, running tests, or human validation, none of the reported percentages measure code quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents WALL, a web application that combines SonarQube static analysis with OpenAI GPT models (GPT-3.5 Turbo, GPT-4o, GPT-4o Mini) to automatically extract code issues, generate revised code, and compare original versus revised files. The authors report experiments on 7,599 issues across 563 proprietary files and a smaller open-source dataset, claiming that GPT-3.5 Turbo revises 71.6% of issues, GPT-4o revises 85.5%, and a hybrid GPT-3.5/GPT-4o workflow reduces cost by up to 40%. The paper concludes that WALL achieves high-quality automated code revisions and reduces human effort.","tokens_in":7306,"tokens_out":5484,"duration_ms":55744,"significance":"If the central claims were valid, WALL would be a practically useful demonstration of using LLMs to automate static-analysis issue repair, and the public availability of the tool is a concrete artifact. However, the evaluation does not measure whether SonarQube issues are actually resolved: 'success' is defined as the LLM having changed lines in the vicinity of the reported issue. The reported success counts and rates are also internally inconsistent with the paper's own Table II, and the 40% cost-saving claim is contradicted by the same table. The paper therefore does not currently provide a sound empirical basis for its main conclusions.","major_comments":[{"comment":"The central metric is not a measure of issue resolution. The success rate is defined in Section V as 'the percentage of issues resolved by each GPT model,' but Section IV-C explains that precision, recall, and F1 are computed from line-level differences between original and revised files, with ground truth defined as the percentage of updated, removed, or changed lines relative to the original files. There is no SonarQube rescan, no test execution, and no human validation to confirm that a 'revised' issue is actually fixed or that no new defect was introduced. A high revision rate only indicates that lines changed. Consequently, the conclusion that WALL produces 'high-quality revisions' is unsupported by the reported evidence, and this is load-bearing for every downstream claim in the paper.","section":"Section V and Section IV-C"},{"comment":"The success counts in the text do not match Table II. The text reports that GPT-3.5 Turbo revised 5,441 issues (71.6%), but the GPT-3.5-only row in Table II sums to 117 + 59 + 3,718 = 3,894 issues, which is 51.2% of 7,599. The text reports GPT-4o revised 6,495 issues (85.5%), but the GPT-4o-only row sums to 234 + 61 + 5,937 = 6,232 issues, which is 82.0%. These discrepancies change the reported rates substantially and must be reconciled before any comparison of model performance can be evaluated.","section":"Section V, Table II"},{"comment":"The claimed 'up to 40%' cost reduction of the hybrid approach is not supported by Table II. GPT-4o-only total cost is $6.20 + $1.01 + $32.57 = $39.78. The GPT-3.5 + GPT-4o total cost is $4.76 + $0.38 + $26.82 = $31.96. This is a reduction of about 19.7%, not 40%. In addition, the 'GPT-4o for Remaining' row reports 2,219 remaining code smells after GPT-3.5 revised 3,718 of 7,304 code smells, but 7,304 - 3,718 = 3,586, not 2,219. The hybrid row's revised count equals the GPT-4o-only count, not the total 7,599, so the definition of 'remaining' and the cost arithmetic are unclear.","section":"Section V, cost claims"},{"comment":"The comparison tool is said to compute precision, recall, and F1-score, but the results section reports no values for these metrics. The only reported quantitative measure is the unvalidated revision rate. If these metrics are intended to support claims about revision quality, the paper should report them and, more importantly, relate them to an external indication of correctness. Without such reporting, the 'evaluation' part of the pipeline is not actually evaluated against any ground truth beyond line edits.","section":"Section IV-C and Section V"}],"minor_comments":[{"comment":"The typographical spacing in 'W ALL' is distracting; the tool name should be written consistently, e.g., 'WALL'.","section":"Throughout"},{"comment":"The company is called 'Team Eagle Ltd.' in the main text but 'Team Eagle Inc.' in the reference list. Please use one official name.","section":"Section III-D and Reference [16]"},{"comment":"The pre-experiments on 350 issues and the claim that 'GPT-4o can resolve any issues GPT-3.5 Turbo can resolve' are mentioned but no data or protocol is provided. Since the hybrid workflow relies on this assumption, the statement should be substantiated or explicitly stated as a limitation.","section":"Section V"},{"comment":"The definition of ground truth as 'the percentage of updated, removed, or changed lines relative to the original files' is conceptually unclear. Precision/recall/F1 require a clear notion of a true positive; the paper should define exactly how each line-level metric is computed from the original and revised files.","section":"Section III-C"}],"recommendation":"reject","confidential_remarks":"The numerical inconsistencies may stem from the authors using different issue counts in the extraction step versus the analysis step, but the burden is on them to reconcile the numbers. More fundamentally, the evaluation's definition of 'revised' does not establish that issues are resolved, so the central quality claim collapses. This is not a mere presentation fix; it would require a redesigned validation protocol with SonarQube rescans, tests, or human inspection and a re-analysis of all reported results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is an engineering write-up of a SonarQube + GPT pipeline, with an evaluation that doesn't measure what it claims. The internal numbers don't add up, and the “success rate” is based on line diffs, not on whether the static-analysis issue is actually resolved. I would not send this to peer review as is.\n\nWhat's genuinely new: the WALL web application itself, which is publicly available. The three-module pipeline (issue extraction to CSV, LLM revision with pre-engineered prompts, and a line-level comparison tool) is a clean integration of known parts. Using a fixed prompt for all models and testing a hybrid GPT-3.5-then-GPT-4o strategy is a sensible, practical idea. The cost heuristic—cheap model for most files, expensive model for the rest—is reasonable and worth reporting. The open-source repo also makes it possible for others to reproduce the tool, which is more than many tool papers offer.\n\nThe soft spots are load-bearing. The paper defines “success rate” as the percentage of issues resolved, but it never checks whether the revised code still triggers the SonarQube rule, passes tests, or preserves behavior. Section IV-C says the ground truth is “the percentage of updated, removed, or changed lines relative to the original files.” So a high F1 or precision just means the revised file is textually close to the original. That cannot support “high-quality revisions” or “revision success rates.” The authors do note the absence of test cases and that human review is required, but then the conclusion overstates the result anyway.\n\nThe numbers also don't reconcile. Section V says GPT-3.5 revised 5,441 issues and GPT-4o revised 6,495; Table II sums to 3,894 and 6,232, respectively. The claimed “up to 40%” cost reduction from the hybrid is not what the table shows—the GPT-4o-only cost is $39.78, and the hybrid is $31.96, about a 20% reduction. That's still a meaningful saving, but the write-up states 40% without a clear derivation.\n\nWho is this for? A developer looking for a quick start on an LLM-based static-analysis repair tool might get value from the code and the workflow. As a research paper, the evaluation needs a proper validation loop: rescan with SonarQube after revision, run available tests, and at minimum have a human label a sample of fixes. Without that, the central quantitative claims don't hold. I'd tell the authors to fix the metric and the accounting, then resubmit. Not ready for referees yet.","headline":"A useful open-source integration, but the evaluation measures line edits, not issue resolution, and the numbers don't add up — not ready for peer review.","tokens_in":7757,"tokens_out":3150,"would_cite":false,"duration_ms":30344,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"WALL is a web pipeline that uses large language models to revise code issues found by static analysis, claiming 71.6% success with a cost-effective model, 85.5% with a stronger model, and up to 40% cost savings from a hybrid strategy.","keywords":["software quality assurance","large language models","code issue revision","static analysis","hybrid model strategy","line-level evaluation","web application","automated code review"],"falsifier":"Re-run WALL's revision step, then rescan every revised file with the same static analyzer and run any available tests; if most of the 7,599 flagged issues still appear in the rescan, or the revised project fails tests, the paper's central claim is false. A small-scale version: take a random sample of 100 issues WALL marked 'revised' and have two developers judge whether the code change actually addresses the reported issue; if most are judged non-fixes, the success-rate headline does not measure quality.","tokens_in":6765,"feed_emoji":"🤖","tokens_out":8002,"duration_ms":78871,"temperature":0.7,"pith_summary":"This paper tries to establish that automated quality assurance is practical for real projects: a three-module web application can extract static-analysis issues, feed each file to a large language model with a fixed few-shot prompt, and produce revised versions that resolve most of the flagged problems. The authors report that on 563 files with 7,599 issues, the cheaper model revised 71.6% of issues and the stronger model revised 85.5%; the stronger model also redid the leftover issues after the cheap model, reaching the same coverage while cutting API cost by up to 40%. They also provide side-by-side comparison and precision/recall/F1 metrics so a human reviews only questionable files. A reader should care because, if true, a large share of routine code-quality maintenance can be automated at modest cost. Success here means the model changed the flagged code; it is not validated by rerunning analysis or executing tests.","feed_headline":"WALL pipeline revises 85.5% of detected code issues","feed_subtitle":"A cheap model fixes most findings; a stronger model takes the rest, and the hybrid claims up to 40% lower cost.","key_machinery":"The load-bearing parts are WALL's fixed revision prompt and its hybrid escalation rule. The prompt, refined through over fifty experiments, packages the original file, the issue list from a CSV (type, line, file, message), a few-shot example, and the programming language, so every model sees the same task; a fixed prompt is what makes fair model-to-model comparison possible. The hybrid rule does the economic work: GPT-3.5 Turbo processes every file first, the project is rescanned, and only leftover issues go to GPT-4o. The comparison module—side-by-side highlighting plus line-level precision, recall, and F1—is the quality gate that tells a human which revisions to inspect.","core_discovery":"The central discovery is that a two-stage LLM workflow can match the revision coverage of the most expensive model while spending less: run a low-cost model across all files, rescan the project, and send only the unresolved issues to an advanced model. In the paper's primary experiment, GPT-3.5 Turbo revised 5,441 of 7,599 issues, and GPT-4o alone revised 6,495; in the per-category table, the hybrid combination of GPT-3.5 then GPT-4o reached the same revised counts as GPT-4o alone (all 234 bugs, all 61 vulnerabilities, and 5,937 of 7,304 code smells) at lower API cost. WALL's comparison module then flags line-level additions and removals and computes precision, recall, and F1, giving a quant","pith_inferences":["The reported success rates are upper bounds on true defect-fixing rates, because a 'revised' issue is one where the model changed lines, not one where the underlying defect is gone; re-running static analysis or tests would give a stricter count.","The cheap-model-first hybrid is a general cost pattern for LLM repair: any analyzer that emits machine-readable findings can drive the same two-stage escalation, with the split point tuned by issue difficulty and model price.","Line-level metrics can mis-rank revisions—a safe refactor that moves many lines looks riskier than a minimal one-line change that actually fixes nothing—so semantic or test-based equivalence would be a more reliable quality gate.","If the revision rates survive validation, the natural next step is unattended patch generation: WALL would submit pull requests for most findings, with humans reviewing only low-confidence files; the paper lists automated test-based evaluation as the missing piece for that."],"forward_implications":["Most static-analysis findings—around 7 in 10 with a cheaper model and 8.5 in 10 with a stronger model—can be automatically revised, leaving a minority for human attention.","A cheap-model-first, rescan, escalate strategy should be considered when running LLM-based repair at project scale; the paper claims cost reductions of up to 40% versus using the advanced model everywhere.","Line-level diff metrics can serve as a practical filter: files with high similarity to the original are lower-risk, while files with many changes are flagged for human review.","Because the pipeline depends only on the static-analysis platform's supported languages, one deployment can span mixed-language projects."],"fun_headline_variants":["Hybrid LLM pipeline revises 85.5% of code issues at 40% lower cost","WALL's cheap-to-smart AI matches top-tier coverage for less","Cost-saving hybrid: GPT-3.5 then GPT-4o fixes 85.5% of issues","Two-step LLM: cheap fixes most, advanced mops up, saves 40%","Cheap AI first, smart AI second: 85.5% fixed, 40% cheaper"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing assumption is that an LLM-produced change to a flagged line counts as fixing the issue; because WALL's success metric is line-level change rather than re-running the analyzer or a passing test, the 71.6% and 85.5% rates would overstate true quality if many revisions do not actually remove the defect.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid LLM pipeline revises 85.5% of code issues at 40% lower cost","WALL's cheap-to-smart AI matches top-tier coverage for less","Cost-saving hybrid: GPT-3.5 then GPT-4o fixes 85.5% of issues","Two-step LLM: cheap fixes most, advanced mops up, saves 40%","Cheap AI first, smart AI second: 85.5% fixed, 40% cheaper"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000832,"raw_usage":{"total_tokens":3468,"prompt_tokens":743,"completion_tokens":2725,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":2604}},"tokens_in":487,"tokens_out":2725,"duration_ms":22934,"temperature":1.0,"reasoning_tokens":2604,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:28:00.607779+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run WALL's revision step, then rescan every revised file with the same static analyzer and run any available tests; if most of the 7,599 flagged issues still appear in the rescan, or the revised project fails tests, the paper's central claim is false. A small-scale version: take a random sample of 100 issues WALL marked 'revised' and have two developers judge whether the code change actually addresses the reported issue; if most are judged non-fixes, the success-rate headline does not measure quality.","supporting_citations":[],"review_version":1}