Pith. sign in

REVIEW 4 major objections 4 minor 21 references

WALL: A Web Application for Automated Quality Assurance using Large Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read WALL is a web pipeline that uses large language models to revise code issues found by static analysis, claiming 71.6% success with a cost-effective model, 85.5% with a stronger model, and up to 40% cost savings from a hybrid strategy.

desk verdict A useful open-source integration, but the evaluation measures line edits, not issue resolution, and the numbers don't add up — not ready for peer review. read the letter →

arxiv 2509.09918 v1 pith:4D3DS4Z2 submitted 2025-09-12 cs.SE cs.AI

Seyed Moein Abtahi , Akramul Azim This is my paper
classification cs.SEcs.AI
keywords softwarequalityassurancelargelanguagemodelscodeissuerevisionstaticanalysishybridmodelstrategyline-levelevaluationwebapplicationautomatedreview
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that automated quality assurance is practical for real projects: a three-module web application can extract static-analysis issues, feed each file to a large language model with a fixed few-shot prompt, and produce revised versions that resolve most of the flagged problems. The authors report that on 563 files with 7,599 issues, the cheaper model revised 71.6% of issues and the stronger model revised 85.5%; the stronger model also redid the leftover issues after the cheap model, reaching the same coverage while cutting API cost by up to 40%. They also provide side-by-side comparison and precision/recall/F1 metrics so a human reviews only questionable files. A reader should care because, if true, a large share of routine code-quality maintenance can be automated at modest cost. Success here means the model changed the flagged code; it is not validated by rerunning analysis or executing tests.

What carries the argument

The load-bearing parts are WALL's fixed revision prompt and its hybrid escalation rule. The prompt, refined through over fifty experiments, packages the original file, the issue list from a CSV (type, line, file, message), a few-shot example, and the programming language, so every model sees the same task; a fixed prompt is what makes fair model-to-model comparison possible. The hybrid rule does the economic work: GPT-3.5 Turbo processes every file first, the project is rescanned, and only leftover issues go to GPT-4o. The comparison module—side-by-side highlighting plus line-level precision, recall, and F1—is the quality gate that tells a human which revisions to inspect.

What would settle it

Re-run WALL's revision step, then rescan every revised file with the same static analyzer and run any available tests; if most of the 7,599 flagged issues still appear in the rescan, or the revised project fails tests, the paper's central claim is false. A small-scale version: take a random sample of 100 issues WALL marked 'revised' and have two developers judge whether the code change actually addresses the reported issue; if most are judged non-fixes, the success-rate headline does not measure quality.

Watch

Extended reading notes

Core claim

The central discovery is that a two-stage LLM workflow can match the revision coverage of the most expensive model while spending less: run a low-cost model across all files, rescan the project, and send only the unresolved issues to an advanced model. In the paper's primary experiment, GPT-3.5 Turbo revised 5,441 of 7,599 issues, and GPT-4o alone revised 6,495; in the per-category table, the hybrid combination of GPT-3.5 then GPT-4o reached the same revised counts as GPT-4o alone (all 234 bugs, all 61 vulnerabilities, and 5,937 of 7,304 code smells) at lower API cost. WALL's comparison module then flags line-level additions and removals and computes precision, recall, and F1, giving a quant

Load-bearing premise

The load-bearing assumption is that an LLM-produced change to a flagged line counts as fixing the issue; because WALL's success metric is line-level change rather than re-running the analyzer or a passing test, the 71.6% and 85.5% rates would overstate true quality if many revisions do not actually remove the defect.

Editorial extensions

If this is right

  • Most static-analysis findings—around 7 in 10 with a cheaper model and 8.5 in 10 with a stronger model—can be automatically revised, leaving a minority for human attention.
  • A cheap-model-first, rescan, escalate strategy should be considered when running LLM-based repair at project scale; the paper claims cost reductions of up to 40% versus using the advanced model everywhere.
  • Line-level diff metrics can serve as a practical filter: files with high similarity to the original are lower-risk, while files with many changes are flagged for human review.
  • Because the pipeline depends only on the static-analysis platform's supported languages, one deployment can span mixed-language projects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported success rates are upper bounds on true defect-fixing rates, because a 'revised' issue is one where the model changed lines, not one where the underlying defect is gone; re-running static analysis or tests would give a stricter count.
  • The cheap-model-first hybrid is a general cost pattern for LLM repair: any analyzer that emits machine-readable findings can drive the same two-stage escalation, with the split point tuned by issue difficulty and model price.
  • Line-level metrics can mis-rank revisions—a safe refactor that moves many lines looks riskier than a minimal one-line change that actually fixes nothing—so semantic or test-based equivalence would be a more reliable quality gate.
  • If the revision rates survive validation, the natural next step is unattended patch generation: WALL would submit pull requests for most findings, with humans reviewing only low-confidence files; the paper lists automated test-based evaluation as the missing piece for that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents WALL, a web application that combines SonarQube static analysis with OpenAI GPT models (GPT-3.5 Turbo, GPT-4o, GPT-4o Mini) to automatically extract code issues, generate revised code, and compare original versus revised files. The authors report experiments on 7,599 issues across 563 proprietary files and a smaller open-source dataset, claiming that GPT-3.5 Turbo revises 71.6% of issues, GPT-4o revises 85.5%, and a hybrid GPT-3.5/GPT-4o workflow reduces cost by up to 40%. The paper concludes that WALL achieves high-quality automated code revisions and reduces human effort.

Significance. If the central claims were valid, WALL would be a practically useful demonstration of using LLMs to automate static-analysis issue repair, and the public availability of the tool is a concrete artifact. However, the evaluation does not measure whether SonarQube issues are actually resolved: 'success' is defined as the LLM having changed lines in the vicinity of the reported issue. The reported success counts and rates are also internally inconsistent with the paper's own Table II, and the 40% cost-saving claim is contradicted by the same table. The paper therefore does not currently provide a sound empirical basis for its main conclusions.

major comments (4)
  1. [Section V and Section IV-C] The central metric is not a measure of issue resolution. The success rate is defined in Section V as 'the percentage of issues resolved by each GPT model,' but Section IV-C explains that precision, recall, and F1 are computed from line-level differences between original and revised files, with ground truth defined as the percentage of updated, removed, or changed lines relative to the original files. There is no SonarQube rescan, no test execution, and no human validation to confirm that a 'revised' issue is actually fixed or that no new defect was introduced. A high revision rate only indicates that lines changed. Consequently, the conclusion that WALL produces 'high-quality revisions' is unsupported by the reported evidence, and this is load-bearing for every downstream claim in the paper.
  2. [Section V, Table II] The success counts in the text do not match Table II. The text reports that GPT-3.5 Turbo revised 5,441 issues (71.6%), but the GPT-3.5-only row in Table II sums to 117 + 59 + 3,718 = 3,894 issues, which is 51.2% of 7,599. The text reports GPT-4o revised 6,495 issues (85.5%), but the GPT-4o-only row sums to 234 + 61 + 5,937 = 6,232 issues, which is 82.0%. These discrepancies change the reported rates substantially and must be reconciled before any comparison of model performance can be evaluated.
  3. [Section V, cost claims] The claimed 'up to 40%' cost reduction of the hybrid approach is not supported by Table II. GPT-4o-only total cost is $6.20 + $1.01 + $32.57 = $39.78. The GPT-3.5 + GPT-4o total cost is $4.76 + $0.38 + $26.82 = $31.96. This is a reduction of about 19.7%, not 40%. In addition, the 'GPT-4o for Remaining' row reports 2,219 remaining code smells after GPT-3.5 revised 3,718 of 7,304 code smells, but 7,304 - 3,718 = 3,586, not 2,219. The hybrid row's revised count equals the GPT-4o-only count, not the total 7,599, so the definition of 'remaining' and the cost arithmetic are unclear.
  4. [Section IV-C and Section V] The comparison tool is said to compute precision, recall, and F1-score, but the results section reports no values for these metrics. The only reported quantitative measure is the unvalidated revision rate. If these metrics are intended to support claims about revision quality, the paper should report them and, more importantly, relate them to an external indication of correctness. Without such reporting, the 'evaluation' part of the pipeline is not actually evaluated against any ground truth beyond line edits.
minor comments (4)
  1. [Throughout] The typographical spacing in 'W ALL' is distracting; the tool name should be written consistently, e.g., 'WALL'.
  2. [Section III-D and Reference [16]] The company is called 'Team Eagle Ltd.' in the main text but 'Team Eagle Inc.' in the reference list. Please use one official name.
  3. [Section V] The pre-experiments on 350 issues and the claim that 'GPT-4o can resolve any issues GPT-3.5 Turbo can resolve' are mentioned but no data or protocol is provided. Since the hybrid workflow relies on this assumption, the statement should be substantiated or explicitly stated as a limitation.
  4. [Section III-C] The definition of ground truth as 'the percentage of updated, removed, or changed lines relative to the original files' is conceptually unclear. Precision/recall/F1 require a clear notion of a true positive; the paper should define exactly how each line-level metric is computed from the original and revised files.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity; mild self-referential success metric because the only stated ground truth is WALL's own line-diff.

  1. other [Section IV-C ('Code Comparison and Evaluation'); Section V ('Results and Analysis'); Table II]
    "In this version of W ALL, the ground truth is determined by the percentage of updated, removed, or changed lines relative to the original files. In the absence of test cases, this approach assists the human reviewer in evaluating the accuracy of the GPT models. ... The tables show the success rate, defined as the percentage of issues resolved by each GPT model, and cost metrics derived from OpenAI's API usage data."

    The headline 'success rate' / 'revision rate' (71.6% GPT-3.5, 85.5% GPT-4o) is presented as issues resolved, but the only operational ground truth given for evaluation is WALL's own line-level diff between original and revised files. No per-model SonarQube rescan, test execution, or human validation is reported for those rates; the paper explicitly says test cases are absent. Therefore a 'successful' revision is effectively whatever changed/removed lines the LLM produced, and the evaluated outcome is the same artifact as the evaluation metric. This is a mild self-referentiality in the measurement, not a case where a predicted quantity is mathematically equal to a fitted input; the LLM outputs remain external and could fail in ways the metric would miss.

full rationale

The paper is an empirical web-tool paper with no fitted parameters, no equations that derive predictions from inputs, and no self-citation chain. The closest thing to circularity is the success definition in Section V versus the line-diff 'ground truth' in Section IV-C. Because there are no test cases (acknowledged in IV-C and VI) and no reported rescan for the per-model rates, the evaluation metric and the measured revision outcome are the same artifact: high scores only mean lines changed, not that the SonarQube issue is fixed or no new defect introduced. This inflates but does not formally force the reported rates. The paper also contains non-circular correctness inconsistencies (Section V's 5,441 and 6,495 revised issues do not match Table II sums of 3,894 and 6,232; the 'up to 40%' hybrid cost saving is ~20% on Table II; a RAG module is mentioned as having limitations but never described). These are validity and reporting problems, not circularity. Overall, the central empirical claim has independent content and no self-citation is load-bearing, so the circularity score is low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's claimed effectiveness rests on three unvalidated premises: that an LLM-generated edit counts as a fix, that line-diff metrics proxy for quality, and that GPT-4o always covers GPT-3.5's output. None are established, and the last is contradicted by the full dataset.

assumptions (3)
  • ad hoc to paper A changed line around the flagged issue constitutes a successful resolution.
    Section V defines success rate as #Revised / #Issues, counting an issue as revised when the LLM generates a modified file, with no verification that the revision fixes the defect.
  • ad hoc to paper Line-level precision, recall, and F1 between original and revised files evaluate revision quality.
    Section IV-C uses these metrics as quality measures, but they capture textual difference rather than semantic correctness.
  • ad hoc to paper GPT-4o can resolve any issue that GPT-3.5 Turbo can resolve.
    The hybrid approach assumes confident escalation from cheap to expensive model; the pre-experiment on 350 issues is small, and the full code-smell success rate for GPT-4o is only 81.2%, so the assumption is contradicted on the full dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WALL: A Web Application for Automated Quality Assurance using Large Language Models." pith.science (2026). https://pith.science/paper/4D3DS4Z2

@misc{pith2026250909918,
  author       = {Pith},
  title        = {Pith review of: WALL: A Web Application for Automated Quality Assurance using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4D3DS4Z2}},
  note         = {Machine review of arXiv:2509.09918}
}
read the original abstract

As software projects become increasingly complex, the volume and variety of issues in code files have grown substantially. Addressing this challenge requires efficient issue detection, resolution, and evaluation tools. This paper presents WALL, a web application that integrates SonarQube and large language models (LLMs) such as GPT-3.5 Turbo and GPT-4o to automate these tasks. WALL comprises three modules: an issue extraction tool, code issues reviser, and code comparison tool. Together, they enable a seamless pipeline for detecting software issues, generating automated code revisions, and evaluating the accuracy of revisions. Our experiments, conducted on 563 files with over 7,599 issues, demonstrate WALL's effectiveness in reducing human effort while maintaining high-quality revisions. Results show that employing a hybrid approach of cost-effective and advanced LLMs can significantly lower costs and improve revision rates. Future work aims to enhance WALL's capabilities by integrating open-source LLMs and eliminating human intervention, paving the way for fully automated code quality management.

Figures

Figures reproduced from arXiv: 2509.09918 by the authors.

Figure 1
Figure 1. Overview of the WALL application workflow, including data process [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 8 linked inside Pith

  1. [1]

    Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation,

    B. A. Becker, P. Denny, J. Finnie-Ansley, A. Luxton-Reilly, J. Prather, and E. A. Santos, “Programming is hard-or at least it used to be: Educational opportunities and challenges of ai code generation,” in Proceedings of the 54th ACM Technical Symposium on Computer Science Education V . 1, 2023, pp. 500–506

    2023

  2. [2]

    Security code review by llms: A deep dive into responses,

    J. Yu, P. Liang, Y . Fu, A. Tahir, M. Shahin, C. Wang, and Y . Cai, “Security code review by llms: A deep dive into responses,”arXiv preprint arXiv:2401.16310, 2024

    arXiv 2024

  3. [3]

    Sorald: Automatic patch suggestions for sonarqube static analysis violations,

    K. Etemadi, N. Harrand, S. Lars ´en, H. Adzemovic, H. L. Phu, A. Verma, F. Madeiral, D. Wikstr¨om, and M. Monperrus, “Sorald: Automatic patch suggestions for sonarqube static analysis violations,”IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 4, pp. 2794–2810, 2023

    2023

  4. [4]

    Learning-based mod- els for vulnerability detection: An extensive study,

    C. Ni, L. Shen, X. Xu, X. Yin, and S. Wang, “Learning-based mod- els for vulnerability detection: An extensive study,”arXiv preprint arXiv:2408.07526, 2024

    arXiv 2024

  5. [5]

    A quantitative analysis of quality and consistency in ai-generated code,

    A. Clark, D. Igbokwe, S. Ross, and M. F. Zibran, “A quantitative analysis of quality and consistency in ai-generated code,” in2024 7th International Conference on Software and System Engineering (ICoSSE). IEEE, 2024, pp. 37–41

    2024

  6. [6]

    When llm-based code generation meets the software development process,

    F. Lin, D. J. Kimet al., “When llm-based code generation meets the software development process,”arXiv preprint arXiv:2403.15852, 2024

    arXiv 2024

  7. [7]

    Experimenting with multi-agent software development: To- wards a unified platform,

    M. A. Sami, M. Waseem, Z. Rasheed, M. Saari, K. Syst ¨a, and P. Abra- hamsson, “Experimenting with multi-agent software development: To- wards a unified platform,”arXiv preprint arXiv:2406.05381, 2024

    arXiv 2024

  8. [8]

    Prompt- enhanced software vulnerability detection using chatgpt,

    C. Zhang, H. Liu, J. Zeng, K. Yang, Y . Li, and H. Li, “Prompt- enhanced software vulnerability detection using chatgpt,” inProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings, 2024, pp. 276–277

    2024

Show all 21 references
  1. [9]

    A hitchhiker’s guide to jailbreaking chatgpt via prompt engineering,

    Y . Liu, G. Deng, Z. Xu, Y . Li, Y . Zheng, Y . Zhang, L. Zhao, T. Zhang, and K. Wang, “A hitchhiker’s guide to jailbreaking chatgpt via prompt engineering,” inProceedings of the 4th International Workshop on Software Engineering and AI for Data Quality in Cyber-Physical Sys- ...

    2024

  2. [10]

    Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag,

    X. Du, G. Zheng, K. Wang, J. Feng, W. Deng, M. Liu, B. Chen, X. Peng, T. Ma, and Y . Lou, “Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag,”arXiv preprint arXiv:2406.11147, 2024

    2024 arXiv

  3. [11]

    Establishing traceability between natural language requirements and software artifacts by combining rag and llms,

    S. J. Ali, V . Naganathan, and D. Bork, “Establishing traceability between natural language requirements and software artifacts by combining rag and llms,” inInternational Conference on Conceptual Modeling. Springer, 2024, pp. 295–314

    2024

  4. [12]

    Sonarqube,

    SonarSource, “Sonarqube,” https://www.sonarsource.com/products/ sonarqube/, 2024, accessed: November 10, 2024

    2024

  5. [13]

    Gpt-3.5 turbo,

    OpenAI, “Gpt-3.5 turbo,” https://platform.openai.com/docs/models/ gpt-3-5-turbo, 2024

    2024

  6. [14]

    Gpt-4 turbo and gpt-4o,

    ——, “Gpt-4 turbo and gpt-4o,” https://platform.openai.com/docs/ models/gpt-4-turbo-and-gpt-4, 2024

    2024

  7. [15]

    Unveiling llm evaluation focused on metrics: Challenges and solutions,

    T. Hu and X.-H. Zhou, “Unveiling llm evaluation focused on metrics: Challenges and solutions,”arXiv preprint arXiv:2404.09135, 2024

    2024 arXiv

  8. [16]

    Team Eagle Inc

    Team Eagle Inc., “Team Eagle Inc.” Available online, 2024. [Online]. Available: https://www.team-eagle.ca/

    2024

  9. [17]

    T\” ulu 3: Pushing frontiers in open language model post-training,

    N. Lambert, J. Morrison, V . Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V . Miranda, A. Liu, N. Dziri, S. Lyuet al., “T\” ulu 3: Pushing frontiers in open language model post-training,”arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  10. [18]

    Open-instruct: Open-source implementation of instruction- tuned models,

    A. I. for AI, “Open-instruct: Open-source implementation of instruction- tuned models,” 2023. [Online]. Available: https://github.com/allenai/ open-instruct

    2023

  11. [19]

    Languages overview,

    SonarSource, “Languages overview,” https://docs.sonarsource.com/ sonarqube/10.5/analyzing-source-code/languages/overview/, 2024, ac- cessed: November 14, 2024

    2024

  12. [20]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  13. [21]

    Llama Models by Meta,

    “Llama Models by Meta,” https://www.llama.com, 2024, accessed: 2024-11-19

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.