{"id":"759b584c-2b22-49ec-ba93-eab5f651f0ae","arxiv_id":"2507.09665","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"4-bit AWQ quantization of CodeLlama and DeepSeek-Coder largely preserves static-analysis measures of code quality on Java and Python tasks, with small readability and maintainability shifts.","lead":"This study compared code written by full-precision and 4-bit quantized versions of CodeLlama and DeepSeek-Coder, finding that quantization does not significantly harm measured code quality like complexity, reliability, or maintainability. This matters because quantized models use less memory and energy, so evidence of preserved quality supports cheaper and greener deployment of AI coding assistants.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 'quality preserved' rests on unnormalized sums of non-commensurable metrics; per-metric non-significance is not equivalence, so the central claim is under-supported.","rationale":"The paper's stated conclusion is that quantization is not a deal-breaker and preserves qualitative aspects and static properties of generated code. For that conclusion to hold, the comparison between full-precision and 4-bit AWQ models must be based on quality measurements that are commensurable, normalized for code size, and capable of supporting a claim of equivalence rather than merely a claim of absence of statistically significant difference. The reader's weakest-assumption analysis correctly identifies the unnormalized aggregation of heterogeneous metrics as the primary soft spot. My independent reading confirms that the aggregate 'total issues' numbers in Section V are not meaningful as constructed: cyclomatic complexity and cognitive complexity are not issues in the same sense as bugs or code smells, and adding them to count-based findings yields a number with no clear interpretation. The lack of LoC normalization matters because several rows of Tables I and II show quantized models producing longer code, and raw counts confound quality with length. The per-metric statistical tests are more defensible, but the paper does not provide confidence intervals or equivalence bounds for the non-significant results, so the headline claim of preservation is not established. A further limitation, not emphasized by the reader, is that the manual readability and consistency assessment in Section IV-G contains no full-precision control group; ratings of quantized outputs alone cannot support the statement that quantization 'introduces' readability shifts. None of these issues make the study valueless; the pass@1 results and per-metric tests support a cautious reading that no large degradation was detected. The appropriate verdict remains conditional rather than a full accept or reject, so the reader's verdict should be unchanged.","tokens_in":20657,"tokens_out":8785,"duration_ms":110832,"concrete_test":"Recompute the headline totals and all per-metric Wilcoxon and Cliff's-delta analyses from the replication package as issue rates per 100 LoC (or with LoC as a covariate), and add two-one-sided equivalence tests with a pre-specified bound (e.g., +/-10% of the full-precision rate) for each metric; if the 6% improvement or the 'no significant degradation' conclusion reverses or falls outside the equivalence bound, the claim needs revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section V, the central quantitative support for the conclusion is the raw 'total issues' comparison: 2,892 (quantized) vs 3,084 (full-precision) issues for McEval-Python and 5,347 vs 5,381 for Java. These totals are computed by summing Security Hotspots, Reliability bugs, Maintainability code smells, cyclomatic complexity, and cognitive complexity (Tables I/II). CyC and CoC are continuous complexity values on a different scale from count-based bugs and smells; summing them into a single 'issue' count has no defined semantics. The totals are also not normalized by LoC despite systematic length differences (e.g., CodeLlama 13B on MultiPL-E-Python increases from 1,130 to 1,180 LoC after quantization). If issue rates per line are the relevant quality measure, the raw totals can mislead. The per-metric Wilcoxon tests avoid aggregation, but the paper reports no confidence intervals or equivalence bounds for the non-significant results; with only 42-161 tasks per configuration, failure to reject equality does not establish that quantized and full-precision models are comparable. The manual readability and consistency check in Section IV-G also rates only quantized outputs, so it cannot by itself attribute any quality difference to quantization. Together these gaps mean the 'not a deal-breaker' conclusion overstates what the data show; a cautious 'no large degradation detected, with reservations' is the defensible reading.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically investigates whether 4-bit Activation-aware Weight Quantization (AWQ) degrades not only functional correctness but also qualitative attributes of code generated by large code models. The study uses CodeLlama (7B, 13B, 34B) and DeepSeek-Coder (1.3B, 6.7B, 33B) on MultiPL-E/HumanEval and McEval benchmarks for Python and Java, compares full-precision and AWQ-4bit models, and evaluates the generated code with SonarCloud, Pylint, Flake8, and PMD. It reports pass@1 and static quality metrics, applies Wilcoxon signed-rank tests with Holm correction and Cliff's delta, and includes a manual readability/consistency assessment. The paper concludes that quantization is not a deal-breaker because it preserves functional correctness and largely maintains qualitative code attributes such as maintainability, reliability, and structural simplicity.","tokens_in":20903,"tokens_out":6803,"duration_ms":79254,"significance":"If the central claim holds, the result is practically valuable: it would indicate that 4-bit quantized code models can be deployed with substantial memory and energy savings without a meaningful loss in code quality. The study addresses a real gap, as prior quantization work for code models focused almost exclusively on functional correctness. Strengths include a publicly available replication package, the use of external standard benchmarks and static analysis tools, paired statistical testing with multiple-comparison correction, and a manual annotation component with reported inter-rater agreement. I found no circularity in the central evaluation: the comparisons rely on external benchmarks and external static-analysis outputs. The main weakness is that the headline aggregate comparisons are computed by summing incommensurable metrics without normalization, while the per-metric statistical results are used as evidence of equivalence without equivalence testing; both issues undermine the strength of the conclusion as currently stated.","major_comments":[{"comment":"The central evidence for the claim that quality is preserved is the raw aggregate comparison (e.g., 2,892 vs. 3,084 issues on McEval-Python). These totals are obtained by summing Security Hotspots, Reliability bugs, Maintainability code smells, Cyclomatic Complexity, and Cognitive Complexity. Cyclomatic and cognitive complexity are continuous scores on a different scale from count-based bug and smell metrics, so the sum has no clear semantic interpretation. The totals are also not normalized by lines of code, although the tables show quantized models producing longer code in several configurations (e.g., CodeLlama 13B on MultiPL-E-Python increases from 1,130 to 1,180 LoC in Table I). The reported 6% reduction is therefore not a valid basis for the conclusion; the authors should either remove the aggregate totals or replace them with a per-LoC or otherwise justified analysis.","section":"Section V, Tables I and II"},{"comment":"The paper interprets non-significant Wilcoxon tests as evidence that quantized models preserve code quality. With 42 to 161 paired tasks per configuration and no confidence intervals, equivalence bounds, or Bayes factors, failure to reject the null hypothesis does not establish comparability. The manuscript should report per-metric effect sizes with confidence intervals and, for the claim of no meaningful degradation, a pre-specified equivalence test (e.g., TOST) or a clear bound on acceptable difference. Without such bounds, the conclusion that quantization is not a deal-breaker is stronger than the statistical evidence supports.","section":"Section V and Section VI"},{"comment":"The favorable aggregate totals mask consistent per-metric degradations that the paper itself notes. For example, PMD Performance issues on MultiPL-E-Java more than double for CodeLlama 34B (14 to 31) and DeepSeek-Coder 33B (13 to 30), and Maintainability code smells increase for several quantized CodeLlama configurations (e.g., CodeLlama 13B on MultiPL-E-Python: 29 to 32). The conclusion that qualitative aspects are preserved needs to address these directional effects explicitly rather than averaging them into a single total, or the claim should be narrowed to specific metrics.","section":"Section V, Tables I and II"},{"comment":"The manual readability and consistency evaluation samples only quantized outputs (50 predictions per language from CodeLlama 34B) and reports that 82% of them are acceptable or good. Because full-precision outputs are not rated in the same manual study, these ratings cannot be used to attribute any quality difference to quantization. In addition, the sample covers only one model family and one size, so it cannot support the cross-family conclusion. The authors should rate paired full-precision outputs as well, or explicitly restrict the claim to the absolute readability of quantized outputs.","section":"Section IV-G"}],"minor_comments":[{"comment":"The cyclomatic complexity formula is written as M = E + 2Q - N; the standard formula is M = E - N + 2P (or E - N + 2 for a single connected component). Please correct the formula and define the terms.","section":"Section IV-E"},{"comment":"The cognitive complexity formula is rendered as C = Cbase + sum_{i=1}^n nc with an undefined summation index; provide the actual definition or a more complete citation.","section":"Section IV-E"},{"comment":"The color-coding description in Section V mentions green, red, and yellow for positive, negative, and neutral effects, but the threshold for neutrality is not defined; please state the rule used to assign colors.","section":"Tables I and II"},{"comment":"The sentence 'We compute the total number of issues as the sum of SonarCloud metrics' appears in the manual-assessment subsection and appears to belong to the quantitative analysis rather than to the manual readability study; please move or clarify it.","section":"Section IV-G"},{"comment":"No versions or configuration profiles are reported for SonarCloud, Pylint, Flake8, or PMD. Static-analysis counts depend heavily on rule sets and tool versions, so the paper should report the versions and enabled rules, at least in the replication package.","section":"Section IV-D"},{"comment":"The number of generated samples per task for the main pass@1 results is not stated; ten predictions per instance are mentioned only for the variability check in Section VI. Please clarify the sampling protocol for the reported pass@1 values.","section":"Section IV-A"},{"comment":"The conclusion generalizes from AWQ 4-bit quantization to 'quantization' generally; the title, abstract, and conclusion should consistently qualify that only 4-bit AWQ was evaluated.","section":"Section VII"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for an empirical software engineering venue, and the replication package makes the proposed fixes feasible. I see no citation or novelty concern beyond minor self-citation. The main issue is interpretive overreach: the data may well support 'no large degradation detected,' but the current aggregate and non-significance-based arguments do not establish that quality is preserved. Adding equivalence bounds and per-LoC analyses would make the paper defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper answers a real question—does 4-bit AWQ quantization degrade the quality of code generated by CodeLlama and DeepSeek-Coder?—and its practical answer (“no large degradation”) is probably correct. But the evidence as presented is shakier than the conclusion, mainly because of how the headline numbers are built.\n\nWhat’s genuinely new: this is the first study I know of that looks at qualitative code attributes—maintainability, complexity, code smells, security—in quantized code models, rather than just pass@k correctness. The design is reasonable: two model families, three sizes each, two languages, two benchmarks, four static analysis tools, Wilcoxon tests with Holm correction, effect sizes, and a replication package. That’s solid empirical practice, and the per-metric results mostly show small, non-significant differences.\n\nThe soft spots are in the interpretation. Section V sums Security Hotspots, Reliability bugs, Maintainability smells, cyclomatic complexity, and cognitive complexity into a single “total issues” number. Those are not commensurable: bugs and smells are counts, while CyC and CoC are continuous complexity scores on a different scale. Summing them gives a number with no clear meaning, and comparing raw totals without normalizing by lines of code is risky because quantized models sometimes generate longer code. The claimed 6% improvement in “quality issues” is largely an artifact of this aggregation.\n\nSecond, the statistical tests mostly yield non-significant p-values, but “not significant” is not evidence of equivalence. With 42–161 tasks per configuration, the study lacks the power to detect small degradations, and there are no confidence intervals or equivalence bounds. So the conclusion should be “no large degradation detected, with reservations,” not “quantization is not a deal-breaker.” The manual readability check also only rates quantized outputs, so it can’t separate quantization effects from the model’s baseline behavior.\n\nThese are real limitations, but they’re fixable. The per-metric tests already support the cautious reading, and the paper’s own “Summary of Experimental Outcomes” is appropriately hedged. The main fix is to de-emphasize the aggregate sums, present per-metric results with CIs or equivalence tests, and soften the headline.\n\nWho gets value: SE researchers working on code generation efficiency, and practitioners considering quantized models. I’d bring it to a reading group; the discussion about metric aggregation and statistical inference would be useful.\n\nBottom line: send it to peer review after major revision. The topic is timely, the study is novel, and the core finding is plausible. Just fix the analysis framing and the conclusion wording.","headline":"Useful first look at how 4-bit AWQ quantization affects code quality, but the headline numbers are over-aggregated and the conclusion outruns the statistics.","tokens_in":21428,"tokens_out":3686,"would_cite":true,"duration_ms":38376,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"4-bit quantization keeps code quality intact in large code models.","keywords":["quantization","large code models","code generation","code quality","AWQ","static analysis","maintainability","empirical study"],"falsifier":"Rerunning the same comparison while normalizing issue counts by lines of code (or by function) would settle whether the quality preservation is real; if 4-bit models show more issues per line on a larger set of tasks, the claim that quantization preserves code quality would be contradicted by the paper's own aggregate evidence.","tokens_in":20435,"feed_emoji":"⚙️","tokens_out":6578,"duration_ms":64411,"temperature":0.7,"pith_summary":"This paper tries to establish that quantizing large code models down to 4-bit precision with Activation-aware Weight Quantization does not silently degrade the code they produce. The authors compare full-precision and AWQ-quantized versions of CodeLlama and DeepSeek-Coder on Java and Python benchmarks, measuring not only whether tests pass but also maintainability, reliability, complexity, security hotspots, and code smells with static analysis tools. Across most metrics the quantized versions look statistically indistinguishable from the full-precision ones, and on one Python benchmark the aggregate issue count is about 6 percent lower for quantized models. The conclusion, stated in the paper's own words, is that quantization is not a deal-breaker: it preserves functional correctness and most qualitative code attributes, with readability showing the only consistent signs of mild degradation.","feed_headline":"Shrinking code models to 4 bits does not wreck code quality","feed_subtitle":"AWQ compression keeps code reliability and maintainability on par with full precision across two model families and languages.","key_machinery":"The load-bearing component is Activation-aware Weight Quantization (AWQ), a post-training quantization method that decides which weights to keep at high precision by looking at activation magnitudes rather than weights alone, then applies per-channel scaling to cut the model to 4-bit with minimal quantization error. AWQ's selective preservation of salient weights is what the paper credits for compressed models behaving like their full-precision selves. The supporting machinery is the paired evaluation: each generation task is run with full-precision and quantized models, the outputs are scored by SonarCloud, Pylint, Flake8, and PMD, and the per-task metric distributions are compared with a Wilcoxon signed-rank test under Holm correction, with Cliff's delta measuring effect sizes.","core_discovery":"The paper's central discovery is that 4-bit AWQ quantization preserves the quality profile of code generated by large code models, not just their pass rates. Across two model families, two benchmarks, and two languages, quantized models produced code whose reliability, maintainability, cyclomatic and cognitive complexity, security-hotspot counts, and code-smell counts were generally comparable to full-precision output; statistical tests found significant differences only for Pylint Warning and PMD Code Style metrics, with effect sizes ranging from negligible to small. Quantized models even showed slightly lower total SonarCloud issue counts than full-precision models on McEval-Python (2,892 versus 3,084) and a marginal reduction on Java. A manual review of 50 sampled predictions per language found 82 percent rated acceptable or good, while readability was the weakest dimension, with lower inter-rater agreement and more poor ratings. The paper interprets these results as contradicting its own starting hypothesis that information loss from quantization would degrade code quality.","pith_inferences":["A likely explanation for the apparent 6 percent improvement on McEval-Python is that quantized DeepSeek models generated shorter code on that benchmark, and fewer lines naturally yield fewer flagged issues; normalizing by lines of code could turn the improvement into a wash.","Because AWQ explicitly protects activation-salient weights, its quality preservation may be specific to AWQ; comparing against weight-only methods such as GPTQ on the same quality metrics would show whether the result generalizes across quantization techniques.","The readability drop may worsen as tasks become more open-ended or as model output is used directly in production codebases, since identifier choice and structure matter more in larger programs; a developer study on real tasks would test this.","The same methodology could be applied to other LCM tasks such as bug fixing, test generation, or summarization, where compression might degrade different quality attributes than code generation."],"forward_implications":["Teams can deploy 4-bit AWQ-quantized code models for code generation with reduced memory and energy use without an anticipated drop in generated-code quality metrics.","Pass@1 differences between full-precision and quantized models are small enough that functional correctness is not the deciding factor in choosing a compressed model.","Readability is the one quality dimension that quantized models tend to hurt, so adoption should pair quantization with identifier and naming review or post-processing.","Larger models exhibit fewer quality issues whether quantized or not, and quantization's negative effects on maintainability mostly show up in the smaller 7B and 13B variants.","For most quality metrics, the absence of statistically significant differences means the practical effect of 4-bit quantization is likely small across the configurations tested."],"supporting_citations":[{"why":"Supplies the AWQ quantization method whose 4-bit models are the treatment in every comparison.","marker":"[16]"},{"why":"Establishes the prior result that quantization preserves functional correctness, the baseline this study extends to code quality.","marker":"[12]"},{"why":"Provides the CodeLlama model family under test.","marker":"[26]"},{"why":"Provides the DeepSeek-Coder model family under test.","marker":"[49]"},{"why":"Supplies the MultiPL-E benchmark used for code generation tasks.","marker":"[58]"},{"why":"Supplies the McEval benchmark used for the second set of Java and Python tasks.","marker":"[60]"},{"why":"Supplies the prior approach of using SonarQube-style static analysis to evaluate AI-generated code quality.","marker":"[44]"},{"why":"Supplies the Wilcoxon signed-rank test used for the paired quantized-versus-full-precision comparisons.","marker":"[75]"}],"fun_headline_variants":["4-bit code models keep quality, not just correctness","Quantization preserves code quality in code LLMs","4-bit AWQ keeps code quality for large code models","Shrinking code models to 4 bits spares code quality","Code model quantization: quality intact, not just accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that different kinds of static-analysis findings, such as bugs, code smells, complexity scores, and security hotspots, can be added into one total-issues number and compared fairly even though quantized models sometimes generate longer code than full-precision models.","fun_headline_variants_meta":{"raw":{"variants":["4-bit code models keep quality, not just correctness","Quantization preserves code quality in code LLMs","4-bit AWQ keeps code quality for large code models","Shrinking code models to 4 bits spares code quality","Code model quantization: quality intact, not just accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000711,"raw_usage":{"total_tokens":3219,"prompt_tokens":986,"completion_tokens":2233,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2154}},"tokens_in":602,"tokens_out":2233,"duration_ms":16619,"temperature":1.0,"reasoning_tokens":2154,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:50:28.660898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerunning the same comparison while normalizing issue counts by lines of code (or by function) would settle whether the quality preservation is real; if 4-bit models show more issues per line on a larger set of tasks, the claim that quantization preserves code quality would be contradicted by the paper's own aggregate evidence.","supporting_citations":[{"cited_title":"Towards greener yet powerful code generation via quantization: An empirical study,","cited_arxiv_id":null,"evidence_quote":"Establishes the prior result that quantization preserves functional correctness, the baseline this study extends to code quality."},{"cited_title":"Multipl-e: a scalable and polyglot approach to benchmarking neural code generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the MultiPL-E benchmark used for code generation tasks."}],"review_version":1}