{"id":"ae7ad879-f775-4600-a258-163d53d05e6d","arxiv_id":"2504.15989","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Refactoring smelly Java code and adding context, role, or token-limit prompts cut LLM chain-of-thought token use by roughly 15-50% in this study, but the 'no quality loss' claim rests only on shallow similarity metrics.","lead":"This paper measures how much 'thinking' an AI uses to repair code, and finds that cleaning up 'smelly' code and adding context, role, and cost prompts can cut token usage by roughly 15 to 50 percent. The practical payoff is lower API bills and faster responses for AI-assisted programming, if the quality claim survives execution-based testing.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2's ~50% refactoring saving excludes the tokens spent by the refactoring step itself; the net pipeline saving on which the paper's practical claim rests is unquantified and may be much smaller.","rationale":"The reader identified the same load-bearing premise: refactoring savings exclude the cost of refactoring. My independent read agrees. The paper's headline practical claim in Section VII and the abstract is about reducing computational overhead in LLM-driven repair workflows; an optimization that merely moves token consumption from the repair call to a preceding refactoring call is not a net optimization. The manuscript itself provides the basis for this concern in Section III-C, which states that token consumption was recorded during refactoring tasks, and in Section IV-B, which does not report or subtract those tokens. This is an omission, not a fabricated problem, and a recomputation of the pipeline total would settle it. I considered other weaknesses—the RQ1 smelly-versus-clean comparison is not matched for code length/complexity, no significance tests accompany the descriptive tables, Table IV contradicts the RQ3 findings prose, and Section IV-E admits cost-sensitive truncation can reduce correctness. These are real and reinforce the reader's CONDITIONAL verdict, but none is more directly tied to the central efficiency claim than the missing refactoring cost. The prompt-annotation result (24.5% in RQ4) compares the same code with and without annotation and is not affected by this concern; it could survive even if RQ2's net claim is revised. Because the required fix is straightforward and the paper is exploratory, CONDITIONAL rather than REJECT remains the right verdict.","tokens_in":14535,"tokens_out":6429,"duration_ms":58678,"concrete_test":"Recompute RQ2 as an end-to-end pipeline: add the DeepSeek-R1 refactoring-step token counts (which Section III-C says were recorded) to the inference tokens on refactored code, for all 300 samples, and compare the total with the baseline inference tokens on original smelly code. Report mean, median, and a paired difference test. If the refactoring was actually Tree-Sitter-based, state that explicitly and show the pipeline costs; if the net reduction is below the advertised ~50% or negative, reframe the claim as inference-only savings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV-B compares token usage on smelly code with token usage on refactored code and reports a saving of roughly 50% (Table II), but the tokens consumed to produce the refactored code are not included. That omission matters because the refactoring step is not free on the paper's own description: Section III-C says all tasks were orchestrated via API calls and that token consumption and latency were recorded 'during both code repair and refactoring tasks' using DeepSeek-R1. If the refactoring call itself consumes a comparable number of tokens, the net pipeline saving shrinks or disappears; if instead Tree-Sitter performed the refactoring at zero token cost, the paper must resolve the contradiction and disclose the pipeline. As reported, the central practical claim that refactoring before CoT reduces computational overhead is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'Token-Aware Coding Flow' to reduce Chain-of-Thought (CoT) token consumption during LLM-based code repair. Using the CodeXGLUE Java dataset, DeepSeek-R1 as the inference engine, and Tree-Sitter-based code-smell annotation, the authors compare token usage between clean and smelly code, before and after refactoring, across different smell types, with and without explicit smell annotations in prompts, and under three prompt-engineering strategies (Context Awareness, Responsibility Tuning, Cost Sensitive). The headline results are that smelly code consumes more tokens than clean code, refactoring reduces token consumption by about 50%, explicit smell annotations reduce it by about 24.5%, and prompt strategies yield further reductions, all while 'without compromising repair quality.' The paper frames this as a preliminary, exploratory study rather than a definitive solution.","tokens_in":14722,"tokens_out":4313,"duration_ms":40900,"significance":"If the empirical claims were properly supported, the paper would offer practically relevant evidence that external code hygiene and prompt design can materially reduce CoT inference cost, and it would be a useful contribution to a growing literature on token-efficient LLM use in software engineering. Strengths include the use of an external model (DeepSeek-R1) and an external dataset (CodeXGLUE), which avoids the circularity of fitting parameters to the reported claims, and the fact that the headline comparisons are, in principle, reproducible and falsifiable. However, the current evidence is largely descriptive and several load-bearing details are missing or internally inconsistent, so the significance is currently conditional on a substantive revision.","major_comments":[{"comment":"The RQ2 headline saving of roughly 50% excludes the token cost of producing the refactored code. Section IV-B states that baseline token usage was measured, 'we applied automated refactoring to the smelly code,' and then token usage was measured on the refactored code; if the refactoring step itself is performed by DeepSeek-R1 API calls, as Section III-C says token consumption was recorded 'during both code repair and refactoring tasks,' then the refactoring tokens are part of the end-to-end pipeline cost and must be reported. If Tree-Sitter performed the refactoring at zero marginal token cost, the paper must state this explicitly and reconcile it with the Section III-C description. Without this information, the central practical claim that refactoring reduces computational overhead is not established.","section":"Section IV-B and Section III-C"},{"comment":"The claims that smelly code 'significantly' increases token consumption, that refactoring yields a 'substantial reduction,' and that explicit smell annotations reduce consumption by 24.5% are based only on descriptive statistics (means, standard deviations, quartiles) with no significance tests, confidence intervals, or effect-size measures. Because RQ2 and RQ4 use paired designs (same samples under two conditions), the authors should report paired tests such as Wilcoxon signed-rank tests, and RQ1 should use an appropriate two-sample test or bootstrap intervals. The word 'significant' appears throughout the analysis but is not supported statistically anywhere.","section":"Tables I, II, VI, and Section IV-A/IV-B/IV-D"},{"comment":"The prose in the RQ3 analysis and Findings inverts the rankings in Table IV. The text says 'complicated regex expression, parameter list too long, and complicated boolean expression' incur the highest token costs and that 'binary operator in name and func name' are lower, but Table IV lists binary operator in name (0.3424) as highest, followed by complicated boolean expression (0.3361), mutation too much (0.3321), cyclomatic complexity (0.3285), and complicated regex expression (0.3221), while too long parameter list (0.1942) and func name (0.1592) are near the bottom. Additionally, Section III-B-3 says 50 samples per smell type (500 total), but Section IV-C says 30 snippets per category (300 total). These inconsistencies directly undermine the RQ3 conclusions and must be corrected.","section":"Section IV-C, Table IV, and Section III-B-3"},{"comment":"The abstract and conclusion claim savings are achieved 'without compromising repair quality,' but the evidence for functional preservation is weak and partly contradictory. In RQ2, functional consistency is assessed via CodeBLEU, but the reported mean CodeBLEU between original and refactored code is only 0.5224, and the '70% code functionality similarity threshold' mentioned in the RQ3/RQ4 analysis is not defined or justified as a functional-equivalence criterion. Moreover, Section IV-E explicitly states that Cost Sensitivity 'sometimes led to the truncation of essential code elements, resulting in reduced functional correctness and lower accuracy,' which is a direct quality compromise. The paper must either qualify the 'without compromising' claim to exclude cost-sensitive truncation or provide a valid functional-equivalence measure that supports it.","section":"Section IV-B, Tables III/V, Section IV-E, and the abstract"}],"minor_comments":[{"comment":"The sample-size inconsistency between 50 samples per smell type and 30 samples per smell type should be resolved, and the naming of smell types should be made consistent across the text and Table IV (e.g., 'parameter list too long' vs. 'too long parameter list').","section":"Section III-B-3 vs. Section IV-C"},{"comment":"The 'Time-Scaled Token Consumption' metric is never formally defined; the text says 'tokens per unit time,' but it is unclear whether this is tokens divided by inference time, tokens per second, or another normalization. A precise definition is needed because this metric is the basis of RQ1 and RQ3 comparisons.","section":"Section IV-A and Table I"},{"comment":"The paper states that all API interactions were automated via Python scripts 'to guarantee reproducibility and efficiency,' but no code, data, or artifact link is provided. At least a public repository with the scripts and the smelly/clean/refactored datasets should be supplied.","section":"Section III-C and reproducibility"},{"comment":"Reference [20] is unrelated to software engineering or LLMs (it concerns molecular dynamics), and reference [31] is cited as 'Tufano et al. (2017)' but the listed paper is by Mastropaolo et al. (2021). Both citations should be corrected or replaced.","section":"References"},{"comment":"Several presentation issues should be fixed: the heading 'I. I NTRODUCTION' contains a spacing artifact, Figure 2 mentions 'GPT-o3' while the text says 'GPT-o1,' and headings such as 'C RAFTING THE FUTURE HOUSE' and 'T RIGGERING THE NANO SURGE' are unclear and should be renamed to reflect their content.","section":"Figures and section headings"},{"comment":"Tables V and VII are referenced only loosely in the analysis, and their relationship to the '70% functionality similarity threshold' is not explained. Each table should be interpreted in the text or removed if not essential.","section":"Tables V and VII"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant topic, and the basic idea of measuring token savings from code refactoring and prompt engineering is worth pursuing. However, the current version does not establish the headline quantitative claims: the refactoring token cost is missing, no inferential statistics are reported, RQ3's text contradicts its own table, and the 'without compromising' claim conflicts with reported truncation. These are fixable with additional analysis and disclosure, so I would not recommend rejection, but the revision must be substantial and should include the omitted pipeline cost and proper statistical testing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the thing to know: this is a preliminary empirical study with one genuinely useful observation and one load-bearing accounting gap. The genuinely useful part is the measured relationship between code smells and Chain-of-Thought token consumption in DeepSeek-R1, with the prompt-annotation result (24.5% reduction) being the most credible. The load-bearing gap is RQ2: the ~50% saving from refactoring excludes the tokens spent on the refactoring API calls themselves. Section III-C says refactoring was done via DeepSeek-R1 and token consumption during refactoring was recorded, but Section IV-B only reports tokens on the original vs. refactored code. If the refactoring call costs comparable tokens, the net saving shrinks or disappears. That needs fixing before the practical claim is taken at face value.\n\nWhat else is solid: the work uses external data (CodeXGLUE) and an external model, so there is no fitting-to-input circularity. The paper is honest about being exploratory and lists Java/DeepSeek-R1 as limitations. The idea that code hygiene and prompt context can reduce inference cost is plausible and not present in the cited literature as a quantified result.\n\nWhere it is soft, in proportion: no significance tests are reported; all headline numbers are descriptive means with large standard deviations. RQ3's findings text contradicts Table IV — the table lists binary-operator-in-name as the highest growth rate, while the text calls it a minor smell and instead names parameter-list-too-long (0.19) as high-cost. The abstract's 'without compromising repair quality' is directly contradicted by Section IV-E's admission that cost-sensitive truncation sometimes produced incomplete outputs. No code, data, or prompts are shipped, so exact reproduction is impossible. The claim to be 'first' to focus on token consumption is not supportable given the cited related work ([6], [8], [10], [26]) and should be toned down.\n\nThis is not a takedown. The empirical direction is worth pursuing, and the paper could become a solid venue-level contribution after a revision that includes the refactoring token cost, significance tests, and corrected RQ3 presentation. Who gets value: researchers working on LLM cost optimization for code, and anyone deciding whether code-smell removal has downstream inference benefits. I'd bring it to a reading group for the discussion, but I would not cite it in its current form. Send it to peer review — a serious referee can handle the accounting fix and the internal contradictions. My honest verdict is major revision rather than reject.","headline":"Useful exploratory evidence that code smells inflate CoT token use, but the headline refactoring saving ignores the refactoring step's own tokens and internal inconsistencies undercut the claims.","tokens_in":15230,"tokens_out":2255,"would_cite":false,"duration_ms":20411,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Code smells inflate token consumption in Chain-of-Thought code reasoning, and refactoring or prompt annotation reduces it by roughly 50% and 24.5% respectively.","keywords":["chain-of-thought reasoning","token consumption optimization","code smells","code refactoring","prompt engineering","LLM inference cost","code repair","cost-sensitive prompting"],"falsifier":"Run the same 300 smelly-code samples through the full pipeline, counting refactoring tokens plus inference tokens on the refactored code, and compare that total with inference tokens on the original code. If the end-to-end total is not clearly lower, the headline savings figure is an accounting artifact rather than a real efficiency gain.","tokens_in":1543,"feed_emoji":"⚡","tokens_out":1606,"duration_ms":89175,"temperature":0.7,"pith_summary":"The paper tries to establish that token consumption in Chain-of-Thought (CoT) code reasoning is substantially driven by the quality of the code fed to the model, not just by the model's internal reasoning. Using code smells—recurring structural problems such as long methods and duplicated logic—as a measurable proxy for code quality, it compares 'smelly' and 'clean' Java samples and finds that smelly code consumes noticeably more tokens during inference. It then shows two external interventions: automated refactoring to remove smells cuts token use by roughly 50%, and adding an explicit smell annotation to the prompt cuts it by about 24.5%, both while preserving functional equivalence as measured by CodeBLEU and similarity scores. A third set of prompt-level strategies—context awareness, role assignment, and cost limits—adds further reductions of 15–30%. If true, the paper's contribution is that code hygiene and prompt design are economically meaningful levers for LLM inference cost, available without modifying model weights.","feed_headline":"Refactoring smelly code halves AI token use for code reasoning","feed_subtitle":"No model changes needed: cleaner code and focused prompts trim Chain-of-Thought costs without hurting repair quality.","key_machinery":"The load-bearing mechanism is the use of code smells as an externally measurable proxy for reasoning difficulty, combined with a preprocessing layer the paper calls the 'Token-Aware Coding Flow.' Code smells (long methods, duplicated logic, complicated boolean or regex expressions, and similar patterns) are detectable before inference; the paper shows they predict extra token expenditure because the model must re-verify structure and intent. On top of that, three prompt-level controls—Context Awareness (injecting file path, function names, and surrounding code), Responsibility Tuning (assigning a role such as software engineer or QA engineer), and Cost Sensitive (limiting output or token budget)—steer the same model toward shorter reasoning traces. The comparison metrics that carry the quantitative claims are tokens normalized by time or by code complexity, rather than raw token counts, so that samples of different size can be compared.","core_discovery":"The central discovery is that the token cost of CoT reasoning over code is not fixed: it reacts strongly to how the input code is written and how the prompt frames it. On smelly code the model enters repeated verification loops—the paper illustrates this with a mask-reshaping example in which the model re-derives C versus Fortran memory layout several times—whereas clean code lets it converge faster. Refactoring away smells lowers complexity-normalized token consumption from 0.1015 to 0.0576 tokens per unit complexity (about 50%), and explicit smell hints lower mean total tokens from 5,876 to 4,431 (about 24.5%). Prompt strategies deliver further savings: context awareness 15–20%, responsibility tuning 10–15%, and cost limits 20–30%, though overly tight cost limits can truncate output. The paper reads these results as evidence that external, model-agnostic interventions—refactoring and prompt engineering—can make CoT-based code repair substantially cheaper without sacrificing repair quality.","pith_inferences":["A testable extension the authors do not run: count the tokens spent by the refactoring pass itself and compare end-to-end totals; only then does the roughly 50% saving represent net cost reduction rather than a shift of cost to a preprocessing step.","The same mechanism suggests a broader principle: any input property that reduces the model's need to re-verify assumptions—clear naming, explicit type information, or stated invariants—should reduce token spend, and this could be tested by perturbing clean code with injected ambiguities and measuring token deltas.","The cost-sensitive results hint at a trade-off frontier: token budgets can be tuned to sit just above the point where output truncation starts, yielding maximum savings per unit of quality loss; mapping that frontier per task would be a practical follow-up.","Because the interventions are external to the model, they are likely portable to other models and languages; comparing savings across model families would reveal whether external code hygiene matters more for weaker or stronger reasoners."],"forward_implications":["If refactoring really halves token use, engineering teams can treat code-smell removal as a direct inference-cost optimization, not just a maintainability practice.","Explicit smell annotations in prompts are nearly free and cut tokens by about 24.5%, suggesting that cheap annotations can substitute for costly refactoring when refactoring is impractical.","Because different smell types carry different token burdens, with expression-related and structural smells costing the most, prioritizing removal of those categories yields the largest efficiency gain.","Prompt-level steering (context, role, and cost limits) offers 10–30% savings and can be combined, indicating that prompt design is a reusable efficiency knob that does not require retraining.","If these results generalize beyond the tested Java and reasoning-model setup, API cost forecasting for LLM-based coding assistants should include input code quality as a variable."],"supporting_citations":[{"why":"Supplies the reasoning model whose Chain-of-Thought traces are measured.","marker":"[1]"},{"why":"Documents token inflation in LLM code generation and motivates the efficiency objective.","marker":"[6]"},{"why":"Documents long Chain-of-Thought traces as a source of token overhead.","marker":"[10]"},{"why":"Defines code smells and refactoring, the conceptual basis for treating smells as a cost driver.","marker":"[12]"},{"why":"Provides the code-smell categories used to construct the ten-type dataset.","marker":"[13]"},{"why":"Shows that LLMs can perform code refactoring, supporting the automated refactoring step.","marker":"[19]"},{"why":"Links Chain-of-Thought prompting to neural code generation, contextualizing CoT token costs.","marker":"[29]"},{"why":"Supplies the benchmark code samples used to build the smelly and clean datasets.","marker":"[32]"}],"fun_headline_variants":["Clean code halves LLM token use for code reasoning","Nano Surge trims tokens without hurting code repair quality","Refactoring cuts AI token costs in code tasks by half","Context-aware prompts reduce LLM token use by 20%","Cost limits decrease LLM token consumption up to 30%"],"cache_read_input_tokens":17536,"weakest_assumption_plain":"The load-bearing premise is that the token cost of the refactoring step itself can be ignored in the comparison; if that cost is comparable to the roughly 50 percent saving, the net efficiency gain shrinks or disappears.","fun_headline_variants_meta":{"raw":{"variants":["Clean code halves LLM token use for code reasoning","Nano Surge trims tokens without hurting code repair quality","Refactoring cuts AI token costs in code tasks by half","Context-aware prompts reduce LLM token use by 20%","Cost limits decrease LLM token consumption up to 30%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3368,"prompt_tokens":940,"completion_tokens":2428,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":2344}},"tokens_in":556,"tokens_out":2428,"duration_ms":21183,"temperature":1.0,"reasoning_tokens":2344,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:13:07.681782+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 300 smelly-code samples through the full pipeline, counting refactoring tokens plus inference tokens on the refactored code, and compare that total with inference tokens on the original code. If the end-to-end total is not clearly lower, the headline savings figure is an accounting artifact rather than a real efficiency gain.","supporting_citations":[{"cited_title":"USA: Addison- Wesley Longman Publishing Co., Inc., 1999","cited_arxiv_id":null,"evidence_quote":"Defines code smells and refactoring, the conceptual basis for treating smells as a cost driver."},{"cited_title":"Bad smells - humans as code critics,","cited_arxiv_id":null,"evidence_quote":"Provides the code-smell categories used to construct the ten-type dataset."},{"cited_title":"When to Stop? Towards Efficient Code Generation in LLMs with Excess Token Prevention","cited_arxiv_id":"2407.20042","evidence_quote":"Documents token inflation in LLM code generation and motivates the efficiency objective."}],"review_version":1}