Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

What makes LLM code smell: prompt wording and architecture matter more than model size.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:44 UTC pith:RPTFRL7P

load-bearing objection Useful metric-robustness and mitigation study, but the headline causal claims for architecture and prompt rest on fixed-snippet PSC, not actual generation. the 4 major comments →

arxiv 2511.15817 v6 pith:RPTFRL7P submitted 2025-11-19 cs.SE

A Causal Perspective on Measuring, Explaining and Mitigating Smells in LLM-Generated Code

Alejandro Velasco , Daniel Rodriguez-Cardenas , Dipin Khati , David N. Palacio , Luftar Rahman Alif , Denys Poshyvanyk This is my paper
classification cs.SE
keywords code smellsLLM-generated codepropensity smelly scorecausal inferenceprompt engineeringmodel architecturecode qualitystatic analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that code smells in LLM-generated code can be measured before generation completes, using a probabilistic score derived from the model's own next-token probabilities, and that causal analysis on that score can say which parts of the generation pipeline cause smells. It claims that prompt formulation and model architecture are strong, consistent levers: changing them reduces the propensity for specific smells, while scaling model size from 0.5B to 7B does little. It also shows that a simple inference-time prompt instructing the model to avoid listed smells lowers median scores for three smell types, and that showing the score to developers changes how they judge ambiguous smells. A sympathetic reader would care because the results point to cheap, actionable interventions for improving maintainability of generated code without retraining.

Core claim

The central discovery is that smell propensity, as captured by PSC, responds differently to different generation factors: average treatment effects computed under a structural causal model show near-zero effects for model size, large effects for architecture and prompt type, and mixed effects for decoding strategy. The paper uses this to establish that the model's architecture and the wording of the prompt shape the token-level probabilities that produce smelly structures, making them the actionable targets for mitigation.

What carries the argument

Propensity Smelly Score (PSC): the average token-level probability the model assigns to the span of tokens aligned with a code-smell instance; median and relative variants are used as outcomes. The causal analysis treats generation strategy, model size, architecture, and prompt type as treatments, adjusts for syntactic and lexical confounders under a back-door criterion, and estimates average treatment effects on PSC.

Load-bearing premise

The paper assumes that PSC computed on fixed code snippets — snippets treated as inputs or as model outputs given a prompt — measures the same smell-generation propensity as PSC computed on code the model actually completes, and this equivalence is asserted rather than validated.

What would settle it

For the same 500-snippet smells, generate completions under each architecture and prompt (as done for the generation-type treatment), score only the generated portion, and recompute the T3/T4 ATEs; if prompt and architecture effects shrink or model-size effects appear, the paper's causal conclusions do not describe real generation.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Prompt-based mitigation is demonstrated at inference time: median PSC falls from .80 to .67 (W0719), .76 to .72 (C0304), .52 to .23 (W0611) when the prompt instructs the model to avoid listed smells.
  • PSC carries information about smell severity that BLEU and CodeBLEU do not, supporting its use as a structural-quality signal in evaluations.
  • Model size between 0.5B and 7B has little causal effect on smell propensity, so simply scaling models will not produce cleaner code.
  • Developers shown PSC became significantly more likely to judge subtle smells as systematically model-introduced and more important to fix (5 of 15 comparisons significant).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • PSC could be repurposed as a fine-grained reward or filter signal during decoding or alignment, since it is computed from the same token probabilities used at inference.
  • The robustness result (76% of smell types stable under semantic-preserving edits) implies per-smell calibration is needed before PSC is used as a universal threshold.
  • If PSC generalizes across languages, it would give a training-free way to audit code generators for maintainability, not just correctness.
  • The fixed-snippet scoring used for architecture and prompt comparisons is a simplification; testing on true completions would tell whether the causal ranking holds in deployment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes using the Propensity Smelly Score (PSC), a token-level probabilistic metric, to measure, explain, and mitigate code smells in LLM-generated Python code. It reports four studies: (RQ1) robustness of PSC under semantic-preserving code transformations and information gain relative to BLEU/CodeBLEU; (RQ2) average treatment effects of generation strategy, model size, model architecture, and prompt type on PSC, interpreted causally; (RQ3) a prompt-based mitigation case study; and (RQ4) a user study on whether PSC helps developers assess code smells. The headline findings are that prompt formulation and model architecture decisively reduce smell propensity, model size has limited impact, and PSC is a more informative and robust quality signal than BLEU/CodeBLEU.

Significance. If the causal claims were supported, the paper would make a useful contribution: it proposes a probability-based quality signal, quantifies actionable levers (prompt design, architecture), and provides a dataset, benchmark, and implementation. The mitigation case study and user study are concrete and generally well-motivated. However, the central measurement for RQ2 is not validated against actual generation behavior for the very interventions that drive the headline conclusions, and the RQ1 information-gain comparison is partly circular. These issues are load-bearing, but they are addressable with additional experiments and control analyses, so the paper merits revision rather than rejection.

major comments (4)
  1. [§5.3.2–5.3.4, Table 4] The causal estimates for model size (T2), architecture (T3), and prompt type (T4) are computed on fixed annotated snippets, not on actually generated completions. §5.3.2 explicitly says the snippet is treated as a fixed input, §5.3.3 follows the same rationale, and §5.3.4 treats the snippet as the model output under the prompt. This measures the model's conditional token probability over a pre-existing smelly span, not the propensity to produce such spans during open-ended generation. The T1 protocol in §5.3.1 actually generates completions and analyzes only the generated portion; T2–T4 should either use the same protocol or demonstrate that fixed-snippet PSC correlates with PSC measured on generated completions. Without such validation, the RQ2 conclusion that architecture and prompt are decisive levers for reducing smell generation is unsupported.
  2. [§4.2, Eq. (3), Fig. 5] The information-gain comparison is partly circular. The severity label in Eq. (3) is defined by the proportion of tokens flagged as smelly by static analysis, while PSC in Eq. (1) averages token probabilities over exactly the smelly span. Thus PSC is constructed from the same span annotations that define the label, so its higher IG over BLEU/CodeBLEU is expected by construction. To support the claim that PSC 'aligns with structural quality,' the authors should compare against a control metric computed on the same smelly spans (e.g., corpus-based token frequency or a different probability-based baseline) or use an independent severity label. As written, RQ1's conclusion about PSC's superiority over BLEU/CodeBLEU is inflated by label leakage.
  3. [§5.3.3, Table 1] The T3 treatment labeled 'model architecture' compares four distinct checkpoints (CodeLlama-7b, Mistral-7B-v0.3, Qwen2.5-Coder-7B, StarCoder2-7b). These models differ not only in architectural design but also in pretraining data, tokenizer, training objective, and other configuration details. The estimated ATEs therefore cannot be attributed to architecture alone. The paper should either use variants that better isolate architectural factors or reframe the finding as 'model family/checkpoint effects.' Combined with the fixed-snippet measurement issue, this undermines the conclusion that architecture is a decisive and actionable lever for reducing smell propensity.
  4. [§6, §7] The paper asserts that 'refutation tests confirm the stability of the estimated ATEs' and that results remain consistent across placebo tests, randomized confounding, and subset validation, but no numerical results, tables, or sensitivity bounds are reported. Given the nonstandard measurement protocol for T2–T4, these robustness checks are essential evidence, not a footnote. The authors should report the refutation-test outputs (e.g., placebo ATE magnitudes, sensitivity-to-unobserved-confounding bounds) so the reader can judge whether the estimates are actually robust.
minor comments (6)
  1. [Table 4] The Model Size columns list T_a2, T_b2, T_b2; the third should presumably be T_c2. Also, several rows have '1.0' entries that appear to be data-rendering artifacts rather than real ATE values; please verify the table formatting.
  2. [Table 3] The table is difficult to read: smile IDs such as 'C01160' are concatenated with F-statistics, and column alignment is broken. Please reformat so each row cleanly shows ID, F_anova, p-value, eta^2, and CI.
  3. [Fig. 5] The information-gain results are shown only as a bar chart with no numeric IG values or confidence intervals. Reporting the actual IG values for PSC, BLEU, and CodeBLEU would make the comparison testable.
  4. [Eq. (2)] The empirical bounds P_min(w_k) and P_max(w_k) are said to be computed 'within the sample,' but it is unclear whether this is the evaluation dataset, a reference corpus, or per-snippet. Please specify the reference dataset and how token positions are aligned.
  5. [§4.5.5, Table 5] The user study performs 15 Mann–Whitney tests without any multiple-comparison correction. With p < .05 thresholds, about one false positive is expected by chance. The five significant results should be interpreted with this in mind, or the authors should report adjusted p-values.
  6. [§5.3.1] The notation 'P(Y_1|do(T_1)' is missing a closing parenthesis. Also, the paper states 'all decoding strategies are applied to the same model instance (M1)' but later says the 50% truncation is used; please clarify whether the model is prompted with the truncated prefix or with the full snippet.

Circularity Check

2 steps flagged

PSC's information-gain 'advantage' and the prompt ATE are partly self-definitional; the rest of the causal analysis rests on an unvalidated fixed-snippet equivalence rather than a circular derivation.

specific steps
  1. self definitional [Sec. 3 Eq. 1; Sec. 4.2 Eq. 3; Sec. 6 RQ1]
    "The PSC is computed by aggregating the model’s token-level probabilities within this span. ... θ_μ(w,i,j)=E_{k=i}^{j}[P(w_k|w_1,...,w_{k−1})] (1) ... A snippet is labeled as high severity if more than fifty percent of its tokens are flagged as smelly based on static analysis. Otherwise, it is labeled as low severity. S=(high, if n_s/n_t >0.5; low, otherwise) (3)"

    PSC averages the model's probabilities over the exact token span (i,j) that defines the smell μ, and the severity label thresholds the density of the same smelly tokens. Thus PSC and S are two different encodings of one annotation: any token-level signal on the flagged spans will raise the information gain. The paper's own text says 'Because PSC directly captures the model's generation behavior with respect to smelly structures, it should align more closely with severity labels derived from token-level analysis.' The RQ1 conclusion that PSC beats BLEU/CodeBLEU is therefore a property of the metric's construction, not an independent finding.

  2. self definitional [Sec. 5.3.4; Sec. 6 RQ2 (prompt effects), Table 4]
    "For each prompt condition, we compute the PSC for every snippet in the evaluation dataset, treating the snippet as the model output given the corresponding prompt. ... without introducing any common code smells such as: - Long functions - Deeply nested logic - Duplicated code - Hardcoded constants - Poor naming conventions - Unused variables"

    The outcome PSC is, by Eq. 1, the average probability of the tokens in a known smelly span, and the T_c^4 prompt explicitly tells the model to avoid those exact smell categories. Measuring the same fixed smelly snippets under that prompt mainly checks whether the model lowers probability for the tokens named in the instruction. The reported 'prompt design consistently reduces smell propensity' is thus close to definitional: the intervention contains the target behavior, and the outcome is computed on artifacts whose smelliness is fixed in advance.

full rationale

There are two places where a claimed result reduces to the construct's own definition. First, the RQ1 information-gain comparison: PSC and the severity label are both derived from the same smelly-token annotation (Eqs. 1 and 3), so PSC's advantage over BLEU/CodeBLEU is partly by construction. Second, the T4 prompt ATE: the strongest prompt variant names the smells to avoid and is scored on fixed snippets already known to contain those smells, making the effect a manipulation check rather than an independent estimate of generative behavior. The T2/T3 protocols also compute PSC on fixed smelly snippets, so those ATEs describe conditional token probabilities over pre-existing artifacts; the leap to 'propensity in generated code' is an unvalidated equivalence, though this is better characterized as a construct-validity threat than as equation-level circularity. Notably, RQ3 does perform actual generation, so the mitigation claim has independent support despite the T4 protocol issue. No load-bearing self-citation chain is present: PSC and the λ=0.5 threshold come from prior work by the same authors, but the paper re-defines and re-validates them here, and no uniqueness theorem is invoked. Overall score 5: the central causal story is not fully forced, but two of its headline results are substantially self-definitional.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The paper's central claims rest on a small number of hand-set thresholds and on the unvalidated equivalence between likelihood scoring on fixed snippets and actual generation behavior. The most consequential assumption is that fixed-snippet PSC measures generation propensity; it is the bridge that turns the T2-T4 ATEs into causal statements about smell generation.

free parameters (5)
  • PSC threshold λ = 0.5
    Binary 'propense' labeling and user-study framing depend on λ=0.5, carried from the authors' prior PSC benchmark [43]; not re-fit here, but it is a hand-set boundary with overlapping authorship.
  • Severity threshold (Eq. 3) = 0.5
    High/low severity label requires >50% smelly tokens; this arbitrary cut drives the information-gain comparison, where PSC is computed on exactly those smelly tokens.
  • Relative PSC min/max bounds (Eq. 2) = data-dependent (empirical P_min, P_max)
    Normalization uses empirical bounds from an unspecified reference dataset; the relative score is therefore sensitive to which dataset supplies the bounds.
  • Completion truncation ratio (T1) = 50%
    Snippets are cut to 50% of original tokens before completion; this choice determines which smells appear in the generated segment and affects all T1 ATEs.
  • Decoding hyperparameters = beam=5, top_k=50, top_p=0.9, contrastive penalty=0.6, top_k=4
    Hand-chosen hyperparameters define the T1 treatment conditions; results are conditional on these settings.
axioms (4)
  • domain assumption Pylint-identified smell spans are a valid ground truth for structural code quality.
    Dataset annotation and all PSC computations rely on Pylint labels (Sec. 5.1); if Pylint is noisy or incomplete, every downstream result is affected.
  • ad hoc to paper PSC computed on a fixed snippet (not a generated completion) is a valid proxy for generation propensity.
    T2/T3/T4 compute PSC while 'treating the snippet as a fixed input' or 'as the model output given the corresponding prompt' (Sec. 5.3.2-5.3.4); this equivalence is asserted, not validated, and is the load-bearing bridge between likelihood scoring and causal claims about generation.
  • domain assumption The SCM in Fig. 2 is correctly specified; the listed confounders are sufficient for back-door adjustment.
    ATE identification relies on no unobserved confounding beyond syntax/lexical/POS features; the authors acknowledge residual confounding in Sec. 7.
  • domain assumption Autoregressive next-token probabilities are accessible and comparable across the selected models.
    PSC is defined on token-level logits; cross-model comparisons assume these probabilities are calibrated enough for meaningful aggregation.
invented entities (1)
  • Relative propensity score θ^rel (Eq. 2) no independent evidence
    purpose: Normalize PSC to a unit range using empirical per-token min/max so smells and models can be compared.
    Introduced in Sec. 3.1 without external validation; its behavior depends on the reference dataset used for P_min/P_max, which is not fully specified.

pith-pipeline@v1.3.0-alltime-deepseek · 21542 in / 14602 out tokens · 130720 ms · 2026-08-04T06:44:48.824871+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language models (_LLMs_) have accelerated their adoption in software engineering contexts. However, concerns persist about the structural quality of the code they produce. In particular, _LLMs_ often replicate poor coding practices, introducing code smells (i.e., patterns that hinder readability, maintainability, or design integrity). Although prior research has examined the detection or repair of smells, we still lack a clear understanding of how and when these issues emerge in generated code. This paper addresses this gap by systematically **_measuring_**, **_explaining_** and **_mitigating_** smell propensity in _LLM_-generated code. We build on the Propensity Smelly Score (_PSC_), a probabilistic metric that estimates the likelihood of generating particular smell types, and establish its robustness as a signal of structural quality. Using _PSC_ as an instrument for causal analysis, we identify how generation strategy, model size, model architecture and prompt formulation shape the structural properties of generated code. Our findings show that prompt design and architectural choices play a decisive role in smell propensity and motivate practical mitigation strategies that reduce its occurrence. A user study further demonstrates that _PSC_ helps developers interpret model behavior and assess code quality, providing evidence that smell propensity signals can support human judgement. Taken together, our work lays the groundwork for integrating quality-aware assessments into the evaluation and deployment of _LLMs_ for code.

Figures

Figures reproduced from arXiv: 2511.15817 by Alejandro Velasco, Daniel Rodriguez-Cardenas, David N. Palacio, Denys Poshyvanyk, Dipin Khati, Luftar Rahman Alif.

Figure 1
Figure 1. Figure 1: Propensity Smelly Score (SCM) Computation. The python snippet at the bottom contains two code smells: C0103 (i.e., invalid-name) and C0415 (i.e., import-outside-toplevel). The aggregation function 𝜃𝜇 (𝑤,𝑖, 𝑗) (Eq. 1) supports multiple computation strategies to estimate PSC. Common approaches in￾clude taking the mean or median of token-level probabilities within the span associated with the smell. Alternati… view at source ↗
Figure 2
Figure 2. Figure 2: Our objective is to estimate the effect of four treatment vari [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Causal Analysis 𝑇1−3 Settings 5.3.1 Intervention Setting 𝑃 (𝑌1 |𝑑𝑜 (𝑇1). The 𝑇1 intervention (see ○1 in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Causal Analysis, 𝑇4 Settings T4 -Prompt Type <smelly_function> Complete the following code: <smelly_function> You are an expert software engineer who writes clean, maintainable, and production-quality code. Your goal is to complete the following code, with high readability and without introducing any common code smells: <smelly_function> You are an expert software engineer who writes clean, maintainable, a… view at source ↗
Figure 5
Figure 5. Figure 5: Information Gain (IG) Results (i.e., RQ1). BLEU CodeBLEU PSC BLEU CodeBLEU PSC BLEU CodeBLEU PSC [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mitigation Case Study Results (i.e., RQ3). Base Prompt 1 2 0 .25 .5 .75 1 C0304 W0611 W0719 You are an expert software engineer who writes clean, maintainable, and production-quality code. Your goal is to complete the provided code without introducing any code smell. Specifically, avoid the following: <smell_description> Complete the following code: <smelly_function> <smelly_function> 1 2 0.8 0.67 0.76 0.7… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair

    cs.SE 2026-05 unverdicted novelty 8.0

    SmellBench evaluates 11 LLM agent setups on 65 architectural smells, finding 47.7% best resolution rate, 63.1% false positives per experts, strong false-positive detection (κ=0.94), but aggressive repairs adding up to...

  2. SmellBench: Evaluating LLM Agents on Architectural Code Smell Repair

    cs.SE 2026-05 unverdicted novelty 7.0

    SmellBench is the first benchmark showing LLM agents resolve 47.7% of architectural code smells while accurately spotting false positives, but aggressive repairs often introduce new smells and degrade overall quality.

Reference graph

Works this paper leans on

53 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [1]

    Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training LLMs for project-specific code-summarization. arXiv:2207.04237 [cs.SE] https://arxiv.org/ abs/2207.04237

    Pith/arXiv arXiv 2022

  2. [2]

    Ahmed Aljohani and Hyunsook Do. 2024. From Fine-tuning to Output: An Em- pirical Investigation of Test Smells in Transformer-Based Test Code Generation. InProceedings of the 39th ACM/SIGAPP Symposium on Applied Computing(Avila, Spain)(SAC ’24). Association for Computing Machinery, New York, NY, USA, 1282–1291. doi:10.1145/3605098.3636058

    arXiv 2024

  3. [3]

    Sebastian Baltes and Paul Ralph. 2021. Sampling in Software Engineering Research: A Critical Review and Guidelines. arXiv:2002.07764 [cs.SE] https: //arxiv.org/abs/2002.07764

    Pith/arXiv arXiv 2021

  4. [4]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  5. [5]

    Matteo Ciniselli, Nathan Cooper, Luca Pascarella, Antonio Mastropaolo, Emad Aghajani, Denys Poshyvanyk, Massimiliano Di Penta, and Gabriele Bavota. 2022. An Empirical Study on the Usage of Transformer Models for Code Completion. IEEE Transactions on Software Engineering48, 12 (2022), 4818–4837. doi:10.1109/ TSE.2021.3128234

    arXiv 2022

  6. [6]

    Cruzes and Tore Dyba

    Daniela S. Cruzes and Tore Dyba. 2011. Recommended Steps for Thematic Synthesis in Software Engineering. In2011 International Symposium on Empirical Software Engineering and Measurement. 275–284. doi:10.1109/ESEM.2011.36

    doi:10.1109/esem.2011.36 2011

  7. [7]

    Benedetta Donato, Leonardo Mariani, Daniela Micucci, and Oliviero Riganelli

  8. [8]

    Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin

  9. [9]

    Minying Fang, Xing Yuan, Yuying Li, Haojie Li, Chunrong Fang, and Junwei Du. 2025. Enhanced Prompting Framework for Code Summarization with Large Language Models.Proc. ACM Softw. Eng.2, ISSTA, Article ISSTA072 (June 2025), 24 pages. doi:10.1145/3728949

    work page doi:10.1145/3728949 2025

  10. [11]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. arXiv:2310.068...

    Pith/arXiv arXiv 2023

  11. [12]

    Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023. InferFix: End-to-End Program Repair with LLMs. arXiv:2303.07263 [cs.SE] https://arxiv.org/abs/2303.07263

    Pith/arXiv arXiv 2023

  12. [13]

    Mohamad Khajezade, Jie Jw Wu, Fatemeh Hendijani Fard, Gema Rodríguez- Pérez, and Mohamed Sami Shehata. 2024. Investigating the Efficacy of Large Language Models for Code Clone Detection. In2024 IEEE/ACM 32nd International Conference on Program Comprehension (ICPC). 161–165

    2024

  13. [14]

    Mohammed Kharma, Soohyeon Choi, Mohammed AlKhanafseh, and David Mo- haisen. 2025. Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis. arXiv:2502.01853 [cs.CR] https://arxiv.org/abs/2502.01853

    arXiv 2025

  14. [15]

    Ranim Khojah, Francisco Gomes de Oliveira Neto, Mazen Mohamad, and Philipp Leitner. 2024. The Impact of Prompt Programming on Function-Level Code Generation. arXiv:2412.20545 [cs.SE] https://arxiv.org/abs/2412.20545

    Pith/arXiv arXiv 2024

  15. [16]

    SEMERU Lab. 2024. CodeSmellExt. https://github.com/WM-SEMERU/ CodeSmellExt. Accessed: 2025-02-12

    2024

  16. [17]

    Thanh Le-Cong, Dat Nguyen, Bach Le, and Toby Murray. 2024. Towards Re- liable Evaluation of Neural Program Repair with Natural Robustness Testing. arXiv:2402.11892 [cs.SE] https://arxiv.org/abs/2402.11892

    Pith/arXiv arXiv 2024

  17. [18]

    Feng Lin, Dong Jae Kim, Tse-Husn, and Chen. 2024. SOEN-101: Code Generation by Emulating Software Process Models Using Large Language Model Agents. arXiv:2403.15852 [cs.SE] https://arxiv.org/abs/2403.15852

    Pith/arXiv arXiv 2024

  18. [19]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy- Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zi- jian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgeni...

    Pith/arXiv arXiv 2024

  19. [20]

    Melin, Adam J

    Eric L. Melin, Adam J. Torek, Nasir U. Eisty, and Casey Kennington. 2024. Preci- sion or Peril: Evaluating Code Quality from Quantized Large Language Models. arXiv:2411.10656 [cs.SE] https://arxiv.org/abs/2411.10656

    Pith/arXiv arXiv 2024

  20. [21]

    Sarker, Leandros Maglaras, and Naeem Janjua

    Ahmad Mohsin, Helge Janicke, Adrian Wood, Iqbal H. Sarker, Leandros Maglaras, and Naeem Janjua. 2024. Can We Trust Large Language Models Generated Code? A Framework for In-Context Learning, Security Patterns, and Code Evaluations Across Diverse LLMs. arXiv:2406.12513 [cs.CR] https://arxiv.org/abs/2406.12513

    Pith/arXiv arXiv 2024

  21. [22]

    Henrique Nunes, Eduardo Figueiredo, Larissa Rocha, Sarah Nadi, Fischer Ferreira, and Geanderson Esteves. 2025. Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects. arXiv:2502.02368 [cs.SE] https: //arxiv.org/abs/2502.02368

    Pith/arXiv arXiv 2025

  22. [23]

    Ouédraogo, Yinghua Li, Kader Kaboré, Xunzhu Tang, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F

    Wendkûuni C. Ouédraogo, Yinghua Li, Kader Kaboré, Xunzhu Tang, Anil Koyuncu, Jacques Klein, David Lo, and Tegawendé F. Bissyandé. 2024. Test smells in LLM-Generated Unit Tests. arXiv:2410.10628 [cs.SE] https://arxiv.org/ abs/2410.10628

    Pith/arXiv arXiv 2024

  23. [24]

    Palacio, Daniel Rodriguez-Cardenas, Alejandro Velasco, Dipin Khati, Kevin Moran, and Denys Poshyvanyk

    David N. Palacio, Daniel Rodriguez-Cardenas, Alejandro Velasco, Dipin Khati, Kevin Moran, and Denys Poshyvanyk. 2024. Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanations. arXiv:2407.08983 [cs.SE] https://arxiv.org/abs/2407.08983

    arXiv 2024

  24. [25]

    Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Fausto Fasano, Rocco Oliveto, and Andrea De Lucia. 2018. [Journal First] On the Diffuseness and the Impact on Maintainability of Code Smells: A Large Scale Empirical Investigation. ICSE ’26, April 12–18, 2026, Rio de Janeiro, Brazil Velasco et al. In2018 IEEE/ACM 40th International Conference on Softw...

    arXiv 2018

  25. [26]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318

    2002

  26. [27]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2021. Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. arXiv:2108.09293 [cs.CR] https://arxiv.org/abs/ 2108.09293

    Pith/arXiv arXiv 2021

  27. [28]

    2018.The Book of Why: The New Science of Cause and Effect(1st ed.)

    Judea Pearl and Dana Mackenzie. 2018.The Book of Why: The New Science of Cause and Effect(1st ed.). Basic Books, Inc., USA

    2018

  28. [29]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,...

    Pith/arXiv arXiv 2025

  29. [30]

    Sánchez-García, and Juan Car- los Pérez-Arriaga

    Leonardo Criollo Ramírez, Xavier Limón, Ángel J. Sánchez-García, and Juan Car- los Pérez-Arriaga. 2024. State of the Art of the Security of Code Generated by LLMs: A Systematic Literature Review. In2024 12th International Confer- ence in Software Engineering Research and Innovation (CONISOFT). 331–339. doi:10.1109/CONISOFT63288.2024.00050

    arXiv 2024

  30. [31]

    Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sun- daresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv:2009.10297 [cs.SE] https://arxiv.org/abs/2009.10297

    Pith/arXiv arXiv 2020

  31. [32]

    Palacio, Dipin Khati, Henry Burke, and Denys Poshyvanyk

    Daniel Rodriguez-Cardenas, David N. Palacio, Dipin Khati, Henry Burke, and Denys Poshyvanyk. 2023. Benchmarking Causal Study to Interpret Large Lan- guage Models for Source Code. arXiv:2308.12415 [cs.SE] https://arxiv.org/abs/ 2308.12415

    Pith/arXiv arXiv 2023

  32. [33]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiao- qing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cris- tian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, ...

    Pith/arXiv arXiv 2024

  33. [34]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. IEEE Transactions on Software Engineering50, 1 (2024), 85–105. doi:10.1109/TSE. 2023.3334955

    arXiv 2024

  34. [35]

    Amit Sharma and Emre Kiciman. 2020. DoWhy: An End-to-End Library for Causal Inference.arXiv preprint arXiv:2011.04216(2020)

    Pith/arXiv arXiv 2020

  35. [36]

    Majumder, Maisha R

    Mohammed Latif Siddiq, Shafayat H. Majumder, Maisha R. Mim, Sourov Jajodia, and Joanna C. S. Santos. 2022. An Empirical Study of Code Smells in Transformer- based Code Generation Techniques. In2022 IEEE 22nd International Working Conference on Source Code Analysis and Manipulation (SCAM). 71–82. doi:10. 1109/SCAM55253.2022.00014

    arXiv 2022

  36. [37]

    Mohammed Latif Siddiq, Lindsay Roney, Jiahao Zhang, and Joanna C. S. San- tos. 2024. Quality Assessment of ChatGPT Generated Code and their Use by Developers. In2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR). 152–156

    2024

  37. [38]

    Weisong Sun, Yun Miao, Yuekang Li, Hongyu Zhang, Chunrong Fang, Yi Liu, Gelei Deng, Yang Liu, and Zhenyu Chen. 2025. Source Code Summarization in the Era of Large Language Models . In2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 1882–1894. doi:10.1109/ICSE55347.2025.00034

    arXiv 2025

  38. [39]

    Zhensu Sun, Chengran Yang, Chao Peng, Pengfei Gao, Xiaoning Du, Li Li, and David Lo. 2025. Bridging Developer Instructions and Code Completion Through Instruction-Aware Fill-in-the-Middle Paradigm. arXiv:2509.24637 [cs.SE] https: //arxiv.org/abs/2509.24637

    arXiv 2025

  39. [40]

    Wannita Takerngsaksiri, Micheal Fu, Chakkrit Tantithamthavorn, Jirat Pasuksmit, Kun Chen, and Ming Wu. 2025. Code Readability in the Age of Large Language Models: An Industrial Case Study from Atlassian. arXiv:2501.11264 [cs.SE] https://arxiv.org/abs/2501.11264

    Pith/arXiv arXiv 2025

  40. [41]

    Desmarais, and Giuliano Antoniol

    Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Giuliano Antoniol. 2024. Bugs in Large Language Models Generated Code: An Empirical Study. arXiv:2403.08937 [cs.SE] https: //arxiv.org/abs/2403.08937

    Pith/arXiv arXiv 2024

  41. [42]

    Michele Tufano, Fabio Palomba, Gabriele Bavota, Rocco Oliveto, Massimiliano Di Penta, Andrea De Lucia, and Denys Poshyvanyk. 2015. When and Why Your Code Starts to Smell Bad. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 1. 403–414. doi:10.1109/ICSE.2015.59

    work page doi:10.1109/icse.2015.59 2015

  42. [43]

    Palacio, and Denys Poshyvanyk

    Alejandro Velasco, Daniel Rodriguez-Cardenas, Luftar Rahman Alif, David N. Palacio, and Denys Poshyvanyk. 2025. How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study. arXiv:2412.18989 [cs.SE] https://arxiv.org/abs/2412.18989

    Pith/arXiv arXiv 2025

  43. [44]

    Cody Watson, Nathan Cooper, David Nader Palacio, Kevin Moran, and Denys Poshyvanyk. 2021. A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research. arXiv:2009.06520 [cs.SE] https://arxiv.org/ abs/2009.06520

    Pith/arXiv arXiv 2021

  44. [45]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshy- vanyk. 2020. On learning meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineer- ing (ICSE ’20). ACM. doi:10.1145/3377811.3380429

    arXiv 2020

  45. [46]

    Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk

  46. [47]

    Martin White, Christopher Vendome, Mario Linares-Vasquez, and Denys Poshy- vanyk. 2015. Toward Deep Learning Software Repositories. In2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. 334–345. doi:10.1109/ MSR.2015.38

    2015

  47. [48]

    Di Wu, Fangwen Mu, Lin Shi, Zhaoqiang Guo, Kui Liu, Weiguang Zhuang, Yuqi Zhong, and Li Zhang. 2024. iSMELL: Assembling LLMs with Expert Toolsets for Code Smell Detection and Refactoring. 1345–1357. doi:10.1145/3691620.3695508

    arXiv 2024

  48. [49]

    Chunqiu Steven Xia and Lingming Zhang. 2024. Automated Program Repair via Conversation: Fixing 162 out of 337 Bugs for $0.42 Each using ChatGPT. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis(Vienna, Austria)(ISSTA 2024). Association for Computing Machinery, New York, NY, USA, 819–831. doi:10.1145/3650212.3680323

    arXiv 2024

  49. [50]

    Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, and Junjie Chen. 2024. On the Evaluation of Large Language Models in Unit Test Generation . In 2024 39th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE). IEEE Computer Society, Los Alamitos, CA, USA, 1607–16...

    2024

  50. [51]

    Jiasheng Zheng, Boxi Cao, Zhengzhao Ma, Ruotong Pan, Hongyu Lin, Yao- jie Lu, Xianpei Han, and Le Sun. 2024. Beyond Correctness: Bench- marking Multi-dimensional Code Generation for Large Language Models. arXiv:2407.11470 [cs.SE] https://arxiv.org/abs/2407.11470

    Pith/arXiv arXiv 2024

  51. [2016]

    InProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore)(ASE ’16)

    Deep learning code fragments for code clone detection. InProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore)(ASE ’16). Association for Computing Machinery, New York, NY, USA, 87–98. doi:10.1145/2970276.2970326

    arXiv

  52. [2023]

    2023), 111741

    Out of the BLEU: How should we assess quality of the Code Generation models?Journal of Systems and Software203 (Sept. 2023), 111741. doi:10.1016/j. jss.2023.111741

    arXiv 2023

  53. [2025]

    arXiv:2502.17450 [cs.SE] https://arxiv.org/abs/2502.17450

    Studying How Configurations Impact Code Generation in LLMs: the Case of ChatGPT. arXiv:2502.17450 [cs.SE] https://arxiv.org/abs/2502.17450

    Pith/arXiv arXiv