Pith. sign in

REVIEW 3 major objections 6 minor 56 references

Five LLM code smells are widespread in open-source systems and detectable with 86% precision.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:05 UTC pith:IZONPCO3

load-bearing objection A useful, honest five-smell catalog with a real detector; the 60.5% prevalence is dataset-bound, not a population estimate. the 3 major comments →

arxiv 2512.18020 v1 pith:IZONPCO3 submitted 2025-12-19 cs.SE cs.AI

Specification and Detection of LLM Code Smells

Brahim Mahmoudi , Zacharie Chenail-Larcher , Naouel Moha , Quentin Sti\'evenart , Florent Avellaneda This is my paper
classification cs.SE cs.AI
keywords LLM code smellscode smellsLLM integrationstatic analysisspecification-driven detectionsoftware qualityprevalence studyPython
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LLM code smells: recurring source-code patterns around LLM inference calls that do not cause explicit bugs but degrade maintainability, reliability, performance, or robustness. It formalizes five such smells — unbounded max metrics, no model version pinning, no system message, no structured output, and temperature not explicitly set — and extends a static analyzer to detect them. Running the detector on 200 open-source Python systems that integrate LLMs, the authors find that 60.50% of systems exhibit at least one smell, with a manual-audit precision of 86.06%. If correct, this gives practitioners a concrete, checkable catalog for LLM integration quality and a tool to enforce it.

Core claim

The paper's claim is that a formal catalog of LLM code smells can be specified at the source-code level and automatically detected, and that these smells are not rare: in a dataset of 200 open-source Python systems that invoke LLMs, 121 (60.50%) trigger at least one of the five smells. Per-smell prevalence ranges from 34.5% for missing system messages to 40.5% for no structured output. A stratified manual audit of detections yields total precision of 86.06%, with per-smell precision between 80.0% and 95.7%. The authors position this as a precision-oriented prevalence study; they do not estimate recall, which would require exhaustive file-level ground truth.

What carries the argument

The central object is the catalog of five LLM code smells, each specified by a fixed schema (name, context, problem, solution, example, sources, effects). The detection mechanism is SpecDetect4LLM, an extension of a specification-driven static analyzer that uses DSL-defined rules and semantic predicates to match LLM call-site patterns without direct AST manipulation. The catalog gives the rules their content; the analyzer makes the smells observable at scale, enabling the prevalence and precision statistics that carry the empirical argument.

Load-bearing premise

The load-bearing premise is that the 200 open-source Python systems assembled for the study represent LLM-integrating software systems generally; if that convenience sample is biased, the 60.50% prevalence and 86.06% precision figures may not generalize.

What would settle it

Re-run SpecDetect4LLM on a probability-based sample of LLM-integrating repositories drawn without star or recency filters, and manually classify every alert rather than a stratified subset. If true prevalence among such a sample falls well below 60%, or if full-audit precision (or recall) comes out far below the reported levels, the paper's claims about widespread, detectable LLM code smells would be weakened.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Teams can adopt the five smells as concrete, reviewable coding guidelines for LLM integration, since each comes with a remediation: bound limits, pin versions, add a system message, enforce an output schema, and set temperature explicitly.
  • Governance-related smells (no version pinning, implicit temperature) concentrate inside projects, so fixing them once through wrappers or configuration can remove many alerts; engineering-related smells (unstructured output, unbounded metrics) spread across projects and likely require cross-cutting refactors.
  • Static detection with precision around 86% is sufficient to surface candidate issues, but the unmeasured recall means the tool's usefulness as a standalone audit depends on future recall measurement.
  • The catalog and detector provide a foundation for extending LLM code-smell checking to other languages and for dynamic detection that captures execution-dependent effects.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the 60.50% prevalence generalizes beyond the study's convenience sample, most production LLM integrations carry technical debt that current testing practices do not flag; a randomized census of LLM-integrating repositories would be a stronger test.
  • Because the smells are defined as quality debt rather than bugs, their business impact (cost, incident rate, maintenance effort) is not yet quantified; a controlled before/after study of remediating the unstructured-output and unbounded-metrics smells could test whether the catalog pays off.
  • The same five patterns likely appear in non-Python LLM code, since the underlying constraints (token limits, model aliases, chat roles, schemas, sampling parameters) are provider-level, not language-level; porting the rules to other ecosystems would be a natural next step.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces the concept of LLM code smells and defines a catalog of five recurrent, source-code-level problematic practices in LLM-integrating systems: Unbounded Max Metrics (UMM), No Model Version Pinning (NMVP), No System Message (NSM), No Structured Output (NSO), and LLM Temperature Not Explicitly Set (TNES). It extends the static analyzer SpecDetect4AI into SpecDetect4LLM, then applies the tool to a manually-curated dataset of 200 open-source Python systems. The reported results are that 60.50% of the analyzed systems are affected by at least one LLM code smell, and a manual audit of 109 detections yields an average precision of 86.06% across the five smells. The paper positions the catalog and tool as a first step toward coding guidelines for LLM integration quality.

Significance. The catalog and detection tool are a timely and useful contribution, filling a gap between high-level taxonomies of LLM defects and concrete source-code-level practices. The five smells are well-defined with examples, and the detection tool is released as a versioned module with tests and a replication package. The prevalence study, while explicitly precision-oriented and limited by the lack of recall estimation, offers an initial quantitative indication that these smells are widespread. If the findings hold, the catalog can serve as a practical basis for API-usage guidelines and for future automated refactoring support. The study is honest about its limitations, and the release of artifacts supports reproducibility.

major comments (3)
  1. [§4.3, §7] The dataset is a convenience sample assembled via GitHub API keyword searches (LLM keywords, Python, stars>20, recency) merged with Shao et al.'s 100-system set. No sampling frame, inclusion/exclusion audit, or representativeness analysis is provided. The abstract carefully qualifies the headline as 'of the analyzed systems,' but the conclusion (§7) states that the findings 'confirm their relevance and the necessity of such a catalog,' which generalizes beyond the studied systems. The external validity of the 60.50% prevalence figure is therefore a load-bearing weakness. Please add a limitations paragraph that restricts prevalence claims to the study sample, temper the concluding generalization, and if possible include basic demographic comparisons to a broader crawl of LLM-integrating repositories.
  2. [§4.3, §5, Table 1] Cohen's κ is mentioned in the methodology but the value is never reported. The precision audit uses only 20–23 detections per smell (109 total), and the average precision of 86.06% is presented without confidence intervals. Given that the prevalence figures are tool outputs, the precision estimate is central to the claim that the detected patterns are real. Please report κ, a precise description of the stratified sampling procedure (random draw boundaries, stratification variables), and exact binomial confidence intervals for each per-smell precision in Table 1.
  3. [§4.1] The catalog admission criterion 'cross-source support (>5 of academic / grey / community / code change)' is ambiguous. It is unclear whether this is a count of documents, of distinct source categories, or of evidence instances, and whether all listed categories must be represented. Since the catalog is the direct basis for the detection rules, this vagueness limits reproducibility. Please specify the exact counting procedure and provide a worked example of how one smell (e.g., NSO) met the threshold.
minor comments (6)
  1. [Abstract] The abstract says the catalog is 'based on relevant literature,' but §4.1 also uses grey literature and empirical artifacts. Please update the wording to reflect the multi-source triangulation.
  2. [§4.2] The statement that semantic predicates avoid 'any direct abstract syntax tree manipulation' is unclear. A one-sentence explanation of how the DSL-based rules work would help readers assess the tool's architecture.
  3. [§2] The distinction between prompt smells (which operate on prompts) and LLM code smells (which operate on source code) is important but only implicit. Please make this contrast explicit in the related-work discussion.
  4. [Table 1] The table footer defines '##' as the number of affected systems, but the table rows show fractions like '76/200'. Please make the notation clearer, e.g., 'Affected systems (out of 200)'.
  5. [§5] The phrase 'in more than half affected projects (69/121)' is correct for NSM but could be misinterpreted; consider rephrasing to 'present in 69 of the 121 systems affected by at least one smell' for clarity.
  6. [Listings] The examples use red/green colors to denote bad/good code. The colors may not be discernible in grayscale printing; consider adding textual labels (e.g., 'bad'/'good') in the code comments.

Circularity Check

0 steps flagged

No significant circularity: the tool operationalizes the catalog, but manual precision audit and a separate dataset prevent circular reduction; one minor self-citation is not load-bearing.

full rationale

The paper's derivation chain is: (1) build a catalog of five LLM code smells from literature, grey literature, and empirical artifacts; (2) implement detection rules in SpecDetect4LLM derived from that catalog; (3) run the tool on 200 systems and manually audit a stratified sample of detections to estimate precision. The potential circularity—that the tool merely detects patterns it was built to detect, so prevalence is self-fulfilling—is mitigated by the manual dual-review audit reporting 86.06% precision (Table 1, §5). This audit uses external human judgment with a pre-defined decision sheet, providing independent support that the tool's alerts correspond to the catalog's intended smells. The prevalence figures are therefore measurements of tool detections that have been validated for precision, not pure tautologies. The only self-citation is [26], the authors' prior SpecDetect4AI, which is used as the base tool; the paper states it selected that tool because it 'demonstrated high accuracy against prior baselines,' but the current precision audit is performed on the new tool, so the self-citation is not load-bearing for the central empirical claims. The dataset combines a GitHub API crawl with Shao et al.'s systems, which is a convenience sample—an external-validity concern about generalization of the 60.50% prevalence, not a circularity concern. The paper also transparently discloses that recall was not estimated and the precision sample size is small. No equation-level reduction, fitted-parameter renaming, or imported uniqueness theorem is present. Overall, the central derivation is self-contained and independently grounded by the audit; any circular flavor is inherent to any tool-based empirical study and does not rise to a significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The main quantities the central claim depends on are the chosen thresholds in dataset assembly and the precision audit (stars>20, ≥20 detections per smell, >5 sources for catalog admission), plus the domain assumptions that the smells harm quality, that static rules capture them, that the sample represents the population, and that manual labeling is reliable. There are no fitted mathematical parameters and no new physical or formal entities.

free parameters (3)
  • Dataset popularity threshold = stars > 20
    GitHub API filter in §4.3; affects which systems are studied and therefore the reported prevalence.
  • Precision audit sample size = ≥20 detections per smell, ≥5 systems
    Manual audit sampling rule in §4.3; chosen by hand and constrains the precision estimates (109 total sampled alarms).
  • Catalog admission threshold = >5 cross-source evidences
    Admission criterion in §4.1; determines which candidate practices are accepted as LLM code smells.
axioms (4)
  • domain assumption The five listed practices degrade robustness, performance, maintainability, or reliability.
    Table 1 and §3 assert effects via cited literature and grey documentation; the claimed harm is never measured in the studied systems. If wrong, calling these practices 'code smells' is unsupported.
  • domain assumption Static detection rules are valid operationalizations of the catalog.
    §4.2 maps each smell to DSL rules and predicates; recall is not estimated, so the fraction of missed instances is unknown.
  • domain assumption The GitHub keyword+stars+recency crawl combined with Shao et al.'s set approximates the population of LLM-integrating Python systems.
    §4.3 dataset extraction; convenience sampling threatens external validity of the 60.50% prevalence figure.
  • domain assumption Manual dual-review labels (true/false positive) are reliable ground truth.
    §4.3 mentions computing Cohen's κ but does not report the value; inter-rater reliability is unverified in the paper.

pith-pipeline@v1.3.0-alltime-deepseek · 9622 in / 12725 out tokens · 112519 ms · 2026-08-03T15:05:50.247104+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) have gained massive popularity in recent years and are increasingly integrated into software systems for diverse purposes. However, poorly integrating them in source code may undermine software system quality. Yet, to our knowledge, there is no formal catalog of code smells specific to coding practices for LLM inference. In this paper, we introduce the concept of LLM code smells and formalize five recurrent problematic coding practices related to LLM inference in software systems, based on relevant literature. We extend the detection tool SpecDetect4AI to cover the newly defined LLM code smells and use it to validate their prevalence in a dataset of 200 open-source LLM systems. Our results show that LLM code smells affect 60.50% of the analyzed systems, with a detection precision of 86.06%.

Figures

Figures reproduced from arXiv: 2512.18020 by Brahim Mahmoudi, Florent Avellaneda, Naouel Moha, Quentin Sti\'evenart, Zacharie Chenail-Larcher.

Figure 1
Figure 1. Figure 1: presents our three-step methodology. 4.1 Step 1: Catalog Construction Goal: Consolidate evidence from multiple sources into an LLM code smells catalog with precise, implementation-oriented specifications. Inputs: i) Academic literature on LLM engineering and efficien￾cy/reliability; ii) Grey literature (engineering blogs, provider docu￾mentations); iii) Empirical artifacts (GitHub, Stack Overflow (SO)). Ou… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 3 canonical work pages · 3 internal anchors

  1. [1]

    Amey Agrawal, Nitin Kedia, Anmol Agarwal, Jayashree Mohan, Nipun Kwatra, Souvik Kundu, Ramachandran Ramjee, and Alexey Tumanov. 2025. On Evaluating Performance of LLM Inference Serving Systems. (2025). arXiv:2507.09019 [cs.LG] https://arxiv.org/abs/2507.09019

    Pith/arXiv arXiv 2025

  2. [2]

    2025.Timeouts, retries and backoff

    Amazon Web Services. 2025.Timeouts, retries and backoff. https://aws.amazon. com/fr/builders-library/timeouts-retries-and-backoff-with-jitter

    2025

  3. [3]

    2025.Claude Documentation (API, Models)

    Anthropic. 2025.Claude Documentation (API, Models). https://docs.claude.com/ Specification and Detection of LLM Code Smells Conference’17, July 2017, Washington, DC, USA

    2025

  4. [4]

    2025.structured-logprobs

    arena-ai. 2025.structured-logprobs. https://github.com/arena-ai/structured- logprobs OS library: enhances OpenAI Structured Outputs with token logprobs

    2025

  5. [5]

    2025.Replica- tion_Package_LLM_code_smells

    Mahmoudi Brahim and Chenail-larcher Zacharie. 2025.Replica- tion_Package_LLM_code_smells. https://github.com/Brahim-Mahmoudi/ SpecDetect4LLM_ICSE

    2025

  6. [6]

    Alessio Bucaioni, Martin Weyssow, Junda He, Yunbo Lyu, and David Lo. 2025. A Functional Software Reference Architecture for LLM-Integrated Systems. (2025). arXiv:2501.12904 [cs.SE] https://arxiv.org/abs/2501.12904

    Pith/arXiv arXiv 2025

  7. [7]

    Carvalho, Maurício Aniche, João Veríssimo, Alessandro Garcia, Vitor Alves, and Rohit Gheyi

    Silvio G. Carvalho, Maurício Aniche, João Veríssimo, Alessandro Garcia, Vitor Alves, and Rohit Gheyi. 2019. An empirical catalog of code smells for the pre- sentation layer of Android apps.Empirical Software Engineering24, 6 (dec 2019), 3546–3586. doi:10.1007/s10664-019-09768-9

    doi:10.1007/s10664-019-09768-9 2019

  8. [8]

    Pan, Shuyi Yang, Lakshya A

    Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ram- chandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. Why Do Multi-Agent LLM Systems Fail? (2025). arXiv:2503.13657 [cs.AI] https: //arxiv.org/abs/2503.13657

    Pith/arXiv arXiv 2025

  9. [9]

    Zixi Chen, Yinyu Ye, and Zijie Zhou. 2025. Adaptively Robust LLM Inference Optimization under Prediction Uncertainty. (2025). arXiv:2508.14544 [cs.LG] https://arxiv.org/abs/2508.14544

    Pith/arXiv arXiv 2025

  10. [10]

    2025.System Messages: Best Practices, Real-world Experiments & Prompt Injection Protectors

    Dan Cleary. 2025.System Messages: Best Practices, Real-world Experiments & Prompt Injection Protectors. https://www.prompthub.us/blog/everything-system- messages-how-to-use-them-real-world-experiments-prompt-injection- protectors PromptHub Blog

    2025

  11. [11]

    2023.What is the use case of System role

    cyz3a5c0v1. 2023.What is the use case of System role. https://stackoverflow.com/ questions/76272624/what-is-the-use-case-of-system-role

    arXiv 2023

  12. [12]

    2025.A Practical Guide on Structuring LLM Outputs with Pydan- tic

    Developer Service. 2025.A Practical Guide on Structuring LLM Outputs with Pydan- tic. https://dev.to/devasservice/a-practical-guide-on-structuring-llm-outputs- with-pydantic-50b4

    2025

  13. [13]

    Ali Mohammadi Esfahani, Nafiseh Kahani, and Samuel A. Ajila. 2024. Un- derstanding Defects in Generated Codes by Language Models. (2024). arXiv:2408.13372 [cs.SE] https://arxiv.org/abs/2408.13372

    Pith/arXiv arXiv 2024

  14. [14]

    Martin Fowler, Kent Beck, John Brant, William Opdyke, and Don Roberts. 1999. Refactoring: improving the design of existing code. Addison-Wesley Longman Publishing Co., Inc., USA

    1999

  15. [15]

    2025.Gemini API - Google AI for Developers

    Google. 2025.Gemini API - Google AI for Developers. https://ai.google.dev/

    2025

  16. [16]

    2025.Quotas and limits - BigQuery

    Google Cloud. 2025.Quotas and limits - BigQuery. https://cloud.google.com/ bigquery/quotas

    2025

  17. [17]

    Tingxu Han, Zhenting Wang, Chunrong Fang, Shunyuan Zhao, Shiqing Ma, and Ziang Chen. 2025. Token-Budget-Aware LLM Reasoning. InACL 2025. Association for Computational Linguistics, 24842–24855. doi:10.18653/v1/2025.findings-acl. 1274

    doi:10.18653/v1/2025.findings-acl 2025

  18. [18]

    2025.Transformers Documentation (Generation, Chat Templates, Model Revisions)

    Hugging Face. 2025.Transformers Documentation (Generation, Chat Templates, Model Revisions). https://huggingface.co/docs/transformers

    2025

  19. [19]

    Minbyul Jeong, Jungho Cho, Minsoo Khang, Dawoon Jung, and Teakgyu Hong

  20. [20]

    Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, Caiming Xiong, and Shafiq Joty. 2025. A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systems. (2025). arXiv:2504.09037 [cs.AI] https://arxiv.org/abs/2504.09037

    arXiv 2025

  21. [21]

    2024.Enforcing JSON Outputs in Commercial LLMs

    Daniel Kharitonov. 2024.Enforcing JSON Outputs in Commercial LLMs. https://medium.com/data-science/enforcing-json-outputs-in-commercial- llms-3db590b9b3c8 Towards Data Science

    2024

  22. [22]

    2024.Uncovering the Reliability and Consistency of AI Language Models: A Systematic Study

    Aisha Khatun. 2024.Uncovering the Reliability and Consistency of AI Language Models: A Systematic Study. Ph. D. Dissertation. University of Waterloo. https: //uwspace.uwaterloo.ca/items/e01e11a6-e033-4f6a-85c6-849fba74e039

    2024

  23. [23]

    2007.Guidelines for perform- ing Systematic Literature Reviews in Software Engineering

    Barbara Kitchenham and Stuart Charters. 2007.Guidelines for perform- ing Systematic Literature Reviews in Software Engineering. Technical Report EBSE-2007-01. EBSE 2007. https://www.elsevier.com/__data/promis_misc/ 525444systematicreviewsguide.pdf

    2007

  24. [24]

    Pierre Le Jeune, Jiaen Liu, Luca Rossi, and Matteo Dora. 2025. RealHarm: A Collection of Real-World Language Model Application Failures. InLLMSEC 2025. 87–100

    2025

  25. [25]

    Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J

    Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J. Cai. 2024. We Need Structured Output: To- wards User-centered Constraints on Large Language Model Output. InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems. Associa- tion for Computing Machinery, New York, NY, USA....

    arXiv 2024

  26. [26]

    Brahim Mahmoudi, Naouel Moha, Quentin Stievenert, and Florent Avellaneda

  27. [27]

    2023.PydanticOutputParser has high chance failing when completion contains new line #3709

    mariafilippa. 2023.PydanticOutputParser has high chance failing when completion contains new line #3709. https://github.com/hwchase17/langchain/issues/3709 GitHub issue

    2023

  28. [28]

    AI-Specific Code Smells: From Specification to Detection. (2025). arXiv:2509.20491 [cs.SE] doi:10.48550/arXiv.2509.20491

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.20491 2025

  29. [29]

    2025.Design to Support Foundation Model Life Cy- cles

    Microsoft Learn. 2025.Design to Support Foundation Model Life Cy- cles. https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/manage- foundation-models-lifecycle

    2025

  30. [30]

    2025.Azure OpenAI - Documentation (Quotas, Structured Outputs, How-To)

    Microsoft Learn. 2025.Azure OpenAI - Documentation (Quotas, Structured Outputs, How-To). https://learn.microsoft.com/en-us/azure/ai-foundry/openai/

    2025

  31. [31]

    2024.How To Ensure LLM Output Adheres to a JSON Schema

    Modelmetry. 2024.How To Ensure LLM Output Adheres to a JSON Schema. https: //modelmetry.com/blog/how-to-ensure-llm-output-adheres-to-a-json-schema

    2024

  32. [32]

    Roush, Andreas Kirsch, and Ravid Shwartz-Ziv

    Nguyen Nhat Minh, Andrew Baker, Clement Neo, Allen G. Roush, Andreas Kirsch, and Ravid Shwartz-Ziv. 2025. Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs. InICLR 2025

    2025

  33. [33]

    Masumi Morishige and Ryo Koshihara. 2025. Ensuring Reproducibility in Gener- ative AI Systems for General Use Cases: A Framework for Regression Testing and Open Datasets. (5 2025). arXiv:2505.02854 doi:10.48550/arXiv.2505.02854

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.02854 2025

  34. [34]

    João Eduardo Montandon, Luciana Lourdes Silva, Cristiano Politowski, Daniel Prates, Arthur de Brito Bonifácio, and Ghizlane El Boussaidi. 2025. Unbox- ing Default Argument Breaking Changes in Data Science Libraries. (2025). arXiv:2408.05129 [cs.SE] doi:10.48550/arXiv.2408.05129 JSS

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2408.05129 2025

  35. [35]

    Kaiwen Ning, Jiachi Chen, Jingwen Zhang, Wei Li, Zexu Wang, Yuming Feng, Weizhe Zhang, and Zibin Zheng. 2024. Defining and Detecting the Defects of the Large Language Model-based Autonomous Agents. (2024). arXiv:2412.18371 [cs.SE] https://arxiv.org/abs/2412.18371

    Pith/arXiv arXiv 2024

  36. [36]

    Anna Neumann, Elisabeth Kirsten, Muhammad Bilal Zafar, and Jatinder Singh

  37. [37]

    InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency

    Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs). InProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency. ACM, 573–598. doi:10.1145/3715275.3732038

    arXiv 2025

  38. [38]

    2025.OpenAI Platform Documentation

    OpenAI. 2025.OpenAI Platform Documentation. https://platform.openai.com/ docs

    2025

  39. [39]

    2025.Modelfile: Valid parameters and values

    Ollama. 2025.Modelfile: Valid parameters and values. https://github.com/ollama/ ollama/blob/main/docs/modelfile.md#valid-parameters-and-values

    2025

  40. [40]

    2025.API Reference - Structured model outputs

    OpenAI. 2025.API Reference - Structured model outputs. https://platform.openai. com/docs/guides/structured-outputs

    2025

  41. [41]

    Frank Reyes, Yogya Gamage, Gabriel Skoglund, Benoit Baudry, and Martin Mon- perrus. 2024. BUMP: A Benchmark of Reproducible Breaking Dependency Up- dates. InSANER 2024. arXiv:2401.09906 doi:10.48550/arXiv.2401.09906

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.09906 2024

  42. [42]

    2025.openai-python

    OpenAI. 2025.openai-python. https://github.com/openai/openai-python

    2025

  43. [43]

    2025.OpenRouter.ai - One API for Any Model

    OpenRouter. 2025.OpenRouter.ai - One API for Any Model. https://openrouter.ai/

    2025

  44. [44]

    Haoye Tian, Chong Wang, BoYang Yang, Lyuye Zhang, and Yang Liu. 2025. A Taxonomy of Prompt Defects in LLM Systems. (2025). arXiv:2509.14404 [cs.SE] https://arxiv.org/abs/2509.14404

    arXiv 2025

  45. [45]

    Krishna Ronanki, Beatriz Cabrero-Daniel, and Christian Berger. 2024. Prompt Smells: An Omen for Undesirable Generative AI Outputs. (2024). arXiv:2401.12611 [cs.LG] https://arxiv.org/abs/2401.12611

    Pith/arXiv arXiv 2024

  46. [46]

    Yuchen Shao, Yuheng Huang, Jiawei Shen, Lei Ma, Ting Su, and Chengcheng Wan. 2025. Are LLMs Correctly Integrated into Software Systems? (2025). arXiv:2407.05138 [cs.SE] https://arxiv.org/abs/2407.05138

    Pith/arXiv arXiv 2025

  47. [47]

    Greg Wilson, D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven H. D. Haddock, Kathryn D. Huff, Ian M. Mitchell, Mark D. Plumbley, Ben Waugh, Ethan P. White, and Paul Wilson. 2014. Best Practices for Scientific Computing.PLOS Biology12, 1 (2014), e1001745

    2014

  48. [48]

    2025.LLM Temperature: How It Works and When You Should Use It

    Vellum AI. 2025.LLM Temperature: How It Works and When You Should Use It. https://www.vellum.ai/llm-parameters/temperature

    2025

  49. [49]

    I depended on you and you broke me: An empirical study of manifesting breaking changes in client packages

    Daniel Venturini, Filipe Roseiro Cogo, Ivanilton Polato, Marco A. Gerosa, and Igor Scaliante Wiese. 2023. I Depended on You and You Broke Me: An Em- pirical Study of Manifesting Breaking Changes in Client Packages. (2023). arXiv:2301.04563 doi:10.48550/arXiv.2301.04563 TOSEM, 2023

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2301.04563 2023

  50. [50]

    Zhiqiu Xia, Lang Zhu, Bingzhe Li, Feng Chen, Qiannan Li, Chunhua Liao, Feiyi Wang, and Hang Liu. 2025. Analyzing 16,193 LLM Papers for Fun and Profits. (2025). arXiv:2504.08619 [cs.DL] https://arxiv.org/abs/2504.08619

    Pith/arXiv arXiv 2025

  51. [51]

    Cailin Winston and René Just. 2025. A Taxonomy of Failures in Tool-Augmented LLMs. InAST 2025. 125–135. doi:10.1109/AST66626.2025.00019

    arXiv 2025

  52. [52]

    2024.How to Validate the Output of LLM-Based Products

    Matt Wyman and Sarah Barber. 2024.How to Validate the Output of LLM-Based Products. https://okareo.com/blog/posts/validate-llm-output

    2024

  53. [53]

    Terry Yue Zhuo, Junda He, Jiamou Sun, Zhenchang Xing, David Lo, John Grundy, and Xiaoning Du. 2025. Identifying and Mitigating API Misuse in Large Language Models. (2025). arXiv:2503.22821 [cs.SE] https://arxiv.org/abs/2503.22821

    arXiv 2025

  54. [54]

    Wenli Yang, Lilian Some, Michael Bain, and Byeong Kang. 2025. A comprehensive survey on integrating large language models with knowledge-based methods. Knowledge-Based Systems318 (2025), 113503. doi:10.1016/j.knosys.2025.113503

    arXiv 2025

  55. [55]

    Haiyin Zhang, Luís Cruz, and Arie van Deursen. 2022. Code Smells for ML Applications. (2022). arXiv:2203.13746 [cs.SE] https://arxiv.org/abs/2203.13746

    Pith/arXiv arXiv 2022

  56. [2025]

    System Message Generation for User Preferences using Open-Source Models. (2025). arXiv:2502.11330 [cs.CL] https://arxiv.org/abs/2502.11330

    Pith/arXiv arXiv 2025