Pith. sign in

REVIEW 3 major objections 6 minor 39 references

Decomposing LLM scientific analysis into specialized agents makes what the code computes inspectable and lets much smaller models finish the job.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 02:19 UTC pith:FEZIQN6M

load-bearing objection Useful multi-agent workbench that makes LLM HEP analysis code inspectable and runnable at 14B; the reliability-vs-CoLLM claim is thinner than the abstract suggests. the 3 major comments →

arxiv 2607.05762 v1 pith:FEZIQN6M submitted 2026-07-07 cs.SE hep-exhep-ph

Articulating Assumptions in AI-Generated Scientific Analyses through Task Decomposition

Ahmed Hammad , Mihoko Nojiri This is my paper
classification cs.SE hep-exhep-ph
keywords LLM code generationscientific computingmulti-agent systemssemantic differencingreproducibilitycollider physicstask decompositionambiguity detection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Language-model code for scientific analyses can run successfully yet still hide which quantities were computed and which assumptions shaped the numbers. This paper introduces a multi-agent workflow that splits the problem into helper selection, code generation, execution-and-repair, quantity-grounded tracing of how outputs are built, and critique against the original request. A pre-generation module also surfaces ambiguities in the user instruction and offers alternative rewrites. On representative collider-physics analyses the modular design is more transparent and reliable than a single-prompt baseline, and it lets models at the 14B–32B scale complete a workflow that previously required models near 70B parameters.

Core claim

The paper establishes that quantity-grounded semantic differencing—assigning helper selection, generation, execution repair, implementation tracing, and critique to separate agents—reconstructs how key scientific outputs are produced and surfaces mismatches between the intended analysis and the implemented code. Validated on collider analyses, this task decomposition improves transparency and reliability relative to the prior single-prompt approach while enabling substantially smaller models to execute the complete pipeline.

What carries the argument

Quantity-grounded semantic differencing: a multi-agent pipeline whose tracer follows each requested quantity through variable definitions and dependency chains back to object collections and helper calls, then feeds the reconstructed “as-implemented” specification to a critique agent that compares it with the original task card.

Load-bearing premise

The method assumes the tracer fully reconstructs every load-bearing definition and dependency chain so the critique can catch semantic mismatches, even when the code is long and the models are only around 14 billion parameters.

What would settle it

Run the five collider benchmarks repeatedly under both the modular pipeline and a single-prompt baseline at matched model sizes; if expert review finds the modular tracer-plus-critique misses a comparable fraction of serious semantic errors, or if 14B-scale models still fail to produce usable end-to-end implementations, the central claim does not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Selected helpers, tracer chains, and critique reports form an auditable record of the assumptions behind each numerical result.
  • Domain packages can be swapped so the same orchestration applies outside collider physics without rewriting the pipeline.
  • Explicit composite-object construction in the task card reduces implementation variability across repeated generations.
  • Ambiguity detection before code generation moves interpretation choices from the model to the researcher.
  • Models at the 14B scale become usable for full generation-plus-review when the task is decomposed this way.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same tracing-and-critique layer could audit any LLM-generated scientific pipeline where silent redefinitions of objects or cuts matter.
  • Forcing reusable composite objects in the task card may matter as much as model scale for semantic fidelity.
  • If tracer completeness is tuned mainly to the validated domain patterns, new analysis styles will need fresh prompt hardening before critique can be trusted.
  • Groups without large-model API access could draft production analyses locally with 14B–32B models under this architecture.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents a multi-agent framework for LLM-generated scientific analysis code that decomposes the pipeline into helper selection, code generation, execution-based repair, quantity-grounded tracing, post-generation critique, and a pre-generation ambiguity module (Golden Axe Oracle). Domain-specific assets (generation profile, helper registry, critique/oracle contexts) are isolated behind a fixed interface so the orchestration can be reused across domains. The system is instantiated for LHCO collider analyses and evaluated on five structured task cards. The authors report that modular decomposition improves transparency and reliability relative to their prior single-prompt CoLLM approach and that Qwen-family models at the 14B–32B scale can complete a workflow that previously required ~70B-scale models.

Significance. If the claims hold, the work is a useful systems contribution for scientific software engineering: it treats LLM code generation as an auditable multi-stage process rather than a single opaque inference step, and it shows a practical path to running such pipelines with smaller, locally hostable models. Strengths include the explicit provenance design (run directories with intermediate artifacts), line-referenced tracer outputs, a domain-asset interface that separates orchestration from HEP-specific resources, and a public GitHub package. The Oracle’s pre-generation ambiguity protocol and the emphasis on composite-object construction in task cards are concrete, transferable practices. The significance is currently limited by the narrow empirical base (one domain, five cases) and by incomplete measurement of semantic reliability versus the single-prompt baseline.

major comments (3)
  1. [Abstract; §4.2–4.3; Table 3] The central claim that modular decomposition “enhances … reliability relative to the previous single prompt approach” is not supported by a head-to-head, multi-case semantic evaluation. Section 4.2 and the surrounding text mainly establish that helper selection makes the generation–fixer loop produce runnable code at 14B where CoLLM “rarely” did—an executability result, not a fidelity result. Table 3 provides the only detailed semantic scorecard and covers only Case 5; with Qwen-14B helper selection, both 14B and 70B generators fail the composite Δφ(ττ,jj) quantity that the framework is designed to protect. There is no multi-case tally of selection-cut / composite / cutflow pass rates for multi-agent vs single-prompt at matched model sizes. Either add that comparison or narrow the abstract/conclusion claim to transparency plus executability at smaller scale.
  2. [§2.1 (Implementation tracing); §4.5] The load-bearing assumption that the tracer’s as-implemented specification is complete and accurate enough for critique to surface all important semantic mismatches is asserted but not measured. Section 2.1 states that the tracer prompt was “extensively validated” on LHCO examples for O(10)B models, yet §4.5 explicitly declines a statistical study of tracing and offers only qualitative excerpts (Case 3 and Case 5). Without a controlled check—e.g., injected definition mismatches / composite violations and measured recall of tracer+critique across the five cases—the claim that quantity-grounded semantic differencing reliably exposes implementation assumptions remains under-supported. A modest injection or multi-run agreement study would address this.
  3. [§4.2 Table 2; §4.3 Table 3] Helper-selection stability (Table 2) is reported carefully, but the paper itself notes that stability is not correctness and that multiple helper sets may be compatible with the same task card. Table 3 then shows that the choice of helper set (Qwen-14B vs Llama-70B) changes whether composite observables are implemented correctly. The manuscript needs a clearer statement of how a “correct” or “acceptable” helper set is defined for evaluation, and whether downstream semantic pass rates should be conditioned on a fixed gold helper list versus the selector’s own output. Without that, the reliability of the full pipeline under realistic selector noise is hard to interpret.
minor comments (6)
  1. [§4.3 Table 3] Table 3 footnote states code generation used Qwen3-14B because Qwen2.5-14B API access became unavailable, while helper-selection tables use Qwen2.5-14B. Please state model identities consistently in all tables and note any cross-version caveats in one place.
  2. [Figure 1; §2.1] Figure 1 is helpful; a short caption note on which stages are domain-dependent vs domain-independent would help readers planning ports to other fields.
  3. [§4.4 Table 4] Oracle evaluation (Table 4) uses manual inspection “using ChatGPT” to judge valid ambiguities. Briefly describe the adjudication criteria and whether a second human rater was involved, so the useful/off-target counts are reproducible.
  4. [§1 Introduction] Several compound words appear without spaces in the PDF source (e.g., “accompaniedbyanaccountofwhatwascomputed”, “missingtransverse”). Clean typesetting before camera-ready.
  5. [Title page; §5] The package is said to be on GitHub, but the manuscript text only has the placeholder “Package: GitHub”. Provide a stable URL or DOI for the artifact.
  6. [§2.1] Clarify default values of free parameters (retry limit N, regeneration temperature 0.3, oracle max three ambiguities) in one configuration table so experiments can be reproduced without reading the code.

Circularity Check

1 steps flagged

No load-bearing circularity; empirical multi-agent systems paper whose claims rest on new runs against independent task cards, with only a minor non-load-bearing self-citation of prior CoLLM as the baseline being improved.

specific steps
  1. self citation load bearing [§4.2 Stability of helper selection; also Abstract and §1]
    "Explicitly separating helper identification from code generation substantially mitigates this instability, making reliable code generation possible with 14B-parameter models, whereas the previous single-agent approach required models of approximately 70B parameters."

    The claim that the prior single-prompt CoLLM approach required ~70B models is justified by reference to the authors’ own earlier system rather than by a fresh matched single-prompt baseline re-run inside this paper. The citation is not load-bearing for the new modular results themselves (which are independently measured), so the circularity is minor and does not force the central claims.

full rationale

This is a software-framework / empirical-evaluation paper, not a theoretical derivation. There is no equation, prediction, or uniqueness claim that reduces by construction to its own inputs. The central results (helper-selection stability Table 2, semantic code evaluation Table 3, oracle ambiguity counts Table 4, and qualitative tracer/critique examples in §4.5) are obtained by running the new multi-agent pipeline on five human-authored task cards that are independent of the generated code; the tracer reconstructs dependency chains from concrete source lines rather than restating the task card. The sole self-citation of CoLLM [36] is used only to identify the prior single-prompt system and its domain assets that the present work improves upon; the paper re-demonstrates the 14B executability gain under the new helper-selector stage rather than importing an unverified uniqueness or ansatz theorem. No fitted parameters are renamed predictions, no self-definitional loops appear, and no known empirical pattern is merely re-labeled. Minor incompleteness of the head-to-head reliability comparison (noted by the skeptic) is a strength-of-evidence issue, not circularity. Score 1 reflects only the ordinary, non-load-bearing self-citation of the authors’ prior system.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 3 invented entities

The central claim rests on standard software-engineering and LLM-behavior assumptions plus domain packaging choices for LHCO collider analyses. No numerical free parameters are fitted to physics data; the free choices are architectural (retry limits, temperature settings, ambiguity categories) and the domain helper library itself. Invented entities are the named software modules, which are engineering constructs rather than physical postulates.

free parameters (3)
  • regeneration temperature = 0.3
    Default 0.3 for the regenerator agent (all other agents temperature 0); chosen by hand to balance diversity vs. determinism.
  • max repair/retry limit N
    Unspecified numeric bound on the execution–fix–regenerate loop; affects whether a usable program is obtained.
  • oracle max ambiguities reported = 3
    Hard-coded ceiling of three high-risk phrases per run; controls recall/precision trade-off of the ambiguity module.
axioms (4)
  • domain assumption An executable program that passes runtime checks may still implement a different semantic analysis than the natural-language task card; therefore post-generation tracing and critique are required.
    Stated in Introduction and Section 2; underpins the entire multi-agent design.
  • domain assumption A structured helper registry plus selection policy can guide small LLMs toward correct domain utilities without requiring the model to rediscover the entire library from context.
    Section 2.1 and 3.2; key enabler of 14B-scale success.
  • ad hoc to paper The tracer can reconstruct complete upstream definition chains for every task-card item by following variable definitions and helper calls, even for O(10)B models when the system prompt is sufficiently long and validated.
    Section 2.1 and 4.5; the paper itself notes many failure modes that were iteratively vetoed by prompt engineering on LHCO examples.
  • domain assumption LHCO event format and the supplied helper library constitute a representative and sufficient domain package for validating the generic workflow.
    Section 3; all empirical claims rest on this package.
invented entities (3)
  • quantity-grounded semantic differencing / tracer agent independent evidence
    purpose: Reconstruct an as-implemented specification that maps every task-card quantity to concrete code expressions and dependency chains.
    Core novel module; independent evidence is the qualitative and quantitative validation on the five benchmarks.
  • Golden Axe Oracle (and oraclet templated variant) independent evidence
    purpose: Surface at most three high-risk ambiguous phrases in the task card before code generation and offer two alternative rewrites.
    Pre-generation ambiguity module; evidence is the detection counts across model scales in Table 4.
  • helper selector + helper registry interface independent evidence
    purpose: Decouple domain utility discovery from code generation so small models can be guided by explicit metadata.
    Enables the claimed reduction from 70B to 14B models; evidence is the stability table and generation success rates.

pith-pipeline@v1.1.0-grok45 · 21457 in / 3206 out tokens · 43150 ms · 2026-07-11T02:19:37.940667+00:00 · methodology

0 comments
read the original abstract

Scientific results produced by LLM generated analysis code must be understandable and reproducible. However, uncertainty can arise at different stages of the process, both in the original natural language specification and in the generated implementation. As a result, even executable code may not provide a clear understanding of which quantities are being computed or which assumptions determine the final results. To address this challenge, we introduce quantity grounded semantic differencing, a multi-agent framework for analyzing and comparing scientific programs generated by LLMs. The framework assigns code generation, execution, tracing, and validation to separate agents, allowing it to reconstruct how key output quantities are produced and to identify differences between the intended analysis and the implemented code. We also introduce a module that inspects ambiguities in the initial user instruction and suggests alternative rewrites before code generation. Its modular design enables application to different scientific domains by replacing domain specific resources while preserving the same workflow. We validate the framework on representative collider physics analyses. The results demonstrate that the modular task decomposition enhances both transparency and reliability relative to the previous single prompt approach, while enabling substantially smaller models to execute the complete workflow.

Figures

Figures reproduced from arXiv: 2607.05762 by Ahmed Hammad, Mihoko Nojiri.

Figure 1
Figure 1. Figure 1: Schematic architecture of the multi-agent workflow. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 39 canonical work pages · 12 internal anchors

  1. [1]

    B. Chen, D. Zan, F. Zhang, et al.,Large language models meet nl2code: A survey, inProceedings of ACL 2023, 2023

    work page 2023

  2. [2]

    A Survey on Large Language Models for Code Generation

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim,A survey on large language models for code generation,arXiv preprint arXiv:2406.00515(2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  3. [3]

    A. Ni, P. Yin, Y. Zhao, et al.,L2ceval: Evaluating language-to-code generation capabilities of large language models,Transactions of the Association for Computational Linguistics12(2024) 1311–1329

    work page 2024

  4. [4]

    X. Hou, Y. Zhao, Y. Liu, et al.,Large language models for software engineering: A systematic literature review,ACM Transactions on Software Engineering and Methodology33(2024), no. 8

    work page 2024

  5. [5]

    Lajkó et al.,Csllm: Code-specific large language models—a survey,Expert Systems with Applications(2025)

    A. Lajkó et al.,Csllm: Code-specific large language models—a survey,Expert Systems with Applications(2025). 18

    work page 2025

  6. [6]

    D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, Y. Wang, and J.-G. Lou,Large language models meet nl2code: A survey,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics(2023) 7443–7464

    work page 2023

  7. [7]

    M. Chen, J. Tworek, H. Jun, Q. Yuan, and Pinto,Evaluating large language models trained on code,arXiv preprint arXiv:2107.03374(2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  8. [8]

    H. Qu, C. Li, and S. Qian,Particle Transformer for Jet Tagging,arXiv:2202.03772

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Bhimji, C

    W. Bhimji, C. Harris, V. Mikuni, and B. Nachman,OmniLearned: A Foundation model framework for all tasks involving jet physics,Phys. Rev. D113(2026), no. 3 032020, [arXiv:2510.24066]

    work page arXiv 2026

  10. [10]

    IAFormer: Interaction-Aware Transformer network for collider data analysis

    W. Esmail, A. Hammad, and M. Nojiri,IAFormer: Interaction-Aware Transformer network for collider data analysis,SciPost Phys.20(2026), no. 4 108, [arXiv:2505.03258]

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  11. [11]

    Streamlined jet tagging network assisted by jet prong structure

    A. Hammad and M. M. Nojiri,Streamlined jet tagging network assisted by jet prong structure, JHEP06(2024) 176, [arXiv:2404.14677]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Multi-scale cross-attention transformer encoder for event classification

    A. Hammad, S. Moretti, and M. Nojiri,Multi-scale cross-attention transformer encoder for event classification,JHEP03(2024) 144, [arXiv:2401.00452]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Heneka, F

    C. Heneka, F. Nieser, A. Ore, T. Plehn, and D. Schiller,Large Language Models – the Future of Fundamental Physics?,SciPost Phys.20(2026) 070, [arXiv:2506.14757]

    work page arXiv 2026

  14. [14]

    S. D. Bakshi et al.,ArgoLOOM: agentic AI for fundamental physics from quarks to cosmos, arXiv:2510.02426

    work page arXiv

  15. [15]

    Song et al.,Iterated Agent for Symbolic Regression,arXiv:2510.08317

    Z.-Y. Song et al.,Iterated Agent for Symbolic Regression,arXiv:2510.08317

    work page arXiv

  16. [16]

    P. L. S. Connor and A. Sulc,QCD in Language Models: What they really know about QCD?, PoSEPS-HEP2025(2026) 658, [arXiv:2512.02072]

    work page arXiv 2026

  17. [17]

    J. Birk, A. Hallin, G. Kasieczka, N. Madzharova, I. Pang, and D. Shih,Enhancing next token prediction based pre-training for jet foundation models,Mach. Learn. Sci. Tech.7(2026), no. 3 035042, [arXiv:2512.04149]

    work page arXiv 2026

  18. [18]

    Gendreau-Distler, J

    E. Gendreau-Distler, J. Ho, D. Kim, L. T. Le Pottier, H. Wang, and C. Yang,Automating High Energy Physics Data Analysis with LLM-Powered Agents, in39th Annual Conference on Neural Information Processing Systems: Includes Machine Learning and the Physical Sciences (ML4PS), 12, 2025.arXiv:2512.07785

    work page arXiv 2025

  19. [19]

    Menzo, A

    T. Menzo, A. Roman, S. Gleyzer, K. Matchev, G. T. Fleming, S. Höche, S. Mrenna, and P. Shyamsundar,HEPTAPOD: Orchestrating High Energy Physics Workflows Towards Autonomous Agency,arXiv:2512.15867

    work page arXiv

  20. [20]

    Rafique, A

    A. Rafique, A. Singh, and R. Srinivas,Large Language Model Integration for Knowledge Retrieval and Interaction for the DUNE Experiment, in32nd International Symposium on Lepton Photon Interactions at High Energies: Lepton-Photon 2025, 12, 2025. arXiv:2601.05278

    work page arXiv 2025

  21. [21]

    Zoccheddu, S

    S. Zoccheddu, S. R. Qasim, P. Owen, and N. Serra,Large Language Models for Physics Instrument Design,arXiv:2601.07580

    work page arXiv

  22. [22]

    Hill and H

    J. Hill and H. J. Ryoo,GRACE: an Agentic AI for Particle Physics Experiment Design and Simulation, 1, 2026.arXiv:2602.15039

    work page arXiv 2026

  23. [23]

    Knipfer, A

    M. Knipfer, A. Roman, K. T. Matchev, K. Matcheva, and S. Gleyzer,AI Agents for Variational Quantum Circuit Design,arXiv:2602.19387. 19

    work page arXiv

  24. [24]

    Atif et al.,CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants, arXiv:2603.01051

    M. Atif et al.,CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants, arXiv:2603.01051. [25]Electron-Positron AllianceCollaboration, A. Badea, Y. Chen, M. Maggi, and Y.-J. Lee, Agentic AI – Physicist Collaboration in Experimental Particle Physics: A Proof-of-Concept Measurement with LEP Open Data,arXiv:2603.05735

    work page arXiv

  25. [25]

    K. Li, B. Liu, B. Mellado, C.-Z. Yuan, and Z. Zhang,AI agents, language, deep learning, and the next revolution in science,Front. Phys. (Beijing)21(2026), no. 9 096401, [arXiv:2603.07940]

    work page arXiv 2026

  26. [26]

    Mallampalli and S

    A. Mallampalli and S. Dasu,MITRA: An AI Assistant for Knowledge Retrieval in Physics Collaborations, in39th Annual Conference on Neural Information Processing Systems: Includes Machine Learning and the Physical Sciences (ML4PS), 3, 2026.arXiv:2603.09800

    work page arXiv 2026

  27. [27]

    S. Qiu, Z. Cai, J. Wei, Z. Li, Y. Yin, Q.-H. Cao, C. Liu, M.-x. Luo, X.-B. Yuan, and H. X. Zhu, An End-to-end Architecture for Collider Physics and Beyond,arXiv:2603.14553

    work page arXiv

  28. [28]

    E. A. Moreno, S. Bright-Thonney, A. Novak, D. Garcia, and P. Harris,AI Agents Can Already Autonomously Perform Experimental High Energy Physics,arXiv:2603.20179

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    Agrawal, N

    P. Agrawal, N. Craig, A. Madden, and I. V. Lombera,The FERMIACC: Agents for Particle Theory,arXiv:2603.22538

    work page arXiv

  30. [30]

    Menzo, A

    T. Menzo, A. Roman, G. T. Fleming, S. Gleyzer, K. T. Matchev, and S. Mrenna,Agentic Diagrammatica: Towards Autonomous Symbolic Computation in High Energy Physics, arXiv:2603.26990

    work page arXiv

  31. [31]

    T. J. Jat, T. Ghosh, and K. Suresh,Retrieval-Augmented Question Answering over Scientific Literature for the Electron-Ion Collider,arXiv:2604.02259

    work page arXiv

  32. [32]

    Learning to Unscramble Feynman Loop Integrals with SAILIR

    D. Shih,Learning to Unscramble Feynman Loop Integrals with SAILIR,arXiv:2604.05034

    work page internal anchor Pith review Pith/arXiv arXiv

  33. [33]

    Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions

    G. Krzmanc, V. Mikuni, B. Nachman, and C. Wilkinson,Cross-Domain Transfer with Particle Physics Foundation Models: From Jets to Neutrino Interactions,arXiv:2604.12364

    work page internal anchor Pith review Pith/arXiv arXiv

  34. [34]

    Large Language Model-Assisted Framework for BSM Model Building

    S. Saad,Large Language Model-Assisted Framework for BSM Model Building, arXiv:2606.21316

    work page internal anchor Pith review Pith/arXiv arXiv

  35. [35]

    Esmail, A

    W. Esmail, A. Hammad, and M. Nojiri,Collm: Ai engineering toolbox for end-to-end deep learning in collider analyses,arXiv preprint arXiv:2602.06496(2026)

    work page arXiv 2026

  36. [36]

    Thaler,How to read lhc olympics data files, 2006

    J. Thaler,How to read lhc olympics data files, 2006

    work page 2006

  37. [37]

    A. Akli, M. Papadakis, M. Cordy, and Y. Le Traon,When prompt under-specification improves code correctness: An exploratory study of prompt wording and structure effects on llm-based code generation,arXiv preprint arXiv:2604.24712(2026) [arXiv:2604.24712]

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  38. [38]

    The Impact of Prompt Programming on Function-Level Code Generation

    R. Khojah, F. Gomes de Oliveira Neto, M. Mohamad, and P. Leitner,The impact of prompt programming on function-level code generation,arXiv preprint arXiv:2412.20545(2024) [arXiv:2412.20545]

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  39. [39]

    Paleyes, R

    A. Paleyes, R. Sendyka, D. Robinson, C. Cabrera, and N. D. Lawrence,Prompt variability effects on llm code generation,arXiv preprint arXiv:2506.10204(2025) [arXiv:2506.10204]. 20

    work page arXiv 2025