Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair

Benjamin Agyekum; Fabio Santos

REVIEW 3 major objections 3 minor 17 references

Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair

T0 review · 3 major / 3 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Iterative LLM repair of infrastructure-as-code can break a previously passing security check, and the paper's conservative estimate is about 3.3% of repair scenarios.

desk verdict The first per-iteration look at security regressions in LLM-driven IaC repair supports the qualitative claim, but the strict 3.3% figure is an order-of-magnitude estimate because the authors admit timelines may mix separate executions. read the letter →

arxiv 2608.13404 v1 pith:Q2NTO4GA submitted 2026-08-13 cs.SE cs.CR

Benjamin Agyekum , Fabio Santos This is my paper · ORCID

classification cs.SEcs.CR

keywords InfrastructureasCodesecurityregressionLLMrepairiterativefeedbackloopCIScomplianceCheckovTerraformcumulative-bestmetrics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether the standard practice of feeding validator errors back to a large language model until generated infrastructure code passes can, in the act of fixing, break security properties that were already satisfied. The authors track individual CIS security checks across up to five repair rounds in roughly six thousand scenario timelines, counting every transition in which a check that passed at one iteration fails at the next. They find that repair does introduce such regressions, but that the defensible rate is modest: about 3.3% of scenarios under a strict counting that excludes multi-resource ambiguity, versus 13.8% under an inclusive counting. The result matters because prior work reports only cumulative-best compliance, which is non-decreasing by construction and therefore cannot see this security cost.

What carries the argument

The load-bearing object is the transition-level regression detector built from Checkov check IDs mapped to CIS controls, together with the standard/strict detection distinction. Each consecutive pair of iterations $(i, i+1)$ with Checkov results forms a transition; a regression is a check that passes at $i$ and fails at $i+1$. The strict variant filters out multi-resource ambiguity by requiring the check to be exclusively passed at $i$ and exclusively failed at $i+1$, and this distinction is what turns a dramatic 13.8% standard-mode result into a conservative 3.3% strict-mode claim. Supporting machinery includes a root-cause taxonomy (resource restructuring, configuration drift, argument removal, unclassified) and the check volatility measure $V = |C_{new}| + |C_{removed}| + |C_{flipped}|$, which is the strongest regression signal in the data.

What would settle it

Re-run a sample of scenarios while recording every attempt with full model attribution and an unbroken per-scenario order, then recompute the strict-mode transition rate directly from those true per-attempt sequences; if the rate departs from the reported 3.3% beyond the confidence interval, the headline estimate is an artifact of timeline reconstruction rather than a property of iterative repair.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that iterative LLM-driven repair of Terraform configurations produces genuine security regressions, but most apparent regressions are measurement artifacts. A security regression is defined at the transition level: a check ID that is in the passed set at iteration $i$ and in the failed set at iteration $i+1$. Because one Checkov check can apply to several resources, a restructured configuration can make one resource fail while another still passes; the paper's strict mode requires the check to be exclusively passed at $i$ and exclusively failed at $i+1$, and this drops the scenario-level rate from 13.8% to 3.3% (and the transition-level rate from 24.8% to 5.2%). Root-cause classification attributes 79.0% of standard-mode regressions to resource restructuring, and regressing transitions carry 2.6x more code churn (Cohen's $d = 0.90$) and 4.9x higher strict-mode check volatility ($d = 1.49$). The authors conclude that cumulative-best reporting hides a real security cost and that iteration 3 is the optimal stopping point.

Load-bearing premise

The paper assumes that each timeline is a coherent sequence of repair attempts for one scenario, so a pass-to-fail move between consecutive log entries reflects one genuine repair step; the authors concede that some timelines may mix iterations from separate executions.

Editorial extensions

If this is right

Cumulative-best reporting should be supplemented with per-iteration trajectory metrics; any pipeline that reports only the best compliance over iterations cannot detect the repair-induced regressions this paper measures.
A bounded iteration budget near three rounds captures most compliance gains (83.1% pass rate at iteration 3 versus 83.4% at iteration 4) while limiting accumulated regression risk.
Feedback loops should flag regressions explicitly and encourage minimal edits, since resource restructuring accounts for 79.0% of standard-mode regressions and large rewrites predict them.
Standard and strict detection support different conclusions: RAG looks worse than non-RAG in standard mode but records zero strict-mode regressions, so the choice of detection mode determines model and strategy comparisons.
Check volatility is cheap to compute online and is the strongest predictor of regression ($d = 1.49$ in strict mode), making it a usable halting or flagging signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: run the same repair loop on CloudFormation or Ansible with per-resource validators; if multi-resource ambiguity is the mechanism, the standard/strict gap should reappear with a similar order-of-magnitude strict rate.
Because non-RAG timelines are not model-attributed, the observed Chain-of-Thought advantage may be a model-composition effect; a model-controlled run with recorded model identity per attempt would settle whether prompting strategy itself reduces regressions.
The clean-timelines-regress-more result could be probed by ablating feedback content (syntax-only versus security-only) to separate the opportunity effect from the feedback-purity effect.
If the 36.6% self-correction rate replicates, single-regression halt rules are premature; an adaptive policy that waits for a check to stay failed across two consecutive iterations may beat a fixed budget.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a genuine first look at per-iteration security trajectories in LLM-driven IaC repair, and the central qualitative finding—repair loops sometimes break previously-passing checks, and cumulative-best reporting hides it—holds up. The precise numbers are softer than they look.

What is new: nobody has tracked individual Checkov checks across repair iterations in Terraform before. The standard/strict distinction is a real methodological contribution; it cleanly separates true exclusive failures from multi-resource bookkeeping, and the strict-mode robustness check gives you confidence that most standard-mode "regressions" are artifacts. The root-cause taxonomy (restructuring dominating) is plausible, and the code churn / volatility results are consistent with the degradation literature in general code.

Where I wince: the timeline-coherence problem is real and the authors admit it. For non-RAG runs, deduplication keeps the most recent execution, and Section 7 concedes consecutive records may come from different attempts. That means every transition-level rate—including the 3.3% strict figure—could be counting artifacts. The confidence intervals are computed over timelines as if they were independent coherent units, so they do not cover this error. The paper discloses this, which I respect, but "disclosed limitation" and "quantified limitation" are different things. There is no estimate of how many timelines are mixed.

Also: single runs per configuration leave variance unquantified; the non-RAG strategy comparisons are confounded by unknown model mix; and the root-cause classifier was not validated against human labels. These are all in the threats section, so the authors are not hiding them, but they do cap the precision of the numbers.

None of that kills the paper. The strict rate being 3.3% or 2% or 5% does not change the practical message: stop iterating around iteration 3, monitor check volatility, and treat cumulative-best metrics with suspicion. The qualitative claims rest on multiple independent signals (churn effect, volatility effect, self-correction rate) that would be hard to manufacture through timeline mixing alone.

Who this is for: anyone building LLM feedback loops for IaC, and empirical SE researchers studying repair dynamics. It deserves a serious referee. If I were the editor, I would send it out and ask the authors to quantify the mixed-timeline problem—at minimum, report the fraction of non-RAG timelines that were deduplicated and run the headline strict rate on the subset of RAG timelines where model attribution is known.

Verdict: engage with it. It's a good paper with an honest, fixable soft spot.

Referee Report

3 major / 3 minor

Summary. The paper studies whether iterative LLM-driven repair of Terraform Infrastructure-as-Code (IaC) can turn a previously passing CIS/Checkov security check into a failing one. Using 5,968 scenario timelines from the IaC-Eval benchmark and 4,440 iteration transitions with Checkov results on both sides, it reports that 13.8% of scenarios (24.8% of transitions) exhibit at least one regression under standard detection, while strict detection — which counts only exclusive check failures — reduces the rate to 3.3% of scenarios (5.2% of transitions). The paper attributes most standard-mode events to multi-resource ambiguity, identifies resource restructuring as the dominant root cause (79.0%), and reports that regression transitions have higher code churn and check volatility, that 36.6% of standard-mode regressions self-correct, and that iteration 3 is the optimal stopping point. The central claim is that cumulative-best metrics hide a real but modest per-iteration security cost of repair.

Significance. If the measurement is sound, this is the first IaC-domain evidence that cumulative-best reporting masks per-iteration security regressions in LLM repair loops, and the standard/strict distinction is a useful methodological contribution for resource-level static analysis. The strict-mode 3.3% scenario rate is a genuinely conservative baseline, and the authors are explicit about their thresholds and about what is descriptive versus causal. The paper also ships a replication package, which supports reproducibility of the derived tables and figures. The main qualitative finding — that repair iterations can break a previously passing check and that cumulative-best metrics hide this — is plausible and important. However, the headline numbers rest on timeline coherence and on a regression definition that the paper itself does not fully validate, so the precise point estimates cannot be certified as stated.

major comments (3)

[§4.1, §4.2, §7 (Internal)] The transition-level definition of regression requires consecutive iterations from one coherent repair run, but the deduplication rule keeps only the most recent execution when a non-RAG scenario ran more than once, and the same-iteration exclusion in §4.2 does not prevent adjacent records from different executions. The threats section concedes that 'some timelines may mix iterations from separate executions.' Since every prevalence, root-cause, churn, volatility, self-correction, and oscillation statistic is computed on these transitions, a mixed timeline can manufacture or hide a pass-to-fail transition that is not a repair step. The strict 3.3% scenario rate is itself derived from such transitions. The authors should quantify how many timelines contain records from more than one execution and re-run all transition-level analyses on the subset of provably single-execution timelines; until that is done, the headline point estimate is not supported.
[§4.2, Definition 1] Standard mode counts a regression whenever a check is in the passed set at iteration i and in the failed set at iteration i+1, without requiring that the check was not already failing at iteration i. For a multi-resource check that is partly failing at iteration i, the same per-resource status can satisfy both conditions, so an unchanged partial failure is counted as a regression. This inflates the 13.8% scenario rate and the 24.8% transition rate and weakens the claim that standard mode is an inclusive upper bound on genuine regressions; it is an upper bound plus a constant-partial-failure artifact. The authors should either require that the check was not in the failed set at iteration i, or track failures per resource instance, and report how many standard-mode events are constant partial failures.
[§5.3, §7, Discussion] All non-RAG strategy comparisons and the RAG-versus-non-RAG comparison are confounded by unknown model identity, because the non-RAG logs do not record the generating model and the model mix differs by strategy (Chain-of-Thought is 79% Gemini, Few-Shot 73%, Zero-Shot 46%). The paper acknowledges this, but it still presents Chain-of-Thought as the lowest-regression non-RAG strategy and uses that ordering in the Discussion's recommendation to prefer it where retrieval is unavailable. Since the model effect is the strongest measured factor in the RAG data (OR = 17.29), the observed ordering is exactly what differing model mixes would produce. The strategy-level findings should be reported only as descriptive of the deployed configurations, and the practical recommendation to prefer Chain-of-Thought should be removed or explicitly labeled as confounded.

minor comments (3)

[§4.5, §7] Transition-level confidence intervals and Mann-Whitney tests treat the 4,440 transitions as independent even though transitions are nested within scenario timelines; within-timeline autocorrelation is acknowledged in the threats section but not quantified or corrected. Adding a cluster-robust analysis or explicitly reporting this as a limitation of the transition-level tests would strengthen the paper.
[Table 3, §5.3] Table 3 reports only standard-mode scenario regression rates, while the text makes important strict-mode comparisons (including the claim of zero RAG strict regressions). Adding strict-mode counts and rates for each configuration row would let readers verify the mode-dependent reversals.
[§6.1, Figure 3] The normalized per-transition rates (22.8%, 26.6%, 29.5%, and 20.8%) are stated without a table of the per-transition denominators; a supplemental table or appendix would allow readers to recompute these rates from the reported event counts.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: this is an empirical measurement study against the external IaC-Eval benchmark, with no fitted parameter masquerading as a prediction and no load-bearing self-citation.

full rationale

The paper's central claims are observational measurements of security regressions in LLM-driven IaC repair, not derivations from first principles. Regression is defined operationally in Definition 1 as a check passing at iteration i and failing at iteration i+1; this is a measurement definition, not a conclusion built into its own premise. The standard/strict detection modes are analytical choices with explicitly stated trade-offs, and the paper reports both as complementary perspectives rather than presenting one as forced. The root-cause taxonomy uses thresholds (e.g., resource-count change >=2, text similarity >0.85) that are described as conservative heuristics, not as fitted parameters later renamed as predictions. No statistical model is fitted and then evaluated on data that generated it; all rates, confidence intervals, and effect sizes are computed directly from the tracked Checkov check results. The only self-citation is the authors' own replication package [1], which is cited for data availability and does not carry any load-bearing argument. The paper does invoke prior work by others (e.g., Shukla et al., Chen et al.) for motivation and comparison, but these citations are external evidence, not a self-citation chain. The most serious validity concern visible in the paper is the timeline-coherence caveat in Section 7: 'Some timelines may mix iterations from separate executions.' This is a legitimate threat to the accuracy of transition-level measurements, but it is a data-quality and construct-validity issue, not circularity: the measurement would be wrong if the premise fails, yet the output is not equivalent to the input by construction. No equation in the paper reduces to its inputs, and no cited uniqueness or prior result is being used to forbid alternatives. The paper is self-contained against an external benchmark and openly reports its limitations, so the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on measurement choices rather than derived parameters. The hand-chosen classification thresholds and stopping threshold shape the headline taxonomy and the iteration-3 guidance. The domain assumptions about Checkov and the 30-check subset bound the external validity. The most fragile premise is timeline coherence, which the paper itself questions.

free parameters (3)

Resource restructuring threshold = >=2 resource block change
Hand-chosen cutoff defines the dominant root-cause category in Section 4.4; not empirically tuned.
Configuration drift similarity threshold = >0.85 text similarity
Hand-chosen heuristic in Section 4.4; authors call it a heuristic.
Optimal stopping gain threshold = <0.5pp pass-rate gain
Subjective threshold in Section 6 that produces the iteration-3 recommendation.

assumptions (3)

domain assumption Checkov results are a valid proxy for security
Section 7 Construct threats: the study uses static validation instead of deployment-time evaluation.
domain assumption The 30 tracked CIS check IDs capture security-relevant behavior
Section 4 tracks only 30 check IDs; the paper does not justify that this set is complete or representative for all scenarios.
ad hoc to paper Consecutive log records in a timeline come from the same repair execution
Section 7 Internal: 'some timelines may mix iterations from separate executions'; every transition-level regression rate depends on this.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair." pith.science (2026). https://pith.science/paper/Q2NTO4GA

@misc{pith2026260813404,
  author       = {Pith},
  title        = {Pith review of: Does Fixing Break Security? An Empirical Study of Security Degradation in Iterative LLM-Driven Infrastructure-as-Code Repair},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q2NTO4GA}},
  note         = {Machine review of arXiv:2608.13404}
}

read the original abstract

Background: Iterative feedback loops are the dominant paradigm for improving LLM-generated Infrastructure-as-Code (IaC): validators such as Checkov and terraform validate feed error signals back for successive repair attempts. Prior work reports cumulative-best metrics, which are non-decreasing by construction, so the raw per-iteration security trajectory has never been examined for IaC. Aims: We study security regression (a previously-passing CIS Benchmark check that fails after a repair iteration) to determine whether and how often iterative LLM repair degrades security while fixing other issues. Method: We analyze 5,968 scenario timelines from the IaC-Eval benchmark, each one scenario run through one configuration for up to 5 repair iterations. The 15 configurations (six model-specific RAG, nine model-aggregated non-RAG, three temperatures each) yield 4,440 iteration transitions with Checkov data on both sides. We track 30 individual CIS check IDs and classify root causes from code diffs, under two detection modes: standard (inclusive) and strict (exclusive check failures only). Results: Under standard detection, 13.8% of scenarios (24.8% of transitions) exhibit at least one regression. Under strict detection the rate falls to 3.3% of scenarios (5.2% of transitions), indicating most apparent regressions are multi-resource measurement artifacts. Resource restructuring (79.0%) is the dominant root cause. Regression transitions show 2.6x more code churn (Cohen's d=0.90) and 4.9x higher strict-mode check volatility (d=1.49). Of standard-mode regressions, 36.6% self-correct within an average of 1.2 iterations; iteration 3 is the optimal stopping point. Conclusions: Iterative IaC repair does introduce security regressions, but the conservative, defensible rate is about 3.3% of scenarios. Our findings motivate security-aware feedback-loop design and actionable iteration-budget guidance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 6 canonical work pages

[1]

Replication package: Does fixing break security? an empirical study of security degradation in iterative llm-driven infrastructure-as-code repair

1 Benjamin Agyekum and Fabio Santos. Replication package: Does fixing break security? an empirical study of security degradation in iterative llm-driven infrastructure-as-code repair. Zenodo, 2026.https://doi.org/10.5281/zenodo.20265184. 2 Enna Basic and Alberto Giaretta. From vulnerabilities to remediation: A systematic literature review of LLMs in code ...

arXiv 2026
[5]

Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021.doi:10.48550/ arXiv.2107.03374

6 Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021.doi:10.48550/ arXiv.2107.03374. 7 Yi Chen, Yun Bian, Haiquan Wang, Shihao Li, and Zhe Cui. SCAFFOLD-C...

arXiv doi:10.48550/arxiv.2107.03374 2021
[6]

11 Michael Felderer and Elizabeta Fourneret

doi:10.1145/3643788.3648014. 11 Michael Felderer and Elizabeta Fourneret. A systematic classification of security regression testing approaches.International Journal on Software Tools for Technology Transfer, 17(3):305– 319, 2015.doi:10.1007/s10009-015-0365-2. 12 EhsanFirouzi, ShardulBhatt, andMohammadGhafari. CandevelopersrelyonLLMsforsecure IaC developm...

arXiv
[8]

14HashiCorp

doi:10.1145/3716848. 14HashiCorp. Terraform by hashicorp. Available:https://www.terraform.io/,

doi:10.1145/3716848
[9]

Agyekum and F

B. Agyekum and F. Santos 49:19 15 Prithwish Jana, Sam Davidson, Bhavana Bhasker, Andrey Kan, Anoop Deoras, and Laurent Callot. TerraFormer: Automated infrastructure-as-code with LLMs fine-tuned via policy-guided verifier feedback.arXiv preprint arXiv:2601.08734, 2026.doi:10.48550/arXiv.2601.08734. 16 Patrick Tser Jern Kon, Jiachen Liu, Yiming Qiu, Weijun ...

work page doi:10.48550/arxiv.2601.08734 2026
[10]

17 Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu

doi:10.52202/079017-4273. 17 Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu. SafeGenBench: A benchmark framework for security vulnerability detection in LLM-generated code.arXiv preprint arXiv:2506.05692, 2025.doi:10.48550/arXiv.2506.05692. 18 Yikun Li, Matteo Grella, Daniel Nahmias, Gal Engelberg, Dan Klein, Gianc...

arXiv doi:10.48550/arxiv.2506.05692
[11]

doi:10.48550/arXiv.2511.12385. 19 AmanMadaan, NiketTandon, PrakharGupta, SkylerHallinan, LuyuGao, SarahWiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback. InAdvances in Neur...

work page doi:10.48550/arxiv.2511.12385
[12]

Using a Feedback Loop for LLM-based Infrastructure as Code Generation

arXiv preprint arXiv:2411.19043.doi:10.48550/arXiv.2411.19043. 21 Ravin Ravi, Dylan Bradshaw, Stefano Ruberto, Gunel Jahangirova, and Valerio Terragni. LLMLOOP: Improving LLM-generated code and tests through automated iterative feedback loops. InProceedings of the 41st IEEE International Conference on Software Maintenance and Evolution (ICSME), Tool Demon...

work page Pith review arXiv doi:10.48550/arxiv.2411.19043

Show all 17 references

[13]

doi:10.1109/ICSME64153.2025.00109

arXiv preprint arXiv:2603.23613. doi:10.1109/ICSME64153.2025.00109. 22 Matthew Renze. The effect of sampling temperature on problem solving in large lan- guage models. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 7346–7356. Association for Com...

2025
[14]

26 Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj, and Suman Saha

arXiv preprint arXiv:2506.11022.doi:10.1109/ISTAS65609.2025.11269659. 26 Vidyut Sriram, Sawan Pandita, Achintya Lakshmanan, Aneesh Shamraj, and Suman Saha. Improving LLM-assisted secure code generation through Retrieval-Augmented-Generation and multi-tool feedback.arXiv prepri...

2025
[15]

doi:10.48550/arXiv.2601. 00509. 27 Hao Tang, Keya Hu, Jin Peng Zhou, Sicheng Zhong, Wei-Long Zheng, Xujie Si, and Kevin Ellis. Code repair with LLMs gives an exploration-exploitation tradeoff. InAdvances in Neural Information Processing Systems (NeurIPS), volume 37,

doi
[16]

doi:10.52202/079017-3746

arXiv preprint arXiv:2405.17503. doi:10.52202/079017-3746. 28 Yiming Xiang, Zhenning Yang, Jingjia Peng, Hermann Bauer, Patrick Tser Jern Kon, Yiming Qiu, and Ang Chen. Automated bug discovery in cloud infrastructure-as-code updates with llm ESEM 2026 49:20 Does Fixing Break S...

arXiv
[17]

2506.05623

doi:10.48550/arXiv. 2506.05623

doi
[2022]

5 Checkov

doi: 10.1145/3544902.3546250. 5 Checkov. Checkov: Infrastructure as code static analysis. Available:https://www.checkov.io. Accessed: Mar. 25,
[2024]

3 Islem Bouzenia, Premkumar Devanbu, and Michael Pradel

doi:10.48550/ arXiv.2412.15004. 3 Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. RepairAgent: An autonomous, LLM-based agent for program repair. InProceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE), pages 2188–2200. IEEE/ACM,

doi
[2025]

4 Larissa Braz, Enrico Fregnan, Vivek Arora, and Alberto Bacchelli

doi: 10.1109/ICSE55347.2025.00157. 4 Larissa Braz, Enrico Fregnan, Vivek Arora, and Alberto Bacchelli. An exploratory study on regression vulnerabilities. InProceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 12–22,

2025
[2026]

13 Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, Jiaxin Yu, and Jinfu Chen

doi:10.48550/arXiv.2602.03648. 13 Yujia Fu, Peng Liang, Amjed Tahir, Zengyang Li, Mojtaba Shahin, Jiaxin Yu, and Jinfu Chen. Security weaknesses of Copilot-generated code in GitHub projects: An empirical study.ACM Transactions on Software Engineering and Methodology, 34(8):218...

doi

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Replication package: Does fixing break security? an empirical study of security degradation in iterative llm-driven infrastructure-as-code repair

1 Benjamin Agyekum and Fabio Santos. Replication package: Does fixing break security? an empirical study of security degradation in iterative llm-driven infrastructure-as-code repair. Zenodo, 2026.https://doi.org/10.5281/zenodo.20265184. 2 Enna Basic and Alberto Giaretta. From vulnerabilities to remediation: A systematic literature review of LLMs in code ...

arXiv 2026

[2] [5]

Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021.doi:10.48550/ arXiv.2107.03374

6 Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021.doi:10.48550/ arXiv.2107.03374. 7 Yi Chen, Yun Bian, Haiquan Wang, Shihao Li, and Zhe Cui. SCAFFOLD-C...

arXiv doi:10.48550/arxiv.2107.03374 2021

[3] [6]

11 Michael Felderer and Elizabeta Fourneret

doi:10.1145/3643788.3648014. 11 Michael Felderer and Elizabeta Fourneret. A systematic classification of security regression testing approaches.International Journal on Software Tools for Technology Transfer, 17(3):305– 319, 2015.doi:10.1007/s10009-015-0365-2. 12 EhsanFirouzi, ShardulBhatt, andMohammadGhafari. CandevelopersrelyonLLMsforsecure IaC developm...

arXiv

[4] [8]

14HashiCorp

doi:10.1145/3716848. 14HashiCorp. Terraform by hashicorp. Available:https://www.terraform.io/,

doi:10.1145/3716848

[5] [9]

Agyekum and F

B. Agyekum and F. Santos 49:19 15 Prithwish Jana, Sam Davidson, Bhavana Bhasker, Andrey Kan, Anoop Deoras, and Laurent Callot. TerraFormer: Automated infrastructure-as-code with LLMs fine-tuned via policy-guided verifier feedback.arXiv preprint arXiv:2601.08734, 2026.doi:10.48550/arXiv.2601.08734. 16 Patrick Tser Jern Kon, Jiachen Liu, Yiming Qiu, Weijun ...

work page doi:10.48550/arxiv.2601.08734 2026

[6] [10]

17 Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu

doi:10.52202/079017-4273. 17 Xinghang Li, Jingzhe Ding, Chao Peng, Bing Zhao, Xiang Gao, Hongwan Gao, and Xinchen Gu. SafeGenBench: A benchmark framework for security vulnerability detection in LLM-generated code.arXiv preprint arXiv:2506.05692, 2025.doi:10.48550/arXiv.2506.05692. 18 Yikun Li, Matteo Grella, Daniel Nahmias, Gal Engelberg, Dan Klein, Gianc...

arXiv doi:10.48550/arxiv.2506.05692

[7] [11]

doi:10.48550/arXiv.2511.12385. 19 AmanMadaan, NiketTandon, PrakharGupta, SkylerHallinan, LuyuGao, SarahWiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback. InAdvances in Neur...

work page doi:10.48550/arxiv.2511.12385

[8] [12]

Using a Feedback Loop for LLM-based Infrastructure as Code Generation

arXiv preprint arXiv:2411.19043.doi:10.48550/arXiv.2411.19043. 21 Ravin Ravi, Dylan Bradshaw, Stefano Ruberto, Gunel Jahangirova, and Valerio Terragni. LLMLOOP: Improving LLM-generated code and tests through automated iterative feedback loops. InProceedings of the 41st IEEE International Conference on Software Maintenance and Evolution (ICSME), Tool Demon...

work page Pith review arXiv doi:10.48550/arxiv.2411.19043