Pith. sign in

REVIEW 4 major objections 6 minor 35 references

Large Language Models for IT Automation Tasks: Are We There Yet?

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current open-source LLMs cannot reliably write working Ansible playbooks: on a 126-task execution benchmark, none exceed 12% pass@10, with failures concentrated in state tracking and module knowledge.

desk verdict The low pass@k result is real and ITAB is a step forward, but the headline error-taxonomy percentages come from a convenience sample and overstate what the data supports. read the letter →

arxiv 2505.20505 v1 pith:BCCG33WC submitted 2025-05-26 cs.CL cs.SE

Md Mahadi Hassan , John Salvador , Akond Rahman , Santu Karmaker This is my paper · ORCID
classification cs.CLcs.SE
keywords LLMcodegenerationITautomationAnsiblebenchmarkstatereconciliationpass@kevaluationerrortaxonomyinfrastructureas
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to answer whether open-source large language models can turn real-world, user-written IT automation requests into Ansible playbooks that actually work, rather than merely into scripts that parse as YAML. To test this it builds ITAB, a benchmark of 126 tasks drawn from 52,727 Stack Overflow posts, each with a defined initial system state, a target state, and automated assertions verified by executing the generated playbook in a fresh Docker environment. Evaluating 14 open-source LLMs between 3B and 14B parameters, it finds that none achieve a $\mathrm{pass@}10$ above 12%, and nearly all stay below 4% at $\mathrm{pass@}1$. An analysis of 1,411 execution failures attributes 44.87% of errors to state-reconciliation reasoning (variable, host, path, and template handling) and 24.37% to weak module-specific execution knowledge, supporting the paper's conclusion that reliable LLM-driven IT automation will require advances in state reasoning and domain-specific execution understanding, not just better prompts.

What carries the argument

The load-bearing object is ITAB itself: an execution-driven benchmark whose unit of evaluation is a state-reconciliation task. Each of its 126 tasks is built from a Stack Overflow post by formalizing an initial system state and a target state, identifying the parameters (file paths, services, hosts) that can change the outcome, and writing assertion scripts that check the target state after the playbook runs inside a fresh, isolated four-node Docker network spanning Ubuntu, Alpine, CentOS, and Red Hat. A playbook passes only if it validates as YAML, executes cleanly, satisfies the assertions, and is idempotent, with $\mathrm{pass@}k$ computed by the unbiased estimator over 15 generated samples per model-prompt-temperature configuration. The benchmark's specific contribution is that it evaluates state reconciliation, the property that an automation tool infers the desired state, compares it with the current state, and applies only the necessary changes, something static correctness checks cannot see. The supporting machinery is a nine-category error taxonomy built from 1,411 execution failures, which sorts the failures into two semantic families: state-reconciliation reasoning (variable, host, path, and template issues, 44.87% combined) and module-specific execution knowledge (attribute/parameter and module errors, 24.37%).

What would settle it

Take a random sample of roughly 30 of the 126 tasks and have experienced Ansible practitioners independently reconstruct the initial state, target state, and validation assertions without seeing the paper's versions; substantial disagreement would show the low scores partly reflect the benchmark's chosen strictness, while convergence would confirm the 12% ceiling as a real measure of current open-source capability. A complementary check is to retry the failed generations with execution feedback (show the model the error and let it revise): if any of the nine error categories disappears entirely, that category is a prompt-or-sampling artifact rather than a fixed model limitation.

Watch

Extended reading notes

Core claim

The paper's central claim is that functional correctness for IT automation is a distinct and harder problem than code generation as normally benchmarked, and that current open-source LLMs are not yet reliable at it. On ITAB, a generated playbook counts as correct only if it is syntactically valid, executes without error inside a fresh containerized network, leaves the system in the asserted target state, and is idempotent; by that standard, the strongest evaluated model solves at most 12% of tasks within ten samples. The failure analysis shows the models generally know which automation action is needed but mishandle the details: they mis-track state across hosts, variables, paths, and Jinja2 templates, and they misconfigure module attributes even when they pick the right module. The authors conclude that the two dominant bottlenecks, state-reconciliation reasoning and module-specific execution knowledge, are precisely the abilities that prompt engineering cannot supply, which is why the paper answers its title's question in the negative.

Load-bearing premise

The load-bearing premise is that the initial states, target states, and validation assertions the authors derived for each of the 126 tasks faithfully capture what the original Stack Overflow posters were asking for, and that requiring a correct playbook to be idempotent matches what practitioners actually need; if those mappings are wrong or too strict, the low $\mathrm{pass@}k$ scores and the error taxonomy measure the benchmark's own strictness rather than real-world model capability.

Editorial extensions

If this is right

  • Syntactic validity is a weak proxy for operational correctness: prompts enriched with the paper's own error taxonomy lift pass@k by only 1-4 percentage points, so the observed failures are not mainly instruction-following problems.
  • State-sensitive domains, Templating (Jinja2) and Variable Management, are the hardest for every evaluated model, while Server Configuration and File Management are comparatively tractable, so benchmarks and training should prioritize state-heavy tasks.
  • Sampling temperature is a genuine control knob with a trade-off: low temperatures (0.2) maximize first-attempt reliability, high temperatures (0.6-0.8) improve ten-sample diversity but also widen the variety of errors.
  • Reasoning-distilled models fail overwhelmingly on basic Ansible syntax, indicating that general reasoning strength does not transfer to structured infrastructure-as-code generation without domain grounding.
  • Progress toward dependable language-guided automation should be measured with execution-based, state-reconciliation-aware benchmarks such as ITAB, because static checks cannot detect the dominant failure modes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The strict correctness definition (execution plus asserted target state plus idempotency) may undercount a model that reaches the desired state on a single run but not on a second run; whether practitioners always demand idempotency is a question the paper leaves open, and relaxing that requirement is a testable variant of the benchmark.
  • The evaluation's 3B-14B open-source range leaves open how larger closed models would fare; if a much larger model clears the 12% ceiling decisively, the bottleneck is scale or training-data breadth rather than the task class itself, an easy comparison to run with the same harness.
  • The nine-category error taxonomy points to interventions the paper does not test, such as fine-tuning on Ansible module documentation, supervision signals for state tracking, or execution-feedback loops that let a model see its failure and revise.
  • One appendix passage (Section C.3.1) leaves the sample count as an unfilled template token ('Specify your value of n'), although the body and other appendix sections consistently report 15 samples; anyone reproducing the pass@k numbers should confirm the intended value.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces ITAB, a benchmark of 126 Ansible automation tasks derived from Stack Overflow posts across seven IaC domains, with containerized execution and task-specific validation assertions. The authors evaluate 14 open-source LLMs (3B–14B) under three TELeR prompt levels and four sampling temperatures using pass@k, finding very low success rates (best pass@10 = 12.0%). They then present an error taxonomy from 1,411 execution failures, reporting 44.87% state-reconciliation reasoning errors and 24.37% module-specific execution knowledge errors, and show that error-aware prompting yields only marginal gains. The paper concludes that current open LLMs lack reliable state tracking and domain-specific execution knowledge for IT automation.

Significance. If the benchmark and the error attribution are valid, the paper makes a useful contribution: execution-based evaluation of IaC generation is underrepresented, and the multi-distribution Docker environment, pass@k estimator, and idempotency-aware correctness criterion are commendable strengths. The low pass@k results are machine-checked through dynamic execution and convincingly show that 3B–14B open models rarely generate functionally correct Ansible playbooks. The explanatory error-taxonomy percentages, however, are not currently supported as global estimates of the 1,411 failures, so the headline bottleneck claim needs reanalysis or careful reframing before the paper's central conclusion is established.

major comments (4)
  1. [Appendix G.1, Section 5, Table 4, Abstract] The 44.87% and 24.37% percentages are computed from a convenience sample, not from all 1,411 failures: Appendix G.1 states that for each IaC category 'we selected a representative task and, for each model, sampled 15 failed playbooks stratified by temperature and TELeR prompt level.' This balanced design gives every model-task pair equal weight regardless of how many failures it actually produced, so rare failure types are over-represented and dominant failure types in high-failure tasks are under-represented. The abstract and Section 5 present these numbers as the distribution of the 1,411 failures; they are only the distribution within this fixed 15-per-model-task sample. Please reweight by actual failure counts, sample proportionally to failures, or explicitly reframe the percentages as 'within the stratified error sample' and adjust the abstract accordingly.
  2. [Table 4, Section 5, Abstract] The aggregation in Table 4 is internally inconsistent: the table shows rows for DeepSeek-Distill-L and DeepSeek-Distill-Q, but the caption says these syntax-heavy distributions are excluded from the aggregated percentages. The abstract then reports the aggregate values globally without this caveat. Please either remove the reasoning-distilled rows from the table, show a separate aggregate that includes them, or clearly state in the abstract that the headline percentages apply only to the 12 non-reasoning-distilled models.
  3. [Abstract, Section 4.2, Table 3] The abstract's claim that 'none of which accomplish pass@10 at a rate beyond 12%' is contradicted by Table 3, where Qwen2.5-Coder-7B-it reaches pass@10 of 15.5% with error-aware prompts. Since the paper itself evaluates this prompting condition, the abstract must qualify the claim as applying to the standard TELeR prompt condition, or report the error-aware results as part of the headline evaluation.
  4. [Section 3.2, Section 9, Appendix B] The benchmark's construct validity rests on the authors' translation of ambiguous Stack Overflow posts into precise initial states, target states, and validation assertions, but no inter-annotator reliability or independent validation is reported, and no artifact or availability statement is provided for the 733 test cases and validation scripts. The correctness definition in Section C.3.1 also adds an idempotency requirement that is not described in Section 3.2's assertion design and is not justified against the source posts. Please include the benchmark materials (or a public link), report curation reliability, and clarify how idempotency was operationalized for every task, since the low pass@k values and the error taxonomy both depend on these choices.
minor comments (6)
  1. [Section C.3.1] The line 'n = \u27e8Specify your value of n\u27e9 samples per task' is an unresolved placeholder; Section C.3.2 and Section C.3.3 state n=15, so the formula should state n=15 directly.
  2. [Appendix F.1, Appendix F.2] Appendix F.1 contains unedited placeholder bullets such as 'Present main pass@k results averaged across all tasks,' and Appendix F.2 has a truncated sentence ('demonstrated exceptionally strong performance, solving'). These should be completed or removed before submission.
  3. [Table 4] The column headers are inconsistent and garbled (e.g., 'Attribute & Template Issues' versus 'Attribute & parameter errors' in the text), and the per-model rows are based on only 15 sampled failures per model-task, so percentages such as 100% for DeepSeek-Distill-Q convey little information; consider reporting counts or confidence intervals.
  4. [Section 7] Section 7 states 44.7% for state-reconciliation errors while the abstract and Table 4 say 44.87%; align these numbers.
  5. [Section 7] The sentence 'incorporating error-aware guidance improved performance significantly' overstates Table 3, which shows gains of only 1–4 percentage points; suggest 'modestly' or 'only marginally.'
  6. [Appendix C.1, Tables 6–8] The TELeR prompt templates are inconsistent: Table 6 uses double-brace placeholders like {{title}} while Tables 7 and 8 use bare 'title' and 'constraint'; this should be unified so the prompt construction is reproducible.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the benchmark is execution-based and externally sourced; the headline failure-taxonomy percentages, though possibly unrepresentative, are not constructed from the claims they support.

full rationale

ITAB tasks are sourced from Stack Overflow posts and correctness is determined by dynamic execution against author-defined assertions; the pass@k numbers are measurements of model outputs, not consequences of the benchmark definition. The error taxonomy in Section 5 is a manual qualitative label set over 1,411 failures, and the abstract's 44.87%/24.37% division is a description of that labeled sample, not a derivation from the pass@k numbers. The error-aware prompting experiment in Section 4.2 uses the taxonomy as an input and tests whether it improves pass@k; that is a downstream intervention, not a circular validation of the taxonomy. Self-citations to Hassan et al. 2024 for state reconciliation and Santu and Feng 2023 for TELeR are background definitions and prompt-design choices, not uniqueness theorems or fitted parameters, and neither is load-bearing in the sense of forcing the empirical results. The Appendix G.1 sampling design ('we selected a representative task and, for each model, sampled 15 failed playbooks') makes the headline failure percentages an unweighted convenience-sample estimate rather than a failure-weighted distribution; this is a validity or generalizability concern about how well the percentages represent all ITAB failures, not a circularity concern, because the percentages are not equal by construction to any input of the benchmark. No equation in the paper reduces a predicted quantity to a fitted parameter or to a self-citation, so no circular step is identified.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central findings rest on three categories of unproven premises: the representativeness of the curated Stack Overflow tasks, the correctness of the author-designed validation scripts and idempotency requirement, and the reliability of the manual error taxonomy without inter-annotator agreement. No fitted parameters or invented entities are introduced.

assumptions (5)
  • domain assumption The 52,727 Stack Overflow posts in the Begoug et al. corpus reflect real-world IT automation tasks that practitioners want to automate.
    Section 3.1 uses this corpus as the population for stratified sampling.
  • domain assumption The author-authored initial states, target states, and verification assertions accurately capture the intent of the original Stack Overflow questions.
    Section 3.2 describes manual transformation from ambiguous posts to precise assertions; no inter-rater or external validation is reported.
  • ad hoc to paper A generated playbook is correct only if it is idempotent, as stated in Section C.3.1.
    Idempotency is a reasonable Ansible ideal but is not shown to be appropriate for all 126 tasks, and its role in pass@k is not analyzed in the main results.
  • domain assumption The nine-category error taxonomy can be applied reliably to the 1,411 failures by the authors without inter-annotator agreement.
    Section 5 describes a qualitative study by the authors, with no reported reliability statistics.
  • domain assumption TELeR Levels 1-3 operationalize prompt specificity as intended for this task.
    Section 3.4 and Appendix C.1 adopt TELeR from Santu and Feng (2023) without calibration for Ansible tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Large Language Models for IT Automation Tasks: Are We There Yet?." pith.science (2026). https://pith.science/paper/BCCG33WC

@misc{pith2026250520505,
  author       = {Pith},
  title        = {Pith review of: Large Language Models for IT Automation Tasks: Are We There Yet?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCCG33WC}},
  note         = {Machine review of arXiv:2505.20505}
}
read the original abstract

LLMs show promise in code generation, yet their effectiveness for IT automation tasks, particularly for tools like Ansible, remains understudied. Existing benchmarks rely primarily on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools, such as Ansible. We present ITAB (IT Automation Task Benchmark), a benchmark of 126 diverse tasks (e.g., configuring servers, managing files) where each task accounts for state reconciliation: a property unique to IT automation tools. ITAB evaluates LLMs' ability to generate functional Ansible automation scripts via dynamic execution in controlled environments. We evaluate 14 open-source LLMs, none of which accomplish pass@10 at a rate beyond 12%. To explain these low scores, we analyze 1,411 execution failures across the evaluated LLMs and identify two main categories of prevalent semantic errors: failures in state reconciliation related reasoning (44.87% combined from variable (11.43%), host (11.84%), path(11.63%), and template (9.97%) issues) and deficiencies in module-specific execution knowledge (24.37% combined from Attribute and parameter (14.44%) and module (9.93%) errors). Our findings reveal key limitations in open-source LLMs' ability to track state changes and apply specialized module knowledge, indicating that reliable IT automation will require major advances in state reasoning and domain-specific execution understanding.

Figures

Figures reproduced from arXiv: 2505.20505 by the authors.

Figure 1
Figure 1. Two syntactically correct code snippets used in Ansible-based IT automation where one is functionally [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The ITAB evaluation pipeline, showing different phases for testing LLM-generated Ansible code. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Impact of decoding parameters for the top 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Pass@1 and Pass@10 Performance Across IT Automation Domain Clusters for Four Model Families [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Performance of Qwen-2.5-Coder-7B model [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]
Figure 6
Figure 6. Figure 6: Performance of Qwen-2.5-7B-instruct-it model. [PITH_FULL_IMAGE:figures/full_fig_p028_6.png]
Figure 7
Figure 7. Figure 7: Performance of CodeGemma-7B-it model [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: Performance of CodeLLaMa-7B-it model [PITH_FULL_IMAGE:figures/full_fig_p029_8.png]
Figure 9
Figure 9. Figure 9: Performance of CodeLLaMa-13B-it model [PITH_FULL_IMAGE:figures/full_fig_p029_9.png]
Figure 10
Figure 10. Figure 10: Performance of DeepSeek-R1-Distil-LLaMa-8B model. [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]
Figure 11
Figure 11. Figure 11: Performance of DeepSeek-Coder-V2-it model. [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: Performance of DeepSeek-R1-Distil-Qwen-7B model. [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: Performance of LLaMa-3.1-8B-it model [PITH_FULL_IMAGE:figures/full_fig_p031_13.png]
Figure 14
Figure 14. Figure 14: Performance of LLaMa-3.2-3B-it model [PITH_FULL_IMAGE:figures/full_fig_p032_14.png]
Figure 15
Figure 15. Figure 15: Performance of Microsoft-phi-3.5-Mini-it model. [PITH_FULL_IMAGE:figures/full_fig_p032_15.png]
Figure 16
Figure 16. Figure 16: Performance of StarCoder2-7B model [PITH_FULL_IMAGE:figures/full_fig_p033_16.png]
Figure 17
Figure 17. Figure 17: Performance of WizardCoder-15B model [PITH_FULL_IMAGE:figures/full_fig_p033_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 31 canonical work pages

  1. [1]

    Replicability: The described problem needed to be potentially replicable within a container- ized Linux environment, excluding issues spe- cific to non-containerizable hardware, propri- etary systems unavailable in Docker, or GUI interactions

    work page

  2. [2]

    Posts relying heavily on obscure, deprecated, or highly specialized external collections were typically excluded

    Core Ansible Modules: The problem or its likely solution needed to involve mod- ules primarily from the ansible.builtin, ansible.netcommon, ansible.utils, or community.general collections, focusing on common, well-supported functionalities. Posts relying heavily on obscure, deprecated, or highly specialized external collections were typically excluded. A....

    work page 2023

  3. [3]

    Networking: Configuring network interfaces, IP addressing, routing, DNS, network services (DHCP, NTP), basic network device interac- tions

    work page

  4. [4]

    Deployment Pipelines: Automating steps in application deployment, including fetching artifacts, managing dependencies, deploying code, database migrations, basic CI/CD tasks

    work page

  5. [5]

    Variable Management: Handling variable definition, scope (host, group, play), prece- dence, lookup plugins, complex data struc- tures, and accessing facts

    work page

  6. [6]

    Templating: Using dynamic templates (pri- marily Jinja2) to generate configuration files based on variables and logic

    work page

  7. [7]

    Server Configuration: Managing server en- vironments, package installation/removal, ser- vice states (start, stop, restart), user/group management, basic system settings

    work page

  8. [8]

    Policy Configuration: Defining security set- tings (firewalls, SELinux), compliance rules, access controls, system policy enforcement

    work page

Show all 35 references
  1. [13]

    A.3 Manual Curation Criteria The set of posts selected via stratified sampling (ini- tially 200 posts) underwent manual review

    File Management: Creating, modifying (line- infile, blockinfile, replace), deleting files and directories, managing permissions, copy- ing/fetching files. A.3 Manual Curation Criteria The set of posts selected via stratified sampling (ini- tially 200 posts) underwent manual re...

  2. [14]

    Feasibility Issues: Upon closer inspection, the task required external services, hardware, credentials, or network configurations imprac- tical or insecure to replicate reliably within the isolated Docker test environment

  3. [15]

    Ambiguity/Incompleteness: The problem de- scription was too vague, lacked crucial details, or the accepted SO solution was unclear, in- complete, or non-functional, preventing the formulation of a clear, testable task and vali- dation criteria

  4. [16]

    upward trend

    Relevance Issues: The post, despite key- words, did not ultimately represent a core An- sible automation task solvable via the targeted modules or involved primarily debugging An- sible itself rather than using it for automation. This rigorous process yielded the final 126 tas...

  5. [17]

    Analyze Stack Overflow posts and transform them into practical Ansible solutions

  6. [18]

    Ensure cross-distribution compatibility in playbooks

  7. [19]

    All solutions should be self-contained and well-documented

    Follow infrastructure-as-code best practices You should: - Think step-by-step when analyzing problems - Consider distribution-specific differences - Validate network requirements - Ensure proper Y AML syntax - Verify solution completeness Never assume external dependencies or ...

  8. [25]

    Production-ready, self-contained Ansible playbook that: • Works across all specified distributions • Includes all necessary variables and files • Features comprehensive error handling • Follows Y AML best practices

  9. [26]

    Table 8: Level 3 TELeR (Santu and Feng, 2023) Prompt Structure for IT Automation Tasks

    Must be enclosed in triple backticks (“‘) for automated Python processing Note: Focus on creating a robust, production-ready playbook that can be directly implemented in the target environment. Table 8: Level 3 TELeR (Santu and Feng, 2023) Prompt Structure for IT Automation Ta...

    2023

  10. [27]

    Strictly comply with given constraint (including output format, filenames, and requirements)

  11. [28]

    Define all variables before use and avoid undefined variable errors

  12. [29]

    Avoid using invalid file paths

  13. [30]

    Carefully examine the constraint and runtime environment, and select modules appropriate for both

  14. [31]

    Use only valid attributes for the selected modules

  15. [32]

    Ensure valid Y AML and correct Ansible syntax and structure

  16. [33]

    prompt Transform the following Stack Overflow post into an Ansible playbook solution

    Make sure the output follows the format specified in the constraint. prompt Transform the following Stack Overflow post into an Ansible playbook solution. STACK OVERFLOW POST: Title: title Body: body TARGET ENVIRONMENT: Network: - Subnet: 10.1.1.0/24 - Gateway: 10.1.1.254 Nodes:

  17. [34]

    ubuntu1 (10.1.1.1) - Ubuntu Linux

  18. [35]

    alpine1 (10.1.1.2) - Alpine Linux

  19. [36]

    centos1 (10.1.1.3) - CentOS Linux

  20. [37]

    redhat1 (10.1.1.4) - Red Hat Linux IMPLEMENTATION CONSTRAINT: constraint DELIVERABLE REQUIREMENTS:

  21. [38]

    Brief analysis of the problem and your solution approach

  22. [39]

    • Carefully examine the constraint and runtime environment, and select modules appropriate for both

    Production-ready, self-contained Ansible playbook that: • Works across all specified distributions • Includes all necessary variables and files • Avoid using invalid file paths. • Carefully examine the constraint and runtime environment, and select modules appropriate for both...

  23. [40]

    {{ lookup ( ' env ', ' username ') }}

    Must be enclosed in triple backticks for automated Python processing Note: Focus on creating a robust, production-ready playbook that can be directly implemented in the target environment. Table 11: Level 3 TELeR (Santu and Feng, 2023) Prompt Structure with error awareness for...

    2023

  24. [2020]

    Empirical Soft- ware Engineering, 25(5):3430–3467

    The ‘as code’ activities: development anti- patterns for infrastructure as code. Empirical Soft- ware Engineering, 25(5):3430–3467. Akond Rahman and Chris Parnin. 2023. Detecting and characterizing propagation of security weaknesses in puppet-based infrastructure management. I...

    2023 arXiv

  25. [2021]

    CoRR, abs/2107.03374

    Evaluating large language models trained on code. CoRR, abs/2107.03374. Qihong Chen, Jiawei Li, Jiecheng Deng, Jiachen Yu, Justin Tian Jin Chen, and Iftekhar Ahmed. 2024. A deep dive into large language model code generation mistakes: What and why? CoRR, abs/2411.01414. DeepSe...

    2024 arXiv

  26. [2024]

    Preprint, arXiv:2402.19173

    Starcoder 2 and the stack v2: The next genera- tion. Preprint, arXiv:2402.19173. Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, and Jör...

    2025 arXiv

  27. [2025]

    OpenReview.net

    Swe-bench multimodal: Do AI systems gen- eralize to visual software domains? In The Thir- teenth International Conference on Learning Repre- sentations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham...

    2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.