{"id":"f97e807e-d28f-4959-88f0-fadc1bb942f6","arxiv_id":"2505.20505","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new execution-based benchmark of 126 Ansible tasks shows open-source LLMs reach at most 12% pass@10, with most failures coming from state-tracking and module-knowledge errors.","lead":"Open-source large language models, tested on a new benchmark of 126 real-world Ansible automation tasks, can generate a working script at most 12% of the time when given 10 tries. The biggest failure points are tracking the target system state and knowing the correct module options, not just writing valid syntax.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 44.87%/24.37% bottleneck split is derived from a fixed 15-failure sample of one representative task per domain, not from a failure-weighted sample, so the central explanatory claim is not yet established.","rationale":"The reader's weakest assumption concerned the author-defined initial states, target states, and validation assertions, and the reader's CONDITIONAL verdict cited missing artifacts, the exclusion of DeepSeek-R1 models from aggregate taxonomy percentages, and missing inter-annotator reliability. I agree that the task-state mapping affects whether pass@k measures real practitioner need, but the more directly load-bearing issue for the paper's central explanatory claim is the construction of the error-taxonomy percentages. This concern is internal to the paper: Appendix G.1 describes a fixed-size, one-representative-task-per-category sample, while the abstract presents the resulting percentages as general properties of LLM failures on ITAB. This is not a disagreement with consensus or a preference for different models; it is a correctness risk in the quantitative support for the stated bottleneck conclusion. The low pass@10 values remain credible and are supported by Table 2, so the paper should not be rejected outright; however, the headline explanation for those failures should not be accepted until the taxonomy is recomputed on a failure-weighted sample or explicitly presented as exploratory. Hence the verdict remains CONDITIONAL, with the additional condition that the taxonomy percentages be recomputed and reported with reliability and sampling details.","tokens_in":29582,"tokens_out":5951,"duration_ms":67507,"concrete_test":"Release the 1,411 sampled failures with task IDs and the total failure count for each (model, task) configuration. Resample failures proportional to actual failure counts across all 126 tasks (target at least 2,000 total cases), have two independent annotators label the resample, report Cohen's kappa, and recompute the aggregate percentages using failure-weighted proportions. If the combined state-reconciliation share (variable + host + path + template) falls below about 35%, or the module-knowledge share (attribute/parameter + module) shifts by more than 5 percentage points, the abstract's bottleneck framing should be revised or explicitly re-scoped to the selected representative tasks.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The low pass@k results (Table 2) support the claim that these 3B–14B open models rarely produce functionally correct playbooks on ITAB. The load-bearing part of the paper's central claim, however, is the explanation of why: the abstract and Section 5 attribute 44.87% of failures to state-reconciliation reasoning and 24.37% to module-specific execution knowledge. Appendix G.1 states that for each IaC category 'we selected a representative task and, for each model, sampled 15 failed playbooks stratified by temperature and TELeR prompt level.' The percentages in Table 4 and the abstract are therefore unweighted proportions of a convenience sample with a fixed 15 samples per model-task pair, not estimates of the actual distribution of the 1,411 failures across the benchmark. This design over-weights rare failure types from model-task pairs with few total failures and under-weights dominant failure types in high-failure tasks or categories. Even with perfect manual labels, the headline percentages do not generalize to all ITAB failures; with only one task per domain, they may reflect task-specific quirks rather than general model weaknesses. The reader flagged the absence of inter-annotator reliability, which is real, but the sampling design is a more fundamental threat: it means the quantitative headline is not a valid estimate of the failure distribution it claims to summarize.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ITAB, a benchmark of 126 Ansible automation tasks derived from Stack Overflow posts across seven IaC domains, with containerized execution and task-specific validation assertions. The authors evaluate 14 open-source LLMs (3B–14B) under three TELeR prompt levels and four sampling temperatures using pass@k, finding very low success rates (best pass@10 = 12.0%). They then present an error taxonomy from 1,411 execution failures, reporting 44.87% state-reconciliation reasoning errors and 24.37% module-specific execution knowledge errors, and show that error-aware prompting yields only marginal gains. The paper concludes that current open LLMs lack reliable state tracking and domain-specific execution knowledge for IT automation.","tokens_in":29807,"tokens_out":6734,"duration_ms":66782,"significance":"If the benchmark and the error attribution are valid, the paper makes a useful contribution: execution-based evaluation of IaC generation is underrepresented, and the multi-distribution Docker environment, pass@k estimator, and idempotency-aware correctness criterion are commendable strengths. The low pass@k results are machine-checked through dynamic execution and convincingly show that 3B–14B open models rarely generate functionally correct Ansible playbooks. The explanatory error-taxonomy percentages, however, are not currently supported as global estimates of the 1,411 failures, so the headline bottleneck claim needs reanalysis or careful reframing before the paper's central conclusion is established.","major_comments":[{"comment":"The 44.87% and 24.37% percentages are computed from a convenience sample, not from all 1,411 failures: Appendix G.1 states that for each IaC category 'we selected a representative task and, for each model, sampled 15 failed playbooks stratified by temperature and TELeR prompt level.' This balanced design gives every model-task pair equal weight regardless of how many failures it actually produced, so rare failure types are over-represented and dominant failure types in high-failure tasks are under-represented. The abstract and Section 5 present these numbers as the distribution of the 1,411 failures; they are only the distribution within this fixed 15-per-model-task sample. Please reweight by actual failure counts, sample proportionally to failures, or explicitly reframe the percentages as 'within the stratified error sample' and adjust the abstract accordingly.","section":"Appendix G.1, Section 5, Table 4, Abstract"},{"comment":"The aggregation in Table 4 is internally inconsistent: the table shows rows for DeepSeek-Distill-L and DeepSeek-Distill-Q, but the caption says these syntax-heavy distributions are excluded from the aggregated percentages. The abstract then reports the aggregate values globally without this caveat. Please either remove the reasoning-distilled rows from the table, show a separate aggregate that includes them, or clearly state in the abstract that the headline percentages apply only to the 12 non-reasoning-distilled models.","section":"Table 4, Section 5, Abstract"},{"comment":"The abstract's claim that 'none of which accomplish pass@10 at a rate beyond 12%' is contradicted by Table 3, where Qwen2.5-Coder-7B-it reaches pass@10 of 15.5% with error-aware prompts. Since the paper itself evaluates this prompting condition, the abstract must qualify the claim as applying to the standard TELeR prompt condition, or report the error-aware results as part of the headline evaluation.","section":"Abstract, Section 4.2, Table 3"},{"comment":"The benchmark's construct validity rests on the authors' translation of ambiguous Stack Overflow posts into precise initial states, target states, and validation assertions, but no inter-annotator reliability or independent validation is reported, and no artifact or availability statement is provided for the 733 test cases and validation scripts. The correctness definition in Section C.3.1 also adds an idempotency requirement that is not described in Section 3.2's assertion design and is not justified against the source posts. Please include the benchmark materials (or a public link), report curation reliability, and clarify how idempotency was operationalized for every task, since the low pass@k values and the error taxonomy both depend on these choices.","section":"Section 3.2, Section 9, Appendix B"}],"minor_comments":[{"comment":"The line 'n = \\u27e8Specify your value of n\\u27e9 samples per task' is an unresolved placeholder; Section C.3.2 and Section C.3.3 state n=15, so the formula should state n=15 directly.","section":"Section C.3.1"},{"comment":"Appendix F.1 contains unedited placeholder bullets such as 'Present main pass@k results averaged across all tasks,' and Appendix F.2 has a truncated sentence ('demonstrated exceptionally strong performance, solving'). These should be completed or removed before submission.","section":"Appendix F.1, Appendix F.2"},{"comment":"The column headers are inconsistent and garbled (e.g., 'Attribute & Template Issues' versus 'Attribute & parameter errors' in the text), and the per-model rows are based on only 15 sampled failures per model-task, so percentages such as 100% for DeepSeek-Distill-Q convey little information; consider reporting counts or confidence intervals.","section":"Table 4"},{"comment":"Section 7 states 44.7% for state-reconciliation errors while the abstract and Table 4 say 44.87%; align these numbers.","section":"Section 7"},{"comment":"The sentence 'incorporating error-aware guidance improved performance significantly' overstates Table 3, which shows gains of only 1–4 percentage points; suggest 'modestly' or 'only marginally.'","section":"Section 7"},{"comment":"The TELeR prompt templates are inconsistent: Table 6 uses double-brace placeholders like {{title}} while Tables 7 and 8 use bare 'title' and 'constraint'; this should be unified so the prompt construction is reproducible.","section":"Appendix C.1, Tables 6–8"}],"recommendation":"major_revision","confidential_remarks":"The pass@k results and the benchmark's execution-based design are solid enough to warrant revision rather than rejection. The main risk is that the error-taxonomy percentages, which are the paper's key explanatory contribution, are derived from a small balanced sample and are presented as global estimates; this is fixable by reweighting or careful reframing. The abstract also needs qualification regarding Table 3. If the authors address the sampling and artifact-availability issues, the paper would be a valuable contribution to executable code-generation evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. ITAB is a genuinely useful benchmark: 126 Ansible tasks mined from Stack Overflow, executed in a multi-OS Docker setup, with pass@10 topping out at 12% for 14 open-source models. That core result is believable and worth having. The second thing is that the paper's explanation for those failures—the 44.87% state-reconciliation versus 24.37% module-knowledge split—is not established by the data in the way the abstract presents it.\n\nThe benchmark construction is thoughtful. Tasks come from real Stack Overflow posts, not synthetic templates, and correctness is judged by execution against author-defined initial and target states, including an idempotency check. That is a real improvement over BLEU-based evaluations like WISDOM-Ansible and over static checks. The pass@k numbers are internally consistent with the pipeline; the temperature and TELeR effects are plausible and qualitatively supported.\n\nThe soft spot is Appendix G.1. For each of the seven domains they picked one representative task, sampled 15 failed playbooks per model, and manually labeled them. The aggregate percentages in Table 4 are an equal-weight blend across models, after excluding the two DeepSeek-R1 distill models. So 44.87% and 24.37% are not estimates of the 1,411 failures; they are proportions of a convenience sample. Models with many failures count no more than models with few, and one task per domain is too thin to support a general claim about why these models fail. The exclusion of the distill models is disclosed in the Table 4 caption, but not in the abstract, where the numbers are presented as covering all evaluated models. There is also no inter-annotator reliability, and no released artifacts or validation scripts, which makes the results hard to independently check.\n\nThese are fixable. The pass@k result does not depend on the taxonomy. The taxonomy should either be re-analyzed as failure-weighted proportions across the full failure set, or recast as qualitative per-task observations rather than global percentages.\n\nWho is this for? Anyone working on execution-based code generation for infrastructure-as-code, or on benchmark design generally. It deserves a serious referee. I would send it to review with a request for major revision—release artifacts, report reliability, and fix the error-taxonomy analysis. The negative result should survive; the reason why needs to be re-earned.\n\nIn short: worth engaging, but don't quote the 44.87%/24.37% split in anything you write until the analysis is corrected.","headline":"The low pass@k result is real and ITAB is a step forward, but the headline error-taxonomy percentages come from a convenience sample and overstate what the data supports.","tokens_in":30342,"tokens_out":3951,"would_cite":true,"duration_ms":37461,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current open-source LLMs cannot reliably write working Ansible playbooks: on a 126-task execution benchmark, none exceed 12% pass@10, with failures concentrated in state tracking and module knowledge.","keywords":["LLM code generation","IT automation","Ansible","benchmark","state reconciliation","pass@k evaluation","error taxonomy","infrastructure as code"],"falsifier":"Take a random sample of roughly 30 of the 126 tasks and have experienced Ansible practitioners independently reconstruct the initial state, target state, and validation assertions without seeing the paper's versions; substantial disagreement would show the low scores partly reflect the benchmark's chosen strictness, while convergence would confirm the 12% ceiling as a real measure of current open-source capability. A complementary check is to retry the failed generations with execution feedback (show the model the error and let it revise): if any of the nine error categories disappears entirely, that category is a prompt-or-sampling artifact rather than a fixed model limitation.","tokens_in":29343,"feed_emoji":"⚙️","tokens_out":12229,"duration_ms":102680,"temperature":0.7,"pith_summary":"The paper sets out to answer whether open-source large language models can turn real-world, user-written IT automation requests into Ansible playbooks that actually work, rather than merely into scripts that parse as YAML. To test this it builds ITAB, a benchmark of 126 tasks drawn from 52,727 Stack Overflow posts, each with a defined initial system state, a target state, and automated assertions verified by executing the generated playbook in a fresh Docker environment. Evaluating 14 open-source LLMs between 3B and 14B parameters, it finds that none achieve a $\\mathrm{pass@}10$ above 12%, and nearly all stay below 4% at $\\mathrm{pass@}1$. An analysis of 1,411 execution failures attributes 44.87% of errors to state-reconciliation reasoning (variable, host, path, and template handling) and 24.37% to weak module-specific execution knowledge, supporting the paper's conclusion that reliable LLM-driven IT automation will require advances in state reasoning and domain-specific execution understanding, not just better prompts.","feed_headline":"No LLM tops 12% on real-world Ansible automation tasks","feed_subtitle":"A 126-task execution benchmark traces the failures to state tracking and Ansible module knowledge.","key_machinery":"The load-bearing object is ITAB itself: an execution-driven benchmark whose unit of evaluation is a state-reconciliation task. Each of its 126 tasks is built from a Stack Overflow post by formalizing an initial system state and a target state, identifying the parameters (file paths, services, hosts) that can change the outcome, and writing assertion scripts that check the target state after the playbook runs inside a fresh, isolated four-node Docker network spanning Ubuntu, Alpine, CentOS, and Red Hat. A playbook passes only if it validates as YAML, executes cleanly, satisfies the assertions, and is idempotent, with $\\mathrm{pass@}k$ computed by the unbiased estimator over 15 generated samples per model-prompt-temperature configuration. The benchmark's specific contribution is that it evaluates state reconciliation, the property that an automation tool infers the desired state, compares it with the current state, and applies only the necessary changes, something static correctness checks cannot see. The supporting machinery is a nine-category error taxonomy built from 1,411 execution failures, which sorts the failures into two semantic families: state-reconciliation reasoning (variable, host, path, and template issues, 44.87% combined) and module-specific execution knowledge (attribute/parameter and module errors, 24.37%).","core_discovery":"The paper's central claim is that functional correctness for IT automation is a distinct and harder problem than code generation as normally benchmarked, and that current open-source LLMs are not yet reliable at it. On ITAB, a generated playbook counts as correct only if it is syntactically valid, executes without error inside a fresh containerized network, leaves the system in the asserted target state, and is idempotent; by that standard, the strongest evaluated model solves at most 12% of tasks within ten samples. The failure analysis shows the models generally know which automation action is needed but mishandle the details: they mis-track state across hosts, variables, paths, and Jinja2 templates, and they misconfigure module attributes even when they pick the right module. The authors conclude that the two dominant bottlenecks, state-reconciliation reasoning and module-specific execution knowledge, are precisely the abilities that prompt engineering cannot supply, which is why the paper answers its title's question in the negative.","pith_inferences":["The strict correctness definition (execution plus asserted target state plus idempotency) may undercount a model that reaches the desired state on a single run but not on a second run; whether practitioners always demand idempotency is a question the paper leaves open, and relaxing that requirement is a testable variant of the benchmark.","The evaluation's 3B-14B open-source range leaves open how larger closed models would fare; if a much larger model clears the 12% ceiling decisively, the bottleneck is scale or training-data breadth rather than the task class itself, an easy comparison to run with the same harness.","The nine-category error taxonomy points to interventions the paper does not test, such as fine-tuning on Ansible module documentation, supervision signals for state tracking, or execution-feedback loops that let a model see its failure and revise.","One appendix passage (Section C.3.1) leaves the sample count as an unfilled template token ('Specify your value of n'), although the body and other appendix sections consistently report 15 samples; anyone reproducing the pass@k numbers should confirm the intended value."],"forward_implications":["Syntactic validity is a weak proxy for operational correctness: prompts enriched with the paper's own error taxonomy lift pass@k by only 1-4 percentage points, so the observed failures are not mainly instruction-following problems.","State-sensitive domains, Templating (Jinja2) and Variable Management, are the hardest for every evaluated model, while Server Configuration and File Management are comparatively tractable, so benchmarks and training should prioritize state-heavy tasks.","Sampling temperature is a genuine control knob with a trade-off: low temperatures (0.2) maximize first-attempt reliability, high temperatures (0.6-0.8) improve ten-sample diversity but also widen the variety of errors.","Reasoning-distilled models fail overwhelmingly on basic Ansible syntax, indicating that general reasoning strength does not transfer to structured infrastructure-as-code generation without domain grounding.","Progress toward dependable language-guided automation should be measured with execution-based, state-reconciliation-aware benchmarks such as ITAB, because static checks cannot detect the dominant failure modes."],"supporting_citations":[{"why":"Supplies the 52,727-post Stack Overflow corpus and the seven IT automation domain categories used for stratified sampling and task curation.","marker":"Begoug et al., 2023"},{"why":"Provides the pass@k metric and the sampling-based evaluation paradigm ITAB adapts for functional correctness.","marker":"Chen et al., 2021"},{"why":"Defines state reconciliation defects in infrastructure as code, the property ITAB is explicitly designed to test.","marker":"Hassan et al., 2024"},{"why":"Characterizes how automation tools like Ansible infer desired states, compare with current states, and apply necessary changes.","marker":"Rahman and Parnin, 2023"},{"why":"Supplies the TELeR prompt taxonomy whose Level of Detail dimension the study varies across Levels 1-3.","marker":"Santu and Feng, 2023"},{"why":"IaC-Eval, the prior IaC benchmark ITAB contrasts against; it uses human-curated synthetic configurations rather than execution-based validation.","marker":"Kon et al., 2024"},{"why":"WISDOM-Ansible, the prior Ansible evaluation ITAB contrasts against; it relies on BLEU scores instead of dynamic execution.","marker":"Pujar et al., 2023"}],"fun_headline_variants":["LLMs fail 88% of real Ansible tasks in new benchmark","State tracking and module knowledge trip up LLMs on IT automation","IT automation stumps open-source LLMs: under 12% success","Why LLMs can't handle Ansible: state and module errors dominate","Benchmark shows LLMs lack state reasoning for IT tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the initial states, target states, and validation assertions the authors derived for each of the 126 tasks faithfully capture what the original Stack Overflow posters were asking for, and that requiring a correct playbook to be idempotent matches what practitioners actually need; if those mappings are wrong or too strict, the low $\\mathrm{pass@}k$ scores and the error taxonomy measure the benchmark's own strictness rather than real-world model capability.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail 88% of real Ansible tasks in new benchmark","State tracking and module knowledge trip up LLMs on IT automation","IT automation stumps open-source LLMs: under 12% success","Why LLMs can't handle Ansible: state and module errors dominate","Benchmark shows LLMs lack state reasoning for IT tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000726,"raw_usage":{"total_tokens":3275,"prompt_tokens":988,"completion_tokens":2287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2196}},"tokens_in":604,"tokens_out":2287,"duration_ms":17506,"temperature":1.0,"reasoning_tokens":2196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:52:53.951576+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of roughly 30 of the 126 tasks and have experienced Ansible practitioners independently reconstruct the initial state, target state, and validation assertions without seeing the paper's versions; substantial disagreement would show the low scores partly reflect the benchmark's chosen strictness, while convergence would confirm the 12% ceiling as a real measure of current open-source capability. A complementary check is to retry the failed generations with execution feedback (show the model the error and let it revise): if any of the nine error categories disappears entirely, that category is a prompt-or-sampling artifact rather than a fixed model limitation.","supporting_citations":[],"review_version":1}