Pith. sign in

REVIEW 2 major objections 4 minor 9 references

By separating AI-generated Terraform failures into validation, planning, and policy checks, this paper shows that retrieval, repair, and prompt optimization improve different failure classes — and that most policy failures disappear when th

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 12:43 UTC pith:Y2MI66EM

load-bearing objection Strong benchmark and stage-based failure analysis, but the GEPA +7.5pp claim is likely in-sample and needs a held-out split before it can be believed. the 2 major comments →

arxiv 2607.20478 v1 pith:Y2MI66EM submitted 2026-05-29 cs.SE cs.AI

Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation

Mohamed Jouini This is my paper
classification cs.SE cs.AI
keywords Infrastructure-as-CodeTerraformLLM code generationverifier-first evaluationretrieval-augmented generationiterative refinementprompt optimizationOpen Policy Agent
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that infrastructure-as-code generation by LLMs should be evaluated verifier-first: instead of a single pass/fail score, failures should be separated into schema-validation, planning, and policy-compliance stages. Under that lens, the paper argues that three standard interventions — active retrieval, iterative repair from verifier errors, and instruction-level prompt optimization — improve different, largely complementary failure classes. Concretely, active retrieval lifts a small local model from 14.0% to 45.7% pass@1 by reducing schema errors, iterative repair reaches 62.9% (and 84.4% for a frontier model), and GEPA instruction optimization adds a further +7.5 points. The paper also finds that 79% of residual policy failures vanish when the policy text is shown to the model, arguing that many 'capability' failures are actually prompt–policy information gaps. If right, the work changes how IaC agents should be benchmarked and where to spend engineering effort.

Core claim

The central discovery is a decomposition: on a three-stage verifier (validate, plan, policy eval), each intervention fixes a different slice. Active retrieval (agents querying provider docs while generating) cuts validation failures from 144 to 66, lifting Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p<0.0001). Verifier-guided retries reach 62.9% (Qwen) and 84.4% (GPT-4o), with binary convergence. GEPA instruction optimization adds +7.5 points (p=0.026) by encoding verifier-revealed constraints. Injecting Rego policy text resolves 11 of 14 residual OPA failures (p=0.016), showing most policy failures are information gaps, not capability gaps. Residual schema hallucination drops from 50% to 2

What carries the argument

The three-stage verifier-first harness — terraform validate, terraform plan, opa eval — with a failure taxonomy (VALIDATE_FAIL, PLAN_FAIL, OPA_FAIL) and paired McNemar tests on 186 tasks. The ReAct agent loop (Thought → Action → Observation) provides active retrieval via structured tools or dense search; GEPA is a reflective prompt optimizer driven by verifier feedback; SIMBA is a teacher-free demonstration injector. The harness is the central object: it converts pass@1 into stage-specific attribution.

Load-bearing premise

The GEPA +7.5pp claim assumes the reported 53.2% pass@1 is not inflated by evaluating on tasks used to optimize the instructions; the paper does not report a held-out test split separate from the 124-task training set.

What would settle it

Run GEPA's optimized instruction on a held-out subset of IaC-Eval v2 tasks that were not part of the 124-task training set; if pass@1 drops to near the Active RAG baseline (~45.7%), the +7.5pp effect is fitted rather than general. Alternatively, inspect the per-task pass/fail vectors to check whether GEPA's improved tasks overlap the training set.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Evaluation should preserve verifier-stage information; a single pass@1 hides whether errors are schema, dependency, or policy failures.
  • Retrieval and repair are complementary: retrieval prevents schema errors before generation, repair corrects them after; they address different failure classes.
  • Instruction-level optimization (GEPA) outperforms demonstration-level injection (SIMBA) for schema-constrained IaC, and SIMBA can match active retrieval (p=1.0) without retrieval infrastructure.
  • Most policy failures are information gaps: making policy text visible resolves 79% of residual OPA failures, motivating policy-aware prompt construction.
  • Schema hallucination (SELF_DEFINED_PROPERTY) is model-scale-limited; instruction rules partially compensate, and scaling largely eliminates it.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same verifier-stage funnel should transfer to Azure/Bicep, GCP, Pulumi, or Kubernetes manifests, where analogous validate/plan/policy gates exist; the paper only claims AWS/Terraform.
  • SIMBA's and GEPA's unique successes (20 vs 35 tasks) imply a routed or sequential single-pass optimizer could reach roughly 64% pass@1, matching the iterative-repair result without any retry budget.
  • Because 79% of policy failures disappear when Rego text is visible, a practical policy summarizer that injects compact constraint rules into prompts could close most of the OPA gap in production, where exposing raw policy may be undesirable.
  • If the binary convergence pattern holds, repair budgets beyond the first retry have sharply diminishing returns for strong models, suggesting dynamic early-exit repair policies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents a verifier-first study of seven agentic strategies for Terraform/IaC generation on a 186-task AWS benchmark (IaC-Eval v2), separating failures into terraform validate, terraform plan, and opa eval gates. It reports five principal findings: active retrieval raises Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1; iterative refinement with verifier feedback reaches 62.9% (Qwen 7B) and 84.4% (GPT-4o); GEPA instruction optimization adds +7.5 pp over Active RAG; SIMBA demonstration injection matches retrieval without retrieval infrastructure; and a Rego-injection diagnostic shows 79% of post-refinement OPA failures are information-gap failures. The paper includes per-task result vectors, MLflow trace IDs, McNemar tests, and appended optimizer artifacts (evolved instruction and SIMBA demonstration).

Significance. If the results hold, the paper's main methodological contribution is valuable: stage-decomposed failure accounting, paired statistical tests, and trace-level reproducibility are exactly what agentic IaC evaluation needs. The retrieval and iterative-refinement findings are supported by the reported tables and are plausible. The GEPA optimizer result, however, is currently evaluated on a set that overlaps the optimization training set, so the central 'prompt optimizers generalize to verifiable IaC generation' claim is not established. The RQ4 p-value and transition-table arithmetic also need correction. Because these issues are load-bearing but addressable with a held-out split and corrected statistics, the paper warrants major revision rather than rejection.

major comments (2)
  1. [§3.5, §4.4, Table 5, Figure 1] GEPA and SIMBA are optimized on a 124-task training set and a 20-task Pareto validation set, but Table 5 and Figure 1 report pass@1 over all 186 tasks. No held-out test split is defined, and no per-split result is reported. Thus 144/186 tasks (77%) directly informed the evolved instruction or selected demonstration, and the +7.5 pp GEPA improvement (p=0.026) is a training-set comparison, not an out-of-sample estimate. Appendix B strengthens this concern: the optimized instruction contains task-specific memorized rules (e.g., aws_lightsail_database blueprint_id = 'postgres_12', aws_elasticache_user engine exactly 'redis'), not general abstract schema knowledge. The McNemar test is computed on paired outcomes that are not independent of the optimizer's selection process. Please re-analyze or re-run on a genuinely held-out set and report train/validation/test rows for both GEPA and SIMBA. U
  2. [§4.5, Table 7, Findings 13–14] Table 7 sums to 183 rather than 186, with no rows explaining the missing three tasks, and the stated p=0.016 for 14 paired OPA failures with 11 resolutions and 3 non-resolutions is not reproducible from the stated counts: the continuity-corrected McNemar chi-square for discordant counts (11,3) is 3.5, corresponding to p≈0.057, and the exact two-sided binomial p is also ≈0.057. Please provide the full 186-task transition matrix and the exact test used. In addition, Finding 14 excludes three 'benchmark artifact' failures post hoc and changes the denominator from 186 to 183; without pre-specified exclusion criteria, the 79% information-gap claim and the 96.2% diagnostic upper bound should be reported with and without exclusions. If the 3 remaining OPA failures are artifacts, then among non-artifact OPA failures 11/11 resolve, not 11/14, so the 79% figure mixes artifact and genuine failure c
minor comments (4)
  1. [§4.2, Table 3, Finding 3] Table 3 reports ActiveRAG vs. MCP as p=0.47, while the text of Finding 3 states p=0.38. Please reconcile the reported p-values.
  2. [Figures 1 and 2] The figure environments contain repeated caption text verbatim in the body text below the captions. Remove the duplicated lines.
  3. [§8 Threats to Validity] The 'Optimizer validity' paragraph does not mention data-split leakage or selection of evaluation tasks relative to optimizer training. Given that the GEPA and SIMBA results are cited in the abstract, this threat should be discussed explicitly.
  4. [§3.5] Please clarify how the 80 verifier-guided rollouts are allocated across the 124-task training set and 20-task validation set. As written, it is unclear which tasks contribute to the 80 rollouts and whether every training task receives at least one rollout.

Circularity Check

1 steps flagged

GEPA +7.5pp is partly a fitted value: the optimizer trains on 124/186 tasks, yet Table 5 reports pass@1 on all 186 with no held-out split.

specific steps
  1. fitted input called prediction [§3.5 DSPy Optimization Configuration; §4.4 Table 5, Finding 8]
    ""The optimizer runs for 80 verifier-guided rollouts with a 20-task Pareto validation set and a 124-task training set." Table 5: "DSPy optimization results (Qwen2.5-Coder 7B, 186 tasks, single-pass)." Finding 8: "GEPA instruction optimization raises the Active RAG agent from 45.7% to 53.2% (+7.5 pp, McNemar p=0.026).""

    The paper never defines a held-out test split: §3.5 says GEPA is optimized on a 124-task training set, while Table 5 and Figure 1 report pass@1 on all 186 benchmark tasks. Since the benchmark contains exactly 186 tasks, the evaluation set necessarily includes the 124 tasks used to evolve GEPA's instruction unless an omitted split is used. The +7.5pp and its McNemar p-value are therefore computed partly on training data, measuring fit to the optimized tasks rather than an out-of-sample prediction. Appendix B confirms this by showing the optimized instruction encodes task-specific rules such as 'engine = "redis"' for aws_elasticache_user, consistent with memorizing training failures. No threat-to-validity section addresses this optimizer data leakage.

full rationale

The main circularity is localized to the GEPA result. The paper presents +7.5pp as evidence that prompt optimizers generalize to verifiable IaC generation, but §3.5 states GEPA is optimized on a 124-task training set and Table 5 evaluates 186 tasks with no reported held-out split. Since 124/186 of the benchmark tasks were used for verifier-guided instruction evolution, the reported aggregate includes training tasks; the +7.5pp is at least partially a fitted value, and the McNemar test on those 186 paired outcomes is not a pure generalization test. This is a fitted-input-called-prediction pattern. The same absence of a held-out split affects SIMBA's evaluation, but that result is less consequential because SIMBA is only claimed to match Active RAG and uses a single demonstration selected from the training set. The retrieval, repair, and Rego-oracle findings are not circular: they are interventions compared on the same benchmark, and their mechanisms are not derived from the quantities they predict. There is no load-bearing self-citation or theorem-import circularity. Score 6, not higher, because the GEPA claim is one of five principal findings and the other results retain independent content; score is not 0 because the primary optimization claim is partially reduced to its training objective by the absent split.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper is an empirical study; the central claims rest on the benchmark's validity, the verifier proxies, and the unstated holdout assumption for GEPA. The GEPA instruction and SIMBA demonstration are fitted artifacts; numeric hyperparameters (budgets, splits) are chosen by hand.

free parameters (5)
  • GEPA evolved instruction text = 40+ line domain specification with rules (e.g., aws_elasticache_user password length, engine lowercase)
    Evolved on 124-task training set using verifier feedback; if evaluation includes these tasks, the pass@1 gain is fitted.
  • SIMBA selected demonstration = Task 265 (aws_iam_group)
    Selected by trajectory contrast on training set; minimal schema guidance.
  • Repair budget k = 4
    Chosen by hand; results (e.g., binary convergence) depend on this budget.
  • GEPA rollout budget and split sizes = 80 rollouts, 20-task validation, 124-task training
    Optimization budget and data split; no search over these hyperparameters and no reported test split.
  • Post-hoc excluded benchmark artifacts = 3 OPA failures excluded for 'self-contradictory Rego' and 'incorrect JSON path'
    Exclusion is used to claim 96.2% upper bound; arithmetic inconsistent.
axioms (5)
  • domain assumption terraform validate/plan/opa eval are valid proxies for IaC correctness
    The methodology treats passing these three gates as 'pass'; the paper acknowledges it is not a deployment test (§8).
  • domain assumption The Rego policies in IaC-Eval v2 faithfully encode intent
    The benchmark was created/modernized by the authors; some tasks are admitted artifacts.
  • domain assumption Temperature=0 makes results deterministic and representative
    Single-run at temperature 0; threats section notes hosted-model variance.
  • domain assumption Manual error coding of SELF_DEFINED_PROPERTY is reliable
    No inter-annotator agreement or coding guide provided.
  • ad hoc to paper The evaluation set for GEPA is disjoint from the optimization training set
    The paper never states this; if false, the GEPA improvement is fitted. This is a required premise for the central claim.

pith-pipeline@v1.3.0-alltime-deepseek · 12994 in / 17284 out tokens · 145461 ms · 2026-08-02T12:43:59.919760+00:00 · methodology

0 comments
read the original abstract

Infrastructure-as-Code (IaC) generation from natural language requires satisfying provider schemas, dependency planning, and organizational policy constraints, not merely producing syntactically plausible configurations. We present a verifier-first empirical study of seven agentic strategies for Terraform generation evaluated on IaC-Eval v2, a modernized 186-task AWS/Terraform benchmark with Rego v1 intent policies. Our evaluation separates failures into three verifier stages (terraform validate, terraform plan, opa eval) and applies McNemar's test with Wilson confidence intervals on all pairwise comparisons (n=186, alpha=0.05). We report five principal findings. (1) Active retrieval via ReAct agents with MCP or ChromaDB-backed RAG raises Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p<0.0001), primarily by reducing VALIDATE_FAIL from 144 to 66 tasks. (2) Iterative refinement with verifier feedback achieves 62.9% (Qwen 7B) and 84.4% (GPT-4o) pass@1, exhibiting binary convergence -- tasks either resolve in one retry or exhaust the budget. (3) GEPA reflective instruction optimization raises the Active RAG baseline by +7.5 pp (p=0.026) using only 80 verifier-guided rollouts, providing evidence that prompt optimizers can improve verifiable IaC generation without weight updates. (4) SIMBA teacher-free demonstration injection achieves performance equivalent to Active RAG (p=1.0) without retrieval infrastructure, but fails to address the dominant SELF_DEFINED_PROPERTY error class (50% of failures). (5) A diagnostic Rego-injection experiment shows that 79% of post-refinement OPA failures are information-gap failures resolvable when policy text is visible (p=0.016), motivating policy

Figures

Figures reproduced from arXiv: 2607.20478 by Mohamed Jouini.

Figure 1
Figure 1. Figure 1: Overall pass@1 rate across all evaluated strategies (IaC-Eval v2, 𝑛 = 186). The performance ladder progresses from CoT through active retrieval, prompt optimization, iterative refinement, and the diagnostic policy oracle [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Iterative refinement convergence curves. GPT-4o achieves most gains by the second attempt; Qwen 7B benefits from additional retries but remains bounded by schema-knowledge failures [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Failure-stage decomposition across Qwen strategies. The figure uses the same columns as [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 3
Figure 3. Figure 3: Failure-stage decomposition across Qwen 7B strategies. Improvements primarily reduce validate_fail; OPA failures grow modestly as configurations progress further through the pipeline (failure promotion). prompt or policy enrichment becomes necessary to address these later-stage failures. 6.2 GEPA and RAG Are Complementary GEPA optimizes reasoning strategy (how to generate Terraform from retrieved docs); RA… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

9 extracted references · 8 linked inside Pith

  1. [1]

    GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning

    [Agrawal et al., 2026] Agrawal, L. A. et al. “GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. ” ICLR 2026 (Oral). arXiv:2507.19457. [Anthropic et al., 2024] Anthropic et al. “Model Context Protocol. ” Specifica- tion, November

    Pith/arXiv arXiv 2026

  2. [3]

    DSPy: Compiling Declara- tive Language Model Calls into Self-Improving Pipelines

    arXiv:2403.14403. [Khattab et al., 2024] Khattab, O. et al. “DSPy: Compiling Declara- tive Language Model Calls into Self-Improving Pipelines. ” ICLR

    Pith/arXiv arXiv 2024

  3. [4]

    IaC-Eval: A Code Generation Bench- mark for Cloud Infrastructure-as-Code Programs

    arXiv:2310.03714. [Kon et al., 2024] Kon, P. T. J. et al. “IaC-Eval: A Code Generation Bench- mark for Cloud Infrastructure-as-Code Programs. ” NeurIPS 2024 (Datasets & Benchmarks). DOI:10.52202/079017-4273. [Kon et al., 2025] Kon, P. T. J. et al. “EXP-Bench: Can AI Conduct AI Research Experiments?” arXiv:2505.24785. [Lewis et al., 2020] Lewis, P. et al. ...

    Pith/arXiv arXiv 2024

  4. [7]

    Is Self-Repair a Silver Bullet for Code Generation?

    [Olausson et al., 2023] Olausson, T. X. et al. “Is Self-Repair a Silver Bullet for Code Generation?” arXiv:2306.09896. [Opsahl-Ong et al., 2024] Opsahl-Ong, K. et al. “Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. ” arXiv:2406.11695. [Yao et al., 2023] Yao, S. et al. “ReAct: Synergizing Reasoning and Acting in Languag...

    Pith/arXiv arXiv 2023

  5. [8]

    Deployability-Centric Infrastructure-as- Code Generation

    arXiv:2210.03629. [Zhang et al., 2025] Zhang, T. et al. “Deployability-Centric Infrastructure-as- Code Generation. ” FSE

    Pith/arXiv arXiv 2025

  6. [2020]

    Self-Refine: Iterative Refinement with Self-Feedback

    arXiv:2005.11401. [Madaan et al., 2023] Madaan, A. et al. “Self-Refine: Iterative Refinement with Self-Feedback. ” NeurIPS

    Pith/arXiv arXiv 2005

  7. [2023]

    Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning

    arXiv:2303.17651. [Nashid et al., 2023] Nashid, N. et al. “Retrieval-Based Prompt Selection for Code-Related Few-Shot Learning. ” ICSE

    Pith/arXiv arXiv 2023

  8. [2024]

    Evaluating Large Language Models Trained on Code

    https://modelcontextprotocol.io [Chen et al., 2021] Chen, M. et al. “Evaluating Large Language Models Trained on Code. ” arXiv:2107.03374. [HashiCorp, 2024] HashiCorp. “Terraform Documentation. ” https://developer.hashicorp.com/terraform/docs [Jeong et al., 2024] Jeong, S. et al. “Adaptive-RAG: Learning to Adapt Retrieval- Augmented Large Language Models ...

    Pith/arXiv arXiv 2021

  9. [2025]

    Generate Ter- raform HCL

    arXiv:2506.05623. 23 A Dataset Composition Dimension Bucket Tasks Share Difficulty L1 41 22.0% L2 43 23.1% L3 52 28.0% L4 22 11.8% L5 11 5.9% L6 17 9.1% Resource count 1 84 45.2% 2–3 51 27.4% 4–6 45 24.2% 7+ 6 3.2% B GEPA Evolved Instruction (Excerpt) The GEPA-optimized instruction evolved from a 3-word default (“Generate Ter- raform HCL”) to a 40+ line do...

    arXiv