{"id":"efc8f9a2-f43d-4e30-815c-2f2432339d07","arxiv_id":"2501.07676","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A seven-smell taxonomy for sustainability issues in Terraform IaC is proposed and its prevalence is measured across thousands of open-source scripts.","lead":"This paper defines seven \"sustainability smells\" in Terraform infrastructure-as-code scripts, such as over-provisioning and monolithic design, and measures their frequency across 28,327 open-source scripts. It validates the smells with a 19-practitioner survey and finds monolithic infrastructure in nearly 10% of scripts, giving cloud teams a starting point for more efficient infrastructure.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported prevalence rates rest on an unquantified regex detector: the paper claims 100% accuracy without reporting validation sample sizes, recall, or the actual patterns, and the SS7 heuristic (the headline 9.67% finding) is undefined; this makes the central RQ3 numbers unsupported.","rationale":"The reader's verdict is CONDITIONAL, and the reader's rationale already flags 'unquantified regex accuracy' and the '395 vs 812' dataset discrepancy as threats. I agree these are real, but I would rank detector validity as the single most load-bearing concern rather than the best-practice-proxy assumption. The paper itself concedes in §7.2 that 'removing these smells may not necessarily result in significant reductions in energy consumption or cost'—so the authors are careful not to claim measured energy harm. The novel, falsifiable contribution is the RQ3 prevalence numbers, and those numbers stand or fall with the regex detector's precision and threshold choices. The paper reports a '100% accuracy' figure without any supporting sample size, and it never defines the SS7 rule that produces the headline 9.67%. Because the actual regex patterns are not in the manuscript, a reader cannot check whether the detection is circular (e.g., matching a comment like '# Overprovisioned' rather than the resource configuration) or arbitrary (e.g., a resource-count threshold). The dataset inconsistency between the abstract (395) and §3.5 (812) compounds the problem by making the denominator unclear. If a replication check shows the detector has materially lower precision or that SS7's prevalence is threshold-sensitive, the central empirical claim weakens; if the check reproduces the numbers and shows robust thresholds, the paper's contribution stands as a useful initial taxonomy. This is exactly the conditionality the reader prescribed, so the verdict remains CONDITIONAL/UNCHANGED.","tokens_in":14005,"tokens_out":3800,"duration_ms":38267,"concrete_test":"Obtain the replication package; extract the exact regex patterns and thresholds for all seven smells. Independently run them on the stated 26,467 scripts and compare counts. Then draw a stratified random sample of 100 detections per smell (700 total) plus 100 non-detected scripts per smell, and have two raters manually label ground truth. Report precision, recall, and F1. If SS7's pattern relies on an undocumented 'many resources' cutoff, test sensitivity by varying that cutoff (e.g., 5, 10, 20 resource blocks) and recompute the 9.67% prevalence; if the prevalence shifts by more than a factor of 2, the headline number is not robust. Also reconcile the 395 vs 812 repository counts from the replication package's repository list.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—the RQ3 prevalence figures, especially 'Monolithic Infrastructure appears in 9.67% of scripts'—depends entirely on the regex-based detector described in §3.6. The paper states 'Our regex-based method achieved 100% accuracy based on the random sampling manual validation' but reports neither the number of manually reviewed detections per smell, the composition of that sample, nor recall (false negatives are never estimated). This matters because the detection rules are not shown in the paper: for SS7 (Monolithic Infrastructure) the code sample suggests a 'many resources follow' heuristic, but no threshold (e.g., number of resource blocks per file) is stated; for SS1 (Over-Provisioning) the only code example is a comment '# Overprovisioned', which a regex cannot detect without a list of instance types. A small change in an undocumented threshold could move the 9.67% figure substantially. The abstract's '395 repositories' vs §3.5's '812 distinct GitHub repositories' (401+137+274) further makes the denominator ambiguous. Thus the headline prevalence numbers are not currently supported by a reproducible, validated measurement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines seven 'sustainability smells' for Terraform Infrastructure-as-Code scripts, grounded in a grounded-theory analysis of AWS, Azure, and GCP sustainability best practices. It validates the smells through a 19-participant practitioner survey and measures their prevalence in a dataset of 28,327 Terraform scripts using regex-based detection. The headline result is that Monolithic Infrastructure (SS7) appears in 9.67% of scripts, followed by State Management (SS6) at 1.59% and Lack of Auto-Scaling (SS2) at 0.73%. The paper also clusters the smells into three categories and derives practical recommendations.","tokens_in":14315,"tokens_out":4275,"duration_ms":39469,"significance":"If the results hold, the paper provides a useful initial taxonomy of sustainability-related anti-patterns in Terraform and the first large-scale prevalence estimates for such patterns. Strengths include the grounding in external cloud-provider best practices, the use of two independent raters with perfect inter-rater agreement, a replication package, and a survey that gives some practitioner perspective. The contribution is weakened, however, by the lack of any direct energy or cost measurement, an unquantified regex detector, and an internal inconsistency in the reported dataset size. The prevalence numbers are therefore best interpreted as preliminary and would need to be substantiated before the paper's central empirical claims can be accepted.","major_comments":[{"comment":"The dataset size is internally inconsistent: the abstract reports 395 open-source repositories, while Section 3.5 reports 812 distinct GitHub repositories (401 AWS, 137 Azure, 274 GCP). Because the repository count is used as the sampling strata and the denominator for prevalence claims, this discrepancy must be resolved and the correct number stated explicitly in both the abstract and Section 3.5.","section":"Abstract, §3.5"},{"comment":"The prevalence figures in Section 4.3 rest on an unquantified regex detector. The paper states that the regex-based method achieved 100% accuracy based on random sampling manual validation, but it reports neither the number of manually reviewed detections per smell nor any recall estimate, and the regex patterns are not included in the manuscript. For SS7 (Monolithic Infrastructure), Table 2 shows only a comment '# Many more resources follow ...' with no explicit rule or threshold. As a result, the headline 9.67% prevalence for SS7 is not reproducible or verifiable from the information given; the detection rules and validation sample sizes must be reported.","section":"§3.6, §4.3"},{"comment":"The detection patterns were iteratively refined on a subset of the same dataset that is later used to measure prevalence, and the manual validation is drawn from the scripts already flagged by those patterns. This procedure can overfit to the dataset and inflate precision; the manuscript does not report the size of the refinement subset, the number of validation iterations, or how the random validation sample was constructed. These details are needed to assess the uncertainty of the RQ3 numbers.","section":"§3.6"},{"comment":"The central premise of the smell definition is that each cloud-provider best practice has a corresponding anti-pattern whose presence indicates a sustainability problem. The paper does not measure energy consumption, cost, or carbon impact; its support is practitioner perception from a 19-participant survey. Section 7.2 explicitly concedes that removing these smells 'may not necessarily result in significant reductions in energy consumption or cost.' This limitation is load-bearing for the claim that the identified patterns are sustainability smells and should be stated in the abstract and conclusion, not only in threats to validity.","section":"§3.2, §7.2"}],"minor_comments":[{"comment":"The code samples in Table 2 contain character-spacing artifacts (e.g., 're so ur ce' instead of 'resource') that make them difficult to read; please use a monospaced font with proper ASCII text.","section":"Table 2"},{"comment":"Figure 5 presents prevalence bars without confidence intervals or raw counts; even after the detector is specified, adding the counts and, where possible, confidence intervals would help interpret the rates.","section":"Figure 5"},{"comment":"The survey had only 19 participants and the paper reports percentages such as 94.7% (SS1) without absolute counts; describing the exact recruitment procedure and the number of responses per question would clarify the generalizability of the perception findings.","section":"§3.4"},{"comment":"The claim that no IaC-specific sustainability standards exist is supported by cloud-provider documents, but the text would benefit from citing recent academic work on IaC sustainability, if any, to better position the contribution.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits an empirical software engineering venue and the replication package is a strength. The main revision challenge is to document the detector and resolve the dataset-count inconsistency; if the authors can do that, the paper could be suitable. The novelty relative to prior security-smell work is moderate but sufficient for a specialized venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper gives the IaC community something it didn't have: a concrete list of seven sustainability smells for Terraform, derived systematically from AWS/Azure/GCP best practices via grounded theory, with a practitioner survey and a large-scale prevalence measurement. That's a real contribution. The taxonomy itself is plausible and the categories (general, demand, application) are sensible. The replication package is a plus, and the two-rater validation with Kappa 1.0 is decent evidence that the smells map to the stated best practices.\n\nThe soft spots are mostly in the measurement. The RQ3 prevalence figures, especially the headline 9.67% for Monolithic Infrastructure, rest on a regex-based detector that is not adequately documented. The paper says it achieved 100% accuracy but doesn't report the validation sample size, the breakdown per smell, or recall at all. More importantly, the detection rules themselves aren't shown; SS7's \"many resources follow\" isn't a defined threshold, and SS1's regex can't detect over-provisioning without a list of instance types. Small changes in those thresholds could move the numbers substantially. There's also an internal inconsistency: the abstract says 395 repositories, while Section 3.5 says 812 distinct repositories (401+137+274). That's not a minor typo—it changes the denominator for prevalence.\n\nThe survey is small (19 participants) but for validating perception that's acceptable. The central assumption—that deviating from provider best practices actually causes inefficiency—is not directly measured, but the paper is honest about this in the threats to validity section. So I wouldn't call that a fatal flaw; it's a scope limitation.\n\nOverall, the taxonomy and qualitative analysis are likely to hold up, but the quantitative prevalence claims should be treated as rough until the detector is made reproducible. The paper deserves a serious referee—it's a legitimate first step that should go through review with major revision. I'd ask the authors to resolve the dataset discrepancy, publish the exact detection rules and thresholds, and report precision/recall with confidence intervals.\n\nWho's it for? Researchers in IaC and sustainable cloud computing, and practitioners who want a concrete audit checklist. I'd cite the taxonomy, but I'd be careful about quoting the prevalence numbers.","headline":"A useful first taxonomy of sustainability smells in Terraform, but the headline prevalence numbers rest on an underspecified regex detector and an inconsistent dataset count.","tokens_in":14798,"tokens_out":2485,"would_cite":true,"duration_ms":23977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper defines seven sustainability smells in Terraform scripts, validates them through a practitioner survey, and measures their prevalence in 28,327 open-source scripts, finding Monolithic Infrastructure in 9.67% of scripts.","keywords":["sustainability smells","Infrastructure as Code","Terraform","anti-patterns","cloud sustainability","empirical software engineering","code smells","resource efficiency"],"falsifier":"Measure, for matched pairs of Terraform configurations that differ only in one smell (for example, auto-scaling enabled versus disabled), the actual cloud bill or energy use over a representative workload; if the smelly configuration is not systematically more expensive or energy-hungry than the clean one, the central claim that these patterns are sustainability problems would collapse.","tokens_in":13808,"feed_emoji":"♻️","tokens_out":5743,"duration_ms":52223,"temperature":0.7,"pith_summary":"The paper tries to establish that Infrastructure as Code scripts, specifically Terraform, contain recurring anti-patterns that undermine cloud sustainability, and that these anti-patterns can be named, recognized, and counted. It extracts sustainability best practices from AWS, Azure, and GCP guidance, converts each into an anti-pattern that it calls a sustainability smell, confirms with 19 practitioners that the patterns count as bad practice, and measures how often seven defined smells appear in a large corpus. The headline empirical result is that the most prevalent smell, Monolithic Infrastructure, appears in 9.67% of scripts, while the other six are far rarer. A sympathetic reader would care because the study turns vague sustainability concerns into concrete, detectable code-level patterns that practitioners and tooling can target.","feed_headline":"Monolithic Infrastructure smell appears in 9.67% of Terraform scripts","feed_subtitle":"A study of 28,327 Terraform scripts names seven anti-patterns and measures how often each appears.","key_machinery":"The central machinery is the sustainability smell concept, transferred from code smells to IaC: a recurring code pattern or deployment choice that violates cloud-provider sustainability guidance. The identification pipeline uses grounded theory on sustainability reports from AWS, Azure, and GCP, producing four attributes, Runtime Dependency, Resource Context, Code Dependency, and Inherent Badness, that categorize the smells. Those attributes feed a binary similarity matrix and hierarchical clustering, which divides the smells into General, Demand, and Application categories. Prevalence is measured with hand-built regex patterns for each smell, refined through iterative testing and manually validated on a random sample with reported 100% accuracy.","core_discovery":"The paper's central discovery is a catalog of seven sustainability smells in Terraform: Over-Provisioning Resources, Lack of Auto-Scaling, Ignoring Resource Lifecycles, Excessive Logging, Unoptimized Data Transfers, State Management, and Monolithic Infrastructure. Each smell is derived by taking a sustainability best practice from AWS, Azure, or GCP and positing the corresponding anti-pattern, that is, a failure to follow the guidance. The study reports that a survey of 19 IaC practitioners widely recognizes most of these patterns as bad practice, with agreement ranging from 47.4% to 94.7%, and that a regex-based scan of 28,327 Terraform scripts finds Monolithic Infrastructure by far the most common at 9.67%, followed by State Management at 1.59% and Lack of Auto-Scaling at 0.73%. The paper concludes that these smells often arise from a combination of script structure, configuration choices, and deployment context, which makes root-cause analysis difficult.","pith_inferences":["Because the study measures pattern frequencies and practitioner opinions rather than energy or cost, the actual sustainability impact of the seven smells remains unquantified; a natural next step is pairing each smell with billing or energy telemetry to test whether removal actually reduces resource use.","The binary similarity matrix treats attribute matches as all-or-nothing, so using graded similarity or continuous features could reveal overlaps among smells that the current three categories hide.","The reported prevalence depends on regex patterns, so smells expressed through varied syntax, such as auto-scaling configured outside the scanned file, may be undercounted; a parser-based detector on the same dataset would be a direct check.","Because the smells were derived only from AWS, Azure, and GCP guidance, applying the same method to other IaC tools like Ansible or to other providers could yield a different or extended catalog."],"forward_implications":["If the catalog is correct, Terraform repositories can be automatically screened for these seven patterns, with Monolithic Infrastructure as the highest-yield target.","The three-category structure implies that mitigation strategies differ: demand smells require workload data, general smells can be addressed through modular design, and application smells require inspecting the code the infrastructure runs.","The very low observed prevalence of Over-Provisioning Resources, 0.01%, suggests either that open-source Terraform scripts rarely over-provision or that the regex detector under-recognizes the pattern, so the reported rates are best treated as lower-bound estimates.","The 9.67% figure for Monolithic Infrastructure gives a concrete baseline that future IaC sustainability tools and studies can compare against.","Because the survey shows weaker consensus for Ignoring Resource Lifecycles and State Management, those two smells are the ones most in need of further evidence before being treated as established sustainability problems."],"supporting_citations":[{"why":"AWS Well-Architected Sustainability Pillar, the primary source of best practices from which several sustainability smells are derived.","marker":"[8]"},{"why":"Google Cloud sustainability framework, used alongside Azure guidance to extract best practices and to justify State Management and Monolithic Infrastructure as smells.","marker":"[19]"},{"why":"Azure Cost Optimization Well-Architected guidance, which supplies practices that become anti-patterns such as Lack of Auto-Scaling.","marker":"[20]"},{"why":"The grounded-theory methodology textbook that structures the qualitative coding of the sustainability reports.","marker":"[46]"},{"why":"The refactoring book that introduces the code-smell concept that the paper transfers to IaC sustainability.","marker":"[10]"},{"why":"Prior work defining security smells in IaC scripts, establishing the anti-pattern catalog approach that this study extends to sustainability.","marker":"[34]"}],"fun_headline_variants":["Terraform scripts hide seven sustainability smells","Monolithic Infrastructure top Terraform sustainability smell","Study: seven sustainability smells found in Terraform code","Most common Terraform smell: Monolithic Infrastructure","Sustainability smells complicate Terraform resource use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a Terraform pattern counts as a sustainability smell when it deviates from a best practice in AWS, Azure, or GCP sustainability guidance, without direct measurement that the deviation actually increases energy use or cost.","fun_headline_variants_meta":{"raw":{"variants":["Terraform scripts hide seven sustainability smells","Monolithic Infrastructure top Terraform sustainability smell","Study: seven sustainability smells found in Terraform code","Most common Terraform smell: Monolithic Infrastructure","Sustainability smells complicate Terraform resource use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000507,"raw_usage":{"total_tokens":2486,"prompt_tokens":976,"completion_tokens":1510,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1442}},"tokens_in":592,"tokens_out":1510,"duration_ms":11652,"temperature":1.0,"reasoning_tokens":1442,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:37:12.653945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, for matched pairs of Terraform configurations that differ only in one smell (for example, auto-scaling enabled versus disabled), the actual cloud bill or energy use over a representative workload; if the smelly configuration is not systematically more expensive or energy-hungry than the clean one, the central claim that these patterns are sustainability problems would collapse.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AWS Well-Architected Sustainability Pillar, the primary source of best practices from which several sustainability smells are derived."},{"cited_title":"Design for environmental sustainability | google cloud,","cited_arxiv_id":null,"evidence_quote":"Google Cloud sustainability framework, used alongside Azure guidance to extract best practices and to justify State Management and Monolithic Infrastructure as smells."},{"cited_title":"Cost Optimization - Microsoft Azure Well-Architected Framework,","cited_arxiv_id":null,"evidence_quote":"Azure Cost Optimization Well-Architected guidance, which supplies practices that become anti-patterns such as Lack of Auto-Scaling."},{"cited_title":"& Strauss, A","cited_arxiv_id":null,"evidence_quote":"The grounded-theory methodology textbook that structures the qualitative coding of the sustainability reports."},{"cited_title":"Fowler, Refactoring: Improving the Design of Existing Code , Addison-Wesley, 1999","cited_arxiv_id":null,"evidence_quote":"The refactoring book that introduces the code-smell concept that the paper transfers to IaC sustainability."},{"cited_title":"The seven sins: Security smells in infrastructure as code scripts,","cited_arxiv_id":null,"evidence_quote":"Prior work defining security smells in IaC scripts, establishing the anti-pattern catalog approach that this study extends to sustainability."}],"review_version":1}