{"id":"f41ae95e-e66f-4e4b-96a0-c051c3233631","arxiv_id":"2501.14131","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using 50-shot in-context learning with GPT-4o, automated Dockerfile refactoring reduces image size by 32% on average and outperforms manual refactoring and smell-repair tools.","lead":"The paper tests whether large language models can automatically improve Dockerfiles, the files that define how software containers are built. It finds that a 50-example prompt strategy shrinks container images by an average of 32% and cuts build time by 6%, beating both human refactoring and existing smell-fixing tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Behavior-preservation definition excludes base-image and dependency changes; the paper's own example shows a slim-base swap that can affect runtime, so the 'refactoring' claim is not yet established.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern here: the study treats functional behavior as determined only by COPY/ADD and ENTRYPOINT/CMD, omitting base images and installed dependencies. This assumption is used both to filter the dataset and to validate LLM refactorings, so if it is too narrow, every downstream quality claim is suspect. The paper provides independent strengths: a replication package, a manual evaluation by six developers with a reported kappa of 0.84, and a plausible demonstration that ICL can reduce image size. The single most important threat is not statistical noise or hyperparameter choice; it is whether the measured improvements are genuinely behavior-preserving refactorings. The paper itself contains the evidence that the definition is too narrow: its illustrative example advertises a switch to a slim base image, and RQ4 shows base-image changes causing dependency-related build failures. A concrete audit of the 50-shot outputs—diffs plus runtime smoke tests—would settle whether the concern lands. If behavior changes are rare, the central claim survives; if not, the headline metrics conflate refactoring with modification. The reader's conditional verdict already places the burden on clarifying behavior preservation, and my analysis confirms that condition rather than moving the verdict.","tokens_in":18131,"tokens_out":6214,"duration_ms":56084,"concrete_test":"Sample 30 of the 128 successful 50-shot refactorings. For each, diff the pre- and post-refactoring Dockerfiles for changes to FROM lines, base-image tags, and RUN instructions that install/remove packages (apt, npm, pip). Build both images and run the container under the same ENTRYPOINT/CMD plus a lightweight smoke check (e.g., invoke the app's default command and verify expected output or a health endpoint). If any sampled case with a base-image/dependency change shows divergent runtime behavior, the behavior-preservation assumption is violated and the quality metrics must be recomputed on strictly behavior-preserving outputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GPT-4o with ICL performs behavior-preserving refactoring that improves quality. This rests on the functional-behavior definition in Section III-B, which counts only COPY/ADD and ENTRYPOINT/CMD as behavior-affecting, explicitly excluding FROM base images and RUN-installed packages. The RQ2 behavior assessment, which excluded only 5 cases across all settings, uses this same narrow notion. Yet the case study in Figure 7 shows the LLM changing node:9.11 to node:9.11-slim as an 'Update Image Tag' refactoring; slim images commonly lack shared libraries and change runtime behavior. RQ4 itself attributes dependency errors and missing-base-image failures to base-image and package-installation changes, so the authors' own analysis confirms these elements affect builds. If base-image or dependency changes are behavior-affecting, the headline 32% average image-size reduction and 91%/77% maintainability/understandability gains may be achieved by behavior-changing modifications rather than safe refactorings. The 2x-vs-developer and 10x-vs-PARFUM comparisons compare these modified images, so the core contribution is not yet supported as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies Dockerfile refactoring as practiced by developers and proposes an LLM-based automation approach using GPT-4o with in-context learning (ICL). The dataset consists of 600 Dockerfile revision pairs mined from 358 open-source projects, split into 398 training and 202 test revisions. RQ1 analyzes how image size and build duration evolve over project lifecycles and when developers tend to refactor. RQ2 evaluates zero-shot, one-shot, and few-shot (20/30/50 demonstration) prompting, using a score-based demonstration-selection strategy defined in Eq. (1). RQ3 compares the best automated refactoring (50-shot) with the developers' own refactorings and with the PARFUM smell-repair tool. RQ4 analyzes build failure causes via qualitative inspection. The headline results are a 63% build success rate for 50-shot, a 32% average image-size reduction (322 MB), a 6% build-duration reduction, and understandability/maintainability improvements in 77%/91% of cases; the paper also claims a 2x image-size reduction over manual refactoring and a 10x reduction over PARFUM.","tokens_in":18272,"tokens_out":6451,"duration_ms":54538,"significance":"If the results are valid, this is a substantial contribution to the emerging area of LLM-based infrastructure-as-code maintenance. The study is the first large-scale evaluation of automated Dockerfile refactoring with ICL, and it includes an original demonstration-selection score, a manual evaluation by six industry developers with a Cohen's kappa of 0.84, a comparison against both human developers and an existing smell-fixing tool, and a qualitative analysis of build failures. The replication package is named, and the authors provide detailed figures. However, the validity of the headline claims depends on a behavior-preservation definition that is too narrow, and the automated-vs-manual comparison is computed on a highly selected subset. These issues are load-bearing and would need to be addressed before the central contribution can be accepted.","major_comments":[{"comment":"The paper defines functional behavior as covering only COPY/ADD and ENTRYPOINT/CMD, explicitly excluding the base image (FROM) and installed packages (RUN). This definition is used both to filter the dataset and to validate LLM refactorings in RQ2. The case study in Figure 7 shows an 'Update Image Tag' refactoring from node:9.11 to node:9.11-slim, and the paper's own RQ4 analysis attributes build failures to base-image changes and dependency errors after refactoring. Since slim images commonly omit runtime libraries and packages can alter application behavior, the refactorings measured in the headline results (32% image-size reduction, 91%/77% maintainability/understandability improvements) are not demonstrated to be behavior-preserving. The authors should broaden the behavior check to include base-image and dependency changes, or clearly scope the claims to the narrow definition and justify why that scope is meaningful for refactoring.","section":"Section III-B and Sections IV-B/IV-C"},{"comment":"The automated-vs-manual comparison is performed on only the 66 Dockerfiles that built successfully under both the 50-shot LLM and the developer, which is 33% of the 202 test files. The abstract's '2x' claim is drawn from medians over this nonrandom subset, while the build-success-rate comparison in Table I uses the full 202-file set. The two analyses therefore address different populations, and the comparison does not account for the trade-off between success rate and quality gains. Please report results on the full test set with an explicit treatment of build failures (for example, as lower bounds or via imputation), and justify the subset analysis as a separate robustness check.","section":"Section IV-C, Table I, Figure 6"},{"comment":"No statistical significance tests are reported for any comparison: between the LLM and developers, between the LLM and PARFUM, or across prompting settings. The improvement/deterioration rates and average reductions in Table I and Figure 6 are presented as point estimates, and the Spearman correlations in Section IV-B are given without p-values or confidence intervals. Given that the central claim is that automated refactoring outperforms manual refactoring, the authors should add appropriate significance tests (for example, paired tests on the 66-case subset, or bootstrap confidence intervals) and report effect sizes.","section":"Section IV-C and Section IV-B"},{"comment":"The demonstration-selection score assigns equal hand-picked weights (0.2 each) to five components, including the same metrics that are later used for evaluation. Because the test set is held out, this is not circular for the evaluation itself, but the paper provides no ablation or sensitivity analysis isolating the effect of the score-based selection. Without a comparison to random demonstration selection or to a simpler textual-similarity-only baseline, the contribution of the scoring function to the improved few-shot results is not established. A short ablation would be sufficient to support the claim that the selection strategy matters beyond the number of demonstrations.","section":"Section III-D, Eq. (1)"}],"minor_comments":[{"comment":"The definition of functional behavior contains the typo 'ENTERPOINT/CMD'; this should read 'ENTRYPOINT/CMD'.","section":"Section III-B"},{"comment":"The developer build success rate is listed as 94/202 in Table I but as 92/202 in the RQ3 text; these numbers should be reconciled.","section":"Table I and Section IV-C"},{"comment":"For the 50-shot row, an average image-size reduction of 322 MB over 128 successful builds implies a total reduction of roughly 41 GB, not the reported 35 GB; please clarify whether the total is computed over the 128 builds or over a different denominator, such as only the improved cases.","section":"Table I"},{"comment":"The criteria for removing the three outliers with 300-500% image-size increases are not specified; state the exact exclusion rule and report whether the RQ1 findings are sensitive to this removal.","section":"Section IV-A"},{"comment":"Figure 1 is very dense and the text is difficult to read; a higher-resolution version and a clearer separation of the four RQ pipelines would improve readability.","section":"Figure 1"},{"comment":"Reference [38] contains a placeholder URL (https://arxiv.org/abs/XXXX.XXXXX%7D%7D) and reference [24] lists the author as 'iwei Xu' (likely 'Yiwei Xu'); both entries should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The topic is timely and the empirical setup is ambitious, with a large dataset, a replication package, and a multi-perspective evaluation. The main reasons for not accepting in present form are (1) the behavior-preservation definition in Section III-B is too narrow relative to the paper's own evidence about base-image and dependency effects, and (2) the RQ3 comparison on 66 of 202 files undercuts the headline superiority claim. Both are fixable within the manuscript's scope, though the behavior-preservation fix may require re-running parts of the evaluation with a broader check. I would not reject the paper outright; a major revision with additional significance testing and a re-analysis of the comparison set would bring the contribution to an acceptable level."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper: it is the first to automate Dockerfile refactoring with in-context learning, and it has a real validity problem in how it defines \"behavior-preserving.\"\n\nThe core idea is straightforward and sensible: use GPT-4o with a score-based demonstration retriever that blends textual similarity with quality-metric improvements, then show that 50-shot ICL beats manual refactoring and smell-fixing tools on image size and other metrics. The dataset is substantial (600 refactored Dockerfile pairs from 358 projects), the evaluation is mostly transparent, and they ship a replication package. They also report build failure rates honestly, which is refreshing.\n\nThe biggest issue is the functional-behavior definition in Section III-B. They treat only COPY/ADD and ENTRYPOINT/CMD as behavior-affecting, explicitly excluding FROM base images and RUN-installed dependencies. But their own case study in Figure 7 shows the LLM switching node:9.11 to node:9.11-slim, and RQ4 attributes failures to dependency and base-image changes. That means the measured gains—32% image-size reduction, 10x vs PARFUM—could come from changes that alter runtime behavior, not safe refactorings. This is not a minor nit; it undercuts the central \"behavior-preserving refactoring\" claim.\n\nSecond, there are no significance tests or confidence intervals anywhere. With only 66 files in the RQ3 intersection, the head-to-head comparisons could easily be noise. Third, the demonstration-selection score in Eq. (1) uses the same metrics (image size, build duration, understandability, maintainability) that the paper later uses for evaluation. The test set is held out, so it is not fitted prediction, but the hand-picked equal weights and the outlier-removal threshold need sensitivity analysis.\n\nWho is this for? People working on IaC quality, LLM-based refactoring, and empirical software engineering. The paper is a serious empirical study with a novel contribution, but it needs major revision: tighten the behavior-preservation definition (either include base-image/dependency changes or show they do not affect the application), add statistical tests, and justify the hand-chosen settings. I would send it to peer review, but with the expectation of substantial work.","headline":"A serious first study on automating Dockerfile refactoring with ICL, but a too-narrow behavior-preservation definition weakens the core claim.","tokens_in":18908,"tokens_out":2422,"would_cite":true,"duration_ms":21098,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated Dockerfile refactoring cuts image size 32%","keywords":["Dockerfile refactoring","in-context learning","large language models","image size reduction","technical debt","infrastructure as code","GPT-4o","build duration"],"falsifier":"Take one of the LLM-refactored Dockerfiles that switched base images (for example, node:9.11 to node:9.11-slim) and run the application's own test suite under both the original and refactored images; if the tests pass differently or the application fails at runtime, the behavior-preservation filter missed functional changes, which would undermine the paper's claim that the quality gains come with unchanged behavior.","tokens_in":17821,"feed_emoji":"🐳","tokens_out":7351,"duration_ms":56338,"temperature":0.7,"pith_summary":"This paper claims that Dockerfile refactoring—a task developers usually put off until a project's middle or late stages—can be automated effectively with a large language model using in-context learning. The authors analyze 600 refactored Dockerfiles from 358 open-source projects and find that image size and build duration grow as projects age, while refactoring activity clusters late. Their method feeds the model up to 50 refactoring examples selected by a score that mixes textual similarity with measured quality gains. In the best configuration, it reduced image size by 32% on average (322 MB per build, 35 GB total), cut build duration by 6%, and improved maintainability and understandability in 91% and 77% of cases, respectively. They also report that automated refactoring reduced image size twice as much as developers' manual refactoring and about ten times as much as the smell-fixing tool PARFUM.","feed_headline":"Automated Dockerfile refactoring cuts image size 32%","feed_subtitle":"LLM in-context learning with scored examples beats manual refactoring and smell-fix tools in a 600-Dockerfile study.","key_machinery":"The load-bearing mechanism is a score-based demonstration selection strategy for in-context learning. For a test Dockerfile, each candidate demonstration is scored as the weighted sum of five components—textual similarity by BM-25, understandability gain, maintainability gain, image-size reduction, and build-duration reduction—and the top-ranked examples are inserted into the prompt ordered from highest to lowest score, closest to the query. The prompt itself is the template $P = \\{N + RD + V_{Before\\_Test}\\}$, where $N$ is the task description with definitions of all refactoring actions, $RD$ is the chosen demonstrations, and $V_{Before\\_Test}$ is the input Dockerfile. This selection is what lets the model see the most relevant and most improved examples before generating its own refactoring; the paper finds that the number of demonstrations is a key driver of improvement, rising from a 38% build success rate at zero shots to 63% at 50 shots.","core_discovery":"The paper's central claim is that in-context learning with a score-based demonstration selection strategy lets a single LLM (GPT-4o) perform refactorings that reduce technical debt across an entire Dockerfile, not just inside RUN instructions. With 50 demonstrations, 63% of the generated Dockerfiles built successfully, successful builds shrank by an average of 32% (322 MB), and build duration fell by 6%, with 77% of refactorings improving understandability and 91% improving maintainability. Comparing only the 66 Dockerfiles that both methods built successfully, the LLM brought median image size from 599 MB to 95 MB (an 85% reduction) while developers only reached 340 MB (43%), and the LLM improved maintainability and understandability in a larger share of cases. The authors conclude that automated Dockerfile refactoring outperforms both manual refactoring and smell-repair tools, and that it could serve as a new standard step in CI/CD pipelines.","pith_inferences":["Inference: Because the paper only checks behavior preservation for COPY/ADD and ENTRYPOINT/CMD, a production deployment of this automation should add a runtime validation step—running the application's tests after refactoring—to catch functional changes the paper's filter would miss.","Inference: The score's equal weighting (0.2 per component) is a design choice the paper does not test; reweighting toward image size or build duration could yield a configuration that improves both, since the observed negative correlations sit at the demonstration-selection level.","Inference: The paper measures developer refactorings as they actually occurred, not as an idealized expert baseline; a controlled study with experts given the same task would clarify whether the 2x gap is due to the LLM's strategy or to developers' lack of time or incentive.","Inference: The finding that build context errors caused 52% of LLM refactoring failures suggests a cheap extension: a pre-check that validates COPY/ADD paths against the declared build context before the refactoring is proposed, which the paper discusses only as future work."],"forward_implications":["CI/CD pipelines could adopt the 50-shot scoring recipe as an automated refactoring step, catching image bloat and maintainability loss before they accumulate.","Teams using this automation would face a measurable trade-off: smaller images tend to come with longer build durations (-0.60 correlation), so the pipeline would need to target one objective.","Developers' habit of refactoring late—after image size and build duration have already grown—means automation could be most valuable early, before the debt builds.","The comparison against PARFUM suggests that whole-file refactoring (restructuring stages and images) is a more powerful lever than smell repair inside RUN commands."],"supporting_citations":[{"why":"Supplies the refactoring detector (DRMiner) used to identify refactoring actions in commit history and populate the pre/post refactoring dataset.","marker":"[28]"},{"why":"PARFUM is the automated smell-repair baseline the paper compares against in RQ3, providing the image-size reduction benchmark.","marker":"[6]"},{"why":"Motivates instance-specific demonstration selection for code few-shot learning, grounding the paper's retrieval-based prompt strategy.","marker":"[17]"},{"why":"Provides ordering findings for in-context demonstrations (highest score closest to query) and supports the general ICL design.","marker":"[36]"},{"why":"Defines the technical debt categories and the ten refactoring actions used in the prompt template and in the score's quality components.","marker":"[8]"},{"why":"BM-25 is the textual-similarity function used in the first term of the demonstration-selection score.","marker":"[45]"},{"why":"Google BigQuery GitHub dataset is the source of Dockerfile commit histories from which refactoring pairs were extracted.","marker":"[42]"}],"fun_headline_variants":["AI refactoring shrinks Docker images by 32% in study","LLM beats manual Docker refactoring: 2x smaller images","In-context learning cuts Dockerfile image size 32%","Automated Dockerfile refactoring: 32% smaller images","Study: LLM refactoring outperforms humans on Dockerfiles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a Dockerfile's functional behavior is fully determined by the files it copies in (COPY/ADD) and the startup command (ENTRYPOINT/CMD), so refactorings that change the base image or installed packages are counted as behavior-preserving even though they can change runtime behavior.","fun_headline_variants_meta":{"raw":{"variants":["AI refactoring shrinks Docker images by 32% in study","LLM beats manual Docker refactoring: 2x smaller images","In-context learning cuts Dockerfile image size 32%","Automated Dockerfile refactoring: 32% smaller images","Study: LLM refactoring outperforms humans on Dockerfiles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1301,"prompt_tokens":1001,"completion_tokens":300,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":208}},"tokens_in":617,"tokens_out":300,"duration_ms":2855,"temperature":1.0,"reasoning_tokens":208,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:20:59.293956+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one of the LLM-refactored Dockerfiles that switched base images (for example, node:9.11 to node:9.11-slim) and run the application's own test suite under both the original and refactored images; if the tests pass differently or the application fails at runtime, the behavior-preservation filter missed functional changes, which would undermine the paper's claim that the quality gains come with unchanged behavior.","supporting_citations":[{"cited_title":"Dr- miner: A tool for identifying and analyzing refactorings in dockerfile,","cited_arxiv_id":null,"evidence_quote":"Supplies the refactoring detector (DRMiner) used to identify refactoring actions in commit history and populate the pre/post refactoring dataset."},{"cited_title":"Retrieval-based prompt selection for code-related few-shot learning,","cited_arxiv_id":null,"evidence_quote":"Motivates instance-specific demonstration selection for code few-shot learning, grounding the paper's retrieval-based prompt strategy."},{"cited_title":"What makes good in-context demonstrations for code intelligence tasks with llms?","cited_arxiv_id":null,"evidence_quote":"Provides ordering findings for in-context demonstrations (highest score closest to query) and supports the general ICL design."},{"cited_title":"Refactorings and technical debt for docker projects,","cited_arxiv_id":null,"evidence_quote":"Defines the technical debt categories and the ten refactoring actions used in the prompt template and in the score's quality components."},{"cited_title":"Okapi at trec-3,","cited_arxiv_id":null,"evidence_quote":"BM-25 is the textual-similarity function used in the first term of the demonstration-selection score."},{"cited_title":"Bigquery github database,","cited_arxiv_id":null,"evidence_quote":"Google BigQuery GitHub dataset is the source of Dockerfile commit histories from which refactoring pairs were extracted."}],"review_version":1}