Pith. sign in

REVIEW 3 major objections 4 minor 22 references

Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Pith's one-line read Automated code modernization, not environment backporting, can restore reproducibility for the majority of legacy machine-learning notebooks.

desk verdict Valuable dataset and an important negative result on backporting, but the 74.2% headline is undercut by a scoring cue in the prompt and by inconsistent numbers across the paper. read the letter →

arxiv 2602.07195 v2 pith:TKZ7W37M submitted 2026-02-06 cs.SE cs.LG

classification cs.SEcs.LG
keywords reproducibilitymachinelearningnotebooksLLMagentsprogramrepairenvironmenterosionscorecalibrationKagglecodemodernization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that environmental erosion—rapid changes in ML software and hardware—makes most published ML notebooks non-reproducible, and that the usual remedy of downgrading dependencies to match the original submission time fails. It proposes instead to treat the current environment as fixed and modernize the notebook code itself. The central result is that an LLM-driven agent, MLEModernizer, makes 5,492 of 7,402 (74.2%) previously non-reproducible notebooks reproducible within a 10% score band. This matters because it suggests a scalable, low-cost path to recovering and reusing legacy ML pipelines as ecosystems evolve, rather than trying to reconstruct historical environments. The paper also finds that the hardest errors to fix are attribute errors caused by API misuse, and that runtime reduction rarely succeeds.

What carries the argument

The central object is MLEModernizer, an LLM-driven agentic loop that treats the current execution environment as a fixed constraint and rewrites the notebook code. At each iteration it executes the notebook, gathers error tracebacks, runtime, and a reproduced score, then classifies the failure into one of three fix types: error-repair (when errors occur), runtime-reduction (when execution times out), or score-calibration (when the code runs but the score deviates). The LLM receives the notebook in a cell-delimited format along with the target score, and returns a full updated notebook in a plan-then-patch format. The success criterion is a score deviation within 10% of the reported target, w

What would settle it

Take a random sample of notebooks that MLEModernizer classified as reproducible via score-calibration, then re-execute them on a different held-out split of the same competition data (or on the original private test set if available) and compare both the score deviation and the actual prediction vectors against the pre-calibration outputs. If a substantial fraction fail the 10% band or produce materially different predictions, then the reported reproducibility rate is inflated by target-chasing rather than genuine modernization.

Watch

Extended reading notes

Core claim

The paper establishes that code modernization is a viable alternative to environment reconstruction for restoring reproducibility in ML notebooks. On a dataset of 12,720 Kaggle notebooks from 79 competitions, only 35.4% reproduce in a contemporary environment, and backporting dependencies to submission timestamps actually lowers this to 35.1% while adding failure modes. MLEModernizer, an agentic LLM framework, iteratively executes a notebook, collects execution feedback, and applies three types of fixes—error-repair, runtime-reduction, and score-calibration—until the reproduced score falls within 10% of the reported target. Applied to 7,402 non-reproducible notebooks, it achieves 74.2% repro

Load-bearing premise

The load-bearing assumption is that bringing the reproduced score within 10% of the reported target, with the target given to the LLM, genuinely indicates that the notebook's original behavior has been restored—rather than that the LLM has tuned the code to hit the target score regardless of semantic fidelity.

Editorial extensions

If this is right

  • Environment backporting, the dominant industry response to reproducibility failures, is shown to be ineffective and sometimes harmful; this shifts attention to code-level modernization.
  • An LLM agent can restore reproducibility for roughly three-quarters of legacy non-reproducible notebooks at low cost, suggesting that automated modernization is economically feasible at scale.
  • The three-tier fix taxonomy means that reproducibility can often be achieved even when residual errors remain; about 17% of modernized notebooks produce reproducible scores despite unresolved errors.
  • The stubborn error types are attribute errors rooted in API misuse, indicating that future repair tools should focus on API evolution and migration patterns.
  • The method's dependence on a target score for calibration creates a clear tension between faithful reproduction and score matching, which future evaluations must address.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 74.2% rate may overstate true behavioral fidelity: score-calibration is given the target score as part of the prompt, so an LLM could adjust hyperparameters or post-processing to hit the band without preserving the original model's predictions; a stricter test would compare predictions on a held-out split.
  • The same agentic modernization loop could be applied outside notebooks, for example to legacy data pipelines or scientific scripts, wherever the goal is to run on today's stack while preserving outputs.
  • The score-calibration mechanism, if not bounded, could be exploited as a form of 'score hacking' in benchmark settings; adding a constraint that predictions must remain close to the pre-calibration model's outputs would guard against this.
  • The finding that runtime-reduction fixes rarely succeed suggests that timeout-prone notebooks need a different strategy, such as code-level algorithmic changes or user-assisted resource allocation, rather than LLM patching alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies the reproducibility of legacy machine-learning notebooks from Kaggle by re-executing them in a current Kaggle-like container. It defines reproducibility as a reproduced score sr within 10% of the reported target score st, reports that only 35.4% of 12,720 notebooks (full-text numbers) are reproducible at baseline, and finds that environment backporting does not help (35.1%). It then introduces MLEModernizer, an LLM-driven agent that iteratively executes notebooks and applies error-repair, runtime-reduction, and score-calibration fixes. The full text reports that MLEModernizer makes 5,492 of 7,402 previously non-reproducible notebooks (74.2%) reproducible. The paper concludes that code modernization is a viable alternative to environment reconstruction.

Significance. If the evaluation is valid, the contribution is significant: a large-scale, containerized re-execution study of real MLE notebooks; a practical, cost-reported agentic repair pipeline; a clear negative result for environment backporting; and an open artifact. The authors are careful to separate error-free execution from score-band reproducibility and to report an ablation study. However, the current version has two load-bearing problems: the abstract and full text report irreconcilable dataset sizes, baseline rates, and headline results; and the score-calibration evaluation is circular with the success metric. These issues must be resolved before the central claim can be interpreted.

major comments (3)
  1. [Abstract vs. §1, §2.1, §4.1] The abstract reports 12,106 notebooks from 75 competitions, 26% baseline reproducibility, backporting 'decreased to 12%', 8,210 modernization subjects, and success rates of 3,292 (40.1%, GPT-5.2) and 3,683 (44.9%, GPT-OSS-120b). The full text reports 12,720 notebooks from 79 competitions, 35.4% baseline reproducibility, 35.1% after backporting, 7,402 subjects, and 5,492 (74.2%) with GPT-5.2 only. These are not cosmetic differences: they change the dataset, the backporting conclusion, and the headline result. The manuscript must be harmonized, and the authoritative numbers must be identified explicitly.
  2. [§2.1.2, §3.3, §4.2 (Table 4)] The success criterion (|sr - st|/|st| <= 10%) is the same quantity that MLEModernizer is given as a target and asked to optimize. The input to MLEModernizer includes the target score st (§3.1); the shared prompt contains 'Scores: targets t, currents r' (§3.3); score-calibration is described as aligning the output 'toward the reported Kaggle score' (§4.2, Table 4 caption); and score-calibration accounts for 21,556 entries in Table 4. With no prediction-level guardrail, the 74.2% rate measures the LLM's ability to hit a target score band, not whether the modernized notebook faithfully reproduces the original computation. The running example in §3.4 accepts sr = 0.93778 for st = 0.87511, i.e., a 7.2% relative score difference, as success. Please add a validation that withholds st during score-calibration (e.g., held-out target scores), and/or a prediction-level comparison (e.g., agreement o
  3. [§4.1 vs. §4.2] Table 3 and §4.1 report 5,492 reproducible notebooks (4,208 Error-Free Reproducible + 1,284 Error Reproducible). Section 4.2, Figure 7b, and the RQ2 summary report 5,042 reproducible notebooks for the same cohort of 7,402 notebooks. This 450-notebook discrepancy affects the headline count and the per-fix statistics. It must be reconciled before the results can be interpreted.
minor comments (4)
  1. [Table 4, Runtime-Reduction row] The listed total for Runtime-Reduction is 4,449, but the five outcome columns sum to 5,449 (3,143 + 835 + 1,365 + 29 + 77). The text repeats 4,449. Please correct the total and recompute fix-type shares.
  2. [§3.4] A 7.2% relative score deviation is described as 'a bit higher' than the target. This is close to the 10% cap; please reword and discuss whether the example demonstrates faithful reproduction or merely entry into the score band.
  3. [§2.2.2] Baseline reproducibility is reported as 4,501/12,720 (35.4%) in the text and as '35%' in the summary. Use one consistent decimal representation.
  4. [Throughout] The terms 'environment erosion' and 'environmental erosion' are used inconsistently. Also, 'reproducible' is sometimes used without qualification even though the paper's key distinction is between Error-Free Reproducible and Error Reproducible; define the qualified terms at first use.

Circularity Check

1 steps flagged · score 6.0 of 10

Score-calibration is prompted with the target score and judged by proximity to it, so a component of the 74.2% reproducibility result is self-fulfilling; the 5,492 vs 5,042 count inconsistency is a separate correctness issue.

  1. self definitional [§2.1.2 (reproducibility criterion), §3.3 (shared prompt), Table 4 (Score-Calibration definition)]
    "a notebook is reproducible if its reproduced score deviation from the target score is within a threshold ... δs = |sr−st|/|st| ≤ τ ... Scores: targets t, currents r, and the metric directionality (higher/lower is better) ... Score-Calibration when the script has no errors and does not need runtime reduction but its output must be aligned toward the reported Kaggle score."

    The reproducibility predicate is defined as δs ≤ 10%. The score-calibration prompt hands the LLM the exact target st and current sr and instructs it to align the output toward st. A notebook is then declared reproducible precisely when δs ≤ 10%, i.e., when the same quantity the fix was asked to minimize is small. No prediction-level check (e.g., prediction agreement or model equivalence) is reported, and the running example treats 0.93778 vs 0.87511 as success. Thus the score-calibration component of the 74.2% result measures the LLM's ability to hit the target it was given, not faithful restoration of the original computation; this part of the derivation reduces to the evaluation definition by construction.

full rationale

The central circularity is localized to score-calibration: the target score is both an input to the fix prompt and the reference point of the success predicate, making those successes partly self-fulfilling. The evaluation is not entirely circular, because error-repair and runtime-reduction fixes are judged by execution success and runtime, and the baseline/backporting numbers are anchored to the external MLE-Bench grader. No load-bearing self-citation or imported uniqueness theorem was found; the one self-citation (Jin et al. [10]) is contextual and not load-bearing. Separately, the manuscript contains an unresolved internal inconsistency: §4.1/Table 3 reports 5,492 reproducible notebooks, while §4.2/Figure 7b and its summary report 5,042. That is a correctness/verification concern rather than circularity, but it must be reconciled before the headline count is interpretable. Overall: partial circularity, score 6.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The main free parameters are threshold choices that define the dependent variable and the subject pool. The central unstated assumption is that the LLM's score-calibration edits do not amount to fitting the target score. No new physical or conceptual entities are introduced beyond the MLEModernizer software system itself.

free parameters (4)
  • Reproducibility threshold τ = 0.10 (10%)
    Set empirically in §2.1.2 and used in all evaluations; the headline success rate is directly sensitive to this hand-chosen band. No sensitivity analysis is reported.
  • Maximum fix iterations N = 16
    Termination cap in §3.1; notebooks reaching the cap are classified as failed or non-reproducible, so this choice affects the reported success rate.
  • Wall-clock timeout = 600 seconds
    Used to filter the dataset (§2.1.1) and to terminate execution (§3.2); long-running notebooks are excluded from the study and evaluation.
  • Token cutoff for subject pool = 13,485 tokens (95th percentile)
    Chosen in §4 to exclude 472 of 7,874 non-reproducible notebooks; changes the evaluated pool and the 74.2% success rate.
assumptions (5)
  • domain assumption MLE-Bench's offline grader with its own train/test split approximates the Kaggle private leaderboard evaluation.
    Stated in §2.1.2 and §5.2; the paper admits the split may differ from Kaggle's, so the 10% band is meant to absorb this mismatch. If the mismatch is larger than 10% for a given competition, the reproducibility labels change.
  • domain assumption The reported Kaggle score st is a valid target for the offline grader's score sr.
    Used in the definition of δs in §2.1.2; if st is not comparable to offline scores (different test data, metric details), the binary reproducible/non-reproducible classification is unreliable.
  • domain assumption The contemporary Kaggle Docker image plus MLE-Bench data is a representative 'contemporary environment'.
    The paper uses one Kaggle-based environment for all experiments (§2.1.2); results may not generalize to other hardware/software stacks.
  • domain assumption LLM code edits preserve the semantic intent of the original pipeline except for repairing breakage.
    The prompt in §3.3 gives the LLM the target score and asks for score calibration; nothing prevents it from altering the model or predictions to hit the target, which would break the 'restore reproducibility' interpretation.
  • domain assumption Error-tolerant execution, continuing past failing cells, yields a meaningful execution outcome.
    Adopted in §2.1.2; some errors are treated as ignorable, and notebooks with residual errors are counted as reproducible if they produce a CSV within the score band.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility." pith.science (2026). https://pith.science/paper/TKZ7W37M

@misc{pith2026260207195,
  author       = {Pith},
  title        = {Pith review of: Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKZ7W37M}},
  note         = {Machine review of arXiv:2602.07195}
}
read the original abstract

Interactive computational notebooks (e.g., Jupyter notebooks) are widely used in machine learning engineering (MLE) to program and share end-to-end pipelines, from data preparation to model training and evaluation. However, environmental erosion-the rapid evolution of hardware and software ecosystems for machine learning-has rendered many published MLE notebooks non-reproducible in contemporary environments, hindering code reuse and scientific progress. To quantify this gap, we study 12,106 notebooks selected from 75 popular Kaggle competitions: only 26% remain reproducible today. Crucially, we find that environment backporting, i.e., downgrading dependencies to match the submission time, does not improve reproducibility (decreased to 12%) but rather introduces additional failure modes. To address environmental erosion, we design and implement MLEModernizer, an LLM-driven agentic framework that treats the contemporary environment as a fixed constraint and modernizes notebook code to restore reproducibility. MLEModernizer iteratively executes notebooks, collects execution feedback, and applies three types of targeted fixes: error-repair, runtime-reduction, and score-calibration. Evaluated on 8,210 notebooks that are non-reproducible under the baseline environment, MLEModernizer makes 3,292 (40.1%, GPT-5.2) and 3,683 (44.9%, GPT-OSS-120b) notebooks reproducible. MLEModernizer presents a best-effort automated recovery and modernization technique that can improve reproducibility for a subset of notebooks. Practitioners can leverage MLEModernizer to validate, reuse, and maintain MLE artifacts as the hardware and software ecosystems continue to evolve.

Figures

Figures reproduced from arXiv: 2602.07195 by the authors.

Figure 1
Figure 1. An example notebook and our attempt to reproduce its results. Left: successful run on Kaggle; Right: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Distribution of notebook runtime. Filtering Criteria. We apply several filtering cri￾teria to focus our study on notebooks where re￾producibility can be meaningfully measured, First, we retain only notebooks that have been success￾fully executed on Kaggle and have a non-zero score, as these represent notebooks that successfully pro￾duced valid predictions and can be evaluated against ground truth. Second, we retain … view at source ↗
Figure 3
Figure 3. Sankey diagram of notebook reproducibility flow from Baseline to Backporting. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: MLEModernizer’s workflow. • if the execution times out, which can be caused by API misuse or not enabling GPU accelera￾tion, MLEModernizer first needs to reduce the runtime; • if the score deviates and the notebook has errors, MLEModernizer attempts to fix the errors a…
Figure 5
Figure 5. Figure 5: MLEModernizer successfully modernizes the notebook in [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Sankey diagram showing the transitions from Baseline to [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Distribution of number of LLM fixes performed by [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Cumulative (left) and per-fix (right) code modification scale by [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Error-type evolution (left) and distribution at end of fixes (right). [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Effectiveness of using cell-level error-repair in [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 1 linked inside Pith

  1. [2]

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. 2024. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. (2024). arXiv:2410.07095 [cs.CL] https://arxiv.org/abs/2410.07095

  2. [3]

    2025.pigar

    Damnever. 2025.pigar. https://pypi.org/project/pigar/

  3. [5]

    Malinda Dilhara, Danny Dig, and Ameya Ketkar. 2023. PyEvolve: Automating Frequent Code Changes in Python ML Systems. InInternational Conference on Software Engineering. 995–1007. doi:10.1109/ICSE48619.2023.00091

  4. [6]

    Mojtaba Mostafavi Ghahfarokhi, Arash Asgari, Mohammad Abolnejadian, and Abbas Heydarnoori. 2024. DistilKaggle: A Distilled Dataset of Kaggle Jupyter Notebooks. InInternational Working Conference on Mining Software Repositories. 647–651. doi:10.1145/3643991.3644882

  5. [7]

    Odd Erik Gundersen and Sigbjørn Kjensmo. 2018. State of the Art: Reproducibility in Artificial Intelligence. InAAAI Conference on Artificial Intelligence, Vol. 32. doi:10.1609/aaai.v32i1.11503

  6. [8]

    Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep Reinforce- ment Learning That Matters. InAAAI Conference on Artificial Intelligence, Vol. 32. doi:10.1609/aaai.v32i1.11694

  7. [9]

    2025.tiktoken

    Shantanu Jain. 2025.tiktoken. https://pypi.org/project/tiktoken/

  8. [10]

    2025.Learning to Edit Interactive Machine Learning Notebooks

    Bihui Jin, Jiayue Wang, and Pengyu Nie. 2025.Learning to Edit Interactive Machine Learning Notebooks. Association for Computing Machinery, New York, NY, USA, 681–685

Show all 22 references
  1. [11]

    2025.How to Use Kaggle

    Kaggle. 2025.How to Use Kaggle. https://www.kaggle.com/docs/notebooks#the-notebooks-environment

  2. [12]

    2025.How to Use Kaggle

    Kaggle. 2025.How to Use Kaggle. https://www.kaggle.com/docs/notebooks#technical-specifications

  3. [13]

    2025.kaggle

    Kaggle. 2025.kaggle. https://pypi.org/project/kaggle/

  4. [14]

    Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter Development Team. 2016. Jupyter Noteboo...

  5. [15]

    João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2019. A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks. InInternational Working Conference on Mining Software Repositories. doi:10.1109/MSR.2019.00077

  6. [16]

    João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, Juliana Freire, Bram Adams, and Sonia Haiduc. 2021. Understanding and improving the quality and reproducibility of Jupyter notebooks.Empirical Software Engineering26, 65 (2021). doi:10.1007/s10664-021-09961-9

  7. [17]

    Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché- Buc, Emily Fox, and Hugo Larochelle. 2021. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program).Journal of...

  8. [18]

    Adam Rule, Aurélien Tabard, and James D. Hollan. 2018. Exploration and Explanation in Computational Notebooks. In CHI Conference on Human Factors in Computing Systems. 32:1–32:12. doi:10.1145/3173574.3173606

  9. [19]

    Muller, Soya Park, Justin D

    April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael J. Muller, Soya Park, Justin D. Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. 2021. Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation , Vol. 1, No. 1, Article . Publication date: Februar...

  10. [20]

    2025.LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion

    Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2025.LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion. IEEE Press, 885–897. https: //doi.org/10.1109/ICSE55347.2025.00245

  11. [21]

    Yiran Wang, Willem Meijer, Jose Antonio Hernandez Lopez, Ulf Nilsson, and Daniel Varro. 2025. Why do Machine Learning Notebooks Crash? An Empirical Study on Public Python Jupyter Notebooks.Transactions on Software Engineering51, 7 (2025), 2181–2196. doi:10.1109/TSE.2025.3574500

  12. [22]

    Linna Xie, Zhong Li, Yu Pei, Zhongzhen Wen, Kui Liu, Tian Zhang, and Xuandong Li. 2025. PReMM: LLM-Based Program Repair for Multi-method Bugs via Divide and Conquer.International Conference on Object-Oriented Programming, Systems, Languages, and Applications9, OOPSLA2, Article...

  13. [23]

    Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. 2023. Natural Language to Code Generation in Interactive Data Science Notebooks. InAnnual Mee...

  14. [24]

    Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. 2024. PyDex: Repairing Bugs in Introductory Python Assignments using LLMs.International Conference on Object-Oriented Programming, Systems, Languages, and Applications...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.