REVIEW 3 major objections 4 minor 22 references
Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility
T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash
Pith's one-line read Automated code modernization, not environment backporting, can restore reproducibility for the majority of legacy machine-learning notebooks.
desk verdict Valuable dataset and an important negative result on backporting, but the 74.2% headline is undercut by a scoring cue in the prompt and by inconsistent numbers across the paper. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is MLEModernizer, an LLM-driven agentic loop that treats the current execution environment as a fixed constraint and rewrites the notebook code. At each iteration it executes the notebook, gathers error tracebacks, runtime, and a reproduced score, then classifies the failure into one of three fix types: error-repair (when errors occur), runtime-reduction (when execution times out), or score-calibration (when the code runs but the score deviates). The LLM receives the notebook in a cell-delimited format along with the target score, and returns a full updated notebook in a plan-then-patch format. The success criterion is a score deviation within 10% of the reported target, w
What would settle it
Take a random sample of notebooks that MLEModernizer classified as reproducible via score-calibration, then re-execute them on a different held-out split of the same competition data (or on the original private test set if available) and compare both the score deviation and the actual prediction vectors against the pre-calibration outputs. If a substantial fraction fail the 10% band or produce materially different predictions, then the reported reproducibility rate is inflated by target-chasing rather than genuine modernization.
Extended reading notes
Core claim
The paper establishes that code modernization is a viable alternative to environment reconstruction for restoring reproducibility in ML notebooks. On a dataset of 12,720 Kaggle notebooks from 79 competitions, only 35.4% reproduce in a contemporary environment, and backporting dependencies to submission timestamps actually lowers this to 35.1% while adding failure modes. MLEModernizer, an agentic LLM framework, iteratively executes a notebook, collects execution feedback, and applies three types of fixes—error-repair, runtime-reduction, and score-calibration—until the reproduced score falls within 10% of the reported target. Applied to 7,402 non-reproducible notebooks, it achieves 74.2% repro
Load-bearing premise
The load-bearing assumption is that bringing the reproduced score within 10% of the reported target, with the target given to the LLM, genuinely indicates that the notebook's original behavior has been restored—rather than that the LLM has tuned the code to hit the target score regardless of semantic fidelity.
Editorial extensions
If this is right
- Environment backporting, the dominant industry response to reproducibility failures, is shown to be ineffective and sometimes harmful; this shifts attention to code-level modernization.
- An LLM agent can restore reproducibility for roughly three-quarters of legacy non-reproducible notebooks at low cost, suggesting that automated modernization is economically feasible at scale.
- The three-tier fix taxonomy means that reproducibility can often be achieved even when residual errors remain; about 17% of modernized notebooks produce reproducible scores despite unresolved errors.
- The stubborn error types are attribute errors rooted in API misuse, indicating that future repair tools should focus on API evolution and migration patterns.
- The method's dependence on a target score for calibration creates a clear tension between faithful reproduction and score matching, which future evaluations must address.
Reading between the lines
- The 74.2% rate may overstate true behavioral fidelity: score-calibration is given the target score as part of the prompt, so an LLM could adjust hyperparameters or post-processing to hit the band without preserving the original model's predictions; a stricter test would compare predictions on a held-out split.
- The same agentic modernization loop could be applied outside notebooks, for example to legacy data pipelines or scientific scripts, wherever the goal is to run on today's stack while preserving outputs.
- The score-calibration mechanism, if not bounded, could be exploited as a form of 'score hacking' in benchmark settings; adding a constraint that predictions must remain close to the pre-calibration model's outputs would guard against this.
- The finding that runtime-reduction fixes rarely succeed suggests that timeout-prone notebooks need a different strategy, such as code-level algorithmic changes or user-assisted resource allocation, rather than LLM patching alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies the reproducibility of legacy machine-learning notebooks from Kaggle by re-executing them in a current Kaggle-like container. It defines reproducibility as a reproduced score sr within 10% of the reported target score st, reports that only 35.4% of 12,720 notebooks (full-text numbers) are reproducible at baseline, and finds that environment backporting does not help (35.1%). It then introduces MLEModernizer, an LLM-driven agent that iteratively executes notebooks and applies error-repair, runtime-reduction, and score-calibration fixes. The full text reports that MLEModernizer makes 5,492 of 7,402 previously non-reproducible notebooks (74.2%) reproducible. The paper concludes that code modernization is a viable alternative to environment reconstruction.
Significance. If the evaluation is valid, the contribution is significant: a large-scale, containerized re-execution study of real MLE notebooks; a practical, cost-reported agentic repair pipeline; a clear negative result for environment backporting; and an open artifact. The authors are careful to separate error-free execution from score-band reproducibility and to report an ablation study. However, the current version has two load-bearing problems: the abstract and full text report irreconcilable dataset sizes, baseline rates, and headline results; and the score-calibration evaluation is circular with the success metric. These issues must be resolved before the central claim can be interpreted.
major comments (3)
- [Abstract vs. §1, §2.1, §4.1] The abstract reports 12,106 notebooks from 75 competitions, 26% baseline reproducibility, backporting 'decreased to 12%', 8,210 modernization subjects, and success rates of 3,292 (40.1%, GPT-5.2) and 3,683 (44.9%, GPT-OSS-120b). The full text reports 12,720 notebooks from 79 competitions, 35.4% baseline reproducibility, 35.1% after backporting, 7,402 subjects, and 5,492 (74.2%) with GPT-5.2 only. These are not cosmetic differences: they change the dataset, the backporting conclusion, and the headline result. The manuscript must be harmonized, and the authoritative numbers must be identified explicitly.
- [§2.1.2, §3.3, §4.2 (Table 4)] The success criterion (|sr - st|/|st| <= 10%) is the same quantity that MLEModernizer is given as a target and asked to optimize. The input to MLEModernizer includes the target score st (§3.1); the shared prompt contains 'Scores: targets t, currents r' (§3.3); score-calibration is described as aligning the output 'toward the reported Kaggle score' (§4.2, Table 4 caption); and score-calibration accounts for 21,556 entries in Table 4. With no prediction-level guardrail, the 74.2% rate measures the LLM's ability to hit a target score band, not whether the modernized notebook faithfully reproduces the original computation. The running example in §3.4 accepts sr = 0.93778 for st = 0.87511, i.e., a 7.2% relative score difference, as success. Please add a validation that withholds st during score-calibration (e.g., held-out target scores), and/or a prediction-level comparison (e.g., agreement o
- [§4.1 vs. §4.2] Table 3 and §4.1 report 5,492 reproducible notebooks (4,208 Error-Free Reproducible + 1,284 Error Reproducible). Section 4.2, Figure 7b, and the RQ2 summary report 5,042 reproducible notebooks for the same cohort of 7,402 notebooks. This 450-notebook discrepancy affects the headline count and the per-fix statistics. It must be reconciled before the results can be interpreted.
minor comments (4)
- [Table 4, Runtime-Reduction row] The listed total for Runtime-Reduction is 4,449, but the five outcome columns sum to 5,449 (3,143 + 835 + 1,365 + 29 + 77). The text repeats 4,449. Please correct the total and recompute fix-type shares.
- [§3.4] A 7.2% relative score deviation is described as 'a bit higher' than the target. This is close to the 10% cap; please reword and discuss whether the example demonstrates faithful reproduction or merely entry into the score band.
- [§2.2.2] Baseline reproducibility is reported as 4,501/12,720 (35.4%) in the text and as '35%' in the summary. Use one consistent decimal representation.
- [Throughout] The terms 'environment erosion' and 'environmental erosion' are used inconsistently. Also, 'reproducible' is sometimes used without qualification even though the paper's key distinction is between Error-Free Reproducible and Error Reproducible; define the qualified terms at first use.
Circularity Check
Score-calibration is prompted with the target score and judged by proximity to it, so a component of the 74.2% reproducibility result is self-fulfilling; the 5,492 vs 5,042 count inconsistency is a separate correctness issue.
-
self definitional
[§2.1.2 (reproducibility criterion), §3.3 (shared prompt), Table 4 (Score-Calibration definition)]
"a notebook is reproducible if its reproduced score deviation from the target score is within a threshold ... δs = |sr−st|/|st| ≤ τ ... Scores: targets t, currents r, and the metric directionality (higher/lower is better) ... Score-Calibration when the script has no errors and does not need runtime reduction but its output must be aligned toward the reported Kaggle score."
The reproducibility predicate is defined as δs ≤ 10%. The score-calibration prompt hands the LLM the exact target st and current sr and instructs it to align the output toward st. A notebook is then declared reproducible precisely when δs ≤ 10%, i.e., when the same quantity the fix was asked to minimize is small. No prediction-level check (e.g., prediction agreement or model equivalence) is reported, and the running example treats 0.93778 vs 0.87511 as success. Thus the score-calibration component of the 74.2% result measures the LLM's ability to hit the target it was given, not faithful restoration of the original computation; this part of the derivation reduces to the evaluation definition by construction.
full rationale
The central circularity is localized to score-calibration: the target score is both an input to the fix prompt and the reference point of the success predicate, making those successes partly self-fulfilling. The evaluation is not entirely circular, because error-repair and runtime-reduction fixes are judged by execution success and runtime, and the baseline/backporting numbers are anchored to the external MLE-Bench grader. No load-bearing self-citation or imported uniqueness theorem was found; the one self-citation (Jin et al. [10]) is contextual and not load-bearing. Separately, the manuscript contains an unresolved internal inconsistency: §4.1/Table 3 reports 5,492 reproducible notebooks, while §4.2/Figure 7b and its summary report 5,042. That is a correctness/verification concern rather than circularity, but it must be reconciled before the headline count is interpretable. Overall: partial circularity, score 6.
Assumptions & free parameters
free parameters (4)
- Reproducibility threshold τ =
0.10 (10%)
- Maximum fix iterations N =
16
- Wall-clock timeout =
600 seconds
- Token cutoff for subject pool =
13,485 tokens (95th percentile)
assumptions (5)
- domain assumption MLE-Bench's offline grader with its own train/test split approximates the Kaggle private leaderboard evaluation.
- domain assumption The reported Kaggle score st is a valid target for the offline grader's score sr.
- domain assumption The contemporary Kaggle Docker image plus MLE-Bench data is a representative 'contemporary environment'.
- domain assumption LLM code edits preserve the semantic intent of the original pipeline except for repairing breakage.
- domain assumption Error-tolerant execution, continuing past failing cells, yields a meaningful execution outcome.
Cite this review
Pith. "Pith review of Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility." pith.science (2026). https://pith.science/paper/TKZ7W37M
@misc{pith2026260207195,
author = {Pith},
title = {Pith review of: Automated Modernization of Machine Learning Engineering Notebooks for Reproducibility},
year = {2026},
howpublished = {\url{https://pith.science/paper/TKZ7W37M}},
note = {Machine review of arXiv:2602.07195}
}
read the original abstract
Interactive computational notebooks (e.g., Jupyter notebooks) are widely used in machine learning engineering (MLE) to program and share end-to-end pipelines, from data preparation to model training and evaluation. However, environmental erosion-the rapid evolution of hardware and software ecosystems for machine learning-has rendered many published MLE notebooks non-reproducible in contemporary environments, hindering code reuse and scientific progress. To quantify this gap, we study 12,106 notebooks selected from 75 popular Kaggle competitions: only 26% remain reproducible today. Crucially, we find that environment backporting, i.e., downgrading dependencies to match the submission time, does not improve reproducibility (decreased to 12%) but rather introduces additional failure modes. To address environmental erosion, we design and implement MLEModernizer, an LLM-driven agentic framework that treats the contemporary environment as a fixed constraint and modernizes notebook code to restore reproducibility. MLEModernizer iteratively executes notebooks, collects execution feedback, and applies three types of targeted fixes: error-repair, runtime-reduction, and score-calibration. Evaluated on 8,210 notebooks that are non-reproducible under the baseline environment, MLEModernizer makes 3,292 (40.1%, GPT-5.2) and 3,683 (44.9%, GPT-OSS-120b) notebooks reproducible. MLEModernizer presents a best-effort automated recovery and modernization technique that can improve reproducibility for a subset of notebooks. Practitioners can leverage MLEModernizer to validate, reuse, and maintain MLE artifacts as the hardware and software ecosystems continue to evolve.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[2]
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. 2024. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. (2024). arXiv:2410.07095 [cs.CL] https://arxiv.org/abs/2410.07095
arXiv 2024
-
[3]
2025.pigar
Damnever. 2025.pigar. https://pypi.org/project/pigar/
2025
-
[5]
Malinda Dilhara, Danny Dig, and Ameya Ketkar. 2023. PyEvolve: Automating Frequent Code Changes in Python ML Systems. InInternational Conference on Software Engineering. 995–1007. doi:10.1109/ICSE48619.2023.00091
arXiv 2023
-
[6]
Mojtaba Mostafavi Ghahfarokhi, Arash Asgari, Mohammad Abolnejadian, and Abbas Heydarnoori. 2024. DistilKaggle: A Distilled Dataset of Kaggle Jupyter Notebooks. InInternational Working Conference on Mining Software Repositories. 647–651. doi:10.1145/3643991.3644882
arXiv 2024
-
[7]
Odd Erik Gundersen and Sigbjørn Kjensmo. 2018. State of the Art: Reproducibility in Artificial Intelligence. InAAAI Conference on Artificial Intelligence, Vol. 32. doi:10.1609/aaai.v32i1.11503
-
[8]
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. 2018. Deep Reinforce- ment Learning That Matters. InAAAI Conference on Artificial Intelligence, Vol. 32. doi:10.1609/aaai.v32i1.11694
-
[9]
2025.tiktoken
Shantanu Jain. 2025.tiktoken. https://pypi.org/project/tiktoken/
2025
-
[10]
2025.Learning to Edit Interactive Machine Learning Notebooks
Bihui Jin, Jiayue Wang, and Pengyu Nie. 2025.Learning to Edit Interactive Machine Learning Notebooks. Association for Computing Machinery, New York, NY, USA, 681–685
2025
Show all 22 references
-
[11]
2025.How to Use Kaggle
Kaggle. 2025.How to Use Kaggle. https://www.kaggle.com/docs/notebooks#the-notebooks-environment
2025
-
[12]
2025.How to Use Kaggle
Kaggle. 2025.How to Use Kaggle. https://www.kaggle.com/docs/notebooks#technical-specifications
2025
-
[13]
2025.kaggle
Kaggle. 2025.kaggle. https://pypi.org/project/kaggle/
2025
-
[14]
Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, Carol Willing, and Jupyter Development Team. 2016. Jupyter Noteboo...
2016 doi
-
[15]
João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2019. A Large-scale Study about Quality and Reproducibility of Jupyter Notebooks. InInternational Working Conference on Mining Software Repositories. doi:10.1109/MSR.2019.00077
2019
-
[16]
João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, Juliana Freire, Bram Adams, and Sonia Haiduc. 2021. Understanding and improving the quality and reproducibility of Jupyter notebooks.Empirical Software Engineering26, 65 (2021). doi:10.1007/s10664-021-09961-9
2021 doi
-
[17]
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché- Buc, Emily Fox, and Hugo Larochelle. 2021. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program).Journal of...
2021
-
[18]
Adam Rule, Aurélien Tabard, and James D. Hollan. 2018. Exploration and Explanation in Computational Notebooks. In CHI Conference on Human Factors in Computing Systems. 32:1–32:12. doi:10.1145/3173574.3173606
2018
-
[19]
Muller, Soya Park, Justin D
April Yi Wang, Dakuo Wang, Jaimie Drozdal, Michael J. Muller, Soya Park, Justin D. Weisz, Xuye Liu, Lingfei Wu, and Casey Dugan. 2021. Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation , Vol. 1, No. 1, Article . Publication date: Februar...
2021
-
[20]
2025.LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion
Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2025.LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion. IEEE Press, 885–897. https: //doi.org/10.1109/ICSE55347.2025.00245
2025
-
[21]
Yiran Wang, Willem Meijer, Jose Antonio Hernandez Lopez, Ulf Nilsson, and Daniel Varro. 2025. Why do Machine Learning Notebooks Crash? An Empirical Study on Public Python Jupyter Notebooks.Transactions on Software Engineering51, 7 (2025), 2181–2196. doi:10.1109/TSE.2025.3574500
2025
-
[22]
Linna Xie, Zhong Li, Yu Pei, Zhongzhen Wen, Kui Liu, Tian Zhang, and Xuandong Li. 2025. PReMM: LLM-Based Program Repair for Multi-method Bugs via Divide and Conquer.International Conference on Object-Oriented Programming, Systems, Languages, and Applications9, OOPSLA2, Article...
2025 doi
-
[23]
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. 2023. Natural Language to Code Generation in Interactive Data Science Notebooks. InAnnual Mee...
2023 doi
-
[24]
Jialu Zhang, José Pablo Cambronero, Sumit Gulwani, Vu Le, Ruzica Piskac, Gustavo Soares, and Gust Verbruggen. 2024. PyDex: Repairing Bugs in Introductory Python Assignments using LLMs.International Conference on Object-Oriented Programming, Systems, Languages, and Applications...
2024 doi
Reviewed August 3, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.