REVIEW 3 major objections 6 minor 24 references
iML claims that code-driven AutoML becomes reliable and competitive when generation is split into a grounded blueprint, isolated modules, and runtime verification, achieving 85% valid submissions and a 45% medal rate on MLE-BENCH.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 23:22 UTC pith:2QYPNTAD
load-bearing objection A genuinely interesting interface-contract mechanism for code-driven AutoML, but the empirical claims are currently unverifiable due to baseline-fairness issues and internal number conflicts. the 3 major comments →
iML: Executable, Problem-Grounded, and Broadly Exploratory Code-Driven AutoML
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central claim is that a multi-agent, code-driven AutoML framework built on three principles—Code-Guided Planning, Code-Modular Implementation, and Code-Verifiable Integration—can achieve a valid submission rate of 85% and a medal rate of 45% on MLE-BENCH, with an average performance score (APS) of 0.77, and an APS of 0.58 on the author-curated iML-BENCH, outperforming LLM-based baselines by 38%–163%. The decisive mechanism is the interface contract: a strict specification of data types, shapes, and formats that decouples preprocessing from modeling and is enforced by actually executing intermediate code, so that errors are localized and repaired before final assembly. The paper a
What carries the argument
The central object is the interface contract (IC), a strict definition of the exact object types, tensor shapes, and data schemas exchanged between the preprocessing and modeling modules. This contract is synthesized during the strategic-planning phase, embedded in a structured blueprint, and enforced at runtime by the Assembly and Debugging agents. The contract's job is to make the postcondition of the preprocessing module (what it guarantees about its output) imply the precondition of the modeling module (what that module expects as input); dynamic execution verifies this implication, turning a probabilistic LLM generation loop into a verifiable engineering pipeline.
Load-bearing premise
The headline comparisons assume the baseline agents—especially MLZero, described as an AutoGluon wrapper—were run with correctly configured runtime environments; if MLZero's wrapper was broken on five of nine iML-BENCH tasks (where AutoGluon alone scores above 0.85), the claimed 38%-163% margins collapse.
What would settle it
Re-run MLZero on the iML-BENCH tasks Dog Breed and Paddy Disease using the standard AutoGluon configuration that alone yields scores of 0.85 and 0.96 respectively; if MLZero then scores well above the reported 0.00, the paper's baseline-comparison claim is falsified. Alternatively, run iML with zero debugging rounds (K=0) and check whether valid submissions drop to the reported 40%—if it stays high, the debugging budget is not the load-bearing factor claimed.
If this is right
- If iML's results hold, code-driven AutoML becomes practical for non-experts: the system produces executable, deployable pipelines with a high success rate rather than plausible-looking scripts that crash.
- The modular contract-based design implies that debugging and optimization can be targeted at single components (e.g., fixing data leakage in preprocessing) without rewriting the whole pipeline, which would directly address a known weakness of monolithic LLM-generated code.
- The reported robustness to stripped task descriptions suggests that empirical data profiling can substitute for detailed textual instructions, implying that AutoML agents could operate on raw data with minimal human specification.
- The finding that a single best-selected track outperforms ensembling in competitive metrics suggests that for Kaggle-style tasks, depth of optimization matters more than combining diverse architectures.
- The 70% success rate under stripped descriptions implies that iML could be used in settings where datasets arrive without documentation, such as automated data integration pipelines.
Where Pith is reading between the lines
- The paper's design, if correct, suggests a template for any LLM-based code generation task beyond AutoML: ground generation in empirical observation, decompose into contract-bound modules, and verify by execution at every step. This could generalize to data cleaning, scientific computing, and software engineering agents.
- The iML-BENCH results indicate that current wrapper-based LLM agents (MLZero) may be fundamentally limited by their underlying library; the paper implies that native code synthesis, despite higher variance, can exceed that ceiling on non-standard data.
- A testable extension would be to apply iML to a fresh set of competitions not curated by its authors—especially multimodal tasks where it scored lowest (APS 0.56)—to check whether the reported advantage persists beyond the author-selected benchmark.
- The paper leaves implicit that the 80/20 split of iML-BENCH does not reflect real competition settings where full training data is available; on full data, baselines like AutoGluon might close the gap, since they already score above 0.85 on some tabular tasks.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes iML, a multi-agent code-driven AutoML framework built on three pillars: code-guided planning grounded in empirical data profiling, modular implementation with strict interface contracts, and runtime-verified integration with iterative debugging. The framework is evaluated on MLE-BENCH (Lite) and an author-created iML-BENCH, and the paper reports higher valid-submission rates, medal rates, and average standardized performance scores than AutoGluon, MLZero, MLE-STAR, and AutoML-Agent. The central claim is that these design choices jointly produce a more reliable and competitive code-driven AutoML system.
Significance. If the empirical results are correct, the paper provides a useful architectural template—profiling-grounded planning, contract-based modularity, and dynamic verification—with ablations that isolate each design contribution. The use of an external benchmark (MLE-BENCH) and the promise of released source code are strengths. However, the comparative claim rests on two currently unverified premises: baseline execution fairness and internal consistency of the reported numbers. Both are essential to the paper's quantitative conclusions and neither is established in the present manuscript. Once these are fixed, the work could be a meaningful step for code-driven AutoML.
major comments (3)
- [Abstract; §5.1, Table 1; §5.2.2] The numerical results are internally inconsistent. The abstract reports a 90% valid submission rate and APS 0.82, with 52%–273% improvements; Table 1 and §5.1 report 85% valid, APS 0.77, and the body claims 38%–163% (from Table 2). In addition, §5.2.2 states that the full iML achieves an 'Any Medal rate of 85%', whereas Table 1 and §5.2.3/§5.2.4 report 45%. Because the central claim is quantitative, the authors must reconcile these figures and state which results are canonical.
- [§4.2, Table 2] Baseline fairness is not established. MLZero is described as a wrapper that uses AutoGluon as its core execution engine, yet on iML-BENCH it scores 0.00 on five tasks, including Dog Breed and Paddy Disease, where AutoGluon alone scores 0.85 and 0.96 respectively. A wrapper around a library that solves these tasks should not produce zero scores unless the wrapper's runtime configuration is broken—e.g., incorrect API usage, premature timeout, or invalid submission formatting. §4.2 only states that baselines were run under 'the similar evaluation procedure as in the existing studies' without pinning package versions, prompts, wrapper code, or per-task logs. Without this evidence, the claimed APS improvements over MLZero—and the framing of MLZero's zeros as a 'dependency ceiling'—are not credible. Please provide exact reproducible configurations and logs, or rerun the baselines.
- [§4.3, §5.1, Table 2] The APS aggregation and the comparative percentages are under-specified. §4.3 defines different normalization transformations for different metric types (raw scores, e^{-S_c}, and affine mapping for bounded scores) and then averages across competitions, making the single APS number difficult to interpret and potentially sensitive to benchmark composition. Moreover, the headline 'significantly outperforms' hides large heterogeneity: on Domain Classify, iML (0.40) trails AutoGluon and MLZero (0.44), and on Plant Traits, iML itself scores only 0.01. The paper should report per-competition details and justify that the chosen aggregation preserves the comparative claims.
minor comments (6)
- [References] Reference [18] is truncated: 'https://github.com/iSE-UET-VNU/iML-.' — provide a complete working URL.
- [Figure 8] The labels contain typos: 'Natual language' should be 'Natural language' and 'Vison' should be 'Vision'.
- [§5.1] The phrase 'perfect execution-to-submission ratio' is misleading: Table 1 shows Made Sub = Valid Sub = 85%, not 100%. Rephrase as 'all made submissions were valid'.
- [§4.3] The abstract and title call the metric 'average standardized performance score (APS)', but §4.3 defines a 'Normalized Performance Score'. Align the terminology.
- [Throughout] The benchmark name is written inconsistently as 'iML-BENCH' and 'iML-Bench'; standardize to one form.
- [§3.1, Eq. (1)] The semantic brackets J·K are used without definition. Please define them or replace with a clearer notation for execution semantics.
Circularity Check
No significant circularity: iML's central claims are empirical benchmark measurements against external MLE-BENCH and Kaggle competitions, not derivations from its own assumptions.
full rationale
The paper makes no formal derivation whose conclusion is equivalent to its inputs. Its central claim is an empirical evaluation: iML is run on MLE-BENCH (an external benchmark, reference [17]) and on iML-BENCH, which consists of external Kaggle competitions with a held-out 80/20 test split. Scores are computed by the benchmarks' own grading or by a stated normalization convention; no parameter is fitted to the target metric and then reported as a prediction. The framework's internal design — code-guided planning, modular implementation, verifiable integration — is described as an architecture, not as a theorem derived from a cited prior result. There are no load-bearing self-citations: the only self-referential artifact, the project website [18], is offered for reproducibility and is not used to justify a performance claim. No 'uniqueness theorem' or prior-work ansatz is imported to force the choice of iML's components. The internal metric inconsistencies (abstract 90% valid/APS 0.82 vs. body 85%/0.77, and Section 5.2.2's 85% Any Medal vs. Table 1's 45%) and the doubtful MLZero zero scores on tasks AutoGluon alone solves are threats to measurement fairness and internal consistency, not circularity. A benchmark-fairness problem does not make the derivation circular. Accordingly, the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (5)
- Maximum debugging rounds K =
10
- iML-BENCH train/test split ratio =
80/20
- Per-dataset wall-clock budget =
5 hours
- Normalized performance score mapping =
e^{-S_c} for losses; (S_c+1)/2 for bounded metrics; 0 for execution failure
- Number of seeded runs =
3
axioms (5)
- domain assumption MLE-BENCH Lite rankings correlate with the full MLE-BENCH rankings
- domain assumption All baseline agents are faithfully executed under the shared protocol
- domain assumption Empirical profiling (e.g., ydata-profiling output) suffices to recover task semantics when descriptions are stripped
- domain assumption Benchmark curation criteria (2021-2025, at least 1000 submissions, domain diversity) yield a representative testbed
- domain assumption Gemini-2.5-Flash is a neutral common backbone
read the original abstract
Automated Machine Learning (AutoML) has improved access to machine learning, yet existing techniques often remain limited in flexibility, transparency, and execution reliability. Code-driven AutoML offers a promising direction by synthesizing executable code for preprocessing, model training, and evaluation. However, current LLM-based approaches frequently generate code that is plausible in text yet brittle in execution, insufficiently grounded in the actual dataset, or restricted to narrow solution paths. In this paper, we introduce iML, a multi-agent code-driven AutoML framework designed around three requirements: executability, problem grounding, and broad exploration of valid solutions. iML first analyzes the task and profiles the data, then synthesizes a structured blueprint that guides modular code generation across multiple implementation tracks, including traditional ML,pretrained adaptation, and custom neural architectures. To improve reliability, iML enforces interface checking, dynamic execution, and iterative debugging during integration. We evaluate iML on MLE-BENCH and the newly introduced iML-BENCH, covering diverse Kaggle-style tasks. On MLE-BENCH, iML attains a 90% valid submission rate and a 45% medal rate, and an APS of 0.82, improving the average standardized performance score (APS) over the LLM-based baselines by 52%-273%. On iML-BENCH, it achieves the highest APS and demonstrates robust performance even when task descriptions are substantially stripped. These results establish iML as a reliable and competitive framework for code-driven AutoML.
Figures
Reference graph
Works this paper leans on
-
[1]
T. Chen, C. Guestrin, Xgboost: A scalable tree boost- ing system, in: Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, 2016, pp. 785–794
2016
-
[2]
Hollmann, S
N. Hollmann, S. Müller, L. Purucker, A. Krishnaku- mar,M.Körfer,S.B.Hoo,R.T.Schirrmeister,F.Hut- ter, Accurate predictions on small data with a tabular foundationmodel,Nature637(8045)(2025)319–326
2025
-
[3]
Prokhorenkova, G
L. Prokhorenkova, G. Gusev, A. Vorobev, A. V. Doro- gush, A. Gulin, Catboost: unbiased boosting with cat- egorical features, Advances in neural information pro- cessing systems 31 (2018)
2018
-
[4]
Hollmann, S
N. Hollmann, S. Müller, F. Hutter, Large language models for automated data science: Introducing caafe for context-aware automated feature engineering, Ad- vances in Neural Information Processing Systems 36 (2023) 44753–44775
2023
-
[5]
J. Nam, K. Kim, S. Oh, J. Tack, J. Kim, J. Shin, Optimized feature generation for tabular data via llms withdecisiontreereasoning,AdvancesinNeuralInfor- mation Processing Systems 37 (2024) 92352–92380
2024
-
[6]
Feurer, K
M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, F. Hutter, Auto-sklearn 2.0: Hands-free automl via meta-learning, Journal of Machine Learning Research 23 (261) (2022) 1–61
2022
-
[7]
X. He, K. Zhao, X. Chu, Automl: A survey of the state-of-the-art, Knowledge-based systems 212 (2021) 106622
2021
-
[8]
M.Feurer,A.Klein,K.Eggensperger,J.Springenberg, M. Blum, F. Hutter, Efficient and robust automated machinelearning,Advancesinneuralinformationpro- cessing systems 28 (2015)
2015
-
[9]
1946–1956
H.Jin,Q.Song,X.Hu,Auto-keras:Anefficientneural architecturesearchsystem,in:Proceedingsofthe25th ACMSIGKDDinternationalconferenceonknowledge discovery & data mining, 2019, pp. 1946–1956
2019
-
[10]
L. Li, H. Wang, L. Zha, Q. Huang, S. Wu, G. Chen, J. Zhao, Learning a data-driven policy network for pre-training automated feature engineering, in: The EleventhInternationalConferenceonLearningRepre- sentations, 2023
2023
-
[11]
N. Erickson, J. Mueller, A. Shirkov, H. Zhang, P. Lar- roy, M. Li, A. Smola, Autogluon-tabular: Robust and accurate automl for structured data, arXiv preprint arXiv:2003.06505 (2020)
Pith/arXiv arXiv 2003
-
[12]
LeDell, S
E. LeDell, S. Poirier, et al., H2o automl: Scalable automatic machine learning, in: Proceedings of the AutoML Workshop at ICML, Vol. 2020, 2020, p. 24
2020
-
[13]
Wang, A comprehensive survey of neural architec- ture search: Challenges and solutions, ACM Comput- ing Surveys (CSUR) 54 (4) (2021) 1–34
P.Ren,Y.Xiao,X.Chang,P.-Y.Huang,Z.Li,X.Chen, X. Wang, A comprehensive survey of neural architec- ture search: Challenges and solutions, ACM Comput- ing Surveys (CSUR) 54 (4) (2021) 1–34
2021
-
[14]
J. Nam, J. Yoon, J. Chen, J. Shin, S. Ö. Arık, T. Pfis- ter, Mle-star: Machine learning engineering agent via search and targeted refinement, arXiv preprint arXiv:2506.15692 (2025)
Pith/arXiv arXiv 2025
-
[15]
H. Fang, B. Han, N. Erickson, X. Zhang, S. Zhou, A. Dagar, J. Zhang, A. C. Turkmen, C. Hu, H. Rang- wala, et al., Mlzero: A multi-agent system for end- to-end machine learning automation, arXiv preprint arXiv:2505.13941 (2025)
Pith/arXiv arXiv 2025
-
[16]
P. Trirat, W. Jeong, S. J. Hwang, Automl-agent: A multi-agent llm framework for full-pipeline automl, arXiv preprint arXiv:2410.02958 (2024)
Pith/arXiv arXiv 2024
-
[17]
J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sher- burn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Pat- wardhan, et al., Mle-bench: Evaluating machine learn- ing agents on machine learning engineering, arXiv preprint arXiv:2410.07095 (2024). [18]https://github.com/iSE-UET-VNU/iML-
Pith/arXiv arXiv 2024
-
[19]
C.Thornton,F.Hutter,H.H.Hoos,K.Leyton-Brown, Auto-weka: Combined selection and hyperparameter optimization of classification algorithms, in: Proceed- ings of the 19th ACM SIGKDD international confer- ence on Knowledge discovery and data mining, 2013, pp. 847–855
2013
-
[20]
Zhang, Z
T. Zhang, Z. A. Zhang, Z. Fan, H. Luo, F. Liu, Q. Liu, W. Cao, L. Jian, Openfe: Automated feature genera- tion with expert-level performance, in: International Conference on Machine Learning, PMLR, 2023, pp. 41880–41901. Le et al.:Preprint submitted to ElsevierPage 15 of 16 𝑖ML
2023
-
[21]
J. G. Hernandez, A. K. Saini, A. Ghosh, J. H. Moore, The tree-based pipeline optimization tool: Tackling biomedical research problems with genetic program- ming and automated machine learning, Patterns 6 (7) (2025)
2025
-
[22]
C. Wang, Q. Wu, X. Liu, L. Quintanilla, Automated machinelearning&tuningwithflaml,in:Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022, pp. 4828–4829
2022
-
[23]
S. Zhang, C. Gong, L. Wu, X. Liu, M. Zhou, Automl- gpt: Automatic machine learning with gpt, arXiv preprint arXiv:2305.02499 (2023)
Pith/arXiv arXiv 2023
-
[24]
Zhang, Y
L. Zhang, Y. Zhang, K. Ren, D. Li, Y. Yang, Mlcopi- lot: Unleashing the power of large language models in solving machine learning tasks, in: Proceedings of the 18th Conference of the European Chapter of the AssociationforComputationalLinguistics(Volume1: Long Papers), 2024, pp. 2931–2959
2024
-
[25]
Le et al.:Preprint submitted to ElsevierPage 16 of 16
S.Guo,C.Deng,Y.Wen,H.Chen,Y.Chang,J.Wang, Ds-agent: Automated data science by empowering large language models with case-based reasoning, arXiv preprint arXiv:2402.17453 (2024). Le et al.:Preprint submitted to ElsevierPage 16 of 16
Pith/arXiv arXiv 2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.