pith. sign in

arxiv: 2508.10177 · v3 · submitted 2025-08-13 · 💻 cs.AI

KompeteAI: Accelerated Autonomous Multi-Agent System for End-to-End Pipeline Generation for Machine Learning Problems

Stepan Kulibaba , Artem Dzhalilov , Roman Pakhomov , Oleg Svidchenko , Alexander Gasnikov , Aleksei Shpilman This is my paper

Pith reviewed 2026-05-18 22:18 UTC · model grok-4.3

classification 💻 cs.AI
keywords AutoMLLLM-based systemspipeline generationpredictive scoringdynamic explorationMLE-Benchmerging stage
0
0 comments X

The pith

KompeteAI advances LLM-based AutoML by merging top partial solutions and predicting performance from early metrics to cut evaluation time.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents KompeteAI as a framework that fixes two core problems in current LLM AutoML systems: exploration that keeps ideas separate and cannot recombine strong pieces, plus slow loops of full code runs for every test. It adds a merging step after search to combine leading candidates, pulls real strategies from Kaggle notebooks and papers through retrieval, and introduces a predictive scorer that judges promise from early signals instead of waiting for complete validation. These changes produce 3 percent higher scores than prior leaders on MLE-Bench and make each pipeline check 6.9 times faster. Readers would care because the result points to practical ways to automate more of the routine work in building machine-learning solutions for everyday problems.

Core claim

KompeteAI is a novel AutoML framework with dynamic solution space exploration that introduces a merging stage to compose top candidates, integrates Retrieval-Augmented Generation sourcing ideas from Kaggle notebooks and arXiv papers, and employs a predictive scoring model together with accelerated debugging to assess solution potential from early-stage metrics. This setup outperforms leading methods such as RD-agent, AIDE, and Ml-Master by an average of 3 percent on the primary AutoML benchmark MLE-Bench while accelerating pipeline evaluation 6.9 times, and it also reaches state-of-the-art results on the newly proposed Kompete-bench.

What carries the argument

Dynamic solution space exploration with a merging stage that composes top candidates rather than treating ideas in isolation, supported by a predictive scoring model that ranks potential from early metrics.

If this is right

  • Merging top candidates improves recombination of partial solutions beyond isolated search methods.
  • Early predictive scoring cuts full validation cycles enough to accelerate evaluations by a factor of 6.9.
  • Retrieval from Kaggle and arXiv sources expands the space of usable real-world strategies.
  • The new Kompete-bench provides a stricter test set on which the same framework remains state of the art.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Merging mechanisms of this type could transfer to other multi-agent LLM systems that currently search ideas in isolation.
  • Early-metric predictors may reduce compute costs in any iterative code-generation loop outside AutoML.
  • Domain-specific retrieval from notebooks and papers could raise solution quality in specialized machine-learning tasks.

Load-bearing premise

The predictive scoring model based on early-stage metrics reliably ranks solution potential without requiring full code execution and validation cycles.

What would settle it

Run an ablation of KompeteAI that disables the predictive scoring model and accelerated debugging, then measure whether the 3 percent performance edge on MLE-Bench vanishes and whether total evaluation time rises back toward the levels of prior systems.

Figures

Figures reproduced from arXiv: 2508.10177 by Aleksei Shpilman, Alexander Gasnikov, Artem Dzhalilov, Oleg Svidchenko, Roman Pakhomov, Stepan Kulibaba.

Figure 1
Figure 1. Figure 1: The KompeteAI AutoML pipeline. 3.1 PIPELINE SETUP This phase sets up the core components required for the next stages. The dataset is ingested by The Reader Agent, which analyzes its structure, produces a detailed task specification, and initializes the data for the RAG based on this description. The Metric Agent constructs unit tests to support submission validation and defines the evaluation metric funct… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of our pipeline with AIDE, RD-agent and ML-Master on Contemporary and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
read the original abstract

Recent Large Language Model (LLM)-based AutoML systems demonstrate impressive capabilities but face significant limitations such as constrained exploration strategies and a severe execution bottleneck. Exploration is hindered by one-shot methods lacking diversity and Monte Carlo Tree Search (MCTS) approaches that fail to recombine strong partial solutions. The execution bottleneck arises from lengthy code validation cycles that stifle iterative refinement. To overcome these challenges, we introduce KompeteAI, a novel AutoML framework with dynamic solution space exploration. Unlike previous MCTS methods that treat ideas in isolation, KompeteAI introduces a merging stage that composes top candidates. We further expand the hypothesis space by integrating Retrieval-Augmented Generation (RAG), sourcing ideas from Kaggle notebooks and arXiv papers to incorporate real-world strategies. KompeteAI also addresses the execution bottleneck via a predictive scoring model and an accelerated debugging method, assessing solution potential using early stage metrics to avoid costly full-code execution. This approach accelerates pipeline evaluation 6.9 times. KompeteAI outperforms leading methods (e.g., RD-agent, AIDE, and Ml-Master) by an average of 3\% on the primary AutoML benchmark, MLE-Bench. Additionally, we propose Kompete-bench to address limitations in MLE-Bench, where KompeteAI also achieves state-of-the-art results

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces KompeteAI, an LLM-based multi-agent AutoML framework that performs dynamic solution-space exploration via a merging stage for top candidates, augments the hypothesis space with RAG over Kaggle notebooks and arXiv papers, and employs a predictive scoring model together with accelerated debugging to evaluate pipelines using only early-stage metrics. The authors report that this yields a 6.9× acceleration in pipeline evaluation and a 3 % average improvement over RD-agent, AIDE and ML-Master on MLE-Bench, together with state-of-the-art results on a newly proposed Kompete-bench.

Significance. If the predictive scoring model reliably ranks pipelines without full execution, the reported acceleration would be a practically useful contribution to LLM-based AutoML. The merging strategy and external-knowledge retrieval are conceptually sound extensions of prior MCTS-style approaches. The 3 % gain is modest yet consistent across named baselines; the introduction of Kompete-bench is a positive step toward more challenging evaluation. However, the absence of validation for the core acceleration mechanism limits the strength of the empirical claims.

major comments (2)
  1. The predictive scoring model is presented as assessing solution potential from early-stage metrics to avoid full-code execution, yet the manuscript provides neither a correlation coefficient (or other quantitative measure) between those early metrics and final validation scores nor an ablation that compares predictive top-k ranking against full-execution ranking. This directly underpins both the 6.9× acceleration claim and the reported performance advantage; without such evidence the efficiency and accuracy benefits remain unsubstantiated.
  2. Experimental section (MLE-Bench results): the 3 % average improvement is stated without per-task breakdowns, standard deviations across multiple runs, or statistical significance tests. It is therefore unclear whether the gains are robust or arise from benchmark-specific tuning of the predictive model or merging heuristics.
minor comments (2)
  1. Notation for the predictive scoring function and the early-stage metrics should be defined explicitly (e.g., in a dedicated subsection or table) rather than described only in prose.
  2. The description of Kompete-bench would benefit from a concise table listing the tasks, data characteristics, and evaluation protocol to allow direct comparison with MLE-Bench.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments and the recommendation for major revision. The points raised highlight important aspects of validating our core acceleration mechanism and strengthening the empirical evaluation. We address each comment below and commit to incorporating the suggested analyses in the revised manuscript.

read point-by-point responses

  1. Referee: The predictive scoring model is presented as assessing solution potential from early-stage metrics to avoid full-code execution, yet the manuscript provides neither a correlation coefficient (or other quantitative measure) between those early metrics and final validation scores nor an ablation that compares predictive top-k ranking against full-execution ranking. This directly underpins both the 6.9× acceleration claim and the reported performance advantage; without such evidence the efficiency and accuracy benefits remain unsubstantiated.

    Authors: We agree that explicit quantitative validation of the predictive scoring model is required to substantiate the 6.9× acceleration and performance claims. In the revised manuscript we will add a correlation analysis (Pearson and Spearman coefficients) between the early-stage metrics and the final validation scores across tasks. We will also include an ablation comparing the top-k pipeline rankings produced by the predictive model against rankings obtained from full execution on a held-out subset of MLE-Bench tasks. These additions will directly demonstrate the reliability of early scoring. revision: yes

  2. Referee: Experimental section (MLE-Bench results): the 3 % average improvement is stated without per-task breakdowns, standard deviations across multiple runs, or statistical significance tests. It is therefore unclear whether the gains are robust or arise from benchmark-specific tuning of the predictive model or merging heuristics.

    Authors: We concur that more detailed and statistically rigorous reporting is needed to establish robustness. In the revision we will expand the experimental section to include per-task performance tables on MLE-Bench, report standard deviations computed over at least five independent runs per method, and apply paired t-tests (or Wilcoxon signed-rank tests where appropriate) to assess statistical significance of the 3 % average improvement relative to each baseline. These changes will clarify that the gains are not artifacts of specific tuning. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical claims rest on external benchmarks and direct comparisons

full rationale

The paper describes an engineering system (KompeteAI) with components including merging of candidates, RAG from external sources, a predictive scoring model using early metrics, and accelerated debugging. Performance claims (3% gain on MLE-Bench, 6.9× acceleration) are presented as results of empirical evaluation against named prior systems on public benchmarks, plus a newly proposed Kompete-bench. No mathematical derivation chain, equations, or first-principles results are claimed that reduce to author-defined inputs by construction. The predictive model is an empirical component whose reliability is asserted via system-level outcomes rather than self-referential fitting or self-citation load-bearing. The work is self-contained against external benchmarks with no reduction of predictions to fitted parameters or ansatzes imported via self-citation.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 1 invented entities

The work relies on standard assumptions about LLM capabilities and benchmark validity; no new mathematical axioms or free parameters are introduced in the abstract description.

invented entities (1)
  • Predictive scoring model no independent evidence
    purpose: Assess solution potential from early metrics to avoid full execution
    New component introduced to address execution bottleneck; no independent evidence or external validation provided in abstract.

pith-pipeline@v0.9.0 · 5797 in / 1168 out tokens · 36169 ms · 2026-05-18T22:18:45.752396+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search

    cs.LG 2026-03 unverdicted novelty 6.0

    Gome reaches 35.1% any-medal rate on MLE-Bench by mapping reasoning to gradient-based updates, outperforming tree search once models are sufficiently capable.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · cited by 1 Pith paper · 5 internal anchors

  1. [1]

    MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learn- ing agents on machine learning engineering.arXiv preprint arXiv:2410.07095,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Sela: Tree-search enhanced llm agents for automated machine learning.arXiv preprint arXiv:2410.17238,

    Yizhou Chi, Yizhang Lin, Sirui Hong, Duyi Pan, Yaying Fei, Guanghao Mei, Bangbang Liu, Tianqi Pang, Jacky Kwok, Ceyao Zhang, et al. Sela: Tree-search enhanced llm agents for automated machine learning.arXiv preprint arXiv:2410.17238,

    work page arXiv

  3. [3]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities.arXiv preprint arXiv:2507.06261,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data

    Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexan- der Smola. Autogluon-tabular: Robust and accurate automl for structured data.arXiv preprint arXiv:2003.06505,

    work page internal anchor Pith review Pith/arXiv arXiv 2003

  5. [5]

    Nasvit: Neural architecture search for efficient vision transformers with gradient conflict-aware supernet training.ICLR Proceedings 2022,

    Chengyue Gong and Dilin Wang. Nasvit: Neural architecture search for efficient vision transformers with gradient conflict-aware supernet training.ICLR Proceedings 2022,

    work page 2022

  6. [6]

    Large language models orchestrating structured reasoning achieve kaggle grandmaster level.arXiv preprint arXiv:2411.03562,

    Antoine Grosnit, Alexandre Maraval, James Doran, Giuseppe Paolo, Albert Thomas, Refinath Shahul Hameed Nabeezath Beevi, Jonas Gonzalez, Khyati Khandelwal, Ignacio Iacobacci, Ab- delhakim Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level.arXiv preprint arXiv:2411.03562,

    work page arXiv

  7. [7]

    DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    10 Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Mlagentbench: Evaluating language agents on ma- chine learning experimentation.arXiv preprint arXiv:2310.03302, 2023

    Qian Huang, Jian V ora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation.arXiv preprint arXiv:2310.03302,

    work page arXiv

  9. [9]

    OpenAI o1 System Card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Llm perfor- mance predictors are good initializers for architecture search.arXiv preprint arXiv:2310.16712, 2023a

    Ganesh Jawahar, Muhammad Abdul-Mageed, Laks VS Lakshmanan, and Dujian Ding. Llm perfor- mance predictors are good initializers for architecture search.arXiv preprint arXiv:2310.16712, 2023a. Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, et al. Mixture-of-supernets:...

    work page arXiv

  11. [11]

    Auto-keras: An efficient neural architecture search sys- tem

    Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search sys- tem. InProceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1946–1956,

    work page 1946

  12. [12]

    DSBench : How Far Are Data Science Agents to Becoming Data Science Experts ?, September 2024

    Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts?arXiv preprint arXiv:2409.07703,

    work page arXiv

  13. [13]

    2410.20424 , archiveprefix =

    URLhttps://www. automl.org/wp-content/uploads/2020/07/AutoML_2020_paper_61.pdf. Ziming Li, Qianbo Zang, David Ma, Jiawei Guo, Tuney Zheng, Minghao Liu, Xinyao Niu, Yue Wang, Jian Yang, Jiaheng Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions.arXiv preprint arXiv:2410.20424,

    work page arXiv 2020

  14. [14]

    arXiv preprint arXiv:2506.16499 (2025)

    Zexi Liu, Yuzhu Cai, Xinyu Zhu, Yujie Zheng, Runkun Chen, Ying Wen, Yanfeng Wang, Siheng Chen, et al. Ml-master: Towards ai-for-ai via integration of exploration and reasoning.arXiv preprint arXiv:2506.16499,

    work page arXiv

  15. [15]

    Mle-star: Machine learning engineering agent via search and targeted refinement

    Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan ¨O Arık, and Tomas Pfister. Mle- star: Machine learning engineering agent via search and targeted refinement.arXiv preprint arXiv:2506.15692,

    work page arXiv

  16. [16]

    Automl-agent: A multi-agent llm framework for full-pipeline automl

    Patara Trirat, Wonyong Jeong, and Sung Ju Hwang. Automl-agent: A multi-agent llm framework for full-pipeline automl.arXiv preprint arXiv:2410.02958,

    work page arXiv

  17. [17]

    Lightautoml: Automl solution for a large financial services ecosystem

    Anton Vakhrushev, Alexander Ryzhkov, Maxim Savchenko, Dmitry Simakov, Rinchin Damdinov, and Alexander Tuzhilin. Lightautoml: Automl solution for a large financial services ecosystem. arXiv preprint arXiv:2109.01528,

    work page arXiv

  18. [18]

    R&d-agent: Automating data-driven ai solution building through llm-powered automated research, development, and evolution, 2025

    11 Xu Yang, Xiao Yang, Shikai Fang, Bowen Xian, Yuante Li, Jian Wang, Minrui Xu, Haoran Pan, Xin- peng Hong, Weiqing Liu, et al. R&d-agent: Automating data-driven ai solution building through llm-powered automated research, development, and evolution.arXiv preprint arXiv:2505.14738,

    work page arXiv

  19. [19]

    percent users beaten

    The benchmark comprises 26 Kaggle competitions, totaling 10.2 GB in size, and is divided into two distinct parts. The first part includes competitions from MLE-Bench that remain open for submissions on Kaggle and are each under 1 GB. These span from 2014 to 2017 and primarily feature straightforward tasks, where strong 12 Algorithm 1Adding Stage at Iterat...

    work page 2014

  20. [20]

    For KompeteAI, we empirically selected a set of hyperparameters that strike a balance between the quality of component exploration and the computational time allocated to each

    For AIDE and the RD-agent, we retained their default configurations as specified in the original implementations except for the time limit. For KompeteAI, we empirically selected a set of hyperparameters that strike a balance between the quality of component exploration and the computational time allocated to each. This tuning was guided by the need to en...

    work page arXiv 2013

  21. [21]

    The full timeout parameter defines the overall time limit for the system

    The debug timeout parameter sets the maximum time, in seconds, that is allowed to debug one generated code. The full timeout parameter defines the overall time limit for the system. The if action choosing based on UCB flag determines whether the agents select their actions using the Upper Confidence Bound (UCB) strategy. The enable knowledge base flag ind...

    work page 2000

  22. [22]

    The max debug depth specifies the maximum depth for recursive code debugging

    The steps parameter defines the maximum number of steps the entire system can perform during a run. The max debug depth specifies the maximum depth for recursive code debugging. The debug prob parameter suggests that debugging is enabled for every generated code, meaning that all relevant information will be recorded without any sampling. Finally, the tim...

    work page 2000