REVIEW 3 major objections 7 minor 26 references
LightAutoDS-Tab: Multi-AutoML Agentic System for Tabular Data
T0 review · 3 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read LightAutoDS-Tab pairs an LLM coding agent with the AutoML tools LightAutoML and FEDOT, and reports an average normalized performance score of 0.839 on eight Kaggle tasks, ahead of AutoKaggle (0.816) and AIDE (0.703).
desk verdict Useful engineering integration, but the paper never evaluates the router-based system it claims to beat the state of the art with. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a selector: a query enters through an Interactor agent, which decides whether the user is asking for dialogue or for an ML pipeline, and if a pipeline is wanted, a router chooses between two routes. In the LLM-driven route, Planner, Generator, Validator, and Improver agents iterate over a skeleton script until validation passes; in the AutoML route, an LLM reads the task text, target column, and data head and emits a JSON configuration that launches LightAutoML (LAMA) or FEDOT. Scores are measured with the normalized performance score, $\mathrm{NPS}=1/(1+s)$ when a lower metric is better and $\mathrm{NPS}=s$ otherwise, the same formula AutoKaggle uses, so the reported averages are directly comparable across systems. The system also produces per-step technical and non-technical reports and can export deployment-ready prediction code alongside trained models.
What would settle it
Run the complete LightAutoDS-Tab system, router included, on the same eight Kaggle tasks, recording the route chosen and the score achieved per task. The claim would be settled by comparing the routed system's realized average against the reported 0.839 and against the per-task best configuration: if the router ever selects a route that is worse than the best available tool on that task, or if the routed average falls below 0.839, the headline number describes the system's component ceiling rather than its actual end-to-end behavior.
Extended reading notes
Core claim
LightAutoDS-Tab claims to establish that a router-equipped team of LLM agents plus AutoML tools can beat pure-LLM agent systems on tabular Kaggle tasks. On the eight-competition benchmark imported from AutoKaggle, the best configuration column (LAMA+LLM) reaches an average normalized performance score of 0.839, above AutoKaggle at 0.816 and AIDE at 0.703 and just above the 0.836 human-leaderboard median reported in the same table. The paper also reports that no single tool configuration wins all eight tasks, which it gives as the justification for keeping several AutoML tools in one system, and that the CodeGen route's score shifts with the underlying LLM (0.832 for GPT-4o versus 0.835 for GigaChat2Max). The authors frame the contribution as improving automation and robustness of pipeline construction, explicitly noting that the system does not yet directly improve the accuracy of the AutoML tools themselves.
Load-bearing premise
The load-bearing premise is that the headline 0.839 average, the score of the best single tool configuration LAMA+LLM, stands for the full LightAutoDS-Tab system, even though no experiment runs the router that is supposed to choose the LLM-driven or AutoML route for each task, so a router that picks the weaker route could leave the integrated system below its best column.
Editorial extensions
If this is right
- Hybrid systems that reserve LLMs for understanding and configuration while delegating model fitting to AutoML tools become a credible design for automated tabular data science.
- The best configuration lands between the human median (0.836) and third quartile (0.841) on the shared benchmark, indicating that near-human-level scores are reachable without manual pipeline code.
- Because no single tool wins all eight tasks, the multi-tool design is load-bearing: collapsing to one tool would sacrifice the tasks where the other route is strongest.
- The reported LLM sensitivity in the CodeGen route means the headline results are tied to the specific language model used; swapping the backend shifts the average.
- Exportable deployment code plus a natural-language interface translate the claimed automation into a tool usable by non-experts.
Reading between the lines
- An untested consequence: the paper reports its three tool configurations but never the router's own end-to-end choices, so the realized system's score is an open question; a run that records route selections per task would show how much of the 0.839 the router actually captures.
- A testable extension the paper does not run: an oracle assignment that gives each task its best-performing tool would fix the upper bound the router is aiming for, making per-task routing quality directly measurable on the same eight benchmarks.
- If the division-of-labor thesis generalizes, adding more AutoML tools with complementary strengths (the paper names AutoGluon as an easy addition) should lift the ceiling; re-running the benchmark with an expanded tool set would test that.
- The paper restricts itself to tabular data and flags time-series and sequential data as future work; the same router-plus-AutoML architecture is a plausible template for forecasting and recommendation tasks if the mechanism carries over.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LightAutoDS-Tab, a multi-AutoML agentic system for tabular data that combines an LLM-driven code generation route with AutoML routes based on LightAutoML and FEDOT. A router is intended to select between these routes, and specialized agents handle planning, generation, validation, improvement, execution, interpretation, and reporting. The experimental section (Section 4.2, Table 1) reports normalized performance scores for the three constituent tool configurations on eight Kaggle datasets, and also lists scores for AutoKaggle and AIDE taken directly from prior publications, along with human leaderboard quartiles. The best average score among the three configurations is 0.839 (LAMA+LLM), which the abstract and conclusion attribute to the LightAutoDS-Tab system as a whole. A second table compares two LLM backbones for the CodeGen route. No end-to-end run of the full router-based system is reported, and the paper's own Section 3.2 states that the system may not yet improve AutoML model accuracy.
Significance. The paper tackles a timely problem: integrating LLM agents with classical AutoML tools to make tabular ML pipelines more flexible and accessible. Its strengths include a modular agent architecture, an open-source implementation, a natural-language interface, and export of production-ready code, all of which could make it practically useful. If the full system were evaluated end-to-end and the accuracy claim were supported by properly matched baselines with variance estimates, this would be a useful contribution to the growing literature on AutoML agents. As it stands, however, the experimental evidence does not directly test the system that is advertised, and the comparison to baselines is not statistically grounded, so the significance of the numerical results is uncertain.
major comments (3)
- [Section 3.1 (Algorithm 1); Section 4.2 (Table 1)] The headline claim in the abstract and conclusion that 'LightAutoDS-Tab outperforms state-of-the-art open-source solutions' is not supported by the experiments as reported, because Table 1 evaluates the three constituent tool configurations (LAMA+LLM, CodeGen, FEDOT+LLM) separately and never evaluates the integrated system that uses the router described in Section 3.1. The router's decision is load-bearing: the routing prompt in Appendix A.3 defaults most queries to CodeGen ('NO' unless the user explicitly mentions automl, LightAutoML, or FEDOT), so an end-to-end run could plausibly achieve no more than the CodeGen column's 0.835, or even less if the router makes a suboptimal choice on some datasets. The 0.839 average cited in the abstract is the best single-column result, not the score of the system being introduced. Please provide an end-to-end evaluation with the router enabled (or a documented routing policy and its success rate), or revise the claim to describe the individual tools.
- [Section 4.1] The baselines AutoKaggle and AIDE are not re-run; Section 4.1 states 'We directly include the normalized results from the aforementioned work.' The comparison therefore mixes potentially incompatible experimental conditions (different validation splits, compute budgets, and possibly different versions of GPT-4o). No error bars, multiple seeds, or significance tests are reported, and the observed differences (0.839 versus 0.816 for AutoKaggle and 0.703 for AIDE, while the CodeGen and FEDOT+LLM columns are 0.835) are small enough that they could easily be within run-to-run noise. Please report variance across at least a few seeds, or re-run baselines under identical conditions; if that is not feasible, the manuscript should explicitly characterize the comparison as a literature comparison and soften the outperformance wording.
- [Section 3.2] The manuscript states that 'the current LightAutoDS-Tab system may not yet be capable of directly improving AutoML model accuracy,' yet the abstract claims the system 'outperforms state-of-the-art open-source solutions.' If the contribution is primarily about automation, robustness, and interpretability rather than final accuracy, the accuracy-based claim in the abstract and in Section 5's conclusion should be revised accordingly. As written, the admission directly weakens the load-bearing claim, and the conclusion cites Table 1 as evidence while Table 1 describes only the constituent tools, not the integrated system.
minor comments (7)
- [Algorithm 1] Lines 20-21 of Algorithm 1 contain apparent typos: the assignments 'Avla ← Agen' and 'Aimpr ← Avla' should presumably read 'Aval ← Agen' and 'Aimpr ← Aval'.
- [Equation (1)] The definition of NPS does not make clear whether a higher value is always better; please give an explicit statement that NPS is always to be maximized and clarify what s denotes for each metric type.
- [Section 4.1] The text says there are 7 classification and 1 regression task, plus 1 multi-target task, which sums to 9, while 8 datasets are listed in Table 1; please reconcile this count and describe the multi-target dataset explicitly.
- [Section 4.1 (Table 1)] The 'Human results from leaderboards' columns (Q25, Q50, Q75) are not described; please state how many leaderboard entries were used to compute these percentiles and whether the distributions are based on all teams or a filtered subset.
- [Abstract; Section 3.4] The abstract and Section 3.4 promise reduced development time and improved efficiency, but the paper reports no runtime, token-cost, or wall-clock measurements; please add at least one such comparison to support the efficiency claim.
- [Section 3.2] The statement that LLMs 'cannot produce functional code consistently' is presented without supporting data; please report the number of code-generation failures or refinement cycles observed in the experiments, or soften the wording.
- [Section 3.1; Appendix A.3] The router is described as an intelligent selection mechanism, but the appendix prompt only checks for explicit tool names; this discrepancy should be discussed in the main text.
Circularity Check
No significant circularity: the empirical comparison is self-contained and externally falsifiable.
full rationale
LightAutoDS-Tab is an empirical systems paper and contains no formal derivation for which an output could be equivalent to an input by construction. The central claim of outperforming state-of-the-art open-source solutions is supported by Table 1, which reports normalized performance scores for three LightAutoDS-Tab tool configurations (LAMA+LLM, CodeGen, FEDOT+LLM) and compares them with AutoKaggle and AIDE scores directly copied from Li et al. (2024). The reported 0.839 average is a measured result on eight public Kaggle datasets, not a fitted parameter or a quantity defined in terms of the claim it is meant to support. The only author-related citations are to LightAutoML (Vakhrushev et al., 2021) and FEDOT (Nikitin et al., 2022), which are used as concrete open-source AutoML tools within the system; this is a benchmark usage of one's own tools rather than a load-bearing self-citation, and the results remain externally falsifiable against the public Kaggle leaderboards. The skeptic's concern that Table 1 evaluates the constituent routes rather than the full router-based system is an evaluation-validity limitation, not a circularity: the 0.839 number is not defined in terms of the headline claim, and no equation or fitted parameter reduces to its own input. The paper itself acknowledges in Section 3.2 that the current system may not directly improve AutoML model accuracy, which further shows the performance numbers are empirical outputs rather than consequences of the system's definition. Therefore no circular step is identified, and the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Kaggle competition datasets and leaderboards are representative of real-world tabular data science problems.
- domain assumption The normalized performance score (NPS) is a valid metric to compare solutions across classification and regression tasks.
- domain assumption Normalized results from prior papers (AutoKaggle, AIDE) are directly comparable to the authors' runs without identical protocols.
- ad hoc to paper The performance of the best single tool column in Table 1 represents the performance of the full LightAutoDS-Tab system.
Cite this review
Pith. "Pith review of LightAutoDS-Tab: Multi-AutoML Agentic System for Tabular Data." pith.science (2026). https://pith.science/paper/SM44NDPA
@misc{pith2026250713413,
author = {Pith},
title = {Pith review of: LightAutoDS-Tab: Multi-AutoML Agentic System for Tabular Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/SM44NDPA}},
note = {Machine review of arXiv:2507.13413}
}
read the original abstract
AutoML has advanced in handling complex tasks using the integration of LLMs, yet its efficiency remains limited by dependence on specific underlying tools. In this paper, we introduce LightAutoDS-Tab, a multi-AutoML agentic system for tasks with tabular data, which combines an LLM-based code generation with several AutoML tools. Our approach improves the flexibility and robustness of pipeline design, outperforming state-of-the-art open-source solutions on several data science tasks from Kaggle. The code of LightAutoDS-Tab is available in the open repository https://github.com/sb-ai-lab/LADS
Figures
Reference graph
Works this paper leans on
-
[1]
Do **not delete any comments** in the provided code
-
[2]
This code is autogenerated and will be regenerated upon project restarts
Do **not modify code enclosed between** the designated markers ( `### comment ### code ### comment ###`). This code is autogenerated and will be regenerated upon project restarts
-
[3]
You are **prohibited from using any methods or attributes** from the Fedot framework classes (e.g ., `Fedot`, `Pipeline`), except those that are **already used in the provided code** or ** explicitly mentioned in the comments**
-
[4]
You are allowed to write and modify code **only within the 'USER CODE ' sections**. All other sections will be regenerated upon project restarts. Write the whole code below. ```python A.2 LightAutoML-specific prompts You are an experienced machine learning developer who can formulate tasks in machine learning terms. Your task is to create a configuration ...
-
[5]
Target Variable: Identify the target variable that needs to be predicted or optimized, which is provided in the training set but not in the test set
-
[6]
Competition Overview: Understand the background and context of the topic
-
[7]
Files: Analyze each provided file, detailing its purpose and how it should be used in the competition
-
[8]
Problem Definition: Clarify the problem 's definition and requirements
Show all 26 references
-
[9]
4.1 Data type: 4.1.1
Data Information: Gather detailed information about the data, including its structure and contents . 4.1 Data type: 4.1.1. ID type: features that are unique identifiers for each data point, which will NOT be used in the model training. 4.1.2. Numerical type: features that are ...
-
[11]
Evaluation Metrics: Determine the evaluation metrics that will be used to assess the submissions
-
[12]
Submission Format: Understand the required format for the final submission
-
[13]
Ensure that the analysis is thorough, with a strong emphasis on :
Other Key Aspects: Highlight any other important aspects that could influence the approach to the competition. Ensure that the analysis is thorough, with a strong emphasis on :
-
[14]
Understanding the purpose and usage of each file provided
-
[15]
Figuring out the target variable and evaluation metrics
-
[16]
Classification of the features. # Available Data File And Content in The File {data_files_and_content} # EDA {dataset_eda} A.7 Reporter prompt You are an expert in machine learning tasked with evaluating and reporting on an ML model designed to address the problem. Your report...
-
[17]
**Overview** - Problem description - Goal: Summarize the purpose of the model in plain terms for a general audience
-
[18]
Normalization ensures all features are on the same scale, improving model performance. For example, a scaling process converts values like 'age' (5-90) to 0-1
**Data Preprocessing** - Describe the data preprocessing steps used before modeling in plain, accessible language. - Provide illustrative examples to clarify specific preprocessing steps. - If data normalization was applied, describe it as: "Normalization ensures all features ...
-
[19]
**Pipeline Summary** - Summarize the steps in `{pipeline}` using accessible language and optionally include illustrative examples. - Key Parameters: | Model | Parameters | Explanation | |---------|--------------|---------------| | CatBoost| num_trees: 3000, learning_rate: 0.03...
-
[20]
**Code Highlights:** - Include relevant code snippets wrapped in Markdown Python blocks: ```python {code} ``` - Add a brief explanation of what the code does and why it 's a key
-
[21]
Data Preprocessing (Short Key Snippets)
-
[22]
Model Training, Evaluation, Prediction
-
[23]
Submission File Creation
-
[24]
Other Key Snippets 10
-
[25]
Accuracy tells us how often the model gets it right
**Metrics** - Share performance metric `{metrics}` and briefly describe what each metric signifies (e.g., " Accuracy tells us how often the model gets it right"):
-
[26]
This model predicts X with an accuracy of Y%, demonstrating its potential in Z applications
**Takeaways** - Wrap up with a concise summary of results, emphasizing their significance in a real-world context. For example: "This model predicts X with an accuracy of Y%, demonstrating its potential in Z applications." Engage your audience with a relatable, professional to...
-
[2020]
Preprint, arXiv:2003.06505
Autogluon-tabular: Robust and accurate au- toml for structured data. Preprint, arXiv:2003.06505. 6 Haoyang Fang, Boran Han, Nick Erickson, Xiyuan Zhang, Su Zhou, Anirudh Dagar, Jiani Zhang, Ali Caner Turkmen, Cuixiong Hu, Huzefa Rangwala, and 1 others. 2025. MLZero: A multi-ag...
2003 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.