REVIEW 4 major objections 4 minor 23 references
Agentomics-ML: Autonomous Machine Learning Experimentation Agent for Genomic and Transcriptomic Data
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A fully autonomous agent, Agentomics-ML, writes working classification code and trained models for genomic and transcriptomic benchmarks in 28 of 30 runs, outperforming all benchmarked LLM and agentic baselines and beating the published…
desk verdict A genuinely useful benchmark for autonomous ML agents on genomic data, but the test-set isolation claim needs one missing sentence before the headline numbers can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a step-gated workflow wrapped around an LLM agent: a fixed order of data exploration, representation and architecture choice, artifact generation, and training, with programmatic validation of every file output so a failed script blocks progress. Around this, a reflection step reads scalar training and validation metrics and emits verbal instructions for the next iteration, creating a closed loop that corrects overfitting and design errors without ever exposing the held-out test set. The power of the approach lies in the combination of mandatory structure and iterative feedback, not in any single model or prompt.
What would settle it
Re-run Agentomics-ML with the same prompt and hyperparameters on a newly generated, never-published genomic sequence dataset with a known ground truth. If its success rate and test accuracy fall to the level of ordinary zero-shot code generation, or if probing the LLM recovers benchmark-specific sequences and labels, the reported advantage reflects training-data memorization rather than the agent's experimentation loop.
Extended reading notes
Core claim
The core discovery is that a constrained agent following a fixed sequence of machine-learning experimentation steps can solve genomics and transcriptomics classification tasks more reliably than both direct LLM code generation and state-of-the-art agentic AutoML systems. The agent writes Python files and runs shell commands inside an isolated container, and it cannot advance to the next step until its outputs pass automated checks, such as running the inference script on dummy data. After each iteration, a reflection step converts scalar training and validation metrics into verbal feedback that guides the next iteration. In this design, Agentomics-ML produced working code in 93.33% of runs, solved the paired-sequence AGO2_CLASH_Hejret task on which every benchmarked baseline failed, and its best-run test metrics beat all other LLM-based methods on all six datasets and the published human baseline on one.
Load-bearing premise
Every reported performance number assumes the closed-source LLM backbones (notably gpt-4.1) did not memorize the public Genomic Benchmarks and miRBench datasets during pretraining; the authors state plainly that they cannot determine this.
Editorial extensions
If this is right
- Other agentic ML systems could adopt step-gated programmatic validation and the reflection loop to raise success rates on heterogeneous, high-dimensional data.
- If the success rate holds, bioinformatics groups without specialized ML staff can obtain reproducible classifiers on new sequence tasks at a reported average cost under $2 per run.
- The single success on AGO2_CLASH_Hejret suggests the workflow generalizes to paired-sequence interaction problems in which each input alone is non-predictive, a setting that defeats zero-shot generation.
- Strict programmatic test-set hiding makes Agentomics-ML a workable template for leakage-free evaluation of closed-source LLMs on fresh biomedical benchmarks.
Reading between the lines
- A reader could test whether the backbone matters as much as the gating by replacing gpt-4.1 with an open-weight model; the paper's architecture suggests the enforced step validation, not the LLM, is the primary driver of reliability.
- The paper's own t-tests show the feedback loop improves test metrics significantly on only two of the six datasets, so the reflection step's main measurable contribution may be to success rate and robustness rather than peak accuracy.
- The sub-$2 per-run cost implies that large-scale agent-driven benchmarking across many -omics datasets is now affordable, which could let researchers screen datasets for learnability before investing in expert-built models.
- A natural extension the authors do not explore is applying the same step-gated workflow to regression or multi-label tasks such as expression level prediction, where validation and reflection would face different overfitting signals.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Agentomics-ML is an LLM-driven agent that automates end-to-end machine learning experimentation for genomic and transcriptomic sequence classification. The agent follows a preset workflow—data exploration, representation/architecture selection, training, inference-script generation—and interacts with the filesystem through Bash and Python tools inside an isolated Docker container, with a reflection loop that converts scalar training/validation metrics into verbal feedback for subsequent iterations. The authors evaluate the system on six public benchmarks from Genomic Benchmarks and miRBench, reporting success rates (93.33% for Agentomics-ML vs. lower rates for zero-shot LLMs, AIDE, and Data Interpreter), test-set performance, and comparisons to published human state-of-the-art, including a claimed SOTA on Drosophila_enhancers_stark. The code and supplementary materials are provided.
Significance. The main contribution is a practical, reproducible agentic workflow with a high rate of producing working code and models, including on AGO2_CLASH_Hejret, a paired/variable-length sequence dataset on which all benchmarked baselines fail. The paper's strengths include public code, programmatic validation of agent outputs, consistent use of held-out test abstraction for the agent, and low runtime cost. If the evaluation is confirmed to be leakage-free, the result is a meaningful step for automated ML in biology. However, the headline 'state-of-the-art' claim rests on two unverified assumptions: that the closed-source LLM backbone was not pretrained on the public benchmarks, and that the container prevented the agent from downloading the benchmarks. Both are addressable, but currently weaken the central claim.
major comments (4)
- [§3.2, §3.4] The paper states that all agent operations occur in a 'secure, isolated Docker container' (§3.2) and that the test set is 'programmatically abstracted' (§3.4), but it never states whether the container has network egress. Because the agent is equipped with a general Bash tool, an enabled network would allow it to download the public Genomic Benchmarks or miRBench datasets, including test labels, and use them for validation or template matching during its iterative loop. This would completely invalidate the leakage-free evaluation claim and all downstream comparisons. The authors should explicitly report the container's network policy, and ideally verify (e.g., with firewall rules or a network-access audit) that no egress was possible during the runs.
- [§4.2, Table 2] The comparison to Human SOTA is not controlled. The text says 'we use metrics reported by those publications to allow for a fair comparison,' but it does not establish that those published numbers were obtained with the same train/test splits, preprocessing, metric definition, or evaluation protocol as in this paper. For example, the claimed win on Drosophila_enhancers_stark (0.736 vs. 0.586) is only meaningful if the same test split was used. The authors must verify and report the exact protocol of the cited SOTA papers, or refrain from claiming to outperform them.
- [§5.4, Tables 1–3] The paper acknowledges that human_nontata_promoters was used as a prototype dataset for prompt/hyperparameter development, yet results for this dataset are included in the main success-rate and performance tables and in the claim of leadership on 'all six datasets.' Results on a development dataset are not independent evidence of generalization; at minimum, the authors should either exclude NTP from the headline claims or demonstrate via a held-out control that the prototyping did not inflate its reported performance.
- [§5.1, Table 2] Pretraining contamination is acknowledged as a limitation, but it is also load-bearing for the central claim. Since gpt-4.1 and other closed backbones may have been trained on the public benchmarks, the 'best LLM-based performance' and 'human SOTA' results could partially reflect memorization rather than autonomous experimentation. The authors should provide a concrete probe, such as evaluating on a never-published dataset (they mention this as future work), or a shuffled-label control, and should condition the state-of-the-art wording on the absence of contamination.
minor comments (4)
- [Tables 1–5] The abbreviations AGO2, DE, HEC, HEE, NTP, and OCRE are used in every table but are never defined in the captions; please expand them or add a table note.
- [§4.2, Table 5] Table 5 is introduced as reporting standard deviation in §4.2 but its caption says 'variance'; clarify the statistic and, if variance is intended, report its square root or correct the caption.
- [§4.2] The statement that feedback improves test metrics '80% of the time' across six datasets is ambiguous (80% of what—runs or datasets?) and should be replaced with an exact count, e.g., '4 of 6 datasets' or the per-run proportion.
- [§4.2, Table 4] The t-test is described without stating whether it is paired and whether assumptions were checked; with only five runs per condition, please specify the test variant and report effect sizes, not only p-values.
Circularity Check
No significant circularity: the agent's test-set results are obtained after programmatic test abstraction, and the self-citations point to external public benchmarks rather than load-bearing derivations.
full rationale
Agentomics-ML is an empirical systems paper rather than a derivation from first principles, so the self-definitional, fitted-prediction, and uniqueness-theorem circularity patterns do not apply. The central quantitative claims, the 93.33% success rate and the reported test-set metrics, are produced by running the agent on training data while the test set is hidden: Section 3.4 states 'the agentic system is programmatically abstracted from the held-out test set throughout its development cycle. Only the final, agent-produced inference script is used to report performance on the test set.' Test labels are therefore not inputs to any fitting or selection procedure; the agent selects models by validation metrics, which is standard practice and does not make the test result an input. The self-citations (Genomic Benchmarks, miRBench, and the Human SOTA baseline from Grešová et al., 2023) cite externally published, publicly available datasets and baselines; they are not invoked to justify design choices or to forbid alternatives, so they are not load-bearing circularity. The pretraining-leakage caveat in Section 5.1 is an acknowledged, unknowable uncertainty about closed-source LLM training corpora, and the 'isolated Docker container' statement in Section 3.2 could raise a network-egress leakage question, but both are data-integrity and security concerns rather than cases where a reported result reduces by construction to its own inputs. No circular step can be exhibited from the paper's own text, so the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption Closed-source LLM outputs are stable enough across API calls that 5-run statistics are representative of method performance.
- domain assumption Public benchmark datasets were not seen by the LLM backbones during pretraining.
- domain assumption Validation metrics are a reliable proxy for test performance during model selection.
Cite this review
Pith. "Pith review of Agentomics-ML: Autonomous Machine Learning Experimentation Agent for Genomic and Transcriptomic Data." pith.science (2026). https://pith.science/paper/DU6P2IFX
@misc{pith2026250605542,
author = {Pith},
title = {Pith review of: Agentomics-ML: Autonomous Machine Learning Experimentation Agent for Genomic and Transcriptomic Data},
year = {2026},
howpublished = {\url{https://pith.science/paper/DU6P2IFX}},
note = {Machine review of arXiv:2506.05542}
}
read the original abstract
The adoption of machine learning (ML) and deep learning methods has revolutionized molecular medicine by driving breakthroughs in genomics, transcriptomics, drug discovery, and biological systems modeling. The increasing quantity, multimodality, and heterogeneity of biological datasets demand automated methods that can produce generalizable predictive models. Recent developments in large language model-based agents have shown promise for automating end-to-end ML experimentation on structured benchmarks. However, when applied to heterogeneous computational biology datasets, these methods struggle with generalization and success rates. Here, we introduce Agentomics-ML, a fully autonomous agent-based system designed to produce a classification model and the necessary files for reproducible training and inference. Our method follows predefined steps of an ML experimentation process, repeatedly interacting with the file system through Bash to complete individual steps. Once an ML model is produced, training and validation metrics provide scalar feedback to a reflection step to identify issues such as overfitting. This step then creates verbal feedback for future iterations, suggesting adjustments to steps such as data representation, model architecture, and hyperparameter choices. We have evaluated Agentomics-ML on several established genomic and transcriptomic benchmark datasets and show that it outperforms existing state-of-the-art agent-based methods in both generalization and success rates. While state-of-the-art models built by domain experts still lead in absolute performance on the majority of the computational biology datasets used in this work, Agentomics-ML narrows the gap for fully autonomous systems and achieves state-of-the-art performance on one of the used benchmark datasets. The code is available at https://github.com/BioGeMT/Agentomics-ML.
Figures
Reference graph
Works this paper leans on
-
[1]
A Multivocal Literature Review on the Benefits and Limitations of Automated Machine Learning Tools
Kelly Azevedo, Luigi Quaranta, Fabio Calefato, and Marcos Kalinowski. A multivocal literature review on the benefits and limitations of automated machine learning tools, 2024. URL https://arxiv.org/abs/2401.11366
work page Pith review arXiv 2024
-
[2]
Mahesh Kumar Bharti, Deepika Chandra, R.A. Siddique, K. Ranjan, and Pankaj Kumar. Recent advancement in high-throughput “omics” technologies, page 343–355. Elsevier, 2024. ISBN 9780443216251. doi:10.1016/b978-0-443-21625-1.00023-3. URL http://dx.doi.org/10.1016/B978-0-443-21625-1.00023-3
-
[3]
Beyond Random Split for Assessing Statistical Model Performance
Carlos Catania, Jorge Guerra, Juan Manuel Romero, Gabriel Caffaratti, and Martin Marchetta. Beyond random split for assessing statistical model performance, 2022. URL https://arxiv.org/abs/2209.03346
work page Pith review arXiv 2022
-
[4]
Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025. URL https://arxiv.org/abs/2410.07095
arXiv 2025
-
[5]
Auto-sklearn 2.0: hands-free automl via meta-learning
Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. Auto-sklearn 2.0: hands-free automl via meta-learning. J. Mach. Learn. Res., 23 0 (1), January 2022. ISSN 1532-4435
work page 2022
-
[6]
Genomic benchmarks: a collection of datasets for genomic sequence classification
Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24 0 (1), May 2023. ISSN 2730-6844. doi:10.1186/s12863-023-01123-8. URL http://dx.doi.org/10.1186/s12863-023-01123-8
-
[7]
Data interpreter: An llm agent for data science, 2024 a
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zhibin Gou, Zongze Xu, and Chenglin Wu. Data interprete...
arXiv 2024
-
[8]
Meta GPT : Meta programming for a multi-agent collaborative framework
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J \"u rgen Schmidhuber. Meta GPT : Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representatio...
work page 2024
Show all 23 references
-
[9]
Mlagentbench: Evaluating language agents on machine learning experimentation, 2024 a
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 2024 a . URL https://arxiv.org/abs/2310.03302
2024 arXiv
-
[10]
Understanding the planning of llm agents: A survey, 2024 b
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of llm agents: A survey, 2024 b . URL https://arxiv.org/abs/2402.02716
2024 arXiv
-
[11]
Aide: Ai-driven exploration in the space of code, 2025
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code, 2025. URL https://arxiv.org/abs/2502.13138
2025 arXiv
-
[12]
Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna, 2024
Lifeng Qiao, Peng Ye, Yuchen Ren, Weiqiang Bai, Chaoqi Liang, Xinzhu Ma, Nanqing Dong, and Wanli Ouyang. Model decides how to tokenize: Adaptive dna sequence tokenization with mxdna, 2024. URL https://arxiv.org/abs/2412.13716
2024 arXiv
-
[13]
mirbench: novel benchmark datasets for microrna binding site prediction that mitigate against prevalent microrna frequency class bias
Stephanie Sammut, Katarina Gresova, Dimosthenis Tzimotoudis, Eva Marsalkova, David Cechak, and Panagiotis Alexiou. mirbench: novel benchmark datasets for microrna binding site prediction that mitigate against prevalent microrna frequency class bias. bioRxiv, 2025. doi:10.1101/...
2025 doi
-
[14]
Robinson, and Giorgio Valentini
Max Schubach, Matteo Re, Peter N. Robinson, and Giorgio Valentini. Imbalance-aware machine learning for predicting rare and common disease-associated non-coding variants. Scientific Reports, 7 0 (1), June 2017. ISSN 2045-2322. doi:10.1038/s41598-017-03011-5. URL http://dx.doi....
2017 doi
-
[15]
Reflexion: Language agents with verbal reinforcement learning, 2023
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366
2023 arXiv
-
[16]
A survey on large language model-based agents for statistics and data science, 2024
Maojun Sun, Ruijian Han, Binyan Jiang, Houduo Qi, Defeng Sun, Yancheng Yuan, and Jian Huang. A survey on large language model-based agents for statistics and data science, 2024. URL https://arxiv.org/abs/2412.14222
2024
-
[17]
Automl in the wild: Obstacles, workarounds, and expectations
Yuan Sun, Qiurong Song, Xinning Gui, Fenglong Ma, and Ting Wang. Automl in the wild: Obstacles, workarounds, and expectations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, page 1–15. ACM, April 2023. doi:10.1145/3544548.3581082. URL...
2023
-
[18]
MSAM amba: Adapting subquadratic models to long-context DNA MSA analysis
Vishrut Thoutam and Dina Ellsworth. MSAM amba: Adapting subquadratic models to long-context DNA MSA analysis. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024. URL https://openreview.net/forum?id=PogdVDgGFT
2024
-
[19]
o rheide, Jan Krumsiek, Gabi Kastenm\
Maria A. W\" o rheide, Jan Krumsiek, Gabi Kastenm\" u ller, and Matthias Arnold. Multi-omics integration in biomedical research – a metabolomics-centric review. Analytica Chimica Acta, 1141: 0 144–162, January 2021. ISSN 0003-2670. doi:10.1016/j.aca.2020.10.038. URL http://dx....
2021 doi
-
[20]
Revealing the barriers of language agents in planning, 2024
Jian Xie, Kexun Zhang, Jiangjie Chen, Siyu Yuan, Kai Zhang, Yikai Zhang, Lei Li, and Yanghua Xiao. Revealing the barriers of language agents in planning, 2024. URL https://arxiv.org/abs/2410.12409
2024 arXiv
-
[21]
Chunming Xu and Scott A. Jackson. Machine learning and complex biological data. Genome Biology, 20 0 (1), April 2019. ISSN 1474-760X. doi:10.1186/s13059-019-1689-0. URL http://dx.doi.org/10.1186/s13059-019-1689-0
2019 doi
-
[22]
Self-distillation improves dna sequence inference, 2024 a
Tong Yu, Lei Cheng, Ruslan Khalitov, Erland Brandser Olsson, and Zhirong Yang. Self-distillation improves dna sequence inference, 2024 a . URL https://arxiv.org/abs/2405.08538
2024 arXiv
-
[23]
Assessing and mitigating batch effects in large-scale omics studies
Ying Yu, Yuanbang Mai, Yuanting Zheng, and Leming Shi. Assessing and mitigating batch effects in large-scale omics studies. Genome Biology, 25 0 (1), October 2024 b . ISSN 1474-760X. doi:10.1186/s13059-024-03401-9. URL http://dx.doi.org/10.1186/s13059-024-03401-9
2024 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.