Pith. sign in

REVIEW 4 major objections 6 minor 69 references

A nested agent design splits strategy from execution so long-horizon machine-learning engineering no longer drowns in context.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 01:36 UTC pith:E5FNZT7K

load-bearing objection Solid hierarchical MLE-agent systems paper with real training recipe and decent controls; the hierarchy-vs-monolith gains are partly confounded by unmatched exploration budget under a shared wall-clock limit. the 4 major comments →

arxiv 2607.25090 v1 pith:E5FNZT7K submitted 2026-07-27 cs.AI cs.LG

Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering

Rushi Qiang , Changhao Li , Haotian Sun , Yuchen Zhuang , Chao Zhang , Bo Dai This is my paper
classification cs.AI cs.LG
keywords hierarchical agentsmachine learning engineeringlong-horizon decision makingorchestrator sub-agentsolution refinement treeonline ranking RLcontext managementMLE agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Machine-learning engineering is a long loop of code, run, debug, and score under costly feedback. A single agent that keeps every log and code trace soon faces noisy, ballooning context and limited capacity. This paper argues that the fix is hierarchical: a persistent Orchestrator keeps only compact scores and summaries and chooses what to try next, while short-lived Sub-Agents implement each attempt in a fresh context through a Tool layer. Training builds a solution-refinement tree that reuses shared prefixes and ranks branches by best downstream score, so the Orchestrator learns from expensive runs without seeing raw traces. On a broad MLE benchmark the design lifts small open models toward frontier closed models as orchestrators and yields large gains for a 30B coder, while keeping context growth far slower than a monolithic agent.

Core claim

Matryoshka Agent is an effective, scalable paradigm for long-horizon MLE: decoupling strategic refinement from concrete execution, then training the Orchestrator with tree-based preference signals and the Sub-Agents on successful runs, produces consistent HumanRank gains across model scales—including a 4B Orchestrator matching a strong closed model’s orchestration score, and up to about 37% relative gain for a 30B coder over its monolithic baseline.

What carries the argument

Matryoshka Agent: a three-layer loop (Orchestrator decides parent attempt, instruction, and references; Tools build fresh Sub-Agent contexts and return scores plus structured summaries; Sub-Agents run bounded code–debug loops). Training uses binary Solution Refinement Trees labeled by branch-level max downstream return, optimized with a ranking NCE / log-ratio objective for the Orchestrator plus SFT self-reinforcement on winning Sub-Agent trajectories.

Load-bearing premise

Compact tool summaries and scalar scores, without raw code or full debug traces in the Orchestrator’s memory, still carry enough signal for better long-horizon choices than an agent that sees the full history.

What would settle it

On the same MLE-Dojo split and budgets, ablate the information boundary: give the Orchestrator raw logs or full Sub-Agent traces (or strip structured summaries to scores only) and check whether All HumanRank and long-horizon rank still beat the matched monolithic Dojo Agent; if hierarchy loses under richer context, the boundary assumption fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Small models can act as high-level decision layers when paired with stronger executors, without carrying full execution history.
  • Orchestrators trained with one Sub-Agent stack can be swapped onto other executors at test time and still improve them.
  • Context growth stays roughly linear and slower than a monolithic agent, supporting more refinement rounds under fixed context budgets.
  • Tree sampling that reuses shared prefixes makes preference learning affordable when each environment run is expensive.
  • The same unfold-orchestrate pattern is offered as a general template for other long-horizon agent tasks beyond MLE.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same summary-mediated hierarchy may transfer to other costly iterative loops—kernel tuning, post-training search, or paper-replication agents—where full traces swamp strategy.
  • If Sub-Agent capacity is the bottleneck, further gains may come more from executor specialization than from longer Orchestrator instructions.
  • Binary refinement trees trade exploration breadth for label quality; wider branching or adaptive depth could change the cost–signal tradeoff on harder tasks.
  • Plug-and-play Orchestrators suggest a market-like split: train cheap decision policies once, rent strong code models only for execution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Matryoshka Agent, a three-layer hierarchical agent for long-horizon machine-learning engineering: an Orchestrator keeps a compact score-annotated history and chooses refinement directions; Tools package instructions/references and launch isolated Sub-Agents; Sub-Agents execute/debug code and return summaries plus scores. Training uses Solution Refinement Trees with binary branching, branch labels R(c)=max_{v in subtree(c)} r(v), and a DPO/NCE-like log-ratio ranking loss (Eqs. 1-4), plus SFT from successful Sub-Agent trajectories. On MLE-Dojo's 150/50 split, the authors report gains over an adapted Dojo Agent across several open/closed model pairings, an SFT-Dojo control, RL-without-SFT and orchestrator/sub-agent swap ablations, plug-and-play transfer to GPT-5-nano Sub-Agents, and context-growth analysis.

Significance. If the results hold, this is a useful and timely systems contribution: it gives a concrete way to decouple strategic search from execution in expensive MLE loops, shows a plausible route for small models to act as orchestrators, and connects inference-time scaffolding to an online preference-optimization recipe. Strengths worth credit: evaluation on held-out MLE-Dojo tasks with an external HumanRank metric; multiple model pairings; an adapted monolithic baseline; an SFT-Dojo control; orchestrator/sub-agent attribution ablations; plug-and-play tests; context-length scaling analysis; and Appendix D's check that mean-of-three preserves the displayed ordering. The main risk is not circularity in the metric but causal attribution: some architecture-vs-baseline comparisons may reflect exploration/context-reset budget rather than hierarchy per se.

major comments (4)
  1. [§4.1, §3.4, App. B/F, Table 1] The key untrained comparisons share only a 12h wall-clock budget, not matched attempts/tokens. §3.4 explicitly allows parallel tool calls, each Sub-Agent gets a fresh context with up to 10 debugging iterations (App. B), and App. F estimates 213 vs 137 supported rounds. Under the max-over-attempts objective (§3.1), more valid attempts mechanically raise expected best score. Please report per-task valid-attempt counts, total/generated tokens, serial vs parallel execution, and add matched-attempt/matched-token controls plus a monolithic baseline with periodic context resets/restarts. This is load-bearing for the inference-only claims (0.3302→0.3652 and 0.4832→0.5465).
  2. [§4.1, App. D, Tables 1 and 5] Headline Table 1 uses per-task best-of-three, while Appendix D's mean-of-three check contains only four rows and omits several rows needed for the strongest claims (e.g., Qwen3-30B-Coder Dojo, SFT-Dojo, untrained/SFT Matryoshka-30B, Qwen3-4B-SFT). Given median per-task SD ≈0.07, gaps such as 0.3652 vs 0.3302 and some category-level differences are within or near run noise, and best-of-three favors higher-variance/higher-attempt systems. Provide mean±SD for every Table 1/2 row, paired per-task statistics, and best-of-k sensitivity for k=1,2,3.
  3. [§3.6, Eqs. (1)-(4)] The optimization derivation is DPO-like, but the preference labels are deterministic branch-return winners from R(c)=max subtree r(v). Eq. (1) presupposes a stochastic reward/Bradley-Terry noise model with temperature β; with hard labels and binary C(u), β is not identifiable and Eq. (4) should be presented as a contrastive algorithmic objective rather than a principled posterior unless a noise model is specified. Also discuss credit assignment: branch-return labeling can credit/blame an edge for later descendants under the current policy. A small labeling ablation (immediate score vs global winner vs branch return) would substantiate the design axiom.
  4. [§4.3, Table 2] The plug-and-play claim is not isolated. Table 2 compares GPT-5-nano Dojo (0.3946) with trained RL Orchestrators plus GPT-5-nano (0.4309/0.4626), but omits Matryoshka with an untrained Orchestrator plus GPT-5-nano. Thus the table cannot distinguish 'trained orchestrator transfers' from 'Matryoshka scaffolding helps GPT-5-nano.' Add the untrained-orchestrator + GPT-5-nano control and, if possible, an SFT-only orchestrator control.
minor comments (6)
  1. [App. A, Eq. (5)] HumanRank s=1-p/N needs tie handling, treatment of invalid/missing submissions, whether N includes the agent, and exact public/private averaging. An off-by-one convention (rank 1 gives 1-1/N) should be stated explicitly.
  2. [Fig. 2 / App. F] Define 'round' consistently for Orchestrator, Sub-Agent, and Dojo Agent; state whether Sub-Agent tokens are per-round averages or sums; include base prompt and output-token growth in the 213/137 round estimate, which currently divides 256k only by input-history slope.
  3. [Tables 2-4] Table 2's model-configuration cells appear malformed ('Qwen3-4B-RL +' with the Sub-Agent implicit). Name GPT-5-nano explicitly in each row. In Table 3, define 'O' inline rather than relying on Table 1.
  4. [Writing] Several typos/grammar issues: 'Matryoshka Agent further develop' (§1), 'the the underlying model' (§4.1), 'results of the refinement' phrasing (§3.3), and inconsistent Qwen3-Coder-30B/Qwen3-Coder-30B-A3B naming.
  5. [§3.6 / Related Work] Because Eqs. (1)-(4) closely follow DPO-style log-ratio reparameterization and NCE ranking, cite DPO/Rafailov et al. and relevant preference-optimization/NCE work, and clarify differences from KTO/IPO-style objectives.
  6. [Reproducibility] Please state code/config/seed availability, closed-model versions/snapshots for o4-mini/GPT-5-nano/Gemini-2.5-Flash, exact 12h enforcement, package installation policy, and whether evaluation used parallel tool calls in the reported runs.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose gains are measured on held-out MLE-Dojo HumanRank, not forced by definition or self-citation.

full rationale

Matryoshka Agent is an architecture-plus-training paper, not a first-principles derivation. The objective (§3.1) is max_t r_t from an external execution environment E; preference labels for the Orchestrator (§3.6) are branch-level downstream returns R(c)=max_{v∈subtree(c)} r(v) from those same environment scores, and evaluation HumanRank is the fraction of human Kaggle competitors outperformed on held-out tasks. That chain does not rewrite the metric into the hierarchy by construction: the ranking NCE loss (Eqs. 1–4) optimizes policy log-ratios against observed execution outcomes, and Table 1/2 gains are external leaderboard comparisons. Self-citations (MLE-Dojo, Matryoshka-Pilot, MLE-Smith) supply the benchmark and prior modular scaffolding but do not force the reported All HumanRank numbers or the 36.7% relative gain. No uniqueness theorem, fitted-parameter-as-prediction, or ansatz-via-self-citation step load-bears the central claim. Minor author-overlap citations are normal and non-circular here. (Separate confounds about unmatched attempt/parallelism budgets vs Dojo Agent are experimental-validity issues, not circularity.)

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 3 invented entities

Load-bearing content is architectural and empirical, not a short formal derivation. The claim rests on domain choices about how MLE feedback is exposed, how much history the orchestrator may see, how preferences are labeled from trees, and evaluation via HumanRank on MLE-Dojo. Free parameters are training and scaffold knobs; invented entities are the named layers of the framework; axioms are standard RL/LM and benchmark assumptions.

free parameters (7)
  • beta (KL/temperature in ranking NCE score)
    Controls strength of regularization toward the previous orchestrator policy in Eqs. (1)–(4); chosen as a training hyperparameter, not derived.
  • binary expansion width and tree depth = 2 children per node
    Two children per non-leaf node for tractable collection; depth/budget choices shape preference data and are design knobs.
  • Sub-Agent debugging budget = 10 iterations
    Caps low-level repair loops before failure is returned to the orchestrator (stated as up to 10 iterations).
  • wall-clock attempt budget = 12 hours
    12-hour per-task limit jointly constrains exploration for Matryoshka and Dojo agents.
  • environment feedback truncation = 5k with 2.5k+2.5k keep
    When env output exceeds 5k tokens, keep first and last 2.5k; affects what both agents observe.
  • SFT/RL optimization hyperparameters = lr 1e-5, 2 epochs (tables)
    Learning rates, epochs, batch sizes, parallel sizes in Tables 6–7 are fitted/selected for training stability and final scores.
  • number and source of SFT trajectories = 100 tasks / trajectories
    100 teacher trajectories from gemini-2.5-flash (plus 300 sub-agent traces for 30B) define the supervised prior before RL.
axioms (5)
  • domain assumption Iterative MLE can be scored by a scalar environment metric (and HumanRank vs human leaderboards) that is a valid objective for max best-score within a budget.
    Problem setup §3.1 and Appendix A treat leaderboard-derived HumanRank as the primary comparable reward across heterogeneous tasks.
  • ad hoc to paper Branch-level downstream return R(c)=max_{v in subtree(c)} r(v) is the right preference label for long-horizon orchestrator decisions.
    §3.6 Preference Construction chooses max-subtree score over immediate scores or global winner-take-all; central to the training claim.
  • standard math Energy-based / ranking NCE policy improvement with log-ratio scores against a reference policy is a valid online update for the orchestrator.
    Eqs. (1)–(4) invoke standard ranking/NCE and KL-anchored policy forms from the RL literature.
  • domain assumption Fresh short-context Sub-Agents plus tool-mediated summaries suffice for faithful execution of orchestrator instructions on real MLE codebases.
    §3.4–3.5 information boundary; Appendix E shows this fails for weak 4B executors, confirming the assumption is load-bearing and scale-sensitive.
  • domain assumption MLE-Dojo’s interactive exact feedback and 50-task eval split are representative enough to support claims about long-horizon MLE agents generally.
    All main tables use this single benchmark family; generality beyond it is assumed rather than multi-benchmark proven.
invented entities (3)
  • Matryoshka Agent (Orchestrator + Tool interface + on-demand Sub-Agents) no independent evidence
    purpose: Name the hierarchical scaffold that decouples long-horizon strategy from isolated execution.
    Framework identity for the paper; composition of known agent ideas into a specific MLE-oriented cut.
  • Solution Refinement Tree with branch-level preference pairs no independent evidence
    purpose: Structure expensive MLE rollouts into reusable prefixes and contrastive orchestrator supervision.
    Training-time data structure defined in §3.6; not an external physical entity, but a postulated organizational object for learning.
  • Dojo Agent (adapted MLE-Dojo monolithic baseline) independent evidence
    purpose: Provide a strengthened single-agent control with long context and truncated env feedback.
    Baseline construct used throughout §4; adapted from prior MLE-Dojo scaffold rather than a new scientific object.

pith-pipeline@v1.2.0-grok45-kimik3 · 25541 in / 4224 out tokens · 82583 ms · 2026-07-31T01:36:32.773937+00:00 · methodology

0 comments
read the original abstract

Machine learning engineering (MLE) tasks require long-horizon decision making over iterative solution debugging and refinement, under expensive and feedback-driven environment interactions. Developing and training a monolithic agent for such tasks is fundamentally challenging, as it must simultaneously manage extremely long and noisy contexts, explore vast solution spaces, and remain effective under limited model capacity and computational budgets. To address these challenges, we propose Matryoshka Agent, a unified hierarchical agent framework for complex long-horizon tasks. Matryoshka Agent decomposes agentic problem solving into a coordinated hierarchy of decision making and execution: a high-level Orchestrator maintains compact, long-horizon exploration states and issues strategic instructions, while lower-level Sub-Agents execute concrete solution attempts through direct environment interaction, mediated by standardized Tool interface. This design decouples strategic exploration from costly execution, substantially reducing the burden of long-context reasoning and enabling efficient iterative refinement. We further develop an efficient training paradigm for Matryoshka Agent. Experimental results on a broad range of MLE tasks with diverse model types and scales demonstrate that Matryoshka Agent is an effective and scalable paradigm for long-horizon MLE tasks and complex agentic problem solving. Notably, Matryoshka Agent enables Qwen3-4B-Instruct to reach Orchestrator performance comparable to o4-mini. Applying Matryoshka Agent to Qwen3-30B-Coder results in at most 36.7% relative performance gain.

Figures

Figures reproduced from arXiv: 2607.25090 by Bo Dai, Changhao Li, Chao Zhang, Haotian Sun, Rushi Qiang, Yuchen Zhuang.

Figure 1
Figure 1. Figure 1: Overview of Matryoshka Agent. Upper: Interaction Logic among the three-layered structure. Lower: Training Paradigm with Solution Refinement Tree–based sampling. characterized by their dependency on task-specific datasets with diverse modalities (e.g., images, text), schemas, data distributions, and evaluation objectives, constitute a particularly challenging and representative instantiation of these diffic… view at source ↗
Figure 2
Figure 2. Figure 2: Cumulated Context Length across rounds. X-axis denotes the number of rounds, while y-axis denotes the token count in units of 103 . Shaded areas indicate the standard deviation. Contexts of Dojo Agent and Orchestrator are aver￾aged over tasks, while contexts of Sub-Agents are averaged per round over tasks. A core advantage of Matryoshka Agent lies in its effective and clean separation of context management… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 33 linked inside Pith

  1. [1]

    Asawa, A

    P. Asawa, A. Zhu, M. Zaharia, A. G. Dimakis, and J. E. Gonzalez. How to train your advisor: Steering black-box llms with advisor models.arXiv preprint arXiv:2510.02453, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    Auffarth

    B. Auffarth. Generative ai with langchain.Birmingham, UK: Packt Publishing, 2023

    2023

  3. [3]

    Bendinelli, A

    T. Bendinelli, A. Dox, and C. Holz. Exploring llm agents for cleaning tabular machine learning datasets.arXiv preprint arXiv:2503.06664, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Bettini, A

    M. Bettini, A. Prorok, and V . Moens. Benchmarl: Benchmarking multi-agent reinforcement learning.Journal of Machine Learning Research, 25(217):1–10, 2024

    2024

  5. [5]

    Cemri, M

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, et al. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering.arXiv preprint arXiv:2410.07095, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    C. H. Chen and M. F. Shiu. Agentflow: Resilient adaptive cloud-edge framework for multi-agent coordination.arXiv preprint arXiv:2505.07603, 2025

    Pith/arXiv arXiv 2025

  8. [8]

    G. Chen, S. Dong, Y . Shu, G. Zhang, J. Sesay, B. F. Karlsson, J. Fu, and Y . Shi. Autoagents: A framework for automatic agent generation, 2024

    2024

  9. [9]

    W. Chen, Y . Su, J. Zuo, C. Yang, C. Yuan, C.-M. Chan, H. Yu, Y . Lu, Y .-H. Hung, C. Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  10. [10]

    Y . Chen, Y . Wang, S. Zhu, H. Yu, T. Feng, M. Zhang, M. Patwary, and J. You. Multi-agent evolve: Llm self-improve through co-evolution.arXiv preprint arXiv:2510.23595, 2025

    arXiv 2025

  11. [11]

    Comanici, E

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced rea- soning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    M. Du, B. Xu, C. Zhu, X. Wang, and Z. Mao. Deepresearch bench: A comprehensive benchmark for deep research agents.arXiv preprint arXiv:2506.11763, 2025

    Pith/arXiv arXiv 2025

  13. [13]

    S. Du, X. Yan, D. Jiang, J. Yuan, Y . Hu, X. Li, L. He, B. Zhang, and L. Bai. Automlgen: Navigating fine-grained optimization for coding agents.arXiv preprint arXiv:2510.08511, 2025

    arXiv 2025

  14. [14]

    Errica, D

    F. Errica, D. Sanvito, G. Siracusano, and R. Bifulco. What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V olume 1: Long Papers), pages 1543–1558, 2025. 10

    2025

  15. [15]

    H. Fang, B. Han, N. Erickson, X. Zhang, S. Zhou, A. Dagar, J. Zhang, A. C. Turkmen, C. Hu, H. Rangwala, et al. Mlzero: A multi-agent system for end-to-end machine learning automation. arXiv preprint arXiv:2505.13941, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Grosnit, A

    A. Grosnit, A. Maraval, J. Doran, G. Paolo, A. Thomas, R. S. H. N. Beevi, J. Gonzalez, K. Khandelwal, I. Iacobacci, A. Benechehab, et al. Large language models orchestrating structured reasoning achieve kaggle grandmaster level.arXiv preprint arXiv:2411.03562, 2024

    arXiv 2024

  17. [17]

    S. Guo, C. Deng, Y . Wen, H. Chen, Y . Chang, and J. Wang. Ds-agent: Automated data science by empowering large language models with case-based reasoning.arXiv preprint arXiv:2402.17453, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, C. Zhang, J. Wang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative framework, 2024

    2024

  19. [19]

    Huang, W

    C. Huang, W. Yu, X. Wang, H. Zhang, Z. Li, R. Li, J. Huang, H. Mi, and D. Yu. R-zero: Self-evolving reasoning llm from zero data.arXiv preprint arXiv:2508.05004, 2025

    Pith/arXiv arXiv 2025

  20. [20]

    Huang, J

    Q. Huang, J. V ora, P. Liang, and J. Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation.arXiv preprint arXiv:2310.03302, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code.arXiv preprint arXiv:2403.07974, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Jiang, J

    P. Jiang, J. Lin, Z. Shi, Z. Wang, L. He, Y . Wu, M. Zhong, P. Song, Q. Zhang, H. Wang, et al. Adaptation of agentic ai.arXiv preprint arXiv:2512.16301, 2025

    arXiv 2025

  23. [23]

    Jiang, D

    Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y . Wu. Aide: Ai-driven exploration in the space of code.arXiv preprint arXiv:2502.13138, 2025

    Pith/arXiv arXiv 2025

  24. [24]

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    L. Jing, Z. Huang, X. Wang, W. Yao, W. Yu, K. Ma, H. Zhang, X. Du, and D. Yu. Ds- bench: How far are data science agents from becoming data science experts?arXiv preprint arXiv:2409.07703, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    C. Li, Y . Zhuang, R. Qiang, H. Sun, H. Dai, C. Zhang, and B. Dai. Matryoshka pilot: Learning to drive black-box llms with llms. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  27. [27]

    Z. Li, Q. Zang, D. Ma, J. Guo, T. Zheng, M. Liu, X. Niu, Y . Wang, J. Yang, J. Liu, et al. Autokaggle: A multi-agent framework for autonomous data science competitions.arXiv preprint arXiv:2410.20424, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Liang, Z

    T. Liang, Z. He, W. Jiao, X. Wang, Y . Wang, R. Wang, Y . Yang, S. Shi, and Z. Tu. Encouraging divergent thinking in large language models through multi-agent debate. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 17889–17904, 2024

    2024

  29. [29]

    S. Liu, C. Gao, and Y . Li. Large language model agent for hyper-parameter optimization.arXiv preprint arXiv:2402.01881, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    M. A. Merrill, A. G. Shaw, N. Carlini, B. Li, H. Raj, I. Bercovich, L. Shi, J. Y . Shin, T. Walshe, E. K. Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces.arXiv preprint arXiv:2601.11868, 2026

    Pith/arXiv arXiv 2026

  31. [31]

    F. Mohr, M. Wever, and E. Hüllermeier. Ml-plan: Automated machine learning via hierarchical planning.Machine Learning, 107(8):1495–1515, 2018

    2018

  32. [32]

    J. Nam, J. Yoon, J. Chen, J. Shin, S. Ö. Arık, and T. Pfister. Mle-star: Machine learning engineering agent via search and targeted refinement.arXiv preprint arXiv:2506.15692, 2025. 11

    Pith/arXiv arXiv 2025

  33. [33]

    Introducing openai gpt-5.OpenAI Blog, 2025

    OpenAI. Introducing openai gpt-5.OpenAI Blog, 2025

    2025

  34. [34]

    Introducing openai o3 and o4-mini.OpenAI Blog, 2025

    OpenAI. Introducing openai o3 and o4-mini.OpenAI Blog, 2025

    2025

  35. [35]

    Y . Ou, Y . Luo, J. Zheng, L. Wei, Z. Yu, S. Qiao, J. Zhang, D. Zheng, Y . Mao, Y . Gao, et al. Automind: Adaptive knowledgeable agent for automated data science.arXiv preprint arXiv:2506.10974, 2025

    arXiv 2025

  36. [36]

    Ouyang, S

    A. Ouyang, S. Guo, S. Arora, A. L. Zhang, W. Hu, C. Ré, and A. Mirhoseini. Kernelbench: Can llms write efficient gpu kernels?arXiv preprint arXiv:2502.10517, 2025

    Pith/arXiv arXiv 2025

  37. [37]

    Patwardhan, R

    T. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. P. Fishman, M. Aljubeh, P. Thacker, L. Fauconnet, et al. Gdpval: Evaluating ai model performance on real-world economically valuable tasks.arXiv preprint arXiv:2510.04374, 2025

    Pith/arXiv arXiv 2025

  38. [38]

    L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Press, B

    O. Press, B. Amos, H. Zhao, Y . Wu, S. K. Ainsworth, D. Krupke, P. Kidger, T. Sajed, B. Stellato, J. Park, et al. Algotune: Can language models speed up general-purpose numerical programs? arXiv preprint arXiv:2507.15887, 2025

    arXiv 2025

  40. [40]

    Qiang, Y

    R. Qiang, Y . Zhuang, Y . Li, R. Zhang, C. Li, I. S.-H. Wong, S. Yang, P. Liang, C. Zhang, B. Dai, et al. Mle-dojo: Interactive environments for empowering llm agents in machine learning engineering.arXiv preprint arXiv:2505.07782, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Qiang, Y

    R. Qiang, Y . Zhuang, A. Singh, P. Liang, C. Zhang, S. Yang, and B. Dai. Mle-smith: Scaling mle tasks with automated multi-agent pipeline.arXiv preprint arXiv:2510.07307, 2025

    arXiv 2025

  42. [42]

    B. Rank, H. Bhatnagar, A. Prabhu, S. Eisenberg, K. Nguyen, M. Bethge, and M. Andriushchenko. Posttrainbench: Can llm agents automate llm post-training?arXiv preprint arXiv:2603.08640, 2026

    arXiv 2026

  43. [43]

    Shang, Y

    Y . Shang, Y . Li, K. Zhao, L. Ma, J. Liu, F. Xu, and Y . Li. Agentsquare: Automatic llm agent search in modular design space, 2025

    2025

  44. [44]

    Starace, O

    G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, et al. Paperbench: Evaluating ai’s ability to replicate ai research.arXiv preprint arXiv:2504.01848, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Talebirad and A

    Y . Talebirad and A. Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents.arXiv preprint arXiv:2306.03314, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    X. Tang, Y . Liu, Z. Cai, Y . Shao, J. Lu, Y . Zhang, Z. Deng, H. Hu, K. An, R. Huang, et al. Ml- bench: Evaluating large language models and agents for machine learning tasks on repository- level code.arXiv preprint arXiv:2311.09835, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Q. Team. Qwen3 technical report, 2025

    2025

  48. [48]

    Toledo, K

    E. Toledo, K. Hambardzumyan, M. Josifoski, R. Hazra, N. Baldwin, A. Audran-Reiss, M. Kuch- nik, D. Magka, M. Jiang, A. M. Lupidi, et al. Ai research agents for machine learning: Search, exploration, and generalization in mle-bench.arXiv preprint arXiv:2507.02554, 2025

    arXiv 2025

  49. [49]

    Z. Wan, Y . Li, X. Wen, Y . Song, H. Wang, L. Yang, M. Schmidt, J. Wang, W. Zhang, S. Hu, and Y . Wen. Rema: Learning to meta-think for llms with multi-agent reinforcement learning, 2025

    2025

  50. [50]

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singh, et al. Openhands: An open platform for ai software developers as generalist agents.arXiv preprint arXiv:2407.16741, 2024

    Pith/arXiv arXiv 2024

  51. [51]

    Y . Wang, Y . Chen, F. Zhong, L. Ma, and Y . Wang. Simulating human-like daily activities with desire-driven autonomy.arXiv preprint arXiv:2412.06435, 2024. 12

    Pith/arXiv arXiv 2024

  52. [52]

    J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprint arXiv:2504.12516, 2025

    Pith/arXiv arXiv 2025

  53. [53]

    X. Wei, Y . Dong, X. Wang, X. Zhang, Z. Zhao, D. Shen, L. Xia, and D. Yin. Beyond react: A planner-centric framework for complex tool-augmented llm reasoning, 2025

    2025

  54. [54]

    T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094, 2024

    2024

  55. [55]

    B. Yan, Z. Zhou, L. Zhang, L. Zhang, Z. Zhou, D. Miao, Z. Li, C. Li, and X. Zhang. Beyond self-talk: A communication-centric survey of llm-based multi-agent systems, 2025

    2025

  56. [56]

    S. Yang, J. He-Yueya, and P. Liang. Reinforcement learning for machine learning engineering agents.arXiv preprint arXiv:2509.01684, 2025

    Pith/arXiv arXiv 2025

  57. [57]

    X. Yang, X. Yang, S. Fang, B. Xian, Y . Li, J. Wang, M. Xu, H. Pan, X. Hong, W. Liu, et al. R&d- agent: Automating data-driven ai solution building through llm-powered automated research, development, and evolution.arXiv preprint arXiv:2505.14738, 2025

    arXiv 2025

  58. [58]

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains.arXiv preprint arXiv:2406.12045, 2024

    Pith/arXiv arXiv 2024

  59. [59]

    Zhang, C

    S. Zhang, C. Gong, L. Wu, X. Liu, and M. Zhou. Automl-gpt: Automatic machine learning with gpt.arXiv preprint arXiv:2305.02499, 2023

    Pith/arXiv arXiv 2023

  60. [60]

    Zhang, L

    W. Zhang, L. Zeng, Y . Xiao, Y . Li, C. Cui, Y . Zhao, R. Hu, Y . Liu, Y . Zhou, and B. An. Agentorchestra: Orchestrating multi-agent intelligence with the tool-environment-agent(tea) protocol, 2026

    2026

  61. [61]

    Zhang, R

    Y . Zhang, R. Sun, Y . Chen, T. Pfister, R. Zhang, and S. Ö. Arik. Chain of agents: Large language models collaborating on long-context tasks, 2024

    2024

  62. [62]

    Zhang, X

    Z. Zhang, X. Li, Y . Lin, H. Liu, R. Chandradevan, L. Wu, M. Lin, F. Wang, X. Tang, Q. He, and S. Wang. Unlocking the power of multi-agent llm for reasoning: From lazy agents to deliberation, 2025

    2025

  63. [63]

    execute_code

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023. 13 A Benchmark and Metric Details MLE-Dojo Benchmark.We conduct all experiments onMLE-Dojo[ 40], a large-scale benchmark designed to evaluate autonomo...

    Pith/arXiv arXiv 2023

  64. [64]

    Combine training and testing data for consistent prepro- cessing

    Data Loading: Load train.csv, test.csv, and sample_submission.csv from the specified data_path. Combine training and testing data for consistent prepro- cessing

  65. [65]

    ,margin_64)

    Feature Engineering: The features are provided as individual columns (e.g., margin_1, margin_2, . . . ,margin_64). Concatenate these 64 attributes for each feature type (shape, margin, texture) to form a single feature vector for each image. For example, for themarginfeature, combine margin_1 through margin_64 into a single column named margin_all. Do the...

  66. [66]

    Target Variable: The target variable in the training data is thespeciescolumn

  67. [67]

    Set is_unbalance=True, n_estimators=1000, learning_rate=0.05, min_child_samples=20, subsample=0.7, colsample_bytree=0.7, reg_alpha=0.1, reg_lambda=0.1,random_state=42, andn_jobs=-1

    Model Training: Use a LightGBM Classifier (LGBMClassifier) with objective=’multiclass’, num_class=99, and metric=’multi_logloss’. Set is_unbalance=True, n_estimators=1000, learning_rate=0.05, min_child_samples=20, subsample=0.7, colsample_bytree=0.7, reg_alpha=0.1, reg_lambda=0.1,random_state=42, andn_jobs=-1

  68. [68]

    Prediction: Generate probability predictions for the preprocessed test set and ensure probabilities are within the [0, 1] range

  69. [69]

    Save submission.csvto theoutput_pathusing absolute paths

    Submission File Generation: Create a submission DataFrame with the id column and species prob- ability columns ordered exactly as in sample_submission.csv. Save submission.csvto theoutput_pathusing absolute paths. Constraints and Guidelines: Do not use any image data. Ensure no data leakage, preserve test IDs, normal- ize probabilities if needed, and mini...

    2020