{"id":"b2a7bb23-c92f-4516-b63a-0a46c531c8da","arxiv_id":"2607.25090","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Matryoshka Agent’s orchestrator–sub-agent hierarchy plus tree-ranked RL raises MLE-Dojo HumanRank, letting a 4B orchestrator approach o4-mini and giving a 30B coder up to 36.7% relative gain.","lead":"A hierarchical LLM agent splits long machine-learning engineering jobs into a strategist that keeps short notes and worker agents that write and debug code. The split cuts context bloat and, with tree-based training, lifts small and mid-size models on Kaggle-style MLE benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The untrained Matryoshka-vs-Dojo comparison may conflate hierarchical decomposition with extra effective exploration (parallel Sub-Agents, fresh per-attempt contexts) inside the same 12h wall-clock budget.","rationale":"The reader flagged the information boundary (do summaries retain decision-relevant signal?). That concern is partially self-answering in the paper: the untrained head-to-head (0.3652 vs 0.3302) and the o4-mini inference-only gain are exactly the empirical probes of whether the boundary discards too much, and they come out positive. So the boundary question, as stated, is less load-bearing than the question of what drives those positive numbers. My concern shares the same locus — the credibility of the untrained architecture comparison — but identifies a different mechanism: unmatched exploration/compute budgets. If parallel Sub-Agents or higher attempt counts explain the untrained gap, the boundary may be neutral-to-harmful and masked by extra sampling, which would directly weaken the reader's strongest_claim phrasing (\"effective scalable paradigm... with hierarchical orchestration\"). I do not think this breaks the paper: the trained results (SFT-Dojo control, role-swap ablation, plug-and-play) stand on their own, the 12h wall-clock budget is a defensible practical constraint for Kaggle-style tasks, and the mean-aggregation robustness check in App. D is a genuine good-faith effort. Hence the verdict stays CONDITIONAL; I would add \"report per-task attempt counts/token budgets and an attempt-matched monolithic baseline\" to the reader's existing conditions on artifact release and evaluation reporting. If the matched-control experiment preserves the gap, the architecture claim is solid; if not, the paper's contribution narrows to the training paradigm, which is still publishable but differently framed.","tokens_in":21022,"tokens_out":2518,"duration_ms":46259,"concrete_test":"For the untrained Qwen3-30B-Coder setting (Table 1), log per-task: number of scored solution attempts, total tokens generated, and whether parallel Sub-Agents were used within each 12h trial. Then run an attempt-matched monolithic baseline: Dojo Agent with k independent rollouts inside the 12h window, best-valid-score selection, with k set to Matryoshka's observed attempt count. If the matched baseline reaches ≈0.36, the untrained architecture claim collapses to a parallelism effect; if it stays near 0.33, the hierarchy claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim's empirical spine is Table 1, and its causal attribution — \"the hierarchical architecture contributes beyond supervised fine-tuning alone\" — rests on comparisons that share only a 12-hour wall-clock budget (§4.1, App. B), not matched compute or matched solution attempts. Section 3.4 explicitly allows \"multiple, parallel tool calls when parallel exploration is desired,\" and each Sub-Agent gets a fresh context with up to 10 debugging iterations (App. B). If parallel Sub-Agents were used during evaluation, Matryoshka effectively samples more candidate solutions per trial than the strictly sequential Dojo Agent; even without parallelism, per-attempt context resets let the framework make many more attempts before context saturation (App. F: 213 vs 137 supported rounds). Since the objective (§3.1) is max-over-attempts score, more attempts mechanically lift the expected best score. The untrained gap (0.3652 vs 0.3302) is modest and could be substantially an exploration-budget effect rather than an orchestration effect. Appendix D addresses best-of-three aggregation but never reports attempt counts, token totals, or parallel vs serial execution, so the confound is invisible in the paper. This does not touch the training-paradigm results (SFT-Dojo control is decent), but it undercuts the \"hierarchy itself helps\" leg of the strongest claim, including the inference-only o4-mini gain (0.4832 → 0.5465).","agreement_with_reader":"partial"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper proposes Matryoshka Agent, a three-layer hierarchical agent for long-horizon machine-learning engineering: an Orchestrator keeps a compact score-annotated history and chooses refinement directions; Tools package instructions/references and launch isolated Sub-Agents; Sub-Agents execute/debug code and return summaries plus scores. Training uses Solution Refinement Trees with binary branching, branch labels R(c)=max_{v in subtree(c)} r(v), and a DPO/NCE-like log-ratio ranking loss (Eqs. 1-4), plus SFT from successful Sub-Agent trajectories. On MLE-Dojo's 150/50 split, the authors report gains over an adapted Dojo Agent across several open/closed model pairings, an SFT-Dojo control, RL-without-SFT and orchestrator/sub-agent swap ablations, plug-and-play transfer to GPT-5-nano Sub-Agents, and context-growth analysis.","tokens_in":21425,"tokens_out":3830,"duration_ms":85502,"significance":"If the results hold, this is a useful and timely systems contribution: it gives a concrete way to decouple strategic search from execution in expensive MLE loops, shows a plausible route for small models to act as orchestrators, and connects inference-time scaffolding to an online preference-optimization recipe. Strengths worth credit: evaluation on held-out MLE-Dojo tasks with an external HumanRank metric; multiple model pairings; an adapted monolithic baseline; an SFT-Dojo control; orchestrator/sub-agent attribution ablations; plug-and-play tests; context-length scaling analysis; and Appendix D's check that mean-of-three preserves the displayed ordering. The main risk is not circularity in the metric but causal attribution: some architecture-vs-baseline comparisons may reflect exploration/context-reset budget rather than hierarchy per se.","major_comments":[{"comment":"The key untrained comparisons share only a 12h wall-clock budget, not matched attempts/tokens. §3.4 explicitly allows parallel tool calls, each Sub-Agent gets a fresh context with up to 10 debugging iterations (App. B), and App. F estimates 213 vs 137 supported rounds. Under the max-over-attempts objective (§3.1), more valid attempts mechanically raise expected best score. Please report per-task valid-attempt counts, total/generated tokens, serial vs parallel execution, and add matched-attempt/matched-token controls plus a monolithic baseline with periodic context resets/restarts. This is load-bearing for the inference-only claims (0.3302→0.3652 and 0.4832→0.5465).","section":"§4.1, §3.4, App. B/F, Table 1"},{"comment":"Headline Table 1 uses per-task best-of-three, while Appendix D's mean-of-three check contains only four rows and omits several rows needed for the strongest claims (e.g., Qwen3-30B-Coder Dojo, SFT-Dojo, untrained/SFT Matryoshka-30B, Qwen3-4B-SFT). Given median per-task SD ≈0.07, gaps such as 0.3652 vs 0.3302 and some category-level differences are within or near run noise, and best-of-three favors higher-variance/higher-attempt systems. Provide mean±SD for every Table 1/2 row, paired per-task statistics, and best-of-k sensitivity for k=1,2,3.","section":"§4.1, App. D, Tables 1 and 5"},{"comment":"The optimization derivation is DPO-like, but the preference labels are deterministic branch-return winners from R(c)=max subtree r(v). Eq. (1) presupposes a stochastic reward/Bradley-Terry noise model with temperature β; with hard labels and binary C(u), β is not identifiable and Eq. (4) should be presented as a contrastive algorithmic objective rather than a principled posterior unless a noise model is specified. Also discuss credit assignment: branch-return labeling can credit/blame an edge for later descendants under the current policy. A small labeling ablation (immediate score vs global winner vs branch return) would substantiate the design axiom.","section":"§3.6, Eqs. (1)-(4)"},{"comment":"The plug-and-play claim is not isolated. Table 2 compares GPT-5-nano Dojo (0.3946) with trained RL Orchestrators plus GPT-5-nano (0.4309/0.4626), but omits Matryoshka with an untrained Orchestrator plus GPT-5-nano. Thus the table cannot distinguish 'trained orchestrator transfers' from 'Matryoshka scaffolding helps GPT-5-nano.' Add the untrained-orchestrator + GPT-5-nano control and, if possible, an SFT-only orchestrator control.","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"HumanRank s=1-p/N needs tie handling, treatment of invalid/missing submissions, whether N includes the agent, and exact public/private averaging. An off-by-one convention (rank 1 gives 1-1/N) should be stated explicitly.","section":"App. A, Eq. (5)"},{"comment":"Define 'round' consistently for Orchestrator, Sub-Agent, and Dojo Agent; state whether Sub-Agent tokens are per-round averages or sums; include base prompt and output-token growth in the 213/137 round estimate, which currently divides 256k only by input-history slope.","section":"Fig. 2 / App. F"},{"comment":"Table 2's model-configuration cells appear malformed ('Qwen3-4B-RL +' with the Sub-Agent implicit). Name GPT-5-nano explicitly in each row. In Table 3, define 'O' inline rather than relying on Table 1.","section":"Tables 2-4"},{"comment":"Several typos/grammar issues: 'Matryoshka Agent further develop' (§1), 'the the underlying model' (§4.1), 'results of the refinement' phrasing (§3.3), and inconsistent Qwen3-Coder-30B/Qwen3-Coder-30B-A3B naming.","section":"Writing"},{"comment":"Because Eqs. (1)-(4) closely follow DPO-style log-ratio reparameterization and NCE ranking, cite DPO/Rafailov et al. and relevant preference-optimization/NCE work, and clarify differences from KTO/IPO-style objectives.","section":"§3.6 / Related Work"},{"comment":"Please state code/config/seed availability, closed-model versions/snapshots for o4-mini/GPT-5-nano/Gemini-2.5-Flash, exact 12h enforcement, package installation policy, and whether evaluation used parallel tool calls in the reported runs.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the venue and is more than a prompt-engineering note, but the empirical story depends heavily on the authors' own MLE-Dojo ecosystem and prior Matryoshka line. I would not reject on novelty grounds, but before acceptance I would want at least the budget/attempt controls above and preferably one external comparison point beyond MLE-Dojo-style tasks (e.g., MLE-bench/AIDE-like setting) to reduce benchmark-fit risk."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a usable Orchestrator/Sub-Agent scaffold plus a tree-preference ranking recipe for expensive MLE loops on MLE-Dojo. The small-model-as-planner result (4B Orchestrator + o4-mini executors ≈ o4-mini Orchestrator) and the SFT→RL lifts are the parts worth remembering. Novelty is moderate—modular agents, controller–generator, and MLE MAS are crowded, including their own Matryoshka-Pilot—but the concrete package (dynamic refine-target + references, branch-level downstream-return preferences, ranking NCE with log-ratio score, Sub-Agent self-SFT) is a real engineering contribution, not a re-label.\n\nWhat they do well: broad model pairings, SFT-Dojo monolithic control, role-swap ablation (RL helps more as Orchestrator), RL-without-SFT, plug-and-play to GPT-5-nano, context-growth curves, and Appendix D showing mean-of-three preserves order. Circularity is low; HumanRank is external leaderboard signal. Math is standard energy/ranking NCE, not load-bearing theory. Citations are fair to the crowded space.\n\nSoft spots, in proportion. The stress-test lands on the untrained hierarchy leg: same 12h wall-clock, explicit parallel tool calls, fresh Sub-Agent contexts, up to 10 debug iters each, and App. F’s 213 vs 137 round capacity. Objective is max-over-attempts, so extra effective attempts mechanically help. They never report attempt counts, tokens, or parallel vs serial, so the modest untrained gap (0.3652 vs 0.3302) and the o4-mini inference-only jump are not cleanly “orchestration.” Training comparisons with SFT-Dojo are on firmer ground. Other real limits: single benchmark, teacher-distilled SFT (gemini-2.5-flash), best-of-three as primary, no code/data release, free knobs (β, tree width/depth, budgets).\n\nWho it’s for: people building long-horizon coding/MLE agents who need a trainable planner–executor split and evidence that small orchestrators can ride strong executors. Not a theory paper. I would send it to peer review; ask for matched-attempt or matched-token tables and artifacts. I’d cite the training recipe and the 4B-orchestrator result if I work in this lane; I’d bring it to reading group with the budget confound as the discussion prompt.","headline":"Solid hierarchical MLE-agent systems paper with real training recipe and decent controls; the hierarchy-vs-monolith gains are partly confounded by unmatched exploration budget under a shared wall-clock limit.","tokens_in":22629,"tokens_out":622,"would_cite":true,"duration_ms":12516,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A nested agent design splits strategy from execution so long-horizon machine-learning engineering no longer drowns in context.","keywords":["hierarchical agents","machine learning engineering","long-horizon decision making","orchestrator sub-agent","solution refinement tree","online ranking RL","context management","MLE agents"],"falsifier":"On the same MLE-Dojo split and budgets, ablate the information boundary: give the Orchestrator raw logs or full Sub-Agent traces (or strip structured summaries to scores only) and check whether All HumanRank and long-horizon rank still beat the matched monolithic Dojo Agent; if hierarchy loses under richer context, the boundary assumption fails.","tokens_in":22379,"feed_emoji":"🪆","tokens_out":936,"duration_ms":24782,"temperature":0.7,"pith_summary":"Machine-learning engineering is a long loop of code, run, debug, and score under costly feedback. A single agent that keeps every log and code trace soon faces noisy, ballooning context and limited capacity. This paper argues that the fix is hierarchical: a persistent Orchestrator keeps only compact scores and summaries and chooses what to try next, while short-lived Sub-Agents implement each attempt in a fresh context through a Tool layer. Training builds a solution-refinement tree that reuses shared prefixes and ranks branches by best downstream score, so the Orchestrator learns from expensive runs without seeing raw traces. On a broad MLE benchmark the design lifts small open models toward frontier closed models as orchestrators and yields large gains for a 30B coder, while keeping context growth far slower than a monolithic agent.","feed_headline":"Nested agents lift small models on long MLE tasks","feed_subtitle":"Strategy stays compact; code runs in fresh sub-agents—up to ~37% gain vs a single agent.","key_machinery":"Matryoshka Agent: a three-layer loop (Orchestrator decides parent attempt, instruction, and references; Tools build fresh Sub-Agent contexts and return scores plus structured summaries; Sub-Agents run bounded code–debug loops). Training uses binary Solution Refinement Trees labeled by branch-level max downstream return, optimized with a ranking NCE / log-ratio objective for the Orchestrator plus SFT self-reinforcement on winning Sub-Agent trajectories.","core_discovery":"Matryoshka Agent is an effective, scalable paradigm for long-horizon MLE: decoupling strategic refinement from concrete execution, then training the Orchestrator with tree-based preference signals and the Sub-Agents on successful runs, produces consistent HumanRank gains across model scales—including a 4B Orchestrator matching a strong closed model’s orchestration score, and up to about 37% relative gain for a 30B coder over its monolithic baseline.","pith_inferences":["The same summary-mediated hierarchy may transfer to other costly iterative loops—kernel tuning, post-training search, or paper-replication agents—where full traces swamp strategy.","If Sub-Agent capacity is the bottleneck, further gains may come more from executor specialization than from longer Orchestrator instructions.","Binary refinement trees trade exploration breadth for label quality; wider branching or adaptive depth could change the cost–signal tradeoff on harder tasks.","Plug-and-play Orchestrators suggest a market-like split: train cheap decision policies once, rent strong code models only for execution."],"forward_implications":["Small models can act as high-level decision layers when paired with stronger executors, without carrying full execution history.","Orchestrators trained with one Sub-Agent stack can be swapped onto other executors at test time and still improve them.","Context growth stays roughly linear and slower than a monolithic agent, supporting more refinement rounds under fixed context budgets.","Tree sampling that reuses shared prefixes makes preference learning affordable when each environment run is expensive.","The same unfold-orchestrate pattern is offered as a general template for other long-horizon agent tasks beyond MLE."],"fun_headline_variants":["Matryoshka Agent unfolds sub-agents for long-horizon MLE","Hierarchical agents decouple strategy from MLE execution","4B Orchestrator matches o4-mini on long MLE tasks","Nested agents yield up to 37% gain on MLE benchmarks","Compact strategy, fresh sub-agents beat monolithic MLE"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Compact tool summaries and scalar scores, without raw code or full debug traces in the Orchestrator’s memory, still carry enough signal for better long-horizon choices than an agent that sees the full history.","fun_headline_variants_meta":{"raw":{"variants":["Matryoshka Agent unfolds sub-agents for long-horizon MLE","Hierarchical agents decouple strategy from MLE execution","4B Orchestrator matches o4-mini on long MLE tasks","Nested agents yield up to 37% gain on MLE benchmarks","Compact strategy, fresh sub-agents beat monolithic MLE"]},"model":"grok-4.5","effort":"low","cost_usd":0.002362,"raw_usage":{"total_tokens":988,"prompt_tokens":837,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":23624000,"prompt_tokens_details":{"text_tokens":837,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":76,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":837,"tokens_out":75,"duration_ms":3011,"temperature":1.0,"reasoning_tokens":76,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T01:36:32.773937+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same MLE-Dojo split and budgets, ablate the information boundary: give the Orchestrator raw logs or full Sub-Agent traces (or strip structured summaries to scores only) and check whether All HumanRank and long-horizon rank still beat the matched monolithic Dojo Agent; if hierarchy loses under richer context, the boundary assumption fails.","supporting_citations":[],"review_version":1}