Recognition: no theorem link
Synthetic Sandbox for Training Machine Learning Engineering Agents
Pith reviewed 2026-05-10 20:03 UTC · model grok-4.3
The pith
SandMLE creates micro-scale synthetic MLE tasks that cut verification time by over 13 times, making on-policy reinforcement learning practical for training machine learning engineering agents.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
SandMLE is a multi-agent framework that generates diverse, verifiable synthetic MLE environments from seed tasks while preserving structural and technical complexity at micro-scale. This shrinks dataset size dramatically, reduces execution time by more than 13 times, and enables trajectory-wise on-policy RL for the first time in the MLE domain, producing higher medal rates on MLE-bench-lite and improved HumanRank scores on MLE-Dojo.
What carries the argument
The SandMLE multi-agent framework that produces synthetic MLE environments from seed tasks, each paired with only 50-200 training samples yet retaining real-world structural complexity.
If this is right
- Trajectory-wise on-policy RL becomes feasible at scale for MLE agents instead of relying on SFT or offline proxies.
- Relative medal rate improvements of 20.3 percent to 66.9 percent over SFT baselines across Qwen3-8B, 14B, and 30B-A3B models.
- Trained policies transfer to unseen agentic scaffolds, yielding up to 32.4 percent higher HumanRank on MLE-Dojo.
- Large-scale on-policy training is now possible in the MLE domain where verification costs previously made it prohibitive.
Where Pith is reading between the lines
- The same shrinking-and-synthesizing strategy could apply to other agent domains where full verification pipelines are expensive, such as scientific computing or hardware design.
- If the synthetic tasks retain enough signal, future work could test whether even smaller or procedurally generated environments suffice for initial policy learning before fine-tuning on real data.
- Success here suggests that data volume is often less critical than structural fidelity when training agents to perform multi-step technical workflows.
Load-bearing premise
Synthetic environments generated from a small number of seed tasks still contain the structural and technical complexity of actual machine learning engineering problems even when each task uses only 50-200 training samples.
What would settle it
Measure whether policies trained entirely on SandMLE environments achieve the reported medal-rate and HumanRank gains when evaluated on full-scale, non-synthetic MLE tasks with real-sized datasets.
read the original abstract
As large language model agents advance beyond software engineering (SWE) tasks toward machine learning engineering (MLE), verifying agent behavior becomes orders of magnitude more expensive: while SWE tasks can be verified via fast-executing unit tests, MLE verification requires running full ML pipelines -- data preprocessing, model training, and metric evaluation -- on large datasets at each rollout step, rendering trajectory-wise on-policy reinforcement learning (RL) prohibitively slow. Existing approaches retreat to supervised fine-tuning (SFT) or offline proxy rewards, sacrificing the exploration and generalization benefits of on-policy RL. We observe that sandbox data size is the primary source of this bottleneck. Based on this insight, we introduce SandMLE, a multi-agent framework that generates diverse, verifiable synthetic MLE environments from a small number of seed tasks, preserving the structural and technical complexity of real-world problems while constraining datasets to micro-scale (each task is paired with only 50-200 training samples). Through extensive experiments, we show that SandMLE reduces execution time by over 13 times, enabling large-scale, on-policy trajectory-wise RL for the first time in the MLE domain. On MLE-bench-lite, SandMLE yields significant gains over SFT baselines across Qwen3-8B, 14B, and 30B-A3B, with relative medal rate improvements ranging from 20.3% to 66.9%. Furthermore, the trained policy generalizes across unseen agentic scaffolds, achieving up to 32.4% better HumanRank score on MLE-Dojo.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SandMLE, a multi-agent framework that generates diverse, verifiable synthetic MLE environments from a small number of seed tasks. By constraining each task to micro-scale datasets (50-200 training samples) while claiming to preserve structural and technical complexity, SandMLE reduces ML pipeline execution time by over 13x. This enables large-scale on-policy trajectory-wise RL for MLE agents for the first time. Experiments show relative medal rate gains of 20.3% to 66.9% over SFT baselines on MLE-bench-lite across Qwen3-8B/14B/30B-A3B models, plus up to 32.4% better HumanRank on MLE-Dojo with generalization to unseen scaffolds.
Significance. If the core preservation claim holds, the work is significant because it removes the primary computational barrier to on-policy RL in MLE (full pipeline verification on large data), opening the door to scalable exploration and better generalization than SFT or offline methods. The reported speedups and cross-benchmark gains, if reproducible, would represent a practical advance for training ML engineering agents.
major comments (2)
- [Abstract and §3] Abstract and §3 (SandMLE generation): The central claim that synthetic tasks with only 50-200 samples preserve the structural and technical complexity of real MLE problems (including scale-dependent phenomena such as memory pressure and hyperparameter landscapes) is asserted without any quantitative validation. No task-difficulty metrics, solution-length distributions, expert-rated complexity scores, or direct comparisons to MLE-bench-lite tasks are provided; this is load-bearing because the 13x speedup, on-policy RL feasibility, and downstream transfer all rest on it.
- [§4] §4 (Experiments): The reported relative medal-rate improvements (20.3%-66.9%) and HumanRank gains lack reporting of the number of independent runs, standard deviations, or statistical significance tests. Without these, it is impossible to determine whether the gains over SFT baselines are reliable or could be explained by variance in the synthetic environments.
minor comments (1)
- [§3] The multi-agent generation pipeline would be clearer with an explicit diagram or pseudocode showing the roles of each agent and the verification step.
Simulated Author's Rebuttal
We thank the referee for the constructive and detailed feedback. We address each major comment point by point below, indicating where revisions will be made to improve the manuscript.
read point-by-point responses
-
Referee: [Abstract and §3] Abstract and §3 (SandMLE generation): The central claim that synthetic tasks with only 50-200 samples preserve the structural and technical complexity of real MLE problems (including scale-dependent phenomena such as memory pressure and hyperparameter landscapes) is asserted without any quantitative validation. No task-difficulty metrics, solution-length distributions, expert-rated complexity scores, or direct comparisons to MLE-bench-lite tasks are provided; this is load-bearing because the 13x speedup, on-policy RL feasibility, and downstream transfer all rest on it.
Authors: We acknowledge that the manuscript would be strengthened by explicit quantitative support for the complexity-preservation claim. SandMLE is constructed to retain the core structural elements (pipeline stages, model choices, evaluation protocols) and technical challenges from the seed tasks while reducing data cardinality; the observed transfer to MLE-Dojo and consistent gains across model scales provide supporting evidence. Nevertheless, we agree that direct metrics are missing. In the revision we will add to §3: (i) solution-length distributions comparing synthetic and original tasks, (ii) a small set of expert-rated complexity scores on a sampled subset, and (iii) a brief discussion clarifying that scale-dependent effects such as memory pressure are intentionally de-emphasized by the micro-scale design, while hyperparameter sensitivity and pipeline structure are preserved. We will also include a side-by-side comparison table with MLE-bench-lite tasks on these metrics. revision: yes
-
Referee: [§4] §4 (Experiments): The reported relative medal-rate improvements (20.3%-66.9%) and HumanRank gains lack reporting of the number of independent runs, standard deviations, or statistical significance tests. Without these, it is impossible to determine whether the gains over SFT baselines are reliable or could be explained by variance in the synthetic environments.
Authors: We agree that statistical reporting is essential. The experiments were performed with multiple independent random seeds to control for variance in both environment generation and training, yet these details were omitted from the submitted version. In the revised §4 we will report the exact number of independent runs per model, include standard deviations for all medal-rate and HumanRank figures, and add paired statistical significance tests (with p-values) against the SFT baselines. This will allow readers to assess the robustness of the reported improvements. revision: yes
Circularity Check
No circularity: results are empirical measurements and benchmark scores
full rationale
The paper's central claims rest on direct timing measurements (13x reduction) and downstream performance gains on MLE-bench-lite and MLE-Dojo. The generation of synthetic micro-scale tasks is presented as an engineering choice whose validity is checked via observed RL improvements rather than any equation, fitted parameter, or self-citation that reduces the outcome to its inputs by construction. No load-bearing derivation step equates a prediction to a prior fit or renames an input as a result.
Axiom & Free-Parameter Ledger
Reference graph
Works this paper leans on
-
[1]
Notion Blog. Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan Ö Arık, and Tomas Pfister. Mle-star: Machine learning engineering agent via search and targeted refinement.arXiv preprint arXiv:2506.15692, 2025. Rushi Qiang, Yuchen Zhuang, Yinghao Li, Rongzhi Zhang, Changhao Li, Ian Shu-Hei Wong, Sherry Yang, Percy Liang, Chao Zhang, Bo Dai, et al...
-
[2]
You are a multi-turn generation agent: in each turn, propose/refine the script or reasoning, then wait for environment/tool feedback
-
[3]
STRICT: each response may contain exactly ONE<tool_call>block—do not emit multiple tool calls
Execute via the tool until it runs cleanly and produces the file. STRICT: each response may contain exactly ONE<tool_call>block—do not emit multiple tool calls
-
[4]
After generating the code, the Python environment will provide feedback. You must observe at least one tool feedback (execution result wrapped in<tool_response>...</tool_response> tags) before deciding to end. Only when feedback looks good do you reply with<answer>submission</answer>; otherwise continue iterating (do not output<answer>tags)
-
[5]
Use PythonInterpreter to run updated code; use Score tool to grade submission.csv. Repeat this refine-grade loop until the submission is acceptable, then end with<answer>submission</answer>. Tool usage: For each function call, return a json object with function name and arguments within <tool_- call>...</tool_call>XML tags: - Wrap executable code exactly ...
-
[6]
Asset generation (deterministic, no external downloads): - Tabular: build DataFrames per the blueprint and save totrain.csv and test.csv (test ∼20% the size of train). - Image: createimages/, draw synthetic images with Pillow or cv2 following the blueprint, save.png, and create train.csv and test.csv mapping filename to label (test labels blank but column...
-
[7]
Be explicit about how features drive labels
Hidden rule logic: - Implement the blueprint’shidden_rule_logic to assign labels deterministically. Be explicit about how features drive labels
-
[8]
Heuristic leaderboard / sanity check: - Implement the threshold logic described in the blueprint (use its rules, ignore specific numeric targets). - Compute thresholds on the generated test data (per the threshold logic). - Save the computed thresholds in threshold.json with exactly these keys: "gold_threshold", "silver_threshold","bronze_threshold","medi...
-
[9]
Sample submission: - Createsample_submission.csv with test IDs and a placeholder prediction column matching the target format; fill predictions with random or dummy values as examples. - Also createanswer.csv containing the true labels for test data in the same format assample_sub- mission.csv(for hidden evaluation). Constraints: - Script must be standalo...
-
[10]
Hardcode metric, direction (is_lower_betterbool), and thresholds at the top (derive metric/direction from Task DNA + schema)
-
[11]
CLI: accept–submission_path(default:sample_submission.csv)
-
[12]
Load submission and ground truth, merge on id column if present; if no id, align by row order with a warning
-
[13]
Compute the specified metric
- [14]
-
[15]
On any error, print a JSON error object to stdout (not stderr) and exit gracefully
-
[16]
Use only standard libraries plus numpy/pandas/sklearn if needed
No external configs or downloads. Use only standard libraries plus numpy/pandas/sklearn if needed
-
[17]
Ensure every string literal is closed and every dict/list is syntactically complete
Emit pure Python: no markdown fences, no stray triple quotes. Ensure every string literal is closed and every dict/list is syntactically complete. 27 Table 9Prompt template for the Technical Writer agent. Technical Writer Prompt Template You are a technical writer for synthetic ML benchmarks. Using the provided Task DNA, an example description from the se...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.