Pith.

Artificial Intelligence · arXiv:2610.22194 v1 · submitted 15 Oct 2026 · 32 references

AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning

Bingxiang He, Pengqi Sun, Shihao Peng, Shaoxuan Li, Huiyu Xiao, Hongjian Shi, Zixuan Fu, Ziqing Qiao

show 13 more authors

Yuxin Zuo, Cheng Qian, Xinqiang Yu, Jiayi Li, Xiusi Chen, Yizhen Song, Xiangyang Wang, Iofai Lei, Yudong Wang, Ning Ding, Zhiyuan Liu, Xiaozhi Wang, Chaojun Xiao

The paper says

“Language model agents increasingly automate post-training, but when given the full training stack they converge on supervised fine-tuning and spend most effort revising training data.”

more of the abstract

Existing systems give the agent one kind of control, selecting rows from a fixed pool or calling a generator, whereas a human data engineer adjusts proportions, prunes, repairs, replaces and synthesizes the data. We design a data action space for these changes and ask whether an automated system can make the full range rather than one of them, and what structure that requires. We first run this loop with a single agent, and it reaches only one action, adjusting category proportions, while its context fills with execution output and is discarded mid-search. We therefore introduce \textbfAutoSFT, a multi-agent system where a meta agent separates execution from analysis, gathers proposals from independent angles, and calls a synthesis subagent for capabilities the pool cannot cover, with the model, the trainer, and the budget fixed within each comparison. Adding each component in turn helps: the executor halves context loss, the analysis subagents lift the best IFEval by + 6.5, and synthesis adds + 3.4 IFEval and + 2.9 IFBench. The same loop transfers to code, where analysis adds + 3.7 MBPP+ over a single agent that does not improve. Every result comes from runs that retrain from the base model each round, so each component's effect is individually attributable. Our findings suggest that automating SFT data research depends more on covering diverse data actions than on a stronger single action.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A well-built agentic system for SFT data research with unusually honest ablations, but every component attribution rests on one search per configuration, so the headline deltas are not yet firm. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 32 references say.

The paper says, at §5.2, §5.3, §5.4, Appendix E

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.87):

“Each comparison runs one search per configuration, and although repeated evaluation bounds the noise of a single checkpoint, it does not bound the variation between independent searches.”

Pith says: Objection 1 of 3 · major

Every component delta (+6.5 IFEval for analysis, +3.4 IFEval and +2.9 IFBench for synthesis, +3.7 MBPP+ for transfer) is the difference between a single search of the base configuration and a single search of the augmented configuration. Appendix E states this explicitly: repeated avg@10 evaluation bounds checkpoint noise but not variation between independent searches.

more of objection 1

Because the paper's main conclusion is causal, these single draws are not sufficient to attribute the gains to the added components. I request repeated independent searches per configuration, with the mean and spread of the best scores, or an equivalent paired design. From-base retraining is a strength, but it does not control search-to-search variance.

The paper says, at §4.3, §5.3, Ethics statement

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.60):

“Because the AutoSFT optimizes benchmark scores, it could in principle be used to inflate leaderboard numbers, and we report off- target and held-out benchmarks so that a gain from overfitting the target is visible rather than hidden.”

Pith says: Objection 2 of 3 · major

The synthesis subagent selects capability targets from evaluation results, and the reported gains are on the same benchmarks used to steer target selection. The n-gram overlap check and verifier reduce direct memorization, but they do not eliminate benchmark-specific overfitting through capability targeting.

more of objection 2

The Ethics statement says the paper reports off-target and held-out benchmarks, but I could not find such results for the instruction-following setting; Section 5.4 transfers to a different domain, which is not a held-out evaluation of the same optimization target. Please either add a held-out or off-target benchmark for the main setting or qualify the claim about overfitting being visible.

The paper says, at §5.2, §5.3, Table 6

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The token budgets differ across the comparisons that together support the central conclusion: the analysis ablation uses a 3M-token cap, while the synthesis and code comparisons use a 100M-token cap. Within each comparison the budget is fixed, which supports the per-component claim, but the paper's broader conclusion that action diversity matters more than a stronger single action requires the component effects to be comparable.

more of objection 3

A budget-dependent interaction is plausible: analysis may add more value when the budget is small. Please discuss this explicitly or add a condition where the analysis comparison is repeated at the larger budget.

4 minor objections

The paper says, at Abstract and §7

Pith says: minor objection 1 of 4

The phrase 'so each component's effect is individually attributable' is stronger than the evidence and than Appendix E; consider replacing it with 'under the fixed within-comparison settings, with one search per configuration' or similar.

The paper says, at §5.4

Pith says: minor objection 2 of 4

The transfer result is reported as 'synthesis reaches a comparable peak in fewer rounds,' but the synthesis configuration completes 6 rounds versus 15 for the other two configurations in the same wall-clock budget. The comparison should be made on wall-clock time or the different round counts should be discussed as a cost trade-off.

show 2 more

The paper says, at Table 4, Appendix C.1

Pith says: minor objection 3 of 4

The new-best rates are computed from single runs and are not accompanied by confidence intervals; the text should note that these rates are descriptive rather than statistically robust.

The paper says, at Ethics statement

Pith says: minor objection 4 of 4

The statement that 'we report off-target and held-out benchmarks' should be reconciled with the actual benchmark list in Table 6, since no off-target benchmark appears there.

What would settle it

Run five independent searches per configuration for the comparisons in Sections 5.2 and 5.3, and five for the code comparison in Section 5.4, varying only the random seed and the initial agent prompt while holding the base model, trainer, token budget, and evaluation protocol fixed. Report the distribution of best scores per configuration and a bootstrap 95% CI on each paired delta: 67.10 vs 60.60 IFEval (analysis), 81.58 vs 78.15 IFEval and 51.95 vs 49.02 IFBench (synthesis), and 65.8 vs the base peak on MBPP+. If any CI includes zero, that component attribution fails and the diversity-over-strength conclusion is not established. The released code and run ledger make this a rerun rather than a new experiment.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 17 s · 31,016 tokens read, 8,971 written
  3. Editor wrote the verdict above 17 s · 31,963 tokens read, 1,650 written
  4. Referee major revision: 3 major objections, 4 minor 32 s · 31,361 tokens read, 3,531 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Quick take: this is a genuinely useful systems paper, and the authors have already written down the main weakness themselves. Each component delta comes from one search per configuration, so the +6.5 IFEval, +3.4 IFEval, +2.9 IFBench, and +3.7 MBPP+ figures could partly be search-to-search variance rather than the added component. The paper says this verbatim in Appendix E (b430): repeated avg@10 evaluation bounds noise of a single checkpoint, not variation between independent searches. That does not sink the paper, but it means the central claim should be read as a strong hypothesis, not a measured fact.

What I think is genuinely new: the six-family data action space labelled from recipe diffs, and the from-base retraining protocol that makes per-component attribution possible in principle. The pilot study is also worth reading on its own. The single-agent collapse to one action family, the context filling and discarding ten times, and the trace evidence are concrete and well documented. The executor/analysis/synthesis decomposition is a sensible response to those failure modes, and the ablations are clearly staged.

Credit where it is due: the paper is unusually candid. It flags the one-search-per-configuration issue, the single model scale (b429), the possibility that benchmark-specific overfitting drives gains, and the fact that analysis actions set new bests only some of the time. The code and run ledger are promised, and the evaluation protocol is fixed and described. This is not a paper that hides its soft spots.

Soft spots beyond the main one. Token budgets differ across comparisons (3M for analysis, 100M for synthesis), so the component magnitudes are not directly comparable with each other. The synthesis subagent selects targets from the same evaluation results the paper reports gains on; n-gram decontamination and the verifier reduce but do not eliminate the overfitting risk. And all results are on Qwen3-1.7B-Base, which the authors acknowledge. These are real limitations, but they are proportionate and mostly stated.

Bottom line: this deserves a serious referee. The right next experiment is repeated independent searches per configuration, ideally at more than one model scale, with the released ledger checked against the reported numbers. The action-diversity thesis is plausible and directionally supported; it just is not numerically pinned down yet.

The strongest objection

AutoSFT's central claim is that covering diverse data actions matters more than a stronger single action (Section 7, sentence b7). The evidence is component-wise attribution: analysis adds +6.5 IFEval at equal rounds (Section 5.2, b127-b128), synthesis adds +3.4 IFEval and +2.9 IFBench (Section 5.3), and the loop transfers to code with +3.7 MBPP+ (Section 5.4). Every one of these deltas is the difference between a single search of the base configuration and a single search of the augmented configuration. Appendix E states the limitation in b430: repeated avg@10 evaluation bounds the noise of one checkpoint, not the variation between independent searches. From-base retraining (Section 2.1) prevents weight history from leaking across rounds, but it does not prevent search luck from favoring one configuration. Because the conclusion is causal and the effect sizes are single draws, the numerical claims are not yet attributable to the added components. The authors deserve credit for flagging this; the flag identifies exactly the experiment needed rather than supplying it.

The reader identified the same weakest premise, and I agree: this is the load-bearing point. The paper is unusually explicit about it in Appendix E, so the task is a verification condition on the central claim rather than a contradiction of it. I considered whether the main concern should instead be that the analysis ablation adds four subagents plus a meta agent and therefore changes agent count and inference budget, not only action diversity. The per-round action traces and per-action new-best rates (Section 5.2, Table 4) provide partial evidence that the mechanism is action coverage, but they are not a controlled counterfactual. However, the search-variance issue is more load-bearing because it threatens every reported delta, including the code transfer, and it is the premise on which the causal wording in Section 7 rests. If repeated independent searches reproduce the deltas with confidence intervals excluding zero, the conditional verdict can be upgraded; if not, the component attribution and the central conclusion are unsupported. Nothing in this pass changes the reader's conditional verdict, and I found no internal inconsistency or missing proof in the system description that would require a stronger action.

The referee's report

If the attribution claims hold, the paper makes a useful contribution: it defines a six-family data action space, documents concrete failure modes of a single-agent loop with trace-level evidence, and proposes a modular architecture whose components are individually motivated. The authors are unusually candid about limitations, and the design choices of from-base retraining, fixed within-comparison budgets, action labeling from recipe diffs, and a verifier plus n-gram decontamination gate for synthesis are all strengths. The released code and planned run ledger are valuable for reproducibility. However, the central causal claim currently rests on one search per configuration, and the paper itself identifies this in Appendix E; the component-level numerical claims are therefore not yet fully supported.

Pith's reading of the paper (a machine's summary, not the authors')

The paper claims that what limits an AI agent automating supervised fine-tuning (SFT) data research is not any single data-manipulation capability but whether the system can cover the full range of actions a human data engineer uses. In a controlled pilot, a single agent given the whole research loop collapsed to one action family, adjusting category proportions, never edited or wrote examples, and had its context window compacted to near-empty ten times. The authors build AutoSFT, a multi-agent loop that delegates execution, analysis, and synthesis to separate subagents, and they report that each added component improves results when tested alone: the executor halves context loss, four analysis subagents lift best IFEval by +6.5, and a synthesis subagent adds +3.4 IFEval and +2.9 IFBench. The same structure transfers to code, where analysis adds +3.7 MBPP+ over a single agent that does not improve. The paper's central claim, that covering diverse data actions matters more than a stronger single action, would matter if true because it redirects how post-training automation should be engineered.

What carries the argument. The carrying object is the data action space, six families of change a data engineer makes to a recipe: mixture (category proportions), pruning, repair, resupervision (replacing responses), synthesis (writing new examples), and ordering, with actions labelled after the run from the diff between consecutive recipes rather than from the agent's stated intent. The carrying mechanism is the from-base research loop: training re-initializes from the fixed base checkpoint every round, so the pool and recipe are the only quantities that carry history, which is what makes each component's effect individually attributable. AutoSFT instantiates the research policy as a meta agent that delegates to an executor subagent running training and evaluation in its own session, four analysis subagents that each propose one data action from a complementary evidence source, and a synthesis subagent that writes new rows only against a programmatic verifier for an undersupplied capability.

Core claim. The central claim is that automating SFT data research depends more on covering diverse data actions than on a stronger single action. The paper supports it with component-wise ablations in which every round retrains from the same base checkpoint, so only the added component differs within each comparison. A single agent owning the full loop runs 33 rounds without improving past round 9 and stays inside the mixture family, reweighting category proportions with 22 different phrasings of the same move; the executor that isolates training and evaluation from the deciding agent reduces context compactions from ten to five, the four analysis subagents reading the same round from different evidence sources raise the best IFEval from 60.60 to 67.10, and the synthesis subagent that generates verifier-checked examples for undersupplied capabilities adds +3.4 IFEval and +2.9 IFBench. In the code domain, where no capability taxonomy exists, analysis adds +3.7 MBPP+ over a single agent that shows no sustained improvement, which the authors read as evidence that the structure transfers.

The premise it rests on. The load-bearing premise is that one search per configuration is enough to attribute the observed gains to the added component: the paper acknowledges in its limitations that repeated evaluation bounds checkpoint noise but not variation between independent searches, so the +6.5, +3.4, and +3.7 deltas could in principle be search luck rather than component effects.

If the paper is right (Pith's inferences, not the authors')

Post-training automation systems should be engineered for action coverage rather than for a single stronger selection or generation operator, since the reported gains come from adding families of action, not from strengthening one.

Long-horizon agent loops need context isolation as a design requirement: execution output must leave the deciding agent's window, or the search collapses onto one repeated move.

Complementary analysis angles are a practical way to keep proposal diversity, and synthesis only pays when the pool undersupplies a capability the model can learn, since not every generated batch helps.

Because rounds retrain from base, accepted batches do not compound through continued training; gains accumulate only through pool and recipe history, so the reported per-component effects are measured at the base model's capacity.

The code transfer suggests the loop can build its own capability structure from benchmark and pool when no taxonomy is given, which widens the method's applicability beyond instruction following.

A testable prediction follows from the synthesis results: per-capability gains should correlate with pool supply deficit for capabilities the model can learn, and should stay near zero for capabilities it cannot follow, as with the odd_even_syllables target; ranking capabilities by supply and correlating with per-capability score change would check this.

The context-compaction analysis implies that any fixed-window agent loop will degrade toward repetitive actions as its window fills; running the single-agent loop with a much larger context window would isolate whether context pressure or proposal-confirmation bias is the binding constraint, a distinction the paper's own trace analysis leaves open.

The from-base protocol caps the loop at the capacity of one base model, so a system that also allowed continued training between rounds might reach higher scores, but at the cost of the clean component attribution this paper relies on; the two designs may serve different purposes.

The single-scale results suggest a direct extension: rerunning the same ablations on a larger base model would show whether the value of the multi-agent structure shrinks, stays flat, or grows with model capacity, which the paper does not address.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Language model agents increasingly automate post-training, but when given the full training stack they converge on supervised fine-tuning and spend most effort revising training data. Existing systems give the agent one kind of control, selecting rows from a fixed pool or calling a generator, whereas a human data engineer adjusts proportions, prunes, repairs, replaces and synthesizes the data. We design a data action space for these changes and ask whether an automated system can make the full range rather than one of them, and what structure that requires. We first run this loop with a single agent, and it reaches only one action, adjusting category proportions, while its context fills with execution output and is discarded mid-search. We therefore introduce \textbfAutoSFT, a multi-agent system where a meta agent separates execution from analysis, gathers proposals from independent angles, and calls a synthesis subagent for capabilities the pool cannot cover, with the model, the trainer, and the budget fixed within each comparison. Adding each component in turn helps: the executor halves context loss, the analysis subagents lift the best IFEval by + 6.5, and synthesis adds + 3.4 IFEval and + 2.9 IFBench. The same loop transfers to code, where analysis adds + 3.7 MBPP+ over a single agent that does not improve. Every result comes from runs that retrain from the base model each round, so each component's effect is individually attributable. Our findings suggest that automating SFT data research depends more on covering diverse data actions than on a stronger single action.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 1.00

“Our findings suggest that automating SFT data research depends more on covering diverse data actions than on a stronger single action.”

How the main claim is stated: hedged possibility 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.53 · human participants 0.05 · controlled comparison 0.88 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.90 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Language model agents increasingly automate post-training, but when given the full training stack they converge on supervised fine-tuning and spend most effort revising training data. background 0.96 0.32 0.05 asserted 1.00
Existing systems give the agent one kind of control, selecting rows from a fixed pool or calling a generator, whereas a human data engineer adjusts proportions, prunes, repairs, replaces and synthesizes the data. gap 0.72 0.08 0.04 asserted 1.00
We design a data action space for these changes and ask whether an automated system can make the full range rather than one of them, and what structure that requires. contribution 0.50 0.39 0.04 unclear 0.73
We first run this loop with a single agent, and it reaches only one action, adjusting category proportions, while its context fills with execution output and is discarded mid-search. result 0.87 0.95 0.03 asserted 1.00
We therefore introduce \textbf{AutoSFT}, a multi-agent system where a meta agent separates execution from analysis, gathers proposals from independent angles, and calls a synthesis subagent for capabilities the pool cannot cover, with the model, the trainer, and the budget fixed within each comparison. method 0.66 0.92 0.08 asserted 0.99
Adding each component in turn helps: the executor halves context loss, the analysis subagents lift the best IFEval by $+6.5$, and synthesis adds $+3.4$ IFEval and $+2.9$ IFBench. result 1.00 0.97 0.08 asserted 1.00
The same loop transfers to code, where analysis adds $+3.7$ MBPP+ over a single agent that does not improve. result 1.00 0.95 0.03 asserted 1.00
Every result comes from runs that retrain from the base model each round, so each component's effect is individually attributable. method 0.94 0.95 0.03 asserted 1.00
Our findings suggest that automating SFT data research depends more on covering diverse data actions than on a stronger single action. interpretation 1.00 0.93 0.90 hedged possibility 1.00

What kind of paper the classifier reports this is: method or tool 0.98 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 7,091 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (16)
Figure 1 of the paper
Figure 1. Overview of AutoSFT. The meta agent commits a recipe. An executor trains and evaluates it separately and returns artifact paths. Analysis subagents read these artifacts from complementary angles, and a synthesis subagent writes data for undersupplied capabilities. The meta agent integrates their proposals into the next recipe. All roles share a candidate data pool and a skill library. data. We write these six kinds of change as a data action space (Section 2), and use it only after a run to measure how much of that space the agent reaches. We first study this question with a…
Figure 2 of the paper
Figure 2. One round of the research loop. Only 𝜋 varies across the configurations we compare. these results suggest that automating SFT data research depends less on a stronger single action than on structure that lets an agent cover the range of changes a data engineer makes. Contributions. Our contributions are threefold. • We frame automated SFT data research around a space of six data-engineering actions, and show that a corrected single-agent loop reaches only one of them (Sections 2 and 3). • We build AutoSFT, a multi-agent system that isolates execution, gathers proposals from…
Figure 3 of the paper
Figure 3. The six families of data action, drawn as their effect on the rows of a recipe. Shading marks what the action changes. 2.2. Data research actions A recipe is built out of rows, where a row carries an instruction, a response, and the metadata that describes it. An action states which rows to train on, how often each appears, in what order, and whether any row is edited or newly written. Agents write recipes in free form with no field schema, operator list, and statement that a particular kind of change is available
Figure 4 of the paper
Figure 4. Left: benchmark scores over the 32 evaluated rounds on separate axes. Right: the terms the agent used to state its chosen data action, sized by how often it used them. sequence length 32,768, with global batch size 16 and learning rate 10−4 . The loop stops when IFEval reaches 80 and IFBench reaches 50. A single Codex agent running GPT-5.5 at medium reasoning effort submitted 33 rounds and completed evaluation on 32 of them within 11.5 hours, at which point neither target had been reached. The search stays inside one family of actions. Every round adjusts how much of each category…
Figure 5 of the paper
Figure 5. Training data per round under a wall￾clock budget. Time Limits Reduce Per-Round Training Data. The open-ended run is allotted ten hours, and
Figure 6 of the paper
Figure 6. The single agent’s context. Dashed lines mark compactions labelled with the round. instead to a few large payloads: training output at 13.4%, repeated reads of the run record at 11.8%, applying the data action at 9.7%, and evaluation output at 8.9%. With the agent’s reasoning (16.3%) and compaction summaries (8.3%), six sources take two thirds of the window. None of them relates to choosing the next data action, which argues for giving those steps their own session. 3.2. From observations to requirements The second observation is answered by the objective itself, and the…
Figure 7 of the paper
Figure 7. Context used by the meta agent over each run and the composition of its recorded events. 5.1. Executor subagent Setup. The base configuration is the single agent of Section 3.1. Adding the executor places that lifecycle in a separate session, so the meta agent receives artifact paths and a brief status report rather than executing the lifecycle. Both runs begin from the same base checkpoint, share the same context window, and are evaluated from snapshots taken at 11.5 hours. All other settings are identical. Result. The executor halves how often the meta agent loses its context.…
Figure 8 of the paper
Figure 8. Left: IFEval per round, with dotted guides at the rounds named in the text. Right: distribution of data actions over
Figure 9 of the paper
Figure 9. Scores per round. Left two panels: synthesis ablation on instruction following (Base vs Base + Synthesis, Section 5.3). Right panel: three configurations on code generation (Section 5.4). Bands give the standard deviation over evaluation repeats. 5.4. Generalization to code domain Setup. To assess whether AutoSFT remains effective beyond instruction following, we evaluate it on code generation with MBPP+2 . Unlike IFEval and IFBench, MBPP+ grades only by test execution and carries no capability breakdown, so the system builds its own capability structure from only the benchmark…
Figure 10 of the paper
Figure 10. Condensed trace of the single-agent run, 33 rounds over 11.5h. Red marks an outcome that contradicts the round’s own description. Rounds not shown repeat the pattern of others. The breakdown is coarse in one respect worth stating. Categories are assigned from the command that produced each record, so evaluation output that the agent reads through a general file read is counted as a file read rather than as evaluation output. The shares establish that a few large payloads dominate. They do not establish that context pressure caused the scores to stop improving. B. Details of…
Figure 11 of the paper
Figure 11. The executor is told what to run and is denied the one thing that would let it influence the round, which is reading the scores. maintaining both the criteria and the verifier as it goes so that omissions are corrected without changing the intended capability. C. Details of the experiments C.1. Analysis subagents
Figure 12 of the paper
Figure 12. One of the four analysis skills. The others replace only the Scope section, so the four subagents start from different evidence and return the same kind of object. the number of questions the capability contains, and capabilities backed by fewer than ten questions are drawn faded because their differences are likely to reflect small-sample noise. Most targeted capabilities show higher rates with synthesis, and the advantage is largest and best supported for repetition, sentence and word counts, and comma avoidance, each backed by dozens of questions. A few capabilities show lower…
Figure 13 of the paper
Figure 13. One skill given to the synthesis subagent. It fixes the internal loop by which a worker turns a target into a delivered batch, while leaving the generation strategy to the worker. The generation, verification, and decontamination checks it enforces are detailed in Appendix B.4. draw from the same open instruction-following pool of Section 3 but under different token budgets, and the code experiments use a Python pool with a shorter sequence length. A benchmark score is reported as avg@k, the mean over 𝑘 stochastic evaluations of one checkpoint. IFEval and IFBench use 𝑘=10 and…
Figure 14 of the paper
Figure 14. The second decision of Section 5.2. Explanations reflect the reasoning recorded at the time, and strict repetition is measured per instruction. 0 4 8 0 4 8 1 5 10 15 19 Round 0 25 50 75 100 Strict Pass (%) 75.61 repeat_prompt 1 5 10 15 19 Round 0 25 50 75 100 odd_even_syllables Rows Synthesized (thousands)
Figure 15 of the paper
Figure 15. Two capabilities over the run with synthesis. The line is the capability’s whole-question strict pass rate each round, and the bars are the rows synthesized for it that round. E. Limitations All results are obtained on one base model at one scale (Qwen3-1.7B-Base), so the effects we report may not hold at other scales, and a capability this model cannot learn from more data may be learnable by a larger one. Each comparison runs one search per configuration, and although repeated evaluation bounds the noise of a single checkpoint, it does not bound the variation between…
Figure 16 of the paper
Figure 16. Per-capability comparison of the two runs of Section 5.3. 25

Figures are the authors', reproduced from the paper.

References (32)
  1. [3] Wanyi Chen, Xiao Yang, Xu Yang, Tianming Sha, Qizheng Li, Zhuo Wang, Bowen Xian, Fang Kong, Weiqing Liu, and Jiang Bian. Agent2 RL-Bench: Can LLM agents engineer agentic RL post-training?, 2026a. URLhttps://arxiv.org/abs/2604.10547. Xiusi Chen, Jyun-Yu Jiang, Wei-Cheng Chang, ... on PithPith's reviewarXiv
  2. [5] URLhttps://arxiv.org/abs/2310.01377. Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving LLM alignment via preference data selection. InAdvances in Neural Information Processing Systems (NeurIPS), on PithPith's reviewarXiv
  3. [6] URLhttps://arxiv.org/abs/2502.14560. Aniketh Garikaparthi, Manasi Patwardhan, and Arman Cohan. ResearchGym: Evaluating language model agents on real-world AI research, on PitharXiv
  4. [7] URLhttps://arxiv.org/abs/2602.15112. BingxiangHe, NingDing, ChengQian, JiaDeng, GanquCui, LifanYuan, HaiwenHong, Huan-angGao, Longtao Huang, Hui Xue, Huimin Chen, Zhiyuan Liu, and Maosong Sun. The right time matters: Dataarrangementaffectszero-shotgeneralizationininstructiontu... on PitharXiv
  5. [8] URLhttps://arxiv.org/abs/2310.03302. Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Xianzhen Luo, Qiufeng Wang, YuanTao Fan, Qingfu Zhu, Zhaoxiang Zhang, Yang Gao,... on PithPith's reviewarXiv
  6. [9] doi: 10.18653/v1/2025.acl-long on Pithdoi:10.18653/v1/2025.acl-long
  7. [11] URLhttps://arxiv.org/abs/2606.11926. Feiyang Kang, Hanze Li, Adam Nguyen, Mahavir Dabas, Jiaqi W. Ma, Frederic Sala, Dawn Song, and Ruoxi Jia. Can generalist agents automate data curation?, on PithPith's reviewarXiv
  8. [12] URLhttps://arxiv.org/ab s/2606.04261. 13 AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning Zaid Khan, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. DataEnvGym: Data generation agents in teacher environments with student feedback. InThe Thirt... on PithPith's reviewarXiv
  9. [13] URLhttps://arxiv.org/abs/2410.06215. Spotlight. Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, Matei Zaharia, and Ang Chen. EXP-Bench: Can AI conduct AI researc... on PithPith's reviewarXiv
  10. [14] URLhttps://arxiv.org/abs/2505.24785. Ilia Kulikov, Chenxi Whitehouse, Tianhao Wu, Yixin Nie, Swarnadeep Saha, Eryk Helenowski, Weizhe Yuan, Olga Golovneva, Jack Lanchantin, Yoram Bachrach, Jakob Foerster, Xian Li, Han Fang, Sainbayar Sukhbaatar, and Jason Weston. Autodata: An ... on PithPith's reviewarXiv
  11. [16] URLhttps://arxiv.org/abs/2604.05550. Hao Liang, Qifeng Cai, Yibo Lin, Jianzhuo Du, Qifeng Xia, Sizhe Qiu, Linzhuang Sun, Meiyi Qiang, ZhaoyangHan, XiaochenMa, BohanZeng, RuichuanAn, ConghuiHe, andWentaoZhang. DataPrep- Bench: Benchmarking LLMs as training data preparators.arXi... on PithPith's reviewarXiv
  12. [17] URLhttps://arxiv.org/abs/2607.20465. Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, and Jeff Clune. Towards end-to-end automation of AI research.Nature, 651:914–919, on PithPith's reviewarXiv
  13. [19] URL https://arxiv.org/abs/2605.30407. Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, Kaicheng Yang, Zitao Chen, Wentao Guo, Junlin Yang, Xinyue Ai, Wenhao Chai, Yadi Cao, Ziran Yang, Kun Wang, ... on PithPith's reviewarXiv
  14. [20] URL https://arxiv.org/abs/2605.08678. Zerun Ma, Guoqiang Wang, Xinchen Xie, Yicheng Chen, He Du, Bowen Li, Yanan Sun, Wenran Liu, Kai Chen, and Yining Li. TREX: Automating LLM fine-tuning via agent-driven tree-based exploration. arXiv preprint arXiv:2604.14116, on PithPith's reviewarXiv
  15. [21] URLhttps://arxiv.org/abs/2604.14116. Fanqing Meng, Lingxiao Du, Qiguang Chen, Ziqi Zhao, Haocheng Lu, Mengkang Hu, and Michael Qizhe Shieh. RSIBench-Data: Benchmarking data-centric research for recursive self- improvement.arXiv preprint arXiv:2607.25886, on PithPith's reviewarXiv
  16. [23] URLhttps://arxiv.org/abs/2407.03502. 14 AutoSFT: An Agentic System for Automated Data Research in Supervised Fine-Tuning Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei Li, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori H... on PithPith's reviewarXiv
  17. [24] URLhttps://arxiv.org/abs/2501.19393. Anne Ouyang, Simon Guo, Simran Arora, Alex L. Zhang, William Hu, Christopher Ré, and Azalia Mirhoseini. KernelBench: Can LLMs write efficient GPU kernels? InProceedings of the 42nd International Conference on Machine Learning, on PithPith's reviewarXiv
  18. [26] URLhttps://arxiv.org/abs/2603.08640. QingyuRen,QianyuHe,JiajieZhu,XingzhouChen,JingwenChang,ZeyeSun,HanXia,FeiYu,Jiaqing Liang, and Yanghua Xiao. SEIF: Self-evolving reinforcement learning for instruction following. arXiv preprint arXiv:2605.07465, on PitharXiv
  19. [27] URLhttps://arxiv.org/abs/2605.07465. Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. AI-Researcher: Autonomous scientific innovation. InAdvances in Neural Information Processing Systems, volume 38, on PithPith's reviewarXiv
  20. [28] doi: 10.52202 /085713-0320. URL https://proceedings.neurips.cc/paper_files/paper/2025/ha sh/0d904d300a105809a2114d727851e759-Abstract-Conference.html. TheEndlessFrontierTeam. BigBang: Pursuingopen-endedintelligencethroughself-evolvingsynthesis of verifiable frontier tasks. Tec... on Pith
show 12 more
  1. [30] URLhttps://alignment.anthropic. com/2026/automated-w2s-researcher/. Published April on Pith
  2. [32] URL https://arxiv.org/abs/2605.12944. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. InThe Twelfth International Conference ... on PithPith's reviewarXiv
  3. [33] URLhttps://arxiv.org/abs/ 2503.02951. Zhangchen Xu, Junda Chen, Yue Huang, Dongfu Jiang, Jiefeng Chen, Hang Hua, Zijian Wu, Zheyuan Liu, Zexue He, Lichi Li, Shizhe Diao, Jiaxin Pei, Jinsung Yoon, Hao Zhang, Mengdi Wang, Radha Poovendran, Misha Sra, Alex Pentland, and Zichen Ch... on PithPith's reviewarXiv
  4. [34] URLhttps://arxiv.org/abs/2606 .05080. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, on PithPith's reviewarXiv
  5. [35] URLhttps://arxiv.org/abs/2607.20468. Yuhao Zhan, Bingxiang He, Zecong Tang, and Chaojun Xiao. PACE-Bench: Benchmarking physics adaptation via code evolution in dynamic environments, on PithPith's reviewarXiv
  6. [36] URLhttps://arxiv.org/abs/ 2608.14441. Yunxiang Zhang, Muhammad Khalifa, Shitanshu Bhushan, Grant D Murphy, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. MLRC-Bench: Can language agents solve machine learning research challenges? InAdvances in Neura... on PithPith's reviewarXiv
  7. [37] URLhttps://arxiv.org/abs/2504.09702. Zhengyang Zhao, Shengjie Ye, Lu Ma, Hao Liang, Hengyi Feng, and Wentao Zhang. ANDES: Agent native data evolving synthesis tool for autonomous instruction alignment.arXiv preprint arXiv:2606.01279, on PitharXiv
  8. [38] URLhttps://arxiv.org/abs/2606.01279. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. InAdvances in Neural I... on PithPith's reviewarXiv
  9. [2023] URLhttps://arxiv.org/abs/2212.10560. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empowering code generation with OSS-instruct. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learnin... on PithPith's reviewarXiv
  10. [2024] Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, and Kai Chen. MIG: Automatic data selection for instruction tuning by maximizing information gain in semantic space. InFindings of the Association for Computational Linguistics: ACL 2025,2025. doi: 10.18653/v1/2025.finding... on PithPith's reviewarXiv
  11. [2025] URLhttps://arxiv.org/abs/2411.04425. Φ-Bench contributors.𝜙-bench: Can large language models engineer the infrastructure that powers them?https://faibench.org/, on PithPith's reviewarXiv
  12. [2026] arXiv preprint forthcoming. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. InInternational Confe... on Pith
Cited by 2 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment