REVIEW 3 major objections 4 minor 24 references
A requirement-driven framework synthesizes executable agent tasks from a high-level capability spec, and supervised fine-tuning on that data lifts LLM agents on both terminal and office benchmarks.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 01:38 UTC pith:3XUTQIGL
load-bearing objection A credible, controlled result on requirement-driven task synthesis; the main caveats are the absence of released code/data and thinner evidence for the headline scaling numbers. the 3 major comments →
NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that scaling executable agent training data is bottlenecked by substrate-bound methods, and that the bottleneck can be removed by making task generation requirement-driven. Concretely, NexForge synthesizes executable tasks and expert trajectories from a high-level capability requirement alone: 3.6K terminal tasks improve Qwen3.5-35B-A3B from 22.5% to 52.0% on Terminal-Bench 2.0, and 2K office tasks improve the same base model from 813 to 1338 Elo on GDPval; scaling to 43.2K terminal tasks reaches 58.4%, and production-scale NexForge data yields open models reaching 75.3% on Terminal-Bench 2.1 and 1585 Elo on GDPval. The paper argues these results show that separating the int
What carries the argument
The central object is the task directive, a structured specification of scenario, task form (primary type, expected deliverable, source strategy, runtime environment), compatible option sets, language, difficulty, and composition rationale, fixed before any material is collected. It carries the argument by serving as the contract that the downstream environment-instantiation stages must honor, so a repository or document is selected to serve the intended task rather than the task being inferred from available substrate. The scenario-conditioned compatibility filter is the key mechanism that keeps directives coherent, reducing the candidate space so that, on average, 4.4 of 24 task types rema
Load-bearing premise
Environment materialization faithfully realizes every sampled directive as a runnable workspace, so the corpus distribution reflects the specified demand rather than the constraints of the Docker/default generation pipeline; if hard, diverse, or long-horizon directives fail more often than easy ones, the data is silently biased.
What would settle it
Compare materialization success rates across directive difficulty, horizon, and novelty (e.g., by instrumenting the 3,600-to-3,501 drop in Terminal-3.6K per difficulty bucket). If hard/novel directives fail substantially more often than easy ones, the realized task distribution is substrate-biased, contradicting the claim that the data reflects real-world demand. Alternatively, apply the unmodified pipeline to a third capability domain (e.g., scientific computing) with only a specification change: if it requires domain-specific adaptations to produce an effective corpus, the 'no domain-specifi
If this is right
- Capability expansion reduces to specification writing: a new domain needs a high-level requirement, not a bespoke pipeline; the paper demonstrates this by producing both terminal and office corpora from the same framework.
- Corpus-level task distribution becomes a controllable design variable, not an emergent property of tools or repositories; ablations show task-form sampling prevents mode collapse and scenario grounding shapes realism.
- Data volume is not bottlenecked by curated materials or skill graphs; increasing tasks from 3.6K to 43.2K improves Terminal-Bench 2.0 accuracy from 52.0% to 58.4% with one rollout per task.
- The data is useful for SFT even without task-specific verifiers or reference answers, as trajectories encode long-horizon tool use, intermediate failures, and recovery behavior (averaging 129K tokens and 123 tool calls per terminal trajectory).
- Improvements hold across very different base models (+26.7 points for Qwen3-32B, +29.5 for Qwen3.5-35B-A3B), suggesting the synthetic data teaches general interaction patterns rather than merely exploiting a specific model's weaknesses.
Where Pith is reading between the lines
- If substrate independence holds, the limiting resource for agent post-training shifts from environment engineering to the quality of the capability specification and to validation that materialization faithfully covers hard and novel directives; the 3,600-to-3,501 usable-task drop in Terminal-3.6K marks the spot to test.
- A stress test for the real-world-demand claim: instrument materialization success by directive difficulty, horizon, and source strategy. If hard directives fail disproportionately, the resulting corpus inherits a substrate bias, and the distributional claims would need to be restated as 'realistic among what the pipeline can instantiate.'
- The lack of machine-verifiable outcomes (explicitly noted by the authors) means the framework currently serves SFT but not RL or benchmarking; adding reference solutions and verifiers would let the same requirement-driven process feed both training paradigms.
- An independent check of distributional realism would compare synthesized task signatures against actual recorded terminal sessions or office workflow logs; the paper does not provide such a comparison, so the demand profile's fidelity remains an open question.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NexForge, a requirement-driven pipeline that synthesizes executable agent tasks and expert trajectories from a high-level capability description. It consists of three stages: research-based demand discovery, distribution-aware task compilation, and environment instantiation with trajectory collection. The authors evaluate the framework on two disparate domains — terminal operations and office work — showing that SFT on NexForge-generated data improves Qwen3.5-35B-A3B from 22.5% to 52.0% on Terminal-Bench 2.0 and from 813 to 1338 GDPval Elo. Additional experiments examine data scaling, ablations of task-form control and scenario grounding, and the contribution of NexForge data to the Nex-N2 model family.
Significance. The central claim is significant: if a single generic pipeline can synthesize executable, diverse, and useful agent training data without domain-specific infrastructure, this would lower the cost of scaling agent post-training across capabilities. The paper provides several concrete strengths: four independent evaluations per controlled run, a clear failure-counting protocol, external benchmarks that are not used in data construction, and publicly released models. The ablations are carefully designed and the distributional analysis of task types is informative. The main risk is that the claimed substrate independence may be weakened by silent materialization failures and by single-run scaling results that lack confidence intervals.
major comments (3)
- [Section 3.4, Table 7] The materialization stage silently drops packages: Table 7 shows 3,600 packages yielding 3,501 terminal tasks, 2,000 yielding 1,947 for Terminal-2K, and 1,916/1,894 for the ablation variants. The manual audit in Section G only inspects kept packages. The paper reports realized distributions (Section 4.2, Table 18) but never compares them to the intended distribution Phi(I) at the directive-sampling stage. If hard, long-horizon, or substrate-demanding directives fail materialization at higher rates, the final corpus is biased toward whatever the CPU-only Docker pipeline can instantiate. Please report drop rates by task type, difficulty, runtime, and source strategy, and compare intended vs. realized weights. If the drop is uniform, state so explicitly with evidence.
- [Section 5.3, Figure 4, Table 11] The 43.2K scaling result (58.4%) is reported from a single run with no confidence interval, while the controlled 2K and 3.6K runs have overlapping CIs (43.8±4.3 and 52.0±3.6). The abstract's scaling claim rests on this point. Reporting four runs for the 3.6K condition but a single run for the headline scaling result is asymmetric and weakens the claim. Please provide multiple evaluations or a confidence interval for the 43.2K run, and clarify whether the same task suite, verifier version, and inference settings were used.
- [Section 5.3, Table 4, Table 12] The office scaling claim is not supported by the reported uncertainty. Office-2K is 1338 Elo with 95% CI [1312, 1364] and Office-22K is 1384 Elo with CI [1358, 1410]. These intervals overlap, so the text's statement that 'the same pattern holds on GDPval' and 'shows consistent gains' is too strong. Either provide a paired significance test across the same evaluation pool or temper the claim to a non-significant trend.
minor comments (4)
- [Tables 3 and 13] Cross-model comparisons in Tables 3 and 13 use different scaffolds, tools, and inference settings from published leaderboards. The paper acknowledges this in Section 5.1, but the main text phrases such as 'on par with Claude Opus 4.6' and 'surpassing several frontier proprietary systems' should more prominently carry the cross-scaffold caveat, since the controlled comparisons with the same harness are the only strictly fair ones.
- [Section 5.5] The manual audit of 100 packages reports scores on five dimensions but does not describe how packages were sampled, whether the audit was single- or multi-rater, or how disagreements were resolved. Adding this information would make the quality claim more reproducible.
- [Conclusion] The limitation that synthesized tasks lack machine-verifiable outcomes is correctly stated. This is a real constraint and should be retained in any future versions, as it delimits the use of NexForge data for benchmarks and RL environments.
- [Appendix E, Table 12] The GDPval Elo values are fit in one internal joint comparison pool with an LLM judge. Please report judge agreement or variance, and clarify how ties are counted, since the confidence intervals are used to support scaling claims.
Circularity Check
No circular derivation: external benchmarks are explicitly excluded from data construction; teacher and students are distinct model families; controlled runs are self-contained.
full rationale
NexForge's derivation chain is self-contained. The input is a high-level capability requirement I; NexForge expands it into a task demand profile Φ(I) and scenario reservoir G(I) through web research and LLM generation, composes task directives, instantiates executable workspaces, and collects teacher rollouts from a different model family (DeepSeek-V4-Pro) to produce SFT data. The resulting models are evaluated on external, published benchmarks: Terminal-Bench 2.0 and GDPval. The paper explicitly decouples evaluation from construction: 'both benchmarks are used exclusively for downstream evaluation and are not involved in data construction' (Sec. 5.1). There is no fitted parameter calibrated to the benchmark, no benchmark-derived feedback into the profile weights, and no self-authored task distribution used as the evaluation target. The controlled experiments compare public base models (Qwen3-32B, Qwen3.5-35B-A3B) against published baselines on an external test suite, making the improvement falsifiable rather than definitional. The GDPval Elo is fit in an in-house joint pool (Appendix E), which is a reproducibility caveat, but the pool includes external baselines and an external anchor (GPT-5.1 at 1000), so it does not make the result circular. The only self-referential elements are the NexAU harness name and the Nex-N2 model-page URL (SII, 2026); neither is load-bearing for the central claim, and the controlled 3.6K/2K runs stand independently of the scaled Nex-N2 claims. The paper acknowledges limitations—synthesized tasks 'lack machine-verifiable outcomes' and a small fraction of packages fail materialization (Table 7)—but these are data-quality caveats, not circular reductions. The materialization-bias concern, while worth testing, questions whether the corpus matches real-world demand, not whether the reported benchmark gains reduce to the construction choices. No step in the chain defines its output in terms of the evaluation target.
Axiom & Free-Parameter Ledger
free parameters (4)
- Task-form profile weights Φk =
Not disclosed (LLM-generated evidence-derived weights)
- Batch-level constraints B (language split, difficulty budget, rollout count) =
50/50 Chinese-English; balanced difficulty; 3 rollouts for controlled runs, 1 rollout for scaled runs
- Embedding deduplication cosine threshold =
0.85
- Trajectory cleaning criteria ('malformed, degenerate, trivially short') =
Not specified numerically
axioms (4)
- domain assumption Web research conducted by LLMs faithfully captures the real-world task demand distribution for a capability.
- domain assumption Teacher rollouts from DeepSeek-V4-Pro on synthetic workspaces provide useful supervised signals for the target agents.
- domain assumption The automatic workspace generation/validation stage can instantiate any compatible directive into a runnable CPU-only Docker environment.
- domain assumption Terminal-Bench 2.0 and GDPval measure the intended general capabilities rather than overfitting to synthetic data.
read the original abstract
Scaling executable agent training data for LLM post-training is bottlenecked by substrate-bound methods that tie task generation to predefined tools, repositories, or skill graphs: expanding coverage requires manual substrate engineering, each new domain demands a bespoke pipeline, and the resulting task distributions often reflect substrate biases rather than real-world demand. We introduce NexForge, a requirement-driven framework that takes high-level capability requirements as input and synthesizes diverse, executable agent tasks and expert trajectories for SFT. NexForge first investigates real-world demand to construct scenarios and task profiles, then performs distribution-aware compilation to generate task directives. For each directive, NexForge automatically retrieves or constructs the required files, dependencies, and runtime configurations, and finally collects expert rollouts to produce training trajectories. Without domain-specific infrastructure, NexForge produces 3.6K terminal and 2K office tasks, improving Qwen3.5-35B-A3B Base from 22.5\% to 52.0\% on Terminal-Bench 2.0 and from 813 to 1338 Elo on GDPval; scaling further to 43.2K terminal tasks yields 58.4\%, on par with Claude Opus 4.6 equipped with Claude Code. Scaled further, NexForge-synthesized data contributes to the training of Nex-N2, a family of publicly available agent models that lift Qwen3.5-397B-A17B to 75.3\% on Terminal-Bench 2.1 and to 1585 Elo on GDPval---achieving state-of-the-art open-source performance and surpassing several frontier proprietary systems. Nex-N2 models are available at https://nex.sii.edu.cn/.
Figures
Reference graph
Works this paper leans on
-
[1]
Smith and Daniel Khashabi and Hannaneh Hajishirzi , title =
Yizhong Wang and Yeganeh Kordi and Swaroop Mishra and Alisa Liu and Noah A. Smith and Daniel Khashabi and Hannaneh Hajishirzi , title =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics , year =
-
[2]
arXiv preprint arXiv:2402.13064 , year =
Haoran Li and Qingxiu Dong and Zhengyang Tang and Chaojun Wang and Xingxing Zhang and Haoyang Huang and Shaohan Huang and Xiaolong Huang and Zeqiang Huang and Dongdong Zhang and Yuxian Gu and Xin Cheng and Xun Wang and Si-Qing Chen and Li Dong and Wei Lu and Zhifang Sui and Benyou Wang and Wai Lam and Furu Wei , title =. arXiv preprint arXiv:2402.13064 , year =
-
[3]
arXiv preprint arXiv:2604.09377 , year =
Hui Liu and Bin Zou and Kecheng Chen and Jie Liu and Wenya Wang and Haoliang Li , title =. arXiv preprint arXiv:2604.09377 , year =
-
[4]
arXiv preprint arXiv:2604.25727 , year =
Zhiyuan Fan and Tinghao Yu and Yuanjun Cai and Jiangtao Guan and Yun Yang and Dingxin Hu and Jiang Zhou and Xing Wu and Zhuo Han and Feng Zhang and Lilin Wang , title =. arXiv preprint arXiv:2604.25727 , year =
-
[5]
arXiv preprint arXiv:2506.10055 , year =
Dingfeng Shi and Jingyi Cao and Qianben Chen and Weichen Sun and Weizhen Li and Hongxuan Lu and Fangchen Dong and Tianrui Qin and King Zhu and Minghao Liu and Jian Yang and Ge Zhang and Jiaheng Liu and Changwang Zhang and Jun Wang and Yuchen Eleanor Jiang and Wangchunshu Zhou , title =. arXiv preprint arXiv:2506.10055 , year =
-
[6]
arXiv preprint arXiv:2506.14205 , year =
Jingxu Xie and Dylan Xu and Xuandong Zhao and Dawn Song , title =. arXiv preprint arXiv:2506.14205 , year =
-
[7]
Pan and Yunhong Wang , title =
Zihao Cheng and Hongru Wang and Zeming Liu and Xinyi Wang and Xiangrong Zhu and Yuhang Guo and Wei Lin and Jeff Z. Pan and Yunhong Wang , title =. arXiv preprint arXiv:2605.20876 , year =
-
[8]
Ruan and Tyler Griggs and Alexander Glenn Shaw and Hritik Bansal and E
Negin Raoof and Richard Zhuang and Marianna Nezhurina and Etash Guha and Atula Tejaswi and Ryan Marten and Charlie F. Ruan and Tyler Griggs and Alexander Glenn Shaw and Hritik Bansal and E. Kelly Buchanan and Artem Gazizov and Reinhard Heckel and Chinmay Hegde and Sankalp Jajee and Daanish Khazi and Emmanouil Koukoumidis and Xiangyi Li and Hange Liu and S...
-
[9]
arXiv preprint arXiv:2604.18292 , year =
Guanting Dong and Junting Lu and Junjie Huang and Wanjun Zhong and Longxiang Liu and Shijue Huang and Zhenyu Li and Yang Zhao and Xiaoshuai Song and Xiaoxi Li and Jiajie Jin and Yutao Zhu and Hanbin Wang and Fangyu Lei and Qinyu Luo and Mingyang Chen and Zehui Chen and Jiazhan Feng and Ji-Rong Wen and Zhicheng Dou , title =. arXiv preprint arXiv:2604.1829...
-
[10]
arXiv preprint arXiv:2603.11076 , year =
Aili Chen and Chi Zhang and Junteng Liu and Jiangjie Chen and Chengyu Du and Yunji Li and Ming Zhong and Qin Wang and Zhengmao Zhu and Jiayuan Song and Ke Ji and Junxian He and Pengyu Zhao and Yanghua Xiao , title =. arXiv preprint arXiv:2603.11076 , year =
-
[11]
arXiv preprint arXiv:2602.10090 , year =
Zhaoyang Wang and Canwen Xu and Boyi Liu and Yite Wang and Siwei Han and Zhewei Yao and Huaxiu Yao and Yuxiong He , title =. arXiv preprint arXiv:2602.10090 , year =
-
[12]
arXiv preprint arXiv:2602.21193 , year =
Renjie Pi and Grace Lam and Mohammad Shoeybi and Pooya Jannaty and Bryan Catanzaro and Wei Ping , title =. arXiv preprint arXiv:2602.21193 , year =
-
[13]
arXiv preprint arXiv:2605.29559 , year =
Xiaoxuan Peng and Kaiqi Zhang and Xinyu Lu and Boxi Cao and Yaojie Lu and Hongyu Lin and Xianpei Han and Le Sun , title =. arXiv preprint arXiv:2605.29559 , year =
-
[14]
arXiv preprint arXiv:2602.01244 , year =
Siwei Wu and Yizhi Li and Yuyang Song and Wei Zhang and Yang Wang and Riza Batista-Navarro and Xian Yang and Mingjie Tang and Bryan Dai and Jian Yang and Chenghua Lin , title =. arXiv preprint arXiv:2602.01244 , year =
-
[15]
John Yang and Kilian Lieret and Carlos E. Jimenez and Alexander Wettig and Kabir Khandpur and Yanzhe Zhang and Binyuan Hui and Ofir Press and Ludwig Schmidt and Diyi Yang , title =. arXiv preprint arXiv:2504.21798 , year =
-
[16]
arXiv preprint arXiv:2504.07164 , year =
Naman Jain and Jaskirat Singh and Manish Shetty and Liang Zheng and Koushik Sen and Ion Stoica , title =. arXiv preprint arXiv:2504.07164 , year =
-
[17]
arXiv preprint arXiv:2606.06462 , year =
Shiyun Xiong and Dongming Wu and Peiwen Sun and Yuang Ai and Bokang Yang and Wencheng Han and Xiao-Hui Li and Xiangyu Yue , title =. arXiv preprint arXiv:2606.06462 , year =
-
[18]
Mike A. Merrill and Alexander G. Shaw and Nicholas Carlini and Boxuan Li and Harsh Raj and Ivan Bercovich and Lin Shi and Jeong Yeon Shin and Thomas Walshe and E. Kelly Buchanan and Junhong Shen and Guanghao Ye and Haowei Lin and Jason Poulos and Maoyu Wang and Marianna Nezhurina and Jenia Jitsev , title =. arXiv preprint arXiv:2601.11868 , year =
-
[19]
arXiv preprint arXiv:2510.04374 , year =
Tejal Patwardhan and Rachel Dias and Elizabeth Proehl and Grace Kim and Michele Wang and Olivia Watkins and Sim. arXiv preprint arXiv:2510.04374 , year =
-
[20]
2026 , howpublished =
2026
-
[21]
arXiv preprint arXiv:2606.22883 , year =
Zhanbo Hua and Yifan Yao and Weihao Xie and Yongchi Zhao and Minghao Liu and Ruizhi Qiu and Zhewei Huang and Zun Wang and Yiyan Ji and Yunhai Ye and Letian Zhu and Xinping Lei and Han Li and Zhiyuan Ma and Zili Wang and Zhaoxiang Zhang and Jiaheng Liu , title =. arXiv preprint arXiv:2606.22883 , year =
-
[22]
arXiv preprint arXiv:2505.09388 , year =
An Yang and Beichen Zhang and Binyuan Hui and Bofei Gao and Bowen Yu and Chengpeng Li and Dayiheng Liu and Jianhong Tu and Jingren Zhou and Junyang Lin and Keming Lu and Kexin Yang and Mengfei Du and Rui Men and Tao Bai and Xingzhang Ren and Xuancheng Ren and Yang Fan and Yichang Zhang and Yu Wan and Yuqiong Liu and Zeyu Cui and Zhenru Zhang and Zhifang S...
-
[23]
arXiv preprint arXiv:2606.19348 , year =
Aixin Xu and Bei Feng and Bingxuan Wang and Chengqi Deng and Deli Chen , title =. arXiv preprint arXiv:2606.19348 , year =
-
[24]
Proceedings of the International Conference on Learning Representations , year =
Can Xu and Qingfeng Sun and Zhengfeng Yang and Feng Ji and Weikang Wang and Shuai Zhang and Xiaoxu Wang and Yeqing He and Chongyang Tao and Qingyu Zhou and Yifan Li and Kai Zhang and Jingjing Liu and Lingling Zhang , title =. Proceedings of the International Conference on Learning Representations , year =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.