REVIEW 3 major objections 5 minor 24 references
Cross-Domain Hybrid OPD for Generalizable Search Agents
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read A two-stage hybrid training method lets a search-specialized AI agent recover and even surpass its general reasoning skills without losing search ability.
desk verdict Competent industrial recipe, but the headline alignment-tax claim is undercut by training/eval contamination—worth revising, not rejecting. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Reverse-KL on-policy distillation inside a single policy-gradient objective. For each general-domain sample, the student generates on-policy tokens, the corresponding expert teacher supplies reference log-probabilities, and the log-ratio is treated as a token-level advantage, replacing the GRPO advantage used for search samples. The teachers themselves are trained with a two-stage curriculum that starts on medium-difficulty examples and later expands to harder ones, which the ablation shows is important for teacher quality. A mixed mini-batch cycles between search trajectories and general-domain prompts in every iteration.
What would settle it
Compute an n-gram or embedding overlap between the teacher training corpora (which include GPQA, BBH, AIME, MATH, LiveCodeBench) and the evaluation benchmarks (GPQA Diamond, BBEH Mini, AIME25, etc.). If overlap is high for the benchmarks that show the largest Stage II gains, run the same hybrid training and evaluate on a fresh, non-overlapping test set; the recovery margin would shrink or vanish.
Extended reading notes
Core claim
The paper's central discovery is an objective that simultaneously maximizes verifier-based search rewards and minimizes reverse-KL divergence between the student and domain-expert teacher policies on general-domain prompts. Measured on their benchmarks, Stage II raises HYEval3.1 Logical Reasoning from 33.95 to 46.90, BBEH Mini from 34.60 to 47.86, AutoCodeBench v2 from 67.74 to 74.15, and AIME25 from 59.00 to 66.33, while AS-MultiHopQA and AS-WideSeekQA continue to improve over Stage I. The authors interpret this as evidence that the alignment tax from search RL is not a fixed cost but can be mitigated by injecting dense token-level supervision from specialists.
Load-bearing premise
The general-capability benchmarks used to demonstrate recovery are not meaningfully contaminated by the same public datasets used to train the expert teachers; if overlap is significant, part of the measured improvement reflects memorization rather than restored competence.
Editorial extensions
If this is right
- Search-focused RL without mitigation systematically degrades general reasoning benchmarks, confirming the alignment tax.
- Adding multi-domain OPD in Stage II restores general capability on most benchmarks, often exceeding the base model.
- Search capability is not sacrificed: three of five search benchmarks improve from Stage I to Stage II, and all remain well above base.
- Ablations show outcome-based RL alone (Mix RL) cannot replicate OPD's recovery, and a single general teacher underperforms domain-routed experts.
- Curriculum learning during teacher training improves the hardest-level teacher accuracy and downstream hybrid performance.
Reading between the lines
- The reported pattern suggests a general recipe: any capability-specialized RL (not just search) might be paired with OPD from complementary experts to prevent regression, since the mechanism is token-level density supervision rather than task-specific rewards.
- The reverse-KL formulation penalizes student tokens the expert considers unlikely, which may act as a regularizer preserving output diversity; a testable hypothesis is that the benefit grows with the diversity of expert domains included.
- The benchmark-overlap caveat means the magnitude of the claimed recovery should be checked on freshly constructed evaluation sets disjoint from teacher training corpora.
- A practical extension would be to scale the number of expert domains (e.g., multilingual, instruction following) and measure whether the hybrid objective continues to improve general capabilities without hurting search.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Hybrid-OPD, a two-stage training framework for a search agent built on the Hunyuan3 backbone. Stage I applies GRPO on agent search trajectories to acquire search behaviors; Stage II jointly optimizes search GRPO and reverse-KL on-policy distillation from four domain-specialized teacher models (math, code, logic, science). The central empirical claim is that Stage II mitigates the alignment tax: relative to Stage I, it improves most general reasoning benchmarks (e.g., HYEval3.1 Logical Reasoning +12.95, BBEH Mini +13.26, AIME25 +7.33) while preserving or improving search rewards (Tables 2 and 3). Ablations attribute the gains to OPD over outcome-based RL, multi-teacher routing, and curriculum learning.
Significance. If the alignment-tax result is clean, the framework is practically significant: it offers a concrete recipe for combining agentic RL with distillation so that search specialization does not degrade general ability. The paper provides a well-specified algorithm (Eq. 2-3, Algorithm 1) and a useful ablation design. However, the headline evidence is materially weakened by the overlap between the teacher training corpora and the general-capability evaluation benchmarks, so the central claim is not yet established as reported.
major comments (3)
- [§6.2, Tables 1 and 2] The central claim that Stage II 'effectively mitigates the alignment tax' rests on improvements on general benchmarks that overlap with teacher training data. Table 1 lists GPQA, BBH, AIME, and LiveCodeBench as public training sources for expert teachers; Table 2 evaluates GPQA Diamond, BBEH Mini, AIME25, and AutoCodeBench v2. GPQA Diamond is a subset of GPQA; AIME25 belongs to the same public contest family as AIME; BBEH Mini and AutoCodeBench v2 are not shown to be disjoint from BBH/LiveCodeBench. No decontamination protocol (n-gram/embedding overlap, temporal cutoff, manual audit) is reported. Since Stage II distills from teachers trained on these corpora, the reported Stage II gains (+6.69 GPQA Diamond, +4.33 AIME25, +7.64 BBEH Mini) could reflect re-injection of training data rather than restored general capability. The paper must provide contamination evidence or re-evaluate on hel
- [§7.2, Table 4] The AIME25 row in Table 4 appears to show Single General Teacher at 70.00 versus Hybrid Optimization at 66.33. If accurate, this contradicts the statement in §7.2 that the single-teacher variant underperforms the full framework on most benchmarks. The numeric entries are also missing separators ('70.0066.33', '53.6850.74'), so this may be a formatting artifact; please clarify. If the value is real, it is a counterexample to the multi-teacher conclusion and needs discussion.
- [§5, Tables 2 and 4] No uncertainty estimates, confidence intervals, or multiple-seed results are reported. Several claimed improvements are very small (e.g., PRBench Legal Hard +0.46, SUPERChem-v3 +0.20, HYEval3.1 Math +0.02) and may lie within run-to-run noise. For the key Stage-I-to-Stage-II comparisons, the paper should report variance or at least multiple seeds, especially for the benchmarks that drive the alignment-tax conclusion.
minor comments (5)
- [After Abstract] There is an unexplained, unlabeled 'Paris itinerary' passage immediately after the abstract, followed by document snippets and a failure/success illustration. It appears to be a leftover example or artifact and should be removed or properly integrated as a figure/example.
- [§4.3] Typo: 'enabling the model to preserving search capabilities' should be 'preserve'; also 'as shown in Figure 3 3' has an extra '3'.
- [Table 4] Numeric formatting is corrupted (missing spaces between columns), e.g., '70.0066.33' and '53.6850.74'. Please regenerate the table.
- [§7.1] The claim that removing OPD causes 'substantial degradation' should acknowledge that Mix RL already matches Hybrid Optimization on AIME25 (66.33) and is close on AutoCodeBench v2; the exception is notable.
- [References] The in-text citation '(Team, 2026)' for Hunyuan3 should match the reference list entry 'Tencent Hunyuan Team'.
Circularity Check
General-capability recovery is partly circular: GPQA and AIME25 eval benchmarks are inside the teacher training corpora.
-
fitted input called prediction
[Section 3.1 Table 1; Section 5 Evaluation Data; claimed in Section 6.2]
"Mathematics MATH, GSM8K, AIME, AMC, AQuA-RAT ... Logical Reasoning ... BBH ... Science GPQA, SciQ, SciBench ... we evaluate general capabilities on ... AIME25 (AIME, 2025) ... GPQA Diamond (Rein et al., 2023) ... AIME25 increases from 59.00 to 66.33, while ... GPQA Diamond improves by 6.69 points over Stage I"
The Stage-II claim of mitigating the alignment tax is supported by improvements on GPQA Diamond (+6.69 vs Stage I) and AIME25 (+4.33). But the expert teachers used in Stage-II OPD were trained on GPQA and AIME (Table 1). GPQA Diamond is the canonical hard subset of GPQA, and AIME25 is an instance of AIME, so these eval benchmarks are contained in the teacher training corpora. Distilling these teachers back into the student on the same general-domain data re-injects the teacher's training signal; the measured 'restoration' on these benchmarks reflects memorization of the training set rather than independent recovery of general reasoning. No decontamination protocol (n-gram overlap removal, temporal cutoff, or held-out split) is reported, so the claimed predictions are partially forced by co
full rationale
The paper's main empirical assertion (Section 6.2) is that Stage-II hybrid training 'effectively mitigates the alignment tax' because it improves on most general benchmarks while retaining search rewards. The strongest evidence for this is real but partly circular: the expert teachers were trained on public corpora that include GPQA and AIME, and the evaluation suite includes GPQA Diamond and AIME25. Since these eval sets are subsets of (or directly drawn from) the teacher training corpora, the Stage-II gains on those specific benchmarks are expected from data re-injection rather than from restored general capability. The framework is not wholly circular: the Mix RL ablation (Table 4) shows that OPD contributes beyond plain GRPO, and several other benchmarks (AutoCodeBench v2, HYEval3.1, PRBench, SUPERChem) are not shown to overlap the training corpora, giving the central claim some independent content. However, the specific 'predictions' of improved GPQA Diamond and AIME25 are compromised by construction, so a moderate circularity score is warranted.
Assumptions & free parameters
free parameters (5)
- Difficulty thresholds for curriculum =
easy: pass@k > 0.9; medium: 0.3-0.9; hard: < 0.3 (Sec 3.1)
- Stage-1 curriculum difficulty selection =
intermediate/high confidence only in Stage 1, full spectrum in Stage 2
- Search/general mixture ratio in hybrid minibatch =
not specified
- GRPO hyperparameters =
learning rate 5e-6, group size 8, max seq 61,440
- Reward model training data and splits =
internal pairwise data for search; binary RMs for domains
assumptions (3)
- domain assumption Reverse-KL distillation via a token-level advantage (the k1 estimator after Schulman 2020) approximates the stated minimization of KL(pi_theta || pi_phi_d).
- domain assumption The AS-* search benchmarks and their LLM judges measure real search capability.
- ad hoc to paper Evaluation benchmarks are free of substantial overlap with the training corpora.
Cite this review
Pith. "Pith review of Cross-Domain Hybrid OPD for Generalizable Search Agents." pith.science (2026). https://pith.science/paper/Z2QPYOFX
@misc{pith2026260802101,
author = {Pith},
title = {Pith review of: Cross-Domain Hybrid OPD for Generalizable Search Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z2QPYOFX}},
note = {Machine review of arXiv:2608.02101}
}
read the original abstract
Recent advances in Reinforcement Learning (RL) have substantially improved the capabilities of autonomous search agents, enabling sophisticated planning, and iterative retrieval over dynamic information sources. However, optimizing language models for specialized search behaviors often incurs an alignment tax, where gains in search performance come at the expense of general-purpose capabilities, limiting their effectiveness as universal assistants. In this technical report, we present the training framework behind the Yuanbao search agent, designed to achieve search specialization without sacrificing general intelligence. Built upon the Hunyuan3 architecture, our framework combines agentic reinforcement learning for autonomous search with a cross-domain expert On-Policy Distillation (OPD) pipeline. Experts specializing in complementary general-purpose domains are distilled into the search-specialized student, restoring and further enhancing its broad capabilities. Rather than treating specialization and general capability as competing objectives, our hybrid training strategy jointly optimizes both, effectively mitigating the alignment tax. Extensive experiments demonstrate that the resulting model achieves competitive search performance while consistently improving its general-purpose capabilities, providing a favorable balance between specialized execution and broad generalization in real-world search scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
On-policy distillation of language models: Learning from self- generated mistakes
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pp. 21246–21263,
2024
-
[5]
Mindsearch: Mimicking human minds elicits deep ai searcher
Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mindsearch: Mimicking human minds elicits deep ai searcher. InInternational Conference On Learning Representations, volume 2025, pp. 90007–90029,
2025
-
[6]
URLhttps://arxiv.org/abs/2508.09101. 12 Cross-Domain Hybrid OPD for Generalizable Search Agents Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,
-
[7]
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang
URLhttps://gemini.google/overview/deep-research/. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. InInternational Conference on Learning Representations, volume 2024, pp. 32694– 32717,
2024
-
[8]
Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938, 2021a
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps.arXiv preprint arXiv:2105.09938, 2021a. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measurin...
-
[9]
Zhouyu Jiang, Mengshu Sun, Lei Liang, and Zhiqiang Zhang
URLhttps://openreview.net/forum?id=chfJJYC3iL. Zhouyu Jiang, Mengshu Sun, Lei Liang, and Zhiqiang Zhang. Retrieve, summarize, plan: Advanc- ing multi-hop question answering with an iterative approach. InCompanion Proceedings of the ACM on Web Conference 2025, pp. 1677–1686,
2025
-
[10]
Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516,
-
[11]
Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1285. URLhttps://aclanthology.org/2025.acl-long.1285/. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quan- titative reasoning problems ...
Show all 24 references
-
[13]
Towards robust mathematical reasoning
13 Cross-Domain Hybrid OPD for Generalizable Search Agents Minh-Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, et al. Towards robust mathematical reasoning. InProceedings of the 202...
2025
-
[15]
R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592,
Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning.arXiv preprint arXiv:2503.05592,
-
[16]
Challenging big-bench tasks and whether chain-of-thought can solve them
Mirac Suzgun, Nathan Scales, Nathanael Sch¨arli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. InFindings of the Association for Computational Lin...
2023
-
[17]
Miles Wang, Robi Lin, Kat Hu, Joy Jiao, Neil Chowdhury, Ethan Chang, and Tejal Patwardhan
GitHub repository. Miles Wang, Robi Lin, Kat Hu, Joy Jiao, Neil Chowdhury, Ethan Chang, and Tejal Patwardhan. Frontierscience: Evaluating ai’s ability to perform expert-level scientific tasks.arXiv preprint arXiv:2601.21165,
-
[18]
Scibench: Evaluating college-level scientific problem-solving abilities of large language models.arXiv preprint arXiv:2307.10635, 2023a
Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models.arXiv preprint arXiv:2307.10635, 2023a. Yil...
2023 arXiv
-
[19]
Deepseek-v4: Towards highly efficient million- token context intelligence.arXiv preprint arXiv:2606.19348,
Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million- token context intelligence.arXiv preprint arXiv:2606.19348,
-
[20]
Recomp: Improving retrieval-augmented lms with compression and selective augmentation.arXiv preprint arXiv:2310.04408,
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation.arXiv preprint arXiv:2310.04408,
-
[22]
Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,
14 Cross-Domain Hybrid OPD for Generalizable Search Agents Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering.arXiv preprint arXiv:2602.15763,
-
[23]
Superchem: A multimodal reasoning benchmark in chemistry.arXiv preprint arXiv:2512.01274,
Zehua Zhao, Zhixian Huang, Junren Li, Siyu Lin, Junting Zhou, Fengqi Cao, Kun Zhou, Rui Ge, Tingting Long, Yuexiang Zhu, Yan Liu, Jie Zheng, Junnian Wei, Rong Zhu, Peng Zou, Wenyu Li, Zekai Cheng, Tian Ding, Yaxuan Wang, Yizhao Yan, Tingru Wei, Haowei Ming, Weijie Mao, Chen Su...
-
[24]
Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments
Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environ- ments. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processi...
2025
-
[2017]
doi: 10.18653/v1/P17-1015
Association for Computational Linguistics. doi: 10.18653/v1/P17-1015. URL https://aclanthology.org/P17 -1015/. Junteng Liu, Yuanxiang Fan, Jiang Zhuo, Han Ding, Yongyi Hu, Chi Zhang, Yiqi Shi, Shitong Weng, Aili Chen, Shiqi Chen, et al. Synlogic: Synthesizing verifiable reason...
-
[2020]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300,
-
[2023]
Qwen3 technical report.arXiv preprint arXiv:2505.09388,
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,
-
[2024]
2025 aime i
AIME. 2025 aime i. Art of Problem Solving Wiki,
2025
-
[2025]
Accessed:
URL https://artofproblemsolving.com/ wiki/index.php/2025 AIME I. Accessed:
2025
-
[2026]
ISBN 979-8-89176-390-6
Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1958. URLhttps://aclanthology.org/2026.acl-long.1958/. Shengnan An, Xunliang Cai, Xuezhi Cao, Xiaoyu Li, Yehao Lin, Junlin Liu, Xinxuan Lv, Dan Ma, Xuanlin Wang, Ziwen Wang, et al...
2026
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.