REVIEW 5 major objections 6 minor 17 references
EAPO shows that a learnable training-time 'consult expert' action, gated off over time, improves independent LLM reasoning by about 5 points over self-exploration.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
EAPO lets a policy model consult a stronger expert during training, anneals that access to zero, and improves independent math reasoning by about 5 points over self-exploratory RL.
T0 review reviewed 2026-08-04 challenge →
load-bearing objection EAPO is a plausible way to let RL policies consult experts during training and anneal that away, but the headline math gain isn't cleanly isolated from a prompt confound and the non-math generalization claim is unsupported. the 5 major comments →
EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The discovery claimed by EAPO is that expert assistance can be used as a training-only scaffold for RL-based reasoning: the model learns when to ask, what to ask, and how to synthesize answers, and the resulting patterns persist after expert access is removed. The objective maximizes verifiable final-answer reward over trajectories that interleave private reasoning with parallel expert queries; the annealing schedule rho_s=s^-1 (and a shrinking turn budget) suppresses consultation over time. Under conditions stated in Section 2.2 (expert responses valid and aligned; policy can extract useful features from them), the framework denseifies sparse rewards and stabilizes optimization. Empirically
What carries the argument
The load-bearing object is the 'consult experts' action, a special token the policy can emit during training. It is optimized jointly with reasoning actions, so the model learns a consultation policy. Its effect is modulated by two annealed gates: acceptance rate rho_s=s^-1 for writing an expert response into history, and a shrinking turn budget. Together these create an implicit curriculum: early episodes get rich external hints, later episodes must solve with the model's own internalized strategies. Parallel querying of up to K expert replicas broadens the evidence gathered per turn, while a verifiable end-to-end reward keeps the objective aligned with answer correctness.
Load-bearing premise
The argument depends on expert responses being valid and aligned with the task objective, and on the policy being able to learn useful reasoning patterns from them; if either fails, the reported gains could come from extra training compute rather than from internalized expert knowledge.
What would settle it
Train EAPO twice with identical compute and consultation schedules, once with real expert responses and once with responses replaced by random text or by the policy's own rollouts. If the second run matches the first on math benchmarks, the claim that expert knowledge is internalized is false. A second check: evaluate the trained model with forced expert calls at test time; a large accuracy jump would show the model never became independent.
If this is right
- A policy trained this way can be deployed without any expert calls, so test-time cost stays that of the small model alone.
- Consultation frequency should drop during training while accuracy holds; the paper reports exactly this dynamic in its call-rate curves.
- Scaling the number of parallel expert queries or the expert model's capacity should further improve the final independent policy.
- If the abstract's generalization claim holds, EAPO's benefit is not limited to contest-style mathematics and should appear on code, science-knowledge, and general-knowledge tasks.
- A fall in response variance across test problems is a predicted corollary of the smoother learning signal, and the paper observes lower variance than self-exploratory RL.
Where Pith is reading between the lines
- A cheap way to test the internalization story: replace expert responses with the policy's own sampled completions, keeping call count and compute fixed. If the five-point gain survives, EAPO's value is denser exploration, not expert knowledge.
- Because rho_s=s^-1 kills consultation very fast, early training quality matters disproportionately; a softer annealing floor might prolong gains on hard problems, a variant the paper does not explore.
- The limitations section says cross-task generalization and richer external signals are future work; the non-math claims in the abstract are not backed by tables in the current text, so they should be read as preliminary rather than established.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes EAPO, a reinforcement-learning framework that augments a reasoning policy with a learnable “consult experts” action during training. The policy model (DeepSeek-R1-Distill-Qwen-7B/14B) can query up to three QwQ-32B experts in parallel, receives multi-turn assistance, and is trained with a verifiable reward; a scheduled acceptance rate ρ_s = s^{-1} and a shrinking turn budget are used to reduce expert dependence. At evaluation the consultation action is disabled. Experiments on AIME 2024, AIME 2025, and AIMO 2025 report Pass@32 and per-problem variance, claiming a 4.91-point average gain over a DAPO-based self-exploratory RL baseline. The abstract additionally claims generalization to non-math benchmarks (HumanEval, HLE, GPQA, MMLU, EvalPlus, HotpotQA, SimpleQA).
Significance. If the central comparison were clean, EAPO would be a useful contribution: it formalizes expert consultation as a trainable action, provides a clear trajectory-level objective, reports rollout paradigms with concrete case studies, and gives implementation details (Appendix B, Table 4) that facilitate reproduction. The paper also explicitly states the two assumptions (valid expert knowledge, policy extractability) on which the mechanism rests, which is a strength. However, the main empirical claim is currently vulnerable to a prompt confound, the internalization evidence is partly forced by construction, and the abstract overstates the scope of the results. These issues are fixable within the manuscript's scope, so the correct disposition is major revision.
major comments (5)
- [Section 3.1 and Appendix D] The comparison between EAPO and Self-Exploratory RL is confounded by the training prompt. The EAPO training prompt (Appendix D) contains a lengthy description of the expert panel, the <agent_calls>/<agent_returns> format, and instructions to decompose problems and consult multiple experts. The self-exploratory baseline is described only as “outcome-driven RL” (DAPO); no matching prompt is reported, and no ablation trains self-exploratory RL with the EAPO prompt but with the consultation action disabled. Without this control, the ~5-point average gain could be caused by the richer system prompt rather than by on-demand expert assistance. This is the load-bearing comparison for the paper's central claim and must be resolved with a matched-prompt ablation.
- [Section 2.3, Eq. (3) and Figure 4] The decrease in consultation calls is used as evidence of capability internalization, but the framework itself forcibly suppresses expert access: the acceptance rate ρ_s = s^{-1} is annealed and the per-episode turn budget is reduced by construction. A declining call count under an externally imposed gate does not show that the policy has learned to stop consulting; it may simply reflect that responses are discarded or that the budget expires. The manuscript should report the policy's attempted consultation rate separately from the accepted response rate, or run a variant with a fixed acceptance gate, before claiming that internalization is demonstrated. The phrase “gates are calibrated” in Section 3.2 does not address this confounding.
- [Abstract and Section 5] The abstract claims EAPO “generalizes to non-math benchmarks, including HumanEval, HLE, GPQA, MMLU, EvalPlus, HotpotQA, and SimpleQA.” No experiments on any of these benchmarks appear in the body; Section 5 explicitly lists cross-task generalization as future work. This is an unsupported claim and must be removed from the abstract unless the corresponding experiments are added.
- [Table 1 and Section 3.2] The paper repeatedly states that EAPO has lower variance than the baselines, and the Table 1 caption says EAPO “consistently outperforms all comparison baselines in terms of mean and variance across all three benchmarks.” This is false on AIME 2024: EAPO reports Var = 0.0673 while Self-Exploratory RL reports Var = 0.0646. The stability claim should be restricted to the average or to the benchmarks where it holds. In addition, no confidence intervals, multiple seeds, or run-to-run variability are reported, so the stability contribution is not statistically supported.
- [Section 3.2, Table 1] The reported improvements are labeled as percentages (e.g., “improving average F1 by 21.54%”), but 64.07 − 42.53 = 21.54 is a percentage-point difference, not a relative percentage improvement. Similar wording appears for other baselines. This should be corrected to avoid misrepresenting the magnitude of the gains.
minor comments (6)
- [Throughout] Typos and grammatical errors: “generate the reasoning tokens independently or or to seek a hint” (Section 1), “consultant occurs on demand” (Section 1), “resoning” (Appendix D), “The agent call query and results are enclosed within<agent_calls> </agent_calls>and<agent_returns> </agent_returns>tags respectively” (missing spaces before tags).
- [Section 2.3] The annealing schedule is described qualitatively. Please report the initial turn budget, the final evaluation budget, and the step at which ρ_s reaches small values; these are necessary to interpret Figure 4.
- [Figure 4] The axes of Figure 4(a) are not fully labeled; it is unclear whether the y-axis counts accepted expert responses, attempted calls, or episodes with at least one consultation. Please clarify.
- [Section 3.1] The definition of Pass@32 is the mean F1 over 32 samples, not the conventional pass@k probability. This is a legitimate measure but should be explicitly distinguished from standard pass@k notation to avoid confusion with the literature.
- [Table 3] Some numeric entries appear misaligned (e.g., the AIMO 2025 column for R1-14B Self Collaboration reads “49.4347.08” and the Expert-Assisted Workflow row reads “64.65 48.6553.98”). The table formatting should be fixed.
- [Table 2] The row labeled “EAPO (Ours) × 32B” lists both “×” and “32B”; the checkmark and cross symbols are not explained in the caption. Please make the notation consistent and self-contained.
Circularity Check
No significant circularity: EAPO's central comparison is self-contained against external benchmarks; the annealing/call-rate analysis is a confounded interpretation, not a definitional reduction.
full rationale
EAPO's central claim is an empirical comparison: RL training with on-demand expert consultation improves AIME/AIMO accuracy over baselines. The paper does not derive this from a fitted parameter or from a cited uniqueness theorem; there are no self-citations, no fitted values presented as predictions, and no equation in which an outcome is defined as its own input. The main evaluation is against external benchmarks (AIME 2024/2025, AIMO 2025) using standard Pass@32, so the result is externally falsifiable. The only potentially circular-looking passage is the consultation-rate analysis in Section 2.3 and Figure 4, where the annealing schedule ρs=s^-1 suppresses expert access by construction and the observed decline in consultations is cited as evidence of internalization. However, the paper explicitly attributes the decline to 'joint effects of capability internalization and gating calibration' and uses the stability of performance under reduced access as the evidence, rather than treating the decline itself as a derived prediction; the schedule is a training design choice, not a fitted parameter, and the performance comparison does not reduce to it. The prompt mismatch between EAPO and the self-exploratory baseline is a possible experimental confound, but that is a validity/correctness concern, not circularity. The abstract's cross-task generalization claim is unsupported in the body (the conclusion lists it as future work), but unsupported overclaiming is not circularity. Therefore no load-bearing circular step is identifiable.
Axiom & Free-Parameter Ledger
free parameters (4)
- expert acceptance annealing rate =
rho_s = s^-1
- format-only reward =
0.1
- max consultation turns =
10
- expert concurrency =
K=3
axioms (5)
- domain assumption Expert knowledge is valid and aligned with the objective.
- domain assumption The policy model can extract useful features from expert assistance and convert them into improved decisions.
- domain assumption Verifiable rewards (F1 on final answer) are an appropriate training signal.
- domain assumption The RL optimization (presumably DAPO) behaves as expected on the augmented action space.
- domain assumption The evaluation sets are not contaminating the training set in a way that favors EAPO over baselines.
Cite this review
Pith. "Pith review of EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance." pith.science (2026). https://pith.science/paper/5UX7KIXH
@misc{pith2026250923730,
author = {Pith},
title = {Pith review of: EAPO: Enhancing Policy Optimization with On-Demand Expert Assistance},
year = {2026},
howpublished = {\url{https://pith.science/paper/5UX7KIXH}},
note = {Machine review of arXiv:2509.23730}
}
read the original abstract
Large language models (LLMs) have recently advanced in reasoning when optimized with reinforcement learning (RL) under verifiable rewards. Existing methods primarily rely on outcome-based supervision to strengthen internal LLM reasoning, often leading to inefficient exploration and sparse rewards. To mitigate this issue, we propose Expert-Assisted Policy Optimization (EAPO), a novel RL framework that enhances exploration by incorporating multi-turn interactions with external experts during training. Unlike prior methods, where policies reason in isolation, EAPO incentivizes the policy to adaptively determine when and how to consult experts, yielding richer reward signals and more reliable reasoning trajectories. External assistance ultimately internalizes expert knowledge into the policy model, amplifying the model's inherent reasoning capabilities. During evaluation, the policy model has been well-optimized to solve questions independently, producing improved reasoning paths and more accurate solutions. On AIME 2024/2025 and AIMO 2025, EAPO consistently outperforms expert-assisted, expert-distilled, and RL baselines, averaging a 5-point gain over self-exploration RL, and also generalizes to non-math benchmarks, including HumanEval, HLE, GPQA, MMLU, EvalPlus, HotpotQA, and SimpleQA.
Figures
Reference graph
Works this paper leans on
-
[3]
Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Wojciech Gajewski, Andrea Ges- mundo, Neil Houlsby, and Wei Wang. Ask the right questions: Active question reformu- lation with reinforcement learning.arXiv preprint arXiv:1705.07830,
-
[4]
Open problems and fundamental limitations of reinforcement learning from human feedback
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217,
-
[6]
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,
-
[8]
Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124,
-
[9]
Learning from peers in reasoning models.arXiv preprint arXiv:2505.07787,
Tongxu Luo, Wenyu Du, Jiaxi Bi, Stephen Chung, Zhengyang Tang, Hao Yang, Min Zhang, and Benyou Wang. Learning from peers in reasoning models.arXiv preprint arXiv:2505.07787,
-
[14]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171,
-
[15]
Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale.arXiv preprint arXiv:2503.14476,
-
[16]
Michelle Zhao, Reid Simmons, Henny Admoni, Aaditya Ramdas, and Andrea Bajcsy. Conformalized interactive imitation learning: Handling expert shift and intermittent feedback.arXiv preprint arXiv:2410.08852,
-
[17]
expert_id
13 A Appendix B Implementation Details The pre-trained policy model of DeepSeek-R1-Distill-Qwen-7B can be download via https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B and DeepSeek- R1-Distill-Qwen-14B can be download via https://huggingface.co/deepseek-ai/ DeepSeek-R1-Distill-Qwen-14B . The pre-trained expert model of QwQ-32B can be downloa...
2048
-
[2015]
Kickstarting deep reinforcement learning.arXiv preprint arXiv:1803.03835,
Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Woj- ciech M Czarnecki, Joel Z Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, et al. Kickstarting deep reinforcement learning.arXiv preprint arXiv:1803.03835,
-
[2017]
Houda Nait El Barj and Théophile Sautory. Reinforcement learning from llm feedback to counteract goal misgeneralization.arXiv preprint arXiv:2401.07181,
-
[2019]
Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692,
Junlin Wang, Jue Wang, Ben Athiwaratkun, Ce Zhang, and James Zou. Mixture-of-agents enhances large language model capabilities.arXiv preprint arXiv:2406.04692,
-
[2020]
Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization in llm with demonstration-guided reinforcement learning.arXiv preprint arXiv:2404.19409,
-
[2022]
Training language models to self-correct via reinforcement learning, 2024.URL https://arxiv
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning, 2024.URL https://arxiv. org/abs/2409.12917, 2(3):4,
Pith/arXiv arXiv 2024
-
[2023]
Critic: Large language models can self-correct with tool-interactive critiquing
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738,
-
[2024]
Emilien Biré, Anthony Kobanda, Ludovic Denoyer, and Rémy Portelas. Efficient active imitation learning with random network distillation.arXiv preprint arXiv:2411.01894,
-
[2025]
Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources.arXiv preprint arXiv:2306.09782,
This paper was first reviewed by deepseek-v4-flash on August 4, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.