RECORD
OPDSearch+: On-Policy Distillation with RL Refinement for Search-Augmented Reasoning
Artificial Intelligence · Submitted 25 Aug 2026
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Across seven QA benchmarks, OPDSearch+ with a 3B model consistently outperforms all prior 3B RL baselines, achieving gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Search-augmented reasoning remains difficult for small language models. On-policy distillation (OPD) from trained teachers offers a promising direction, but suffers from two issues: (1) high-quality multi-turn search trajectories depend on dynamic retriever responses, making SFT data prohibitively expensive to collect at scale; (2) task-specifically trained teachers incur substantial training cost, while directly applying OPD with an off-the-shelf teacher without task-specific fine-tuning constrains the student to the teacher's performance ceiling and suffers from severe training instability. We propose OPDSearch+, the first distillation paradigm that requires no teacher fine-tuning for search-augmented reasoning. We investigate the role of a frozen off-the-shelf instruct model as the teacher in on-policy distillation, and reveal a key insight: the teacher reshapes the student's policy distribution so that subsequent RL converges to a superior solution that RL alone cannot reach. In stage one, the student interacts with a live search engine and is distilled via a per-position forward KL objective, transferring reasoning decomposition and evidence integration skills without any task-specific teacher training. In stage two, RL refines the distilled student from a richer behavioral foundation, achieving performance that RL alone cannot reach from scratch. Across seven QA benchmarks, OPDSearch+ with a 3B model consistently outperforms all prior 3B RL baselines, achieving gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Search-augmented reasoning remains difficult for small language models. | background 0.99 | 0.07 | 0.02 | asserted 1.00 |
| s1 | On-policy distillation (OPD) from trained teachers offers a promising direction, but suffers from two issues: (1) high-quality multi-turn search trajectories depend on dynamic retriever responses, making SFT data prohibitively expensive to collect at scale; (2) task-specifically trained teachers incur substantial training cost, while directly applying OPD with an off-the-shelf teacher without task-specific fine-tuning constrains the student to the teacher's performance ceiling and suffers from severe training instability. | gap 0.51 | 0.10 | 0.03 | asserted 0.82 |
| s2 | We propose OPDSearch+, the first distillation paradigm that requires no teacher fine-tuning for search-augmented reasoning. | contribution 1.00 | 0.84 | 0.30 | asserted 1.00 |
| s3 | We investigate the role of a frozen off-the-shelf instruct model as the teacher in on-policy distillation, and reveal a key insight: the teacher reshapes the student's policy distribution so that subsequent RL converges to a superior solution that RL alone cannot reach. | interpretation 0.69 | 0.93 | 0.47 | asserted 1.00 |
| s4 | In stage one, the student interacts with a live search engine and is distilled via a per-position forward KL objective, transferring reasoning decomposition and evidence integration skills without any task-specific teacher training. | method 1.00 | 0.90 | 0.06 | asserted 1.00 |
| s5 | In stage two, RL refines the distilled student from a richer behavioral foundation, achieving performance that RL alone cannot reach from scratch. | method 0.80 | 0.90 | 0.21 | asserted 1.00 |
| s6 | Across seven QA benchmarks, OPDSearch+ with a 3B model consistently outperforms all prior 3B RL baselines, achieving gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA. | result 1.00 | 0.95 | 0.53 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Search-augmented reasoning remains difficult for small language models. On-policy distillation (OPD) from trained teachers offers a promising direction, but suffers from two issues: (1) high-quality multi-turn search trajectories depend on dynamic retriever responses, making SFT data prohibitively expensive to collect at scale; (2) task-specifically trained teachers incur substantial training cost, while directly applying OPD with an off-the-shelf teacher without task-specific fine-tuning constrains the student to the teacher's performance ceiling and suffers from severe training instability. We propose OPDSearch+, the first distillation paradigm that requires no teacher fine-tuning for search-augmented reasoning. We investigate the role of a frozen off-the-shelf instruct model as the teacher in on-policy distillation, and reveal a key insight: the teacher reshapes the student's policy distribution so that subsequent RL converges to a superior solution that RL alone cannot reach. In stage one, the student interacts with a live search engine and is distilled via a per-position forward KL objective, transferring reasoning decomposition and evidence integration skills without any task-specific teacher training. In stage two, RL refines the distilled student from a richer behavioral foundation, achieving performance that RL alone cannot reach from scratch. Across seven QA benchmarks, OPDSearch+ with a 3B model consistently outperforms all prior 3B RL baselines, achieving gains of 13.1% on HotpotQA and 8.5% on 2WikiMultihopQA.
Human review
References
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 5 sentences without a confident label
Uses the method or tool of 1 paper
-
Both stages use the veRL framework [ref] on 4$×$ H200 GPUs with lr $1×10^-6$, $G=8$ rollouts, $T_max=4$ search turns.
HybridFlow: A Flexible and Efficient RLHF Framework · Experimental Setup · 1.00
Compares with 2 papers
-
We compare against Search-R1 [ref] , ReSearch [ref] , AutoRefine [ref] , StepSearch [ref] , GiGPO [ref] , and non-RL baselines (Direct Generation, SFT, Naive RAG, Search-o1).
Group-in-Group Policy Optimization for LLM Agent Training · Experimental Setup · 1.00
-
We compare against Search-R1 [ref] , ReSearch [ref] , AutoRefine [ref] , StepSearch [ref] , GiGPO [ref] , and non-RL baselines (Direct Generation, SFT, Naive RAG, Search-o1).
StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization · Experimental Setup · 1.00
Cites as background 12 papers
-
Recent work explores the interplay between KL direction and training stability: EOPD [ref] uses entropy-adaptive KL mixing, Decoupled-KL [ref] analyzes the design space of prefix source and KL direction and finds that forward KL is essential for preventing entropy collapse in long-sequence distillation, and KDRL [ref] jointly optimizes KL distillation with RL objectives.
Entropy-Aware On-Policy Distillation of Language Models · Related Work · 0.93
-
Recent work explores the interplay between KL direction and training stability: EOPD [ref] uses entropy-adaptive KL mixing, Decoupled-KL [ref] analyzes the design space of prefix source and KL direction and finds that forward KL is essential for preventing entropy collapse in long-sequence distillation, and KDRL [ref] jointly optimizes KL distillation with RL objectives.
KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning · Related Work · 0.93
-
Recent work explores the interplay between KL direction and training stability: EOPD [ref] uses entropy-adaptive KL mixing, Decoupled-KL [ref] analyzes the design space of prefix source and KL direction and finds that forward KL is essential for preventing entropy collapse in long-sequence distillation, and KDRL [ref] jointly optimizes KL distillation with RL objectives.
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation · Related Work · 0.93
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.