Pith. sign in

REVIEW 3 major objections 5 minor 291 references

Parallel Shapley replaces uniform outcome rewards with per-path Shapley values, sharply improving multi-path math reasoning and cutting training steps.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:48 UTC pith:MH7VRS4N

load-bearing objection A sensible Shapley-based credit-assignment recipe for parallel-reasoning RL, with suggestive but statistically fragile results and a load-bearing GRM that is never validated against the summary. the 3 major comments →

arxiv 2607.18979 v1 pith:MH7VRS4N submitted 2026-07-21 cs.AI

Fishing Out Free Riders: Shapley-Based Reward Attribution for Parallel Reasoning via Reinforcement Learning

Wentao Zhang , Haoyu Zhang , Xinke Jiang , Yuxuan Cheng , Yuhan Pan , Miao Li , Zhipeng Qiao , Tao Feng
show 2 more authors
Zhen Tao Dengji Zhao
This is my paper
classification cs.AI
keywords Shapley valueparallel reasoningreward attributionreinforcement learninggenerative reward modelcredit assignmentGRPOmathematical reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to show that the free-rider problem in parallel reasoning—where redundant or misleading paths collect the same reward as truly helpful ones—is measurable and fixable at training time. Its proposed fix, Parallel Shapley, assigns each reasoning path a process reward equal to its average marginal contribution to the utility of all path subsets, where that utility is scored by a generative reward model. The authors argue this dense, attribution-based signal lets a model learn from 40 reinforcement-learning steps what a uniform-reward baseline cannot learn in 200, and that it raises the upper bound of multi-path correctness (Pass@16) by 42.3% on average across competition-math benchmarks. The central consequence, if true, is that outcome-level reward is not a necessary bottleneck for learning to reason in parallel.

Core claim

The central claim is that replacing the single outcome-level reward shared by all reasoning paths with a Shapley-based process reward eliminates free-riding and yields better, cheaper training. Each path is a player in a cooperative game; the utility of any subset of paths is scored in [0,5] by a generative reward model prompted against the ground-truth answer, and each path's Shapley value—its average marginal utility contribution over all subset sizes and permutations—becomes its token-level reward, injected at the path's closing tag and normalized before GRPO advantage estimation. The paper reports that on four mathematics benchmarks, a 4-billion-parameter model trained this way for 40 st

What carries the argument

The load-bearing object is the Monte Carlo Shapley value of a reasoning path: the path's expected marginal contribution to the utility of every coalition of other paths, U(S)=GRM(S), averaged over sampled path permutations. The generative reward model supplies the utility of each path subset from a five-dimension rubric (method soundness, formula completeness, computational accuracy, result correctness, and information purity); the Shapley average turns those subset scores into a per-path credit signal. Its role is to disambiguate learning: paths that add no information or actively mislead receive low or negative rewards even when the final summary is correct, which is exactly what uniform o

Load-bearing premise

The method's correctness hinges entirely on the generative reward model's subset scores being a faithful measure of how much a set of reasoning paths helps produce the right final answer; if those scores are miscalibrated, the Shapley rewards are precise attributions of the wrong quantity.

What would settle it

Using the authors' stored rollouts, compare GRM subset utilities against an outcome-derived ground truth: for each subset of paths, force the summarizer to write the summary from only those paths and check whether the mean GRM score orders the resulting correctness (Pass@1) monotonically. If a higher-GRM subset is not more likely to yield the correct answer, the process reward is not tracking marginal contribution to correctness, and the reported Pass@16 gains would be expected to survive even with random permutation of path rewards.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Redundant and misleading paths no longer receive positive gradient when the summary happens to be correct; the policy is pushed toward paths with positive marginal contribution.
  • Training efficiency improves: 40 GRPO steps suffice to match or beat a 200-step parallel-thinking baseline, with stable convergence around step 30.
  • The induced policy produces more complementary, specialized paths; masking a fraction of paths degrades Parallel Shapley-trained rollouts more sharply than uniform-reward rollouts.
  • Pass@16, a proxy for the upper bound of multi-path capability, rises by an average of 42.3% on three competition-math benchmarks, while Mean@16 improves modestly.
  • The benefit is not tied to one evaluator: the method beats the uniform-reward baseline even when the generative reward model is replaced by weaker open-source models, and preliminary multi-hop QA results show the pattern generalizes beyond mathematics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same Shapley-over-a-proxy-utility recipe could be applied to any generation problem where a final output is assembled from multiple components (code modules, retrieved evidence, tool-call chains); the paper's mechanism is agnostic to the component content as long as a subset scorer exists.
  • A cheap discriminating test: swap the GRM for a non-generative, outcome-derived utility (e.g., majority-vote correctness), and keep everything else fixed; if gains persist, the credit comes from the Shapley weighting rather than from the richer rubric.
  • The Mean@16 vs Pass@16 gap suggests the policy may be over-specializing individual paths; a distillation or summary-reward term that rewards synthesizing across paths could convert upper-bound gains into single-generation accuracy, a next step the paper itself flags.
  • Because the GRM never reads the summary, the paper's measure of 'contribution' is indirect; feeding the GRM the full response (paths plus summary) would test whether the current utility is actually aligned with the final output quality that the outcome reward measures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Parallel Shapley, a reinforcement learning framework for multi-path ('parallel thinking') LLM reasoning. Each reasoning path is treated as a player in a cooperative game; a generative reward model (GRM) scores subsets of paths, and Monte Carlo Shapley values are used as token-level path rewards in GRPO, combined with a final outcome reward. Experiments on AIME24/25, AMC23, and MATH with a Qwen3-4B base report an average 42.3% relative improvement in Pass@16 over the Parallel-R1-Unseen baseline while using 40 rather than 200 RL steps, together with ablations (leave-one-out, independent evaluation, GRM strength, path count) and an appendix on multi-hop QA.

Significance. The idea of using Shapley-based credit assignment for parallel reasoning paths is timely and relevant to RL training for LLMs. The implementation is concrete (built on VERL, with an external GRM), and the GRM-strength ablation and multi-hop QA extension are useful sanity checks. If the attribution mechanism were properly validated, the work could influence reward design for multi-path generation. However, the central evidence is single-run benchmark numbers, and the path-level reward is computed from an unvalidated GRM proxy rather than from the actual summary-generation process, so the significance is conditional on additional validation.

major comments (3)
  1. [§3.2, Appendix B] The process reward is the Monte Carlo Shapley value of U(S)=GRM(S), where the GRM is prompted (Appendix B) to score a bare subset of paths against the ground-truth answer. The GRM never sees the policy's <Summary>, yet a path's true marginal contribution in parallel reasoning is defined by how the summarizer uses it. The paper provides no evidence that GRM subset utilities track the summary's use of paths, nor that GRM(P) is consistent with the outcome reward r_out. Appendix A's motivating theorem assumes Σ_j φ_j = r_out (Eqs. 18–20), but the implemented reward tensor (§3.3) sums the Shapley rewards to GRM(P) and injects r_out separately at the final token. The claimed 'less biased, free-rider-free' signal is therefore unsupported as stated. Please add a direct validation, e.g., measure the correlation between GRM(P) and r_out over rollouts, or compare Shapley path rewards against human/
  2. [Table 1, Table 2] All reported results are single-run, with no confidence intervals or repeated seeds. On the 30-problem AIME sets, the AIME25 Pass@16 difference (50.0 vs. 37.8, +12.2 points) is about 1.0 binomial standard error, and the AIME24 difference (63.3 vs. 33.2, +30.1 points) is about 2.4 standard errors. The 'consistently outperforms' claim is not statistically established. Moreover, K=4 is selected from the sensitivity sweep in Table 2 using the same test benchmarks, so the reported K=4 numbers are the best case over the path count, further inflating apparent gains. Please report multiple seeds, bootstrap confidence intervals, and either a validation split for hyperparameter selection or an appropriate multiple-comparison correction.
  3. [Appendix A, Eqs. (2), (13), (16), (18)–(20)] The theoretical justification for the harm of uniform rewards contains derivational gaps. Eq. (16) converts a triangle-inequality bound into an approximate equality; Eq. (2) does not follow from Eq. (13) without an additional assumption about the sum of cosine-similarity terms; and Eq. (18)'s decomposition U(P)=Σ_j φ_j(P−j) is not the Shapley decomposition used in the implementation (which averages over all coalitions, not just P−j). Since this theorem is the paper's stated motivation for needing path-level rewards, it should be corrected or explicitly labeled as heuristic motivation rather than a formal result.
minor comments (5)
  1. [§4.3, Table 1] The text says 'Independent path evaluation achieves the lowest performance (Avg. 46.5)' and 'LOO performs slightly worse (Avg. 46.2)', but Table 1 shows LOO at 46.2 and Independent at 46.5. The text and table contradict each other; LOO is lower.
  2. [§3.3] The normalization of the two reward signals is not specified. It is unclear whether the Shapley rewards are normalized per rollout, per group, or globally, and how λ_o=0.5, λ_p=0.5 interact with that normalization. Please define 'normalized Shapley reward' explicitly.
  3. [Appendix B] The evaluation prompt says it assesses 'the solution-path summary for {query}', but it then refers to '{path}' in the singular and does not specify how a subset of multiple paths is serialized into the prompt. This makes the GRM's exact input for a coalition S ambiguous.
  4. [§4.2] The abstract and Section 4.2 claim 'consistently outperforms' strong baselines, but Table 1 shows AIME25 Mean@16 for Parallel Shapley (17.2) is below Parallel-R1-Unseen (17.7). Please qualify the claim to Pass@16 or overall average.
  5. [Appendix D.1] Monte Carlo sampling uses M=5 for K=3,4,5. With such a small number of permutations, the Shapley estimates are likely to be noisy, yet no variance diagnostics or sensitivity to M are reported. Please report the variance of φ across permutations or increase M.

Circularity Check

0 steps flagged

No significant circularity: the central empirical claim is validated against held-out exact-match benchmarks and the path rewards come from an external, ablated generative reward model.

full rationale

The paper's main claim—that Shapley-based path-level rewards improve parallel reasoning—is not circular. The path-level reward is the Monte Carlo Shapley value of U(S)=GRM(S), where GRM is an external generative reward model scoring subsets of paths against ground truth; it is not fitted to the evaluation metric. Evaluation is exact-match accuracy on held-out benchmarks (AIME, AMC, MATH, multi-hop QA), which is independent of the reward model's internal scores. The GRM-sensitivity ablation (Qwen2.5-3B through Qwen2.5-7B) shows the result does not depend on a particular evaluator. The Shapley efficiency identity (sum of Shapley values equals U(P)) is a mathematical theorem, not a circular loading of the conclusion. The Appendix A assumption that Σφ_j=r_out is a theoretical motivation, and the gap between GRM(P) and r_out is a validity limitation, not a circularity: no derived quantity is identical by construction to an input. The cited Parallel-R1 tags and cold-start dataset are not self-citations by the present authors and are used as external building blocks, not as the sole justification of the central claim. Therefore no step satisfies the quoted-reduction standard for circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

The central claim rests on (i) the unvalidated identification U(S)=GRM(S) as the utility whose Shapley values constitute process rewards; (ii) hand-set hyperparameters λo=λp=0.5, K=4 (best of a sweep), M=5; and (iii) an appendix theorem whose assumptions (Σφ=rout; factorized path log-probs) are not satisfied by the implementation. Shapley-value mathematics and MC convergence are standard background. No new entities are invented — GRM is an existing model class used as a component. Net contribution: a new application of textbook Shapley attribution, with the scientific load carried by whether GRM subset scores are a valid proxy for outcome contribution and whether single-run benchmark deltas survive statistical scrutiny.

free parameters (4)
  • reward combination weights λ_o, λ_p = 0.5 / 0.5 (all experiments)
    Hand-set in §3.3 and App. D.1 ('set to λo=0.5 and λp=0.5 throughout'); never ablated, so the balance between outcome and process signals is a free choice that shapes the reward tensor.
  • path count K = 4 (best of sweep {2,3,4,5})
    K=4 chosen from the parameter sensitivity sweep (§4.3, Table 2); all main results and the GRM-sensitivity block use the best-scoring K, and K=5 degrades uniformly — the headline configuration is a selection over K.
  • Monte Carlo permutation count M = 5 for K=3,4,5 (full enumeration at K=2)
    Hand-chosen in App. D.1; no convergence check is reported for the O(1/√M) error bound, so the process-reward noise floor is unknown.
  • GRM utility rubric = 0-5 scale; five 1-point dimensions (Table 3)
    The utility U(S) whose Shapley values become process rewards is defined by a hand-authored rubric (Method Reliability, Formula Completeness, Computational Accuracy, Result Correctness, Information Purity). The scale and dimension weights are ad hoc design choices that define every Shapley value in the paper.
axioms (6)
  • standard math Shapley-value axioms and Monte Carlo convergence O(1/√M)
    §3.2 uses the standard permutation formulation of Shapley values and its MC estimator; these are textbook results, not the paper's contribution.
  • domain assumption U(S)=GRM(S) is a valid proxy for a path subset's contribution to the final outcome
    §3.2: 'the utility function U(·) cannot be reliably estimated using verified reward signals such as RLVR. Therefore, we instantiate U(·) using Generative Reward Model.' The GRM is never validated against outcome correctness, and it never reads the <Summary>, so the proxy link is assumed.
  • ad hoc to paper Path log-probabilities factorize as log π(paths) = Σ_j log π(path_j)
    Appendix A, Eqs. (4)-(5): the free-rider theorem treats paths as independently sampled terms in a REINFORCE gradient. Autoregressive generation of nested <Parallel>/<Path> blocks produces a single coupled sequence, so the factorization is not satisfied by the system being analyzed.
  • ad hoc to paper Σ_j φ_j = rout: Shapley values exhaust the outcome reward
    Appendix A, Eqs. (18)-(20) assert the true utility decomposes with Σφ_j = rout. In the implementation, Shapley values are computed for U = GRM, so Σφ_j = GRM(P), a separate unvalidated quantity — the theorem's premise does not match the system.
  • domain assumption GRPO group-relative normalization is valid for a mixed outcome+process reward tensor
    §3.3: both reward signals are normalized and combined into one token-level tensor, then group-normalized for advantage. No analysis is given for how the two heterogeneous signals interact under mean/std normalization.
  • standard math First-order Taylor expansion of the policy probability at small learning rate
    Appendix A, Eqs. (6)-(9): the conclusion that misleading paths get positive updates relies on a first-order expansion in μ; standard but unproven here, and it ignores the KL and clip terms in the actual GRPO objective.

pith-pipeline@v1.3.0-alltime-deepseek · 20567 in / 21426 out tokens · 202253 ms · 2026-08-01T13:48:29.984514+00:00 · methodology

0 comments
read the original abstract

Large Language Models (LLMs) excel at multi-step reasoning, yet current parallel reasoning approaches often fail to distinguish the contributions of individual reasoning paths. Many paths may be redundant, misleading, or even detrimental, but outcome-level rewards assign uniform reward, leading to ambiguous learning signals and unstable training. We propose Parallel Shapley, a reinforcement learning framework that attributes fine-grained, path-level contributions in multi-path reasoning. Treating each path as a player in a cooperative game, we leverage Shapley values to quantify marginal contributions, using a generative reward model to evaluate path utilities and Monte Carlo sampling for efficient approximation. Experiments on mathematical reasoning benchmarks show that Parallel Shapley outperforms existing baselines while providing more stable and interpretable training. Our framework effectively "fishes out the free riders," assigning reward proportionally and improving multi-path reasoning in LLMs.

Figures

Figures reproduced from arXiv: 2607.18979 by Dengji Zhao, Haoyu Zhang, Miao Li, Tao Feng, Wentao Zhang, Xinke Jiang, Yuhan Pan, Yuxuan Cheng, Zhen Tao, Zhipeng Qiao.

Figure 1
Figure 1. Figure 1: Misleading path and helpful path shared pol [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of Parallel Shapley Pipeline. Left: Comparison of training frameworks. Parallel-R1 assigns uniform outcome rewards to all paths, while Parallel Shapley computes individualized marginal contributions for each reasoning path. Right: The Shapley-based reward attribution mechanism. We sample M path permutations via Monte Carlo, compute marginal contributions ∆ πm j under each ordering, and aggregate t… view at source ↗
Figure 3
Figure 3. Figure 3: Training-curve comparison among Parallel [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Path dependency analysis (K = 4). Solid lines denote full-path baselines; dashed lines denote the corresponding masking conditions. meration, or boundary verification. Parallel-R1’s uniform rewards, by contrast, produce redundant paths that rehearse the same logic, so surviving paths still suffice to reconstruct the answer. These results confirm that Parallel Shapley successfully induces complementary, non… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

291 extracted references · 51 linked inside Pith

  1. [1]

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher R \'e , and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787

    Pith/arXiv arXiv 2024

  2. [2]

    Keyu Chen, Zhifeng Shen, Daohai Yu, Haoqian Wu, Wei Wen, Jianfeng He, Ruizhi Qiao, and Xing Sun. 2025. Aspd: Unlocking adaptive serial-parallel decoding by exploring intrinsic parallelism in llms. arXiv preprint arXiv:2508.08895

    Pith/arXiv arXiv 2025

  3. [3]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261

    Pith/arXiv arXiv 2025

  4. [6]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 175 others. 2025. https://doi.org/10.1038/s41586-025-09422-z Deepseek-r1 incentivizes reasoning in llms through reinforcement lear...

    doi:10.1038/s41586-025-09422-z 2025

  5. [8]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset . Preprint, arXiv:2103.03874

    Pith/arXiv arXiv 2021

  6. [9]

    Xanh Ho, Anh-Khoa Duong, Quoc-Huy Nguyen, and Suong Nguyen. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In COLING

    2020

  7. [10]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    Pith/arXiv arXiv 2020

  8. [11]

    David Scott Lewis. 2025. From brute force to brain power: How stanford's s1 surpasses deepseek-r1

    2025

  9. [12]

    Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. 2024. https://arxiv.org/abs/2410.12832 Generative reward models . Preprint, arXiv:2410.12832

    Pith/arXiv arXiv 2024

  10. [13]

    Mathematical Association of America . 2023. American mathematics competitions ( AMC ). https://maa.org/student-programs/amc/. Problems from the 2023 AMC 10 and AMC 12 examinations

    2023

  11. [14]

    Mathematical Association of America . 2024. American invitational mathematics examination ( AIME ). https://maa.org/maa-invitational-competitions/. Problems from AIME 2024 and AIME 2025 editions

    2024

  12. [15]

    OpenAI. 2025. https://openai.com/o1/ Openai-o1 . Accessed: 2025-05-16

    2025

  13. [16]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. https://arxiv.org/abs/2305.18290 Direct preference optimization: Your language model is secretly a reward model . Preprint, arXiv:2305.18290

    Pith/arXiv arXiv 2024

  14. [18]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  15. [19]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256

    Pith/arXiv arXiv 2024

  16. [20]

    Ibne Farabi Shihab, Sanjeda Akter, and Anuj Sharma. 2025. Detecting and mitigating reward hacking in reinforcement learning systems: A comprehensive empirical study. arXiv preprint arXiv:2507.05619

    arXiv 2025

  17. [21]

    Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, Yue Wu, Wenhai Wang, Junsong Chen, Zhangyue Yin, Xiaozhe Ren, Jie Fu, Junxian He, Wu Yuan, Qi Liu, and 15 others. 2024. https://arxiv.org/abs/2312.11562 A survey of reasoning with foundation models . Preprint, arXiv:2312.11562

    Pith/arXiv arXiv 2024

  18. [22]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Musique: Multihop reasoning dataset with explanation. arXiv preprint arXiv:2108.00573

    Pith/arXiv arXiv 2022

  19. [23]

    Minh Duc Vu, Han Wang, Zhuang Li, Jieshan Chen, Shengdong Zhao, Zhenchang Xing, and Chunyang Chen. 2024. https://arxiv.org/abs/2401.14268 Gptvoicetasker: Llm-powered virtual assistant for smartphone . Preprint, arXiv:2401.14268

    Pith/arXiv arXiv 2024

  20. [25]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    Pith/arXiv arXiv 2022

  21. [26]

    Hao Wen, Yifan Su, Feifei Zhang, Yunxin Liu, Yunhao Liu, Ya-Qin Zhang, and Yuanchun Li. 2025. Parathinker: Native parallel thinking as a new paradigm to scale llm test-time compute. arXiv preprint arXiv:2509.04475

    Pith/arXiv arXiv 2025

  22. [27]

    Fang Wu, Weihao Xuan, Heli Qi, Aaron Tu, Ximing Lu, Li Erran Li, and Yejin Choi. 2026. https://openreview.net/forum?id=Kx0G6v2c2S Deepsearch: Overcome the bottleneck of reinforcement learning with verifiable rewards via tree-based search . In The Fourteenth International Conference on Learning Representations

    2026

  23. [28]

    Tong Wu, Yang Liu, Jun Bai, Zixia Jia, Shuyi Zhang, Ziyong Lin, Yanting Wang, Song-Chun Zhu, and Zilong Zheng. 2025. Native parallel reasoner: Reasoning in parallelism via self-distilled reinforcement learning. arXiv preprint arXiv:2512.07461

    Pith/arXiv arXiv 2025

  24. [29]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025 a . Qwen3 technical report

    2025

  25. [30]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  26. [31]

    Xinyu Yang, Yuwei An, Hongyi Liu, Tianqi Chen, and Beidi Chen. 2025 b . Multiverse: Your language models secretly decide how to parallelize and merge generation

    2025

  27. [32]

    Zhilin Yang, Peng Qi, Saizheng Zhang, and 1 others. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP

    2018

  28. [33]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models . Preprint, arXiv:2305.10601

    Pith/arXiv arXiv 2023

  29. [34]

    Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. DAPO : An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Proces...

    2025

  30. [35]

    Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. 2024. Accessing gpt-4 level mathematical olympiad solutions via monte carlo tree self-refine with llama-3 8b. arXiv preprint arXiv:2406.07394

    Pith/arXiv arXiv 2024

  31. [36]

    Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, Irwin King, Xue Liu, and Chen Ma. 2025. https://arxiv.org/abs/2503.24235 A survey on test-time scaling in large language models: What, how, where, and how well? Preprint, arXiv:2503.24235

    Pith/arXiv arXiv 2025

  32. [37]

    Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Xinyu Yang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, and Dong Yu. 2025. Parallel-r1: Towards parallel thinking via reinforcement learning

    2025

  33. [38]

    Zijian Zhou, Ao Qu, Zhaoxuan Wu, Sunghwan Kim, Alok Prakash, Daniela Rus, Jinhua Zhao, Bryan Kian Hsiang Low, and Paul Pu Liang. 2025. MEM1 : Learning to synergize memory and reasoning for efficient long-horizon agents. arXiv preprint arXiv:2506.15841

    Pith/arXiv arXiv 2025

  34. [39]

    Guo, Daya and Yang, Dejian and Zhang, Haowei and Song, Junxiao and Wang, Peiyi and Zhu, Qihao and Xu, Runxin and Zhang, Ruoyu and Ma, Shirong and Bi, Xiao and Zhang, Xiaokang and Yu, Xingkai and Wu, Yu and Wu, Z. F. and Gou, Zhibin and Shao, Zhihong and Li, Zhuoshu and Gao, Ziyi and Liu, Aixin and Xue, Bing and Wang, Bingxuan and Wu, Bochao and Feng, Bei ...

  35. [40]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  36. [41]

    Publications Manual , year = "1983", publisher =

    1983

  37. [42]

    2021 , eprint=

    Measuring Mathematical Problem Solving With the MATH Dataset , author=. 2021 , eprint=

    2021

  38. [43]

    2024 , journal =

    HybridFlow: A Flexible and Efficient RLHF Framework , author =. 2024 , journal =

    2024

  39. [44]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  40. [45]

    arXiv preprint arXiv:2506.01939 , year=

    Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning , author=. arXiv preprint arXiv:2506.01939 , year=

    Pith/arXiv arXiv

  41. [46]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  42. [47]

    Dan Gusfield , title =. 1997

    1997

  43. [48]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  44. [49]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  45. [50]

    Xinyu Yang and Yuwei An and Hongyi Liu and Tianqi Chen and Beidi Chen , title =

  46. [51]

    Tong Zheng and Hongming Zhang and Wenhao Yu and Xiaoyang Wang and Xinyu Yang and Runpeng Dai and Rui Liu and Huiwen Bao and Chengsong Huang and Heng Huang and Dong Yu , title =

  47. [52]

    Hao Wen and Yifan Su and Feifei Zhang and Yunxin Liu and Yunhao Liu and Ya-Qin Zhang and Yuanchun Li , title =

  48. [53]

    Advances in Neural Information Processing Systems , volume=

    Rest-mcts*: Llm self-training via process reward guided tree search , author=. Advances in Neural Information Processing Systems , volume=

  49. [54]

    2025 , eprint=

    Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL , author=. 2025 , eprint=

    2025

  50. [55]

    2024 , eprint=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. 2024 , eprint=

    2024

  51. [56]

    arXiv preprint arXiv:2305.14992 , year=

    Reasoning with language model is planning with world model , author=. arXiv preprint arXiv:2305.14992 , year=

    Pith/arXiv arXiv

  52. [57]

    arXiv preprint arXiv:2406.11176 , year=

    Watch every step! llm agent learning via iterative step-level process refinement , author=. arXiv preprint arXiv:2406.11176 , year=

    Pith/arXiv arXiv

  53. [58]

    arXiv preprint arXiv:2210.03629 , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. arXiv preprint arXiv:2210.03629 , year=

    Pith/arXiv arXiv

  54. [59]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  55. [60]

    Journal of artificial intelligence research , volume=

    Reinforcement learning: A survey , author=. Journal of artificial intelligence research , volume=

  56. [61]

    Journal of Cognitive Neuroscience , volume=

    Reinforcement learning , author=. Journal of Cognitive Neuroscience , volume=

  57. [62]

    P lan RAG : A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers

    Lee, Myeonghwa and An, Seonho and Kim, Min-Soo. P lan RAG : A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.na...

    doi:10.18653/v1/2024.naacl-long.364 2024

  58. [63]

    arXiv preprint arXiv:2501.17161 , year=

    Sft memorizes, rl generalizes: A comparative study of foundation model post-training , author=. arXiv preprint arXiv:2501.17161 , year=

    Pith/arXiv arXiv

  59. [64]

    arXiv preprint arXiv:2408.09199 , year=

    TC-RAG: Turing-Complete RAG's Case study on Medical LLM Systems , author=. arXiv preprint arXiv:2408.09199 , year=

    Pith/arXiv arXiv

  60. [65]

    arXiv preprint arXiv:2412.12881 , year=

    RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement , author=. arXiv preprint arXiv:2412.12881 , year=

    Pith/arXiv arXiv

  61. [66]

    Inference Scaling for Long-Context Retrieval Augmented Generation , author=

  62. [67]

    arXiv preprint arXiv:2410.20753 , year=

    Plan times RAG: Planning-guided Retrieval Augmented Generation , author=. arXiv preprint arXiv:2410.20753 , year=

    Pith/arXiv arXiv

  63. [68]

    arXiv preprint arXiv:2504.10198 , year=

    DioR: Adaptive Cognitive Detection and Contextual Retrieval Optimization for Dynamic Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2504.10198 , year=

    Pith/arXiv arXiv

  64. [69]

    IEEE Transactions on Computational Intelligence and AI in games , volume=

    A survey of monte carlo tree search methods , author=. IEEE Transactions on Computational Intelligence and AI in games , volume=. 2012 , publisher=

    2012

  65. [70]

    arXiv preprint arXiv:2502.01142 , year=

    DeepRAG: Thinking to Retrieval Step by Step for Large Language Models , author=. arXiv preprint arXiv:2502.01142 , year=

    Pith/arXiv arXiv

  66. [71]

    Statistical Rejection Sampling Improves Preference Optimization , author=

  67. [72]

    arXiv preprint arXiv:2504.11343 , year=

    A minimalist approach to llm reasoning: from rejection sampling to reinforce , author=. arXiv preprint arXiv:2504.11343 , year=

    Pith/arXiv arXiv

  68. [73]

    arXiv preprint arXiv:2501.05366 , year=

    Search-o1: Agentic search-enhanced large reasoning models , author=. arXiv preprint arXiv:2501.05366 , year=

    Pith/arXiv arXiv

  69. [74]

    arXiv preprint arXiv:2503.09516 , year=

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning , author=. arXiv preprint arXiv:2503.09516 , year=

    Pith/arXiv arXiv

  70. [75]

    arXiv preprint arXiv:2510.14545 , year=

    Agentic entropy-balanced policy optimization , author=. arXiv preprint arXiv:2510.14545 , year=

    arXiv

  71. [76]

    arXiv preprint arXiv:2507.19849 , year=

    Agentic reinforced policy optimization , author=. arXiv preprint arXiv:2507.19849 , year=

    Pith/arXiv arXiv

  72. [77]

    arXiv preprint arXiv:2506.04185 , year=

    R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning , author=. arXiv preprint arXiv:2506.04185 , year=

    Pith/arXiv arXiv

  73. [78]

    arXiv preprint arXiv:2503.20201 , year=

    Open deep search: Democratizing search with open-source reasoning agents , author=. arXiv preprint arXiv:2503.20201 , year=

    Pith/arXiv arXiv

  74. [79]

    arXiv preprint arXiv:2411.00744 , year=

    CORAG: A Cost-Constrained Retrieval Optimization System for Retrieval-Augmented Generation , author=. arXiv preprint arXiv:2411.00744 , year=

    arXiv

  75. [80]

    arXiv preprint arXiv:2501.10053 , year=

    Airrag: Activating intrinsic reasoning for retrieval augmented generation via tree-based search , author=. arXiv preprint arXiv:2501.10053 , year=

    Pith/arXiv arXiv

  76. [81]

    CoRR , year=

    SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation , author=. CoRR , year=

  77. [82]

    Proceedings of the 31st International Conference on Computational Linguistics , pages=

    LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation , author=. Proceedings of the 31st International Conference on Computational Linguistics , pages=

  78. [83]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Self-Knowledge Guided Retrieval Augmentation for Large Language Models , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  79. [84]

    Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

    Im-rag: Multi-round retrieval-augmented generation through learning inner monologues , author=. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages=

  80. [85]

    arXiv preprint arXiv:2410.01428 , year=

    Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks , author=. arXiv preprint arXiv:2410.01428 , year=

    Pith/arXiv arXiv

Showing first 80 references.