Pith. sign in

REVIEW 3 major objections 5 minor 33 references

CoVRL claims that coupling question-only and answer-guided reasoning-trace sampling through a composite distribution makes verifier-free RL for LLM reasoning more sample-efficient and coherent, improving average benchmark performance by 12.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 16:36 UTC pith:NC3NA5XP

load-bearing objection CoVRL is a plausible incremental extension of verifier-free RL with a real but fixable gap in the KL estimator; worth refereeing, not yet ready as is. the 3 major comments →

arxiv 2512.12576 v3 pith:NC3NA5XP submitted 2025-12-14 cs.CL cs.AI

Coupled Variational Reinforcement Learning for Language Model General Reasoning

Xueru Wen , Jie Lou , Yanjiang Liu , Hongyu Lin , Ben He , Xianpei Han , Le Sun , Yaojie Lu
show 1 more author
Debing Zhang
This is my paper
classification cs.CL cs.AI
keywords verifier-free reinforcement learningvariational inferencereasoning traceshybrid samplingcomposite distributionlarge language modelschain-of-thoughtGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that verifier-free RL for LLM reasoning is limited when reasoning traces are sampled only from a question-conditioned prior: exploration is inefficient and traces can be incoherent with the final answer. CoVRL couples that prior with an answer-conditioned posterior by defining a composite distribution over reasoning traces and hybrid-sampling from both modes, then optimizes a variational objective with a GRPO-trained reconstruction term and a KL regularization term. On Qwen2.5 and Qwen3 base models trained on non-mathematical questions, it reports average gains of 12.4% over the base model and 2.3% over the strongest verifier-free baseline, with reasoning gains transferring to mathematical benchmarks. A sympathetic reader would care because, if correct, the method removes the need for external verifiers or reward models while keeping answer-guided exploration.

Core claim

The central discovery is that the two failure modes of prior-only verifier-free RL—low sample efficiency and trace-answer incoherence—can be addressed by optimizing a composite distribution p'(z|x,y) = 1/2 p_phi(z|x) + 1/2 q_psi(z|x,y) over reasoning traces, rather than sampling from either distribution alone. All three distributions (prior, posterior, and answer decoder) are implemented by the same LLM through different prompt templates, so no extra model or verifier is needed. The paper derives an evidence lower bound with a reconstruction term trained by GRPO and a KL term that keeps the composite distribution close to the prior, and makes training tractable through hybrid sampling with i

What carries the argument

The central object is the composite distribution p'(z_t|z_<t,x,y) = 1/2 p_phi(z_t|z_<t,x) + 1/2 q_psi(z_t|z_<t,x,y), defined token-by-token, which couples the question-only prior with the answer-guided posterior. Because sampling from this composite directly is expensive, the paper samples from a hybrid distribution phybrid (prior with probability alpha, posterior with probability 1-alpha) and corrects with importance ratios r_t = p'_new(z_t|·)/phybrid(z_t|·) inside GRPO (Group Relative Policy Optimization, a policy-gradient method that estimates advantages within a group). The KL regularizer D_KL(p'||p_phi) is estimated by two Bregman-divergence control-variate estimators—one for traces sam

Load-bearing premise

The central claim rests on the off-policy importance correction: the GRPO ratio r_t = p'_new/p_hybrid and the KL estimators are assumed to give unbiased or correctly signed gradients of the composite objective, so the KL regularizer really keeps the posterior close to the prior; if that estimator is biased downward, the posterior can drift and the reported gains could come from answer leakage rather than from the variational coupling.

What would settle it

Run CoVRL on a fixed dataset while replacing the KL estimators with an exact Monte-Carlo estimate of D_KL(p'||p_phi) computed from many p'-samples, and compare training trajectories; if the exact-KL run does not preserve the 12.4% gain, the reported result depends on the estimator rather than the coupling. Independently, mask the ground-truth answer when computing the reconstruction reward log p_theta(y|z,x) on posterior-sampled traces: if the gain over prior-only sampling collapses, the benefit is answer leakage, not guided exploration.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim is right, verifier-free RL no longer has to choose between answer guidance during training and inference-time transferability: the same model can be trained with both modes.
  • Reasoning gains learned on non-mathematical questions transfer to mathematical benchmarks, suggesting general-purpose training data can substitute for domain-specific verifiable rewards.
  • Balanced hybrid sampling (alpha near 0.5) outperforms both prior-dominated and posterior-dominated sampling, indicating the coupling itself is the active ingredient rather than either mode alone.
  • Because the prior, posterior, and decoder all share one LLM and differ only in prompt templates, the method can be inserted into existing GRPO pipelines without a second model or external reward signal.
  • The reported gains hold across Qwen2.5 and Qwen3 base models from 7B to 14B parameters, suggesting the mechanism scales with model capacity.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the posterior template reveals the ground-truth answer before the reasoning trace, so part of the measured reward may come from answer leakage into the context; ablating the answer when computing log p_theta(y|z,x) on posterior-sampled traces would separate 'guided exploration' from 'format memorization.'
  • Editorial inference: the fixed 1/2 mixing weight is arbitrary; a learned or annealed alpha could trade exploration and coherence more finely, and the paper's own alpha sweep (0.1 vs 0.9) suggests the optimum may shift with training data or model scale.
  • Editorial inference: selective NLL loss on positive-advantage traces is a second mechanism that filters low-quality traces; an ablation applying the same filtering without the composite distribution would isolate how much of the 12.4% gain comes from the variational coupling itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CoVRL, a verifier-free reinforcement learning method for training LLM reasoning. Reasoning traces are treated as latent variables, and the method couples a question-only prior p_phi(z|x) with an answer-guided posterior q_psi(z|x,y) through a token-level composite distribution p'(z|x,y). A hybrid sampling strategy draws traces from either the prior or the posterior, and the reconstruction term is optimized with GRPO-style updates plus a selective NLL loss, while a KL regularizer is intended to keep the composite distribution close to the prior. Experiments on Qwen2.5/Qwen3 base models show consistent gains on nine reasoning benchmarks, with an overall 12.4% improvement over the base model and a 2.3% improvement over the strongest verifier-free baseline, despite training only on non-mathematical WebInstruct data.

Significance. If the empirical results hold, CoVRL would be a useful contribution to verifier-free RL: it reports consistent improvements across multiple base models and across both general and mathematical reasoning benchmarks after training on non-mathematical data, and it ablates key hyperparameters (alpha, KL coefficient, NLL coefficient, reward formulation). The controlled baseline setup — all methods trained on the same base model with identical GRPO hyperparameters — strengthens the empirical comparison. However, the paper's central claimed contribution is a principled variational-coupling framework, and that theoretical claim is not currently supported. The main gap is the off-policy KL estimator and the GRPO surrogate: as written, the regularizer being optimized is not shown to be the sequence-level KL divergence in Eq. (8), and the importance-weighted policy update lacks the standard surrogate-objective guarantee. No code is provided, and the headline 2.3% margin has no error bars or significance tests. The variational interpretation is plausible but needs either a corrected derivation or a repositioning as a heuristic with supporting ablations.

major comments (3)
  1. [Section 2.5 and Appendix A, Eqs. (14)-(15) vs. Eqs. (23)-(25)] The paper claims the KL estimators are unbiased for D_KL(p'||p_phi) in Eq. (8). The main-text estimators are written with full-sequence ratios and are unbiased if those ratios are used. The appendix, however, derives token-level estimators w_t log w_t - (w_t - 1) and (1/(2r_t)+1/2) log(...) + (1/(2r_t)-1/2), with no importance weight for the prefix distribution. Their expectation is Sigma_t E_{z_<t ~ p_phi or q_psi}[KL_t(z_<t)], not Sigma_t E_{z_<t ~ p'}[KL_t(z_<t)] = D_KL(p'||p_phi). Soft clipping and skipping truncated sequences add further bias. The exact aggregate estimator used in training is never specified, so the regularizer actually optimized is not shown to be the one in Eq. (8). Please provide the exact estimator, its aggregation, and a proof of unbiasedness, or state the actual objective and justify it in another way.
  2. [Section 2.4, Eqs. (11)-(12)] The importance-weighted GRPO objective is presented as a 'mathematically principled' optimization of the composite distribution, but the standard PPO/GRPO clipped surrogate requires the sampling distribution to be the old policy. Here the behavior policy is phybrid, while the target is p'_new, and p'_old does not appear in the ratio. Clipping against phybrid does not inherit the trust-region or lower-bound properties of the standard objective, and no off-policy policy-gradient theorem is supplied. The update may still work as a heuristic, but the claim that it optimizes the variational objective in Eq. (8) is unsupported. Please provide a derivation or explicitly present the update as an off-policy approximation with empirical justification.
  3. [Section 3.2, Table 1] The headline claim is an additional 2.3% over RLPR. Tables report only point estimates at Average@N, with no standard errors, confidence intervals, or paired significance tests. Several per-task margins are small (AIME'24 7.5 vs. 6.5; MMLU-Pro 46.5 vs. 44.9), and the overall margin could reflect sampling noise. Please report the variance across the N evaluation runs and a significance test for the Overall column, and release code (or at least detailed training/evaluation logs) to allow verification of the reported numbers and the exact KL aggregation.
minor comments (5)
  1. [Table 1] Formatted text has missing spaces (e.g., '31.642.7' and '30.446.5'), making the table appear misaligned. Check column alignment in the camera-ready version. The numerical values themselves are internally consistent with Tables 2 and 3.
  2. [Section 2.3 and Section 3.4] Eq. (6) fixes the composite weight at 1/2 for prior and posterior, while alpha in Eq. (9) and Figure 5 controls the sampling frequency only. Please state explicitly whether the composite target p' remains at 1/2 when alpha != 0.5, since this affects the importance ratios and the interpretation of the alpha ablation.
  3. [Section 2.4] The text says optimizing the prior corresponds to standard maximum likelihood and can be computed with NLL loss, but then applies NLL only to samples with positive advantage. This is a filtered/weighted NLL, not MLE. Clarify the terminology.
  4. [Appendix B] The baseline gradient formulas are difficult to parse; e.g., the RA VR expression appears to mix notation (nabla_theta R(z) times D_KL) and lacks definitions of P({pi|ti in y*}). Please rewrite for clarity or add a table with clearly defined symbols.
  5. [Section 2.5] The sentence 'we adopt the latter approach, which we find is more stable' is anecdotal. If this is a design choice, give the comparison or cite a systematic ablation; otherwise rephrase as a practical choice.

Circularity Check

0 steps flagged

No significant circularity: the central claims are empirical and the variational/RL objectives are not constructed from the benchmark results.

full rationale

The paper's headline claims are benchmark numbers (12.4% over base, 2.3% over the verifier-free RL baselines) obtained on held-out external tasks with math-verify answer checking; no fitted parameter is renamed as a prediction. The variational derivation (Eqs. 5-8) is a standard ELBO with q(z) replaced by the composite distribution p′(z|x,y), and the subsequent GRPO importance ratio (Eq. 12) and KL estimators (Eqs. 14-15) are off-policy/importance-sampling estimators rather than definitions of the measured outcome. The self-referential reward—the model's own probability of the reference answer—is the shared training signal of all verifier-free baselines and is not a hidden input-to-output reduction. The references to GRPO, Schulman (2020), and template/verifier tools are external and non-load-bearing for the headline claim. The skeptic's concern about the KL estimators (token-level aggregation and clipping introducing bias relative to the sequence-level KL) is a mathematical-correctness or attribution issue, not a circularity: the paper does not derive the benchmark result by construction from the estimator. No load-bearing self-citation chain or fitted-input-called-prediction step is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central method rests on standard variational inference plus several domain assumptions about verifier-free rewards and the validity of the off-policy estimators. The free hyperparameters α, λ_KL, λ_NLL are hand-chosen and shown to affect results; no invented entities are introduced.

free parameters (4)
  • α (hybrid sampling probability) = 0.5 (default)
    Probability of sampling from the prior vs posterior; hand-set to 0.5; Figure 5 and Figure 7 show performance and training dynamics are sensitive to it.
  • λ_KL (KL regularization coefficient) = 1.0 (default, inferred from ablation λ_KL=0.1)
    Weights D_KL(p'||p_φ); Table 4 shows λ_KL=0.1 collapses overall accuracy from 50.2 to 27.4, so the method depends critically on this hand-tuned weight.
  • λ_NLL (NLL loss coefficient) = 1.0 (default, inferred from ablation λ_NLL=0.1)
    Weights the selective negative log-likelihood term; Table 4 shows moderate sensitivity (50.2→44.7 at 0.1).
  • GRPO clip threshold ε = 0.3
    Set because of the off-policy nature of the algorithm; not ablated, but affects the surrogate objective.
axioms (5)
  • standard math Jensen's inequality yields the ELBO in Eq 5 and Eq 8
    The variational lower bound is standard; no issue.
  • domain assumption Reasoning traces are latent variables mediating question and answer, p(y|x)=∫p(y|z,x)p(z|x)dz (Eq 4)
    The generative-process assumption underlying the whole verifier-free latent-reasoning framework.
  • domain assumption The model's own answer probability log p_θ(y|z,x) is a valid training reward
    Shared by verifier-free baselines; if the model can inflate this probability via format or shortcut learning, the training signal is not reasoning quality.
  • ad hoc to paper Off-policy importance ratio r_t = p'_new / phybrid (Eq 12) yields a valid GRPO-style update for the composite distribution
    The paper replaces the old composite policy with the behavior distribution without proof that clipping this ratio is a valid surrogate objective.
  • ad hoc to paper The KL estimators in Eqs 14–15 are unbiased for sequence-level D_KL(p'||p_φ)
    The derivation in Appendix A is per-token and samples prefixes from p_φ or q_ψ, not from p'; the claim of unbiased sequence-level KL is not established.

pith-pipeline@v1.3.0-alltime-deepseek · 17832 in / 26034 out tokens · 216604 ms · 2026-08-03T16:36:49.495313+00:00 · methodology

0 comments
read the original abstract

While reinforcement learning has achieved impressive progress in language model reasoning, it is constrained by the requirement for verifiable rewards. Recent verifier-free RL methods address this limitation by utilizing the probabilities that LLMs generate reference answers as reward signals. However, these approaches typically sample reasoning traces conditioned only on the question. This design decouples reasoning-trace sampling from answer information, leading to inefficient exploration and incoherence between traces and final answers. In this paper, we propose \textit{\b{Co}upled \b{V}ariational \b{R}einforcement \b{L}earning} (CoVRL), which bridges variational inference and reinforcement learning by coupling prior and posterior distributions through a hybrid sampling strategy. By constructing and optimizing a composite distribution that integrates these two distributions, CoVRL enables efficient exploration while preserving strong thought-answer coherence. Extensive experiments on mathematical and general reasoning benchmarks show that CoVRL improves performance by 12.4\% over the base model and achieves an additional 2.3\% improvement over state-of-the-art verifier-free RL baselines, providing a principled framework for enhancing the general reasoning capabilities of language models.

Figures

Figures reproduced from arXiv: 2512.12576 by Ben He, Debing Zhang, Hongyu Lin, Jie Lou, Le Sun, Xianpei Han, Xueru Wen, Yanjiang Liu, Yaojie Lu.

Figure 1
Figure 1. Figure 1: Comparison between verifier-free RL with question-only training and CoVRL. Unlike prior methods that sample reasoning traces conditioned only on the question, CoVRL couples question￾conditioned prior sampling with answer-conditioned posterior sam￾pling via a hybrid variational framework, enabling efficient explo￾ration while preserving strong trace-answer coherence. In this paradigm, the LLM generates a ch… view at source ↗
Figure 2
Figure 2. Figure 2: CoVRL employs hybrid sampling between prior pϕ(z|x) and posterior qψ(z|x, y) to generate reasoning traces. It optimizes the reconstruction term using GRPO and NLL loss, with KL regularization applied to ensure training-inference coherence. from the observed data. This lower bound is called the evidence lower bound (ELBO), which is widely used in variational autoencoders (Kingma & Welling, 2022) and diffusi… view at source ↗
Figure 3
Figure 3. Figure 3: Prompt templates after applying chat template for Prior and Posterior distributions. The key difference lies in the order of reasoning and answer components within the assistant’s response. samples drawn from the posterior distribution. In practice, this importance ratio can be computed via two forward passes for each sampled reasoning trace z: one with the prior template to obtain pϕ(z|x) and one with the… view at source ↗
Figure 4
Figure 4. Figure 4: Training dynamics of CoVRL across different metrics. We observe stable improvements in reasoning quality alongside effective optimization of both reconstruction and regularization objectives. AIME'24 AQuA CARP-EN GPQA MATH-500 Minerva MMLU-Pro SAT-Math TheoremQA Overall 0 20 40 60 80 100 Performance (%) 5.4 67.0 63.1 30.5 61.8 20.4 42.2 88.7 29.1 45.4 7.5 77.3 65.1 30.4 66.3 25.5 46.5 97.1 36.3 50.2 0.1 38… view at source ↗
Figure 5
Figure 5. Figure 5: Impact of hybrid sampling probability α. (mcaleste, 2023). For evaluation, we use temperature=0.6 and max tokens=4096. We utilize Math-Verify to check answer correctness; more details are provided in Appendix C. To reduce evaluation variance, we repeat the evaluation N times for each dataset, where N depends on the test set size. We report the average performance across these N runs as Average@N to account… view at source ↗
Figure 6
Figure 6. Figure 6: shows the behavior of both estimators across different likelihood ratios. When r > 1, the posterior assigns higher probability to the sample than the prior, indicating reasoning traces well-aligned with target answers. When 0 < r < 1, the prior assigns higher probability, representing more exploratory reasoning paths. Our hybrid sampling strategy leverages this complementary behavior to ensure stable optim… view at source ↗
Figure 7
Figure 7. Figure 7: Training dynamics comparison between different prior sampling probabilities (α = 0.1 vs α = 0.9). All methods aim to improve reasoning without external verifiers but differ in their approach. JLB (Tang et al., 2025) uses log-probability as reward with fixed answer term weighting. LaTRO (Chen et al., 2024) incorporates KL regularization between policy and reference models. VeriFree (Zhou et al., 2025a) uses… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 29 linked inside Pith

  1. [6]

    DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y ., Wu, Z

    URLhttps://arxiv.org/abs/2502.01456. DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y ., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., Liu, A., Xue, B., Wang, B., Wu, B., Feng, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Chen, D., Ji, D., Li, E., Lin, F.,...

    Pith/arXiv arXiv

  2. [7]

    Ho, J., Jain, A., and Abbeel, P

    URL https://arxiv.org/abs/2501.12948. Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models,

    Pith/arXiv arXiv

  3. [9]

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y ., Zheng, L., Yu, 9 Coupled Variational Reinforcement Learning for Language Model General Reasoning C

    URL https://arxiv.org/ abs/1312.6114. Kwon, W., Li, Z., Zhuang, S., Sheng, Y ., Zheng, L., Yu, 9 Coupled Variational Reinforcement Learning for Language Model General Reasoning C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. InProceedings of the ACM SIGOPS 29th Symposium o...

    Pith/arXiv arXiv

  4. [10]

    Li, R., Li, X., Lin, C., Collinson, M., and Mao, R

    URL https://arxiv.org/abs/2206.14858. Li, R., Li, X., Lin, C., Collinson, M., and Mao, R. A stable variational autoencoder for text modelling,

    Pith/arXiv arXiv

  5. [12]

    Ling, W., Yogatama, D., Dyer, C., and Blunsom, P

    URLhttps://arxiv.org/abs/2510.25206. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation : Learning to solve and explain algebraic word problems,

    arXiv

  6. [14]

    Mathematical Association of America

    URL https://arxiv.org/abs/ 2505.14652. Mathematical Association of America. Aime problems and solutions. Art of Problem Solving Wiki,

    Pith/arXiv arXiv

  7. [15]

    Con- tains 32 math questions from May 2023 SAT

    URL https: //huggingface.co/datasets/mcaleste/ sat_multiple_choice_math_may_23. Con- tains 32 math questions from May 2023 SAT. Accessed: 2025-01-27. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P.,...

    2023

  8. [16]

    URL https: //arxiv.org/abs/2203.02155. Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., ...

    Pith/arXiv arXiv

  9. [17]

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C

    URL https: //arxiv.org/abs/2412.15115. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model,

    Pith/arXiv arXiv

  10. [18]

    Rein, D., Hou, B

    URL https://arxiv.org/abs/2305.18290. Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y ., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark,

    Pith/arXiv arXiv

  11. [19]

    Schulman, J

    URL https://arxiv.org/abs/2311.12022. Schulman, J. Approximating KL divergence,

    Pith/arXiv arXiv

  12. [20]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y

    URL https://arxiv.org/abs/ 1707.06347. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y . K., Wu, Y ., and Guo, D. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models,

    Pith/arXiv arXiv

  13. [21]

    Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y ., Lin, H., and Wu, C

    URL https://arxiv.org/abs/2402.03300. Sheng, G., Zhang, C., Ye, Z., Wu, X., Zhang, W., Zhang, R., Peng, Y ., Lin, H., and Wu, C. Hybridflow: A flexi- ble and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv

  14. [22]

    Tang, Y ., Wang, S., Madaan, L., and Munos, R

    URLhttps://arxiv.org/abs/2209.13085. Tang, Y ., Wang, S., Madaan, L., and Munos, R. Beyond verifiable rewards: Scaling reinforcement learning for language models to unverifiable data,

    Pith/arXiv arXiv

  15. [23]

    URLhttps: //arxiv.org/abs/2503.19618. Team, S. Sglang: A fast serving framework for large language models and vision language models,

    Pith/arXiv arXiv

  16. [24]

    Accessed: 2025-01-23

    URL https://github.com/ sgl-project/sglang. Accessed: 2025-01-23. Wang, Y ., Ma, X., Zhang, G., Ni, Y ., Chandra, A., Guo, S., Ren, W., Arulraj, A., He, X., Jiang, Z., Li, T., Ku, M., Wang, K., Zhuang, A., Fan, R., Yue, X., and Chen, 10 Coupled Variational Reinforcement Learning for Language Model General Reasoning W. Mmlu-pro: A more robust and challengi...

    2025

  17. [25]

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D

    URL https://arxiv.org/abs/2406.01574. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models,

    Pith/arXiv arXiv

  18. [26]

    URL https://arxiv.org/abs/ 2201.11903. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., Zheng, C., Liu, D., Zhou, F., Huang, F., Hu, F., Ge, H., Wei, H., Lin, H., Tang, J., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Zhou, J., Lin, J., Dang, K., Bao, K., Yang, K., Yu, L., Deng, L., Li, M., Xue, ...

    Pith/arXiv arXiv

  19. [27]

    URL https: //arxiv.org/abs/2505.09388. Yu, Q., Zhang, Z., Zhu, R., Yuan, Y ., Zuo, X., Yue, Y ., Dai, W., Fan, T., Liu, G., Liu, L., Liu, X., Lin, H., Lin, Z., Ma, B., Sheng, G., Tong, Y ., Zhang, C., Zhang, M., Zhang, W., Zhu, H., Zhu, J., Chen, J., Chen, J., Wang, C., Yu, H., Song, Y ., Wei, X., Zhou, H., Liu, J., Ma, W.-Y ., Zhang, Y .-Q., Yan, L., Qia...

    Pith/arXiv arXiv

  20. [28]

    Yue, X., Zheng, T., Zhang, G., and Chen, W

    URLhttps://arxiv.org/abs/2401.10020. Yue, X., Zheng, T., Zhang, G., and Chen, W. Mammoth2: Scaling instructions from the web,

    Pith/arXiv arXiv

  21. [29]

    URL https: //arxiv.org/abs/2405.03548. Yue, Y ., Yuan, Y ., Yu, Q., Zuo, X., Zhu, R., Xu, W., Chen, J., Wang, C., Fan, T., Du, Z., Wei, X., Yu, X., Liu, G., Liu, J., Liu, L., Lin, H., Lin, Z., Ma, B., Zhang, C., Zhang, M., Zhang, W., Zhu, H., Zhang, R., Liu, X., Wang, M., Wu, Y ., and Yan, L. Vapo: Efficient and reliable reinforce- ment learning for advan...

    Pith/arXiv arXiv

  22. [30]

    Zhang, B., Zhou, K., Wei, X., Zhao, W

    URL https://arxiv.org/abs/2504.05118. Zhang, B., Zhou, K., Wei, X., Zhao, W. X., Sha, J., Wang, S., and Wen, J.-R. Evaluating and improving tool- augmented computation-intensive math reasoning,

    Pith/arXiv arXiv

  23. [31]

    Zhao, Y ., Liu, Y ., Liu, J., Chen, J., Wu, X., Hao, Y ., Lv, T., Huang, S., Cui, L., Ye, Q., Wan, F., and Wei, F

    URLhttps://arxiv.org/abs/2306.02408. Zhao, Y ., Liu, Y ., Liu, J., Chen, J., Wu, X., Hao, Y ., Lv, T., Huang, S., Cui, L., Ye, Q., Wan, F., and Wei, F. Geometric-mean policy optimization,

    Pith/arXiv arXiv

  24. [32]

    Zhou, X., Liu, Z., Sims, A., Wang, H., Pang, T., Li, C., Wang, L., Lin, M., and Du, C

    URL https://arxiv.org/abs/2507.20673. Zhou, X., Liu, Z., Sims, A., Wang, H., Pang, T., Li, C., Wang, L., Lin, M., and Du, C. Reinforcing gen- eral reasoning without verifiers, 2025a. URL https: //arxiv.org/abs/2505.21493. Zhou, X., Liu, Z., Wang, H., Du, C., Lin, M., Li, C., Wang, L., and Pang, T. Variational reasoning for language mod- els, 2025b. URL ht...

    arXiv

  25. [33]

    org/abs/2504.16084

    URL https://arxiv. org/abs/2504.16084. 11 Coupled Variational Reinforcement Learning for Language Model General Reasoning A. KL Divergence Estimator In this section, we provide the derivation and intuition for our KL divergence estimators used in CoVRL. A.1. Derivation via Control Variates and Bregman Divergence Our KL estimators are derived following the...

    Pith/arXiv arXiv 2020

  26. [2016]

    org/abs/1509.00519

    URL https://arxiv. org/abs/1509.00519. Chan, A., Silva, H., Lim, S., Kozuno, T., Mahmood, A. R., and White, M. Greedification operators for policy op- timization: Investigating forward and reverse kl diver- gences,

    Pith/arXiv arXiv

  27. [2017]

    Ma, X., Liu, Q., Jiang, D., Zhang, G., Ma, Z., and Chen, W

    URL https: //arxiv.org/abs/1705.04146. Ma, X., Liu, Q., Jiang, D., Zhang, G., Ma, Z., and Chen, W. General-reasoner: Advancing llm reasoning across all domains,

    Pith/arXiv arXiv

  28. [2019]

    Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K

    URL https://arxiv.org/abs/1911.05343. Lightman, H., Kosaraju, V ., Burda, Y ., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., and Cobbe, K. Let’s verify step by step.arXiv preprint arXiv:2305.20050,

    Pith/arXiv arXiv 1911

  29. [2020]

    Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y

    URL https://arxiv.org/ abs/2006.11239. Hu, J., Wu, X., Zhu, Z., Xianyu, Wang, W., Zhang, D., and Cao, Y . Openrlhf: An easy-to-use, scalable and high-performance rlhf framework.arXiv preprint arXiv:2405.11143,

    Pith/arXiv arXiv 2006

  30. [2022]

    Chen, C., Liu, Z., Du, C., Pang, T., Liu, Q., Sinha, A., Varakantham, P., and Lin, M

    URL https://arxiv.org/abs/ 2107.08285. Chen, C., Liu, Z., Du, C., Pang, T., Liu, Q., Sinha, A., Varakantham, P., and Lin, M. Bootstrapping language models with dpo implicit rewards,

    Pith/arXiv arXiv

  31. [2023]

    URL https: //arxiv.org/abs/2305.12524. Cui, G., Yuan, L., Wang, Z., Wang, H., Zhang, Y ., Chen, J., Li, W., He, B., Fan, Y ., Yu, T., Xu, Q., Chen, W., Yuan, J., Chen, H., Zhang, K., Lv, X., Wang, S., Yao, Y ., Han, X., Peng, H., Cheng, Y ., Liu, Z., Sun, M., Zhou, B., and Ding, N. Process reinforcement through implicit rewards,

    Pith/arXiv arXiv

  32. [2024]

    Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y ., Ma, X., Xu, J., Wang, X., and Xia, T

    URLhttps://arxiv.org/abs/2411.04282. Chen, W., Yin, M., Ku, M., Lu, P., Wan, Y ., Ma, X., Xu, J., Wang, X., and Xia, T. Theoremqa: A theorem- driven question answering dataset,

    Pith/arXiv arXiv

  33. [2025]

    Chen, H., Feng, Y ., Liu, Z., Yao, W., Prabhakar, A., Hei- necke, S., Ho, R., Mui, P., Savarese, S., Xiong, C., and Wang, H

    URL https: //arxiv.org/abs/2406.09760. Chen, H., Feng, Y ., Liu, Z., Yao, W., Prabhakar, A., Hei- necke, S., Ho, R., Mui, P., Savarese, S., Xiong, C., and Wang, H. Language models are hidden reasoners: Unlock- ing latent reasoning capabilities via self-rewarding,

    Pith/arXiv arXiv