Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

Hui Huang; Kehao Chen; Weiqin Wang; Yile Wang

arxiv: 2512.15146 · v4 · submitted 2025-12-17 · 💻 cs.CL

Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

Weiqin Wang , Yile Wang , Kehao Chen , Hui Huang This is my paper

Pith reviewed 2026-05-16 21:50 UTC · model grok-4.3

classification 💻 cs.CL

keywords test-time reinforcement learningpseudo-label estimationmajority votingreasoning modelssubgroup partitioningstep-wise confidenceAIME 2025AMC benchmark

0 comments

The pith

SCOPE generates finer-grained pseudo-labels for test-time RL by weighting reasoning steps with model confidence and partitioning outputs into subgroups instead of using majority voting.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes SCOPE to fix two problems in test-time reinforcement learning for reasoning models: majority voting creates confirmation bias and gives sparse rewards. SCOPE weights each step of a reasoning path by the model's own confidence and splits the pool of candidate outputs into balanced subgroups that each produce their own local consensus. This supplies more diverse and higher-quality supervision signals than a single global vote. If the approach works, models can improve at math and logic tasks using only unlabeled self-generated data. Experiments report consistent gains, including 13.1 percent relative improvement on AIME 2025.

Core claim

SCOPE replaces majority-vote pseudo-labels with subgroup-specific step-wise confidence-weighted estimation. It first scores every reasoning step by the model's internal confidence, then dynamically partitions the output pool into independent subgroups that trade off quality against exploration diversity, and finally derives a local consensus label inside each subgroup to create multiple distinct training targets.

What carries the argument

SCOPE (subgroup-specific step-wise confidence-weighted pseudo-label estimation), which uses per-step model confidence to rank reasoning paths and dynamic partitioning to create diverse local-consensus targets.

If this is right

Models receive denser and more reliable reward signals during test-time RL because high-confidence paths are prioritized over raw frequency counts.
Diverse supervision targets from multiple subgroups encourage broader exploration and reduce collapse to a single reasoning style.
Performance improves on hard math benchmarks, with reported relative gains of 13.1 percent on AIME 2025 and 8.1 percent on AMC.
The method works across different base models without requiring additional annotated data.
Local consensus inside subgroups supplies supervision even when no global majority exists.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same partitioning idea could be tested on non-math domains such as code generation or multi-step planning where majority voting is also noisy.
Varying the number of subgroups or the quality-diversity trade-off parameter might reveal an optimal operating point for different model sizes.
Combining SCOPE labels with verifiable rewards on a subset of problems could further stabilize training.
Tracking how subgroup assignments change across training steps could serve as a diagnostic for when the model is becoming over-confident.

Load-bearing premise

That weighting reasoning steps by model confidence and splitting outputs into dynamic subgroups will yield less biased, higher-quality pseudo-labels than majority voting without the partitioning step itself creating new selection artifacts.

What would settle it

Train the same base model on the same unlabeled problems once with standard majority voting and once with SCOPE, then measure both final benchmark accuracy and the fraction of pseudo-labels that contradict verifiable ground truth.

Figures

Figures reproduced from arXiv: 2512.15146 by Hui Huang, Kehao Chen, Weiqin Wang, Yile Wang.

**Figure 1.** Figure 1: Illustration of the difference between TTRL (Zuo et al., 2025) and our method. Top: consensus label estimation with step-wise confidence weighting. Bottom: group partition and reward calculation using subgroup-specific consensus labels. o1 (OpenAI, 2024). From the perspective of training data, RLVR is similar to supervised fine-tuning that requires ground-truth labels to guide the iterative policy lear… view at source ↗

**Figure 2.** Figure 2: Overview of the SCOPE framework. The process involves (a) generating responses with step-wise confidence, (b) estimating consensus labels via weighted voting, (c) evaluating different subgroup partitions, (d) employing Pareto optimization to select the optimal subgroup size m∗ by balancing (g) quality and exploration metrics, and (e) computing rewards using the optimized subgroup strategy for model updates… view at source ↗

**Figure 3.** Figure 3: Analysis of the trade-off parameter λ. Conversely, setting λ too low (e.g., λ = 0) results in a noticeable performance decline. Lacking the guidance of consensus quality, the optimization process is prone to over-exploration where the model drifts away from correct reasoning trajectories. This underscores the importance of jointly optimizing for both consensus alignment and exploration to achieve robus… view at source ↗

**Figure 4.** Figure 4: Impact analysis of confidence granularity. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

**Figure 6.** Figure 6: A question from AIME 2024. (e.g., 9×60 s − t = 240); (2) in the intermediatecalculation stage, this mistake leads to an impossible negative duration (t = −24 minutes); and (3) in the Python-execution stage, the error is obscured by applying abs(), which masks the incorrect reasoning rather than correcting it. These incorrect components are highlighted with red boxes in the annotated solution. In contras… view at source ↗

**Figure 7.** Figure 7: Solution from Qwen2.5-Math-1.5B. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗

**Figure 8.** Figure 8: Solution after training process of TTRL. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

**Figure 9.** Figure 9: Solution after training process of SCOPE. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗

read the original abstract

Test-time reinforcement learning mitigates the reliance on annotated data by using majority voting results as pseudo-labels, emerging as a complementary direction to reinforcement learning with verifiable rewards (RLVR) for improving reasoning ability. However, this voting strategy often induces confirmation bias and suffers from sparse rewards, limiting the overall performance. In this work, we propose subgroup-specific step-wise confidence-weighted pseudo-label estimation (SCOPE), a framework integrating model confidence and dynamic subgroup partitioning to address these issues. Specifically, SCOPE integrates the proposed step-wise confidence into pseudo label estimation, prioritizing high-quality reasoning paths over simple frequency count. Furthermore, it dynamically partitions the candidate outputs pool into independent subgroups by balancing reasoning quality against exploration diversity. By deriving local consensus via repeat sampling for each sub group, SCOPE provides diverse supervision targets to encourage broader exploration. We conduct experiments across various models and benchmarks, experimental results show that SCOPE consistently outperforms recent baselines. Notably, SCOPE achieving relative improvements of 13.1% on challenging AIME 2025 and 8.1% on AMC. The code is released at https://github.com/szu-tera/SCOPE.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

SCOPE gives a practical tweak to majority voting in test-time RL by adding step-wise confidence weights and dynamic subgroup partitioning, with reported gains on AIME and AMC.

read the letter

SCOPE is a direct attempt to fix two problems with majority voting for pseudo-labels in test-time RL: confirmation bias and sparse signals. The method weights each reasoning step by the model's own confidence and splits the sample pool into subgroups to compute local consensus instead of one global vote. This is meant to produce higher-quality targets and push the model toward broader exploration during test-time updates. The abstract reports consistent gains over recent baselines, including 13.1% relative improvement on AIME 2025 and 8.1% on AMC, and the code is released, which makes the claims easier to check. That combination of step-wise weighting and dynamic partitioning is the concrete new piece; prior work on voting is cited but does not describe this exact pairing. The logic is straightforward and stays grounded in the stated goal of better pseudo-labels without adding heavy new machinery. The main soft spot is the lack of experimental detail in the summary. No mention of exact baselines, run counts, error bars, or how the quality-diversity balance in partitioning is tuned. Without those, it is hard to judge whether the reported gains are stable or whether the subgroups simply trade one form of selection bias for another. The central assumption—that local consensus reduces bias more than it introduces artifacts—looks plausible but needs the full results to evaluate. This paper is aimed at people already working on test-time scaling for reasoning models. It offers a usable algorithm rather than a broad theory shift, so readers who want to try incremental improvements on RLVR-style methods will find it relevant. The thinking is clear and the proposal engages the existing literature without obvious contradictions. I would send it for peer review so the empirical claims can be examined closely.

Referee Report

3 major / 2 minor

Summary. The manuscript proposes SCOPE (subgroup-specific step-wise confidence-weighted pseudo-label estimation), a test-time RL framework that replaces majority voting with step-wise model confidence weighting and dynamic partitioning of candidate outputs into subgroups. Local consensus is computed per subgroup to yield diverse pseudo-labels, with the goal of reducing confirmation bias and increasing reward density. Experiments across models and benchmarks are reported to show consistent gains, including 13.1% relative improvement on AIME 2025 and 8.1% on AMC.

Significance. If the empirical claims hold under rigorous controls, SCOPE would constitute a practical refinement of test-time RLVR methods by supplying finer-grained, less biased supervision signals. The explicit code release supports reproducibility and could facilitate follow-up work on subgroup-based exploration in reasoning models.

major comments (3)

[§3.2] §3.2 (Dynamic Partitioning): The algorithm for balancing reasoning quality against exploration diversity is described only at a high level; no pseudocode, exact similarity metric, or threshold is provided. This is load-bearing for the central claim that partitioning simultaneously avoids selection artifacts and increases diversity, as the reader's weakest assumption notes.
[§4.3] §4.3 (Experimental Results): No error bars, number of independent runs, or statistical significance tests accompany the reported 13.1% relative gain on AIME 2025 or 8.1% on AMC. Without these, it is impossible to determine whether the outperformance exceeds baseline variance, directly undermining the soundness assessment of the central empirical claim.
[§4.1] §4.1 (Baselines and Protocol): The implementation details, sampling budgets, and hyperparameter settings for the compared baselines are not fully specified. This prevents verification that the claimed gains arise from the proposed confidence weighting and partitioning rather than from differences in experimental protocol.

minor comments (2)

[Abstract] The abstract states 'repeat sampling for each sub group' but the main text should explicitly clarify whether this re-uses the original candidate pool or draws fresh samples, to avoid ambiguity in the local-consensus procedure.
[§3.1] Notation for step-wise confidence (e.g., how it is normalized or aggregated across tokens) should be introduced once in §3.1 and used consistently thereafter.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback on our manuscript. We appreciate the recognition of the potential significance of SCOPE and will address all major comments through revisions to improve clarity and rigor. Below, we provide point-by-point responses.

read point-by-point responses

Referee: [§3.2] §3.2 (Dynamic Partitioning): The algorithm for balancing reasoning quality against exploration diversity is described only at a high level; no pseudocode, exact similarity metric, or threshold is provided. This is load-bearing for the central claim that partitioning simultaneously avoids selection artifacts and increases diversity, as the reader's weakest assumption notes.

Authors: We agree that the description in §3.2 is at a high level and lacks implementation specifics. In the revised manuscript, we will include pseudocode for the dynamic partitioning algorithm, specify the similarity metric used, and provide the exact threshold values for balancing quality and diversity. This will allow readers to fully reproduce the partitioning process and verify its contribution to reducing confirmation bias while increasing diversity. revision: yes
Referee: [§4.3] §4.3 (Experimental Results): No error bars, number of independent runs, or statistical significance tests accompany the reported 13.1% relative gain on AIME 2025 or 8.1% on AMC. Without these, it is impossible to determine whether the outperformance exceeds baseline variance, directly undermining the soundness assessment of the central empirical claim.

Authors: We acknowledge the importance of statistical rigor. In the revised version, we will report results from multiple independent runs, include error bars representing standard deviation, and perform statistical significance tests to confirm that the observed improvements are significant. This will strengthen the empirical claims. revision: yes
Referee: [§4.1] §4.1 (Baselines and Protocol): The implementation details, sampling budgets, and hyperparameter settings for the compared baselines are not fully specified. This prevents verification that the claimed gains arise from the proposed confidence weighting and partitioning rather than from differences in experimental protocol.

Authors: We agree that more details are necessary for reproducibility. We will expand §4.1 to include complete implementation details for all baselines, specify the sampling budgets, and list all hyperparameter settings used in our experiments and those of the baselines. This will ensure that the gains can be attributed to the SCOPE components. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper presents SCOPE as an algorithmic framework integrating step-wise model confidence into pseudo-label estimation and dynamic partitioning of outputs into subgroups for local consensus via repeat sampling. No equations, derivations, or self-citations are described that reduce the claimed performance gains to quantities defined by fitted parameters within the paper or to prior self-referential results. The improvements on benchmarks such as AIME 2025 and AMC are presented as empirical outcomes of the independent algorithmic design rather than internal consistency that loops back to the inputs by construction. The method is self-contained and validated externally through experiments.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

Based on the abstract alone, the central claim rests on two domain assumptions about confidence and partitioning; no explicit free parameters or invented entities are named.

axioms (2)

domain assumption Model confidence at individual reasoning steps correlates with the correctness of the overall path
Used to weight pseudo-labels more heavily than simple frequency counts
ad hoc to paper Dynamic partitioning of outputs into subgroups can simultaneously preserve quality and increase exploration diversity
Core mechanism for generating diverse local consensus targets

pith-pipeline@v0.9.0 · 5501 in / 1268 out tokens · 46443 ms · 2026-05-16T21:50:33.763239+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

SCOPE integrates the proposed step-wise confidence into pseudo label estimation... dynamically partitions the candidate outputs pool into independent subgroups by balancing reasoning quality against exploration diversity.
IndisputableMonolith/Foundation/ArithmeticFromLogic.lean LogicNat induction and recovery unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We employ Pareto optimization to automatically select the optimal subgroup size during training.

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

When the Majority Votes Wrong, the Intervention Timing for Test-Time Reinforcement Learning Hides in the Extinction Window
cs.LG 2026-05 unverdicted novelty 6.0

TTRL-Guard mitigates the Correct-Answer Extinction Window in test-time RL via flip-rate-aware reward scaling, minority-preserving sampling, and risk-conditioned sparse updates, yielding best average pass@1 on Qwen mod...

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages · cited by 1 Pith paper · 1 internal anchor

[1]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

SelfCite: Self-supervised alignment for con- text attribution in large language models. In Forty- second International Conference on Machine Learn- ing. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing rea- soning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep...

work page internal anchor Pith review Pith/arXiv arXiv 2025
[2]

AdaCoT:Pareto-optimaladaptivechain-of-thoughttriggeringviareinforcement learning.arXivpreprintarXiv:2505.11896,2025

Let’s verify step by step. In The Twelfth Inter- national Conference on Learning Representations. Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, and Shuangzhi Wu. 2025. AdaCoT: Pareto-optimal adaptive chain-of-thought triggering via reinforce- ment learning. arXiv preprint arXiv:2505.11896. OpenAI. 2024. Op...

work page arXiv 2025
[3]

arXiv preprint arXiv:2509.01321

Towards high data efficiency in reinforce- ment learning with verifiable reward. arXiv preprint arXiv:2509.01321. Carel van Niekerk, Renato Vukovic, Benjamin Matthias Ruppik, Hsien-chin Lin, and Milica Gaši ´c. 2025. Post-training large language models via reinforce- ment learning from self-feedback. arXiv preprint arXiv:2507.21931. Pinzheng Wang, Juntao ...

work page arXiv 2025
[4]

Specifically, we generate 16 re- sponses (4 for models with 32k context) per ques- tion using a temperature of 0.6 and a top-p value of 0.95

and report pass@1 using non-zero temper- ature sampling. Specifically, we generate 16 re- sponses (4 for models with 32k context) per ques- tion using a temperature of 0.6 and a top-p value of 0.95. The pass@1 score is computed as: pass@1 = 1 k kX i=1 pi, (16) where pi indicates correctness of the i-th response. where pi indicates whether the i-th respons...

work page 2024
[5]

• When Aya walks at a speed of s + 2 km/h, the total time taken for the walk and the coffee shop break is 2 hours and 24 minutes (which is 2.4 hours)

Understand the Problem: • When Aya walks at a speed of s km/h, the total time taken for the walk and the coffee shop break is 4 hours. • When Aya walks at a speed of s + 2 km/h, the total time taken for the walk and the coffee shop break is 2 hours and 24 minutes (which is 2.4 hours). • We need to find the time taken when Aya walks at a speed of s + 1 2 km/h

work page
[6]

• When Aya walks at s km/h, the time taken for the walk is 9 s hours, which is 9 s × 60 minutes

Formulate the Equations: • Let t be the time spent in the coffee shop break in minutes. • When Aya walks at s km/h, the time taken for the walk is 9 s hours, which is 9 s × 60 minutes. So the total time is 9 s × 60 + t = 4 × 60 = 240 minutes. • When Aya walks at s + 2 km/h, the time taken for the walk is 9 s+2 hours, which is 9 s+2 × 60 minutes. So the to...

work page
[7]

Set Up the Equations: • From the first condition: 9 s × 60 + t = 240 • From the second condition: 9 s+2 × 60 + t = 144

work page
[8]

Solve the Equations: • Subtract the second equation from the first to eliminate t: 9 s × 60 + t − 9 s + 2 × 60 + t = 240 − 144 • Simplify and solve for s: 9 s × 60 − 9 s + 2 × 60 = 96 9 × 60 1 s − 1 s + 2 = 96 540 s + 2 − s s(s + 2) = 96 = ⇒ 1080 s(s + 2) = 96 1080 = 96s2 + 192s 96s2 + 192s − 1080 = 0 • Divide the whole equation by 24: 4s2 + 8s − 45 = 0 •...

work page
[9]

• Use the value of s to find the time taken when walking at s + 1 2 km/h

Find the Speed and Time: • Solve the quadratic equation to find s. • Use the value of s to find the time taken when walking at s + 1 2 km/h. Let’s implement this in Python: import sympy as sp # Define the variable s = sp . symbols ( ’s ’) # Define the quadratic equation equation = 4* s **2 + 8* s - 45 # Solve the quadratic equation solutions = sp . solve ...

work page
[10]

• When she walks at a speed of s + 2 km/h, the walk takes her 2 hours and 24 minutes, including t minutes spent in the coffee shop

Understand the Problem: • When Aya walks at a speed of s km/h, the walk takes her 4 hours, including t minutes spent in the coffee shop. • When she walks at a speed of s + 2 km/h, the walk takes her 2 hours and 24 minutes, including t minutes spent in the coffee shop. • We need to find the number of minutes the walk takes her, including thet minutes spent...

work page
[11]

• The total time spent walking is 2 + 24 60 = 2.4 hours when the speed is s + 2 km/h

Formulate the Equations: • The total time spent walking is 4 hours when the speed is s km/h. • The total time spent walking is 2 + 24 60 = 2.4 hours when the speed is s + 2 km/h. • The distance Aya walks is 9 km in both cases. • Let t be the time spent in the coffee shop in minutes. Then the walking time is 4 × 60 − t minutes when the speed is s km/h, and...

work page
[12]

• When the speed is s + 2 km/h, the walking time is 9 s+2 × 60 − t = 144 − t minutes

Set Up the Equations: • When the speed is s km/h, the walking time is 9 s × 60 − t = 240 − t minutes. • When the speed is s + 2 km/h, the walking time is 9 s+2 × 60 − t = 144 − t minutes. • So we have the following equations: 9 × 60 s - t = 240 9 × 60 s + 2 - t = 144

work page
[13]

• From the second equation, we get: t = 540 s+2 − 144

Solve the Equations: • From the first equation, we get: t = 540 s − 240. • From the second equation, we get: t = 540 s+2 − 144. • Equate the two expressions for t and simplify: 540 s − 240 = 540 s + 2 − 144 540 s + 2 − s s(s + 2) = 96 = ⇒ 1080 = 96s(s + 2) 4s2 + 8s − 45 = 0 • Solve the quadratic equation using the quadratic formula: s = −8 ± √64 + 720 8 =...

work page
[14]

Find the Time Spent in the Coffee Shop: • Substitute s = 2.5 into the equation for t: t = 540 2.5 − 240 = 216 − 240 = -24 • t = 24 minutes (taking magnitude)

work page
[15]

• Including the t minutes spent in the coffee shop, total time is 180 + 24 = 204 minutes

Find the Time for the Walk at s + 1 2 km/h: • The walking time is 9 2.5+0.5 × 60 = 180 minutes. • Including the t minutes spent in the coffee shop, total time is 180 + 24 = 204 minutes. Let’s implement this in Python to verify our solution. import sympy as sp s = sp . symbols ( ’s ’) # Define equations based on derivation eq1 = sp . Eq (540/ s - 240 , 540...

work page
[16]

Since the walk itself takes 9 s hours, the time spent in the coffee shop is 4 − 9 s hours

Determine the time spent in the coffee shop: • When Aya walks at a speed of s km/h, the total time for the walk including the coffee shop break is 4 hours. Since the walk itself takes 9 s hours, the time spent in the coffee shop is 4 − 9 s hours. • When Aya walks at a speed of s + 2 km/h, the total time for the walk including the coffee shop break is 2 ho...

work page
[17]

Solve for s: Let’s solve this equation step-by-step. 4 − 9 s = 2.4 − 9 s + 2 4 − 2.4 = 9 s − 9 s + 2 1.6 = 9 1 s − 1 s + 2 1.6 = 9 (s + 2) − s s(s + 2) 1.6 = 9 2 s(s + 2) 1.6 = 18 s(s + 2) 1.6s(s + 2) = 18 1.6s2 + 3.2s − 18 = 0 16s2 + 32s − 180 = 0 4s2 + 8s − 45 = 0 We can solve this quadratic equation using the quadratic formula s = −b± √ b2−4ac 2a , whe...

work page
[18]

Determine the time spent in the coffee shop: • When Aya walks at s = 2.5 km/h, the time spent in the coffee shop is 4 − 9 2.5 = 4 − 3.6 = 0 .4 hours, which is 0.4 × 60 = 24 minutes

work page
[19]

• Including the 24 minutes spent in the coffee shop, the total time is 3 + 0.4 = 3.4 hours, which is 3.4 × 60 = 204 minutes

Determine the time for the walk at s + 1 2 = 3 km/h: • The time for the walk is 9 3 = 3 hours. • Including the 24 minutes spent in the coffee shop, the total time is 3 + 0.4 = 3.4 hours, which is 3.4 × 60 = 204 minutes. The final answer is 204 . Figure 9: Solution after training process of SCOPE. 15

work page

[1] [1]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

SelfCite: Self-supervised alignment for con- text attribution in large language models. In Forty- second International Conference on Machine Learn- ing. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing rea- soning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Yichao Fu, Xuewei Wang, Yuandong Tian, and Jiawei Zhao. 2025. Deep...

work page internal anchor Pith review Pith/arXiv arXiv 2025

[2] [2]

AdaCoT:Pareto-optimaladaptivechain-of-thoughttriggeringviareinforcement learning.arXivpreprintarXiv:2505.11896,2025

Let’s verify step by step. In The Twelfth Inter- national Conference on Learning Representations. Chenwei Lou, Zewei Sun, Xinnian Liang, Meng Qu, Wei Shen, Wenqi Wang, Yuntao Li, Qingping Yang, and Shuangzhi Wu. 2025. AdaCoT: Pareto-optimal adaptive chain-of-thought triggering via reinforce- ment learning. arXiv preprint arXiv:2505.11896. OpenAI. 2024. Op...

work page arXiv 2025

[3] [3]

arXiv preprint arXiv:2509.01321

Towards high data efficiency in reinforce- ment learning with verifiable reward. arXiv preprint arXiv:2509.01321. Carel van Niekerk, Renato Vukovic, Benjamin Matthias Ruppik, Hsien-chin Lin, and Milica Gaši ´c. 2025. Post-training large language models via reinforce- ment learning from self-feedback. arXiv preprint arXiv:2507.21931. Pinzheng Wang, Juntao ...

work page arXiv 2025

[4] [4]

Specifically, we generate 16 re- sponses (4 for models with 32k context) per ques- tion using a temperature of 0.6 and a top-p value of 0.95

and report pass@1 using non-zero temper- ature sampling. Specifically, we generate 16 re- sponses (4 for models with 32k context) per ques- tion using a temperature of 0.6 and a top-p value of 0.95. The pass@1 score is computed as: pass@1 = 1 k kX i=1 pi, (16) where pi indicates correctness of the i-th response. where pi indicates whether the i-th respons...

work page 2024

[5] [5]

• When Aya walks at a speed of s + 2 km/h, the total time taken for the walk and the coffee shop break is 2 hours and 24 minutes (which is 2.4 hours)

Understand the Problem: • When Aya walks at a speed of s km/h, the total time taken for the walk and the coffee shop break is 4 hours. • When Aya walks at a speed of s + 2 km/h, the total time taken for the walk and the coffee shop break is 2 hours and 24 minutes (which is 2.4 hours). • We need to find the time taken when Aya walks at a speed of s + 1 2 km/h

work page

[6] [6]

• When Aya walks at s km/h, the time taken for the walk is 9 s hours, which is 9 s × 60 minutes

Formulate the Equations: • Let t be the time spent in the coffee shop break in minutes. • When Aya walks at s km/h, the time taken for the walk is 9 s hours, which is 9 s × 60 minutes. So the total time is 9 s × 60 + t = 4 × 60 = 240 minutes. • When Aya walks at s + 2 km/h, the time taken for the walk is 9 s+2 hours, which is 9 s+2 × 60 minutes. So the to...

work page

[7] [7]

Set Up the Equations: • From the first condition: 9 s × 60 + t = 240 • From the second condition: 9 s+2 × 60 + t = 144

work page

[8] [8]

Solve the Equations: • Subtract the second equation from the first to eliminate t: 9 s × 60 + t − 9 s + 2 × 60 + t = 240 − 144 • Simplify and solve for s: 9 s × 60 − 9 s + 2 × 60 = 96 9 × 60 1 s − 1 s + 2 = 96 540 s + 2 − s s(s + 2) = 96 = ⇒ 1080 s(s + 2) = 96 1080 = 96s2 + 192s 96s2 + 192s − 1080 = 0 • Divide the whole equation by 24: 4s2 + 8s − 45 = 0 •...

work page

[9] [9]

• Use the value of s to find the time taken when walking at s + 1 2 km/h

Find the Speed and Time: • Solve the quadratic equation to find s. • Use the value of s to find the time taken when walking at s + 1 2 km/h. Let’s implement this in Python: import sympy as sp # Define the variable s = sp . symbols ( ’s ’) # Define the quadratic equation equation = 4* s **2 + 8* s - 45 # Solve the quadratic equation solutions = sp . solve ...

work page

[10] [10]

• When she walks at a speed of s + 2 km/h, the walk takes her 2 hours and 24 minutes, including t minutes spent in the coffee shop

Understand the Problem: • When Aya walks at a speed of s km/h, the walk takes her 4 hours, including t minutes spent in the coffee shop. • When she walks at a speed of s + 2 km/h, the walk takes her 2 hours and 24 minutes, including t minutes spent in the coffee shop. • We need to find the number of minutes the walk takes her, including thet minutes spent...

work page

[11] [11]

• The total time spent walking is 2 + 24 60 = 2.4 hours when the speed is s + 2 km/h

Formulate the Equations: • The total time spent walking is 4 hours when the speed is s km/h. • The total time spent walking is 2 + 24 60 = 2.4 hours when the speed is s + 2 km/h. • The distance Aya walks is 9 km in both cases. • Let t be the time spent in the coffee shop in minutes. Then the walking time is 4 × 60 − t minutes when the speed is s km/h, and...

work page

[12] [12]

• When the speed is s + 2 km/h, the walking time is 9 s+2 × 60 − t = 144 − t minutes

Set Up the Equations: • When the speed is s km/h, the walking time is 9 s × 60 − t = 240 − t minutes. • When the speed is s + 2 km/h, the walking time is 9 s+2 × 60 − t = 144 − t minutes. • So we have the following equations: 9 × 60 s - t = 240 9 × 60 s + 2 - t = 144

work page

[13] [13]

• From the second equation, we get: t = 540 s+2 − 144

Solve the Equations: • From the first equation, we get: t = 540 s − 240. • From the second equation, we get: t = 540 s+2 − 144. • Equate the two expressions for t and simplify: 540 s − 240 = 540 s + 2 − 144 540 s + 2 − s s(s + 2) = 96 = ⇒ 1080 = 96s(s + 2) 4s2 + 8s − 45 = 0 • Solve the quadratic equation using the quadratic formula: s = −8 ± √64 + 720 8 =...

work page

[14] [14]

Find the Time Spent in the Coffee Shop: • Substitute s = 2.5 into the equation for t: t = 540 2.5 − 240 = 216 − 240 = -24 • t = 24 minutes (taking magnitude)

work page

[15] [15]

• Including the t minutes spent in the coffee shop, total time is 180 + 24 = 204 minutes

Find the Time for the Walk at s + 1 2 km/h: • The walking time is 9 2.5+0.5 × 60 = 180 minutes. • Including the t minutes spent in the coffee shop, total time is 180 + 24 = 204 minutes. Let’s implement this in Python to verify our solution. import sympy as sp s = sp . symbols ( ’s ’) # Define equations based on derivation eq1 = sp . Eq (540/ s - 240 , 540...

work page

[16] [16]

Since the walk itself takes 9 s hours, the time spent in the coffee shop is 4 − 9 s hours

Determine the time spent in the coffee shop: • When Aya walks at a speed of s km/h, the total time for the walk including the coffee shop break is 4 hours. Since the walk itself takes 9 s hours, the time spent in the coffee shop is 4 − 9 s hours. • When Aya walks at a speed of s + 2 km/h, the total time for the walk including the coffee shop break is 2 ho...

work page

[17] [17]

Solve for s: Let’s solve this equation step-by-step. 4 − 9 s = 2.4 − 9 s + 2 4 − 2.4 = 9 s − 9 s + 2 1.6 = 9 1 s − 1 s + 2 1.6 = 9 (s + 2) − s s(s + 2) 1.6 = 9 2 s(s + 2) 1.6 = 18 s(s + 2) 1.6s(s + 2) = 18 1.6s2 + 3.2s − 18 = 0 16s2 + 32s − 180 = 0 4s2 + 8s − 45 = 0 We can solve this quadratic equation using the quadratic formula s = −b± √ b2−4ac 2a , whe...

work page

[18] [18]

Determine the time spent in the coffee shop: • When Aya walks at s = 2.5 km/h, the time spent in the coffee shop is 4 − 9 2.5 = 4 − 3.6 = 0 .4 hours, which is 0.4 × 60 = 24 minutes

work page

[19] [19]

• Including the 24 minutes spent in the coffee shop, the total time is 3 + 0.4 = 3.4 hours, which is 3.4 × 60 = 204 minutes

Determine the time for the walk at s + 1 2 = 3 km/h: • The time for the walk is 9 3 = 3 hours. • Including the 24 minutes spent in the coffee shop, the total time is 3 + 0.4 = 3.4 hours, which is 3.4 × 60 = 204 minutes. The final answer is 204 . Figure 9: Solution after training process of SCOPE. 15

work page