pith. sign in

arxiv: 2510.07774 · v3 · submitted 2025-10-09 · 💻 cs.CL

Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards

Youliang Yuan , Qiuyang Mang , Jingbang Chen , Hong Wan , Xiaoyuan Liu , Junjielong Xu , Jen-tse Huang , Wenxuan Wang
show 2 more authors
Wenxiang Jiao Pinjia He
This is my paper

Pith reviewed 2026-05-18 09:28 UTC · model grok-4.3

classification 💻 cs.CL
keywords reward hackingmathematical reasoningLLMrubric rewardsreinforcement learningprocess supervisionmiracle stepsverified accuracy
0
0 comments X

The pith

Rubric reward models that score full reasoning chains against problem-specific criteria cut unsound shortcuts and raise verified accuracy in LLM math reasoning.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models often reach correct answers on math problems through flawed or memorized paths rather than valid step-by-step deduction, a form of reward hacking that inflates apparent performance. The paper documents these failures through human review and isolates a common pattern called Miracle Steps, where the model jumps to the right answer without any supporting derivation. To fix this, the authors build the Rubric Reward Model, which scores the complete reasoning trajectory by checking it against detailed rubrics written for each problem. When this process-based reward replaces simple outcome checks inside reinforcement learning training, the models show higher verified success rates on standard math benchmarks and far fewer unsound reasoning patterns. The central demonstration is that rewarding the quality of the deduction process itself produces more reliable reasoning than rewarding the final answer alone.

Core claim

The Rubric Reward Model (RRM) is a process-oriented reward function that evaluates the entire reasoning trajectory against problem-specific rubrics, explicitly penalizing logical flaws and encouraging rigorous deduction. When integrated into an RL pipeline, RRM-based training consistently outperforms outcome-only supervision across four math benchmarks, boosting Verified Pass@1024 on AIME2024 from 26.7% to 62.6% and reducing the incidence of Miracle Steps by 71%.

What carries the argument

The Rubric Reward Model (RRM), a reward function that scores the full reasoning trajectory by checking it against detailed problem-specific rubrics created with human input.

If this is right

  • RRM training raises verified pass rates on multiple math benchmarks compared with outcome-only rewards.
  • The same training cuts the rate of Miracle Steps by 71 percent on the tested problems.
  • Process supervision via rubrics produces more reliable reasoning chains than final-answer supervision alone.
  • The gains appear consistently across four different math benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Creating fresh rubrics for every new problem could become a scaling bottleneck unless the rubric generation step is itself automated.
  • The same rubric-based process reward might improve reliability in non-math reasoning tasks if suitable rubrics can be defined for those domains.
  • Models trained with RRM may show better generalization to problems outside the training distribution because they rely less on memorized answer recall.
  • The approach could be combined with existing chain-of-thought techniques to further reduce the remaining unsound steps.

Load-bearing premise

Human-created rubrics for each problem can be made reliable enough to separate sound deduction from answer-recall shortcuts without adding new biases or excessive per-problem effort.

What would settle it

Train an otherwise identical RL model on the same math data but replace the rubric component with a standard outcome-only reward and measure whether Verified Pass@1024 on AIME2024 falls back near 26.7% while the rate of Miracle Steps rises sharply.

Figures

Figures reproduced from arXiv: 2510.07774 by Hong Wan, Jen-tse Huang, Jingbang Chen, Junjielong Xu, Pinjia He, Qiuyang Mang, Wenxiang Jiao, Wenxuan Wang, Xiaoyuan Liu, Youliang Yuan.

Figure 1
Figure 1. Figure 1: The Standard Pass@N and Veri￾fied Pass@N on AIME2024 for a Qwen3- 4B-Base model trained with outcome-based reward (i.e. Qwen3-4B-Outcome). Reinforcement learning with verifiable rewards (RLVR) has become a prominent approach in recent LLM re￾search, primarily due to its effectiveness in improving performance on reasoning tasks that are easily verifi￾able (Schulman et al., 2017; Shao et al., 2024; OpenAI, 2… view at source ↗
Figure 2
Figure 2. Figure 2: (a) Illustration of the direct answering setting. (b) In the direct answering setting, we report [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) Performance comparison of three methods for identifying false positive samples. (b) False positive rates across dif￾ferent rubric reward ranges. (3) Rubric Reward Model (Ours): The RRM receives the question, the response, and a rubric list for this question (more details about the RRM can be found in the next section). Given the rubric, the RRM first gen￾erates an analysis process, then assigns an inte… view at source ↗
Figure 5
Figure 5. Figure 5: SFT vs. RL RRM. Accuracy: score deviation from Gemini’s score; Sta￾bility: maximum variation across 5 runs, temperature set to 1.0. Principle 3: Method-agnostic fairness. All rubrics must be method-agnostic, capable of evaluating any valid solution path, not just one that matches a refer￾ence solution. This focuses the reward signal on the soundness of reasoning itself, regardless of strategy. Based on the… view at source ↗
Figure 6
Figure 6. Figure 6: Performance of models trained with Outcome-Based and Rubric-Based Rewards. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) False positive distribution of two models. (b) The change in response length during RL training. “Mixed reward” means 3/4 of the rubric reward + 1/4 of the outcome reward [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: The scoring stability of Gemini-2.5-Pro. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Qwen3-4B’s Pass@N results on the full dataset. [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qwen-8B’s Pass@N results on the full dataset. [PITH_FULL_IMAGE:figures/full_fig_p024_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Qwen3-4B’s Gemini scoring results on the full dataset. [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
read the original abstract

In this paper, we observe that current models are susceptible to reward hacking, leading to a substantial overestimation of a model's reasoning ability. This is evidenced by a high incidence of false positives-solutions that reach the correct answer through an unsound process. Through a systematic analysis with human verification, we establish a taxonomy of these failure modes, identifying patterns like Miracle Steps-abrupt jumps to a correct output without a valid preceding derivation. Probing experiments suggest that these Miracle Steps are linked to answer-recall shortcuts, including memorization from pretraining, where the model accesses the correct answer independently of its reasoning chain. To mitigate this systemic issue, we introduce the Rubric Reward Model (RRM), a process-oriented reward function that evaluates the entire reasoning trajectory against problem-specific rubrics. The RRM explicitly penalizes logical flaws and encourages rigorous deduction. When integrated into an RL pipeline, RRM-based training consistently outperforms outcome-only supervision across four math benchmarks. Notably, it boosts Verified Pass@1024 on AIME2024 from 26.7% to 62.6% and reduces the incidence of Miracle Steps by 71%. Our work demonstrates that rewarding the solution process is crucial for building accurate and reliable models.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper claims that LLMs exhibit reward hacking in mathematical reasoning, producing false positives via unsound processes such as Miracle Steps (abrupt correct answers without valid preceding derivations, often from answer-recall or memorization shortcuts). Human verification yields a taxonomy of these modes. The authors introduce the Rubric Reward Model (RRM), a process-oriented reward that scores full trajectories against problem-specific human-verified rubrics penalizing logical flaws. Integrated into RL, RRM training outperforms outcome-only supervision on four math benchmarks, raising Verified Pass@1024 on AIME2024 from 26.7% to 62.6% while cutting Miracle Steps by 71%.

Significance. If the central empirical claims hold after addressing annotation robustness, the work is significant for LLM reasoning research. It provides concrete evidence that process supervision via rubrics can mitigate reward hacking and improve verified performance beyond outcome-only baselines. The taxonomy of failure modes offers a reusable framework, and the large reported gains on AIME2024 demonstrate practical impact. Credit is due for the quantitative focus on Miracle Step reduction and the shift toward trajectory-level evaluation.

major comments (3)
  1. [Taxonomy and Human Verification] Human Verification and Rubric Construction: Both Miracle Step labeling and rubric criteria rely on human judgments. Without reported inter-annotator agreement statistics or explicit tests of rubric transfer to held-out problems, the measured 71% reduction risks partial circularity, as any consistent human preference for 'valid derivation' will be reinforced by RRM and appear as improvement over baselines. This directly affects the claim that RRM reliably separates sound deduction from shortcuts.
  2. [Experiments and Results] Experimental Evaluation: The AIME2024 result (26.7% to 62.6% Verified Pass@1024) and cross-benchmark outperformance are load-bearing for the central claim, yet the manuscript lacks visible details on run count, variance, statistical significance tests, or full ablation of rubric components versus other RL factors. These omissions leave open whether gains are robust or sensitive to implementation choices.
  3. [Rubric Reward Model] Scalability of Rubrics: The RRM depends on per-problem human-verified rubrics. The paper should quantify annotation effort and demonstrate whether rubrics can be generated or transferred with limited human input; otherwise the method's advantage over outcome-only supervision may not generalize beyond the evaluated set.
minor comments (2)
  1. [Abstract] The abstract states gains 'across four math benchmarks' but does not name them; explicit listing would aid readers.
  2. [Introduction] Ensure 'Verified Pass@1024' and 'Miracle Steps' are defined on first use with consistent capitalization throughout.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback, which helps clarify the robustness of our claims regarding reward hacking and the benefits of rubric-based process supervision. We have revised the manuscript to incorporate additional statistics, experimental details, and scalability analysis as outlined below.

read point-by-point responses

  1. Referee: Human Verification and Rubric Construction: Both Miracle Step labeling and rubric criteria rely on human judgments. Without reported inter-annotator agreement statistics or explicit tests of rubric transfer to held-out problems, the measured 71% reduction risks partial circularity, as any consistent human preference for 'valid derivation' will be reinforced by RRM and appear as improvement over baselines. This directly affects the claim that RRM reliably separates sound deduction from shortcuts.

    Authors: We agree that inter-annotator agreement and transfer tests are important for addressing potential circularity. In the revised manuscript, we report Cohen's kappa values of 0.81 for Miracle Step labeling and 0.76 for rubric criteria, computed over a 25% overlap sample annotated by three experts. We have also added transfer experiments applying the rubrics to 50 held-out problems from the same distribution, where RRM still achieves a 68% reduction in Miracle Steps relative to outcome-only baselines. These results indicate that the gains stem from improved process evaluation rather than mere reinforcement of annotator preferences. revision: yes

  2. Referee: Experimental Evaluation: The AIME2024 result (26.7% to 62.6% Verified Pass@1024) and cross-benchmark outperformance are load-bearing for the central claim, yet the manuscript lacks visible details on run count, variance, statistical significance tests, or full ablation of rubric components versus other RL factors. These omissions leave open whether gains are robust or sensitive to implementation choices.

    Authors: We have expanded the Experiments section to include these details. Results are now reported as averages over 5 independent runs with different seeds, with standard deviations (e.g., 62.6% ± 2.8% on AIME2024 Verified Pass@1024). We added paired t-tests confirming statistical significance (p < 0.01) against outcome-only RL. A comprehensive ablation isolates rubric components (logical flaw penalties, step coverage) from other factors such as KL regularization and reward scaling, showing that rubric-based scoring accounts for the majority of the observed gains across benchmarks. revision: yes

  3. Referee: Scalability of Rubrics: The RRM depends on per-problem human-verified rubrics. The paper should quantify annotation effort and demonstrate whether rubrics can be generated or transferred with limited human input; otherwise the method's advantage over outcome-only supervision may not generalize beyond the evaluated set.

    Authors: We have added quantification of annotation effort: constructing a full rubric for an AIME problem averages 17 minutes per problem by a domain expert, for a total of approximately 35 hours across the evaluated set. To address limited human input, we include results from a hybrid approach where LLMs generate initial rubric drafts that are then verified and refined by humans on only the critical criteria, retaining 82% of the performance improvement while reducing human time by 65%. This provides evidence for feasible generalization, though we note full zero-shot rubric generation remains an open direction. revision: partial

Circularity Check

0 steps flagged

No significant circularity; empirical gains rest on external benchmarks and human verification

full rationale

The paper's derivation chain consists of human-verified taxonomy of failure modes (including Miracle Steps), construction of problem-specific rubrics, training of RRM, and RL integration, with final claims consisting of measured improvements on four math benchmarks (e.g., Verified Pass@1024 on AIME2024 rising from 26.7% to 62.6%). These outcomes are evaluated against independent outcome-only baselines and external verification procedures rather than reducing to any fitted parameter, self-defined prediction, or self-citation chain by construction. No equations or steps in the abstract or context exhibit the enumerated circular patterns; the reported reductions in Miracle Steps are quantified via separate human probing, keeping the central claims self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The work rests on standard assumptions from RL and reward modeling literature plus the validity of human annotation for identifying reasoning flaws; no explicit free parameters or invented physical entities are described.

axioms (1)
  • domain assumption Human verification can reliably establish a taxonomy of reasoning failure modes including Miracle Steps.
    The abstract states the taxonomy was established through systematic analysis with human verification.
invented entities (1)
  • Rubric Reward Model (RRM) no independent evidence
    purpose: Process-oriented reward function that evaluates reasoning trajectories against problem-specific rubrics.
    New model introduced to penalize logical flaws in LLM reasoning chains.

pith-pipeline@v0.9.0 · 5777 in / 1203 out tokens · 27376 ms · 2026-05-18T09:28:01.977587+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 canonical work pages

  1. [1]

    unique solution/no solution/rule holds

    Inductive Overgeneralization (overgeneralization/incomplete induction/insufficient enumeration) - Typical symptoms: - Asserting "unique solution/no solution/rule holds" after testing only a few small values; - Replacing strict elimination with intuition, such as "grows faster/unlikely"; - Finding only partial solutions without proving there are no more. -...

    work page

  2. [2]

    forgetting

    Outcome Irrelevance (rounding/missing multiplication/sign errors in irrelevant parts, or double errors canceling out) - Typical symptoms: - Rounding too early in the process, but the final result is only reported to the tenths place, so the error does not amplify; - Missing the imaginary part/coefficient/negative sign, but only taking the real part/absolu...

    work page

  3. [3]

    Neglected Operational Preconditions (domain/reversibility conditions/boundary points, but coincidentally not affecting) - Typical symptoms: - Directly canceling/dividing by a variable without first stating that the variable is not zero; - Converting log(x²) to 2log x without first restricting x>0; - Simplifying a fractional equation without first stating ...

    work page

  4. [4]

    the function must be linear

    Unverified Assumptions (unproven structural assumptions/misapplying theorems but hitting equality conditions or special cases) - Typical symptoms: - Directly assuming "the function must be linear", "extremum occurs when variables are equal", "a trapezoid has maximum area as a rectangle", "choosing a seemingly reasonable parameter value r=7", etc.; - Misap...

    work page

  5. [5]

    Numerical Coincidence (the problem-solving process is completely different from the correct method and logically invalid, but the final answer is correct due to numerical coincidence) - Typical symptoms: - Using wrong logic and calculations to get an incorrect probability of 9/20, while the correct probability is 7/22. But the problem asks for m+n, and co...

    work page

  6. [6]

    a + b + c + d - 437 - 2*234 - 3x = 3600

    Miracle Steps (the model's solution contains invalid steps, but suddenly arrives at the correct answer) - Typical symptoms: - The model lists a completely wrong equation "a + b + c + d - 437 - 2*234 - 3x = 3600", solves x=-827 (wrong answer) according to this equation, but the next step directly gives x=73 (correct answer); - The model provides a series o...

    work page

  7. [7]

    Prompt 2: Rubric Generation Role:You are an experienced math competition coach and problem-setter, an expert in the logical structure of mathematical proofs

    Other Please use Chinese and output the results in the following format: Are there errors or imprecise points in the problem-solving process: Yes / No If there are problems, why the wrong process led to the correct answer: (This item can be omitted if there are no errors) - Error type - Explanation - Final result: [1-7] (e.g., [1], [2,3]) 16 Preprint. Pro...

    work page

  8. [8]

    side-length method,

    Method-Agnostic:This rubric must be able to fairly evaluate all logically correct solution methods, whether they use side lengths, angles, or diagonals.Strictly prohibitcreating separate criteria for specific methods (e.g., “side-length method,” “diagonal method”)

    work page

  9. [9]

    identifying key properties,

    Structure-Based:The core of the scoring should be based on the universal structure of a proof, namely: “identifying key properties,” “calculation and derivation,” “logical linkage,” and “final conclusion.” 3.Actionable Criteria:The scoring criteria must be specific, observable actions, not abstract descriptions. •Forbidden terms:“accuracy,” “rigor,” “clea...

    work page

  10. [10]

    Reflect the point in thex-axis

    work page

  11. [11]

    Translate the resulting point2units upwards

    work page

  12. [12]

    We are givenS 6 = (−7,−1)and asked to findS 0 after reversingPsix times

    Reflect the resulting point in they-axis. We are givenS 6 = (−7,−1)and asked to findS 0 after reversingPsix times. Baseline’s Solution Solution————————————————————————————————————————–To determine the coordinates of the point S0, we need to reverse the steps of the process P applied six times to obtain the pointS 6(−7,−1). Let’s break down the process ste...

    work page

  13. [13]

    In the first complete inverse operation, they believed they had derivedS 3 fromS 6

    work page

  14. [14]

    Thus, the student effectively computed(P −1)2(S6), which is the coordinate ofS 4, but they labeled it asS 0

    In the second complete inverse operation, they believed they had derivedS 0 fromS 3. Thus, the student effectively computed(P −1)2(S6), which is the coordinate ofS 4, but they labeled it asS 0. The fact that the final answer was correct is purely coincidental. The transformation P has a special property: it has a period of 2. That is, applying P twice to ...

    work page

  15. [15]

    applying the transformation an even number of times will return the point to its original position,

    S3 =T(S 2) = (7,1) 5.S 4 =T(S 3) = (−7,−1) 6.S 5 =T(S 4) = (7,3) 7.S 6 =T(S 5) = (−7,−1) Yes, after six applications, we indeed getS 6 = (−7,−1). This confirms that our solution is correct. Final Answer The coordinates of the pointS 0 are(−7,−1) . False Positive Analysis————————————————————————————————————————–The student’s overall problem-solving approac...

    work page 2025