Pith. sign in

REVIEW 5 major objections 6 minor 22 references

Test-Time Scaling with Reflective Generative Model

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A 32B language model can both generate and score its own reasoning traces, matching o3-mini with only 53M extra parameters.

desk verdict A genuinely useful shared-backbone PRM that reportedly matches o3-mini, but the absent decontamination analysis leaves the headline comparison unverified. read the letter →

arxiv 2507.01951 v2 pith:SQWQORPK submitted 2025-07-02 cs.LG cs.CL

classification cs.LGcs.CL
keywords test-timescalingprocessrewardmodelself-supervisedlearningreasoningtrajectoryselectionbest-of-nsamplinglargelanguagemodelsGRPOreflectivegenerative
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single 32B language model can both generate and score its own reasoning traces, and that this reflective generative form closes most of the gap to OpenAI's o3-mini without process-level annotations. The authors add a 53M-parameter scoring head on top of a shared backbone, train it with outcome-only supervision, then use it to pick the best of k sampled thinking trajectories at test time. On AIME24, AIME25, LiveCodeBench, and C-Eval, the high-effort mode reaches 85.2, 73.6, 64.2, and 89.7, which they read as outperforming listed open-source models and matching o3-mini-medium. If true, test-time scaling becomes a cheap add-on to an existing policy rather than a separate large verifier.

What carries the argument

The load-bearing object is the Self-supervised Process Reward Model (SPRM), a binary classifier of two linear layers and a dropout layer attached to the shared backbone at the second-to-last transformer layer. For each step-token, a token containing '.\n\n', the head outputs a process score, and the trajectory score is the geometric mean of step scores; inference selects the trajectory with the highest score. The SPR Loss in Eq. 6 does the training work: it back-propagates only through steps whose current score is consistent with whether the final answer was correct, so the head is asked to amplify distinctions it already makes rather than to imitate noisy outcome labels on every step. This mechanism is what lets one network predict and select trajectories with only 53M extra parameters.

What would settle it

Take a trained MetaStone-S1 and score reasoning traces whose final answers are correct but that contain one known logical or arithmetic error injected at a specific step; if the SPRM does not assign those error steps reliably lower scores than clean steps across many traces, then the reported Best-of-N gains are not explained by learned process discrimination.

Watch

Extended reading notes

Core claim

The central claim is that a process reward model can be folded into the policy itself and learned without step labels, and that the resulting selector is what makes test-time scaling work. The paper calls this the Reflective Generative Form: the same network generates k reasoning trajectories, segments each one at tokens containing '.\n\n', scores every step with a lightweight head over the second-to-last layer's hidden states, and aggregates step scores by geometric mean to choose the trajectory that then produces the final answer. Training couples GRPO on the policy with a self-supervised process reward loss whose per-step weight is 1 only when the head's own score already agrees with the final-answer correctness, a dynamic filter meant to suppress noisy supervision. The authors report that this yields monotone gains as k grows from 2 to 32 and that the 32B model's high mode is comparable to o3-mini-medium on the four benchmarks, with only 53M extra parameters.

Load-bearing premise

The method assumes that the SPRM head's early step scores are informative enough for the Eq. 6 self-filter to keep useful steps and discard noise; if the head starts near random, the filter can reinforce its own errors instead of learning process quality.

Editorial extensions

If this is right

  • With the reflective form, Best-of-N sampling using the shared SPRM outperforms a separately trained 72B outcome reward model and a 72B process reward model as verifiers, while adding only 5M, 26M, or 53M parameters at the 1.5B, 7B, and 32B scales.
  • Increasing the number of sampled trajectories from k=2 to k=32 raises AIME24 accuracy from 44.0 to 57.9 at 1.5B, from 60.7 to 70.2 at 7B, and from 82.0 to 85.2 at 32B.
  • SPRM's step scores transfer to LiveCodeBench without task-specific fine-tuning, which the paper takes as evidence that the learned notion of a good step is not math-specific.
  • The same SPRM can guide Monte Carlo Tree Search: raising the search-token budget from 0 to 160k lifts AIME24 accuracy from 39.3 to 52.8 on the 1.5B model.
  • Final performance grows roughly with the logarithm of the test-time compute budget, so within the studied range, exponentially more compute buys linear gains in accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the self-referential filter in Eq. 6 makes training a bootstrapping procedure, so the reported 'aha moment' at 10/20/50 steps is a testable claim; tracking early-training agreement between SPRM scores and final correctness would show whether the filter amplifies signal or noise.
  • Editorial inference: because the SPRM head reads the shared backbone's second-to-last layer, the method implies that reasoning quality is encoded in the policy's internal representations; if so, the same head could be adapted to other selection tasks such as tool choice or self-correction with minimal retraining.
  • Editorial inference: the geometric-mean aggregation means a single very low-scoring step vetoes a trajectory, a strong prior that any logical slip is fatal; this may suit math and code but could be too strict for open-ended or creative tasks.
  • Editorial inference: the reported parity with o3-mini is specific to the four benchmarks and to the k=32 effort mode; it does not by itself establish parity on other reasoning distributions or under matched sampling budgets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The manuscript introduces MetaStone-S1, a family of reasoning LLMs (1.5B, 7B, 32B) initialized from DeepSeek-R1-Distill and QWQ backbones, augmented by a shared-backbone Self-supervised Process Reward Model (SPRM) head of 5M/26M/53M parameters. Training uses GRPO for the policy and a new SPR loss (Eq. 6) that supervises step scores with final-answer correctness, filtered by the model's own current step scores. At inference, k sampled think-trajectories (k=2, 8, 32 for low, medium, high effort) are scored by the SPRM via geometric mean and the best is selected. The paper reports consistent improvements over the corresponding baselines on AIME24, AIME25, LiveCodeBench, and C-Eval, ablates SPRM against 72B ORM/PRM reward models, studies scaling laws and an aha moment, and extends the approach to a step-level MCTS variant. The headline claim is that MetaStone-S1-32B-high is comparable to OpenAI o3-mini-medium.

Significance. If the benchmark results are valid, the central contribution is valuable: a 53M-parameter shared-backbone scoring head trained without process annotations can provide effective Best-of-N selection, outperform 72B reward models, and bring a 32B open model close to o3-mini-medium. Strengths include the external final-answer evaluation for the main claim, the SPRM ablation against two 72B reward baselines, the repeated 64-run evaluation protocol, and the public release of code and models. The main threats to significance are the absence of a contamination analysis for the training corpus, the incompleteness of the o3-mini comparison, and the under-analyzed self-referential nature of the SPR loss, each of which is load-bearing for a specific part of the paper's claims.

major comments (5)
  1. [Section 5.1 and Table 1] The training set is described as built from NuminaMath, OpenR1-Math-220k, DeepScaleR, LIMR, and OREAL-RL, and the headline results are on AIME24, AIME25, LiveCodeBench, and C-Eval. No contamination check is reported: there is no exact-match or n-gram overlap analysis against these evaluation sets, and no temporal filtering for LiveCodeBench. Because public competition-math corpora are known to contain benchmark items, the reported +5.3/+3.1 gains of MetaStone-S1-32B-high over Baseline-32B on AIME24/AIME25 could in part reflect memorization rather than SPRM-guided selection. Since the o3-mini comparison is built directly on these four numbers, this gap is load-bearing. Please report contamination checks (e.g., exact and near-duplicate matching, 10-gram overlap, and date filtering for LiveCodeBench) and, if overlap is found, retrain on a cleaned subset or exclude the contaminated items.
  2. [Section 5.2 and Table 1] The abstract and introduction claim that MetaStone-S1 achieves comparable performance to the OpenAI o3-mini series, but Table 1 includes only o3-mini-medium (marked by an asterisk) and no o3-mini-low or o3-mini-high, and there is no matching of inference budgets or reasoning-effort levels. The claim of series-level comparability is therefore not supported by the evidence presented. Please add the missing o3-mini effort modes with comparable sampling budgets, or explicitly restrict the claim to o3-mini-medium.
  3. [Section 4.2, Eq. (6)] The SPR loss sets w_i=1 only when the SPRM head's own current step score agrees in sign with the final-answer correctness. Early in training, if the head's scores are unreliable, this filter can retain only samples that confirm the head's initial biases, reinforcing initial errors and discarding informative steps. The paper provides no diagnostic of the pseudo-label accuracy or agreement rate over training, and no ablation that varies the filter (e.g., unfiltered BCE, random gating at the same retention rate, or confidence thresholds). This is load-bearing for the internal claim that SPRM learns genuine step-level trajectory selection rather than self-confirmation. Please add such analyses.
  4. [Section 5.3, Fig. 3] The scaling-law section states that beyond 32x the baseline the improvement is slow and therefore focuses on Best-of-32, but Fig. 3 appears to truncate at Best-of-32 and no saturated data points are shown or tabulated. As written, the claim of a positive logarithmic correlation with compute is not supported by the displayed curve, and the saturation claim is not quantifiable. Please include the full curve at least up to the point where saturation is observed, and report the underlying values.
  5. [Section 5.1 and Table 1] The manuscript states that the evaluation is repeated 64 times and averaged, but Table 1 reports only point estimates. Several headline margins are small (for example, +0.8 on LiveCodeBench and +0.3 on C-Eval for MetaStone-S1-32B-high over Baseline-32B), so without confidence intervals or standard deviations it is unclear whether the differences are meaningful. Please report confidence intervals or per-run variances, at least for the 32B models and the o3-mini comparison.
minor comments (6)
  1. [Section 4.2, Eq. (6)] The notation y_i is reused for every step of a trajectory; clarify that y_i is the same in each step and equals the final-answer correctness of the whole trajectory.
  2. [Section 5.1] The evaluation metric is called Pass@1, but for MetaStone-S1 the final answer is produced after selecting one of the k thinking trajectories; clarify whether Pass@1 refers only to the selected trajectory and how the 64 repeats are aggregated.
  3. [Section 5.6 and Fig. 6] The symbol k is used both for the number of candidate trajectories and for the maximum MCTS search tokens; please disambiguate these two quantities.
  4. [Section 5.4] The term aha moment is used without a quantitative definition; specify the criterion used to detect when the score curves for correct and incorrect trajectories diverge.
  5. [References] The citation for Beeching et al. is missing a year and venue, and several non-archival sources (Labs, Zeff, Chollet) should be marked as web/blog references rather than journal or conference citations.
  6. [General] Figure numbering in the full text appears duplicated (Figure 4 appears twice); the figures should be renumbered consecutively.

Circularity Check

2 steps flagged · score 4.0 of 10

Training signal is self-referential and the 'aha moment' is a by-construction phase transition, but the headline o3-mini comparison rests on external benchmarks, so the paper is only partially circular.

  1. self definitional [Section 4.2, Eq. (6)]
    "LSPR = 1/N Σ w_i * BCELoss(Score_i, y_i), where w_i = {1, if y_i =1 & Score_i >0.5; 1, if y_i =0 & Score_i <0.5; 0, others} ... we use the SPRM head's own prediction on each step as the pseudo label and set w_i =1 only if the pseudo label is consistent with the final answer's correctness."

    The SPRM's learning signal is gated by the SPRM's own current score: a step is trained only when Score_i already agrees with the outcome label y_i. Hence the learned scores are self-confirming; the claimed 'process evaluation ability' is, on the filtered subset, the model's initial opinion reinforced by the loss. The final external benchmarks are not tautological, but the internal training evidence for SPRM is circular.

  2. fitted input called prediction [Section 5.4, Fig. 4]
    "However, after a certain number of training steps (e.g., around 10/20/50 steps, 1280/2560/6400 samples for MetaStone-S1-1.5B/7B/32B, respectively), we observe a distinct 'aha moment' point where the optimization trends of different reasoning trajectories begin to diverge."

    This divergence is the direct effect of Eq. (6): gradient is applied only after the SPRM score crosses the 0.5 threshold in the direction of the outcome label, and then the loss pushes the score further in that direction. The 'aha moment' is therefore a phase transition in the self-filtering loss, not independent evidence that the model has learned to evaluate reasoning content. Reporting it as an emergent discovery presents the training objective's intended effect as a prediction.

full rationale

The paper's central claim—MetaStone-S1-32B-high is comparable to o3-mini-medium—is evaluated on AIME24/25, LiveCodeBench, and C-Eval, all external benchmarks, so the headline comparison is not circular. The main circularity risk is internal: Eq. (6) makes the SPRM's training signal depend on the SPRM's own current scores, and Sec. 5.4's 'aha moment' is the resulting self-reinforcing phase transition, not an independent discovery. This does not make the external numbers tautological, but it weakens the mechanistic claim that SPRM acquires genuine step-level understanding. Separately, the paper reports no decontamination against AIME24/25, LiveCodeBench, or C-Eval for the 40k training set assembled from NuminaMath, OpenR1-Math-220k, DeepScaleR, LIMR, and OREAL-RL; that is a data-integrity gap rather than a circularity and should be assessed as such. Overall, partial circularity in the internal training narrative, but the benchmark comparison retains independent content: score 4.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central method rests on a small number of hand-chosen design decisions and domain assumptions: the step segmentation rule, the choice of the second-to-last layer for scoring, the 0.5 pseudo-label threshold, and the self-referential SPR loss. No new physical entities are introduced; the SPRM head is a learned model component with external benchmark evidence. The scaling-law and aha-moment analyses are qualitative and not fitted, so they contribute little to the ledger.

free parameters (3)
  • candidate count k (Best-of-N budget) = 2, 8, 32
    Chosen by hand to define low/medium/high effort modes; directly controls test-time compute and is central to the performance claims (Section 5.1, Eq.7).
  • SPR loss pseudo-label threshold = 0.5
    In Eq.6, a step contributes to the loss only when Score_i and final correctness agree, using the classifier's 0.5 boundary; no sensitivity analysis is given.
  • MCTS budget = max 160k tokens, 4 children, 1024 tokens per child
    Hand-set in Section 5.6; these choices affect the MCTS extension result but not the main comparison.
assumptions (5)
  • domain assumption Step tokens are defined as occurrences of a period followed by two newline characters, and these delimit meaningful reasoning steps.
    Section 4.1: the entire step-level scoring pipeline depends on this segmentation heuristic.
  • domain assumption Second-to-last-layer hidden states at step-token positions encode step quality.
    Section 4.1: the SPRM head reads only these representations to produce process scores.
  • ad hoc to paper Final answer correctness, filtered by the model's own step predictions, is a usable supervisory signal for process quality.
    Section 4.2, Eq.6: this is the core self-supervised assumption and the most fragile one.
  • domain assumption Sharing the backbone between policy and reward model does not degrade reward quality.
    Section 4.1: the unified interface is motivated by parameter efficiency; Table 2 is the only empirical check.
  • domain assumption Computation budget C = Params_policy x tokens_infer captures test-time scaling.
    Section 5.3: follows Snell et al., but the product conflates model size and search budget and is not derived.
invented entities (1)
  • Self-supervised Process Reward Model (SPRM) head independent evidence
    purpose: Scores each reasoning step from shared-backbone hidden states, enabling Best-of-N trajectory selection without a separate reward model.
    The head is evaluated on AIME24/25, LiveCodeBench, and C-Eval and compared against 72B reward models (Table 2), so it has a falsifiable external handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Test-Time Scaling with Reflective Generative Model." pith.science (2026). https://pith.science/paper/SQWQORPK

@misc{pith2026250701951,
  author       = {Pith},
  title        = {Pith review of: Test-Time Scaling with Reflective Generative Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQWQORPK}},
  note         = {Machine review of arXiv:2507.01951}
}
read the original abstract

We introduce our first reflective generative model MetaStone-S1, which obtains OpenAI o3-mini's performance via the new Reflective Generative Form. The new form focuses on high-quality reasoning trajectory selection and contains two novelties: 1) A unified interface for policy and process reward model: we share the backbone network and use task-specific heads for reasoning trajectory predicting and scoring respectively, introducing only 53M extra parameters for trajectory scoring. 2) Eliminating the reliance on process-level annotation: we provide a self-supervised process reward model, which can directly learn the high-quality reasoning trajectory selection from the outcome reward. Equipped with the reflective generative form, MetaStone-S1 is naturally suitable for test-time scaling, and we provide three reasoning effort modes (low, medium, and high) based on the controllable thinking length. Experiments demonstrate that our MetaStone-S1 achieves comparable performance to OpenAI o3-mini's series with only 32B parameter size. To support the research community, we have open-sourced MetaStone-S1 at https://github.com/MetaStone-AI/MetaStone-S1.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 3 canonical work pages

  1. [3]

    Accessed: 2024-12-20

    URL https://arcprize .org/blog/oai-o3-pub-breakthrough#:~:text=o3%27s%20improvement%20over %20the%20GPT,progress%20is%20about%20new%20ideas. Accessed: 2024-12-20. T. GLM, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Rojas, G. Feng, H. Zhao, H. Lai, H. Yu, H. Wang, J. Sun, J. Zhang, J. Cheng, J. Gui, J. Tang, J. Zhang, J. Li, L. Zhao, L. Wu, L. Zhong, M. ...

  2. [4]

    X. Guan, L. L. Zhang, Y. Liu, N. Shang, Y. Sun, Y. Zhu, F. Yang, and M. Yang. rstar-math: Small llms can master math reasoning with self-evolved deep thinking. arXiv preprint arXiv:2501.04519,

  3. [5]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P . Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

  4. [7]

    N. Jain, K. Han, A. Gu, W.-D. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974,

  5. [8]

    M. Jin, Q. Yu, D. Shu, H. Zhao, W. Hua, Y. Meng, Y. Zhang, and M. Du. The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.04925,

  6. [9]

    Accessed: 2025-02-18

    URL https://labs.ada line.ai/p/inside-reasoning-models-openai-o3/. Accessed: 2025-02-18. J. Li, E. Beeching, L. Tunstall, B. Lipkin, R. Soletskyi, S. Huang, K. Rasul, L. Yu, A. Q. Jiang, Z. Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9,

  7. [10]

    X. Li, H. Zou, and P . Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502.11886,

  8. [11]

    15 R. Liu, J. Gao, J. Zhao, K. Zhang, X. Li, B. Qi, W. Ouyang, and B. Zhou. Can 1b llm surpass 405b llm? rethinking compute-optimal test-time scaling. arXiv preprint arXiv:2502.06703,

Show all 22 references
  1. [12]

    L. Luo, Y. Liu, R. Liu, S. Phatale, M. Guo, H. Lara, Y. Li, L. Shu, Y. Zhu, L. Meng, et al. Improve mathematical reasoning in language models by automated process supervision.arXiv preprint arXiv:2406.06592,

  2. [13]

    Notion Blog. C. Lyu, S. Gao, Y. Gu, W. Zhang, J. Gao, K. Liu, Z. Wang, S. Li, Q. Zhao, H. Huang, et al. Exploring the limit of outcome reward for learning mathematical reasoning. arXiv preprint arXiv:2502.06781,

  3. [14]

    Muennighoff, Z

    N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P . Liang, E. Candès, and T. Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393,

  4. [15]

    Accessed: 2025-04-13

    URL https://openai.com/index/learnin g-to-reason-with-llms/. Accessed: 2025-04-13. OpenAI. Openai o3-mini evaluation,

  5. [16]

    Accessed: 2025-01-31

    URL https://openai.com/index/openai-o 3-mini/. Accessed: 2025-01-31. Z. Shao, P . Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

  6. [17]

    Snell, J

    C. Snell, J. Lee, K. Xu, and A. Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314,

  7. [18]

    X. Tan, T. Yao, C. Qu, B. Li, M. Yang, D. Lu, H. Wang, X. Qiu, W. Chu, Y. Xu, et al. Aurora: Automated training framework of universal process reward models via ensemble prompting and reverse verification. arXiv preprint arXiv:2502.11520,

  8. [19]

    URL https: //qwenlm.github.io/blog/qwq-32b/. P . Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. arXiv preprint arXiv:2312.08935,

  9. [20]

    E. Yeo, Y. Tong, M. Niu, G. Neubig, and X. Yue. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373,

  10. [21]

    Accessed: 2024-12-23

    URL https://techcrunch.com/2024/12/23/openais-o3-suggests-ai-models-are-s caling-in-new-ways-but-so-are-the-costs/?guccounter=1#:~:text=AI%20mo dels%20were%20showing%20diminishing,with%20drawbacks%20of%20its%20own . Accessed: 2024-12-23. 16 Z. Zeng, Q. Cheng, Z. Yin, Y. Zhou, ...

  11. [22]

    Zhang, C

    Z. Zhang, C. Zheng, Y. Wu, B. Zhang, R. Lin, B. Yu, D. Liu, J. Zhou, and J. Lin. The lessons of de- veloping process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301,

  12. [2023]

    Jaech, A

    A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720,

  13. [2024]

    14 Q. Chen, L. Qin, J. Wang, J. Zhou, and W. Che. Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought. Advances in Neural Information Processing Systems, 37:54872–54904, 2024a. X. Chen, J. Xu, T. Liang, Z. He, J. Pang...

  14. [2025]

    URL https://artofproblemsolving.com/wi ki/index.php/AIME_Problems_and_Solutions. E. Beeching, L. Tunstall, and S. Rush. Scaling test-time compute with open models. URL https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-tim e-compute. B. Brown, J. Juravsky, R. Eh...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.