{"id":"6d4344f0-a1fe-4902-8bbc-e499d1adaa2b","arxiv_id":"2501.13622","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Merging adjacent reasoning steps into coarser training steps for process reward models improves best-of-n accuracy on GSM-Plus and MATH500 by about 0.5 to 3.4 percentage points.","lead":"This paper proposes a simple data-augmentation trick for training process reward models: merge adjacent reasoning steps into coarser steps and train at multiple granularities. It reports small, mostly consistent accuracy gains in best-of-n math reasoning evaluation across two datasets and three loss functions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No equal-data control: the reported CFPRM gains may reflect added training samples rather than the coarse-to-fine merging strategy.","rationale":"The reader identified the last-step labeling rule as the weakest assumption. That is a plausible mechanism-level concern, but the data-quantity confound is more load-bearing for the central claim. CFPRM adds merged samples to the original data, so the treated model always sees more training examples than the baseline. Without an equal-data control, the empirical claim 'CFPRM consistently improves' could be true even if the coarse-to-fine merging and relabeling are completely irrelevant; simply adding more samples (or duplicating the original ones) might yield the same gains. The paper's own figures show that performance varies with C, including regressions at C=4, which already undermines the unqualified 'consistently' language, but the missing control is a structural gap in the experimental design. The proposed test isolates the mechanism by matching corpus size and by removing the ordering/merging effect. If CFPRM still wins under these controls, the central claim is substantiated; if not, the contribution reduces to a data-augmentation effect. The reader's labeling-rule concern would remain relevant for understanding why the merged data helps or hurts, but the data-quantity control should come first. I therefore recommend keeping the CONDITIONAL verdict, with the equal-data control added to the required conditions.","tokens_in":6833,"tokens_out":7194,"duration_ms":64276,"concrete_test":"Retrain the three baselines (ShepHerd/BCE, ReSTMCTS*/MSE, PQM/Q-ranking) on two augmented versions of the original training data: (I) duplicate the original fine-grained samples so the corpus size matches CFPRM's, without merging; (II) apply the same window-C=2 merging and last-step labeling rule, but use only the merged samples (or replace an equal number of originals) so the corpus size matches the baseline. Compare BoN@64 accuracy on GSM-Plus and MATH500. If either control reproduces CFPRM's gains, the reported improvements are attributable to data quantity, not to the coarse-to-fine merging strategy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CFPRM consistently improves PRM accuracy across backbones and loss objectives. However, the experimental protocol compares each baseline trained on the original fine-grained data with the same baseline trained on the original data plus all merged coarse samples (Algorithm 1 lines 7-9, Section 2.2 'we combine the merged coarse training samples with the original fine-grained data'). The CFPRM training corpus is therefore strictly larger than the baseline's. Improvements in best-of-n accuracy could be a trivial data-volume effect, independent of the proposed coarse-to-fine mechanism or the last-step labeling rule. The paper provides no ablation that trains a baseline on an equally sized augmented corpus (e.g., duplicated original steps or randomly paired windows with the same labeling rule). Without such a control, the reported gains do not uniquely support the claim that merging and relabeling steps at multiple granularities is what helps. The Limitations section acknowledges that redundant-step detection and adaptive C are open, but it does not mention this data-quantity confound, which is more basic and directly threatens the causal interpretation of every reported improvement.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CFPRM, a coarse-to-fine process reward model (PRM) training strategy. Reasoning traces are augmented by merging adjacent steps with a sliding window of size C, labeling each merged window by the label of its last step, and adding these merged samples to the original fine-grained data. The window size is gradually reduced from Cmax to 1, producing training data at multiple granularities. The authors evaluate CFPRM on GSM-Plus and MATH500 using two Qwen2.5-7B backbones and three loss objectives (BCE, MSE, Q-ranking) under best-of-n selection. The main results in Table 2 report consistent improvements when C=2; Figure 3 examines C=2, 3, 4 and finds mixed results, including a negative effect for C=4 with Q-ranking on MATH500.","tokens_in":6968,"tokens_out":2752,"duration_ms":24649,"significance":"If the central claim holds, CFPRM is a simple, plug-and-play data augmentation method that improves PRM accuracy without changing the architecture or loss function. The paper addresses a real issue—redundant steps in process supervision—and evaluates on external benchmarks with multiple backbones and loss criteria. The design is easy to reproduce, and the authors explicitly note that the method is applicable to arbitrary loss criteria. However, the significance is moderated by the absence of an equal-data control, which leaves open the possibility that the gains are a data-volume effect rather than a coarse-to-fine effect, and by the lack of statistical significance testing despite five repetitions.","major_comments":[{"comment":"The experimental protocol compares each baseline trained on the original fine-grained data with the same baseline trained on the original data plus all merged coarse samples. The CFPRM training corpus is therefore strictly larger than the baseline's. Reported improvements in best-of-n accuracy could be a trivial data-volume effect, independent of the proposed coarse-to-fine merging or the last-step labeling rule. The paper provides no ablation that trains a baseline on an equally sized augmented corpus (e.g., duplicated original steps, or randomly paired windows with a different labeling rule). Without such an equal-data control, the gains in Table 2 do not uniquely support the claim that merging and relabeling steps at multiple granularities is what helps. I request an additional experiment that isolates the data-volume factor.","section":"§2.2 and §3.2 (Algorithm 1, Table 2)"},{"comment":"The main results fix C=2 and claim consistent improvements across all configurations. However, Figure 3 shows that C=2 is not uniformly best and that C=4 can hurt performance (e.g., Q-ranking on MATH500 drops below the raw baseline). The paper does not describe how C is selected; it appears to be chosen based on test-set performance. This undermines the generalizability claim and the statement that CFPRM 'consistently brings performance improvements.' Please report results for a C selected on a held-out validation split, or show that the main conclusions are robust to reasonable choices of C. In addition, the paper states that each experiment is repeated five times to report the mean, but no variance or significance test is given; with small effect sizes (0.5–1.8 points), error bars or a paired test are necessary to assess reliability.","section":"§3.2 and Figure 3"},{"comment":"The label of a merged window is set to the label of its last step. This is a strong assumption: a correct step merged with a later wrong step is labeled entirely negative, and a wrong step followed by a correct step is labeled positive. The paper provides no justification or ablation for this relabeling rule, even though the entire data pipeline depends on it. I request an ablation comparing the last-step labeling rule against alternatives such as majority label, OR-label (any positive), or using the label of the final step in the window. This would clarify whether the proposed labeling contributes to the observed gains or merely preserves the original trajectory's outcome signal.","section":"§2.2 'Merged steps labeling'"}],"minor_comments":[{"comment":"The 'Avg.' column is described as the average across four sampling conditions, but it is not clear whether this is the mean of @8/@16/@32/@64 or some other aggregate. Please make the definition explicit in the table caption.","section":"Table 2"},{"comment":"The abstract states 'Extensive experiments on two reasoning datasets across three loss criteria validate the CFPRM's effectiveness and versatility.' Given the mixed results for C=4 in Figure 3, the term 'extensive' is acceptable but 'validate' is too strong; consider softening to 'support' or 'demonstrate'.","section":"Abstract and §1"},{"comment":"The label annotation in Figure 2 is confusing ('+ -Label + + - + +'). It would help to explicitly show which merged window receives which label and how the label is derived from the last step.","section":"Figure 2"},{"comment":"The Limitations section mentions redundant-step detection and adaptive C, but does not acknowledge the increased training-data volume or the last-step labeling assumption as potential confounds. Adding these would give a more complete picture of the method's scope.","section":"§5 Limitations"},{"comment":"The paper says 'Our code will be released in the future.' For a methods paper, releasing code or at least providing detailed hyperparameters and data-processing scripts is important for reproducibility. Please clarify the release timeline or include a pseudo-code supplement beyond Algorithm 1.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is plausible and the experimental breadth is a strength, but the missing equal-data control is a standard and serious confound. The paper's own Figure 3 presents a negative result for C=4 with Q-ranking, which tempers the word 'consistent' in the abstract and Section 3.2. The lack of variance reporting and the apparent test-set selection of C also need attention. I believe the paper can be revised to address these points, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CFPRM is a simple, honest, plug-in data-formatting trick for process reward modeling. The contribution is real but small; the measured gains (0.5–3.4 points BoN) are plausible. The main thing to know: the paper's central claim—\"consistent improvements irrespective of backbone or loss\"—is supported only in the weak sense that adding merged coarse samples to the original fine-grained data helps. There is no control that adds the same number of extra training samples without the merging/relabeling. So we don't learn whether the mechanism is coarse-to-fine granularity or just more supervised data.\n\nWhat's new and good: sliding-window merging of consecutive steps and relabeling by the last step is not in the cited baselines ShepHerd, ReSTMCTS*, or PQM. The idea is clearly presented, easy to reproduce (if code ships), and applied uniformly across three loss objectives. The Limitations section is refreshingly direct: it admits no redundant-step detection and no adaptive C. The paper doesn't oversell.\n\nSoft spots, in order of importance:\n\n1. The equal-data confound. Algorithm 1 adds all merged windows to the original data. Every CFPRM run trains on a strictly larger corpus than its baseline. Gains of 1–3 points could easily come from seeing more samples. A control that trains on the original data plus duplicated original steps, or on randomly paired windows with the same last-step labeling, would settle this. This is the biggest gap.\n\n2. No variance or significance. They say five runs but report only means. With small effects (0.5–2 points on some cells), the consistent-improvement claim needs error bars, especially for the smaller gains.\n\n3. C is chosen on the test sets. Figure 3 sweeps C=2,3,4 on GSM-Plus/MATH500, and Table 2 uses C=2. No validation split. If C were chosen on a held-out set, the result would be cleaner.\n\n4. The last-step labeling rule is an assumption, not a mechanism. They label a merged window by its final step. If that rule corrupts training signal, the gains wouldn't come from the stated redundancy-reduction story. The paper doesn't test this directly.\n\nNone of these are fatal to the practical usefulness of the trick. If the code ships and the equal-data control confirms the effect, this is a decent minor contribution for PRM practitioners. But as it stands, the causal claim outruns the evidence.\n\nWho should read it: people training PRMs or doing data-centric reasoning work. It's not a theory paper and doesn't need deep math; the value is empirical.\n\nRecommendation: send it to review, but the referee should ask for the equal-data control, error bars, and a validation split for C. It's a legitimate short-paper contribution, not a desk reject.","headline":"A simple data-augmentation trick for PRM training that looks useful at C=2, but the reported gains aren't cleanly attributable to the coarse-to-fine mechanism without an equal-data control.","tokens_in":7583,"tokens_out":2230,"would_cite":false,"duration_ms":19221,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that merging adjacent reasoning steps into coarse windows before training—then shrinking the window—improves process reward models for mathematical reasoning across every backbone, loss, and dataset tested.","keywords":["process reward model","mathematical reasoning","coarse-to-fine learning","step merging","redundant reasoning steps","best-of-n evaluation","sliding window","PRM training"],"falsifier":"Hold the backbone, loss, and C fixed at the paper's best setting (C=2, MSE on the MATH backbone), and add a third training corpus matched in sample count but built by duplicating individual steps instead of merging windows. If best-of-n accuracy matches CFPRM, the coarse-to-fine relabeling is not the active ingredient; if it is clearly lower, the merging mechanism is confirmed.","tokens_in":6576,"feed_emoji":"🧮","tokens_out":10505,"duration_ms":87604,"temperature":0.7,"pith_summary":"The paper argues that the step-level labels used to train process reward models are imprecise because LLM-generated reasoning steps often repeat or restate earlier information instead of adding new content. It proposes CFPRM, a coarse-to-fine strategy: merge C consecutive steps into a single coarse step, label that merged step by the label of its final step, then shrink C down to 1 so the training corpus contains the original fine steps alongside coarser ones. Tested on GSM-Plus and MATH500, with two backbone language models and three losses (MSE, BCE, Q-ranking), the added coarse samples improve average best-of-n accuracy for every baseline method when C=2, with gains of up to 3.4 points. If the claim holds, CFPRM is a plug-in data-augmentation method that improves PRM accuracy without changing the model architecture or loss function.","feed_headline":"Merging reasoning steps improves math problem-solving in every test","feed_subtitle":"A coarse-to-fine step-merging window boosts best-of-n accuracy on GSM-Plus and MATH500 for every baseline tested.","key_machinery":"The operative mechanism is a sliding-window merging-and-relabeling loop. For a trajectory $s_1,\\dots,s_N$, a window size $C$ turns each block $s_{i:i+C-1}$ into one merged step whose label is the label of the last step $s_{i+C-1}$; the window slides over the whole trajectory, and the process repeats for every $C$ from $C_{\\max}$ down to 1. The merged samples are combined with the original fine-grained steps into one corpus, which is traversed coarse-to-fine during training. This is the component that carries the reported gains: it supplies multi-granularity supervision without needing a redundant-step detector.","core_discovery":"The central discovery is that deliberately coarsening the granularity of process supervision—rather than detecting redundancy—is enough to make process reward models better. CFPRM slides a window of size C over a reasoning trajectory, merges each window into one holistic step, assigns it the label of its last individual step, and then repeats with C decreasing to 1. This produces a training set in which the model sees the same reasoning at several granularities; with C=2, all three tested PRM methods improve average best-of-n accuracy on both GSM-Plus and MATH500 under their native losses, and the MSE-trained baseline gains 1.3 and 3.4 points on the two benchmarks respectively.","pith_inferences":["Inference: the same coarse-to-fine relabeling could help PRMs in other multi-step domains, such as code repair or tool-use traces, where self-corrections and repeated statements are common, though the paper only tests mathematical reasoning.","Inference: the last-step labeling rule is a design choice rather than a necessity; majority voting over the window, or a mixed label, might handle windows that contain both a correct prefix and an incorrect conclusion, and comparing rules would separate the merging effect from the relabeling effect.","Inference: the reported gains could also be read as a data-augmentation effect rather than redundancy mitigation, because CFPRM adds extra training samples; a sample-matched control would test the stated mechanism.","Inference: the authors' own limitation notes point to adaptive window-size selection, and a loss-aware schedule for C could remove the C=4 regression seen with Q-ranking."],"forward_implications":["CFPRM is a plug-in: it can be layered onto any existing PRM training pipeline without altering the loss objective, the backbone, or the sampling procedure.","At window size C=2, every base method improves average best-of-n accuracy on both GSM-Plus and MATH500, under both backbone models.","The optimal window size depends on the loss: Q-ranking, which evaluates transitions between adjacent steps, is more sensitive to large C, and at C=4 it can fall below the raw baseline.","Because the corpus is traversed from coarse to fine, the model first absorbs merged holistic steps and then the original fine steps, preserving fine-grained supervision.","CFPRM requires no explicit detection of redundant steps; it simply adds merged samples at multiple granularities to the existing training corpus."],"supporting_citations":[{"why":"Supplies the PRM800K step-labeled corpus used to train the PRMs and establishes step-by-step verification as the task being improved.","marker":"Lightman et al., 2023"},{"why":"Provides the ShepHerd baseline with automatically labeled step data, the BCE loss, and the label-by-final-answer-potential idea that CFPRM adapts for merged windows.","marker":"Wang et al., 2024"},{"why":"Provides the ReSTMCTS* baseline that produces tree-search step labels and is evaluated under the MSE objective.","marker":"Zhang et al., 2024"},{"why":"Provides the PQM baseline under the Q-ranking objective and the observation that redundant, non-incremental steps can distort process supervision.","marker":"Li and Li, 2024"},{"why":"Supplies the MATH benchmark from which MATH-500 is drawn and the mathematical problems used for evaluation.","marker":"Hendrycks et al., 2021"},{"why":"Supplies GSM-Plus, the perturbed grade-school benchmark used for evaluation.","marker":"Li et al., 2024"},{"why":"Supports the paper's interpretation that a later step can repair an earlier wrong step, which motivates labeling a merged window by its last step.","marker":"Setlur et al., 2024"}],"fun_headline_variants":["Coarse-to-fine window beats redundancy detection for math PRMs","Merging reasoning steps boosts math reward models across the board","Coarse-to-fine step merging: better math PRMs without redundancy checks","Sliding window refines process rewards, improving math problem solving","CFPRM: merge reasoning steps to improve process reward models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method depends on treating the last step of a merged window as the truth of the entire window; if a correct earlier part followed by a wrong conclusion is too coarsely labeled, the relabeling could erase partial-progress signal instead of removing redundancy.","fun_headline_variants_meta":{"raw":{"variants":["Coarse-to-fine window beats redundancy detection for math PRMs","Merging reasoning steps boosts math reward models across the board","Coarse-to-fine step merging: better math PRMs without redundancy checks","Sliding window refines process rewards, improving math problem solving","CFPRM: merge reasoning steps to improve process reward models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000485,"raw_usage":{"total_tokens":2334,"prompt_tokens":827,"completion_tokens":1507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":1417}},"tokens_in":443,"tokens_out":1507,"duration_ms":9407,"temperature":1.0,"reasoning_tokens":1417,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:45:13.816896+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold the backbone, loss, and C fixed at the paper's best setting (C=2, MSE on the MATH backbone), and add a third training corpus matched in sample count but built by duplicating individual steps instead of merging windows. If best-of-n accuracy matches CFPRM, the coarse-to-fine relabeling is not the active ingredient; if it is clearly lower, the merging mechanism is confirmed.","supporting_citations":[],"review_version":1}