{"id":"2499931f-d07f-4df7-929b-fc63710beebf","arxiv_id":"2607.05199","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"A step-level reward framework using GPT-4o as a training-time verifier reduces reasoning errors in small language models on physics benchmarks by 10-20% over baselines.","lead":"This paper trains small language models to fix their own physics reasoning errors by spotting the first wrong step and giving targeted feedback during training. It matters because it improves small models on multi-step reasoning without needing expensive human annotations.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The central claim depends on GPT-4o's training-time verifier accuracy, which is never independently validated; additionally, the reward computation code's regex step-counter may not match the XML-tagged solution format, potentially making the step-level reward uninformative.","rationale":"The reader's identification of the GPT-4o verifier dependency as the load-bearing concern is correct. The paper's central claim — accuracy gains of 17–20% over CoT and 10–16% over baselines — depends on the reward signal being meaningful, which in turn depends on GPT-4o correctly identifying error steps and types during training. The paper is commendably honest about this limitation (§8), but acknowledging a gap does not close it. The consistency of gains across 4 models and 5 benchmarks provides partial evidence that the method works, but without knowing the verifier's accuracy, we cannot distinguish whether gains come from the step-level reward mechanism specifically or from the feedback content (which contains correct physics from GPT-4o and retrieval) acting as a general training signal. The regex step-counting concern compounds this: if rewards are systematically miscalculated due to format mismatch, the 'step-level' nature of the reward is compromised, and the method reduces to a simpler feedback-conditioned training signal without position-dependent shaping — weakening the novelty claim. The paper has independent support in the form of consistent cross-model results, detailed error analysis showing differential effects across error types (CE reduces most consistently, consistent with code-execution feedback being more reliable), and released code/prompts. But the verifier accuracy gap and the potential reward computation bug remain the two most important unaddressed conditions for the central claim. The CONDITIONAL verdict with MODERATE confidence is appropriate; I would not adjust it.","tokens_in":33210,"tokens_out":9128,"duration_ms":148510,"concrete_test":"Sample 200 training instances where GPT-4o identified a non-zero error step. Have two physics experts independently annotate the first error step and error type. Compute Cohen's Kappa between GPT-4o and human consensus. Separately, run the reward computation code (Appendix B, Listing 2) on 50 model-generated XML-tagged solutions and log the `total_steps` value returned; if it is consistently 1, the regex mismatch is confirmed and the step-level reward is systematically miscalculated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the most load-bearing concern. The entire training loop depends on GPT-4o correctly identifying the first error step (e_first) and classifying the error type (c1 ∈ {MC, CM, CE}) against ground truth solutions (§3.2, §3.3). The reward r = e_first/(n+1) is computed from e_first, and feedback is routed to one of three channels based on c1. If GPT-4o misidentifies the error step, the reward is wrong; if it misclassifies the error type, feedback goes to the wrong channel. The paper acknowledges this in §8: 'systematic GPT-4o classification errors would corrupt both the reward signal and feedback conditioning y2, without being detectable from accuracy metrics alone.' The human evaluation (Cohen's Kappa 0.75–0.89, Table 6) validates the evaluation pipeline, not the training-time verifier. No accuracy metric for GPT-4o's verifier role is reported. A secondary, more concrete concern: the reward computation code (Appendix B, Listing 2) uses regex `r'(?:#+ )?Step :d+: '` to count total steps, but the model generates XML-tagged solutions with seven tags (<problem_analysis>, <principle>, etc.) per the SFT format in Appendix A. If the regex does not match, `total_steps` defaults to 1, making reward = e_first/2 regardless of solution length. This breaks the intended position-dependent penalty where earlier errors receive lower rewards, reducing the step-level reward to a near-binary signal and undermining the core mechanism that distinguishes this work from response-level methods.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes a step-level reward framework for physics reasoning in small language models (SLMs). The method operates in three stages: (1) SFT warm-up on structured XML-tagged chain-of-thought solutions, (2) a step-level reward mechanism where GPT-4o identifies the first reasoning error against ground truth and computes a position-dependent reward r = e_first/(n+1), and (3) error-type-conditioned structured feedback generation (MC, CM, CE channels) followed by policy gradient updates with KL regularization. The framework is evaluated on four SLMs (1B–3.8B) across five physics benchmarks, reporting accuracy gains of 17–20% over CoT and 10–16% over the strongest baseline, with error-type-specific reductions in calculation and miscomprehension errors. The paper includes ablations on feedback channels, a comparison table with prior work across five dimensions, and an honest limitations section.","tokens_in":33438,"tokens_out":1447,"duration_ms":79807,"significance":"The paper addresses a genuine gap: step-level reward methods for reasoning typically require annotated preference data, and this work proposes a mechanism that avoids that requirement while operating in a domain (physics) where SLMs struggle structurally. The structured feedback routing by error type is a reasonable design choice, and the error taxonomy (MC/CM/CE) is well-motivated. The authors provide code listings, full hyperparameter tables, training prompt templates, and dataset samples, supporting reproducibility. The limitations section is notably candid about several threats to validity. However, the central empirical claims rest on uncharacterized variance and an unvalidated training-time verifier, which together weaken the strength of the evidence presented.","major_comments":[{"comment":"§8; Tables 3–4: All results are from single training runs (seed 42, §B). No multi-seed validation or variance estimates are reported. Given that the RL training loop involves stochastic generation, GPT-4o verifier queries, and feedback routing, the reported accuracy differences—some as small as 2–3 percentage points between conditions—cannot be distinguished from run-to-run noise. This is load-bearing for the claim of consistent gains across models and benchmarks. At minimum, 3 seeds with mean ± std for the primary accuracy tables would be needed to support the comparative claims in Tables 3–4 and the ablation in Table 7.","section":null},{"comment":"§3.2, §3.3, Algorithm 1 lines 4–5, 11–12: The entire reward signal and feedback routing depend on GPT-4o correctly identifying the first error step (e_first) and classifying the error type (c1 ∈ {MC, CM, CE}) against ground truth solutions. The human evaluation (Cohen's Kappa 0.75–0.89, Table 6) validates the evaluation pipeline (Step 3 error analysis in Appendix F), not the training-time verifier. No accuracy metric for GPT-4o's verifier role is reported. The paper acknowledges this in §8: 'systematic GPT-4o classification errors would corrupt both the reward signal and feedback conditioning y2, without being detectable from accuracy metrics alone.' This is a correctness-risk concern: if GPT-4o systematically misclassifies error types or misidentifies the first error step, the reward r = e_first/(n+1) is wrong and feedback is routed to the wrong channel, making the policy gradientupdate","section":null},{"comment":"Appendix B, Listing 2, line 15: The reward computation uses regex `r'(?:#+ )?Step :d+: '` to count total steps (total_steps), but the SFT format (Appendix A) uses seven XML tags (<problem_analysis>, <principle>, etc.) without 'Step N:' headers. If the regex does not match, total_steps defaults to 1 (line 20: `total_steps = len(matches) if matches else 1`), making reward = e_first/2 regardless of solution length. This would collapse the position-dependent penalty—the core mechanism distinguishing this work from response-level reward methods—into a near-binary signal. The authors should either confirm that the model generates 'Step N:' headers during training (contradicting the XML format in Appendix A) or report the frequency of regex match failures and their impact on the reward distribution.","section":null}],"minor_comments":[{"comment":"§3.4, Eq. (2): The loss expression mixes log-probability terms and KL divergence without clear grouping. Parentheses around the REINFORCE term vs. the KL term would improve readability.","section":null},{"comment":"Table 1: The caption says 'Cell shading indicates per-row rank' but no shading is visible in the text rendering. If shading is present in the PDF, this is fine; otherwise, rank should be indicated numerically.","section":null},{"comment":"§4.4: The three-step evaluation protocol is described in the main text but the Step 3 prompt is in Appendix F.2. A forward reference would help readers locate it.","section":null},{"comment":"Table 5: The caption references 'Benchmarks' generically but the text (§5.3) specifies PhysicsQA only. The caption should state 'PhysicsQA' for precision.","section":null},{"comment":"§6.1: The phrase 'the only model where this channel consistently shows negative values' refers to Qwen 2.5 1.5B for Concept feedback, but the prior sentence discusses Statement feedback for LLaMA 3.2 3B. The transition is abrupt and could confuse readers.","section":null},{"comment":"Appendix B vs. Appendix A: The SFT configuration appendix is labeled 'B SFT Training Configuration' but appears after 'A Training Dataset Samples.' The lettering is inconsistent with standard ordering (A, B, C...).","section":null},{"comment":"§8: 'parse failures default to minimal reward and their frequency is not reported' — this is acknowledged but the default value is not specified. Is it 0? This should be stated explicitly.","section":null},{"comment":"References: Lai et al. 2024a and 2024b are identical citations (same arXiv ID). One should be removed.","section":null}],"recommendation":"major_revision","confidential_remarks":"The reader's report and stress-test note correctly identify the two most load-bearing concerns: (1) no multi-seed validation and (2) the training-time GPT-4o verifier is never independently validated. Both concerns are acknowledged by the authors in §8, which is commendable, but acknowledgment does not substitute for evidence. The regex mismatch concern (Listing 2 vs. Appendix A format) is concrete and potentially serious—if confirmed, it would mean the step-level reward mechanism is not functioning as described. I recommend the authors be asked to address all three major comments with empirical evidence (seed runs, verifier accuracy measurement, regex match rate) rather than additional discussion alone. The paper's framing and honest limitations section suggest the authors are capable of addressing these; the question is whether the data supports the current claims."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The three major comments are all valid concerns. We agree that (1) multi-seed validation is needed to support the comparative claims, (2) the training-time GPT-4o verifier accuracy should be directly measured and reported, and (3) the regex-based step-counting mechanism has a format mismatch with the XML-tagged SFT format that must be investigated and corrected. We describe our planned revisions for each point below.","responses":[{"response":"The referee is correct. Single-seed results without variance estimates are insufficient to support the comparative claims, particularly for differences in the 2–3 percentage point range. We acknowledge this limitation explicitly in §8 of the current manuscript but agree that acknowledgment alone is not adequate for the central empirical claims of the paper. We will rerun all four models across all five benchmarks with at least 3 seeds (42, 123, 456) and report mean ± standard deviation for Tables 3, 4, and 7. We will also report whether the gains over the strongest baseline remain statistically significant (paired t-test or Wilcoxon signed-rank, as appropriate). If any model-benchmark comparison fails to show significance after multi-seed validation, we will qualify the corresponding claim accordingly rather than presenting it as a consistent gain. We note that the largest gains (e.g., LLaMA 3.2 3B on JEEBench: 30.21% → 57.34%, a 27-point improvement) are unlikely to be attributable to seed variance, but we agree that the smaller differences require validation. The revised manuscript will present multi-seed results as the primary tables, with the current single-seed numbers available as supplementary material.","revision_made":"yes","referee_comment":"§8; Tables 3–4: All results are from single training runs (seed 42, §B). No multi-seed validation or variance estimates are reported. Given that the RL training loop involves stochastic generation, GPT-4o verifier queries, and feedback routing, the reported accuracy differences—some as small as 2–3 percentage points between conditions—cannot be distinguished from run-to-run noise. This is load-bearing for the claim of consistent gains across models and benchmarks. At minimum, 3 seeds with mean ± std for the primary accuracy tables would be needed to support the comparative claims in Tables 3–4 and the ablation in Table 7."},{"response":"The referee correctly identifies a gap in our validation: the Cohen's Kappa scores in Table 6 measure inter-annotator agreement for the evaluation pipeline (Appendix F, Step 3), not the training-time GPT-4o verifier. These are different systems—the evaluation pipeline uses a separate LLM judge with a different prompt (Appendix F.2), while the training-time verifier uses the prompt in Appendix A (Listing 1). We agree that the training-time verifier's accuracy in (a) identifying the first error step and (b) classifying the error type must be measured directly. In the revision, we will conduct a human evaluation of the training-time verifier on a random sample of at least 200 model-generated solutions from the training loop. For each sample, two human annotators (from the same pool used for Table 6) will independently identify the first error step and error type. We will report: (i) exact-match accuracy for the first error step index, (ii) accuracy for error type classification, and (iii) Cohen's Kappa between GPT-4o and human annotators for both dimensions. We will also report the distribution of reward values (r = e_first/(n+1)) as actually computed during training, which will show whether the reward signal has meaningful variance or is collapsed. If the verifier accuracy is low in certain error categories, we will report this transparently and discuss its implications for the reward signal quality. We acknowledge that we cannot fully eliminate the correctness-risk concern—the framework's dependence on GPT-4o is a fundamental design choice, not a validation gap that can be closed entirely. However, providing direct accuracy metrics for the training-time verifier will allow readers to assess the magnitude of this risk.","revision_made":"yes","referee_comment":"§3.2, §3.3, Algorithm 1 lines 4–5, 11–12: The entire reward signal and feedback routing depend on GPT-4o correctly identifying the first error step (e_first) and classifying the error type (c1 ∈ {MC, CM, CE}) against ground truth solutions. The human evaluation (Cohen's Kappa 0.75–0.89, Table 6) validates the evaluation pipeline (Step 3 error analysis in Appendix F), not the training-time verifier. No accuracy metric for GPT-4o's verifier role is reported. The paper acknowledges this in §8: 'systematic GPT-4o classification errors would corrupt both the reward signal and feedback conditioning y2, without being detectable from accuracy metrics alone.' This is a correctness-risk concern: if GPT-4o systematically misclassifies error types or misidentifies the first error step, the reward r = e_first/(n+1) is wrong and feedback is routed to the wrong channel, making the policy gradientupdate"},{"response":"The referee has identified a genuine inconsistency in our code that we must investigate and address honestly. The SFT training format (Appendix A) uses seven XML tags without 'Step N:' headers, yet the reward computation in Listing 2 (Appendix B) uses a regex pattern that matches 'Step N:' headers. There are two possible explanations: (1) the model generates 'Step N:' headers during the RL training loop despite being trained on XML-tagged format (models sometimes introduce formatting variations during generation), in which case the regex matches but the step count may not correspond to the intended XML-tag-based steps; or (2) the regex does not match, total_steps defaults to 1, and the reward collapses to e_first/2, which would significantly weaken the position-dependent penalty mechanism. We cannot determine which scenario occurred without re-examining the training logs, which we will do in the revision. Specifically, we will: (i) inspect a sample of model-generated solutions from the training loop to determine whether 'Step N:' headers appear, (ii) compute the regex match rate across all training samples, (iii) report the actual distribution of total_steps values and reward values as computed during training, and (iv) if the regex is failing, correct the step-counting mechanism to parse XML tags instead (e.g., counting <substitution> and <calculation> sub-blocks as steps) and rerun the experiments with the corrected reward computation. If the correction changes the results, we will report both the original and corrected numbers transparently. If the regex was matching due to model-generated 'Step N:' headers, we will document this and clarify the relationship between the XML-tagged SFT format and the actual generation format during RL training. We agree with the ref's","revision_made":"no","referee_comment":"Appendix B, Listing 2, line 15: The reward computation uses regex `r'(?:#+ )?Step :d+: '` to count total steps (total_steps), but the SFT format (Appendix A) uses seven XML tags (<problem_analysis>, <principle>, etc.) without 'Step N:' headers. If the regex does not match, total_steps defaults to 1 (line 20: `total_steps = len(matches) if matches else 1`), making reward = e_first/2 regardless of solution length. This would collapse the position-dependent penalty—the core mechanism distinguishing this work from response-level reward methods—into a near-binary signal. The authors should either confirm that the model generates 'Step N:' headers during training (contradicting the XML format in Appendix A) or report the frequency of regex match failures and their impact on the reward distribution."}],"tokens_in":33093,"tokens_out":1662,"duration_ms":66838,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"Bottom line: this paper combines step-level rewards with error-type-conditioned feedback channels to train small language models on physics reasoning, and the gains are real and consistent — 10–16% over the strongest baseline across four models and five benchmarks. The main concern is that the entire training loop depends on GPT-4o as the verifier, and its accuracy in that role is never independently measured. A secondary issue: the reward computation code's regex step-counter likely doesn't match the XML-tagged solution format the models actually generate, which could degrade the step-level reward to a near-binary signal. Both deserve scrutiny, but neither is a knockout — the accuracy improvements are large enough that something is working, even if the mechanism isn't fully validated yet.","headline":"Step-level reward + structured feedback for physics SLMs: real gains, but verifier dependency and regex mismatch need referee attention","tokens_in":34117,"tokens_out":222,"would_cite":false,"duration_ms":51096,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Step-level error rewards cut physics reasoning failures in small language models by up to 33%","keywords":[],"falsifier":"Run the framework with a verifier whose error-classification accuracy is independently measured and deliberately degraded (e.g., by injecting controlled misclassification rates). If accuracy gains degrade proportionally to verifier error rate, the framework's success is verifier-dependent; if gains persist, the feedback signal is robust to misclassification.","tokens_in":33164,"feed_emoji":"🔬","tokens_out":1479,"duration_ms":47831,"temperature":0.7,"pith_summary":"The paper proposes a training method for small language models (under 4 billion parameters) that aims to fix physics reasoning at the exact step where it first goes wrong, rather than scoring the entire solution as a single unit. The central mechanism is a step-level reward r = e_first/(n+1), where e_first is the index of the first reasoning error and n is the total number of steps: solutions that stay correct longer receive higher reward, and earlier failures are penalized more heavily. During training, an external verifier (a large language model) compares the small model's solution against a ground-truth solution, identifies the first erroneous step, classifies the error into one of three types (problem miscomprehension, conceptual misapplication, or calculation error), and generates structured feedback routed through a type-specific channel. The small model then produces a revised solution conditioned on that feedback, and a policy gradient update with KL regularization adjusts the model's weights to favor the revision. The model never sees correct solutions as generation targets — it learns only from feedback on its own mistakes. The framework requires no annotated step-level preference data and uses the external verifier only at training time, not at inference. Across four open-source models and five physics benchmarks, the authors report accuracy gains of 17–20 percentage points over chain-of-thought prompting and 10–16 points over the strongest baseline, with calculation errors dropping from 56.9% to 23.5% and miscomprehension errors from 22.3% to 12.0% in the best cases. Conceptual misapplication errors, however, remain above 42% across all conditions and are identified as the hardest unresolved failure mode.","feed_headline":"Step-level error rewards cut physics reasoning failures in small language models by up to ","feed_subtitle":"A training method that pinpoints the first wrong step in a physics solution and gives type-specific feedback lets sub-4B models rival much","key_machinery":"step-level reward r = e_first/(n+1), error-type-conditioned feedback channels, policy gradient with KL regularization against a frozen SFT reference policy","core_discovery":"The paper's central claim is that pinpointing the first reasoning error in a multi-step physics solution — and conditioning feedback on the error type — lets a small language model learn to revise its own reasoning more effectively than whole-response preference optimization or supervised fine-tuning, without requiring annotated step-level data or an inference-time verifier. The reward formula r = e_first/(n+1) is the load-bearing object: it converts the position of the first error into a scalar training signal, making earlier errors costlier than later ones. The three-channel feedback architecture (re-reading the problem for miscomprehension, retrieving correct physical laws for conceptual,","pith_inferences":["The reward formula r = e_first/(n+1) implicitly assumes that error position is the most informative feature for training signal quality. An alternative — weighting by error type severity or by downstream propagation distance — could be tested by ablating the reward shape while holding feedback constant.","The framework's training loop skips samples where the first attempt is already correct (r > threshold). If a small model's initial accuracy is low, most training samples enter the revision loop; if initial accuracy is high, the effective training set shrinks. This suggests the method may have a natural ceiling tied to the model's first-attempt accuracy distribution.","The coupling of verifier and feedback generator in the same large model (GPT-4o) means that error-type classification accuracy and feedback quality are not independent variables. A controlled study using a high-accuracy classifier with a low-quality feedback generator (or vice versa) would isolate which component drives the gains.","The observation that DPO increases conceptual misapplication while the proposed framework reduces it suggests that whole-response preference signals may actively distort conceptual reasoning in small models, and that step-level localization is not merely an incremental improvement but may prevent a specific failure mode introduced by response-level training."],"forward_implications":["If the step-level reward mechanism generalizes, the same identify-first-error-and-condition-feedback approach could apply to other sequential reasoning domains — chemistry, multi-step arithmetic, logical proof — wherever an external verifier can locate the first deviation from a reference solution during training.","The persistence of conceptual misapplication errors above 42% suggests that retrieval-augmented feedback supplying the correct formula is insufficient for teaching correct application; a feedback channel that demonstrates principle application, not just retrieval, may be needed.","The framework's dependence on a large external verifier at training time but not at inference means the cost of verifier quality is paid entirely during training, making the deployed model lightweight — but any systematic verifier errors are baked into the policy and invisible at evaluation time."],"fun_headline_variants":["Step-level error rewards cut physics reasoning failures in small language models by up to ","Pinpointing the first wrong step in a physics solution lets small language models self-cor","Step-level feedback on the first reasoning error cuts physics calculation errors from 57% ","Targeted feedback on the first wrong step outperforms whole-response tuning in small langu"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire reward signal and feedback routing depend on a single large language model (GPT-4o) correctly identifying and classifying the first reasoning error against a ground-truth solution during training. If that verifier systematically misclassifies error types or misidentifies which step first goes wrong, the reward is wrong, feedback is sent to the wrong correction channel, and the policy gradient update optimizes against a corrupted signal — with no way to detect the腐败","fun_headline_variants_meta":{"raw":{"variants":["Step-level error rewards cut physics reasoning failures in small language models by up to 20%","Pinpointing the first wrong step in a physics solution lets small language models self-correct","Step-level feedback on the first reasoning error cuts physics calculation errors from 57% to 24%","Targeted feedback on the first wrong step outperforms whole-response tuning in small language models"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":633,"prompt_tokens":551,"completion_tokens":82,"prompt_tokens_details":null},"tokens_in":551,"tokens_out":82,"duration_ms":8350,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-07T23:46:39.881031+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Run the framework with a verifier whose error-classification accuracy is independently measured and deliberately degraded (e.g., by injecting controlled misclassification rates). If accuracy gains degrade proportionally to verifier error rate, the framework's success is verifier-dependent; if gains persist, the feedback signal is robust to misclassification.","supporting_citations":[],"review_version":1}