{"id":"b8c4373d-3f3b-4875-9fd1-d07662760b7e","arxiv_id":"2505.17454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CORE-PO self-trains LLMs to prefer reasoning paths with high self-estimated confidence, improving answer and reasoning accuracy on several benchmarks.","lead":"This paper introduces CORE-PO, a self-training method that teaches LLMs to prefer reasoning paths their own confidence checks judge as correct, rather than only answers that get majority votes. It reports small accuracy gains over existing self-training baselines on six reasoning benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No controlled comparison isolates reasoning-level from answer-level confidence: CORE-PO differs from SC-PO in both the confidence estimator (P(True) vs majority voting) and the training framework, so the central claim is underdetermined.","rationale":"The reader's weakest assumption concerns the trustworthiness of P(True) as a self-evaluation measure. I agree that calibration is a real risk, and the paper's Limitations section explicitly acknowledges it. However, the more precise, load-bearing problem is that the experimental design does not isolate the paper's central variable. CORE-PO and SC-PO differ not only in whether confidence is measured at the reasoning level or the answer level, but also in the confidence estimator itself (P(True) vs majority voting) and in the training protocol (online vs offline DPO, N=5 vs N=8, threshold vs no threshold, two iterations vs one). The base-model rows in Tables 1 and 2 show that P(True|r,a) inference-time selection is already competitive with or better than SC selection, so part of the reported gain could come from the estimator swap alone. The greedy decoding comparisons also cannot separate the effect of reasoning-level confidence from the effect of the training framework. Table 6 is the only ablation that varies reasoning-level confidence while holding the answer-level signal fixed, but it is conducted in a ground-truth fine-tuning setting rather than in the self-training setting that constitutes the paper's contribution. Because of this missing control, the central claim is underdetermined, not refuted. The condition I propose would settle it: if CORE-PO with Cθ(r|x) removed (answer-level P(True) only) performs the same, then the reasoning-level component is not the driver. My recommendation remains CONDITIONAL, matching the reader's verdict, but the condition should explicitly require this ablation rather than only robustness checks like error bars.","tokens_in":16720,"tokens_out":9163,"duration_ms":104258,"concrete_test":"Run a matched ablation: under the exact CORE-PO online-DPO protocol (N=5, same prompts, same checkpoint selection), replace the preference score with answer-level P(True) only, i.e., set Cθ(r|x)≡1, and compare full CORE-PO on all six benchmarks over at least 5 seeds. Also compute, on a held-out set, whether Cθ(r|x) adds pairwise ranking signal over Cθ(a|x,r) against ground-truth reasoning correctness. If the answer-level-P(True) variant is statistically indistinguishable, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines the preference score as Cθ(a,r|x)=Cθ(a|x,r)·Cθ(r|x), while the SC-PO baseline uses majority-vote answer confidence Cθ(a|x) and an offline DPO setup (N=8, score gap >=3, two iterations). The comparison therefore changes two variables at once: the confidence estimator (P(True) vs self-consistency voting) and the object of confidence (reasoning+answer vs answer). The base-model rows in Tables 1 and 2 show that P(True|r,a) inference-time selection already beats SC selection on several benchmarks (e.g., Qwen GSM8K 93.2 vs 92.0; GPQA 34.1 vs 33.7), indicating the estimator itself carries part of the effect. For greedy decoding, CORE-PO still beats SC-PO on most datasets, but this may reflect online vs offline DPO, sample count, or thresholding rather than reasoning-level confidence. Table 6's ablation adds reasoning confidence only in a ground-truth fine-tuning setup, not in the self-training setting. Consequently, the central claim—that preferring high reasoning-level confidence is what improves reasoning—is not causally isolated. The acknowledged calibration risk (Limitations) makes this worse: if P(True) is miscalibrated, the model can increase its own confidence scores without improving reasoning, and the reported o4-mini-judged 'Reason Acc.' can be inflated by selection bias.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CORE-PO, a self-training method for LLM reasoning that uses reasoning-level confidence, estimated via P(True), as the preference signal in DPO instead of answer-level majority-vote confidence. The authors motivate the approach with observational evidence that answer-level confidence can be high even when the reasoning path is incorrect, and that reasoning-level confidence correlates better with reasoning accuracy. They report accuracy improvements over SR-PO and SC-PO on four in-distribution benchmarks (GSM8K, ARC-Challenge, GPQA, MATH) and two out-of-distribution benchmarks (CRUXEval, Game of 24), using Llama3.1-8B-Instruct and Qwen2.5-7B-Instruct. They also present ablations comparing monolithic versus statement-wise P(True) and a ground-truth fine-tuning variant that adds reasoning confidence to answer-accuracy rewards.","tokens_in":17039,"tokens_out":4271,"duration_ms":49175,"significance":"If the central claim holds, the paper makes a useful contribution by identifying a limitation of answer-level confidence in self-training and by providing a simple, DPO-based recipe that uses reasoning-level confidence. The empirical coverage is reasonable: two base models, four in-distribution and two out-of-distribution benchmarks, plus ablations for the confidence estimator and a ground-truth fine-tuning variant. The paper is also candid in its Limitations section about overconfidence risk, language scope, and model scale. However, the main evidence does not yet isolate the proposed reasoning-level confidence signal from other changes in the training protocol, and the reliance on P(True) without calibration checks leaves the mechanism under-supported.","major_comments":[{"comment":"The comparison between CORE-PO and SC-PO changes two variables at once: the confidence estimator (P(True) product vs. majority-vote answer confidence) and the DPO protocol (online, N=5, no score-gap threshold, fixed reference model vs. offline, N=8, score-gap >=3, updated reference model). The base-model rows in Tables 1 and 2 show that P(True|r,a) inference-time selection already outperforms SC selection on several benchmarks (e.g., Qwen GSM8K 93.2 vs. 92.0; GPQA 34.1 vs. 33.7), so part of the reported gain may come from the inference-time selector rather than from training on reasoning-level confidence. The only ablation that adds reasoning confidence (Table 6) is in a ground-truth fine-tuning setting, not in the self-training setting. Please add a controlled comparison that isolates the confidence estimator from the training protocol, for example SC-PO with answer-level P(True) or CORE-PO with majority-vote answer confidence.","section":"Section 3.2, Eq. (1); Tables 1 and 2; Appendix B.4"},{"comment":"The authors select the checkpoint with the highest accuracy on the ARC-Challenge validation split, and ARC-Challenge is also one of the four in-distribution evaluation benchmarks in Tables 1 and 2. This selection procedure can inflate the reported ARC-Challenge results and any other numbers reported for the selected checkpoint. Please report results for a checkpoint selected on a validation set disjoint from all evaluated benchmarks, or report results across all checkpoints.","section":"Appendix B.4"},{"comment":"All results are from a single run with no error bars, standard deviations, or significance tests. Several reported differences are small (e.g., GSM8K 93.0 vs. 93.5 for Qwen; ARC-Challenge 92.0 vs. 92.8), and without variance estimates it is unclear whether these differences are reliable. Please provide multiple seeds and report means and standard deviations, or otherwise justify that the differences are not within run-to-run noise.","section":"Tables 1, 2, and 4"},{"comment":"The method uses P(True) both as the training signal and as the inference-time selection score, and Table 3 shows that confidence increases after training. However, the paper does not report calibration or precision-recall analysis of P(True) before and after training, and the Limitations section explicitly acknowledges that the confidence measure 'can suffer from overconfidence.' Without such analysis, the observed confidence increase could reflect miscalibrated inflation rather than improved reasoning, and the reported Reasoning Acc. gains under P(True|r,a) selection may be partly a selection artifact. Please report calibration curves or reliability diagrams for P(True) and show that the post-training confidence increase coincides with improved external reasoning quality rather than only with higher self-assigned scores.","section":"Section 3.2; Table 3; Limitations"}],"minor_comments":[{"comment":"The column header 'QPGA ext' is a typo and should read 'GPQA ext'.","section":"Table 2"},{"comment":"CRUXEval is attributed to 'Guo et al. (2024)' in the data-statistics text, but the reference list entry is by Gu et al. (2024); please correct the citation.","section":"Appendix B.3"},{"comment":"The benchmark name is written inconsistently as 'Cruxout', 'CRUXout', and 'CRUXEval'; please use one consistent name.","section":"Table 4 and Section 4.1"},{"comment":"The x-axis label 'Top-N%' is undefined; please specify what N denotes and how the curves are computed.","section":"Figure 2"},{"comment":"The learning-rate search is reported as {1e-6, 5e-6} but the final chosen learning rate for each model is not stated; please report the selected value.","section":"Appendix B.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real gap and the proposed method is plausible, but the central claim is currently underdetermined by the experimental design. The missing controlled ablation, checkpoint-selection issue, and lack of multi-run variance are all fixable within the scope of a revision. I would support acceptance after these points are addressed; if the controlled ablation shows the effect largely comes from the training-protocol change, the paper would need to be reframed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid empirical paper with a clean idea—use P(True) as a reasoning-level confidence signal for DPO self-training—and the results point in the right direction. But the headline claim, that reasoning-level confidence is what drives the gains, isn't actually isolated by the experiments. CORE-PO differs from SC-PO in two ways at once: the confidence estimator (P(True) vs majority voting) and the training setup (online vs offline DPO, N=5 vs N=8, different pair construction). The base-model rows in Tables 1 and 2 show that P(True|r,a) inference selection already beats SC selection on several benchmarks, so part of the effect is in the estimator itself. The paper needed at least one ablation that changes one variable at a time—e.g., online DPO with majority-vote confidence, or offline DPO with P(True)—to make the central claim stick.\n\nWhat's genuinely new: the specific combination of reasoning-level P(True) with online DPO for self-training. The motivation is well-argued, and Observations 1 and 2 in Figure 2 are nice empirical support. The paper is also honest about the calibration issue in the Limitations, and it covers six benchmarks and two base models, which is more than many papers in this area. Table 6, where adding reasoning confidence to ground-truth fine-tuning improves both answer and reasoning accuracy, is a good supporting piece, even though it's not the self-training setting.\n\nSoft spots beyond the missing isolation: no comparison to ReST-MCTS*, the closest process-reward self-training baseline, which is cited but never run. Single-run results with no error bars or significance tests, and checkpoint selection on ARC-Challenge validation for a benchmark that is later reported—minor risk, but it's a leak. No code or data released.\n\nThe citation pattern looks fine; the related work is reasonably thorough. I don't see a fatal flaw. The calibration concern is real but it's a known issue for any P(True)-based method, and the authors flag it themselves.\n\nBottom line: this paper deserves a serious referee. The recipe is practical, the gains are modest but consistent, and the failure to isolate the mechanism is a fixable experimental gap rather than a fundamental error. I'd want to see the controlled ablation and an error-bar run before accepting, but I'd bring it to a reading group and cite it as a useful reference.","headline":"A practical self-training recipe with real promise, but the key experiment that would separate reasoning confidence from other changes is missing.","tokens_in":17571,"tokens_out":2936,"would_cite":true,"duration_ms":32969,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an LLM to prefer reasoning paths it itself judges true, rather than answers it repeats most often, improves accuracy and reduces reasoning errors on six benchmarks.","keywords":["reasoning-level confidence","self-training","P(True)","direct preference optimization","chain-of-thought","self-consistency","confidence calibration","LLM reasoning"],"falsifier":"On held-out questions where the base model's P(True) is known to be overconfident, measure P(True) and an external judge's reasoning-accuracy score before and after CORE-PO training: if P(True) rises while reasoning accuracy stays flat or falls, the training signal is being gamed rather than improved. A cleaner control is to run the same DPO loop with the P(True) scores randomly shuffled or reversed; answer accuracy that still improves under that control would show the gain does not come from reasoning-confidence information.","tokens_in":16535,"feed_emoji":"🧠","tokens_out":9109,"duration_ms":61043,"temperature":0.7,"pith_summary":"To improve chain-of-thought reasoning without human labels, the paper proposes training an LLM to prefer its own reasoning paths that the model itself judges to be correct. Existing confidence-based self-training relies on majority-voted answers, which can reward a wrong chain of thought that happens to end in a correct answer. The paper's method, CORE-PO, measures reasoning-level confidence with P(True) and fine-tunes the model with direct preference optimization to favor high-confidence reasoning. The authors claim this improves answer accuracy and reduces reasoning errors on four in-distribution and two out-of-distribution benchmarks, beating answer-confidence-based self-training.","feed_headline":"Better LLM self-training from reasoning confidence, not answer votes","feed_subtitle":"CORE-PO prefers reasoning paths the model itself judges true, gaining accuracy on six benchmarks.","key_machinery":"The load-bearing object is the reasoning-level confidence score $C_\\theta(r|x)$, defined as $P(\\text{True})$: the probability that the LLM returns “true” when prompted to judge whether its reasoning path $r$ is correct for question $x$. It comes in two variants, a monolithic one that asks for a single verdict on the whole chain and a statement-wise one that averages per-step verdicts, and it is multiplied by the answer-level confidence $C_\\theta(a|x,r)$ to form the preference reward. That reward feeds an online direct preference optimization update that raises the likelihood of high-confidence reasoning–answer sequences and lowers the likelihood of low-confidence ones, which is the mechanism that transfers the model's self-assessment into a better policy.","core_discovery":"Stated on the paper's own terms, the discovery is that answer-level confidence, estimated by majority voting over sampled answers, is an unreliable proxy for reasoning quality: a chain of thought can be wrong even when its final answer is confidently repeated. Reasoning-level confidence, obtained by asking the model whether its reasoning is true, tracks reasoning accuracy more closely and separates sound chains of thought from faulty ones. When the model is fine-tuned to prefer reasoning–answer pairs with high combined confidence $C_\\theta(r|x)C_\\theta(a|x,r)$ using online direct preference optimization, it produces fewer erroneous reasoning steps and more accurate answers than models trained to prefer high-confidence answers alone, both on in-distribution tasks and on held-out tasks from a different domain.","pith_inferences":["A natural next step the paper leaves implicit is monitoring calibration after DPO: if P(True) rises faster than externally judged reasoning accuracy, the preference signal eventually becomes self-confirming, and a calibration penalty or held-out truthfulness filter would be needed.","Statement-wise P(True) is effectively a parameter-free process reward; distilling it into a learned process reward model and using it for tree search at inference is a direct extension that could combine CORE-PO's signal with search-time scaling.","Because the paper's gains are largest on multiple-choice benchmarks, where wrong reasoning can reach the right answer by chance, the method should transfer especially well to tasks with many plausible distractors, such as medical or legal multiple-choice QA."],"forward_implications":["Self-training with only unlabeled questions can improve chain-of-thought reasoning when the training signal is reasoning quality rather than answer agreement.","The accuracy gains appear on out-of-distribution tasks such as code execution and arithmetic puzzles, suggesting the preference signal teaches a transferable disposition toward careful reasoning.","Even in the supervised setting where ground-truth answers are available, adding reasoning-level confidence to an answer-accuracy reward improves both answer accuracy and reasoning accuracy.","Because both the monolithic and the statement-wise P(True) variants improve over the base model, the benefit is attributed to the reasoning-confidence principle itself, not to one particular confidence implementation."],"supporting_citations":[{"why":"Defines P(True), the self-evaluation probability that the paper uses as its reasoning-level confidence measure.","marker":"Kadavath et al., 2022"},{"why":"Introduces self-consistency and majority voting, the answer-level confidence on which the SC-PO baseline and the paper's comparison rest.","marker":"Wang et al., 2023"},{"why":"Proposes SC-PO, the answer-level-confidence self-training baseline the paper must beat.","marker":"Prasad et al., 2024"},{"why":"Supplies the online direct preference optimization algorithm used to fine-tune the model toward high-confidence reasoning.","marker":"Guo et al., 2024"},{"why":"Shows that reasoning-level confidence selects high-quality outputs at inference, motivating the paper's training-time use of the same signal.","marker":"Becker and Soatto, 2024"},{"why":"Demonstrates that confidence improves self-consistency selection, evidence that high reasoning-level confidence correlates with accuracy.","marker":"Taubenfeld et al., 2025"},{"why":"Proposes SR-PO, the linguistic self-assessment baseline compared against CORE-PO.","marker":"Kumar et al., 2024"}],"fun_headline_variants":["Reasoning confidence beats answer votes for LLM self-training","Self-train LLMs with reasoning confidence, not vote counts","CORE-PO: Self-train LLMs by trusting reasoning over votes","CORE-PO: Prefer reasoning truth over vote-based answers","LLM self-training: trust reasoning, not answer votes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on the model's P(True) self-evaluation being a trustworthy, non-gameable measure of reasoning quality; if that confidence is miscalibrated or becomes inflated during training, the model can be trained to prefer incorrect reasoning that merely sounds confident.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning confidence beats answer votes for LLM self-training","Self-train LLMs with reasoning confidence, not vote counts","CORE-PO: Self-train LLMs by trusting reasoning over votes","CORE-PO: Prefer reasoning truth over vote-based answers","LLM self-training: trust reasoning, not answer votes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3252,"prompt_tokens":873,"completion_tokens":2379,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":2291}},"tokens_in":489,"tokens_out":2379,"duration_ms":12367,"temperature":1.0,"reasoning_tokens":2291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:46:24.657250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On held-out questions where the base model's P(True) is known to be overconfident, measure P(True) and an external judge's reasoning-accuracy score before and after CORE-PO training: if P(True) rises while reasoning accuracy stays flat or falls, the training signal is being gamed rather than improved. A cleaner control is to run the same DPO loop with the P(True) scores randomly shuffled or reversed; answer accuracy that still improves under that control would show the gain does not come from reasoning-confidence information.","supporting_citations":[],"review_version":1}