{"id":"989c8d9c-2171-4c59-b700-5fceb0bca2da","arxiv_id":"2608.07931","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"REIN trains reasoning models with reflection-veracity and abstention rewards, letting them answer or abstain in a single pass, and reports large drops in a false-endorsement metric across four benchmarks.","lead":"The paper introduces REIN, a training method that teaches large reasoning models to add a self-reflection step before answering and to say 'I don't know' when their reasoning does not support an answer. The authors report that this lowers a hallucination proxy by 58 to 72 percent while keeping most questions answered.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"H-Proxy for untrained baselines is measured under a forced <think>/<reflection>/<answer> template they were never trained for; without per-method draft-extraction and stance distributions, the 58–72% reduction may be a format artifact.","rationale":"The reader's weakest_assumption identifies exactly the concern that the H-Proxy comparison is compromised by the forced structured-output protocol: untrained baselines are evaluated under a template they were never trained to use, and the metric's dependence on draft parsing and stance classification can bias the comparison. My reading of Sections 4 and A.2 confirms this is the most load-bearing issue for the central claim, because the headline 58–72% reduction is stated relative to those same baselines. The paper's internal sensitivity analyses (Appendix G) address parser robustness for REIN itself, but they do not establish comparability of the baselines' reflection stances or draft-extraction rates. The duplicated DeepSeek-R1-Distill-8B Base rows and missing error bars are additional reproducibility concerns, but they are secondary to the measurement validity issue. Since the reader's CONDITIONAL verdict already hinges on this assumption, my stress-test does not move the verdict; it sharpens the condition by specifying a concrete check that would validate or refute the comparison.","tokens_in":23736,"tokens_out":4957,"duration_ms":51640,"concrete_test":"Recompute H-Proxy for Base and Instruct on GSM8K without the forced template: generate the model's native output, then elicit a reliability judgment in a separate post-hoc call (e.g., 'Is this answer correct? Yes/No/Unsure') for every substantive answer. If the resulting H-Proxy for the baselines differs from the forced-template values by more than, say, 5 points, the reported 58–72% reduction is partly an artifact of the protocol. Also report the draft-extraction rate and stance distribution for each method in Table 1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 states that for H-Proxy 'every method is prompted to generate the same <think>-><reflection>-><answer> structure within a single autoregressive completion.' Appendix A.2 defines H-Proxy as N_nat_e / N_w, where N_w counts incorrect parseable drafts and N_nat_e requires the native reflection stance to be 'correct'. Untrained Base/Instruct models never received the reflection-veracity reward; their 'reflection' stances are arbitrary responses to an unfamiliar template. Two concrete failure modes make the comparison unfair. First, the denominator N_w depends on the draft-extraction rate: if a baseline frequently produces malformed structure so that no draft is parseable, those cases are silently dropped, changing the population over which H-Proxy is computed. The paper reports no per-method draft-extraction rates. Second, the stance parser maps wrong/uncertain/⊥ to non-endorsement; a baseline that emits 'I am not sure' or fails to produce a stance is automatically treated as non-endorsing, which can either inflate or deflate H-Proxy relative to a model trained to output 'correct' only when right. The 58–72% reduction is therefore not a controlled comparison of reliability; it compares REIN's trained reflection with an untrained model's arbitrary text in the same slot. The claimed 86–91% coverage and 6.6–14.2% selective accuracy improvements are also computed after forcing all methods into the same template; the selective accuracy numbers are less format-dependent, but the H-Proxy headline is the primary reliability claim. The reader correctly identifies this as the weakest assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"REIN trains a large reasoning model to emit a structured <think> -> <reflection> -> <answer> completion in a single forward pass. During GRPO training, a reflection-veracity reward aligns the reflection stance with the verified correctness of the pre-reflection draft conclusion, and a boundary-aware IDK reward encourages abstention when none of K=16 sampled completions yields a correct final answer. The method is evaluated on GSM8K, MATH-500, StrategyQA, and ARC-Challenge with Qwen2.5-7B and Llama-3.1-8B, training only on GSM8K and reporting selective accuracy, H-Proxy, coverage, and effective accuracy. The paper reports a 58-72% relative reduction in H-Proxy, 86-91% average coverage, and 6.6-14.2% selective-accuracy improvement, plus ablations, a draft-to-final transition analysis, and a robustness appendix.","tokens_in":24084,"tokens_out":6260,"duration_ms":72850,"significance":"If the main results hold, REIN would be a meaningful single-pass reliability alignment method: it makes abstention a trainable decision, connects reflection to verified draft correctness, and does not require process supervision, external search, or test-time controllers. The cross-task transfer design (training only on GSM8K and evaluating on MATH, StrategyQA, and ARC-Challenge) is a genuine strength, as are the component ablation, the K-sensitivity analysis with confidence bounds, the draft-to-final transition matrix, and the robustness appendix that fixes prompt, parser, and decoding choices. The significance is partly conditional, however, because the headline hallucination metric is closely tied to the reflection-veracity training reward and because the baseline comparison for that metric uses a forced reflection template that untrained models were never trained to follow.","major_comments":[{"comment":"H-Proxy as defined in Eq. (24) is the conditional probability that an incorrect draft is endorsed by the native reflection, Pr[J(r)=1 | T(x,y0)=0]. This is exactly the quantity that the reflection-veracity reward in Eq. (9) rewards the model to minimize: the reward gives +1 when J(r)=T(x,y0) and -1 when they disagree, so incorrect drafts with J=1 are directly penalized. The reported 58-72% H-Proxy reductions therefore report, to a large degree, the model's success at optimizing its own training objective rather than an independent measurement of hallucination. The cross-task results on MATH, StrategyQA, and ARC-Challenge are less subject to this concern because REIN was trained only on GSM8K, and those results do provide independent support. I recommend reframing H-Proxy as an alignment metric and adding at least one independent hallucination measure that does not appear in any reward term, such as post-hoc verifier disagreement or a factuality benchmark, reported separately for the held-out tasks.","section":"Appendix A.3, Eq. (24) vs. Section 3.2, Eq. (9)"},{"comment":"The H-Proxy comparison for untrained baselines is not fully controlled. Section 4 states that for H-Proxy every method, including Base and Instruct, is prompted to generate the same <think>-><reflection>-><answer> structure, and Appendix A.2 defines 'native' solely by whether the reflection is emitted before the final answer, regardless of whether the method was trained with a reflection objective. An untrained base model has never been trained to produce a meaningful reliability stance in that slot, so its 'reflection' is arbitrary text generated under an unfamiliar template. Two concrete consequences follow. First, the denominator N_w^(0) in Eq. (23) counts only parseable drafts; if baselines frequently produce malformed structure, their H-Proxy is computed over a different, smaller population, and the paper reports no per-method draft extraction rates. Second, the stance parser maps wrong, uncertain, and unparsable to non-endorsement, so a baseline that emits 'I am not sure' or no stance is automatically treated as non-endorsing, which can artificially lower or raise its H-Proxy for reasons unrelated to reliability. Please report per-method draft extraction rates and stance distributions, and either restrict H-Proxy to completions with parseable drafts and valid stances or supplement the forced-template comparison with a post-hoc judging protocol that applies the same judge to every method's draft.","section":"Section 4 and Appendix A.2"},{"comment":"The supervised fine-tuning stage that initializes reflection behavior is not specified. Section 3.4 says REIN first applies 'a short supervised fine-tuning stage to initialize the structured output format and the reflection labels,' but the paper does not state how the reflection labels were produced, whether they were written by humans, generated by an LLM, derived from reference answers, or rule-based, nor how many examples were used. This detail is load-bearing because the reflection-veracity reward aligns the model to these initial labels, and because the paper claims the method requires 'no process supervision.' If the SFT labels are generated with an LLM or an oracle, that qualification needs to be stated; if they are rule-based, the construction should be described. Please provide the SFT data construction procedure, label source, and dataset size.","section":"Section 3.4 and Appendix F"},{"comment":"The headline relative reduction in H-Proxy is not consistent with the reported tables. In Table 1, the eight backbone-benchmark cells give relative reductions from Base to REIN ranging from about 62.5% (Llama-3.1-8B on MATH, 48.0 to 18.0) to about 75.0% (Qwen2.5-7B on ARC-Challenge, 36.0 to 9.0). Including Appendix Table 5, the Mistral-7B cell on MATH shows a reduction of about 54%, and the DeepSeek-R1-Distill-8B cell on MATH shows about 81%. The Abstract's '58~72%' range does not match either set of numbers. The text in Section 5 also reports a macro-averaged reduction of 28.5 percentage points, which is a different quantity from the 58-72% relative reduction. Please state exactly which subset of cells the 58-72% range refers to, or report a consistent range computed from all evaluated backbone-benchmark pairs.","section":"Abstract, Section 5, Table 1, and Appendix Table 5"}],"minor_comments":[{"comment":"The H-proxy column in Table 3 reports values as 0.11, 0.08, and 0.06, while Table 1 and the rest of the paper report H-Proxy as percentages (e.g., 11.0, 8.0, 6.0). Please use one consistent convention.","section":"Table 3"},{"comment":"R-Tuning is cited as 'Cohen et al., 2024,' but the reference list attributes R-Tuning to Zhang et al., 2024; Cohen et al. is the IDK-token work. Please correct the citation.","section":"Section 4, Models and Baselines"},{"comment":"The third qualitative example shows 'Wrong. There are two roots ±2; I only listed one.' followed by a final answer of \\boxed{2}. This appears to be an example where the reflection identifies the error but the final answer is not corrected; please clarify whether this is an intended failure-mode illustration or a typo in the final answer.","section":"Appendix H"},{"comment":"The sentence 'REIN reduces H-Proxy by 28.5%' says '28.5%' but the surrounding text compares percentage-point differences (e.g., 'by 14.0%' and 'by 7.0%' for selective and effective accuracy). Please use 'percentage points' consistently to avoid confusing relative and absolute improvements.","section":"Section 5, first paragraph"},{"comment":"The MCC formula is split across a line break with the equation typeset as 'MCC = TP·TN−FP·FNp' followed by a continuation; the square-root term and the denominator are not readable. Please typeset the full formula on one line.","section":"Appendix B.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is worth pursuing: the cross-task design, the component ablations, and the robustness appendix show careful empirical work. The main issue is that the headline hallucination metric is essentially the reflection-veracity training objective, and the forced-template H-Proxy comparison for untrained baselines needs per-method extraction rates and stance distributions to be interpretable. Neither problem looks unfixable within the manuscript's scope. I do not see evidence of misconduct, but the abstract should be reframed so that the central claim is about the alignment objective and its cross-task generalization rather than an independent hallucination measurement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuinely solid piece of work in the reliability-alignment subfield. The core idea is clean: train the model to emit a structured <think> → <reflection> → <answer> completion, reward the reflection for correctly assessing the pre-reflection draft, and use a group-level boundary estimate to reward abstention only when no sampled rollout succeeds. The two-failure-mode framing (reasoning hallucination vs. knowledge hallucination) is useful, and the design of a single-pass mechanism that avoids external verifiers or iterative loops is a real practical advantage. The cross-task transfer experiment, training only on GSM8K and evaluating on MATH, StrategyQA, and ARC-Challenge, is the most convincing part of the paper—it suggests the reliability behavior generalizes rather than just memorizing a reward. I also want to credit the authors for the robustness analysis in Appendix G: they fix the parser, the prompt, and the decoding protocol and show the headline numbers move only slightly, which is more honesty than most papers in this space show.\n\nThe soft spots are real, but not fatal. The H-Proxy metric is essentially the reflection-veracity training objective measured at inference; the 58–72% reduction is partly a report of the reward being optimized. Worse, the evaluation forces untrained Base and Instruct baselines into the same <think>/<reflection>/<answer> template they were never trained for. Their 'reflection' stances are arbitrary text produced under an unfamiliar format, so the comparison is not controlled. The stress-test note is right that this could inflate or deflate the headline number; without per-method draft-extraction rates and stance distributions, the 58–72% claim should be treated with caution. That said, the selective accuracy and effective accuracy gains are less format-dependent, and those still look meaningful.\n\nTwo smaller issues: Table 5 has the DeepSeek-R1-Distill-8B Base rows identical to the Llama-3.1-8B Base rows, which looks like a copy-paste error. No code or data are linked, and there are no error bars or significance tests anywhere. These are addressable and do not by themselves undermine the central claim.\n\nThis paper is for researchers working on reliability alignment for reasoning models. It deserves a serious referee—the idea is worth engaging with and the empirical foundation is mostly sound, but the evaluation protocol needs tightening before publication. If I were the editor, I would send it out with a clear request to address the baseline-template artifact and the missing reproducibility artifacts.","headline":"REIN is a competent single-pass reflection-and-abstention alignment method whose headline H-Proxy reduction is partly a reward-echo artifact, but whose cross-task transfer and selective accuracy gains are real enough to warrant serious review.","tokens_in":24642,"tokens_out":1617,"would_cite":true,"duration_ms":18884,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"REIN trains a large reasoning model to think, reflect on its own draft reasoning, and then either answer or say 'I don't know', reducing the fraction of wrong drafts it confidently endorses by 58–72% while keeping 86–91% of questions…","keywords":["large reasoning models","hallucination","reflection","abstention","self-correction","reinforcement learning","uncertainty calibration","selective answering"],"falsifier":"Recompute the hallucination proxy for the untrained base models under a free-format protocol, where the endorsement decision is made by an independent judge instead of the model's own forced reflection line; if the base models' proxy under free formatting is much lower than the reported 35–48% range, then part of the 58–72% relative reduction is a measurement artifact. As a second check, test abstention against an oracle boundary built from 64 independent samples: if REIN's IDK precision and recall measured against that oracle do not stay in the paper's reported 78–83% and 66–73% ranges, the learned 'I don't know' behavior is chasing the estimator, not the true knowledge gap.","tokens_in":23526,"feed_emoji":"🧠","tokens_out":17938,"duration_ms":171267,"temperature":0.7,"pith_summary":"Large reasoning models hallucinate through two separable routes: reasoning hallucination, where a flawed inference step propagates to a wrong conclusion, and knowledge hallucination, where the model simply lacks the relevant knowledge. This paper claims that one alignment pass can handle both, by training the model to emit a structured completion <think> then <reflection> then <answer>, and rewarding the reflection for correctly judging whether its own draft conclusion is sound. A second, boundary-aware reward makes the model answer 'I don't know' when none of a group of sampled attempts produced a correct answer, which the paper treats as evidence the question sits beyond the model's current knowledge. On math and commonsense benchmarks across four backbones, REIN endorses 58–72% fewer incorrect draft conclusions than the untrained base models while keeping 86–91% of questions answered, all in a single forward pass with no verifier or retry loop at test time. If this holds, reliability is something that can be trained into the model's own self-assessment rather than bolted on with extra inference-time machinery.","feed_headline":"58–72% fewer confident but wrong answers after reflection training","feed_subtitle":"A reward for honest self-judgment and 'I don't know' abstention cuts hallucination in half while keeping most answers.","key_machinery":"The load-bearing object is the structured completion <think> → <reflection> → <answer>, parsed into a pre-reflection reasoning trajectory, the draft conclusion it implies, a reflection span, and a final answer that is either substantive or 'I don't know'. Two rewards carry the argument: the reflection-veracity reward aligns the reflection's reliable/unreliable stance with whether the draft conclusion was actually correct, and the boundary-aware IDK reward estimates a policy-relative knowledge boundary from a group of $K=16$ on-policy completions—'within' if any sample succeeded, 'beyond' otherwise—and prices abstention only against that boundary. The rewards are combined into one scalar with accuracy and format rewards and optimized by GRPO, a group-relative policy-gradient update, under a two-phase curriculum: the first phase explicitly advertises 'I don't know' as an available final action, and the second withdraws the cue so the model must infer when to abstain on its own. The boundary indicator is the mechanism that lets a single training signal tell a fixable inference slip apart from genuinely missing knowledge.","core_discovery":"REIN's central claim is that the two failure modes behind hallucination are distinguishable inside one completion and jointly addressable by two aligned reward signals. A reflection-veracity reward gives +1 when the model's reflection stance agrees with the verified correctness of the draft conclusion implied by its reasoning, and −1 when it disagrees, so the model learns to endorse only drafts that are actually right. A boundary-aware IDK reward marks a prompt as within or beyond the policy's sampled knowledge boundary depending on whether any of $K=16$ sampled completions produced a verified-correct final answer, then rewards correct substantive answers inside the boundary, penalizes unnecessary abstention inside it, and rewards 'I don't know' beyond it. Trained with GRPO on GSM8K and evaluated on MATH-500, StrategyQA, and ARC-Challenge, REIN achieves the best selective accuracy and the lowest hallucination proxy (the fraction of incorrect draft conclusions the reflection nonetheless endorses as reliable) in every reported cell, cutting the proxy by 58–72% relative to the base models at 86–91% average coverage.","pith_inferences":["The selective-prediction table shows the trained reflection stance and sixteen-sample self-consistency agreement are complementary reliability signals (the combination gives the best AUROC), which the paper leaves unexploited; a production variant could run the single-pass reflection as the default gate and add sampling only on borderline cases.","The paper's confidence bound—no successes in $K$ draws implies the true success probability is at most $1-\\alpha^{1/K}$, about 13.4% at $K=16$ and 90% confidence—hands deployments a knob the paper does not turn: choose $K$ by how many mistaken abstentions the application can tolerate, since a 'beyond-boundary' label still leaves a measurable chance the model could have answered.","REIN's reflection-veracity reward presupposes an automatic check of whether the draft conclusion is correct, which math and multiple-choice benchmarks provide; moving to open-ended generation would force a learned correctness proxy, in which case the framework is only as reliable as that proxy."],"forward_implications":["Reliability alignment no longer requires process supervision, inference-time verifiers, majority voting, or multi-round critique loops: the decision to trust or abstain happens inside a single forward pass.","Abstention becomes policy-relative and difficulty-sensitive: a prompt with any successful sample is treated as answerable, and only zero-success groups license 'I don't know', which concentrates refusals near the capability frontier (IDK rate rises from 2.1% on two-step GSM8K problems to 14.8% on five-plus-step problems, and from 3.2% on MATH Level 1 to 31.8% on Level 5).","Because training on GSM8K alone improves reliability on StrategyQA and ARC-Challenge, the reflect-then-abstain behavior transfers across task families rather than being memorized per format.","Reflection acts as a correction mechanism before finalization: across the four benchmarks it repairs 11.0% of initially incorrect drafts while corrupting only 0.9% of initially correct ones.","Deeper reflection is counterproductive: adding nested reflection cycles further suppresses hallucination but reduces coverage and effective accuracy through overcorrection, so the single reflection layer is the preferred operating point."],"supporting_citations":[{"why":"Establishes the reasoning-model paradigm and the GRPO-based RL loop whose outer structure REIN reuses.","marker":"[DeepSeek-AI, 2025]"},{"why":"Supplies the GRPO training objective, the policy-optimization engine for REIN's joint reward.","marker":"[Shao et al., 2024]"},{"why":"Evidence that LLMs fail to self-correct without external feedback; motivates aligning reflection to verified correctness instead of trusting self-critique.","marker":"[Huang et al., 2023]"},{"why":"Survey of when self-correction actually works; underpins the paper's split between recoverable reasoning slips and irrecoverable knowledge gaps.","marker":"[Kamoi et al., 2024]"},{"why":"Self-Refine, the iterative critiquing baseline that REIN is designed to outperform in a single pass.","marker":"[Madaan et al., 2023]"},{"why":"Process-reward-model verification, the step-level supervision approach REIN deliberately avoids.","marker":"[Lightman et al., 2024]"},{"why":"R-Tuning, the refusal fine-tuning baseline REIN is compared against on abstention behavior.","marker":"[Zhang et al., 2024]"},{"why":"IDK-token explicit uncertainty modeling, a core abstention baseline for the reliability comparison.","marker":"[Cohen et al., 2024]"},{"why":"TruthRL, the strongest RL abstention baseline; the matched-coverage head-to-head carries REIN's main claim.","marker":"[Wei et al., 2026]"}],"fun_headline_variants":["Reflection training cuts hallucination by up to 72%","Teach AI to say 'I don't know' and cut wrong answers","Self-reflecting AI reduces confident errors 58-72%","Alignment method boosts AI honesty without extra compute","REIN: fewer hallucinations, more honest self-judgment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline reduction is measured by forcing every model, trained or not, to print a <reflection> verdict before its answer, and the numbers assume both that an untrained model's forced verdict is a meaningful self-endorsement and that the knowledge boundary estimated from $K=16$ training samples is still the right line to abstain beyond when the model is deployed.","fun_headline_variants_meta":{"raw":{"variants":["Reflection training cuts hallucination by up to 72%","Teach AI to say 'I don't know' and cut wrong answers","Self-reflecting AI reduces confident errors 58-72%","Alignment method boosts AI honesty without extra compute","REIN: fewer hallucinations, more honest self-judgment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000188,"raw_usage":{"total_tokens":1384,"prompt_tokens":1051,"completion_tokens":333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":250}},"tokens_in":667,"tokens_out":333,"duration_ms":4377,"temperature":1.0,"reasoning_tokens":250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:39:21.744844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the hallucination proxy for the untrained base models under a free-format protocol, where the endorsement decision is made by an independent judge instead of the model's own forced reflection line; if the base models' proxy under free formatting is much lower than the reported 35–48% range, then part of the 58–72% relative reduction is a measurement artifact. As a second check, test abstention against an oracle boundary built from 64 independent samples: if REIN's IDK precision and recall measured against that oracle do not stay in the paper's reported 78–83% and 66–73% ranges, the learned 'I don't know' behavior is chasing the estimator, not the true knowledge gap.","supporting_citations":[{"cited_title":"When Can","cited_arxiv_id":null,"evidence_quote":"Survey of when self-correction actually works; underpins the paper's split between recoverable reasoning slips and irrecoverable knowledge gaps."},{"cited_title":"2024 , url=","cited_arxiv_id":null,"evidence_quote":"Process-reward-model verification, the step-level supervision approach REIN deliberately avoids."},{"cited_title":"I Don't Know: Explicit Modeling of Uncertainty with an","cited_arxiv_id":null,"evidence_quote":"IDK-token explicit uncertainty modeling, a core abstention baseline for the reliability comparison."}],"review_version":1}