{"id":"e8fb62f3-e2ce-42dd-83df-5973bb3c84a4","arxiv_id":"2505.23646","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Post-training pipeline choice (SFT+RL vs RL-only vs SFT-only) reliably shifts hallucination rates in large reasoning models on fact-seeking benchmarks.","lead":"Some new AI models \"think\" step by step before answering, and this paper asks whether that thinking makes them more or less truthful. It finds that the training recipe matters: models trained with both supervised fine-tuning and reinforcement learning stay more factual, while models trained with only one stage tend to make up more wrong facts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Controlled baselines are the load-bearing issue: two of four SFT+RL rows use non-initialization baselines and one is an inference-mode self-comparison, leaving one clean pipeline test.","rationale":"The most load-bearing condition for the paper's headline is that each LRM is compared with the model from which it was actually post-trained; otherwise accuracy deltas cannot be attributed to the pipeline. The reader's weakest assumption identified this for two Table 1 rows. I agree and sharpen it: the Qwen3-32B row is not a pipeline comparison at all, leaving one clean row in Table 1. This is not an external-consensus objection; it is a correctness risk about causal attribution, and Section 6/Appendix A concede the confound. A direct evaluation of DeepSeek-V3-Base (and ideally the true GLM base) would settle the flagship case. Other weaknesses—missing error bars, below-chance probing for MiMo, and the parameter-update inconsistency—are real but secondary; they affect confidence in auxiliary analyses, not the central comparison. Therefore the conditional verdict stands.","tokens_in":18199,"tokens_out":10025,"duration_ms":96015,"concrete_test":"Run DeepSeek-V3-Base, the true initialization for R1, under the paper's exact decoding and LLM-as-a-judge protocol on SimpleQA and TriviaQA. If V3-Base accuracy is already at or above R1's (28.5/79.0), the reported +4.7/+2.2 delta is a baseline artifact; if V3-Base is clearly below R1, the DeepSeek row supports Takeaway #1. The same substitution with GLM-4-9B-Base (if it is the true base for GLM-4-Z1-9B) would test the remaining uncontrolled SFT+RL row.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that the post-training pipeline, not reasoning capability per se, determines hallucination—hinges on Table 1's SFT+RL comparisons. Three of the four rows do not test the pipeline against the true initialization. DeepSeek-R1 is compared with DeepSeek-V3-Instruct even though R1 was initialized from V3-Base (Section 3.1); GLM-4-Z1-9B is compared with GLM-4-9B-Instruct because the actual base is unreleased; and the Qwen3-32B row is the same checkpoint run with and without the thinking template, which isolates inference-time reasoning mode, not the post-training recipe. Only GLM-4-Z1-32B vs GLM-4-32B-Base is a controlled pipeline comparison. Since instruction-tuned baselines can differ from base checkpoints on factuality tasks, the positive deltas in the uncontrolled rows (4.7/2.2 and 1.0/2.4) could be baseline artifacts. The paper itself concedes in Section 6 and Appendix A that pretraining corpus, instruction-tuning data, and post-training schedules are uncontrolled. If these two rows drop out, the SFT+RL takeaway rests on one clean model pair plus a within-model reasoning-mode comparison, which is too thin for the claimed causal generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether large reasoning models (LRMs) are more prone to hallucination than their non-reasoning counterparts, and answers by comparing factuality on SimpleQA and TriviaQA across models grouped by post-training pipeline. It reports three central takeaways: SFT+RL LRMs improve factuality, RL-only LRMs degrade it, and SFT-only LRMs also degrade it (Section 3.2, Tables 1-3). The paper then introduces two behavioral categories, Flaw Repetition and Think-Answer Mismatch (Section 4), and an uncertainty analysis using self-consistency confidence, ECE calibration, and hidden-state probing (Section 5). It also includes a parameter-shift analysis to argue that catastrophic forgetting is not the main driver (Section 6, Table 7).","tokens_in":18426,"tokens_out":5664,"duration_ms":47971,"significance":"The paper addresses a timely and contested question, releasing evaluation code and covering several model families. Its central direction is plausible: the post-training recipe, not reasoning capability per se, may determine factual reliability. The uncertainty-probing analysis is a distinctive contribution, and the three pipeline-level takeaways are falsifiable. However, the evidence does not yet support the categorical causal generalization in Takeaways #1-3, primarily because the SFT+RL comparison set contains only one initialization-controlled pair, and because most quantitative differences are reported without uncertainty estimates. The paper itself concedes in Section 6 and Appendix A that pretraining corpora, instruction-tuning data, and post-training schedules are uncontrolled.","major_comments":[{"comment":"The SFT+RL takeaway is underdetermined by the comparisons. Only GLM-4-Z1-32B vs GLM-4-32B-Base is a true initialization-controlled test. DeepSeek-R1 is compared against DeepSeek-V3-Instruct rather than its actual initialization (V3-Base); GLM-4-Z1-9B is compared against GLM-4-9B-Instruct because the base is unreleased; and the Qwen3-32B row compares the same checkpoint with and without the thinking template, which isolates inference-time reasoning mode rather than the post-training recipe. The Qwen3 comparison is further confounded because the two modes use different decoding parameters (temperature 0.6 vs 0.7, top-p 0.95 vs 0.8; Table 9). Because instruction-tuned baselines can differ from base checkpoints on factuality tasks, the positive deltas in the uncontrolled rows (4.7/2.2 and 1.0/2.4) do not establish that the SFT+RL recipe improves factuality. The paper acknowledges this limitation in Appendix A, but the abstract and Section 3.2 state the takeaway categorically. Please either add controlled model pairs or downgrade the claim to a conditional statement.","section":"Section 3.2, Table 1"},{"comment":"The behavioral analysis uses one representative model per pipeline (GLM-4-Z1-32B, DeepMath-Zero, DPSK-Qwen-32B), and the annotation is performed by Qwen3-32B, the same model family used for answer extraction and correctness grading. The claim that flaw repetition and think-answer mismatch are 'important causes' of hallucination is too strong with n=1 per pipeline, no inter-annotator agreement, and statistics computed only on hallucinated outputs rather than contrasted with correct outputs. In addition, the assertion that non-reasoning counterparts produce no such behaviors may be tautological if those models do not generate a long CoT. Please add more models per pipeline or soften the causal language.","section":"Section 4.1, Table 4"},{"comment":"The calibration results are reported without error bars or significance tests. The ECE differences are small (e.g., 0.121 vs 0.146 for GLM-Z1-32B vs its base; 0.127 vs 0.112 for DPSK-Qwen-14B vs Qwen2.5-14B-Base), and P(a) is estimated with only N=10 samples per question. It is unclear whether these deltas are within sampling noise. Bootstrap confidence intervals or per-question variance estimates are needed before concluding that SFT+RL improves calibration while SFT-only and RL-only corrupt it.","section":"Section 5.1, Figure 2"},{"comment":"Several accuracy deltas in the RL-only and SFT-only tables are very small and likely within noise: DeepMath-Zero improves by 0.6 points on SimpleQA, Qwen3-14B drops by only 0.4 on average, and DPSK-Qwen-14B improves by 0.1 on SimpleQA. Without error bars or repeated runs, these deltas cannot support the categorical claim that these pipelines 'are more prone to hallucination.' The paper should report variance estimates or restrict the claims to the larger, consistent effects.","section":"Section 3.2, Tables 2 and 3"},{"comment":"The parameter-shift analysis compares raw MAE values across models of different sizes and architectures (Qwen2.5-7B vs Qwen3-14B vs GLM-32B), so the absolute values are not directly comparable. The observation that GLM-Z1-32B has the largest MAE yet mild hallucination, while smaller-MAE models hallucinate more, could be an artifact of scale or architecture differences rather than evidence against catastrophic forgetting. A normalized or within-family comparison would be more convincing.","section":"Section 6, Table 7"}],"minor_comments":[{"comment":"The header contains a typo: 'Fraw Rep.' should be 'Flaw Rep.'","section":"Table 4"},{"comment":"The text refers to 'DPSK-Qwen-7B' in the behavior discussion, but the model collection in Section 3.2 lists DPSK-Qwen-14B, DPSK-Qwen-32B, DPSK-LLaMA-8B, and DPSK-LLaMA-70B; the 7B name appears only in Table 4 and should be resolved.","section":"Section 4.1, Table 4"},{"comment":"There is a typo: 'verision 0.8.3' should be 'version 0.8.3'.","section":"Appendix E"},{"comment":"The text says 'we omit the provided excepts' — 'excepts' should be 'excerpts'.","section":"Section 3.1"},{"comment":"The table only uses the ↓ arrow, but the text notes that DPSK-LLaMA-70B improves on TriviaQA (74.1 to 75.2); please mark this improvement with ↑ or explain the convention in the caption.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and addresses a practically important question. The main risk is overclaiming from uncontrolled public models; the categorical takeaways in the abstract should be aligned with the evidence strength. I would be comfortable with a revised version that adds at least one more controlled SFT+RL comparison, reports uncertainty estimates, and tempers the causal language in the behavior and calibration sections."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the comparison design, not the conclusions. The question is a good one—does post-training for reasoning change factuality?—and the three-way lineup (SFT+RL, RL-only, SFT-only) on SimpleQA and TriviaQA is a useful synthesis. That framing alone makes the paper worth a look.\n\nCredit where due: the direction of the effect is consistent across several model families, the two behavior descriptors (flaw repetition, think-answer mismatch) are memorable and appear in the case studies, and the uncertainty probe is a reasonable mechanistic check. They also put honest limitations in Appendix A.\n\nThe soft spots are real. The load-bearing one is baselines for the SFT+RL claim. DeepSeek-R1 is compared to DeepSeek-V3-Instruct even though R1 starts from V3-Base; GLM-4-Z1-9B is compared to an instruct model whose initialization is unreleased; and the Qwen3-32B row is the same checkpoint with and without the thinking template, which isolates inference mode, not training. That leaves GLM-4-Z1-32B vs GLM-4-32B-Base as the only controlled SFT+RL comparison. With the paper's own appendix conceding that pretraining, instruction data, and schedules are uncontrolled, Takeaway #1 is not established at the strength claimed. The RL-only and SFT-only tables have cleaner baselines, but still no error bars and single runs.\n\nThe behavior analysis uses one model per pipeline and the same Qwen3-32B judge for extraction, correctness, and behavior annotation. Since that judge is itself an SFT+RL model, there is at least a risk of bias, though I would call this a minor concern. The probing table has an anomaly: MiMo-7B-RL-Zero's probe accuracy is 15.5%, below chance for a binary task, and the paper doesn't explain it. There is also an internal contradiction: Section 6 says the parameter-shift data challenge a simple forgetting explanation, while the conclusion says SFT-only/RL-only pipelines show a 'significant drop in the volume of parameter updating, which may explain their hallucination.' Those cannot both be the takeaway.\n\nWho this is for: people building or auditing reasoning models, and benchmark designers. It does not need to be right at full strength to be worth a serious referee. Send it to peer review, but ask for tighter baselines, error bars, and a fix for the probe anomaly and the contradiction. I would not desk-reject it; I also would not accept it as is.","headline":"Useful three-way comparison but the SFT+RL takeaway rests on one clean baseline; the behavioral and probing analyses are suggestive, not conclusive.","tokens_in":18985,"tokens_out":3450,"would_cite":true,"duration_ms":29171,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Whether a reasoning model hallucinates is set by its post-training recipe: cold-start SFT plus verifiable-reward RL reduces hallucination; RL-only and SFT-only increase it.","keywords":["large reasoning models","hallucination","chain-of-thought","post-training","reinforcement learning","supervised fine-tuning","model calibration","factuality"],"falsifier":"A controlled experiment using one base model, for example Qwen2.5-7B-Base, post-trained three ways—cold-start SFT then verifiable-reward RL, zero-style RL only, and distillation SFT only—with matched data and compute would settle the claim if the RL-only and SFT-only versions no longer fall below the base on SimpleQA and TriviaQA while the SFT+RL version improves. A cheaper check is to compare DeepSeek-R1 against DeepSeek-V3-Base rather than DeepSeek-V3-Instruct once the base is accessible; if the +4.7 SimpleQA gain disappears, that example no longer supports the SFT+RL takeaway.","tokens_in":17976,"feed_emoji":"🧠","tokens_out":8359,"duration_ms":68535,"temperature":0.7,"pith_summary":"This paper asks whether large reasoning models—models that think through long chains of reasoning before answering—are more prone to making up facts. It finds the answer depends on how the model was post-trained, not on reasoning ability alone. Models built with a cold-start supervised fine-tuning stage followed by reinforcement learning with verifiable rewards consistently answered more fact-seeking questions correctly than their non-reasoning backbones. Models built by reinforcement learning alone or by distillation alone were generally worse, often repeating the same flawed logic or producing final answers that contradict their own reasoning. The authors conclude that hallucination in reasoning models is largely a training-pipeline effect, and they call for factuality to be tracked during LRM development.","feed_headline":"Training recipe, not reasoning itself, decides model hallucination","feed_subtitle":"Cold-start fine-tuning plus verifiable-reward RL improves factuality; other recipes worsen it.","key_machinery":"The argument is carried by a post-training pipeline taxonomy that sorts reasoning models into three recipes: cold-start SFT plus verifiable-reward RL, zero-style RL without cold start, and distillation-only SFT. Each tested model is assigned to one of these categories and compared with its closest non-reasoning backbone on SimpleQA and TriviaQA. The behavioral analysis adds two named failure modes, Flaw Repetition and Think-Answer Mismatch, scored by an LLM judge on wrong outputs. The mechanism analysis then measures expected calibration error and trains a probe on the hidden states of the last question token to test whether correctness information survives post-training.","core_discovery":"The paper's central claim is that the post-training pipeline determines a reasoning model's tendency to hallucinate on fact-seeking questions. Across SimpleQA and TriviaQA, all tested SFT+RL LRMs—DeepSeek-R1, Qwen3-32B, GLM-4-Z1-9B, and GLM-4-Z1-32B—outperformed their non-reasoning counterparts, with gains up to +4.7 points on SimpleQA and +4.0 points on TriviaQA. In contrast, RL-only models such as MiMo-7B-RL-Zero, DeepMath-Zero, and DAPO-Qwen-32B mostly lost accuracy, with drops as large as -29.9 points on TriviaQA, and SFT-only distilled models such as the DeepSeek-R1-Distill series and Qwen3-14B also dropped on average. The authors trace the extra errors to two behaviors: flaw repetition, where the model cycles through surface-different but logically identical wrong paths, and think-answer mismatch, where the final answer diverges from the chain-of-thought. They further show the failure is visible as corrupted calibration: RL-only and SFT-only models have higher expected calibration error and lose answer-correctness information in their hidden states, whereas SFT+RL models are better calibrated and their hidden states remain more probeable for correctness.","pith_inferences":["The paper itself flags that it could not reproduce full-parameter post-training and relies on publicly released models, so the pipeline attribution is an inference across pre-existing models rather than a controlled experiment; a strict test would train one base model through all three pipelines with matched data and compute.","Flaw Repetition and Think-Answer Mismatch could be turned into inexpensive real-time hallucination alarms: a decoder that detects semantically repetitive thinking loops or a final answer that contradicts the chain-of-thought could flag high-risk outputs before they are delivered.","Because SFT+RL models retained answer-correctness information in hidden states while RL-only and SFT-only models lost it, uncertainty probing could become a model-selection criterion during post-training, separate from benchmark accuracy.","Several comparisons use baselines that are not the exact initialization weights, such as DeepSeek-R1 versus DeepSeek-V3-Instruct and GLM-4-Z1-9B versus GLM-4-9B-Instruct; if unmeasured differences in base-model quality or instruction tuning drive part of the deltas, the practical recommendation may become 'evaluate factuality after every recipe' rather than 'prefer SFT+RL unconditionally'."],"forward_implications":["Developers of reasoning models should not assume that long chain-of-thought reasoning improves factuality; fact-seeking accuracy should be evaluated after each post-training stage.","RL-only and SFT-only pipelines should carry extra factuality safeguards if used, since models trained this way hallucinate more than their non-reasoning backbones.","Calibration error can serve as a practical monitoring signal for hallucination during LRM development, complementing benchmark accuracy.","The finding that parameter-update magnitude does not predict hallucination suggests that hallucination is not simply catastrophic forgetting from large weight shifts; the training objective and data matter more.","Factuality benchmarks such as SimpleQA and TriviaQA should become standard evaluation companions to formal reasoning tasks when releasing reasoning models."],"supporting_citations":[{"why":"Supplies the SimpleQA fact-seeking benchmark and its LLM-as-a-Judge grading template used for the main hallucination measurements.","marker":"[34]"},{"why":"Supplies TriviaQA, the second fact-seeking benchmark, evaluated without its provided passages.","marker":"[16]"},{"why":"Provides DeepSeek-R1 as the flagship SFT+RL example and the DeepSeek distilled SFT-only models.","marker":"[11]"},{"why":"Supplies DAPO-Qwen-32B, an RL-only model whose accuracy drops anchor Takeaway #2.","marker":"[38]"},{"why":"Supplies DeepMath-Zero, the RL-only model used for behavior and calibration analysis.","marker":"[12]"},{"why":"Supplies MiMo-7B-RL-Zero as another RL-only baseline with large accuracy drops.","marker":"[31]"},{"why":"Documents the Qwen3 model family, including the SFT+RL Qwen3-32B and its fused thinking modes.","marker":"[28]"},{"why":"Qwen3 technical report underpins the Qwen3-14B SFT-only distilled model comparison.","marker":"[36]"},{"why":"Documents the GLM-4-Z1 series, the SFT+RL models compared with GLM-4 backbones.","marker":"[13]"},{"why":"Records OpenAI's observation of worsened hallucination on newer reasoning models, the motivating discrepancy the paper addresses.","marker":"[25]"}],"fun_headline_variants":["Post-training recipe, not reasoning, sets hallucination risk","SFT+RL curbs hallucination; other recipes worsen it","Cold-start SFT plus RL keeps reasoning models honest","Training recipe explains reasoning-model hallucinations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the accuracy gaps can be blamed on the post-training pipeline, but the comparisons are not fully controlled: DeepSeek-R1 is compared with DeepSeek-V3-Instruct, GLM-4-Z1-9B with GLM-4-9B-Instruct, and the authors state they could not run the training themselves, so unmeasured differences in base models, instruction tuning, or training schedules could explain part of the effect.","fun_headline_variants_meta":{"raw":{"variants":["Post-training recipe, not reasoning, sets hallucination risk","SFT+RL curbs hallucination; other recipes worsen it","Cold-start SFT plus RL keeps reasoning models honest","Training recipe explains reasoning-model hallucinations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000962,"raw_usage":{"total_tokens":4189,"prompt_tokens":1130,"completion_tokens":3059,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":746,"completion_tokens_details":{"reasoning_tokens":2996}},"tokens_in":746,"tokens_out":3059,"duration_ms":20611,"temperature":1.0,"reasoning_tokens":2996,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:40:33.698820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled experiment using one base model, for example Qwen2.5-7B-Base, post-trained three ways—cold-start SFT then verifiable-reward RL, zero-style RL only, and distillation SFT only—with matched data and compute would settle the claim if the RL-only and SFT-only versions no longer fall below the base on SimpleQA and TriviaQA while the SFT+RL version improves. A cheaper check is to compare DeepSeek-R1 against DeepSeek-V3-Base rather than DeepSeek-V3-Instruct once the base is accessible; if the +4.7 SimpleQA gain disappears, that example no longer supports the SFT+RL takeaway.","supporting_citations":[{"cited_title":"ReFT: Reasoning with reinforced fine-tuning","cited_arxiv_id":null,"evidence_quote":"Supplies the SimpleQA fact-seeking benchmark and its LLM-as-a-Judge grading template used for the main hallucination measurements."},{"cited_title":"Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension","cited_arxiv_id":null,"evidence_quote":"Supplies TriviaQA, the second fact-seeking benchmark, evaluated without its provided passages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the Qwen3 model family, including the SFT+RL Qwen3-32B and its fused thinking modes."},{"cited_title":"Openai o3 and o4-mini system card","cited_arxiv_id":null,"evidence_quote":"Records OpenAI's observation of worsened hallucination on newer reasoning models, the motivating discrepancy the paper addresses."}],"review_version":1}