{"id":"4bab15dd-f969-45ec-bb54-c6f507027ae0","arxiv_id":"2607.05861","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"MARGO mitigates thinking-induced hallucination in large reasoning models by using mixed-mode GRPO rollout groups that compare thinking trajectories against same-model non-thinking references.","lead":"This paper identifies a failure mode where explicit reasoning traces in large reasoning models can overturn correct direct answers, causing factual hallucination. It proposes MARGO, a reinforcement learning method that mixes thinking and non-thinking rollouts in the same training group so the model learns when reasoning helps versus hurts factual accuracy.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Same judge (Qwen3-32B) used for data selection, training rewards, and all evaluation creates a measurement loop; the disproportionately large SimpleQA-V gains (+5.9% on 4B, +4.4% on 8B) are the most likely place where judge bias could be inflating results.","rationale":"The paper identifies a real phenomenon (thinking-induced hallucination, well-characterized in Tables 1, 6–8), proposes a clean and mathematically correct advantage decomposition (Appendix A, Proposition 1), and the mixed-mode rollout group design is a genuinely novel mechanism. The ablation in Table 3 (random vs. selected data) and the transition analysis in Figure 2 provide supporting evidence that the gains come from the mixed-mode regularization rather than RL alone. The math evaluation (Table 4) shows reasoning ability is preserved. However, the central empirical claim — 'consistently improves factual QA accuracy across six benchmarks' — rests entirely on a single judge model used throughout the pipeline. The disproportionately large SimpleQA-V gains are the specific place where this concern is most acute: a 144% relative improvement on a benchmark designed to be more reliable, from training on only 5,660 TriviaQA examples, with no RL baseline showing any gain on that benchmark, warrants cross-judge validation. The reader's verdict of CONDITIONAL with MODERATE confidence is appropriate. The concern is not that the method is wrong — the advantage decomposition is correct and the design is well-motivated — but that the measurement infrastructure cannot independently confirm the magnitude of the claimed gains. A single cross-judge re-evaluation would settle this.","tokens_in":20233,"tokens_out":2978,"duration_ms":188444,"concrete_test":"Re-evaluate all methods in Table 2 (both Qwen3-4B and Qwen3-8B rows) using a different judge model — e.g., GPT-4o or Claude-3.5-Sonnet — with the same ground-truth-conditioned prompt from Appendix B. If MARGO's gains persist across all six benchmarks (especially SimpleQA-V) within ±1% of the reported values, the judge-circularity concern is resolved. If the SimpleQA-V gain shrinks by more than 2% absolute or any benchmark's ranking changes, the gains are partially judge-dependent. This requires no retraining — only re-scoring existing model outputs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader correctly identifies the most load-bearing concern: Qwen3-32B serves as the sole judge across the entire pipeline — data filtering (§C.1, Eqs. 25–32), reward computation during training (§4.1, Eq. 13), and all six test-benchmark evaluations (§4.1). The paper notes in Appendix B that the judge performs ground-truth-conditioned semantic matching rather than open-ended factual verification, which mitigates but does not eliminate the concern. A judge can have systematic biases in semantic matching — e.g., favoring concise direct answers over hedged ones, or being more lenient toward certain phrasings — and MARGO could learn to exploit these during RL training. The evidence that this might be happening is the SimpleQA-V column: on Qwen3-4B, MARGO jumps from 4.10% to 10.00% (+5.90 absolute, +144% relative), while FixedThink+RL stays flat at 4.10%. This is more than double the gain on any other benchmark, and it occurs on a benchmark (SimpleQA-V) that was specifically designed to be more reliable. On Qwen3-8B, the pattern repeats: SimpleQA-V goes from 4.60% to 9.00% (+4.40), again the largest gain. If MARGO were genuinely suppressing thinking-induced hallucination, we'd expect proportional gains across benchmarks; instead the gains are concentrated where the baseline is lowest and the judge has the most discretion over borderline cases. No error bars or significance tests are reported, so we cannot assess whether the 1–2% gains on TriviaQA/NQ_Open/PopQA are within noise. The advantage decomposition (Eq. 8/19) is algebraically correct and the method design is sound; the concern is purely about measurement validity.","agreement_with_reader":"agree"},"referee_report":{"model":"glm-5.2","summary":"This paper introduces MARGO, a mixed-mode GRPO variant that places thinking and non-thinking trajectories in the same rollout group for factuality-oriented QA. The core idea is that non-thinking responses serve as same-model references in advantage estimation, so that thinking trajectories are penalized when they degrade factuality relative to direct answering. The advantage decomposition (Proposition 1, Appendix A) shows that the mixed-mode advantage of a thinking trajectory decomposes into a within-thinking advantage plus a residual-value adjustment proportional to the thinking/non-thinking reward gap. Experiments on Qwen3-4B and Qwen3-8B across six factuality QA benchmarks show consistent improvements over fixed-mode and adaptive baselines, while mathematical reasoning ability is preserved. The transition analysis (Tables 1, 6–8) documenting thinking-induced hallucination is a useful empirical contribution.","tokens_in":20481,"tokens_out":1191,"duration_ms":272887,"significance":"The paper addresses a real and timely problem: thinking-induced hallucination in large reasoning models on factuality tasks. The mixed-mode advantage decomposition (Proposition 1) is a clean, correct mathematical result that directly motivates the method. The transition analysis across multiple benchmarks and model scales (Tables 1, 6–8, including the GSM8K contrast in Table 8) provides solid empirical grounding for the phenomenon. The data selection ablation (Table 3) and the transition-ratio analysis (Figure 2a–b) are well-designed controls that isolate the contribution of mixed-mode regularization from RL or data selection alone. The method is simple to implement and the experimental setup is reasonably comprehensive.","major_comments":[{"comment":"§4.1, Appendix B, Appendix C.1: The same judge model (Qwen3-32B) with the same prompt is used for (a) training data filtering (Eqs. 25–32), (b) reward computation during RL training (Eq. 13), and (c) all six test-benchmark evaluations. This creates a measurement loop: if Qwen3-32B has systematic biases in semantic matching (e.g., favoring certain answer formats, lengths, or phrasings), MARGO could learn to exploit these during RL training, and the same biases would inflate test scores. The paper acknowledges that the judge performs ground-truth-conditioned matching rather than open-ended verification (Appendix B), which mitigates but does not eliminate this concern. The SimpleQA-V column in Table 2 is particularly striking: on Qwen3-4B, MARGO jumps from 4.10% (FixedThink) to 10.00% (+5.90 absolute, +144% relative), while FixedThink+RL stays flat at 4.10%. This is more than double the per","section":null}],"minor_comments":[{"comment":"Table 2: No error bars, confidence intervals, or significance tests are reported for any benchmark. Given that several gains are in the 1–3% range (e.g., TriviaQA 4B: +1.68% over FixedThink+RL), it is difficult to assess whether these are within sampling noise. Adding bootstrap confidence intervals or at least reporting results over multiple random seeds would strengthen the claims.","section":null},{"comment":"§4.1: The evaluation uses temperature 0.6 for thinking and 0.7 for non-thinking mode. It would help to clarify whether the reported numbers are single-run or averaged over multiple samples, and whether the same decoding settings were used for all baselines.","section":null},{"comment":"Table 3 (data selection ablation) is only reported on Qwen3-4B. Including the 8B ablation would make the data-selection claim more robust.","section":null},{"comment":"Figure 2a–b: The y-axis labels and bar values are somewhat hard to read. Consider using a table format or increasing font size.","section":null},{"comment":"§C.1, Eqs. 29–30: The threshold for thinking-favored examples on Qwen3-4B requires Δ(x) ≥ 1.0 and S_T(x) ≥ 1.0. Since S_T is computed from N=6 samples, S_T(x) ≥ 1.0 means all 6 thinking samples are correct. This is a very strict threshold; it would help to report how many examples satisfy each condition.","section":null},{"comment":"The paper would benefit from a brief discussion of how α (the thinking ratio) was chosen. Only α=0.75 is tested; a small ablation over α values would clarify sensitivity.","section":null},{"comment":"References: Several entries have incomplete formatting (e.g., 'Guibin Zhang et al.' missing venue/year; 'Pingzhi Li et al.' missing year).","section":null}],"recommendation":"major_revision","confidential_remarks":"The core methodological contribution (mixed-mode advantage decomposition) is sound and the paper is well-written. The primary concern is the single-judge evaluation loop, which is load-bearing for the empirical claims. If the authors can show that gains hold under an independent judge or human evaluation — particularly on SimpleQA-V where the gains are anomalously large — this paper would be a solid contribution. Without that, the reviewer cannot distinguish genuine factual improvement from judge-bias exploitation. The SimpleQA-V anomaly is the most concrete red flag: a benchmark designed to be more reliable showing the largest gains, concentrated where the baseline is lowest, is exactly the pattern one would expect if the model learned to exploit judge tendencies rather than genuinely improve factuality."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for the careful and constructive review. The referee identifies a valid concern about the measurement loop created by using the same judge model (Qwen3-32B) for training data filtering, RL reward computation, and test evaluation. We agree this is a legitimate methodological concern and will address it in revision by adding an independent evaluation judge and discussing the limitations of the shared-judge design. The core contributions—the thinking-induced hallucination phenomenon, the advantage decomposition, and the mixed-mode regularization mechanism—are independent of the specific judge used, but we acknowledge the referee's point that the current evaluation setup does not fully rule out judge-bias exploitation.","responses":[{"response":"We agree with the referee that using the same judge model for training data filtering, RL reward computation, and test evaluation creates a potential measurement loop. This is a legitimate methodological concern. We will address it through the following revisions: (1) We will add an independent evaluation using a different judge model (e.g., GPT-4o or Llama-3.3-70B) on at least a subset of benchmarks to verify that MARGO's improvements are not artifacts of Qwen3-32B's biases. (2) We will add an explicit discussion of this limitation in the paper, acknowledging the shared-judge concern and explaining why ground-truth-conditioned semantic matching (as opposed to open-ended verification) mitigates—but does not eliminate—the risk. (3) Regarding the SimpleQA-V jump specifically: we note that FixedThink+RL, which uses the same judge for reward computation and the same training data, does NOT show this gain (staying at 4.10%), which suggests the improvement is attributable to the mixed-mode advantage mechanism rather than simple judge exploitation. If MARGO were merely learning to exploit judge biases, we would expect FixedThink+RL to benefit similarly since it shares the same reward signal. That said, we acknowledge this argument is not fully conclusive without an independent judge, and we will provide the cross-judge evaluation to properly address the concern.","revision_made":"yes","referee_comment":"§4.1, Appendix B, Appendix C.1: The same judge model (Qwen3-32B) with the same prompt is used for (a) training data filtering, (b) reward computation during RL training, and (c) all six test-benchmark evaluations. This creates a measurement loop where MARGO could learn to exploit Qwen3-32B's systematic biases during RL training, and the same biases would inflate test scores. The SimpleQA-V result on Qwen3-4B (4.10% to 10.00%, +144% relative) is particularly striking."}],"tokens_in":19781,"tokens_out":559,"duration_ms":156926,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"MARGO's core idea is simple and genuinely new: put thinking and non-thinking trajectories in the same GRPO rollout group, so the advantage signal reflects whether thinking actually adds factual value over direct answering. The advantage decomposition (Proposition 1) is correct — it's straightforward algebra showing the mixed-group baseline introduces a residual-value term that standard all-thinking GRPO lacks. The transition analysis (Table 1) documenting that 6-8% of factual QA examples flip from correct to incorrect when thinking is enabled is a clean empirical observation, and the contrast with GSM8K (where r_{1,0} is under 2%) is a nice touch. The method is lightweight — it only changes rollout group composition — and the ablation showing random data selection fails (Table 3) supports the design choices. Math benchmarks are preserved, which matters for a method that could have degenerated into suppressing all reasoning. Credit where due: the mechanism is well-motivated, the experiments are systematic, and the writing is clear. The concurrent work on thinking-induced hallucination means the observation isn't unique, but the specific solution is. The main soft spot is evaluation validity. Qwen3-32B serves as the sole judge across the entire pipeline — data filtering, reward computation, and all six test benchmarks. The judge does ground-truth-conditioned semantic matching rather than open-ended factual verification, which mitigates the circularity concern but doesn't eliminate it. A judge can have systematic biases in how it interprets semantic consistency, and MARGO could learn to exploit those during RL training. The SimpleQA-V column is where this concern is most visible: +5.9% on 4B and +4.4% on 8B, more than double the gains on any other benchmark, concentrated where baselines are lowest and the judge has the most discretion over borderline cases. If MARGO were genuinely suppressing thinking-induced hallucination, I'd expect more proportional gains across benchmarks. No error bars or significance tests are reported, so we can't tell whether the 1-2% gains on TriviaQA, NQ_Open, and PopQA are within sampling noise. Experiments are limited to Qwen3 at two scales. These are addressable concerns, not structural flaws. The method is sound, the math checks out, and the core contribution — the mixed-mode advantage mechanism — is real. A revision with a second judge model (or human spot-checks on SimpleQA-V), plus significance testing, would substantially strengthen the paper. This deserves a serious referee.","headline":"MARGO mixes thinking and non-thinking rollouts in the same GRPO group — a clean, genuinely new mechanism for a real problem. The math is correct, the method is lightweight, and math ability is preserved. The main concern is evaluation validity: one judge model (Qwen3-32B) is used for data selection, training rewards, and all test evaluation, with no error bars. Still worth a serious referee.","tokens_in":21121,"tokens_out":1203,"would_cite":false,"duration_ms":75671,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Thinking can make AI less factual — now there's a fix","keywords":[],"falsifier":"Run MARGO with a different judge model for evaluation than for training, or add human evaluation on a sample of outputs, and check whether the gains persist.","tokens_in":20257,"feed_emoji":"🧠","tokens_out":593,"duration_ms":183976,"temperature":0.7,"pith_summary":"Large reasoning models generate explicit thinking traces before answering questions. This paper identifies a failure mode where thinking actually corrupts correct answers: the model knows the right answer without thinking, but generating a reasoning trace introduces entity confusion, unsupported associations, and factual drift that flips the correct answer into a wrong one. The authors call this 'thinking-induced hallucination.' They show this happens in 5-8% of factual QA questions across multiple model scales, and is largely absent in math reasoning. To address it, they propose MARGO, a reinforcement learning method that places both thinking and non-thinking responses to the same question into one comparison group during training. This lets the optimizer see whether thinking actually adds factual value over direct answering, rather than just ranking thinking trajectories against each other. The key mathematical insight is that mixing modes introduces a 'residual value' term into the advantage computation: if thinking helps, it gets reinforced; if thinking hurts, it gets penalized relative to the non-thinking reference.","feed_headline":"Thinking can make AI less factual — now there's a fix","feed_subtitle":"A new RL method uses the model's own direct answers as a reference to detect when reasoning corrupts facts, improving QA accuracy by 2-3%.","key_machinery":"MARGO constructs mixed rollout groups containing both thinking and non-thinking trajectories for each question, then applies standard GRPO advantage normalization across the mixed group. The thinking ratio α controls the proportion of thinking vs non-thinking samples (set to 0.75 in experiments, meaning 6 thinking and 2 non-thinking rollouts per group).","core_discovery":"The paper's central contribution is the decomposition of mixed-mode advantage into two terms: a within-mode advantage (how good this thinking trajectory is compared to other thinking trajectories) and a residual-value adjustment (whether thinking as a whole helps or hurts factuality for this question, measured against the non-thinking baseline). This second term is absent in standard GRPO, which only compares thinking trajectories to each other. By including non-thinking rollouts in the same group, MARGO automatically suppresses thinking that corrupts correct direct answers while preserving thinking that recovers missing knowledge, without requiring a separate classifier or hand-crafted mode","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Reasoning models hallucinate facts by overthinking. MARGO fixes that","When AI thinks too much, facts drift. A new RL method catches it","Mixed-mode RL curbs thinking-induced hallucination in reasoning models","AI reasoning can corrupt correct answers. MARGO learns when to stop","Non-thinking rollouts as reference: new way to tame LRM hallucination"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The same model (Qwen3-32B) serves as both the training reward signal and the evaluation judge, using the same prompt. If this judge has systematic biases toward certain answer formats or phrasings that MARGO's outputs happen to match, the reported gains could be partially artifactual rather than reflecting genuine factual improvement.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning models hallucinate facts by overthinking. MARGO fixes that","When AI thinks too much, facts drift. A new RL method catches it","Mixed-mode RL curbs thinking-induced hallucination in reasoning models","AI reasoning can corrupt correct answers. MARGO learns when to stop","Non-thinking rollouts as reference: new way to tame LRM hallucination","Thinking isn't always better for factual QA — now models can tell","MARGO: RL framework that checks if reasoning actually helps facts","Some AI reasoning overturns correct answers. This method fixes it"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":759,"prompt_tokens":581,"completion_tokens":178,"prompt_tokens_details":null},"tokens_in":581,"tokens_out":178,"duration_ms":18844,"temperature":1.0,"reasoning_tokens":43,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T22:08:18.145567+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Run MARGO with a different judge model for evaluation than for training, or add human evaluation on a sample of outputs, and check whether the gains persist.","supporting_citations":[],"review_version":1}