Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM

Hao Chen; Jiaqi Hu; Junbo Zhao; Qi Zhang; Wentao Ye; Xiaomeng Hu; Zhanming Shen

REVIEW 4 major objections 5 minor 18 references

Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SSC-GRPO rewards reasoning steps that agree across rollouts, cutting the dominant class of LLM reasoning hallucinations.

desk verdict A sensible but thinly evidenced step-reward scheme for GRPO; worth a referee, but the paper's central mechanism is not actually validated. read the letter →

arxiv 2607.18915 v1 pith:DZEJCEA6 submitted 2026-07-21 cs.CL

Xiaomeng Hu , Jiaqi Hu , Hao Chen , Qi Zhang , Zhanming Shen , Wentao Ye , Junbo Zhao This is my paper

classification cs.CL

keywords context-sensitivefactualhallucinationself-consistencygrouprelativepolicyoptimizationreinforcementlearningreasoningstep-levelrewardnaturallanguageinferencemathematical

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that most hallucinations inside LLM reasoning are context-sensitive factual hallucinations: the model has the relevant fact but states it wrongly because of the surrounding reasoning context. To target these, the authors propose SSC-GRPO, which turns step-level self-consistency—whether a step agrees with other sampled rollouts—into a reward signal, rescaling per-token advantages so that consistent steps get credit and inconsistent steps get penalized. This needs no external knowledge base: the model being trained judges its own steps via natural-language-inference-style comparisons. Experiments on math and hallucination benchmarks report consistent gains over a standard GRPO baseline across three base models. The value, if true, is a training signal for factual soundness of intermediate reasoning steps without extra supervision.

What carries the argument

The load-bearing object is Equation (9), the SSC-GRPO objective, where the standard GRPO per-token advantage is multiplied by a per-step weight w_{i,k} that is derived from step-level self-consistency scores (Equations 5–8). The consistency score is computed by prompting the model being trained to classify each step as entailment, neutral, or contradiction relative to each other rollout, aggregated with higher weight for rollouts whose final answer is correct. This step weight redistributes advantage within a rollout while preserving its total, so the model increases probability mass on steps that are self-consistent and lowers it on steps that diverge.

What would settle it

Construct a set of reasoning traces where the same factually wrong step appears in every rollout (for example, by prompting the model with a fixed flawed intermediate step), train with SSC-GRPO, and measure whether the wrong step's probability changes. If consistently-wrong steps receive positive advantage or the final-answer error persists, the self-consistency reward is not locating hallucinations. A simpler measurement: compute the correlation between SSC-GRPO step rewards and human annotations of step correctness on a new step-level labeled dataset; a zero or negative correlation would ref

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that context-sensitive factual hallucinations are the dominant error type in long reasoning traces, and that self-consistency across rollouts locates them well enough to serve as a step-level reward. The method computes, for each reasoning step, an NLI-style consistency score against all other rollouts in the same group, weighted by whether those rollouts reach a correct final answer. It then rescales the GRPO advantage within each step, keeping the total advantage of a rollout fixed, so optimization differentially rewards steps that are both consistent and part of a correct trajectory. The result is a parameter-free-in-external-knowledge wa

Load-bearing premise

The method assumes that the model's own cross-rollout self-consistency judgment, made by the model being trained, reliably identifies the hallucinated step, so a wrong step that agrees with other rollouts would be reinforced rather than penalized.

Editorial extensions

If this is right

If SSC-GRPO works as claimed, training on plain math questions can mitigate mid-reasoning factual errors without external fact-checking tools, reducing reliance on expensive verifiers.
The step-level reward offers a dense credit-assignment signal, potentially improving sample efficiency compared to outcome-only rewards in long reasoning tasks.
The method's reliance on self-consistency suggests that reasoning models can be made more robust to contextual interference by rewarding adherence to the fact already present in the model's own knowledge.
The reported reduction in context-sensitive factual hallucinations could improve trustworthiness in deployed chain-of-thought systems, not just benchmark accuracy.
The framework naturally extends the GRPO rollout structure, so any existing GRPO-based reasoning trainer can adopt the step-level rescaling with minimal changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One extension the authors do not explore: the same step-consistency signal could be used during decoding, not just training, as a lightweight detector that re-samples or flags suspicious intermediate steps.
Another: the reward's quality depends on the judge's calibration; if step-level NLI accuracy is low on a different domain, the method may need a calibrated judge or a confidence threshold. The paper's own SNLI results suggest the base model is weak on neutral-vs-entailment distinctions.
The hallucination taxonomy itself was produced by an LLM judge; a human-annotated step-level dataset would test whether the 70% figure holds, and whether the 'context-sensitive' category is an artifact of the judge's rewriting procedure.
Because the method only separates steps that agree from steps that diverge, it may be unable to correct a consistently wrong step shared by all rollouts; a multi-run or temperature schedule that encourages diversity could expose this failure mode.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a modest combination of known ideas—SelfCheckGPT-style NLI consistency, GRPO, advantage reweighting—into a step-level reward. That is not a criticism. The method is clearly described, the motivation is sensible, and the authors did the baseline work: same dataset, same framework, three model families, plus an ablation. The context-sensitive factual hallucination taxonomy is the more interesting contribution, and the context-augmentation experiments in Appendix D provide some independent support that contextual interference exists.

The soft spots are real but not fatal. The self-consistency judge is validated on MathNLI but not on the actual training distribution of reasoning steps; the base model at 53.6% on SNLI is worrying. The paper argues that mispredictions are mostly entailment/contradiction judged as neutral, which gives zero weight and thus no negative signal; that is a reasonable defense but not a substitute for measuring the reward on hallucinated vs. correct steps. The stress-test note is right that a consistently wrong step could be rewarded—the outcome weighting helps only if correct rollouts exist with entailing premises. The paper never checks this.

The taxonomy is also not human-validated; cross-model agreement shows reproducibility, not correctness. And the hyperparameters λ and ε_c are tuned on the same final benchmarks, with no error bars on AIME Pass@32 (only two runs). Some cells worsen. These are standard weaknesses for this kind of RL paper, but they matter because the SOTA claim is modest (1–2% average) and the method adds compute overhead.

All in all, it's a serious paper that a good referee could improve. The central idea is plausible and the paper is honest about limitations. I'd send it to peer review with the requirement that the authors either validate the step reward against human-labeled hallucination steps or show consistency correlates with correctness on a held-out set, and report variance.

Referee Report

4 major / 5 minor

Summary. This paper analyzes hallucinations in long LLM reasoning traces and introduces a new category, "context-sensitive factual hallucination," in which the model possesses the relevant knowledge but makes a factual error because of surrounding reasoning context. It then proposes SSC-GRPO, an extension of GRPO that computes step-level rewards by asking the training model itself to judge each reasoning step against other rollouts via an NLI-style consistency check, and uses these rewards to rescale per-token advantages (Eqs. (5)--(9)). The paper reports a taxonomy in which context-sensitive factual hallucinations account for roughly 67--70% of hallucinations (Fig. 2), and presents experiments on Qwen3-4B-Base, Qwen3-4B-Instruct, and Llama3-8B-Instruct across GSM8K, MATH-500, AIME-24/25, HaluEval, TruthfulQA, and HotpotQA, claiming state-of-the-art performance over GRPO, FSPO, and SEED-GRPO.

Significance. If valid, the contribution is useful: it proposes an external-knowledge-free step-level reward for policy optimization and offers a concrete, testable taxonomy of reasoning hallucinations. The method is described in enough detail to reproduce, with prompts in Appendix I, hyperparameters in Appendix A, and a cross-model robustness check of the taxonomy in Appendix C. The augmented-context experiments in Appendix D are a nice additional probe. However, the two load-bearing empirical components are not yet established: the hallucination taxonomy in Section 3.3 is built entirely on an LLM-judge pipeline with no human validation, and the self-consistency judge in Section 5.5 is validated only on standard NLI benchmarks, not on the actual step-versus-full-rollout judgment used during training. The reported gains are also small, and the paper provides no error bars or significance testing. If the judge does not separate hallucinated steps from correct steps on the training distribution, Eq. (8) may reward consistently wrong steps, so the central mechanism could reinforce the errors it is designed to remove.

major comments (4)

[§5.5, Eqs. (5)–(6)]
[§3.3, Figure 2]
[Table 1, §5.1]
[Appendix B, Table 5]

minor comments (5)

[Eq. (2)]
[§4.3, Eq. (8)]
[Appendix I]
[Table 4]
[Table 10]

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity; one minor self-referential validation step in the consistency-tracking analysis.

other [§4.2 Eq. (5)–(6) and §5.4 / Fig. 4]
"SCi,k←j = NLI πold (oj, oi,k) ... rsc i,k = Σ_{j≠i} αj · SC i,k←j ... We expect that, as training progresses, reasoning consistency will increase, thereby reducing hallucinations. To verify this hypothesis, we track the proportion of consistent and inconsistent steps over the course of training. ... This indicates that reinforcement learning progressively enhances reasoning consistency and reduces hallucinations."

The quantity tracked in Fig. 4 is exactly the step-level reward r^sc defined in Eq. (6) and optimized in Eq. (9). Observing that this quantity rises during training is a direct effect of maximizing it, so it cannot independently substantiate a reduction in factual hallucinations unless one already assumes that cross-rollout NLI consistency equals hallucination detection—the very premise the method is intended to establish. This is a circular validation of the auxiliary consistency-dynamics evidence, although the main benchmark results are external and non-circular.

full rationale

The paper's central derivation is not circular by construction: the step-level reward in Eqs. (5)–(6) is computed from cross-rollout NLI consistency, not from the evaluation labels, and the main results are measured on external math and hallucination benchmarks. No load-bearing self-citation chain appears; GRPO and self-consistency checking are attributed to prior external work (Shao et al.; Manakul et al.). The main validity risk is calibration rather than circularity: §5.5 concedes that training-time step judgments have no ground-truth labels and validates only on standard NLI sets, with Qwen3-4B-Base at 53.6% SNLI accuracy. This weakens the claim that consistency can locate hallucinated steps, but it does not make the benchmark evaluation circular. The only mild circularity is the use of the optimized consistency signal itself as evidence of hallucination reduction in Fig. 4, which keeps the score just above zero.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central method depends on the self-consistency-as-factuality proxy and the model's own NLI judgments; free parameters λ, ε_c, α± are selected by hand or ablation on the final benchmarks. There are no invented physical entities; the only invented construct is the hallucination taxonomy category, which lacks independent evidence.

free parameters (5)

α+ (correct-rollout weight) = 0.8
Hand-set in Eq. (7) to weight consistency signals from outcome-correct rollouts; no principled choice or search reported.
α− (incorrect-rollout weight) = 0.2
Hand-set in Eq. (7); balances influence of incorrect rollouts.
λ (advantage rescale factor) = 0.05
Selected by ablation (Table 5) on the final evaluation benchmarks; training collapses without scaling/clipping (Appendix B).
ε_c (rescale clip ratio) = 0.2
Selected by ablation on final benchmarks; larger/smaller values harm stability (Appendix B).
NLI label mapping = +1 entailment / 0 neutral / −1 contradiction
Chosen by convention in Eq. (5); neutral contributes zero to the consistency reward.

assumptions (4)

domain assumption Cross-rollout self-consistency is a valid proxy for step-level factual correctness in reasoning.
Central to the reward; wrong-but-consistent steps would be reinforced. Invoked in §4.2.
domain assumption The policy model π_old is a reliable NLI judge of its own reasoning steps.
Used in Eq. (5). §5.5 validates on MathNLI only; Appendix G shows base model SNLI accuracy 53.6%.
domain assumption LLM-based hallucination classification and the isolated-query test reliably measure whether the model possesses the knowledge.
§3.3 uses Qwen3-235B as judge to locate, rewrite, and re-query; no human annotation or exact-match verification.
domain assumption Reasoning traces can be decomposed into well-defined steps.
Step segmentation is not specified; Eq. (9) maps tokens to steps via k(t) without an algorithm.

invented entities (1)

Context-sensitive factual hallucination (construct)
purpose: Categorizes reasoning errors where the model answers correctly in isolation but errs in context; motivates the reward design.
Operationalized through the paper's own LLM-judge pipeline (§3.3); no independently human-labelled benchmark or external measurement.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM." pith.science (2026). https://pith.science/paper/DZEJCEA6

@misc{pith2026260718915,
  author       = {Pith},
  title        = {Pith review of: Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DZEJCEA6}},
  note         = {Machine review of arXiv:2607.18915}
}

read the original abstract

With the rapid advancement of large language models (LLMs), modern systems not only possess strong foundational capabilities and extensive knowledge, but can also solve complex problems via long, multi-step reasoning. However, as reasoning traces become longer, LLMs may produce a substantial amount of hallucinated content during the reasoning process, which is often difficult to detect. In this work, we conduct a fine-grained analysis of hallucinations arising in LLM reasoning and find that the reasoning traces are particularly prone to Context-Sensitive Factual Hallucinations: cases where the model actually has the relevant knowledge, yet makes factual errors due to contextual interference during reasoning. To address this issue, we propose Step-level Self-Consistency Group Relative Policy Optimization (SSC-GRPO), which assigns step-level rewards to reasoning traces by computing self-consistency scores of individual steps across multiple rollouts. Compared with prior methods, SSC-GRPO achieves state-of-the-art performance on both mathematical reasoning benchmarks and hallucination leaderboards. Our results offer a new perspective for detecting and mitigating hallucinations in the reasoning process of large language models.

Figures

Figures reproduced from arXiv: 2607.18915 by the authors.

**Figure 1.** Context-Sensitive Factual Hallucination in Large Language Models. During the reasoning process, large language models may produce factual errors that subsequently lead to incorrect final answers, as illustrated in Scenario A. However, when the erroneous part is extracted and reformulated as an independent question, the model is able to answer it correctly, as shown in Scenario B. This phenomenon indicates that the m… view at source ↗

**Figure 2.** Analysis of hallucination types in reasoning models. Among all cases where hallucinations occur, context-sensitive factual hallucinations account for the majority—nearly 70%. The other categories of hallucinations make up only a small fraction. As model capability improves, knowledge-missing hallucinations become increasingly rare, yet a substantial number of context-sensitive factual hallucinations still persist. … view at source ↗

**Figure 3.** Overview of Our Method. We first follow the standard GRPO procedure to sample a group of rollouts. Then, for each reasoning step in each rollout, SSC-GRPO computes a consistency score with respect to other rollouts, which serves as a step-level reward. Based on these step-level rewards, we rescale the rollout-level advantages by allocating higher advantages to reasoning steps with higher consistency scores. Finally,… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Consistency Ratio over Training. As training progresses, the proportion of consistent steps steadily increases while the proportion of inconsistent steps decreases, demonstrating that hallucinations are gradually reduced during training. Model Accuracy F1-score Qwen3-4…

**Figure 5.** Figure 5: presents a comparison example before and after augmentation. Although extra conditions are added, these conditions are redundant and irrelevant to the actual solution. Nevertheless, the model produces an incorrect answer, indicating that it is indeed influenced by the…

**Figure 6.** Figure 6: The Confusion Matrix of NLI task for models I Prompts Template In this section, we provide detailed prompt templates used in various stages of our experiments. These templates ensure the consistency and reproducibility of our experimental setup. Figures 7 and 8 show…

**Figure 7.** Figure 7: System prompt for detecting hallucinations during reasoning in Section [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: User prompt for detecting hallucinations during reasoning in Section [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Prompt for augmented dataset in section D [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

**Figure 10.** Figure 10: Prompt for NLI judge during training [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 5 linked inside Pith

[1]

To find the proper divisors of284, we first factorize284

Find the proper divisors of284. To find the proper divisors of284, we first factorize284. (284 = 22×71). The divisors of284are:1,2,4,71,142,284. The proper divisors of 284 are all divisors except284itself, so they are:1,2,4,71,142
[2]

Now, we need to find the sum of the proper divisors of220

Calculate the sum of the proper divisors of 284.[1 + 2 + 4 + 71 + 142 = 220]3. Now, we need to find the sum of the proper divisors of220. First, find the proper divisors of220. To find the proper divisors of 220, we first factorize 220.(220 = 22×5×11)The divisors of220are:1,2,4,5,10,11,20,22,44,55, 110,220. The proper divisors of220are all divisors except...
[3]

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal

Math natural language inference: this should be easy! InProceedings of the 14th Joint Conference on Lexical and Computational Semantics (* SEM 2025), pages 179–188. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630. Aaron Grattafiori, Abhi...

arXiv 2025
[4]

inclusionAI

Halucheck: Explainable and verifiable automa- tion for detecting hallucinations in llm responses.Ex- pert Systems with Applications, 272:126712. inclusionAI. 2025. Areal-boba-data. https: //huggingface.co/datasets/inclusionAI/ AReaL-boba-Data. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and P...

arXiv 2025
[5]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empiri- cal methods in natural language processing, pages 236...

arXiv 2018
[6]

Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Ke- qing He, Zejun Ma, and Junxian He

Are reasoning models more prone to halluci- nation?arXiv preprint arXiv:2505.23646. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Ke- qing He, Zejun Ma, and Junxian He. 2025. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. InSecond Conference on Language Modeling. Yaocheng Zhang, Haohuan Huang, Zijun S...

arXiv 2025
[9]

hallucination localization/rewriting

Calculate the sum of the proper divisors of220. [1 + 2 + 4 + 5 + 10 + 11 + 20 + 22 + 44 + 55 + 110 = 284] The sum of the proper divisors of the sum of the proper divisors of284is284. <answer> 284</answer> Table 10:The case of our method in Qwen3-4B-base Entailment Neutral Contradiction Predicted label Entailment Neutral ContradictionTrue label 291 385 2 1...
[10]

Final Answer

Determine whether the final answer is correct - Extract the standard final answer from ‘<Standard_Solutions>’ (prefer ‘\boxed{...}’; otherwise take the last explicitly stated final value). - Extract the model’s final submitted answer from ‘<Model_Solution>’ (use the final conclusion/the last ‘\boxed{...}’/“Final Answer ...” as the reference; ignore interm...

Show all 18 references

[11]

earliest hallucinated sentence that directly affects subsequent reasoning

If the final answer is incorrect: locate the “earliest hallucinated sentence that directly affects subsequent reasoning” - Treat ‘<Model_Solution>’ as consisting of multiple sentences (or multiple lines/steps). Audit it sentence by sentence. - Find the first sentence that will...
[12]

AB= 5 ” when “AB= 3

Determine the hallucination type: context inconsistency vs factual error - Context inconsistency (‘context inconsistency’): the sentence contradicts the problem conditions, previously established conclusions in the preceding derivation, symbol definitions, variable meanings, e...
[13]

independently solvable

If it is a factual error: rewrite the hallucinated sentence into a new, “independently solvable” problem ‘hallucination_question’ - Goal: rewrite the erroneous step that caused the mistake into a standalone mini-problem, so that one can verify/derive the correct conclusion bas...
[14]

other parts

Add several conditions/background/intermediate conclusions in the **same domain** as the original problem (from “other parts”), which should be **misleading** but **irrelevant to solving the current question**, and must not change the original answer
[15]

sub-question is embedded in a larger context,

Create the sense that the “sub-question is embedded in a larger context,” but **do not use** explicit phrases such as “big problem/subproblem/step”; embed only via natural sentences
[16]

Compute. . . /What is. . . ,

**Keep only one question**, and the **last sentence** must ask that question; **keep the asking style consistent with the original** (e.g., if the original is “Compute. . . /What is. . . ,” the augmented version must follow the same paradigm)
[17]

seemingly related but actually irrelevant

Do not introduce conflicting conditions that would change the answer: do not alter given dimensions/definitions/constraints; do not replace key numeric values; you may introduce “seemingly related but actually irrelevant” definition reminders, side theorems, neighboring concep...
[18]

?” or English “?

Add 1–3 distractors of moderate length; preferably insert them in the front or middle of the question body with natural transitions. **[Invariance & Consistency]** - The ‘answer’ **must match the input exactly** (character-for-character, including formatting/units/case). - If ...
[2021]

Valeria de Paiva, Qiyue Gao, Hai Hu, Pavel Kovalev, Yikang Liu, Lawrence S Moss, and Zhiheng Qian

Training verifiers to solve math word prob- lems.arXiv preprint arXiv:2110.14168. Valeria de Paiva, Qiyue Gao, Hai Hu, Pavel Kovalev, Yikang Liu, Lawrence S Moss, and Zhiheng Qian

arXiv
[2025]

Chain-of-thought prompting obscures halluci- nation cues in large language models: An empirical evaluation.arXiv preprint arXiv:2506.17088. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reii...

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

To find the proper divisors of284, we first factorize284

Find the proper divisors of284. To find the proper divisors of284, we first factorize284. (284 = 22×71). The divisors of284are:1,2,4,71,142,284. The proper divisors of 284 are all divisors except284itself, so they are:1,2,4,71,142

[2] [2]

Now, we need to find the sum of the proper divisors of220

Calculate the sum of the proper divisors of 284.[1 + 2 + 4 + 71 + 142 = 220]3. Now, we need to find the sum of the proper divisors of220. First, find the proper divisors of220. To find the proper divisors of 220, we first factorize 220.(220 = 22×5×11)The divisors of220are:1,2,4,5,10,11,20,22,44,55, 110,220. The proper divisors of220are all divisors except...

[3] [3]

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal

Math natural language inference: this should be easy! InProceedings of the 14th Joint Conference on Lexical and Computational Semantics (* SEM 2025), pages 179–188. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy.Nature, 630(8017):625–630. Aaron Grattafiori, Abhi...

arXiv 2025

[4] [4]

inclusionAI

Halucheck: Explainable and verifiable automa- tion for detecting hallucinations in llm responses.Ex- pert Systems with Applications, 272:126712. inclusionAI. 2025. Areal-boba-data. https: //huggingface.co/datasets/inclusionAI/ AReaL-boba-Data. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and P...

arXiv 2025

[5] [5]

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning

Qwen3 technical report.arXiv preprint arXiv:2505.09388. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 conference on empiri- cal methods in natural language processing, pages 236...

arXiv 2018

[6] [6]

Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Ke- qing He, Zejun Ma, and Junxian He

Are reasoning models more prone to halluci- nation?arXiv preprint arXiv:2505.23646. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Ke- qing He, Zejun Ma, and Junxian He. 2025. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. InSecond Conference on Language Modeling. Yaocheng Zhang, Haohuan Huang, Zijun S...

arXiv 2025

[7] [9]

hallucination localization/rewriting

Calculate the sum of the proper divisors of220. [1 + 2 + 4 + 5 + 10 + 11 + 20 + 22 + 44 + 55 + 110 = 284] The sum of the proper divisors of the sum of the proper divisors of284is284. <answer> 284</answer> Table 10:The case of our method in Qwen3-4B-base Entailment Neutral Contradiction Predicted label Entailment Neutral ContradictionTrue label 291 385 2 1...

[8] [10]

Final Answer

Determine whether the final answer is correct - Extract the standard final answer from ‘<Standard_Solutions>’ (prefer ‘\boxed{...}’; otherwise take the last explicitly stated final value). - Extract the model’s final submitted answer from ‘<Model_Solution>’ (use the final conclusion/the last ‘\boxed{...}’/“Final Answer ...” as the reference; ignore interm...