{"id":"88f8ccaf-4433-448c-9bb2-3380672a5316","arxiv_id":"2507.22940","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper reports that a fact-checker trained on replaced entities, used both as a reward and as the evaluation metric, raises measured step-factuality of small open LLMs by up to 49.9 percentage points.","lead":"The paper introduces RELIANCE, a pipeline that checks each step of an LLM's visible reasoning for factual errors, trains models to avoid those errors through reinforcement learning, and inspects how internal activations change. If the method truly works, it would give a practical way to make reasoning models state fewer false facts, which matters for medical, legal, and scientific advice.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 49.9% reasoning-step factuality gain is measured with the same classifier that serves as the RL reward (Eq. 9 vs.","rationale":"The reader's weakest assumption — that the entity-substitution-trained fact-checker is a valid and generalizable factuality measure when used both as metric and reward — is exactly the load-bearing point. My stress-test confirms this concern is not speculative: Eq. (9) and §4.4 use the same classifier, so the 49.9-point gain is a measure of optimization against the reward, not of external factual correctness. The lack of human validation on real model reasoning makes this circularity decisive rather than merely a methodological nicety. The case study in §6/Figure 6 adds internal evidence against the claim, because the 'enhanced' output for the pediatric morphine query retains the very factual errors the method is supposed to fix. Independent support in the paper — the synthetic dataset construction, the LoRA training setup, the ablation study, and the released models — does not rescue the central claim, because none of these validates the classifier against human judgment on realistic reasoning chains. Therefore the reported improvements do not support the conclusion that RELIANCE enhances factual robustness in a meaningful sense, and the reader's REJECT verdict should stand unchanged.","tokens_in":38492,"tokens_out":3516,"duration_ms":34984,"concrete_test":"Recompute the headline SFA results on the 100 held-out factuality-enhancement test instances from §4.1 using independent human annotation instead of the trained classifier: have three domain-expert annotators label each segmented reasoning step (segmentation as in Appendix A.2) as factually consistent or not for both the base model and the RELIANCE-enhanced model, then compute SFA from these labels. If the human-labeled improvement for Qwen2.5-0.5B-Open-R1-Distill is substantially below the reported 49.90 percentage points (or negative), the claimed gain is an artifact of optimizing the evaluation metric. As a secondary check, also run an off-the-shelf factual consistency scorer (e.g., FactScore or SAFE) on the same generations to see whether the gain transfers to an independent measure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of RQ2 is that RELIANCE improves factual consistency of reasoning steps by up to 49.90 percentage points. The load-bearing weakness is that the evaluation metric and the training reward are the same model. In §4.4, Reasoning Chain Steps Factuality Accuracy (SFA) is defined as the mean over steps of f_θ(q, t_i), where f_θ is the fact-checking classifier. In §3.2, Eq. (9) defines the factuality reward R_fact as the proportion of steps for which the same classifier's probability exceeds threshold τ. Thus the policy is directly optimized to maximize the exact quantity later reported as the headline improvement. Any increase in SFA under this setup is expected by construction and does not by itself demonstrate that reasoning became more factually correct. The classifier's own validity is only established on synthetic entity-substitution counterfactuals (§3.1, Table 3); no human evaluation on real model-generated reasoning is reported, so systematic differences between the synthetic training distribution and actual reasoning errors could make the reward an easily gameable proxy. The paper's own §6 case study deepens this concern: the post-training response for the pediatric morphine query shown in Figure 6 still contains the same critical factual errors the framework claims to remove — morphine described as a stimulant, a 1-year-old's weight given as 2.2 kg, and a recommended 0.22 mg dose for vomiting — directly contradicting the claimed factual enhancement. Downstream benchmark changes (Figure 2) are small and no significance tests are reported, so the 'maintain or improve final performance' claim is also not established. The ablation study (Table 7) is consistent with reward hacking: removing the fact-checking reward collapses SFA to 43.53%, but because SFA is measured with the same classifier, this only shows the model learned to satisfy the classifier, not that it learned facts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RELIANCE, a framework for detecting and improving factual accuracy in the intermediate reasoning steps of LLMs. It comprises three components: (1) a fact-checking classifier trained on counterfactually augmented (entity-substituted) reasoning chains; (2) a GRPO-based reinforcement learning procedure with multi-dimensional rewards combining factuality, semantic similarity, format compliance, and length constraints; and (3) a mechanistic interpretability analysis of hidden-state activations across reasoning steps. The authors evaluate reasoning-step factuality for ten white-box and black-box models, report that even strong models achieve only ~82% step-level factuality, and claim that RELIANCE improves reasoning-step factuality by up to 49.90 percentage points while preserving final-answer performance on Math-500, AIME-2024, GPQA, and LiveCodeBench. The central claim is that the RL procedure genuinely increases the factual correctness of intermediate reasoning steps.","tokens_in":38794,"tokens_out":5525,"duration_ms":50253,"significance":"The problem addressed is important and timely: visible reasoning chains that are fluent but factually flawed can mislead users in high-stakes settings, and most existing evaluations focus only on final answers. The paper's pipeline is broad and touches on a real gap. However, the core result is not supported as presented. The evaluation metric is identical to the reward used for training, the fact-checker is validated only on synthetic entity-substitution data, and the paper's own case study shows the enhanced model reproducing the same clinically dangerous errors it claims to remove. These issues are load-bearing, so the contribution as stated is not established. The mechanistic interpretability analysis is descriptive and may interest specialists, but it does not provide an independent check on factuality. The paper would need a substantially redesigned evaluation to support its claims.","major_comments":[{"comment":"The primary evaluation metric, Reasoning Chain Steps Factuality Accuracy (SFA), is defined in Section 4.4 as Acc(T) = 1/n * sum_i f_theta(q, t_i), which is exactly the quantity optimized as the factual reward R_fact in Eq. (9) (with a threshold and indicator function). Because the policy is trained to maximize R_fact, the reported SFA improvements in Table 5, including the headline +49.90% gain, are expected by construction and do not constitute independent evidence that reasoning steps became more factually correct. To support the RQ2 claim, the authors must evaluate with a metric not used in training, such as a separately trained classifier, human annotation, or verification against an external knowledge base.","section":"Section 4.4 vs. Eq. (9)"},{"comment":"The paper's own case study contradicts the claimed enhancement. The post-RELIANCE response to the pediatric morphine query still states that morphine is a stimulant, that a 1-year-old weighs 2.2 kg, and recommends 0.22 mg of morphine for vomiting, and Figure 6 explicitly annotates all of these as errors. The text describes 'remarkable improvements' and says RELIANCE 'prioritized safety by avoiding speculative dosing recommendations,' yet the displayed enhanced output contains the same clinically dangerous errors as the baseline. This internal inconsistency directly undercuts the central RQ2 claim.","section":"Section 6, Figure 6"},{"comment":"The fact-checking classifier is trained and evaluated solely on synthetic entity-substitution counterfactuals. No evaluation is performed on real model-generated reasoning chains, and no human agreement study is reported. Without evidence that the classifier generalizes beyond this synthetic distribution, its validity as a measure of reasoning-step factuality is not established. When the same classifier is used both as the training reward and as the evaluation metric, the circularity is compounded and the reported gains become uninterpretable.","section":"Section 3.1, Table 3"},{"comment":"The claim that RELIANCE maintains or improves final-answer performance is supported for only one of the four enhanced models (DeepSeek-R1-Distill-Qwen-1.5B) on the four benchmarks. The other three models are not shown, and no statistical significance tests or confidence intervals are provided. This is insufficient to support the general claim stated in the abstract and Section 5.2 that performance is 'maintained or slightly improved' across the approach.","section":"Section 5.2, Figure 2"}],"minor_comments":[{"comment":"The reported 'Overall Acc.' for DeepSeek-R1-Distill-Qwen-7B is 71.46%, which is lower than every per-temperature value listed (70.37%–78.56%), so the aggregation appears inconsistent or misreported.","section":"Table 4"},{"comment":"The KL divergence formula appears to be written in the wrong direction: D_KL(pi_theta || pi_ref) should involve pi_theta/pi_ref - log(pi_theta/pi_ref) - 1, not the inverse ratio shown, which may also violate non-negativity.","section":"Equation (8)"},{"comment":"The word 'False' is misspelled as 'Fasle' in Figure 1, and several labels are otherwise inconsistent with the text (e.g., 'Mechanical Interpretability' vs. 'mechanistic interpretability').","section":"Figure 1"},{"comment":"The model name 'FactCompass' appears in Figure 6 but is never defined in the text; it appears to be a leftover from another tool and should be removed or explained.","section":"Figure 6"},{"comment":"The caption mentions shaded cells indicating relative improvement, but no shading is visible in the table as provided; this should be corrected for clarity.","section":"Table 5"},{"comment":"The ablation study reports that removing the format or length reward causes training to fail completely, and the fact-checking reward alone also fails. This suggests the framework is brittle and that the attribution of the reported gains to the factual reward is not straightforward; this point deserves fuller discussion in the main text.","section":"Appendix A.3, Table 7"}],"recommendation":"reject","confidential_remarks":"The most serious issue is the internal contradiction between the abstract's claim of 'up to 49.90% improvement' and the paper's own Figure 6, where the post-training response still contains the same clinically dangerous errors that the framework claims to remove. Combined with the reward/metric circularity (Eq. 9 vs. Section 4.4), the central empirical claims are not supported by the evidence presented. These are not local presentation issues; they call for a fundamental re-evaluation of the method and its measurement. I therefore recommend rejection, even though the target problem is clearly important."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result doesn't survive contact with the paper. The reasoning-step accuracy metric (SFA) in Section 4.4 is exactly the same fact-checking function f_theta used as the factual reward in Eq. 9, so training to maximize the reward is training to maximize the metric. The 49.9% improvement is therefore expected by construction and tells us little about real factual accuracy. More damning: the case study in Figure 6 shows the post-training morphine response still calls morphine a stimulant, gives a 1-year-old's weight as 2.2 kg, and recommends 0.22 mg for vomiting. The paper claims this is a 'remarkable improvement' but the response shown is clinically dangerous. That internal contradiction is hard to wave away.\n\nWhat is genuinely useful here is the RQ1 evaluation. Measuring reasoning-step factuality across ten models (including black-box ones) with a consistent classifier is a nice empirical contribution, and the numbers like GPT-o1 at 82.57% are interesting even if the classifier's validity rests only on synthetic entity-swapped data. The counterfactual augmentation pipeline is straightforward but sensible, and the LoRA-based fact-checker beats prompt/CoT baselines by a lot, which is a real if narrow result. The multi-reward GRPO integration is a reasonable combination of known pieces, and the ablation at least shows the factual reward is doing something—though what it is doing is inflating the classifier's score, not necessarily improving facts.\n\nOther soft spots: the downstream benchmark changes are small and no significance tests are reported, so 'maintain or improve' is not established. The mechanistic analysis is descriptive and does not feed back into the method. The fact-checker has no human validation on real model reasoning, only on synthetic substitutions, so we do not know whether it rewards genuine factuality or surface patterns. The paper also does not compare against recent step-level factuality methods, which weakens the novelty claim.\n\nWho is this for? Someone working on factuality evaluation might get a useful data point from the RQ1 numbers, and the circular-evaluation failure is a good teaching example. But the central enhancement claim is not credible as presented. I would send this to peer review only because the topic matters and a serious referee could force the authors to redo the evaluation with independent ground truth or human annotation; if they cannot show improvement on a non-circular metric, it should be rejected. As is, the paper's own evidence contradicts its conclusion.","headline":"The 49.9% factuality gain is measured with the same classifier used as the RL reward, and the paper's own case study shows the enhanced model still making the same medical errors.","tokens_in":39422,"tokens_out":3272,"would_cite":false,"duration_ms":32128,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a counterfactual-trained fact-checker plus GRPO reinforcement learning raises reasoning-step factuality by up to 49.9 points.","keywords":["reasoning factuality","chain-of-thought","counterfactual augmentation","GRPO","reinforcement learning","fact-checking classifier","mechanistic interpretability","LLM evaluation"],"falsifier":"Take a held-out set of naturally occurring reasoning errors (human-annotated, not entity-substituted), and compare the fact-checker's verdicts and the RELIANCE-trained model's step accuracy against human labels. If the classifier's agreement with humans on natural errors is no better than chance, or if the enhanced model's outputs are rated by blind experts as more evasive rather than more accurate, the central claim is falsified.","tokens_in":38270,"feed_emoji":"🧠","tokens_out":6647,"duration_ms":59529,"temperature":0.7,"pith_summary":"This paper tries to establish that the visible intermediate reasoning steps of large language models are pervasively factually unreliable, even when the final answer is correct, and that this can be fixed. The proposed RELIANCE framework trains a fact-checking classifier on counterfactually corrupted reasoning chains, then uses that classifier as a reward in a GRPO reinforcement loop. On ten models, the authors report that improving reasoning-step factuality by up to 49.90 percentage points does not degrade final-answer performance on Math-500, AIME-2024, and GPQA. If right, this redirects LLM evaluation and training away from final answers and toward the reasoning users actually see.","feed_headline":"Fact-checking reward lifts LLM reasoning accuracy by 49.9 points","feed_subtitle":"A counterfactual-trained classifier plus RL makes visible thinking more factual without hurting final answers.","key_machinery":"The load-bearing object is the fact-checking classifier: a small LoRA-adapted language model trained with supervised fine-tuning on 38,539 counterfactually augmented reasoning chains, where entities are replaced by same-type entities to create plausible but false statements. This classifier is then reused as the factual reward inside a GRPO reinforcement learning objective, alongside semantic, format, and length rewards. The mechanism works by making factual consistency a per-step optimization signal rather than a final-output property. The activation trajectory metrics (mean step distance, angular deviation, and adjacent-step cosine similarity) serve as the interpretability lens that connects training-induced behavior changes to internal representational changes.","core_discovery":"The central claim is that reasoning-step factuality is a measurable, trainable property distinct from final-answer accuracy. The authors build a binary classifier that labels each reasoning step as factually consistent or not, training it on chain-of-thought data where named entities have been replaced by same-type entities to create subtle counterfactual errors; the best LoRA-tuned version reaches 97.20% accuracy on their test set. They then feed this classifier's per-step verdicts into GRPO as one of four rewards (factual, semantic-similarity, format, length), reporting that the resulting policy improves step factuality from 42.20% to 92.10% on the weakest small model and from 68.53% to 82.65% on a 1.5B model, while extractive-match scores on standard benchmarks stay flat or improve. Mechanistic analysis of hidden-state activations shows that enhanced models have shorter step-to-step trajectory distances, smaller rotation angles, and higher adjacent-step cosine similarity, which the paper interprets as more coherent and semantically grounded reasoning.","pith_inferences":["Editorial inference: because the fact-checker is trained only on entity-substitution counterfactuals, it may miss errors that do not involve named entities, such as wrong causal links or miscalculations; a natural extension is to generate counterfactuals by corrupting relations, quantities, and logical connectives.","Editorial inference: if the classifier becomes the optimization target, the policy could learn to game it by producing evasive, hedged reasoning that the classifier labels 'true' while a human would call it unhelpful; the safety case study hints at this trade-off.","Editorial inference: the activation 'aha moment' signature could be exploited as a training signal or regularizer to encourage epistemic caution and step reconsideration, beyond what the current four-reward design achieves."],"forward_implications":["Reasoning chains become auditable: each step can be scored by a fact-checking classifier, so 'correct answer but wrong thinking' becomes a detectable failure mode rather than an invisible one.","Small models can be brought closer to large-model factuality: the 0.5B model jumps from 42.20% to 92.10% step factuality after RELIANCE training, suggesting parameter-efficient RL is a viable reliability route.","RL rewards for reasoning quality can be multi-dimensional without sacrificing benchmark accuracy; Math-500, AIME-2024, GPQA, and LiveCodeBench scores are maintained or slightly improved.","Activation trajectory patterns (reduced divergence, higher step coherence) correlate with factual correctness, offering observable internal signatures of trustworthy reasoning."],"supporting_citations":[{"why":"Supplies the GRPO algorithm that the authors adapt as the reinforcement learning backbone for factuality enhancement.","marker":"[24]"},{"why":"Provides the large reasoning model used to generate the 6,000 chain-of-thought Q&A pairs that seed both the fact-checking dataset and enhancement training data.","marker":"[7]"},{"why":"Provides the named-entity-recognition model used for entity substitution in counterfactual augmentation.","marker":"[60]"},{"why":"Supplies the LoRA adaptation method used to train the fact-checking classifier efficiently.","marker":"[62]"},{"why":"Defines the chain-of-thought prompting format for reasoning chains and serves as the strongest non-fine-tuned baseline in fact-checking comparison.","marker":"[59]"},{"why":"Provides the LLM-as-a-judge procedure used to filter the 6,000 reasoning samples used for enhancement training.","marker":"[67]"},{"why":"Provides the base training framework that the authors extend with their multi-dimensional reward design.","marker":"[68]"}],"fun_headline_variants":["RL with fact-check reward lifts LLM reasoning truth by 49.9%","Counterfactual-trained classifier plus RL boosts step-level factuality","LLM chain-of-thought factuality improved 49.9% via new RL method","Turning LLM reasoning fact-checkable: up to 49.9% accuracy gain","Measure and train reasoning factuality in LLMs, up to +49.9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the fact-checking classifier being a valid measure of reasoning-step factuality; if the entity-substitution training makes it blind to other error types, or if the policy learns to fool it, then the reported 49.90-point gain is not genuine factual improvement.","fun_headline_variants_meta":{"raw":{"variants":["RL with fact-check reward lifts LLM reasoning truth by 49.9%","Counterfactual-trained classifier plus RL boosts step-level factuality","LLM chain-of-thought factuality improved 49.9% via new RL method","Turning LLM reasoning fact-checkable: up to 49.9% accuracy gain","Measure and train reasoning factuality in LLMs, up to +49.9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1729,"prompt_tokens":1008,"completion_tokens":721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":615}},"tokens_in":624,"tokens_out":721,"duration_ms":6304,"temperature":1.0,"reasoning_tokens":615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:59:01.121711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of naturally occurring reasoning errors (human-annotated, not entity-substituted), and compare the fact-checker's verdicts and the RELIANCE-trained model's step accuracy against human labels. If the classifier's agreement with humans on natural errors is no better than chance, or if the enhanced model's outputs are rated by blind experts as more evasive rather than more accurate, the central claim is falsified.","supporting_citations":[{"cited_title":"FLAIR: An easy-to-use framework for state-of-the- art NLP,","cited_arxiv_id":null,"evidence_quote":"Provides the base training framework that the authors extend with their multi-dimensional reward design."}],"review_version":2}