{"id":"f2f2bff5-f5d7-4b21-af98-a275b72541f0","arxiv_id":"2506.02126","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM reasoning can be scored separately for knowledge and step-by-step information gain, and doing so shows SFT and RL affect these two capacities differently across medicine and math.","lead":"The paper proposes a fine-grained evaluation that splits an LLM's step-by-step reasoning into knowledge correctness and reasoning quality, then uses it to compare supervised fine-tuning and reinforcement learning on medical and math tasks. A general reader should care because the results suggest SFT boosts accuracy but can make reasoning less efficient, while RL trims wrong knowledge, which matters for training reliable models in high-stakes fields.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"InfoGain conflates answer leakage with reasoning: because Eq. (1) scores P(A*|Q, s1:i), any step that states or restates the answer mechanically reduces PPL, so the SFT/RL reasoning-quality findings may be an artifact of where and whether the answer appears.","rationale":"The reader's weakest assumption is that the perplexity-based InfoGain, computed with an untrained Qwen2.5-7B judge, is a faithful measure of step-level reasoning quality. I agree, and I sharpen the concern: InfoGain is not merely noisy or unvalidated; it has a directional mechanical bias. Because it scores the probability of the correct answer's tokens after conditioning on the step text, any step that mentions or restates the answer inflates InfoGain, while a concise high-quality inference that does not echo the answer is penalized. This is exactly the kind of confound that could produce the paper's headline pattern—SFT lowering InfoGain by 38.9% and RL raising it—without any actual change in logical reasoning quality. The knowledge-index half of the framework is less vulnerable because it checks extracted claims against external sources, and the accuracy results stand on their own. The internal 38.9%/37.1% discrepancy and the single-seed training runs are secondary and do not change the architecture of the argument. Given that the central reasoning-quality claim depends on this metric, the paper should not be accepted as a definitive characterization until the proposed masking/control test is run. That is precisely the CONDITIONAL status the reader assigned, so the verdict is unchanged.","tokens_in":14791,"tokens_out":4439,"duration_ms":44218,"concrete_test":"Recompute InfoGain after masking answer leakage: before scoring P(A*|Q, s1:i), remove from each step any token span that matches tokens of A*, then compare the SFT-vs-base and RL-vs-SFT deltas against the reported 38.9% drop and the RL gains. If the deltas largely disappear or reverse, InfoGain is driven by answer mention rather than reasoning. A paired control on a held-out set of base-model traces—appending 'The correct answer is ...' as an extra step—would confirm the mechanism if InfoGain jumps sharply.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—SFT degrades reasoning (38.9% InfoGain drop in the abstract; 37.1% in Section 4.2) while RL improves it—rests entirely on the InfoGain metric defined by Eqs. (1)–(4). Pi is the probability of the correct answer A* given the question and steps s1:i, and Delta-I averages adjacent PPL reductions. This measures how predictable A*'s tokens become after reading each step, not whether the step contains a genuine logical inference. A step that explicitly says 'the correct answer is B' or paraphrases answer tokens will produce a large PPL drop regardless of the reasoning it contains, while a terse step that supplies the single crucial deduction but does not mention the answer will produce a small drop. The GPT-4o decomposition is unconstrained about this: steps are arbitrary sentence spans, and the final-answer sentence can easily become its own step. The observed SFT and RL differences are therefore confounded with verbosity, answer placement, and decomposition choices. RL could look good simply because RL traces front-load answer-relevant tokens, and SFT could look bad because SFT adds redundant restatements. This concern does not undermine the knowledge/reasoning framing itself, but the InfoGain half of the decomposition currently measures answer-token surprisal, not reasoning informativeness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to decompose LLM reasoning traces into two components: knowledge correctness, measured by a Knowledge Index (KI), and reasoning informativeness, measured by an Information Gain (InfoGain) metric defined as the average reduction in perplexity of the correct-answer tokens over successive reasoning steps. The framework is applied to Qwen2.5-7B-base and DeepSeek-R1-Distill-Qwen-7B models after supervised fine-tuning (SFT) and reinforcement learning (RL) on medical and mathematical benchmarks. The main reported findings are that (1) R1-distilled models do not transfer their reasoning ability to medicine, (2) SFT improves accuracy but lowers InfoGain by 38.9% (also stated as 37.1%), and (3) RL improves medical reasoning and knowledge by pruning incorrect or irrelevant knowledge from reasoning paths. The authors conclude that medical tasks are more knowledge-intensive while math tasks are more reasoning-intensive, and that SFT and RL target different capacities.","tokens_in":15032,"tokens_out":4920,"duration_ms":45311,"significance":"If the proposed decomposition is valid, the paper offers a useful lens for understanding post-training effects beyond final-answer accuracy, and the two-domain comparison (medical vs. math) is well motivated. The authors release code, project pages, and detailed prompts in Appendix B.2, which is a concrete reproducibility strength. The paper also includes case studies and a correlation analysis linking the proposed metrics to accuracy. However, the significance is conditional on establishing that the InfoGain metric actually measures reasoning quality rather than answer-token leakage or verbosity, and that the Knowledge Index measures factual correctness rather than GPT-4o self-consistency. As it stands, the headline quantitative claims rest on metrics whose construct validity has not been demonstrated.","major_comments":[{"comment":"The InfoGain metric is computed as the reduction in perplexity of the correct-answer tokens A* conditioned on Q and the steps s1:i. Because the score is evaluated on the tokens of A* itself, any step that states, restates, or paraphrases the answer will mechanically reduce PPL even if it contains no new logical inference; conversely, a terse step that supplies the key deduction without mentioning the answer will receive a small ΔPPL. The GPT-4o decomposition in Appendix B.2 does not exclude the final-answer sentence, and Figure 3 explicitly labels a step as \"Reasoning + Answer\" with step text \"The final answer is.....\". This means the Section 4.2 conclusion that SFT lowers InfoGain by 37.1% and RL raises it conflates reasoning quality with answer-token leakage, answer placement, and decomposition choices. I ask for a control that removes or masks answer-containing sentences before computing PPL, or a variant that scores steps against the probability of the next reasoning step rather than the final answer, and a report of how the SFT/RL differences survive that control.","section":"§3.3, Eqs. (1)–(4)"},{"comment":"No evidence is presented that the perplexity-gap quantity ΔI tracks independent judgments of reasoning quality. The paper asserts that \"a higher ΔI indicates that more information emerges during reasoning, reflecting stronger reasoning capabilities,\" but this interpretation is not validated against human annotations or an established step-quality benchmark. Table 2 shows only that a different metric (ROSCOE-SA) is insensitive to the differences the authors care about; it does not establish that InfoGain is a faithful measure of reasoning quality. Given that all three headline findings depend on InfoGain, I request a validation study comparing ΔI with human-rated step informativeness or with a known reasoning-quality metric on the same responses. Without this, the central claim that SFT harms reasoning while RL improves it is an assumption rather than a demonstrated result.","section":"§3.3 (InfoGain validity)"},{"comment":"All reported comparisons are point estimates from single training runs, with no error bars, confidence intervals, or significance tests. Several differences used to support the claims are extremely small (e.g., InfoGain 0.157 vs. 0.161 and 0.192 vs. 0.190 in Table 1; KI 63.40 vs. 64.23). The reader cannot assess whether the reported 38.9%/37.1% InfoGain drop, the \"RL improves knowledge index by 12.4 points\" claim in Figure 8, or the correlation coefficients in Figure 5 are robust to seed variation and to randomness in the GPT-4o decomposition and judgment steps. At minimum, the authors should report variance over multiple random seeds and provide a significance test for the main comparisons; without this, the quantitative strength of the findings is not established.","section":"§4.1–4.2, Tables 1 and Figures 6–8"},{"comment":"The Knowledge Index pipeline uses GPT-4o for knowledge extraction, for query generation, and for the final consistency judgment against retrieved passages, and the manuscript does not report human agreement on KI or a comparison with a curated gold standard. Because the same model produces both the extracted knowledge point and the consistency verdict, KI may measure GPT-4o's self-consistency rather than objective factual correctness. The paper also states that math knowledge is not evaluated (\"as the knowledge base of math\" is an incomplete sentence in §4.2), so the knowledge side of the decomposition is only tested in the medical domain. This weakens the knowledge component of the framework and the medical-domain claims that depend on KI differences, such as the 6.2% and 12.4% KI gains discussed in Sections 4.1 and 4.2.","section":"§3.4 and Appendix B.2 (KI validation)"}],"minor_comments":[{"comment":"The abstract reports an average InfoGain drop of 38.9% after SFT, while Section 4.2 reports an average reduction of 37.1%. These numbers should be reconciled or the discrepancy explained.","section":"Abstract vs. §4.2"},{"comment":"The math reasoning decomposition prompt begins \"You are a helpful, pattern-following medical assistant,\" which appears to be a copy-paste error from the medical prompt.","section":"Appendix B.2, Figure 11"},{"comment":"The sentence \"We only present the knowledge index metric for medical benchmarks, as the knowledge base of math\" is incomplete; please clarify whether math knowledge is omitted and, if so, why.","section":"§4.2, Figure 6"},{"comment":"The text states \"the SFT-ed model outperforms both its RL-ed and base counterparts, with average gains of 4.6% and 6.2%, respectively (SFT-ed 49.8% > RL-ed 49.8% > base 48.2%)\", but SFT-ed and RL-ed are both listed as 49.8%, which contradicts \"outperforms\"; the numbers and the comparison should be checked.","section":"§4.2, Figure 6(a)"},{"comment":"The caption contains a typo: \"knowlegde\" should be \"knowledge\".","section":"Figure 8 caption"}],"recommendation":"major_revision","confidential_remarks":"The manuscript presents an appealing framework and a substantial set of experiments, but the central quantitative claims hinge on the construct validity of InfoGain, which is currently confounded with answer-token leakage, and on the absence of statistical uncertainty. I do not see evidence of misconduct or fatal internal inconsistency; rather, the paper needs additional controls and validation before the claims can be accepted. The scope fit with a CS/CL venue is reasonable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague], quick take on arXiv:2506.02126. The paper separates knowledge correctness from reasoning informativeness in LLM reasoning traces and applies it to SFT/RL in medicine and math. That lens is genuinely useful, and the comparison with ROSCOE makes a good point: embedding-similarity metrics are too blunt to see training-induced changes. The KI metric, with retrieval against a medical knowledge base, is a reasonable way to score factual correctness, though it's GPT-4o-heavy.\n\nThe soft spot is InfoGain. Equation (1) computes the probability of the correct answer's tokens given the question plus steps, so any step that mentions or restates the answer will mechanically drop perplexity. The stress-test note is right: this is answer-token surprisal, not logical informativeness. A terse step that makes the key deduction but doesn't name the answer scores low; a verbose restatement that echoes answer tokens scores high. The SFT-vs-RL difference (38.9% drop in the abstract, 37.1% in Section 4.2 — a discrepancy that needs fixing) could be confounded with verbosity and answer placement. The authors do show in Figure 1 that redundant steps get low InfoGain, so the metric is not pure noise, but they haven't validated it against human judgments or against reasoning quality independent of answer overlap. No error bars, single runs on one 7B family, so the quantitative claims are shaky.\n\nThat said, I don't think the framing collapses. The KI findings and the domain difference (medical knowledge-heavy, math reasoning-heavy) are plausible and worth testing. The paper is honest about its limitations (single model family). The main thing referees should demand is a validation of InfoGain against human-rated reasoning quality, or a redefinition that controls for answer leakage.\n\nBottom line: It deserves a serious referee. I'd send it out, but the referees should push on the metric. The framework is a useful contribution even if the specific 'SFT hurts reasoning' numbers don't survive.","headline":"Useful two-axis evaluation lens, but the reasoning metric measures answer-token surprisal, not validated reasoning quality; the SFT/RL claims are plausible but not yet established.","tokens_in":15593,"tokens_out":2372,"would_cite":true,"duration_ms":21632,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM's step-by-step reasoning can be split into knowledge correctness and reasoning informativeness, and the two respond differently to training.","keywords":["LLM reasoning evaluation","Knowledge Index","Information Gain","step-wise perplexity","supervised fine-tuning","reinforcement learning","medical reasoning","mathematical reasoning"],"falsifier":"Take a set of medical reasoning traces that are logically equivalent but differ in wording, and compute InfoGain for each; if the score moves substantially across paraphrases that preserve the logical content, the metric is tracking style, not reasoning informativeness, and the paper's SFT-versus-RL conclusions lose their support.","tokens_in":14579,"feed_emoji":"🧠","tokens_out":9331,"duration_ms":85179,"temperature":0.7,"pith_summary":"This paper argues that final-answer accuracy hides how an LLM actually thinks, and that the thinking trace itself can be scored along two separate axes. It introduces Knowledge Index (KI), the fraction of reasoning steps whose factual content agrees with an external medical knowledge source, and InfoGain, the average reduction in a judge model's perplexity toward the correct answer as each reasoning step is read. Applied to Qwen2.5-7B base and DeepSeek-R1-distilled models trained with supervised fine-tuning (SFT) and reinforcement learning (RL) on medicine and math, the framework shows that SFT raises accuracy and factual knowledge but cuts InfoGain by 38.9% on average, while RL prunes inaccurate or irrelevant knowledge from reasoning paths. The authors conclude that medical tasks lean more on knowledge and math tasks lean more on reasoning, so the two training regimes should be chosen by domain.","feed_headline":"SFT makes LLMs more accurate but 39% less informative","feed_subtitle":"Step-level scores separate knowledge from reasoning: medical tasks reward SFT, math tasks reward RL.","key_machinery":"The machinery is a two-metric decomposition of a model's thinking trace. A response is split into successive steps $s_1, \\dots, s_t$ by GPT-4o. InfoGain is then computed as $\\Delta I = \\frac{1}{t}\\sum_{i=1}^t(\\mathrm{PPL}_{i-1} - \\mathrm{PPL}_i)$, where $\\mathrm{PPL}_i$ is the perplexity that an untrained Qwen2.5-7B judge assigns to the correct answer $A^*$ given the question and the first $i$ steps; a positive gap means the step concentrates probability mass on the correct answer. Knowledge Index is the average of Boolean judgments $\\mathrm{consistency}_i$: for each step GPT-4o extracts its knowledge claim, the claim is looked up in a medical textbook database, and GPT-4o decides whether the step matches the retrieved fact. These two scalar metrics are what let the authors compare SFT and RL training regimes step by step, and they are the basis for the claim that the two capacities are distinct.","core_discovery":"The paper's central discovery is that reasoning traces can be decomposed into two capacities that respond differently to training: knowledge correctness, measured as Knowledge Index, and reasoning informativeness, measured as InfoGain. In medical fine-tuning the base Qwen2.5-7B outperforms the R1-distilled variant by 14.7 points with SFT alone and by 22.6 points after RL, showing that general reasoning skills distilled from R1 do not transfer to medicine. SFT increases accuracy and knowledge (average KI up 6.2 points on medical tasks) but lowers InfoGain by 38.9% on average, producing verbose or suboptimal reasoning paths; RL restores reasoning quality by steering the model away from steps containing wrong knowledge, raising KI by 12.4 points on average when applied alone. Across five medical benchmarks KI correlates with accuracy at 0.998 on average, about 0.3 higher than InfoGain's correlation, while in math RL yields the largest accuracy gains, supporting the conclusion that knowledge and reasoning are distinct, domain-dependent abilities.","pith_inferences":["If InfoGain is a valid measure, it could be used as a training signal: rewarding steps that reduce a judge model's perplexity toward the correct answer should yield more concise, informative reasoning chains than rewarding only final accuracy.","The same decomposition could be applied to other structured domains such as legal IRAC reasoning or financial forecasting, where step-level knowledge and inference can be separated; a concrete test would be whether KI and InfoGain correlate with human expert ratings in those domains.","A natural stress test is to replace the untrained Qwen2.5-7B judge with models of different size and domain familiarity; if the ranking of training regimes reverses under that substitution, the reported effects are judge-dependent rather than properties of the reasoning itself.","The RL 'pruning' behavior suggests RL improves knowledge correctness not by adding facts but by suppressing unreliable paths, which could be verified by tracking which knowledge points disappear from reasoning traces after RL."],"forward_implications":["SFT should not be assumed to improve reasoning quality: on these models it raises accuracy and knowledge while cutting InfoGain by 38.9% on average.","RL is the better lever for reasoning efficiency: it consistently raises InfoGain in both medicine and math, and in medicine it raises Knowledge Index by 12.4 points on average by pruning wrong-knowledge paths.","Accuracy gains in medicine come mostly from knowledge: KI correlates with accuracy at 0.998 average across the five medical benchmarks, about 0.3 higher than InfoGain's correlation.","General reasoning distilled from R1 does not transfer to medicine: the Qwen base model beats the R1-distilled variant by 14.7 points with SFT and by 22.6 points after RL in average medical accuracy.","Training strategy should differ by domain: SFT matters more for knowledge-intensive medicine, while RL matters more for reasoning-intensive math, where RL-ed accuracy (61.7% average) exceeds SFT-ed accuracy (51.9%)."],"supporting_citations":[{"why":"Qwen2.5-7B: the universal base model all training and evaluation experiments start from.","marker":"[37]"},{"why":"DeepSeek-R1-Distill-Qwen-7B: the R1-distilled variant whose medical transfer fails, a central comparison.","marker":"[11]"},{"why":"HuatuoGPT-o1: supplies the medical SFT and RL data splits used to train the compared models.","marker":"[9]"},{"why":"RLHFlow: supplies the math-domain SFT and RL training pipeline and data.","marker":"[12]"},{"why":"MedQA-USMLE: the medical textbook source used as external ground truth in Knowledge Index retrieval.","marker":"[19]"},{"why":"Language models are few-shot learners: the perplexity formulation that InfoGain is built from.","marker":"[7]"},{"why":"GPT-4o: runs the step decomposition, knowledge extraction, and consistency judgements in the pipeline.","marker":"[27]"},{"why":"ROSCOE: the prior step-level reasoning metric shown to be insensitive across models, motivating the new metrics.","marker":"[14]"},{"why":"SFT memorizes, RL generalizes: cited as consistent with the observed SFT/RL tradeoff in reasoning quality.","marker":"[10]"}],"fun_headline_variants":["SFT boosts accuracy but cuts reasoning info by 39%","Knowledge vs reasoning: SFT helps medicine, RL helps math","R1-distilled skills don't transfer to medicine, study finds","SFT hurts reasoning quality: InfoGain drops 39%","LLM training trade-off: accuracy up, insight down"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the drop in a small untrained language model's confusion about the correct answer from one reasoning step to the next really measures how informative that step is, rather than how the answer is phrased or how familiar the model already is with the topic.","fun_headline_variants_meta":{"raw":{"variants":["SFT boosts accuracy but cuts reasoning info by 39%","Knowledge vs reasoning: SFT helps medicine, RL helps math","R1-distilled skills don't transfer to medicine, study finds","SFT hurts reasoning quality: InfoGain drops 39%","LLM training trade-off: accuracy up, insight down"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000535,"raw_usage":{"total_tokens":2608,"prompt_tokens":1016,"completion_tokens":1592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":1505}},"tokens_in":632,"tokens_out":1592,"duration_ms":12632,"temperature":1.0,"reasoning_tokens":1505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:29:20.816928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of medical reasoning traces that are logically equivalent but differ in wording, and compute InfoGain for each; if the score moves substantially across paraphrases that preserve the logical content, the metric is tracking style, not reasoning informativeness, and the paper's SFT-versus-RL conclusions lose their support.","supporting_citations":[{"cited_title":"Deepseek -r1-distill-qwen-7b","cited_arxiv_id":null,"evidence_quote":"DeepSeek-R1-Distill-Qwen-7B: the R1-distilled variant whose medical transfer fails, a central comparison."},{"cited_title":"Rlhf workflow: From reward modeling to online rlhf","cited_arxiv_id":null,"evidence_quote":"RLHFlow: supplies the math-domain SFT and RL training pipeline and data."},{"cited_title":"Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, andet al.Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901, 2020","cited_arxiv_id":null,"evidence_quote":"Language models are few-shot learners: the perplexity formulation that InfoGain is built from."}],"review_version":1}