{"id":"7c50569f-61f6-4d62-9fa8-f9730113f6ca","arxiv_id":"2505.19949","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Influence-function attribution shows high-difficulty math and low-difficulty code training data best improve math and code reasoning, and difficulty-based reweighting improves benchmark performance.","lead":"This paper measures how individual training examples, sentences, and tokens contributed to a large language model's math and code reasoning skills, using influence functions. It reports that hard math and easy code examples help most, and that flipping difficulty in a training set doubled AIME24 accuracy and raised LiveCodeBench accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated influence rankings are the load-bearing weak point: Appendix C shows weak stability for small n and no retraining check, so the difficulty-flip claim (AIME 10→20, LCB 33.8→35.3) may not be attributable to difficulty.","rationale":"The reader's weakest assumption is the right one. The central quantitative claims hinge on influence scores computed with EK-FAC on MLP parameters only, attention held fixed, and 4096-token truncation, and those scores are never checked against actual retraining. Appendix C reports correlations only among different choices of n, not against ground truth; values of 0.51–0.62 at n=10/25 indicate substantial instability, and n=100 is itself arbitrary. Given that the difficulty-level conclusions and the flip recipe are built on these rankings, a direct retraining validation is the minimal evidence needed. I do not reject the paper: the exploration-removal SFT ablation (Table 2, right) independently supports the sequence-level finding, and the flipped-versus-reverse comparison partly controls for data source because both arms use OpenThoughts-114k, so the issue is about missing validation rather than internal inconsistency or fraud. The appropriate verdict remains conditional pending this check, hence UNCHANGED relative to the reader's CONDITIONAL verdict.","tokens_in":14477,"tokens_out":4280,"duration_ms":51437,"concrete_test":"Validate the influence rankings against retraining in the cheaper LLaMA3-8B short-CoT setting: sample 40 training examples from the MetaMathQA+OSS-Instruct mixture; for each example, fine-tune the base model to the same SFT objective twice, once with the example included and once with it excluded (or with its loss upweighted by a small epsilon), and measure the change in mean log-likelihood on a fixed D_correct of 100 math and code problems. Compute Spearman/Pearson correlation between predicted influence scores and actual log-likelihood changes. If the correlation is below roughly 0.5, the difficulty-level findings and the flip recipe lose their foundation; if it is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that influence scores identify hard math and easy code as optimal, and that flipping difficulty yields the reported gains. For this to hold, the ranking computed via Equation 3 with the Section 3.1 approximations must track the true effect of training examples on D_correct. The evidence is missing: (i) the Hessian is EK-FAC over MLP parameters only, attention is held fixed, and sequences are truncated at 4096 tokens, so long-CoT signals may be distorted; (ii) D_correct is a random sample of n=100 correct answers with no reported variance; (iii) Appendix C validates only self-consistency against n=100, giving Pearson 0.51–0.62 for n=10/25, not a validation against any ground-truth retraining effect; (iv) no leave-one-out or subset-retraining check is reported anywhere. Without such a check, the difficulty-level averages in Figure 5 and the flip results in Table 1 could reflect noise or the data-source difference between BS-17k and OpenThoughts-114k rather than difficulty. The exploration-removal SFT ablation (Table 2, right) independently supports Finding 3, but the difficulty findings and the headline AIME24/LiveCodeBench gains rest entirely on unvalidated ranks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Infra, an influence-function framework for attributing the effects of SFT training examples on math and code reasoning in LLMs. It computes instance-, sequence-, and token-level influence scores using an EK-FAC Hessian approximation over MLP parameters, and uses these scores to identify dataset-level patterns. The main claims are that cross-domain examples help both math and code reasoning, that high-difficulty math and low-difficulty code examples are optimal, and that a 'difficulty-flip' reweighting strategy doubles AIME24 accuracy from 10% to 20% and improves LiveCodeBench from 33.8% to 35.3% on Qwen2.5-7B-Instruct. The paper also claims that exploratory behaviors in long CoTs are beneficial, supported by both influence scores and an SFT ablation.","tokens_in":14734,"tokens_out":5009,"duration_ms":56065,"significance":"If the central claims held, the paper would provide a practical, influence-based method for data-centric reasoning SFT and a nontrivial account of cross-domain transfer. The strengths include the clear formulation of instance-, sequence-, and token-level influence in Eqs. (3), (6), and (8), the explicit use of two SFT settings, and the independent SFT ablation in Table 2 (right) that directly validates the exploration finding. The difficulty-flip claim, however, currently rests on influence rankings that are not validated against retraining and on a dataset swap that changes both difficulty and data source, so the significance of the headline result cannot be assessed without additional validation.","major_comments":[{"comment":"The difficulty-flip experiment is confounded by dataset provenance. The text says that 'we replace simple math problems in the original dataset with more challenging ones sourced from a larger scale OpenThoughts-114k dataset' and similarly replace hard code problems with simpler ones. This changes not only difficulty but also the source of the examples, so the reported improvements (AIME24 10.0→20.0, LiveCodeBench 33.8→35.3) could be due to source distribution rather than difficulty. The footnote that OpenThoughts-114k is 'curated using the same pipeline' is an assumption, not a control. The authors should run a source-controlled comparison, for example by swapping difficulty labels within BS-17k itself or by matching examples across sources on difficulty and content, before attributing the gain to difficulty.","section":"§4.2 / Table 1"},{"comment":"The influence rankings that underlie Findings 1 and 2 are not validated against ground-truth retraining effects. The Hessian is approximated with EK-FAC over MLP parameters only, with attention held fixed and sequences truncated to 4096 tokens, as acknowledged in the Limitations section. Appendix C validates only the self-consistency of rankings across different sizes of D_correct; it does not test whether a high predicted influence actually corresponds to a larger change in the target function f(θ) when the example is removed or upweighted. This is a load-bearing gap because the difficulty-level averages in Figure 5 and the flip results in Table 1 depend on the correctness of those rankings. The authors should add a small-scale leave-one-out or subset-retraining validation comparing predicted influence with measured changes in f(θ).","section":"§3.1 / §4.1 / Limitations"},{"comment":"The choice of n = 100 for D_correct is a free parameter, and the reported stability is weak. Table 3 shows Pearson correlations of 0.51–0.60 when comparing n=10 or n=25 to n=100, which is far from stable, and no variance or confidence intervals are reported for the average influence scores in Figure 5. Because the difficulty ordering is computed from a single random draw of D_correct, the authors should report bootstrap or repeated-sampling intervals and demonstrate that the difficulty-level ordering is robust to the choice of n.","section":"§4.1 / Appendix C / Figure 5"},{"comment":"The claim that 'symbolic math examples and high-difficulty math problems are particularly effective in improving code reasoning' is based on grouping MATH data into difficulty levels, but the classification of data into categories such as FOBAR, SV, and 'college-level' versus 'high-school' is not precisely defined in the main text. Since these categories are used to draw conclusions about symbolic versus conversational data, the authors should define the taxonomy and report inter-annotator or classifier agreement.","section":"§2 / §4.2"}],"minor_comments":[{"comment":"The text 'Bespoke-Stratos-17k2' has a stray superscript that appears to be a misplaced footnote marker; please move the footnote marker to the correct location.","section":"§1 / footnote 2"},{"comment":"The right panel appears to show both a distribution of cognitive behaviors and average influence bars, but the caption does not explain the axes and scale of the two plots; please clarify which bars correspond to which metric and add error bars or per-behavior sample sizes.","section":"Figure 6"},{"comment":"The caption says 'We report pass@1 accuracy of LiveCodeBench,' but the table also reports AIME and MATH500 scores; please specify the decoding and metric conditions for each benchmark.","section":"Table 1"},{"comment":"The notation 'Math− →Code' in the Figure 5 caption is confusing; define the arrow notation in the text or use a consistent legend.","section":"Figure 5 / §4.2"},{"comment":"The GPT-4o truncation prompt in Appendix D asks for the 'number of distinct backtracking instances' even though the behavior being truncated is exploration; this appears to be a copy-paste error in the prompt template and should be corrected.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.LG and the exploration ablation is a solid contribution, but the difficulty-flip claim needs a source-controlled experiment and a retraining-based validation of the influence scores before it can be considered established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The genuinely new thing is fine-grained attribution—sequence and token level—applied to reasoning SFT data, and the specific claim that low-difficulty code examples help code reasoning when mixed with math CoT. The paper also does a clean thing: it tests the exploration finding with an actual SFT ablation (Table 2, right), and that result stands on its own. Credit where due: the method is reasonable, the writeup is clear, and the reverse-flip control is a decent idea.\n\nThe soft spots are the usual suspects for influence-function papers, but they land directly on the main claim. The rankings computed via Equation 3 with EK-FAC on MLP parameters only are never validated against retraining. Appendix C only checks self-consistency across n, and the Pearson values for n=10/25 are 0.51–0.62—moderate at best. That is not enough to support the difficulty-level averages in Figure 5. The difficulty-flip experiment in Table 1 is also confounded: easy math is replaced by hard math from OpenThoughts-114k and hard code by easier code, so the gains could come from source or content rather than difficulty. The paper argues the sources are curated with the same pipeline, but that is not a controlled comparison. No error bars, no multiple seeds. The stress-test note is right about all of this.\n\nThe exploration finding is the one I'd trust most. The token-level analysis is interesting but qualitative, and the claims about 'distinct paradigms' need more support than a few highlighted examples.\n\nOverall: the central difficulty-flip claim is plausible but not established. The method is worth publishing, but only after the rankings are checked against leave-one-out or subset retraining, and the flip experiment is redone with source controlled and variance reported.\n\nWho should read this: anyone working on reasoning SFT data curation. It deserves a serious referee, but it needs major revision before the findings are credible. I'd bring it to reading group and would consider citing it only after the validation is in.","headline":"A promising influence-based attribution method for reasoning SFT, but the flagship difficulty-flip result is not yet established: the rankings are unvalidated and the flip experiment is confounded.","tokens_in":15283,"tokens_out":2557,"would_cite":false,"duration_ms":26787,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper uses influence functions to show that high-difficulty math examples improve both math and code reasoning, while low-difficulty code examples most benefit code reasoning, and that flipping difficulty in this way doubles AIME24…","keywords":["influence functions","data attribution","math reasoning","code reasoning","chain-of-thought","supervised fine-tuning","dataset difficulty","cross-domain transfer"],"falsifier":"An independent check would retrain the model after deleting the examples that influence scores rank as most positive and compare accuracy against deleting random or low-ranked examples; if removing the highest-ranked examples does not hurt most, or if the rankings are unstable across retraining runs, the difficulty and flip findings lose their foundation.","tokens_in":1634,"feed_emoji":"🧮","tokens_out":2897,"duration_ms":81635,"temperature":0.7,"pith_summary":"The paper tries to establish that influence-based data attribution can reveal which attributes of supervised fine-tuning data actually stimulate math and code reasoning. Its central finding is a difficulty asymmetry: high-difficulty math examples improve both math and code reasoning, while low-difficulty code examples most effectively benefit code reasoning. Acting on this by swapping easy math for hard math and hard code for easy code doubles AIME24 accuracy from 10% to 20% and lifts LiveCodeBench from 33.8% to 35.3% on Qwen2.5-7B-Instruct. The same attribution machinery also shows that post-correctness exploration in long chain-of-thought traces helps both domains, and that the most influential tokens are logical connectors in math but structural syntax in code.","feed_headline":"Hard math plus easy code is the best reasoning mix","feed_subtitle":"High-difficulty math plus low-difficulty code lifts AIME24 from 10% to 20%, an influence-function study finds.","key_machinery":"The machinery is the influence function with an EK-FAC (Eigenvalue-Corrected Kronecker-Factored Approximate Curvature) Hessian estimated on MLP parameters only, with attention held fixed and sequences truncated to 4096 tokens. Influence scores feed a mean-log-likelihood surrogate for benchmark accuracy, and the method extends to sequence-level attribution by removing a sentence and measuring the score change, and to token-level attribution by exploiting the autoregressive gradient decomposition. This lets the paper rank individual examples, cognitive behaviors, and tokens by their estimated contribution to math and code reasoning.","core_discovery":"Infra, the paper's influence-based attribution method, computes for every supervised fine-tuning example its estimated contribution to the mean log-likelihood of correctly answered math and code benchmark problems. The central empirical claim is that high-difficulty math examples improve both math and code reasoning, whereas low-difficulty code examples are the most effective target for code reasoning, so the optimal data-mixing strategy is to combine hard math with easy code. Acting on this by flipping difficulty doubles AIME24 accuracy from 10% to 20% and boosts LiveCodeBench accuracy from 33.8% to 35.3% on Qwen2.5-7B-Instruct, with smaller gains on the 14B model. At finer granularity, the paper claims that exploratory behavior after reaching a correct answer improves both math and code reasoning, and that token-level influence patterns diverge across domains: math rewards natural-language logical connectors, while code rewards explicit structural syntax.","pith_inferences":["A testable extension would replace influence-score computation with a cheaper difficulty-and-domain heuristic and check whether it replicates the AIME and LiveCodeBench gains, which would suggest influence functions are identifying an effect that simpler proxies can capture.","The token-level divergence suggests a curriculum-ordering hypothesis the paper does not pursue: begin code training with structurally clean examples, then increase logical complexity through math, which could be tested by staged supervised fine-tuning.","Whether the difficulty-flip result transfers to reinforcement-learning post-training, larger models, or domains beyond math and code is left open; the proposed logic-versus-structure mechanism predicts transfer, but the paper demonstrates only supervised fine-tuning on 7B and 14B instruct models.","The sequence-level attribution depends on the classifier used to detect exploration and other cognitive behaviors, so varying the classifier threshold and re-measuring influence would test how sensitive the conclusions are to behavior detection."],"forward_implications":["If the difficulty finding is correct, dataset reweighting by difficulty is a cheap and effective lever: within a fixed 17k-example budget, replacing easy math with hard math and hard code with easy code improves both math and code benchmarks together.","Cross-domain data deserves an explicit role in data curation: code reasoning can be improved by math examples, especially high-difficulty, formally symbolic ones, not only by in-domain code examples.","Exploratory overthinking in training traces should be preserved rather than pruned, since removing it drops MATH500 from 77.2 to 73.8 and LiveCodeBench from 33.8 to 32.0.","Token-level findings suggest that math training traces benefit from explicit logical connectors, while code training traces benefit from preserved structural formatting and syntax markers.","The reverse difficulty-mixing strategy degrades performance, supporting the direction of the finding rather than a generic preference for harder or easier data."],"supporting_citations":[{"why":"It provides the influence-function formula for upweighting a training point and the implicit-function derivation that the method relies on.","marker":"[17]"},{"why":"It shows how to scale influence functions to large Transformers via EK-FAC on MLP parameters, which Infra adopts for Hessian approximation.","marker":"[8]"},{"why":"It supplies the Kronecker-factored curvature approximation that the EK-FAC Hessian estimate builds on.","marker":"[6]"},{"why":"It establishes a pretraining link between code data and reasoning, motivating the search for cross-domain effects during supervised fine-tuning.","marker":"[30]"},{"why":"It argues that a small set of challenging math problems can elicit strong reasoning, the prior difficulty result that this paper extends and refines.","marker":"[37]"},{"why":"It provides the LiveCodeBench benchmark used to measure code reasoning in the long-chain-of-thought experiments.","marker":"[16]"},{"why":"It supplies MetaMathQA, one of the two training datasets in the short-chain-of-thought cross-domain attribution setup.","marker":"[39]"},{"why":"It supplies OSS-Instruct, the code-instruction dataset used alongside MetaMathQA for the basic cross-domain experiments.","marker":"[33]"}],"fun_headline_variants":["Hard math plus easy code best for LLM reasoning","Data mix: high-difficulty math, low-difficulty code","Difficulty flip doubles AIME24 score in reasoning","Influence functions find ideal reasoning data blend","Hard math, easy code lifts math and code skills"],"cache_read_input_tokens":17408,"weakest_assumption_plain":"The entire analysis rests on believing that the influence scores, computed with an approximate Hessian that only updates MLP weights while freezing attention and truncating long traces, accurately rank how much each training example truly contributes to the final model's reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Hard math plus easy code best for LLM reasoning","Data mix: high-difficulty math, low-difficulty code","Difficulty flip doubles AIME24 score in reasoning","Influence functions find ideal reasoning data blend","Hard math, easy code lifts math and code skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1339,"prompt_tokens":985,"completion_tokens":354,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":601,"tokens_out":354,"duration_ms":4418,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:03:19.281028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent check would retrain the model after deleting the examples that influence scores rank as most positive and compare accuracy against deleting random or low-ranked examples; if removing the highest-ranked examples does not hurt most, or if the rankings are unstable across retraining runs, the difficulty and flip findings lose their foundation.","supporting_citations":[{"cited_title":"Fast approximate natural gradient descent in a kronecker factored eigenbasis.Advances in neural information processing systems, 31, 2018","cited_arxiv_id":null,"evidence_quote":"It supplies the Kronecker-factored curvature approximation that the EK-FAC Hessian estimate builds on."},{"cited_title":"Livecodebench: Holistic and contamination free evaluation of large language models for code","cited_arxiv_id":null,"evidence_quote":"It provides the LiveCodeBench benchmark used to measure code reasoning in the long-chain-of-thought experiments."},{"cited_title":"Magicoder: Em- powering code generation with oss-instruct.Proceedings of Machine Learning Research, 235: 52632–52657, 2024","cited_arxiv_id":null,"evidence_quote":"It supplies OSS-Instruct, the code-instruction dataset used alongside MetaMathQA for the basic cross-domain experiments."}],"review_version":1}