Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper uses influence functions to show that high-difficulty math examples improve both math and code reasoning, while low-difficulty code examples most benefit code reasoning, and that flipping difficulty in this way doubles AIME24…

desk verdict A promising influence-based attribution method for reasoning SFT, but the flagship difficulty-flip result is not yet established: the rankings are unvalidated and the flip experiment is confounded. read the letter →

arxiv 2505.19949 v1 pith:OAH7242M submitted 2025-05-26 cs.LG

Siqi Kou , Qingyuan Tian , Hanwen Xu , Zihao Zeng , Zhijie Deng This is my paper
classification cs.LG
keywords influencefunctionsdataattributionmathreasoningcodechain-of-thoughtsupervisedfine-tuningdatasetdifficultycross-domaintransfer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that influence-based data attribution can reveal which attributes of supervised fine-tuning data actually stimulate math and code reasoning. Its central finding is a difficulty asymmetry: high-difficulty math examples improve both math and code reasoning, while low-difficulty code examples most effectively benefit code reasoning. Acting on this by swapping easy math for hard math and hard code for easy code doubles AIME24 accuracy from 10% to 20% and lifts LiveCodeBench from 33.8% to 35.3% on Qwen2.5-7B-Instruct. The same attribution machinery also shows that post-correctness exploration in long chain-of-thought traces helps both domains, and that the most influential tokens are logical connectors in math but structural syntax in code.

What carries the argument

The machinery is the influence function with an EK-FAC (Eigenvalue-Corrected Kronecker-Factored Approximate Curvature) Hessian estimated on MLP parameters only, with attention held fixed and sequences truncated to 4096 tokens. Influence scores feed a mean-log-likelihood surrogate for benchmark accuracy, and the method extends to sequence-level attribution by removing a sentence and measuring the score change, and to token-level attribution by exploiting the autoregressive gradient decomposition. This lets the paper rank individual examples, cognitive behaviors, and tokens by their estimated contribution to math and code reasoning.

What would settle it

An independent check would retrain the model after deleting the examples that influence scores rank as most positive and compare accuracy against deleting random or low-ranked examples; if removing the highest-ranked examples does not hurt most, or if the rankings are unstable across retraining runs, the difficulty and flip findings lose their foundation.

Watch

Extended reading notes

Core claim

Infra, the paper's influence-based attribution method, computes for every supervised fine-tuning example its estimated contribution to the mean log-likelihood of correctly answered math and code benchmark problems. The central empirical claim is that high-difficulty math examples improve both math and code reasoning, whereas low-difficulty code examples are the most effective target for code reasoning, so the optimal data-mixing strategy is to combine hard math with easy code. Acting on this by flipping difficulty doubles AIME24 accuracy from 10% to 20% and boosts LiveCodeBench accuracy from 33.8% to 35.3% on Qwen2.5-7B-Instruct, with smaller gains on the 14B model. At finer granularity, the paper claims that exploratory behavior after reaching a correct answer improves both math and code reasoning, and that token-level influence patterns diverge across domains: math rewards natural-language logical connectors, while code rewards explicit structural syntax.

Load-bearing premise

The entire analysis rests on believing that the influence scores, computed with an approximate Hessian that only updates MLP weights while freezing attention and truncating long traces, accurately rank how much each training example truly contributes to the final model's reasoning.

Editorial extensions

If this is right

  • If the difficulty finding is correct, dataset reweighting by difficulty is a cheap and effective lever: within a fixed 17k-example budget, replacing easy math with hard math and hard code with easy code improves both math and code benchmarks together.
  • Cross-domain data deserves an explicit role in data curation: code reasoning can be improved by math examples, especially high-difficulty, formally symbolic ones, not only by in-domain code examples.
  • Exploratory overthinking in training traces should be preserved rather than pruned, since removing it drops MATH500 from 77.2 to 73.8 and LiveCodeBench from 33.8 to 32.0.
  • Token-level findings suggest that math training traces benefit from explicit logical connectors, while code training traces benefit from preserved structural formatting and syntax markers.
  • The reverse difficulty-mixing strategy degrades performance, supporting the direction of the finding rather than a generic preference for harder or easier data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would replace influence-score computation with a cheaper difficulty-and-domain heuristic and check whether it replicates the AIME and LiveCodeBench gains, which would suggest influence functions are identifying an effect that simpler proxies can capture.
  • The token-level divergence suggests a curriculum-ordering hypothesis the paper does not pursue: begin code training with structurally clean examples, then increase logical complexity through math, which could be tested by staged supervised fine-tuning.
  • Whether the difficulty-flip result transfers to reinforcement-learning post-training, larger models, or domains beyond math and code is left open; the proposed logic-versus-structure mechanism predicts transfer, but the paper demonstrates only supervised fine-tuning on 7B and 14B instruct models.
  • The sequence-level attribution depends on the classifier used to detect exploration and other cognitive behaviors, so varying the classifier threshold and re-measuring influence would test how sensitive the conclusions are to behavior detection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Infra, an influence-function framework for attributing the effects of SFT training examples on math and code reasoning in LLMs. It computes instance-, sequence-, and token-level influence scores using an EK-FAC Hessian approximation over MLP parameters, and uses these scores to identify dataset-level patterns. The main claims are that cross-domain examples help both math and code reasoning, that high-difficulty math and low-difficulty code examples are optimal, and that a 'difficulty-flip' reweighting strategy doubles AIME24 accuracy from 10% to 20% and improves LiveCodeBench from 33.8% to 35.3% on Qwen2.5-7B-Instruct. The paper also claims that exploratory behaviors in long CoTs are beneficial, supported by both influence scores and an SFT ablation.

Significance. If the central claims held, the paper would provide a practical, influence-based method for data-centric reasoning SFT and a nontrivial account of cross-domain transfer. The strengths include the clear formulation of instance-, sequence-, and token-level influence in Eqs. (3), (6), and (8), the explicit use of two SFT settings, and the independent SFT ablation in Table 2 (right) that directly validates the exploration finding. The difficulty-flip claim, however, currently rests on influence rankings that are not validated against retraining and on a dataset swap that changes both difficulty and data source, so the significance of the headline result cannot be assessed without additional validation.

major comments (4)
  1. [§4.2 / Table 1] The difficulty-flip experiment is confounded by dataset provenance. The text says that 'we replace simple math problems in the original dataset with more challenging ones sourced from a larger scale OpenThoughts-114k dataset' and similarly replace hard code problems with simpler ones. This changes not only difficulty but also the source of the examples, so the reported improvements (AIME24 10.0→20.0, LiveCodeBench 33.8→35.3) could be due to source distribution rather than difficulty. The footnote that OpenThoughts-114k is 'curated using the same pipeline' is an assumption, not a control. The authors should run a source-controlled comparison, for example by swapping difficulty labels within BS-17k itself or by matching examples across sources on difficulty and content, before attributing the gain to difficulty.
  2. [§3.1 / §4.1 / Limitations] The influence rankings that underlie Findings 1 and 2 are not validated against ground-truth retraining effects. The Hessian is approximated with EK-FAC over MLP parameters only, with attention held fixed and sequences truncated to 4096 tokens, as acknowledged in the Limitations section. Appendix C validates only the self-consistency of rankings across different sizes of D_correct; it does not test whether a high predicted influence actually corresponds to a larger change in the target function f(θ) when the example is removed or upweighted. This is a load-bearing gap because the difficulty-level averages in Figure 5 and the flip results in Table 1 depend on the correctness of those rankings. The authors should add a small-scale leave-one-out or subset-retraining validation comparing predicted influence with measured changes in f(θ).
  3. [§4.1 / Appendix C / Figure 5] The choice of n = 100 for D_correct is a free parameter, and the reported stability is weak. Table 3 shows Pearson correlations of 0.51–0.60 when comparing n=10 or n=25 to n=100, which is far from stable, and no variance or confidence intervals are reported for the average influence scores in Figure 5. Because the difficulty ordering is computed from a single random draw of D_correct, the authors should report bootstrap or repeated-sampling intervals and demonstrate that the difficulty-level ordering is robust to the choice of n.
  4. [§2 / §4.2] The claim that 'symbolic math examples and high-difficulty math problems are particularly effective in improving code reasoning' is based on grouping MATH data into difficulty levels, but the classification of data into categories such as FOBAR, SV, and 'college-level' versus 'high-school' is not precisely defined in the main text. Since these categories are used to draw conclusions about symbolic versus conversational data, the authors should define the taxonomy and report inter-annotator or classifier agreement.
minor comments (5)
  1. [§1 / footnote 2] The text 'Bespoke-Stratos-17k2' has a stray superscript that appears to be a misplaced footnote marker; please move the footnote marker to the correct location.
  2. [Figure 6] The right panel appears to show both a distribution of cognitive behaviors and average influence bars, but the caption does not explain the axes and scale of the two plots; please clarify which bars correspond to which metric and add error bars or per-behavior sample sizes.
  3. [Table 1] The caption says 'We report pass@1 accuracy of LiveCodeBench,' but the table also reports AIME and MATH500 scores; please specify the decoding and metric conditions for each benchmark.
  4. [Figure 5 / §4.2] The notation 'Math− →Code' in the Figure 5 caption is confusing; define the arrow notation in the text or use a consistent legend.
  5. [Appendix D] The GPT-4o truncation prompt in Appendix D asks for the 'number of distinct backtracking instances' even though the behavior being truncated is exploration; this appears to be a copy-paste error in the prompt template and should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the influence-based findings are tested by independent SFT interventions, not derived from fitted outcomes.

full rationale

The paper's load-bearing claims (difficulty mixing, exploration benefit) are derived from influence scores computed with Eq. 3, then validated by independent supervised fine-tuning experiments in Table 1 and Table 2 (right). The flip strategy is an intervention constructed from the influence-based hypothesis, and the AIME24/LiveCodeBench gains are new measurements, not values fitted from the influence scores. No parameter is fitted to the evaluation outcomes, and no conclusion reduces to its own definition: the influence score measures effect on the surrogate f(θ), while the SFT results measure actual benchmark accuracy. The method relies on the external EK-FAC approximation from Grosse et al. (2023), and the paper states the approximation as a limitation rather than importing a uniqueness theorem. There are no self-citations, no imported uniqueness arguments, and no renamed known results presented as derivations. The main weakness—that influence rankings are not checked against leave-one-out retraining—is a validity threat, not a circularity. Accordingly, no circular step can be quoted with a specific equation-to-equation reduction.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central empirical claims depend on several modeling choices (sample size n, truncation, difficulty labels) and on approximations inherited from prior influence-function work. The paper acknowledges the EK-FAC approximation but does not validate influence rankings against retraining. Its main experiment also assumes that replacement data from OpenThoughts-114k differs only in difficulty.

free parameters (4)
  • n, size of D_correct subsample = 100
    The surrogate objective in Eq. 5 averages over n=100 correctly answered benchmark examples. Appendix C reports Pearson correlations with the n=100 ranking as low as 0.51-0.60 for smaller n, so the choice affects instance-level influence rankings.
  • Sequence truncation length = 4096 tokens
    Influence computation truncates training sequences to 4096 tokens to fit memory, which can remove long CoT content and alter attribution.
  • Code difficulty labels in BS17k = Not specified
    Figure 5(a) groups code training examples by difficulty, but the paper does not describe how code difficulty levels are assigned, so the difficulty-flip design is not fully reproducible.
  • Token-level thresholds = top 100 examples, top 5% tokens
    Section 4.4 selects the top 100 influential examples and highlights the top 5% tokens; these thresholds are chosen by hand and the result is presented qualitatively.
assumptions (6)
  • standard math First-order influence function approximation via the implicit function theorem (Koh and Liang, 2017; Grosse et al., 2023).
    Equation 3 and Appendix A derive influence scores using the inverse Hessian and a first-order Taylor expansion, which the paper inherits from prior work.
  • domain assumption The SFT model is close enough to a minimum for the first-order influence approximation to be valid.
    Footnote 3 notes that for non-converged models the response function is the Proximal Bregman response, but the experiments use the standard Hessian-based formula over an SFT model that is not guaranteed to be converged.
  • domain assumption Mean log-likelihood on correctly answered benchmark examples is a valid surrogate for benchmark accuracy.
    Section 3.2 replaces non-differentiable accuracy with f(theta) in Eq. 5, and the paper does not show that increasing this likelihood transfers to accuracy in general.
  • domain assumption EK-FAC with MLP-only parameters and fixed attention approximates the true Hessian well enough to rank training data.
    Section 3.1 and the Limitations paragraph state that attention is treated as fixed and only MLP parameters are used, an approximation inherited from Grosse et al.
  • ad hoc to paper OpenThoughts-114k data differs from BS17k only in difficulty, because it is 'curated using the same pipeline'.
    Section 4.2 uses this assumption to attribute the Table 1 gains to difficulty flipping, but the examples, sources, and lengths may differ, confounding the experiment.
  • domain assumption LLM-based classifiers accurately identify exploration, verification, backtracking, subgoal setting, and backward chaining in CoT traces.
    Appendices D and E use Qwen-32B and GPT-4o prompts to label and truncate behaviors; errors in these labels would change the sequence-level influence and ablation results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions." pith.science (2026). https://pith.science/paper/OAH7242M

@misc{pith2026250519949,
  author       = {Pith},
  title        = {Pith review of: Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OAH7242M}},
  note         = {Machine review of arXiv:2505.19949}
}
read the original abstract

Large language models (LLMs) have demonstrated remarkable reasoning capabilities in math and coding, often bolstered by post-training on the chain-of-thoughts (CoTs) generated by stronger models. However, existing strategies for curating such training data predominantly rely on heuristics, limiting generalizability and failing to capture subtleties underlying in data. To address these limitations, we leverage influence functions to systematically attribute LLMs' reasoning ability on math and coding to individual training examples, sequences, and tokens, enabling deeper insights into effective data characteristics. Our Influence-based Reasoning Attribution (Infra) uncovers nontrivial cross-domain effects across math and coding tasks: high-difficulty math examples improve both math and code reasoning, while low-difficulty code tasks most effectively benefit code reasoning. Based on these findings, we introduce a simple yet effective dataset reweighting strategy by flipping task difficulty, which doubles AIME24 accuracy from 10\% to 20\% and boosts LiveCodeBench accuracy from 33.8\% to 35.3\% for Qwen2.5-7B-Instruct. Moreover, our fine-grained attribution reveals that the sequence-level exploratory behaviors enhance reasoning performance in both math and code, and the token-level influence patterns are distinct for math and code reasoning: the former prefers natural language logic connectors and the latter emphasizes structural syntax.

Figures

Figures reproduced from arXiv: 2505.19949 by the authors.

Figure 1
Figure 1. An illustration of our key findings towards the question: [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Cross-domain influence analysis of LLaMA3-8B-Base fine-tuned on combined Meta [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Average influence score of the training dataset combining MetaMathQA and OSS-Instruct, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Different types of MATH questions from Meta￾MathQA [39] dataset. To investigate how various training data types influence code reason￾ing, we further aggregate training samples by category and compute average influence scores per category. As illustrated in [PITH_FULL…
Figure 5
Figure 5. Figure 5: Left: Average influence scores of math and code training data from varying difficulty levels on reasoning performance. For instance, Math−→Code denotes the influence of math data on code reasoning tasks. Right: Distribution of math and code samples across difficulty le…
Figure 6
Figure 6. Figure 6: Left: An example of long CoT illustrating cognitive behaviors: verification (systematic error-checking) and exploration (searching for another approach after reaching the correct answer). Right: Distribution of different cognitive behaviors in BS-17k training dataset a…
Figure 7
Figure 7. Figure 7: Left: Visualization of top 5% influential tokens in math CoT. Right: Visualization of top 5% influential tokens in code CoT. is more impactful in programming tasks. This may be because programming requires breaking down high-level goals into modular components, making …
Figure 8
Figure 8. Figure 8: Cross-domain influence analysis of Qwen2.5-7B-Instruct fine-tuned on Bespoke-Stratos [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: To assess whether the exploration behavior has a positive or negative impact, we use [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Left: Exploration: When performing reasoning, seeking alternative approaches after reaching a correct answer. We capture this behavior and calculate the number of exploration steps by analyzing the content like "Another way to look at this is..." etc.. Right: Verifica…
Figure 11
Figure 11. Figure 11: Left: Backtracking: The behavior of realizing a path won’t work and explicitly going back to try a different approach. We capture this behavior and calculate the number of backtracking steps by finding the content like "This approach leads to a contradiction. Going ba…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

41 extracted references · 10 canonical work pages

  1. [1]

    Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models.arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  2. [2]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187, 2024

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms.arXiv preprint arXiv:2412.21187, 2024

    arXiv 2024

  3. [3]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  4. [4]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars.arXiv preprint arXiv:2503.01307, 2025

    arXiv 2025

  5. [5]

    Interpretable contrastive monte carlo tree search reasoning.arXiv preprint arXiv:2410.01707, 2024

    Zitian Gao, Boye Niu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, and Lijie Wen. Interpretable contrastive monte carlo tree search reasoning.arXiv preprint arXiv:2410.01707, 2024

    arXiv 2024

  6. [6]

    Fast approximate natural gradient descent in a kronecker factored eigenbasis.Advances in neural information processing systems, 31, 2018

    Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in a kronecker factored eigenbasis.Advances in neural information processing systems, 31, 2018

    work page 2018

  7. [7]

    The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv e-prints, pages arXiv–2407, 2024

    2024

  8. [8]

    Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296, 2023

    Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, and Ethan Perez. Studying large language model generalization with influence functions.arXiv preprint arXiv:2308.03296, 2023

    arXiv 2023

Show all 41 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  2. [10]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benc...

  3. [11]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track ...

  4. [12]

    Towards reasoning in large language models: A survey.arXiv preprint arXiv:2212.10403, 2022

    Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey.arXiv preprint arXiv:2212.10403, 2022

    2022 arXiv

  5. [13]

    O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson?arXiv preprint arXiv:2411.16489, 2024

    Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson?arXiv preprint arXiv:2411.16489, 2024

    2024 arXiv

  6. [14]

    Datamodels: Understanding predictions with data and data with predictions

    Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Understanding predictions with data and data with predictions. InInternational Conference on Machine Learning, pages 9525–9587. PMLR, 2022

    2022

  7. [15]

    Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card.arXiv preprint arXiv:2412.16720, 2024. 10

    2024 arXiv

  8. [16]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Ar- mando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. InThe Thirteenth International Conference on Lear...

    2021

  9. [17]

    Understanding black-box predictions via influence functions

    Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. InInternational conference on machine learning, pages 1885–1894. PMLR, 2017

    2017

  10. [18]

    Llm post-training: A deep dive into reasoning large language models.arXiv preprint arXiv:2502.21321, 2025

    Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models.arXiv preprint arXiv:2502.21321, 2025

    2025 arXiv

  11. [19]

    T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T\" ulu 3: Pushing frontiers in open language model post-training.arXiv preprint arXiv:2411.15124, 2024

    2024 arXiv

  12. [20]

    Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repository, 13:9, 2024

    Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions.Hugging Face repositor...

    2024

  13. [21]

    Codei/o: Condensing reasoning patterns via code input-output prediction.arXiv preprint arXiv:2502.07316, 2025

    Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. Codei/o: Condensing reasoning patterns via code input-output prediction.arXiv preprint arXiv:2502.07316, 2025

    2025 arXiv

  14. [22]

    Residuals and influence in regression, 1984

    Robert F Ling. Residuals and influence in regression, 1984

    1984

  15. [23]

    Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems.arXiv preprint arXiv:2412.09413, 2024

    Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems.arXiv preprint arXiv:2412.09413, 2024

    2024 arXiv

  16. [24]

    s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling.arXiv preprint arXiv:2501.19393, 2025

    2025 arXiv

  17. [25]

    Sky-t1: Train your own o1 preview model, 2024

    NovaSky. Sky-t1: Train your own o1 preview model, 2024. URL https://novasky-ai. github.io/posts/sky-t1

    2024

  18. [26]

    Gpt-3.5-turbo

    OpenAI. Gpt-3.5-turbo. Technical report, OpenAI, 2022

    2022

  19. [27]

    Estimating training data influence by tracing gradient descent.Advances in Neural Information Processing Systems, 33: 19920–19930, 2020

    Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent.Advances in Neural Information Processing Systems, 33: 19920–19930, 2020

    2020

  20. [28]

    Reasoning with language model prompting: A survey.arXiv preprint arXiv:2212.09597, 2022

    Shuofei Qiao, Yixin Ou, Ningyu Zhang, Xiang Chen, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, and Huajun Chen. Reasoning with language model prompting: A survey.arXiv preprint arXiv:2212.09597, 2022

    2022 arXiv

  21. [29]

    O1 replication journey: A strategic progress report–part 1

    Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report–part 1. arXiv preprint arXiv:2410.18982, 2024

    2024 arXiv

  22. [30]

    Pro- cedural knowledge in pretraining drives reasoning in large language models.arXiv preprint arXiv:2411.12580, 2024

    Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwarak Talupuru, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. Pro- cedural knowledge in pretraining drives reasoning in large language models.arXiv preprint arXiv:2411.12...

    2024 arXiv

  23. [31]

    A value for n-person games

    Lloyd S Shapley et al. A value for n-person games. 1953

    1953

  24. [32]

    Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419, 2025

    Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models.arXiv preprint arXiv:2503.16419, 2025. 11

    2025 arXiv

  25. [33]

    Magicoder: Em- powering code generation with oss-instruct.Proceedings of Machine Learning Research, 235: 52632–52657, 2024

    Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Em- powering code generation with oss-instruct.Proceedings of Machine Learning Research, 235: 52632–52657, 2024

    2024

  26. [34]

    Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10460, 2025

    Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond.arXiv preprint arXiv:2503.10460, 2025

    2025 arXiv

  27. [35]

    Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though.arXiv preprint arXiv:2501.04682, 2025

    Violet Xiang, Charlie Snell, Kanishk Gandhi, Alon Albalak, Anikait Singh, Chase Blagden, Duy Phung, Rafael Rafailov, Nathan Lile, Dakota Mahan, et al. Towards system 2 reasoning in llms: Learning how to think with meta chain-of-though.arXiv preprint arXiv:2501.04682, 2025

    2025 arXiv

  28. [36]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  29. [37]

    Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning.arXiv preprint arXiv:2502.03387, 2025

    2025 arXiv

  30. [38]

    Representer point selection for explaining deep neural networks.Advances in neural information processing systems, 31, 2018

    Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks.Advances in neural information processing systems, 31, 2018

    2018

  31. [39]

    Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models.arXiv preprint arXiv:2309.12284, 2023

    2023 arXiv

  32. [40]

    Mammoth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    2023 arXiv

  33. [41]

    Alternatively, maybe there’s a different way to approach the problem?

    Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, and Linda Ruth Petzold. Unveiling the impact of coding data instruction fine-tuning on large language models reasoning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 3...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.