Pith. sign in

REVIEW 4 major objections 5 minor 9 cited by

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On math reasoning tasks, large language models rely on pretraining documents that demonstrate the procedure, not documents that contain the answer.

desk verdict Careful influence-function study with a real finding, but the 'not retrieval' claim is weaker than it looks because influence is aggregated over the whole chain-of-thought. read the letter →

arxiv 2411.12580 v2 pith:XNCSOJBC submitted 2024-11-19 cs.CL cs.LG

Laura Ruis , Maximilian Mozes , Juhan Bae , Siddhartha Rao Kamalakara , Dwarak Talupuru , Acyr Locatelli , Robert Kirk , Tim Rocktäschel
show 2 more authors
Edward Grefenstette Max Bartolo
This is my paper · ORCID
classification cs.CLcs.LG
keywords influencefunctionsproceduralknowledgelanguagemodelreasoningpretrainingdataattributionchain-of-thoughtmathematicalcodeinselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models solve reasoning tasks by retrieving memorised answers from pretraining or by applying a generalisable procedure. To answer it, the authors compute influence-function rankings over 5 million pretraining documents for 80 factual and reasoning queries answered by two models of different sizes (7B and 35B). They find that a document's influence on one reasoning query strongly predicts its influence on another query of the same mathematical task, which is not true for factual queries. The exact answers to reasoning questions almost never appear among the most influential documents, while answers to factual questions often do. The authors conclude that the models synthesise procedural knowledge from documents that demonstrate the same kind of reasoning, rather than retrieving answers.

What carries the argument

The central machinery is the EK-FAC influence function, an approximate counterfactual attribution that estimates how much removing a pretraining document would change the log-probability of a completion, per nat of completion information. The paper applies it to rank 5 million documents for each query, using a Hessian estimated from a 100,000-document sample and query gradients compressed by approximate SVD; only MLP parameters are used, with the instruction-tuning stage treated as identity. The rankings are then compared across queries with Pearson correlations, searched for literal answers, inspected qualitatively for procedural content, and analysed by source dataset.

What would settle it

Retrain a 7B model on a pretraining corpus from which the top-ranked procedural documents, such as code that computes slope, are removed while answer-containing documents are retained; if accuracy on unseen reasoning queries does not drop, the claimed causal role of procedural documents fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLM reasoning on simple mathematical tasks is driven by procedural knowledge in pretraining documents: documents that show how to perform the relevant calculation, in formula or code, have similar influence across different queries requiring the same procedure, while documents containing the particular answer or intermediate steps are not highly influential. The paper contrasts this with factual question answering, where the answer itself regularly surfaces at the top of the influence ranking. Together with lower and less volatile per-document influence for reasoning, the authors interpret this as evidence for a generalisation strategy that combines procedural knowledge from many documents, unlike retrieval. They also report that code data is overrepresented among the most influential documents for reasoning, with StackExchange and mathematics sources appearing far more often than their pretraining share would predict.

Load-bearing premise

The analysis depends on the assumption that influence scores computed from MLP parameters alone, with the instruction-tuning stage treated as identity, identify the documents that actually drive the reasoning; if attention layers or fine-tuning carry the reasoning, the rankings could miss the true drivers.

Editorial extensions

If this is right

  • Pretraining data selection for reasoning does not need to cover every numerical case; documents that demonstrate procedures across diverse inputs should carry more weight.
  • Code and mathematics-heavy sources are disproportionately influential for mathematical reasoning, suggesting code data is a particularly valuable procedural training signal.
  • Factual question answering remains more retrieval-like: answers show up in top influential documents in 30 to 55 percent of cases, so contamination concerns are more acute for fact recall than for the simple reasoning tasks studied.
  • Larger models show an even stronger separation: the 35B model never had a reasoning answer among its top influential documents and relied on each individual document even less.
  • The same procedural documents can explain many different queries, with correlations in influence scores reaching 0.9 for queries that share a procedure, indicating a shared source of generalisable knowledge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this pattern generalises, reasoning benchmark contamination may be subtler than exact-answer overlap: a model could score well after seeing worked procedures or code patterns during pretraining even when the test question's answer never appears in training data.
  • A direct testable extension would be to pretrain two models on corpora that differ only in the presence of procedure-demonstrating documents, then measure transfer to unseen numbers; the paper's account predicts a clear accuracy gap.
  • Because the influence scores come only from MLP parameters, the actual arithmetic may be performed in attention layers while the feed-forward influence rankings surface the documents that supply the procedure; the paper's correlation patterns are consistent with that division of labour.
  • The near-zero correlation between the influence rankings of the 7B and 35B models suggests that procedural knowledge is instantiated in different documents for different model sizes, so data-selection conclusions may not transfer across scales without re-measuring.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies how two Cohere Command R models (7B and 35B) use pretraining data when producing chain-of-thought reasoning traces for three simple mathematical tasks, compared with factual question answering. The authors use EK-FAC influence functions to rank 5 million sampled pretraining documents (2.5B tokens) by their estimated influence on the log-likelihood of each query completion. They report four main findings: (1) influence scores for queries sharing a reasoning task are positively correlated, suggesting shared procedural knowledge; (2) reasoning queries rely on less individual per-nat influence than factual queries; (3) answers to reasoning questions rarely appear in the top 0.01% of influential documents, unlike factual answers; and (4) procedural documents, especially code, are overrepresented among influential documents for reasoning. The paper concludes that reasoning is 'unlike retrieval' and instead synthesizes procedural knowledge from documents demonstrating similar reasoning. The appendix contains counterfactual retraining experiments validating influence functions on accuracy, detailed approximation justifications, control-query analyses, and extensive qualitative results.

Significance. If the central claim holds, this is a valuable contribution to the interpretability and data-selection literatures: it moves beyond contamination-based explanations and suggests that pretraining data demonstrating procedures, rather than containing exact answers, drives reasoning generalization. The paper is methodologically ambitious, applying influence functions at 7B/35B scale, and it ships unusually strong supporting evidence: counterfactual retraining experiments (Appendix A.1) show that influence-based document removal degrades accuracy more than random or TracIn baselines; the authors verify their internal influence implementation against the public kronfluence code with Pearson R > 0.99; and they provide queries, completions, keywords, and prompts as supplementary material. These strengths make the empirical observations credible. However, the central interpretation rests on several load-bearing approximations and on inference from full-sequence influence rankings, so the evidence as presented is not yet conclusive enough for the broad 'unlike retrieval' conclusion.

major comments (4)
  1. [Section 4 and Eq. (1)] The claim that reasoning is 'unlike retrieval' rests heavily on Finding 3, but the influence score in Eq. (1) is computed on the full completion, which includes the chain-of-thought. As stated in Section 4, 'we calculate influence over the full completion.' For a long reasoning trace dominated by generic step-by-step phrasing, a document that mildly boosts many generic tokens can accumulate high total influence, while a document that strongly boosts only the final answer token may rank far lower. The per-nat normalization described in Section 3 is a scalar and does not reweight individual token contributions. Thus Finding 3 does not establish that the model is not retrieving the answer; it only shows that answer documents are not dominant influences on the whole sequence. The comparison to factual queries is confounded because factual completions are short and answer-heavy. I recommend token-level influence rankings, or at least a supplementary analysis restricting influence to the final answer token, to test whether the answer document rises in the ranking.
  2. [Section 4 (Models) and Appendix A.7] The experimental setup computes document gradients and EK-FAC Hessians with the base models, while query gradients and completions come from the supervised fine-tuned models, 'assuming the EK-FAC for the fine-tuning phase is the identity' (Section 4). This is a strong assumption: if SFT substantially changes the relevant parameters, the influence scores do not correspond to a well-defined counterfactual for the model that actually produced the completions. Appendix A.7 lists the assumption but does not validate it for the 7B and 35B models used here. Given that the central claim is about how the final models reason, this gap needs to be addressed, for example by comparing rankings computed from base-model query gradients with those from SFT-model query gradients on a small subset, or by citing direct validation evidence that SFT shifts are negligible for these specific models.
  3. [Finding 1 and Appendix A.9.1] The correlation analysis is presented as evidence for procedural knowledge, but the appendix shows that formatting and superficial similarity can explain a substantial portion of the effect. For example, the reasoning control queries, which require no reasoning but share formatting, sometimes achieve within-group correlations up to 0.38 (Appendix A.9.1), and for the 35B linear equations only 34 of 190 within-task correlations exceed the maximum correlation with control queries. The paper acknowledges part of the correlation is driven by formatting, but the quantitative decomposition is not provided. Without controlling for completion template and document-gradient overlap, Finding 1 is weaker evidence for shared procedures than the abstract claims suggest. I recommend reporting correlations on token-level or template-normalized influences, or explicitly quantifying how much of the same-task correlation survives when generic reasoning tokens are excluded.
  4. [Appendix A.7] The influence calculations only cover MLP parameters because EK-FAC is not properly defined for attention layers. The paper acknowledges this in Appendix A.7 and in the discussion, but the central claim is phrased broadly as 'the approach to reasoning the models use.' If procedural reasoning is substantially implemented in attention layers, the rankings could miss the true influential documents. This is a load-bearing scope limitation for the 'unlike retrieval' conclusion. I suggest either restricting the claims to knowledge stored in MLP parameters or adding evidence, such as attention-head attribution or ablation studies, that attention-layer contributions do not change the qualitative findings.
minor comments (5)
  1. [Section 3] There is a typo: 'more accuractely' should be 'more accurately.'
  2. [Section 4] The phrase 'distributed similary as the training distribution' should read 'distributed similarly to the training distribution.'
  3. [Appendix A.8.1] In the PEMDAS example, 'the perform the operations' should be 'they perform the operations.'
  4. [Section 5.1 and Appendix A.9.3] The power-law slope comparison in Table 28 is reported with significance stars, but the discussion in Appendix A.9.3 notes that the steepest 35B slope may be noise and that the top document is unrelated. Please clarify whether the reported slope differences are robust to excluding this query, since the current presentation is potentially misleading.
  5. [Figure 1] The figure is dense and the redacted document snippets are hard to read; consider enlarging the document text or splitting the figure into two panels with clearer annotations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical observations from influence-function rankings, and the method is validated by independent counterfactual retraining and a public-code correlation check.

full rationale

The paper's central claims are empirical findings about which pretraining documents receive high influence scores for reasoning versus factual completions. The influence definition in Equation 1 (log-likelihood of the completion as a function of pretraining documents) is an input assumption, not a disguised version of the conclusion that reasoning is procedural rather than retrieval-based. The comparison between reasoning and factual queries is made on real rankings over 5M documents, and the qualitative checks (answer presence, source-dataset overrepresentation, procedural content) are separate measurements. The heavy reliance on Grosse et al. (2023) and Bae et al. (2024) involves overlapping authors (Juhan Bae is a co-author of the present paper and of both cited works), but the load-bearing method does not reduce to an unverified self-citation: the authors report a Pearson's R of 0.993 between their internal implementation and the public kronfluence codebase, and Appendix A.1 contains independent counterfactual retraining experiments showing that EK-FAC influence scores identify documents that causally affect downstream accuracy more than random removal or TracIn. The stated assumption that the EK-FAC for the SFT stage is the identity (Bae et al., 2024) is explicit and is treated as a limitation rather than silently used to define the result. The strongest potential critique, that influence is computed over the full chain-of-thought completion so answer-token influence may be diluted by procedural tokens, is a validity threat to the interpretation of Finding 3, not a circularity: no equation in the paper defines 'answer retrieval' as 'top-ranked influence on the full completion' and then presents that definition as the finding. Similarly, the correlation in Finding 1 is discussed alongside control queries and the authors explicitly note that part of the correlation is driven by formatting, which is an honest analysis rather than a constructed equivalence. Overall, the derivation chain is self-contained: the findings are conditional on an explicitly stated influence measure, and the empirical content does not collapse into the input by construction.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the validity of the EK-FAC influence approximations, the representativeness of the 2.5B-token sample, the exclusion of attention layers and the SFT stage, and the adequacy of the control and qualitative analyses. No free parameters are fitted to the target result, but the task selection threshold and sampling choices shape the scope.

free parameters (4)
  • Task selection accuracy threshold = 80%
    Reasoning tasks were selected only if the model achieved at least 80% zero-shot chain-of-thought accuracy on a 100-question pilot (Section 4). This determines which tasks are studied and may bias toward tasks where procedural generalization already occurs.
  • EK-FAC damping factor = 0.1
    Used for Hessian estimation for both models (Appendix A.2). Standard value from Grosse et al. (2023), not fitted to the queries.
  • Block-diagonal approximation blocks = 2 (7B), 4 (35B)
    Chosen for tractability in the eigendecomposition of EK-FAC matrices (Appendix A.2). Not fitted to data; affects the approximation error.
  • Number of sampled pretraining documents = 5 million (2.5B tokens)
    Computational budget decision (Section 4). The representativeness of this sample is a key assumption.
assumptions (6)
  • domain assumption EK-FAC influence functions with the listed approximations (first-order Taylor, block-diagonal, MC sampling, low-rank SVD) estimate the true counterfactual influence of pretraining documents on completions.
    The paper relies on Grosse et al. (2023) for these approximations and validates only on small fine-tuning counterfactuals (Appendix A.1).
  • domain assumption Influence on MLP parameters alone captures the documents that drive reasoning completions.
    Attention layers are excluded from EK-FAC because the method is not defined for them (Appendix A.7). Reasoning implemented in attention would be invisible.
  • domain assumption The supervised fine-tuning (SFT) stage has identity EK-FAC, so pretraining influence is unaffected by SFT.
    Assumed from Bae et al. (2024), stated in Section 4 and Appendix A.7.
  • domain assumption The 5M-document sample, drawn uniformly from batches, is representative of the full pretraining distribution and contains any documents that would be highly influential.
    Stated in Section 4 and Discussion; the authors acknowledge the alternative hypothesis that influential documents are outside the sample.
  • ad hoc to paper Control queries with superficial similarity but no required reasoning isolate the effect of procedure from formatting and topic.
    The control set was constructed for this paper; correlations in controls reach up to 0.38, so the isolation is imperfect (Appendix A.9.1).
  • ad hoc to paper The qualitative classifications by Command R+ and manual keyword searches correctly identify answers and procedural content in top documents.
    Used to support Findings 3 and 4; the paper verifies Command R+ against manual hits but does not provide exhaustive ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models." pith.science (2026). https://pith.science/paper/XNCSOJBC

@misc{pith2026241112580,
  author       = {Pith},
  title        = {Pith review of: Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNCSOJBC}},
  note         = {Machine review of arXiv:2411.12580}
}
read the original abstract

The capabilities and limitations of Large Language Models have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.

Figures

Figures reproduced from arXiv: 2411.12580 by the authors.

Figure 1
Figure 1. A summary of our most important findings towards answering the question [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The total influence per nat of query completion information for different portions of the [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. We search for the answer in the top 500 (top 0.01%) documents, and find it relatively frequently for the factual ques￾tions. For the reasoning questions, we find the answer twice for the 7B, and never for the 35B. Both those times, the answers to the steps occur in separate documents. To find the answer to the questions in the queries in the top documents manually, we construct keywords for each query that should be… view at source ↗
Figures from the paper (22 more)
Figure 4
Figure 4. Figure 4: (a) Counterfactual retraining experiments on Wikitext-2. We finetuned GPT-2 (124M) [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 5
Figure 5. Figure 5: Counterfactual retraining experiments on reading comprehension benchmark DROP (a) [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: The pipeline for creating rankings of the most influential pretraining documents for a [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: For the reasoning and factual sets, we compare the amount of documents from a certain source dataset that show up in the top portions of the rankings to the amount you would expect to show up if you randomly sample from the pretraining distribution (indicated by ‘Train…
Figure 8
Figure 8. Figure 8: For the reasoning and factual sets, We compare the amount of documents from a certain source dataset that show up in the bottom portions of the rankings to the amount you would expect to show up if you randomly sample from the pretraining distribution (indicated by ‘Tr…
Figure 9
Figure 9. Figure 9: For the query control sets, we also compare the amount of documents from a certain source dataset that show up in the top portions of the rankings to the amount you would expect to show up if you randomly sample from the pretraining distribution (indicated by ‘Training…
Figure 10
Figure 10. Figure 10: For the query control sets, we also compare the amount of documents from a certain source dataset that show up in the bottom portions of the rankings to the amount you would expect to show up if you randomly sample from the pretraining distribution (indicated by ‘Trai…
Figure 11
Figure 11. Figure 11: Comparison of capability categories identified for the most and least influential docu [PITH_FULL_IMAGE:figures/full_fig_p047_11.png]
Figure 12
Figure 12. Figure 12: The correlation between the influence scores of all 5 million documents for pairs of [PITH_FULL_IMAGE:figures/full_fig_p048_12.png]
Figure 13
Figure 13. Figure 13: The average correlations between the influences of all documents for queries of a specific [PITH_FULL_IMAGE:figures/full_fig_p048_13.png]
Figure 14
Figure 14. Figure 14: The total influence per nat of query completion information for different portions of the [PITH_FULL_IMAGE:figures/full_fig_p057_14.png]
Figure 15
Figure 15. Figure 15: The total influence per nat of query completion information for different portions of the [PITH_FULL_IMAGE:figures/full_fig_p057_15.png]
Figure 16
Figure 16. Figure 16: The total influence per nat of query completion information for different portions of [PITH_FULL_IMAGE:figures/full_fig_p058_16.png]
Figure 17
Figure 17. Figure 17: The total influence per nat of query completion information for different portions of [PITH_FULL_IMAGE:figures/full_fig_p058_17.png]
Figure 18
Figure 18. Figure 18: The total influence per nat of query completion information for different number of [PITH_FULL_IMAGE:figures/full_fig_p059_18.png]
Figure 19
Figure 19. Figure 19: The total influence per nat of query completion information for different number of [PITH_FULL_IMAGE:figures/full_fig_p059_19.png]
Figure 20
Figure 20. Figure 20: The ranked influence scores per query nat for each query shown separately in log-log [PITH_FULL_IMAGE:figures/full_fig_p060_20.png]
Figure 21
Figure 21. Figure 21: The ranked influence scores per query nat for each query shown separately in log-log [PITH_FULL_IMAGE:figures/full_fig_p060_21.png]
Figure 22
Figure 22. Figure 22: The percentage of total influence per nat of query completion information for different [PITH_FULL_IMAGE:figures/full_fig_p062_22.png]
Figure 23
Figure 23. Figure 23: The percentage of total influence per nat of query completion information for different [PITH_FULL_IMAGE:figures/full_fig_p062_23.png]
Figure 24
Figure 24. Figure 24: The percentage of total influence per nat of query completion information for different [PITH_FULL_IMAGE:figures/full_fig_p063_24.png]
Figure 25
Figure 25. Figure 25: The percentage of total influence per nat of query completion information for different [PITH_FULL_IMAGE:figures/full_fig_p063_25.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A K-5-only pretraining corpus and 5B model show that language model capabilities track the knowledge boundary of the training data, and standard post-training methods do not cross it.

  2. Can Large Language Models Generalize Procedures Across Representations?

    cs.CL 2026-02 conditional novelty 6.0 of 10

    Post-training on graph or code versions of a planning task does not transfer to natural-language versions, but a symbolic-then-natural-language RL curriculum achieves strong transfer.

  3. Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Different procedural pretraining tasks create complementary, transferable structures in a transformer's attention and MLP weights, and structures from different tasks can be combined into one initialization.

  4. Which Data Attributes Stimulate Math and Code Reasoning? An Investigation via Influence Functions

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Influence-function attribution shows high-difficulty math and low-difficulty code training data best improve math and code reasoning, and difficulty-based reweighting improves benchmark performance.

  5. Procedural Pretraining: Warming Up Language Models with Abstract Data

    cs.CL 2026-01 conditional novelty 5.0 of 10

    A short warm-up on procedural data (brackets, sorting, sets) makes language models more accurate and more data-efficient on language, code, and informal math.

  6. Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs

    cs.CL 2025-11 reject novelty 5.0 of 10

    Reasoning LLMs seem better at retrieving hierarchical facts not because they know more but because they navigate better; the key supporting RL experiment is missing from the paper.

  7. Beyond Explainable AI (XAI): An Overdue Paradigm Shift and Post-XAI Research Directions

    cs.CY 2026-02 unverdicted novelty 4.0 of 10

    Current XAI methods for DNNs and LLMs rest on paradoxes and false assumptions that demand a paradigm shift to verification protocols, scientific foundations, context-aware design, and faithful model analysis rather th...

  8. Not Every AI Problem is a Data Problem: We Should Be Intentional About Data Scaling

    cs.LG 2025-01 unverdicted novelty 4.0 of 10

    Data shape, including compositional and structural patterns, should guide decisions about which AI tasks benefit from data scaling.

  9. A Survey on Explainable Deep Reinforcement Learning

    cs.LG 2025-02 conditional novelty 3.0 of 10

    A survey that organizes explainable DRL methods into feature-, state-, dataset-, and model-level approaches and reviews their evaluation, security, and LLM-related uses.

Reference graph

Works this paper leans on

22 extracted references · 16 canonical work pages · cited by 9 Pith papers

  1. [1]

    Identify the coordinates of the two points: - Point A: (93, 28) - Point B: (74, 47)

    work page

  2. [2]

    URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, Jeff Schneider, Eduard Hovy, Roger Grosse, and Eric Xing. What is your data worth to gpt? llm-scale...

    arXiv 2020

  3. [3]

    Calculate the difference in y-coordinates between the two points: - Difference in y-coordinates: y2 - y1 = 47 - 28 = 19

    work page

  4. [4]

    Determine the ratio of the difference in y-coordinates to the difference in x-coordinates: - Ratio: 19 / -19 = 1 / -1

    work page

  5. [5]

    23 Table 9: Example from the reasoning set that requires solving for x in a linear equation

    Multiply the ratio by -1 to reverse the direction of the line: - Slope: -1 / 1 = -1 Therefore, the slope of the line passing through the points (93, 28) and (74, 47) is -1. 23 Table 9: Example from the reasoning set that requires solving for x in a linear equation. We have twenty other examples like these on different numbers, only for the 35B model. The ...

    work page

  6. [6]

    Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P

    URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf. Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P. Dick, Hidenori Tanaka, Tim Rockt¨aschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. In The Twelfth I...

    arXiv 2022

  7. [9]

    Calculate the difference in x-coordinates between the two points: - Difference in x-coordinates: x2 - x1 = 74 - 93 = -19

    work page

  8. [13]

    Subtract 21 from both sides of the equation: 5x + 21 - 21 = 91 - 21 5x = 70

    work page

Show all 22 references
  1. [14]

    Goat Nebula

    Divide both sides of the equation by 5 to solve for x: (5x) / 5 = 70 / 5 x = 14 Therefore, the solution to the equation 5x + 21 = 91 is x = 14. Table 10: Example from the factual control set that resembles the question about the tallest mountain in the world from the 7B factua...

    2023

  2. [15]

    What is the longest word in the English language? Answer: The longest word in the English language is ”pneumonoultramicroscopic- silicovolcanoconiosis”, which is a type of lung disease

  3. [16]

    How many bones are in the human body? Answer: There are 206 bones in the human body

  4. [17]

    What is the largest ocean in the world? Answer: The largest ocean in the world is the Pacific Ocean, which covers approxi- mately one-third of the Earth’s surface

  5. [18]

    What is the longest river in the world? Answer: The longest river in the world is the Nile River, which is 6,650 kilometers (4,130 miles) long

  6. [19]

    What is the highest mountain in the world? Answer: The highest mountain in the world is Mount Everest, which is 8,848 meters (29,029 feet) high

  7. [20]

    Calculate the answer: (5 - 3) * 12. Think step-by-step

    What is the most common element in the human body? Answer: The most common element in the human body is oxygen, which makes up approximately 65% of the human body’s mass. ... redacted 36 Another document has the answer to part of an arithmetic query for the 7B (“Calculate the ...

    2023

  8. [21]

    Identify the coordinates of the two points: - Point A: (79, 47) - Point B: (80, 25)

  9. [22]

    Au”. Factual prompt 2 What is the common name for the larva of a housefly? Completion to factual prompt 2 The common name for the larva of a housefly is “maggot

    Calculate the difference in x-coordinates between the two points: - Point A x-coordinate: 79 - Point B x-coordinate: 80 ... redacted completion. Slopes (control prompt) The slope of the line is -22. What is the slope of the line? Think step-by-step. Completion Given that the s...

    2023

  10. [2011]

    URL https://doi.org/10.1137/090771806

    doi: 10.1137/090771806. URL https://doi.org/10.1137/090771806. Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association , 69(346):383–393, 1974. doi: 10.1080/01621459.1974. 12 10482962. URL https://www.tandfonline....

    1974

  11. [2018]

    URL https://proceedings.neurips.cc/paper_files/paper/2018/ file/48000647b315f6f00f913caa757a70b3-Paper.pdf. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamil˙e Lukoˇsi¯u...

    2018

  12. [2020]

    URL https://proceedings.mlr.press/v108/barshan20a.html. Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nichol...

    2023

  13. [2022]

    URL https://arxiv.org/abs/2204.02311. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ar...

    arXiv

  14. [2023]

    Tomer Ullman

    URL https://arxiv.org/abs/2307.09288. Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks, 2023. URL https://arxiv.org/abs/2302.08399. Boshi Wang, Xiang Yue, Yu Su, and Huan Sun. Grokked transformers are implicit reasoners: A mechanistic jou...

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.