REVIEW 5 major objections 6 minor 2 cited by
Modular Arithmetic: Language Models Solve Math Digit by Digit
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Simple arithmetic in LLMs is solved by independent, position-specific MLP circuits, and swapping a circuit's activations changes only that digit of the answer.
desk verdict Causal evidence for digit-position circuits is suggestive but the missing full-distribution analysis leaves the headline claim under-supported; worth a serious referee but needs revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the digit-position circuit: a thresholded set of MLP neurons per layer, selected only from layers after operand information has reached the residual stream, whose activations have high Fisher Score for the digit-pair class at one position (units, tens, hundreds). The Fisher Score $F_{i,d} = \frac{\sum_c |X_{c,d}|(\mu_{i,c,d}-\mu_{i,d})^2}{\sum_c |X_{c,d}|\sigma^2_{i,c,d}}$ measures how well neuron $i$ separates digit-pair classes at position $d$, with high scores meaning the neuron reliably discriminates the subtask. Validation uses interchange interventions, also called activation patching, which replace circuit activations from a base prompt with those from a source prompt and measure the probability over the eight digit-wise combinations of the two results. The selective rise of the single expected variant, such as sbb for a hundreds intervention, is the evidence that the circuit carries that digit position.
What would settle it
Run the same Fisher-Score selection and interchange intervention on random three-digit additions with carries at every position. If patching the units circuit with a carry-producing source raises the probability of the carry-adjusted variant (bb+1s) as much as or more than the source-units-only variant (bbs), then digit circuits are not independent in the general setting.
Extended reading notes
Core claim
Digit-position-specific arithmetic circuits, meaning subgroups of MLP neurons distributed across mid-to-late layers, generate the units, tens, and hundreds digits of arithmetic results in parallel and independently. A Fisher Score selects neurons whose activations discriminate between digit-pair classes at a given position; linear classification confirms these neuron groups are sufficient to represent the subtask, and interchange interventions show their causal role. Patching a circuit's activations shifts probability only toward the result variant with the source digit at that position, for example bbs for a units intervention, by 15 to 50 percentage points, while other variants barely move. Addition and subtraction circuits are largely distinct neuron sets, carry effects are not embedded in the digit circuits, and prompts sharing a digit subtask produce high within-circuit activation similarity. In a single-digit-tokenization model the same modular principle appears as one dominant circuit for the digit being emitted plus a smaller circuit plausibly responsible for carry bits.
Load-bearing premise
The main demonstrations use only no-carry problems, so the independence of digit circuits in ordinary arithmetic with carries rests on two supplementary scenarios rather than a full test.
Editorial extensions
If this is right
- Interventions on a digit-position circuit act as a digit editor: they can change a predicted answer at the units, tens, or hundreds position without disturbing the other positions.
- The same structured circuits appear in 7B, 8B, 9B, and 70B models and under both multi-digit and single-digit numeric tokenization, so the mechanism is not an artifact of a particular tokenizer or scale.
- Addition and subtraction are implemented by largely non-overlapping neuron sets, so the model maintains separate modular machinery for the two operations.
- Carry information is not stored inside the digit circuits and is processed by separate mechanisms, implying that full multi-digit arithmetic requires composition of position-local results with a distinct carry-handling component.
- Heuristic neurons reported in earlier work, such as neurons responding to result parity or result range, align with the digit circuit they belong to, suggesting heuristics are single-neuron fragments of modular circuits rather than an alternative explanation.
Reading between the lines
- If digit circuits are genuinely independent and carry handling is separate, then arithmetic errors should be position-specific: a model that has learned 3+1 but not 3+2 should fail only at the position where that subtask appears, not across the whole answer.
- The independence result suggests practical activation-editing applications: a wrong digit could be corrected by patching a single digit circuit rather than retraining, provided the carry machinery is left intact.
- The carry experiments cover only units-to-tens and tens-to-hundreds; a natural extension is chained carries such as 999+1, where propagation across two positions would test whether distinct carry mechanisms compose cleanly.
- The same Fisher-Score circuit discovery procedure could be applied to multiplication, division, or multi-operand addition to test whether modularity by digit position is a general arithmetic strategy or specific to two-operand addition and subtraction.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper claims to identify digit-position-specific circuits in the MLP layers of LLMs that independently compute the units, tens, and hundreds digits of addition and subtraction results. Using Fisher Score feature selection on one dataset and interchange interventions on separate paired prompts, the authors report that patching a circuit for a given digit position selectively changes that digit in the output while leaving other positions unchanged, across three multi-digit-tokenization models (Llama 3 8B, Llama 3 70B, Olmo 2 7B) and one single-digit-tokenization model (Gemma 2 9B, in an appendix). They additionally report that addition and subtraction circuits are largely distinct, that carry information is handled outside the digit circuits, and that previously described 'heuristic' neurons can be reinterpreted within the digit-circuit framework.
Significance. If the central claim holds, the paper would be a meaningful advance in mechanistic interpretability: it offers causal evidence that LLMs implement simple arithmetic through independent, position-specific computational modules rather than only a bag of unrelated heuristics. The study is well positioned relative to work on digit-wise number representations and on heuristic neurons. Strengths include the use of separate datasets for neuron identification and intervention, replication across four models and two tokenization schemes, and the public release of code and data. The main weakness is that the intervention evaluation is currently incomplete: the reported metrics do not account for probability mass outside eight hand-picked output strings, and the in-table evidence for selectivity is weaker than the text claims. These issues are load-bearing for the paper's central causal claim.
major comments (5)
- [Section 3.2, Table 2] The intervention evaluation only tracks probability changes on the 8 digit-wise combinations of the base and source results (bbb, bbs, ..., sss), but the next-token distribution over the full vocabulary is never reported. For the Llama 3 8B addition unit intervention, the reported deltas sum to roughly -30 percentage points, implying that about 30 percentage points of probability mass moves to outputs outside the 8-variant set after the intervention. Those unmeasured outputs are not constrained to preserve the base tens and hundreds digits, so the observed increases in bbs, bsb, or sbb do not by themselves establish that non-target digits are unaffected. The authors should report the full output distribution before and after intervention, or at least the combined probability of all non-variant outputs, and preferably show that the non-target digits in those other outputs remain at their base values.
- [Section 3.2, Table 2] The text states that 'non-target digits remain largely unaffected, as indicated by minimal changes observed in other result variants,' but Table 2 shows substantial increases in variants that contain multiple source digits. For the Llama 3 8B addition unit intervention, bss increases by +10.59 percentage points and sbs by +2.57; for the Olmo 2 7B unit intervention, sbb increases by +11.27 and sbs by +5.89. These changes are not minimal relative to the targeted increase (e.g., +30.93 for bbs in Llama 3 8B), and they indicate that the intervention also changes the tens or hundreds digit in a non-negligible fraction of cases. The specificity claim should be quantified with a metric that directly penalizes changes in non-target digit positions, such as the probability mass on outputs whose non-target digits differ from the base.
- [Section 3.2, threshold selection] The paper reports results for the 'best threshold t*' for each circuit, selected from t ∈ {0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0}, and Table 2 uses a different t* per circuit. If t* is chosen after inspecting the same intervention data that produces the reported effect sizes, the results are optimistically biased. The authors should either pre-register the thresholds, select them on a separate validation set of intervention prompts, or report results for all thresholds together with a correction for multiple comparisons so that the reported effect sizes are unbiased estimates.
- [Section 2.2 and Section 4.2] The main experiments are restricted by construction to problems with no carry at any digit position, so the claim that 'language models solve math digit by digit' is directly tested only in a regime where digit positions are independent by design. The carry study in Section 4.2 examines only two specific carry scenarios (units-to-tens and tens-to-hundreds) and shows that carry information is not embedded in the digit circuits, but it does not test cases with simultaneous carries at multiple positions or carry propagation through two digits. To support the general wording of the title and abstract, the authors should either extend the carry analysis to a broad carry-inclusive dataset or explicitly frame the main claim as applying to the no-carry regime.
- [Tables 2 and 3] All effect sizes are reported as single point estimates with no error bars, confidence intervals, or per-sample variability, even though they are averaged over 200 intervention pairs. Given the wide range of flip rates in Table 3 (from 7.5% to 68.5%), the stability of the reported effects is unclear. The authors should report the distribution of per-sample probability changes (e.g., mean ± standard deviation or bootstrapped confidence intervals) for at least the main Llama 3 8B results so that the reliability of the effects can be assessed.
minor comments (6)
- [Section 1] There is a typo in 'This finding is suprising' — it should be 'surprising'.
- [Table 2] In the Llama 3 8B addition hundreds row, the entry '+0.50' lacks a percent sign and is inconsistent with the other entries; it should read '+0.50%'.
- [Section 2.3] The text says 'we construct Dsub,op1 and Dsub,op1 where o = −' — the second dataset name should be Dsub,op2.
- [Section 3.1] There is a typo: 'we identify hich neurons' should be 'which neurons'.
- [Section 6] The sentence contains 'digt-wise representation in of numbers'; it should be 'digit-wise representation of numbers'.
- [Appendix E] The paper describes the identified sets as 'circuits' while noting that they comprise about 60% of MLP neurons per layer at the best thresholds; the authors should briefly discuss whether such broad sets are consistent with the usual notion of a sparse circuit, or clarify the intended meaning of 'circuit'.
Circularity Check
Core Fisher-score intervention tests are externally validated and not tautological; residual circularity is limited to a load-bearing self-citation in the single-digit-tokenization appendix.
-
self citation load bearing
[Appendix B (Results on Single Digit Tokenization Models)]
"Based on the intuition given above and previous work that finds LLMs internally generate one result digit more than needed for their current generation to account for carry bits, but no further digits (Baeumel et al., 2025), we conclude the following: Single-digit tokenization models also have digit-position-specific arithmetic circuits, which consist of one dominant circuit responsible for predicting the digit needed for the output generation, and one smaller and shorter circuit which may be responsible for determining whether a carry-bit influences the generation."
The claim that digit-position-specific circuits exist independently of tokenization strategy is extended to Gemma by invoking the authors' own prior result to reinterpret the absence of any unit-digit circuit as expected. This prior work is not independently established inside the present paper, and it carries the load of making the Gemma observations consistent with the hypothesis. If the self-citation were removed, the observed non-existence of unit-digit neurons would weigh against the universal claim rather than support it. The central multi-model evidence is independent, so this is a partial rather than total circularity.
full rationale
The main localization loop is not circular: Fisher Scores are computed from input operand digit-pair class labels, not from the intervention outcomes, and the causal test patches selected neurons on paired base/source prompts and checks whether only the targeted result digit changes. That is an external validation, not a tautology. The reported threshold t* is a hyperparameter, and the evaluation over eight digit-wise combinations in Section 3.2 is an evidential limitation (unmeasured probability mass outside those variants) rather than a definitional equivalence. The construction of no-carry data in Section 2.2 does make digit-wise subtasks independent by design, but the paper still provides causal evidence about where and how the network implements them. The only load-bearing self-citation is in Appendix B, where a missing unit circuit in the single-digit-tokenization model is reinterpreted using the authors' prior lookahead result. That step gives the paper a mild circularity score of 2, with the main contribution remaining independently supported.
Assumptions & free parameters
free parameters (3)
- Fisher Score threshold t* per circuit =
e.g., Llama 3 8B addition: unit 0.6, tens 0.5, hundreds 0.9
- Layer range L =
e.g., Llama 3 8B add: 15-24 (op2), 16-24 (op1)
- No-carry data constraint =
n/a
assumptions (3)
- domain assumption MLP-only analysis captures the relevant digit circuits
- domain assumption Fisher Score thresholding selects computational circuits
- ad hoc to paper No-carry tasks are representative of arithmetic strategy
invented entities (1)
-
Digit-position-specific arithmetic circuits
independent evidence
Cite this review
Pith. "Pith review of Modular Arithmetic: Language Models Solve Math Digit by Digit." pith.science (2026). https://pith.science/paper/SKVDAGUG
@misc{pith2026250802513,
author = {Pith},
title = {Pith review of: Modular Arithmetic: Language Models Solve Math Digit by Digit},
year = {2026},
howpublished = {\url{https://pith.science/paper/SKVDAGUG}},
note = {Machine review of arXiv:2508.02513}
}
read the original abstract
While recent work has begun to uncover the internal strategies that Large Language Models (LLMs) employ for simple arithmetic tasks, a unified understanding of their underlying mechanisms is still lacking. We extend recent findings showing that LLMs represent numbers in a digit-wise manner and present evidence for the existence of digit-position-specific circuits that LLMs use to perform simple arithmetic tasks, i.e. modular subgroups of MLP neurons that operate independently on different digit positions (units, tens, hundreds). Notably, such circuits exist independently of model size and of tokenization strategy, i.e. both for models that encode longer numbers digit-by-digit and as one token. Using Feature Importance and Causal Interventions, we identify and validate the digit-position-specific circuits, revealing a compositional and interpretable structure underlying the solving of arithmetic problems in LLMs. Our interventions selectively alter the model's prediction at targeted digit positions, demonstrating the causal role of digit-position circuits in solving arithmetic tasks.
Figures
Figures from the paper (35 more)
Forward citations
Cited by 2 Pith papers
-
Are Arithmetic Heuristic Neurons Form-Invariant? A Mechanistic Analysis of Symbols, Text, and Code in LLMs
Llama-3 arithmetic is computed by a shared neuron set across symbolic, text, and code formats, and cross-format failures are activation-state differences, not distinct circuits.
-
Automated Attention Pattern Discovery at Scale in Large Language Models
AP-MAE reconstructs masked attention patterns in LLMs with high accuracy, generalizes across models, predicts generation correctness at 55-70%, and enables 13.6% accuracy gains via targeted interventions.
Reference graph
Works this paper leans on
-
[5]
The llama 3 herd of models. Preprint, arXiv:2407.21783. Quanquan Gu, Zhenhui Li, and Jiawei Han
-
[7]
Lan- guage models use trigonometry to do addition. Preprint, arXiv:2502.00873. Amit Arnold Levy and Mor Geva
-
[9]
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov
What is a number, that a large language model may know it? arXiv preprint arXiv:2502.01540. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov
-
[10]
Language Models Encode Numbers Using Digit Representations in Base 10
arXiv preprint arXiv:2410.11781. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, et al
-
[11]
Locating and Editing Factual Asso- ciations in GPT. arXiv preprint. ArXiv:2202.05262 [cs]. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt
-
[12]
Progress mea- sures for grokking via mechanistic interpretability. arXiv preprint. ArXiv:2301.05217 [cs]. Yaniv Nikankin, Anja Reusch, Aaron Mueller, and Yonatan Belinkov
-
[13]
Arithmetic without algo- rithms: Language models solve math with a bag of heuristics. Preprint, arXiv:2410.21272. Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter
-
[14]
2 olmo 2 furious. Preprint, arXiv:2501.00656. Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan
Show all 19 references
-
[15]
arXiv preprint
A Mechanistic Interpretation of Arithmetic Reasoning in Language Models us- ing Causal Mediation Analysis. arXiv preprint . ArXiv:2305.15054 [cs]. Lin Sun, Tianxiang Wang, Weiping Ding, Jiucheng Xu, and Yaojin Lin
-
[16]
pyvene: A library for understanding and improving PyTorch models via interventions. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demon- strations), pages 158–...
2024
-
[17]
Preprint, arXiv:2409.01659
Interpreting and improving large language models in arithmetic calculation. Preprint, arXiv:2409.01659. Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, and Denny Zhou
-
[18]
Preprint, arXiv:2402.09371
Trans- formers can achieve length generalization but not robustly. Preprint, arXiv:2402.09371. Fangwei Zhu, Damai Dai, and Zhifang Sui
-
[19]
3”, “4”, “7
to perform in- terchange interventions, where we intervene at a specific layer on a specific module at the last token position of a base prompt, with the correspond- ing activation from a source prompt (Interchange Intervention). B Results on Single Digit Tokenization Models W...
2025
-
[2012]
Preprint, arXiv:1202.3725
Gen- eralized fisher score for feature selection. Preprint, arXiv:1202.3725. Subhash Kantamneni and Max Tegmark
-
[2021]
Transformer Circuits Thread
A mathematical framework for transformer circuits. Transformer Circuits Thread. Https://transformer- circuits.pub/2021/framework/index.html. Rhys Gould, Euan Ong, George Ogden, and Arthur Conmy
2021
-
[2023]
arXiv preprint arXiv:2312.09230
Successor heads: Recurring, inter- pretable attention heads in the wild. arXiv preprint arXiv:2312.09230. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al
-
[2024]
arXiv preprint arXiv:2410.15580
Language models are symbolic learners in arithmetic. arXiv preprint arXiv:2410.15580. P. E. H. R. O. Duda and D. G. Stork
-
[2025]
ArXiv, abs/2502.19981
The lookahead limitation: Why multi-operand addition is hard for llms. ArXiv, abs/2502.19981. Chunyuan Deng, Zhiqi Li, Roy Xie, Ruidi Chang, and Hanjie Chen
-
[3584]
157 + 431 =
Figure 8: Gemma 2 9B, Dadd: Circuit statistics across digit positions and thresholds. (a) Unit and Tens (b) Unit and Hundreds (c) Tens and Hundreds (d) Circuit Overlap: Overlap in neurons (%) between digit-position circuits. (e) Unit (f) Tens (g) Hundreds (h) Circuit Size: Num...
2023
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.