Pith. sign in

REVIEW 3 major objections 2 minor 8 references

The Discrete-Log Clock: How a Transformer Learns Modular Multiplication

T0 review · 3 major / 2 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Transformers reduce modular multiplication to addition in discrete-log space when analyzed with the multiplicative character transform.

desk verdict The paper shows that the multiplicative character basis turns a dense spectrum into a sparse one with 4 key frequencies and 96.9% single-tuned neurons for mod mult, but the evidence stays descriptive rather than causal. read the letter →

arxiv 2606.17399 v1 pith:ZZJFDQRQ submitted 2026-06-16 cs.LG cs.AI

Huu Danh Nguyen (Stanford University) This is my paper · ORCID
classification cs.LGcs.AI
keywords transformerinterpretabilitymodularmultiplicationgrokkingdiscretelogarithmmultiplicativecharactersFourieranalysisneuralnetworkcircuits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that the dense spectrum seen in prior additive Fourier analyses of grokked transformers on modular multiplication is an artifact of using an mismatched basis. Switching to the multiplicative character transform on a model trained for a times b mod 113 produces a sparse embedding spectrum with only four dominant frequencies and a Gini coefficient of 0.58. Most MLP neurons become tuned to single frequencies, and their activation maps display clear two-dimensional periodicity once inputs are reordered by discrete logarithm. A reader would care because this indicates the network has discovered an internal algorithm that converts the multiplicative task into an additive one inside the discrete-log domain. The result highlights how aligning the analysis tool with the algebraic structure of the operation can expose computation that otherwise appears noisy.

What carries the argument

The multiplicative character transform, which decomposes functions on the multiplicative group into irreducible representations and thereby isolates the frequencies that implement addition after the discrete logarithm is taken.

What would settle it

Measure neuron activations on a grid of input pairs during a forward pass and test whether the pattern of which neurons fire matches the addition of the discrete logs of the two inputs rather than their direct product.

Watch

Extended reading notes

Core claim

Applying the multiplicative character transform to the embeddings of a transformer trained on a · b mod 113 yields a sparse spectrum in which only four frequencies carry significant energy. In this basis, 96.9 percent of MLP neurons are each tuned to one multiplicative frequency, and neuron activation heatmaps exhibit two-dimensional periodic structure when the inputs are reordered according to the discrete logarithm. These observations demonstrate that the transformer reduces multiplication to addition inside discrete-log space.

Load-bearing premise

The sparsity, single-frequency tuning, and periodic heatmaps observed after the basis change correspond to the actual steps executed during the model's forward pass rather than an alternative mathematical description of the same weights.

Editorial extensions

If this is right

  • The network solves the task by first mapping inputs to discrete-log coordinates, performing addition there, and mapping the result back.
  • Only a small number of frequencies in the multiplicative basis are required, in contrast to the full set needed in the additive basis.
  • Reordering activations by discrete logarithm exposes the periodic structure that implements the addition step.
  • The same basis-matching approach can be used to analyze other operations whose natural group structure differs from the additive integers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same analysis could be applied to modular exponentiation or other group operations to check whether transformers similarly exploit homomorphisms.
  • If the discrete-log reduction is causal, editing the weights associated with the four key frequencies should selectively impair multiplication while leaving other behaviors intact.
  • The finding suggests that interpretability tools for arithmetic circuits may need to be chosen according to the operation's group rather than applied uniformly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper claims that when transformers grok modular multiplication a·b mod p, the learned embeddings exhibit a dense spectrum under the additive DFT but become sparse (Gini 0.58, only 4 key frequencies) under the multiplicative character transform on (Z/pZ)*. It further reports that 96.9% of MLP neurons are tuned to a single multiplicative frequency and that activation heatmaps become 2D-periodic when inputs are reordered by discrete logarithm, demonstrating that the model implements a 'Discrete-Log Clock' algorithm that reduces multiplication to addition in log space, analogous to the Clock algorithm for addition.

Significance. If the causal interpretation holds, the work supplies a task-matched analysis basis that converts apparently dense representations into sparse, interpretable structure, extending mechanistic interpretability techniques from additive to multiplicative group operations. The explicit matching of transform to algebraic structure is a methodological contribution that could apply to other structured tasks.

major comments (3)
  1. [Abstract and §4] Abstract and §4 (results on spectrum and neuron tuning): the central claim that the observed Gini coefficient, 4-frequency sparsity, and 96.9% single-frequency tuning 'demonstrate' the Discrete-Log Clock algorithm is not supported by any causal intervention (frequency ablation, neuron knockout, or path patching). The statistics are consistent with a descriptive re-expression of the weights but do not rule out alternative circuits such as direct lookup tables that happen to be sparse in the character basis.
  2. [§3 and §4] §3 (methods) and §4 (neuron heatmaps): the paper reports 2D-periodic structure after discrete-log reordering but provides no verification that these neurons participate in the forward-pass computation of a·b mod p (e.g., via activation patching or counterfactuals). Without such tests the periodic heatmaps remain correlational and do not establish that the model performs addition in log space.
  3. [§4] §4 (Gini and frequency counts): the claim that only 4 frequencies carry significant energy is load-bearing for the sparsity argument, yet the manuscript gives no equation or threshold defining 'significant energy' nor reports the corresponding quantities in the additive basis for the same model, preventing direct comparison of the two bases on identical data.
minor comments (2)
  1. [§2] Notation for the multiplicative character transform is introduced without an explicit equation relating it to the standard DFT; adding the definition (e.g., as Eq. (X)) would improve readability.
  2. [Figures in §4] Figure captions for the reordered heatmaps should state the exact discrete-log mapping used and the range of p=113 to allow reproduction.

Simulated Author's Rebuttal

3 responses · 0 unresolved

Thank you for the constructive and detailed feedback. We address each major comment below and indicate the revisions we will incorporate.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (results on spectrum and neuron tuning): the central claim that the observed Gini coefficient, 4-frequency sparsity, and 96.9% single-frequency tuning 'demonstrate' the Discrete-Log Clock algorithm is not supported by any causal intervention (frequency ablation, neuron knockout, or path patching). The statistics are consistent with a descriptive re-expression of the weights but do not rule out alternative circuits such as direct lookup tables that happen to be sparse in the character basis.

    Authors: We acknowledge that the analysis is observational and that causal interventions such as frequency ablation or path patching are absent. The strength of the evidence lies in the extreme sparsity (Gini 0.58 versus 0.07) appearing only in the task-matched multiplicative character basis, together with 96.9% of neurons exhibiting clean single-frequency tuning; such structure is not a generic property of sparse lookup tables. We will revise the abstract and §4 to replace 'demonstrate' with 'provide strong evidence for' and add a limitations paragraph discussing the correlational nature of the results and the desirability of future interventional tests. revision: partial

  2. Referee: [§3 and §4] §3 (methods) and §4 (neuron heatmaps): the paper reports 2D-periodic structure after discrete-log reordering but provides no verification that these neurons participate in the forward-pass computation of a·b mod p (e.g., via activation patching or counterfactuals). Without such tests the periodic heatmaps remain correlational and do not establish that the model performs addition in log space.

    Authors: The emergence of clean 2D periodicity exclusively under discrete-log reordering constitutes a specific, non-trivial prediction of the clock hypothesis. While we agree that activation patching would strengthen the causal claim, the consistency of the pattern across the MLP layer and its absence in the additive basis already constrain alternative explanations. We will add a short limitations subsection in §4 noting the correlational status of the heatmaps and identifying interventional verification as valuable future work. revision: partial

  3. Referee: [§4] §4 (Gini and frequency counts): the claim that only 4 frequencies carry significant energy is load-bearing for the sparsity argument, yet the manuscript gives no equation or threshold defining 'significant energy' nor reports the corresponding quantities in the additive basis for the same model, preventing direct comparison of the two bases on identical data.

    Authors: We will add an explicit definition and equation in §4: a frequency is deemed significant when its absolute coefficient exceeds 5% of the largest coefficient in that spectrum. We will also report the corresponding count of significant frequencies under the additive DFT on the identical embeddings, which is substantially larger and consistent with the reported Gini values. These additions will appear as a new paragraph with the relevant formulas. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: external basis applied to post-training weights yields empirical observations

full rationale

The paper trains a transformer on a·b mod p, then applies the multiplicative character transform (an external algebraic tool) to the resulting embeddings. Reported quantities (Gini 0.58, 4 key frequencies, 96.9% single-frequency neurons, 2D-periodic heatmaps) are direct measurements in that basis, not quantities defined by the paper's own equations or fitted parameters. No step reduces by construction to inputs, no self-citation chain is load-bearing, and the 'Discrete-Log Clock' claim is an interpretive inference rather than a definitional equivalence. The derivation is self-contained against external benchmarks.

Assumptions & free parameters 0 free parameters · 2 assumptions · 1 invented entities

The central claim rests on standard facts about the cyclic multiplicative group (Z/pZ)* for prime p and the decomposition property of multiplicative characters; the discrete-log clock itself is introduced as an interpretive construct without independent falsifiable evidence outside the observed sparsity.

assumptions (2)
  • standard math The multiplicative group (Z/pZ)* is cyclic for prime p, so the discrete logarithm is well-defined.
    Invoked implicitly to justify reordering inputs and activations by discrete log.
  • standard math The multiplicative character transform decomposes functions on (Z/pZ)* into its irreducible representations.
    Basis for claiming that sparsity in this transform reveals the model's internal algorithm.
invented entities (1)
  • Discrete-Log Clock
    purpose: The internal algorithm by which the transformer computes modular multiplication
    Proposed as the mechanism that reduces multiplication to addition; no independent evidence outside the reported sparsity is supplied.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Discrete-Log Clock: How a Transformer Learns Modular Multiplication." pith.science (2026). https://pith.science/paper/ZZJFDQRQ

@misc{pith2026260617399,
  author       = {Pith},
  title        = {Pith review of: The Discrete-Log Clock: How a Transformer Learns Modular Multiplication},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZZJFDQRQ}},
  note         = {Machine review of arXiv:2606.17399}
}
abstract

When small transformers grok modular multiplication, prior work reports that the learned embedding has a "dense" Fourier spectrum requiring all frequencies. This contrasts with modular addition, where only a sparse set of key frequencies suffices. We show this density is an artifact of analyzing in the wrong basis. The natural Fourier transform for multiplication is not the standard additive DFT but the multiplicative character transform, which decomposes functions on the multiplicative group $(\mathbb{Z}/p\mathbb{Z})^*$ into its irreducible representations. Applying this transform to a grokked transformer trained on $a \cdot b \bmod 113$, we find the embedding spectrum becomes highly sparse (Gini coefficient 0.58 vs. 0.07 in the additive basis) with only 4 key frequencies carrying significant energy. Furthermore, 96.9% of MLP neurons are cleanly tuned to a single multiplicative frequency, and neuron activation heatmaps reveal 2D-periodic structure when reordered by the discrete logarithm. These results demonstrate the transformer reduces multiplication to addition in discrete-log space, implementing a "Discrete-Log Clock" algorithm analogous to Nanda et al.'s Clock algorithm for addition. The methodology generalizes: matching the analysis basis to the algebraic structure of the task reveals interpretable structure where standard tools see noise.

Figures

Figures reproduced from arXiv: 2606.17399 by the authors.

Figure 1
Figure 1. Training curve for a · b mod 113. The model memorizes quickly (train loss drops by epoch 500), then generalizes suddenly during grokking (epochs 9K–14K). Weight decay drives the transi￾tion from memorization to the structured algorithm. means log3 (a) = α. We write α = logg (a) and β = logg (b) for the two inputs. This is an exact bijection (a permutation of the 112 elements) that defines a group iso￾morphism: logg … view at source ↗
Figure 2
Figure 2. Embedding Fourier spectrum. Left: additive basis shows flat, dense spectrum. Right: multiplicative basis reveals 4 sparse peaks. Same y-axis scale. The “density” reported by prior work is a basis artifact [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 5
Figure 5. Principal components of WE reordered by discrete logarithm show sinusoidal structure, confirming the embedding encodes multiplicative characters. same Fourier/trigonometric mechanism known from addi￾tion. The key methodological insight is that the analysis basis must match the algebraic structure of the task. In follow-up experiments, we confirmed the Discrete-Log Clock across 10 primes (p = 59 to 113), all exhibiti… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Neuron heatmaps in integer order (left) vs. discrete-log order (right). In log-order, diagonal stripes emerge, indicating periodicity in logg (a) + logg (b). The “all-frequencies” finding is correct in the additive basis but misleading because it conflates basis mismat…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 1 canonical work pages

  1. [1]

    ICLR 2022 Workshop on MATH-AI , year=

    Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets , author=. ICLR 2022 Workshop on MATH-AI , year=

    2022

  2. [2]

    International Conference on Learning Representations , year=

    Progress Measures for Grokking via Mechanistic Interpretability , author=. International Conference on Learning Representations , year=

  3. [3]

    Advances in Neural Information Processing Systems , year=

    The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networks , author=. Advances in Neural Information Processing Systems , year=

  4. [4]

    arXiv preprint arXiv:2406.03495 , year=

    Grokking Modular Polynomials , author=. arXiv preprint arXiv:2406.03495 , year=

    work page arXiv

  5. [5]

    Transactions on Machine Learning Research , year=

    Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials , author=. Transactions on Machine Learning Research , year=

  6. [6]

    International Conference on Machine Learning , year=

    A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations , author=. International Conference on Machine Learning , year=

  7. [7]

    Advances in Neural Information Processing Systems , year=

    Uncovering a Universal Abstract Algorithm for Modular Addition in Neural Networks , author=. Advances in Neural Information Processing Systems , year=

  8. [8]

    International Conference on Machine Learning , year=

    Grokking Group Multiplication with Cosets , author=. International Conference on Machine Learning , year=

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.