{"id":"3cb6a929-de55-4c03-850f-aa057b75c987","arxiv_id":"2606.17399","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Transformers grok a*b mod p by implementing a discrete-log clock that converts multiplication to addition, shown by sparse spectrum and single-frequency neuron tuning in the multiplicative character basis.","lead":"The paper shows that transformers learning modular multiplication reduce it to addition via the discrete logarithm, visible as sparse structure only when analyzed with the multiplicative character transform instead of the standard additive Fourier basis. A smart generalist might read it to see how matching analysis tools to the algebraic structure of a task can turn apparent model complexity into interpretable mechanisms.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Sparsity in multiplicative basis and neuron tuning may describe weights without proving discrete-log addition is the forward-pass algorithm","rationale":"The reader's weakest_assumption directly identifies the same mechanistic gap. Because the provided abstract contains only correlational observations and the full text is referenced but not reproduced here, the same interpretability-to-mechanism leap remains the load-bearing step; no stronger internal inconsistency or formal error is visible from the given material.","tokens_in":1819,"tokens_out":341,"duration_ms":14149,"concrete_test":"Select the 4 dominant multiplicative frequencies identified in the embedding spectrum; zero their components in the embedding matrix and re-evaluate accuracy on held-out a·b mod 113 examples. Compare against zeroing an equal number of random frequencies of matched magnitude. If accuracy remains >90% after targeted ablation, the frequencies are not mechanistically required for the claimed algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that observed sparsity (Gini 0.58, 4 key frequencies), 96.9% single-frequency MLP neurons, and 2D-periodic heatmaps after discrete-log reordering demonstrate the model reduces a·b mod p to addition in log space. This interpretation assumes the basis-aligned statistics are not merely a re-expression of the trained weights but reflect the actual computation performed by the embedding + MLP + unembedding during inference. The abstract provides no causal intervention (e.g., frequency ablation, neuron knockout, or path patching) that would falsify alternative mechanisms such as direct lookup tables or other non-logarithmic circuits that happen to be sparse in the character basis.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that when transformers grok modular multiplication a·b mod p, the learned embeddings exhibit a dense spectrum under the additive DFT but become sparse (Gini 0.58, only 4 key frequencies) under the multiplicative character transform on (Z/pZ)*. It further reports that 96.9% of MLP neurons are tuned to a single multiplicative frequency and that activation heatmaps become 2D-periodic when inputs are reordered by discrete logarithm, demonstrating that the model implements a 'Discrete-Log Clock' algorithm that reduces multiplication to addition in log space, analogous to the Clock algorithm for addition.","tokens_in":2004,"tokens_out":667,"duration_ms":22792,"significance":"If the causal interpretation holds, the work supplies a task-matched analysis basis that converts apparently dense representations into sparse, interpretable structure, extending mechanistic interpretability techniques from additive to multiplicative group operations. The explicit matching of transform to algebraic structure is a methodological contribution that could apply to other structured tasks.","major_comments":[{"comment":"Abstract and §4 (results on spectrum and neuron tuning): the central claim that the observed Gini coefficient, 4-frequency sparsity, and 96.9% single-frequency tuning 'demonstrate' the Discrete-Log Clock algorithm is not supported by any causal intervention (frequency ablation, neuron knockout, or path patching). The statistics are consistent with a descriptive re-expression of the weights but do not rule out alternative circuits such as direct lookup tables that happen to be sparse in the character basis.","section":"Abstract and §4"},{"comment":"§3 (methods) and §4 (neuron heatmaps): the paper reports 2D-periodic structure after discrete-log reordering but provides no verification that these neurons participate in the forward-pass computation of a·b mod p (e.g., via activation patching or counterfactuals). Without such tests the periodic heatmaps remain correlational and do not establish that the model performs addition in log space.","section":"§3 and §4"},{"comment":"§4 (Gini and frequency counts): the claim that only 4 frequencies carry significant energy is load-bearing for the sparsity argument, yet the manuscript gives no equation or threshold defining 'significant energy' nor reports the corresponding quantities in the additive basis for the same model, preventing direct comparison of the two bases on identical data.","section":"§4"}],"minor_comments":[{"comment":"Notation for the multiplicative character transform is introduced without an explicit equation relating it to the standard DFT; adding the definition (e.g., as Eq. (X)) would improve readability.","section":"§2"},{"comment":"Figure captions for the reordered heatmaps should state the exact discrete-log mapping used and the range of p=113 to allow reproduction.","section":"Figures in §4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's fit to the journal is reasonable given the mechanistic-interpretability focus, but the absence of any training hyperparameters or code release makes independent verification of the reported percentages difficult; this could be addressed in revision."},"author_rebuttal":{"model":"grok-4.3","summary":"Thank you for the constructive and detailed feedback. We address each major comment below and indicate the revisions we will incorporate.","responses":[{"response":"We acknowledge that the analysis is observational and that causal interventions such as frequency ablation or path patching are absent. The strength of the evidence lies in the extreme sparsity (Gini 0.58 versus 0.07) appearing only in the task-matched multiplicative character basis, together with 96.9% of neurons exhibiting clean single-frequency tuning; such structure is not a generic property of sparse lookup tables. We will revise the abstract and §4 to replace 'demonstrate' with 'provide strong evidence for' and add a limitations paragraph discussing the correlational nature of the results and the desirability of future interventional tests.","revision_made":"partial","referee_comment":"[Abstract and §4] Abstract and §4 (results on spectrum and neuron tuning): the central claim that the observed Gini coefficient, 4-frequency sparsity, and 96.9% single-frequency tuning 'demonstrate' the Discrete-Log Clock algorithm is not supported by any causal intervention (frequency ablation, neuron knockout, or path patching). The statistics are consistent with a descriptive re-expression of the weights but do not rule out alternative circuits such as direct lookup tables that happen to be sparse in the character basis."},{"response":"The emergence of clean 2D periodicity exclusively under discrete-log reordering constitutes a specific, non-trivial prediction of the clock hypothesis. While we agree that activation patching would strengthen the causal claim, the consistency of the pattern across the MLP layer and its absence in the additive basis already constrain alternative explanations. We will add a short limitations subsection in §4 noting the correlational status of the heatmaps and identifying interventional verification as valuable future work.","revision_made":"partial","referee_comment":"[§3 and §4] §3 (methods) and §4 (neuron heatmaps): the paper reports 2D-periodic structure after discrete-log reordering but provides no verification that these neurons participate in the forward-pass computation of a·b mod p (e.g., via activation patching or counterfactuals). Without such tests the periodic heatmaps remain correlational and do not establish that the model performs addition in log space."},{"response":"We will add an explicit definition and equation in §4: a frequency is deemed significant when its absolute coefficient exceeds 5% of the largest coefficient in that spectrum. We will also report the corresponding count of significant frequencies under the additive DFT on the identical embeddings, which is substantially larger and consistent with the reported Gini values. These additions will appear as a new paragraph with the relevant formulas.","revision_made":"yes","referee_comment":"[§4] §4 (Gini and frequency counts): the claim that only 4 frequencies carry significant energy is load-bearing for the sparsity argument, yet the manuscript gives no equation or threshold defining 'significant energy' nor reports the corresponding quantities in the additive basis for the same model, preventing direct comparison of the two bases on identical data."}],"tokens_in":1515,"tokens_out":662,"duration_ms":33487,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that the usual additive DFT makes modular multiplication embeddings look dense, but the multiplicative character transform on (Z/pZ)* makes them sparse for a grokked model on a*b mod 113, with Gini 0.58, only four frequencies carrying energy, and 96.9% of MLP neurons tuned to one frequency. Reordering activations by discrete log also produces clean 2D periodic patterns. This extends the clock idea from addition to multiplication by matching the basis to the group operation.\n\nThe work does a clean job of showing why the prior dense-spectrum claim was an artifact of the wrong basis. The quantitative contrast (0.58 vs 0.07 Gini) and the neuron-tuning statistic are straightforward to check from the reported numbers, and the generalization note about choosing the analysis basis to fit the algebraic structure is useful. It gives a concrete template for other group operations.\n\nThe soft spot is that everything stays at the level of weight description. The abstract gives no ablation, frequency knockout, or path-patching result that would show these frequencies are actually used in the forward pass rather than just appearing when you project the weights onto this basis. Without that, the discrete-log clock remains a re-expression that fits the data but has not been shown to be the circuit the model runs. The results are also from a single p=113 case with no reported checks on other primes or training seeds.\n\nThis is for people already working on mechanistic interpretability of grokking on arithmetic tasks. A reader who wants to see how basis choice changes what you see in embeddings will get something concrete; someone looking for verified algorithms will want the causal tests added.\n\nI would send it to peer review. The observation is sharp enough to be worth referee time even if the causal step needs strengthening.","headline":"The paper shows that the multiplicative character basis turns a dense spectrum into a sparse one with 4 key frequencies and 96.9% single-tuned neurons for mod mult, but the evidence stays descriptive rather than causal.","tokens_in":2478,"tokens_out":456,"would_cite":false,"duration_ms":22291,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Transformers reduce modular multiplication to addition in discrete-log space when analyzed with the multiplicative character transform.","keywords":["transformer interpretability","modular multiplication","grokking","discrete logarithm","multiplicative characters","Fourier analysis","neural network circuits"],"falsifier":"Measure neuron activations on a grid of input pairs during a forward pass and test whether the pattern of which neurons fire matches the addition of the discrete logs of the two inputs rather than their direct product.","tokens_in":2719,"feed_emoji":"🕒","tokens_out":715,"duration_ms":27339,"temperature":0.7,"pith_summary":"The paper establishes that the dense spectrum seen in prior additive Fourier analyses of grokked transformers on modular multiplication is an artifact of using an mismatched basis. Switching to the multiplicative character transform on a model trained for a times b mod 113 produces a sparse embedding spectrum with only four dominant frequencies and a Gini coefficient of 0.58. Most MLP neurons become tuned to single frequencies, and their activation maps display clear two-dimensional periodicity once inputs are reordered by discrete logarithm. A reader would care because this indicates the network has discovered an internal algorithm that converts the multiplicative task into an additive one inside the discrete-log domain. The result highlights how aligning the analysis tool with the algebraic structure of the operation can expose computation that otherwise appears noisy.","feed_headline":"Transformer reduces modular multiplication to discrete-log addition","feed_subtitle":"Multiplicative character basis exposes sparse spectrum, single-frequency neurons, and periodic activations aligned with the discrete log.","key_machinery":"The multiplicative character transform, which decomposes functions on the multiplicative group into irreducible representations and thereby isolates the frequencies that implement addition after the discrete logarithm is taken.","core_discovery":"Applying the multiplicative character transform to the embeddings of a transformer trained on a · b mod 113 yields a sparse spectrum in which only four frequencies carry significant energy. In this basis, 96.9 percent of MLP neurons are each tuned to one multiplicative frequency, and neuron activation heatmaps exhibit two-dimensional periodic structure when the inputs are reordered according to the discrete logarithm. These observations demonstrate that the transformer reduces multiplication to addition inside discrete-log space.","pith_inferences":["The same analysis could be applied to modular exponentiation or other group operations to check whether transformers similarly exploit homomorphisms.","If the discrete-log reduction is causal, editing the weights associated with the four key frequencies should selectively impair multiplication while leaving other behaviors intact.","The finding suggests that interpretability tools for arithmetic circuits may need to be chosen according to the operation's group rather than applied uniformly."],"forward_implications":["The network solves the task by first mapping inputs to discrete-log coordinates, performing addition there, and mapping the result back.","Only a small number of frequencies in the multiplicative basis are required, in contrast to the full set needed in the additive basis.","Reordering activations by discrete logarithm exposes the periodic structure that implements the addition step.","The same basis-matching approach can be used to analyze other operations whose natural group structure differs from the additive integers."],"fun_headline_variants":["Transformer implements discrete-log clock for multiplication","Multiplicative transform reveals transformer discrete-log clock","Sparse spectrum reveals discrete-log structure in transformer","Transformer adds in discrete-log space to multiply"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The sparsity, single-frequency tuning, and periodic heatmaps observed after the basis change correspond to the actual steps executed during the model's forward pass rather than an alternative mathematical description of the same weights.","fun_headline_variants_meta":{"raw":{"variants":["Transformer implements discrete-log clock for multiplication","Multiplicative transform reveals transformer discrete-log clock","Sparse spectrum reveals discrete-log structure in transformer","Transformer adds in discrete-log space to multiply"]},"model":"grok-4.3","cost_usd":0.007449,"raw_usage":{"total_tokens":3430,"prompt_tokens":686,"num_sources_used":0,"completion_tokens":52,"cost_in_usd_ticks":74487000,"prompt_tokens_details":{"text_tokens":686,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2692,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":686,"tokens_out":52,"duration_ms":22641,"temperature":1.0,"reasoning_tokens":2692,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T02:07:28.125580+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Measure neuron activations on a grid of input pairs during a forward pass and test whether the pattern of which neurons fire matches the addition of the discrete logs of the two inputs rather than their direct product.","supporting_citations":[],"review_version":1}