MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers

Atri Rudra; Christopher R\'e; Jerry Liu; Roberto Garcia; Ronny Junkins; Sabri Eyuboglu

MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers

T0 review · 4 major / 0 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Closed-form MLPs store facts at optimal capacity inside Transformers, using far fewer parameters than prior constructions.

desk verdict Abstract-only claim of a closed-form, Transformer-compatible fact-storing MLP that hits IT-optimal capacity; interesting if true, but currently uncheckable. read the letter →

arxiv 2607.10034 v1 pith:2I6S3H6I submitted 2026-07-10 cs.LG

Roberto Garcia , Jerry Liu , Ronny Junkins , Sabri Eyuboglu , Atri Rudra , Christopher Ré This is my paper · ORCID

classification cs.LG

keywords factstorageMLPTransformerinformation-theoreticcapacitydecodingmarginclosed-formconstructionmodulareditingkey-valuememory

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language models store facts in their MLP layers, yet existing constructive accounts of that storage do not explain how real models reach information-theoretically optimal density. This paper supplies the missing construction: a closed-form MLP that is fully compatible with the residual stream and attention geometry of a Transformer, stores an arbitrary set of key–value facts, and matches the optimal capacity scaling under isotropic embeddings. The technical step is to analyze the decoding margin of the MLP rather than only its storage capacity; once that margin is controlled, the same construction continues to work for non-isotropic embeddings (up to geometry-dependent penalties) and can be dropped directly into Transformer blocks for factual recall. Empirically the resulting modules need one to two orders of magnitude fewer parameters than earlier fact-storing MLPs at the same number of facts, and they support modular editing by simple MLP swap.

What carries the argument

Decoding-margin analysis of the MLP. By optimizing the geometric margin that separates correct from incorrect value embeddings after the MLP, rather than merely counting storage capacity, the authors obtain an explicit weight construction that simultaneously hits the information-theoretic optimum and stays compatible with residual-stream geometry.

What would settle it

Construct the claimed MLP for a known set of facts with isotropic embeddings of dimension d and verify whether its parameter count scales as Θ(n d) (optimal) while the decoding margin remains positive and factual recall accuracy stays high when the MLP is substituted into a Transformer block; a clear super-linear blow-up or collapse of the margin falsifies the claim.

Watch

Extended reading notes

Core claim

A closed-form, Transformer-compatible fact-storing MLP attains information-theoretically optimal fact-storage capacity scaling under isotropic embeddings, remains capacity-optimal (up to geometry penalties) for arbitrary embeddings, and can be inserted into Transformer blocks for factual recall while requiring 10–104× fewer parameters than prior constructions at matched fact count.

Load-bearing premise

That controlling the decoding margin of the MLP is enough to guarantee both information-theoretic optimality and seamless insertion into real Transformer residual streams for embeddings that are not perfectly isotropic.

Editorial extensions

If this is right

Fact-storing MLPs can be written down in closed form and inserted into Transformers without destroying residual-stream geometry.
At matched fact count the construction needs 10–104× fewer parameters than earlier constructive baselines under isotropic embeddings, and 15–63× fewer when used inside Transformer blocks.
Arbitrary key/value geometries are handled with only a multiplicative capacity penalty that depends on the embedding condition numbers.
Modular fact editing becomes possible by simply swapping one fact-storing MLP for another inside a frozen Transformer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the margin analysis generalizes, practitioners could pre-compute or surgically insert compact fact banks rather than relying solely on gradient-based memorization.
The same construction may supply a diagnostic: measure the effective decoding margin of a trained MLP to estimate how close it sits to the information-theoretic storage limit.
Geometry-dependent penalties suggest that embedding-regularization schemes that keep keys and values well-conditioned could further reduce the parameter overhead of fact storage.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing you need to know: this abstract claims the first closed-form MLP construction that stores facts at information-theoretically optimal scaling, handles arbitrary key/value geometries, and still works inside Transformer residual streams. If the full paper delivers, that is a real within-field advance for mechanistic interpretability and efficient fact packing.

What looks new is the shift from pure storage capacity to decoding margin. Prior constructive work apparently left a gap between theory and the empirical packing rates people see in LLMs; the authors say analyzing the margin closes that gap and yields 10–104× parameter reductions at matched fact count (15–63× when the MLP sits inside a Transformer block). They also sketch a modular editing proof-of-concept by swapping the MLP. That package is cleanly framed against external IT bounds rather than self-referential fitting, which is good structure.

The soft spots are exactly what you expect from abstract-only material. The optimality claim under isotropic embeddings and the “same scaling up to geometry penalization factors” for arbitrary embeddings are numerical and geometric assertions we cannot verify. The stress-test note is right that “decoding margin” is never defined here, so we cannot check the conditions that supposedly guarantee both the capacity lower bound and residual-stream compatibility. Free parameters (geometry factors, width choices) remain opaque. None of that proves the claims false; it just means soundness is currently unscoreable.

Who it is for: people who care about constructive models of factual recall, capacity allocation, and editing primitives. A serious referee should see the full theorems, proofs, and experiments. I would send it to peer review rather than desk-reject; the claimed combination of closed form, optimality, and Transformer compatibility is important enough to deserve that time. Bring it to reading group only after the PDF appears and someone has checked the margin analysis. I would not cite it yet.

Referee Report

4 major / 0 minor

Summary. The manuscript claims a closed-form construction of Transformer-compatible fact-storing MLPs that simultaneously (i) attains information-theoretically optimal fact-storage capacity scaling under isotropic embeddings, (ii) extends to arbitrary key/value geometries up to geometry-dependent penalization factors, and (iii) functions inside Transformer residual streams for factual recall. The stated technical lever is analysis of the MLP decoding margin (rather than storage capacity alone). Under isotropic embeddings the construction is reported to need 10–104× fewer parameters than prior constructions at matched fact count; inside Transformer blocks the reported reduction is 15–63×. A proof-of-concept modular fact-editing application (swapping an MLP) is also claimed.

Significance. If the optimality, geometric generality, and residual-stream compatibility claims hold with the stated parameter reductions, the work would supply the first constructive account of near-optimal fact storage in Transformer MLPs, substantially improve on prior constructive baselines, and open a modular route to fact editing. Those outcomes would be of clear interest to both the theory and mechanistic-interpretability communities. The abstract-only submission, however, does not yet allow those claims to be verified.

major comments (4)

The abstract presents “analyzing the decoding margin” as the single step that simultaneously yields IT-optimal capacity and Transformer compatibility, yet supplies neither a definition of the margin, the precise conditions under which a positive margin guarantees correct retrieval, nor a sketch linking that margin to a capacity lower bound. Without those, the central optimality claim cannot be checked.
The reported 10–104× (isotropic) and 15–63× (in-Transformer) parameter reductions are numerical assertions whose validity depends on the unspecified margin conditions, the choice of baselines, embedding dimension / width, and experimental protocol. None of these appear in the abstract, so the efficiency claims are unanchored.
Extension from isotropic embeddings to arbitrary geometries is stated only as “up to penalization factors depending on the embedding geometries.” The abstract does not define those factors, bound them, or state when they remain O(1). If the factors grow with the number of facts or with embedding anisotropy, the claimed optimal scaling does not transfer.
Transformer compatibility is asserted (“works inside Transformers,” “used within Transformer blocks for factual recall tasks at optimal capacity scaling”) without any statement of residual-stream or attention-geometry interference conditions. Residual addition and attention can introduce extra cross-fact interference that would invalidate a pure-MLP margin analysis; the abstract gives no argument that this does not occur.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity detectable from abstract; closed-form construction claims are framed against external IT capacity bounds and prior constructions.

full rationale

Only the abstract is available. It presents a closed-form Transformer-compatible fact-storing MLP construction that is claimed to attain information-theoretically optimal storage capacity scaling under isotropic embeddings, to handle arbitrary geometries up to geometry-dependent penalties, and to function inside Transformer blocks, with parameter reductions of 10–104× and 15–63× relative to prior constructions. These claims are positioned as comparisons to external information-theoretic capacity and to earlier constructive models, not as quantities fitted from the same data being predicted. No equations, uniqueness theorems, self-citations, or ansatz adoptions appear in the provided text, so none of the six circularity patterns (self-definitional loops, fitted-input-as-prediction, load-bearing self-citation, uniqueness imported from authors, ansatz smuggled via citation, or renaming of a known result) can be exhibited by quote and reduction. Per the analyzer rules, absence of quotable circular reductions yields score 0 with empty steps; residual concerns about unspecified decoding-margin conditions are correctness/definition issues, not circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

Abstract-only review: free parameters, background axioms, and invented entities cannot be exhaustively extracted from equations. The ledger records what the abstract itself makes load-bearing: information-theoretic capacity as the external benchmark, isotropic (and then arbitrary) embedding geometry assumptions, and the decoding-margin analysis as the enabling technical device. No new physical entities are introduced; the 'Hebbian' framing is interpretive.

free parameters (2)

geometry penalization factors (arbitrary embeddings)
Abstract states that for non-isotropic key/value embeddings the same scaling holds 'up to penalization factors depending on the embedding geometries'; those factors are not given numerical values or closed forms here and may act as effective free or data-dependent multipliers.
embedding dimension / width choices for capacity comparisons
Parameter-count ratios (10–104×, 15–63×) depend on how width, depth, and fact count are matched; abstract does not specify the exact matching protocol, so those design choices function as free experimental parameters.

assumptions (4)

domain assumption Factual knowledge in LLMs resides primarily in MLP layers and can be modeled as key–value associations.
Stated as background from recent work; the construction and editing claims rest on this localization.
domain assumption Information-theoretic optimal fact-storage capacity is the correct external benchmark for MLP constructions.
Used throughout to claim 'optimal scaling'; standard in coding/memory theory but still an assumption about the right figure of merit for LLM fact storage.
ad hoc to paper Isotropic embeddings are a valid regime in which to prove optimality, and results extend to arbitrary geometries via penalization.
Abstract explicitly conditions the strongest optimality claim on isotropic embeddings and then claims extension; the precise mathematical conditions are not visible.
ad hoc to paper Decoding margin (not only storage capacity) is the right quantity to optimize for Transformer-compatible readout.
Presented as the key technical insight distinguishing this work from prior constructions that only studied storage.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers." pith.science (2026). https://pith.science/paper/2I6S3H6I

@misc{pith2026260710034,
  author       = {Pith},
  title        = {Pith review of: MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2I6S3H6I}},
  note         = {Machine review of arXiv:2607.10034}
}

abstract

Large language models (LLMs) store factual knowledge in their parameters. While recent work has shown that this knowledge resides in MLP layers, existing constructive and mechanistic interpretability models of fact-storage in LLMs fail to explain the surprising empirical phenomenon that they store facts at an information-theoretically optimal rate. In this work, we develop a theoretical account of this phenomenon. We develop the first Transformer-compatible fact-storing MLP closed-form construction that satisfies the following three properties empirically observed in LLMs: it (i) attains optimal fact storage scaling, (ii) handles arbitrary input/output geometries, and (iii) works inside Transformers. Key to our work is to analyze the decoding margin of MLPs, whereas prior work only studies MLP fact storage. Under isotropic embeddings, our construction achieves information-theoretically optimal storage capacity scaling and requires $10$-$104\times$ fewer parameters at matched fact count than prior constructions. For arbitrary key and value embeddings, we show that our construction attains the same storage capacity scaling, up to penalization factors depending on the embedding geometries. Moreover, we demonstrate that our constructed MLPs can be used within Transformer blocks for factual recall tasks at optimal capacity scaling, requiring $15$-$63\times$ fewer parameters at matched fact count than prior constructions. Finally, as a proof-of-concept, we show that fact-storing MLPs enable modular fact editing by swapping a Transformer's MLP with a new one.

Discussion (0). Continue with ORCID to comment.

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.