Pith. sign in

REVIEW 2 major objections 5 minor 60 references

Under a fixed per-token inference budget, capability converges toward the access-complete class — a compressive state channel paired with a scalable verbatim index — not toward a single representation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:56 UTC pith:5NAMPT2Q

load-bearing objection Worth engaging: an unusually honest, well-scoped framework paper whose central strict-inclusion claim is real but conditional on an unverified separability assumption that the paper's own measurements show is strained for numeric and code keys. the 2 major comments →

arxiv 2607.14144 v2 pith:5NAMPT2Q submitted 2026-07-14 cs.AI cs.ITmath.IT

The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

Wenhui Chen , Jianlin Chen , Ziyao Lin , Chi Man Vong This is my paper
classification cs.AI cs.ITmath.IT
keywords capability convergenceaccess structurehybrid architecturesstate-space modelsinformation-theoretic lower boundspre-registered experimentslong-context retrievalrepresentational convergence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that while representations converge with scale, capability does not follow automatically. Its Capability Convergence Hypothesis states that, under a fixed per-token inference budget, what a model can do converges toward a class of architectures: any design holding both an O(1)-state compressive channel and a scalable verbatim-index channel. The paper formalizes three resource walls that bar pure families on a witness task — one information-theoretic, one about finite windows, one about fixed-depth composition — and shows a hybrid paying each wall's price crosses all three. It follows that capability is strictly super-additive under composition: the solvable-task family of the hybrid strictly contains the union of the pure families. The first pre-registered small-scale tests measure the predicted scissors gap and state-tracking bifurcation, while one prediction failed and is reported as such.

Core claim

The central claim is the Capability Convergence Hypothesis: under a fixed per-token inference budget, representational convergence does not entail capability convergence. Capability instead converges toward a class — the access-complete hybrid — any architecture holding both a compressive state channel (O(1) memory, the 'Idea') and a scalable verbatim-index channel (Θ(L) memory, the 'Shadow'). On the witness task, no pure state-space family can solve exact long-range retrieval at any scale (the Shannon wall, unconditional), no fixed-window architecture can see past its horizon (the horizon wall, unconditional), and no fixed-depth attention-only stack can compose a serial reference chain (the

What carries the argument

The load-bearing object is the query-time accessible state Σt with capacity B=H(Σt). Attention realizes a near-lossless index channel (B=Θ(L)); a state-space model realizes a lossy compressive channel (B=O(1)). The witness protocol plants N independent key–value bindings, interleaves Θ(L) semantic-overlap distractors, and drives a reference chain through the permutation group S5; an information-theoretic floor bounds per-binding accessible information by B/N. The hybrid combines the two channels, and the contraction of its advantage rests on Assumption 1 (write-time code separability): target bindings carry anchor codewords recoverable from the query, and all distractors stay at distance ≥δκ

Load-bearing premise

The hybrid's advantage on the witness rests on Assumption 1 — that target bindings carry anchor codewords recoverable from the query and that distractors stay code-separated in the index's matching space; if natural workloads do not supply this separability, the hybrid's edge collapses to the information-theoretic ambiguity floor.

What would settle it

Measure, on a natural corpus, the fraction of retrieval-relevant key pairs that are within an ε-ball of each other in the write-time embedding; if that fraction is high, the hybrid's advantage should shrink toward the chance floor, directly testing whether the separability assumption holds for real workloads.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Pure state-space models are barred from exact long-range retrieval at any scale under the fixed-budget regime, regardless of parameter count or representational alignment.
  • Fixed-window architectures cannot escape a cliff: near-perfect retrieval inside the window, chance outside it, and no amount of scale fixes the blind spot.
  • A hybrid that pays the Θ(ρL) index capacity can recover a specific binding after 10^7 tokens of noise, deferring — not eliminating — the horizon by a factor of 1/ρ.
  • Capability closure is strictly super-additive under composition: there exists a witness task family that neither pure family solves at any scale while the hybrid does, a phenomenon the paper calls compositional emergence.
  • If the convergence conjecture holds, production models should increasingly land in a narrow band of global-attention share (roughly 1/12 to 1/4), while pure families remain at a characteristic capability distance.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same two-channel logic should apply one level up to agentic systems: an agent with external retrieval tools is access-complete as a system, so capability should track the presence of a scalable index at some level rather than the base architecture alone.
  • The separability assumption is the empirical crux. A natural measurement of the fraction of retrieval-relevant key pairs that are code-separable in the write-time embedding would directly predict where hybrid advantage is real; the paper's own surface-form numbers suggest entity-like keys are separable while numeric IDs and code identifiers are not.
  • The failed channel-commensurability prediction hints that cross-channel alignment is the wrong signature of successful composition; channel differentiation or complementarity may be the better predictor, testable with probing or intervention studies.
  • The load-ratio formulation (difficulty set by Nb/B, not absolute scale) implies small-scale experiments can be legitimate mechanism probes of frontier behavior; an explicit test would train the same hybrid at several sizes and check whether the scissors gap scales with load ratio.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes the Capability Convergence Hypothesis (CCH): under a fixed per-token inference budget, representational convergence does not imply capability convergence; instead capability converges toward an 'access-complete' class of architectures that combine a compressive O(1)-state channel and a scalable verbatim-index channel. The authors formalize a witness task (Newton's apple in an infinite stream) and identify three walls: a Shannon wall (Theorem 1), a horizon wall (Proposition 1), and a circuit wall (Proposition 2, conditional on TC0≠NC1). These lead to Proposition 3, a strict inclusion of capability closures under regime R, Assumption 1, and TC0≠NC1. The paper also reports pre-registered small-scale experiments, a census of production architectures, and explicitly separates proven claims from conjectures. The central formal result is conditional on Assumption 1, whose natural-language incidence is only partially measured.

Significance. If the access-completeness principle is correct, it offers a principled explanation for the practical success of hybrid SSM–attention designs and clarifies why scale alone cannot buy long-range retrieval and state tracking. The formal lower bound in Theorem 1 is clean, and the two unconditional walls (Shannon and horizon) are valuable independent of the rest of the framework. The paper is exemplary in its pre-registration, timestamped amendment log, public code and data, and honest reporting of a failed prediction (P3). The main caveat is that the strict-inclusion result is conditional on Assumption 1 and TC0≠NC1; the paper's own measurement in G.6 shows the assumption is strained for numeric IDs (50.7%) and code identifiers (56.2%), which are canonical long-context retrieval targets. The field-level convergence conjecture is explicitly labeled as a conjecture and is not tested by the reported data.

major comments (2)
  1. [Section 2.10, Assumption 1; Appendix G.6] The positive clause of Proposition 3 depends on Assumption 1, which requires that write-time anchor codewords be separable from all Θ(L) distractors in the index's matching space. The paper's own surface-form measurement (G.6) finds this holds for entity-like keys (87.5%) but not for numeric IDs (50.7%) or code identifiers (56.2%). Numeric IDs and code identifiers are not exotic: they are among the most common targets in long-context retrieval. The collision control (G.2 ix) confirms the hinge: at zero code distance the trained hybrid lands exactly on the architecture-independent ambiguity floor c/(c+1). Without a demonstration that natural workloads supply separable anchors, the strict inclusion is a result about a constructed witness family, not about typical long-context workloads. The paper's scoping statements are honest, but the title and abstract risk overstating the practical rea
  2. [Section 4.2 and G.2, Figure 4] The quantitative collapse of the information floor in Figure 4(b,c) uses a single fitted constant peff = 0.39 bits/scalar. The paper correctly states that this is 'consistent with the floor, not an independent verification of H(Σ)' (Section 4.2). However, the same section presents P1 as supported with the floor-shape collapse as a key element; as a pre-registered prediction, the collapse is only a one-parameter fit, while the independent falsifiable content is the scissors gap (0.994 vs 0.000), which is measured outright. To make the empirical support clean, either derive peff a priori from an independent capacity estimate or explicitly label the curve collapse as a consistency check rather than a test. This would strengthen the paper's own methodological standards and avoid the appearance of post-hoc calibration.
minor comments (5)
  1. [Section 2.2, Eq. (1)] Postulate 1 uses Shannon entropy H(Σ_t) for a state that is continuous for SSMs. Finite-precision regularization is introduced only later in Definition 1. Consider adding a forward reference or reordering so the entropy is well-defined at the point of use.
  2. [Section 2.10 and Appendix G.2] The collision control is introduced in Section 2.10 as 'point 3' in one place and as 'point 2' elsewhere; likewise the internal references to 'Section 2.10 point 2' vs 'point 3' are inconsistent. Please harmonize these cross-references.
  3. [Figure 4 caption] The caption states '3 seeds' for Experiment B, but the doubled-budget run used 6 seeds and the pure-state m=64 runs had 6 runs. The caption should state both phases and seed counts explicitly.
  4. [Section 6.3, Limitation (iii)] The percentages 87%, 51%, 56% in the limitation text are rounded from G.6's 87.5%, 50.7%, 56.2%. Use the exact values or a consistent rounding convention.
  5. [Section 5, AV3] The discussion of AV3 (economics all the way down) is clear, but the paper could more explicitly say in the abstract that the field-level convergence trend is a conjecture with no discriminating evidence, not a measured phenomenon.

Circularity Check

1 steps flagged

Only the auxiliary P1 floor-collapse is partially circular (fitted peff); the formal separation is independent and no self-citation is load-bearing.

specific steps
  1. fitted input called prediction [Section 4.2 (P1, Experiment B), Appendix G.2 (item ii), and G.7 scorecard]
    "P1 is supported in two senses kept separate: the scissors gap is measured outright, while the floor-shape collapse holds under one fitted effective capacity, consistent with but not an independent verification of Theorem 1 at nominal capacity. ... with the single frozen constant peff=0.39 bits/scalar (fit on m=256), the 50%-crossings land at x = Nb/(m peff) ∈ [0.82, 1.03]"

    The empirical 'tracking the floor' claim is evaluated at x = Nb/(m peff), where peff is fit to the same experimental data (m=256). Since the Fano floor crosses 0.5 near x=1, fitting peff to a curve's observed 50% crossing places that curve at the floor by construction. The m=1024 curve provides a genuine partial check, but the m=256 'collapse' is not an independent test. The paper explicitly disclaims independence, so this is an auxiliary calibration issue, not the load-bearing formal derivation.

full rationale

The central claim, Proposition 3, rests on Theorem 1 (a data-processing/Fano lower bound), the horizon/circuit walls, and Assumption 1; none of these are fitted or self-referential. The pre-registered scissors gap, bifurcation, and collision control are independent mechanism-level tests, and the collision control directly probes Assumption 1 by removing it. The only circularity-adjacent step I can exhibit is the P1 information-floor collapse, where one effective-capacity constant peff is fit on the data and then used in the abscissa x=Nb/(m peff); the paper itself marks this as 'not an independent verification of H(Σ)' and scores P1 as partial. R1-R4 are explicitly labeled consistency-level, and the convergence conjecture is explicitly not demonstrated, so no circularity is hidden there. There is also no load-bearing self-citation: the cited walls, TC0 bound, and hybrid primitives are external results, and the paper repeatedly disclaims inventing them. Overall, the formal derivation is self-contained; only one auxiliary empirical prediction is partly circular, yielding a low score.

Axiom & Free-Parameter Ledger

1 free parameters · 7 axioms · 0 invented entities

The central claim rests on standard information-theoretic facts, one unproved complexity conjecture (TC0≠NC1), one explicit data-geometry assumption (Assumption 1) with partial surface-form measurement, and the paper's finite-precision model of state capacity. The only fitted number in the empirical core is peff.

free parameters (1)
  • peff (effective bits per scalar of SSM state) = 0.39 bits/scalar
    Fitted to the m=256 pure-state curve in Experiment B; used to collapse pure-state error curves onto the floor b(1−1/x). The paper states it is the single permitted fitted constant and that the collapse is consistent with Theorem 1 rather than a measurement of H(Σ).
axioms (7)
  • domain assumption State-channel postulate: Σt is the unique data-bearing channel from history to prediction, so I(X1:t; ŷt | qt, θ) ≤ H(Σt) ≤ B
    Section 2.2 Postulate 1; underlies all walls. If readout could access history outside Σt without counting, the bound fails; paper counts external stores in H(Σ).
  • domain assumption Finite-precision regularization B = dstate·p with idealized p = O(log L)
    Definition 1; makes B finite for continuous SSM states. The fitted peff=0.39 bits/scalar is far below hardware precision, so the nominal floor is not directly triggered.
  • domain assumption Assumption 1: write-time code separability of bindings and distractors
    Section 2.10; load-bearing for the hybrid crossing. G.6 surface-form measurement: entity-like keys 87.5% separable, numeric IDs 50.7%, code identifiers 56.2%.
  • domain assumption TC0 ≠ NC1
    Section 2.9 Prop. 2; needed to bar fixed-depth attention-only stacks from length-generalizing S5 composition. Unproved; Walls I-II do not depend on it.
  • standard math Barrington's theorem: S5 word problem is NC1-complete
    Section 2.9; connects S5 composition to NC1.
  • standard math DPI, Fano, Gibbs, Yao minimax
    Sections 2.6-2.7 and Appendix B; standard lower-bound tools.
  • standard math Hopfield retrieval error bound (M e^{-β_H Δ})
    Appendix A.4; used to control exact retrieval in the index channel.

pith-pipeline@v1.3.0-alltime-deepseek · 39079 in / 25846 out tokens · 267171 ms · 2026-08-03T01:56:27.260674+00:00 · methodology

0 comments
read the original abstract

The Platonic Representation Hypothesis (PRH) holds that as models scale, representations of heterogeneous networks converge toward a shared model of reality. We propose its sequel and boundary, the Capability Convergence Hypothesis (CCH): under a fixed per-token inference budget, representational convergence does not entail capability convergence. Capability instead converges toward a class, the access-complete hybrid: any architecture holding both a compressive O(1)-state channel and a scalable verbatim-index channel. We anchor it on a witness task, the Newton's-apple problem in an infinite stream, and name three resource walls: a Shannon wall barring any o(Nb)-state architecture, a horizon wall barring any fixed window, and a circuit wall barring fixed-depth attention-only composition (conditional on TC0 != NC1). Under an explicit separability assumption a hybrid crosses all three by paying each wall's price, so capability is strictly super-additive under composition. We separate what we prove from what we conjecture: the access-completeness principle rests on information-theoretic lower bounds and pre-registered experiments, while the field-level convergence trend is an economics-motivated conjecture. We report the first pre-registered small-scale tests under criteria frozen before the data: the predicted scissors gap is measured (exact-retrieval error 0.994 vs. 0.000 once a 64-scalar state gains one global-attention layer), the state-tracking bifurcation lands at the registered boundary, and a conjunction witness shows an irreducibly two-channel solution; one prediction failed with its direction reversed and is reported as such. Representational convergence is given freely by scale; capability convergence must be purchased by access structure.

Figures

Figures reproduced from arXiv: 2607.14144 by Chi Man Vong, Jianlin Chen, Wenhui Chen, Ziyao Lin.

Figure 1
Figure 1. Figure 1: Overview of the Capability Convergence Hypothesis [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The CCH concept diagram, mirroring [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Transformer and SSM as two ends of one channel-capacity spectrum (Postulate 1): the SSM at the compressive end (B = O(1), lossy), the Transformer at the near-lossless end (B = Θ(L)), and the hybrid operating point B = Θ(ρL) meeting both task necessities at a reduced servable-memory coefficient (ρ ≪ 1), not a smaller growth class (Proposition 4). Remark 1. Were Σt the full history (the Transformer limit, B … view at source ↗
Figure 4
Figure 4. Figure 4: The information floor and the scissors gap, measured (Experiment B, Appendix G.2: NA(N, b=8, B; κ), L=512, 3 seeds; bands span seeds). (a) Exact-retrieval error vs. load N: every pure-state curve (m ∈ {64, 256, 1024} scalars) rises to the chance ceiling while the hybrid (the same m=64 state plus one global-attention layer) stays near zero, the scissors of Prediction P1.; dashed green = doubled budget (24k,… view at source ↗
Figure 5
Figure 5. Figure 5: The global-attention share ρ against release date (N = 61 hybrids; source-verified). The band [1/12, 1/4] (shaded): median 0.167, 57% in band, a 3.4× enrichment over uniform, behaving as a steady state (membership stable since 2024, dispersion 0.148→0.119→0.067 but p ≈ 0.06). Above-band outliers are distillation sweeps and edge models; below-band, single-shared-cache designs (YOCO, SambaY, Nemotron-H). Rig… view at source ↗
Figure 6
Figure 6. Figure 6: Published long-context retrieval stratifies by access class (the CCH analogue of PRH’s [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: CCH’s ontological prediction (Prediction P5.): frontier access-complete models narrow on a capability fingerprint generation by generation, while pure-family baselines stay at a characteristic distance at any scale. Schematic, not measured. parameters, public checkpoints ≤ 3B, so these are mechanism-level tests (existence-level for the single-axis walls, training-reliability-level for the conjunction), not… view at source ↗
Figure 8
Figure 8. Figure 8: The conjunction witness, measured (Experiment E, Appendix G.5; the composite [address→value] × S5 task, 3 seeds/cell). (a) Accuracy vs. load N at the long-extrapolation length T=256 (mean lines, per-seed dots, pass line 0.9): the recurrent-front hybrid (green) passes the registered cell (N=16) on every seed while no other arm meets the criterion; at the supra-capacity load N=64 the hybrid’s worst seed (0.7… view at source ↗
Figure 9
Figure 9. Figure 9: The state-tracking bifurcation, measured (Experiment C, Appendix G.1: running S5 products, trained at T ≤ 40; 3 seeds per cell, 8 for nh=4 general; bands span seeds). (a,b) Accuracy vs. length on the swap and general streams. (c,d) The measured grid: the bifurcation appears at the pre-registered boundary for this configuration. β ∈ (0, 1) (∗, the sharpest claim) cannot even fit the training distribution; o… view at source ↗
Figure 9
Figure 9. Figure 9: The state-tracking bifurcation, measured (Experiment C, Appendix G.1: running S5 products, trained at T ≤ 40; 3 seeds per cell, 8 for nh=4 general; bands span seeds). (a,b) Accuracy vs. length on the swap and general streams. (c,d) The measured grid: the bifurcation appears at the pre-registered boundary for this configuration. β ∈ (0, 1) (∗, the sharpest claim) cannot even fit the training distribution; o… view at source ↗
Figure 10
Figure 10. Figure 10: The Newton’s-apple thought experiment in an infinite stream. A single fact planted at t ≈ 0 must survive 107 tokens of semantically related noise to be retrieved at the query t = N. A pure SSM is barred by the Shannon wall (Theorem 1); a budgeted Transformer by the horizon wall; an idealized Transformer by the circuit wall (Proposition 2). Only the access-complete hybrid crosses, and it does so by paying:… view at source ↗
Figure 11
Figure 11. Figure 11: KV-memory per 1M tokens (bf16, log scale) across 15 census architectures, spanning ≈160× from Llama-3.1-70B (305 GiB) to Jet-Nemotron-2B (1.9 GiB): dense GQA (red) caches at every layer, MLA (amber) a compressed latent at ρ=1, layer-wise hybrids (green) only at global-attention layers; pure SSM/linear models have no Θ(L) bar. The memory-side signature of R1. Source-verified via Eq. (6) ( [PITH_FULL_IMAGE… view at source ↗
Figure 11
Figure 11. Figure 11: KV-memory per 1M tokens (bf16, log scale) across 15 census architectures, spanning ≈160× from Llama-3.1-70B (305 GiB) to Jet-Nemotron-2B (1.9 GiB): dense GQA (red) caches at every layer, MLA (amber) a compressed latent at ρ=1, layer-wise hybrids (green) only at global-attention layers; pure SSM/linear models have no Θ(L) bar. The memory-side signature of R1. Source-verified via Eq. (6) ( [PITH_FULL_IMAGE… view at source ↗
Figure 12
Figure 12. Figure 12: R4 made concrete: fourteen distinct compressive operators (left) route to four structural outcomes (right); the same operator family appears in both the access-complete keystone (green) and the degenerate corners, so membership is set by pairing with a scalable global index, not by operator. Edge width counts census models (N=140). 67%; the lone outlier is the LFM2 edge line, 25%). (iv) Null sensitivity: … view at source ↗
Figure 12
Figure 12. Figure 12: R4 made concrete: fourteen distinct compressive operators (left) route to four structural outcomes (right); the same operator family appears in both the access-complete keystone (green) and the degenerate corners, so membership is set by pairing with a scalable global index, not by operator. Edge width counts census models (N=140). The access spectrum, visualized. A deterministic PCA of seven architectura… view at source ↗
Figure 13
Figure 13. Figure 13: Robustness of the ρ concentration, in the spirit of PRH’s metric-agnosticism check (its [PITH_FULL_IMAGE:figures/full_fig_p029_13.png] view at source ↗
Figure 13
Figure 13. Figure 13: Robustness of the ρ concentration, in the spirit of PRH’s metric-agnosticism check (its [PITH_FULL_IMAGE:figures/full_fig_p030_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Architecture-space embedding of the census (deterministic PCA of seven standardized architectural properties; PRH’s [PITH_FULL_IMAGE:figures/full_fig_p030_14.png] view at source ↗
Figure 14
Figure 14. Figure 14: Architecture-space embedding of the census (deterministic PCA of seven standardized architectural properties; PRH’s [PITH_FULL_IMAGE:figures/full_fig_p031_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Representational convergence ̸= capability convergence (Experiment A: Pythia / Mamba / RWKV-4, all pretrained on The Pile with a shared tokenizer). (a) Retrieval capability stratifies by access structure at matched scale and data. (b) The failure signature at ∼3B: recency slope for the compressive-state family, near-flat for the indexed family, floor for the linear RNN. (c) PRH replication: cross-family m… view at source ↗
Figure 15
Figure 15. Figure 15: Representational convergence ̸= capability convergence (Experiment A: Pythia / Mamba / RWKV-4, all pretrained on The Pile with a shared tokenizer). (a) Retrieval capability stratifies by access structure at matched scale and data. (b) The failure signature at ∼3B: recency slope for the compressive-state family, near-flat for the indexed family, floor for the linear RNN. (c) PRH replication: cross-family m… view at source ↗
Figure 16
Figure 16. Figure 16: Channel commensurability (Prediction P3.): solving hybrids were predicted to align across channels above random initialization. Schematic. The pre-registered test failed, direction reversed (Appendix G.4): solving hybrids align below random initialization, which is itself high because two random channels echo the same input geometry; the schematic is retained as a record of the original claim. complementa… view at source ↗
Figure 16
Figure 16. Figure 16: Channel commensurability (Prediction P3.): solving hybrids were predicted to align across channels above random initialization. Schematic. The pre-registered test failed, direction reversed (Appendix G.4): solving hybrids align below random initialization, which is itself high because two random channels echo the same input geometry; the schematic is retained as a record of the original claim. G.5 The con… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 38 linked inside Pith

  1. [1]

    Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Simple linear attention language models balance the recall-throughput tradeoff.arXiv preprint arXiv:2402.18668, 2024

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff.arXiv preprint arXiv:2402.18668, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Hybrid architectures for language models: Systematic analysis and design insights.arXiv preprint arXiv:2510.04800, 2025

    Sangmin Bae, Bilge Acun, Chien-Yu Lin, Haroun Habeeb, Seungyeon Kim, Liang Luo, Junjie Wang, and Carole-Jean Wu. Hybrid architectures for language models: Systematic analysis and design insights.arXiv preprint arXiv:2510.04800, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Barrington

    David A. Barrington. Bounded-width polynomial-size branching programs recognize exactly those languages in NC1.Journal of Computer and System Sciences, 38(1):150–164, 1989

    1989

  5. [5]

    Nemotron- H: A family of accurate and efficient hybrid Mamba-Transformer models.arXiv preprint arXiv:2504.03624, 2025

    Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, et al. Nemotron- H: A family of accurate and efficient hybrid Mamba-Transformer models.arXiv preprint arXiv:2504.03624, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Priming: Hybrid state space models from pre-trained transformers.arXiv preprint arXiv:2605.08301, 2026

    Aditya Chattopadhyay, Elvis Nunez, Prannay Kaul, Benjamin Bowman, Evan Becker, Luca Zancato, David Thomas, Wei Xia, and Stefano Soatto. Priming: Hybrid state space models from pre-trained transformers.arXiv preprint arXiv:2605.08301, 2026

    Pith/arXiv arXiv 2026

  7. [7]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas.Elements of Information Theory. Wiley-Interscience, 2nd edition, 2006

    2006

  8. [8]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InInternational Conference on Machine Learning (ICML), 2024. arXiv:2405.21060

    Pith/arXiv arXiv 2024

  9. [9]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  10. [10]

    Smith, Anushan Fernando, et al

    Soham De, Samuel L. Smith, Anushan Fernando, et al. Griffin: Mixing gated linear recur- rences with local attention for efficient language models.arXiv preprint arXiv:2402.19427, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Language modeling is compression

    Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression. InInternational Conference on Learning Representations (ICLR), 2024. arXiv:2309.10668. 39

    Pith/arXiv arXiv 2024

  12. [12]

    Hymba: A hybrid-head architecture for small language models.arXiv preprint arXiv:2411.13676, 2024

    Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabalesh- warkar, et al. Hymba: A hybrid-head architecture for small language models.arXiv preprint arXiv:2411.13676, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Springer, 1999

    Bernhard Ganter and Rudolf Wille.Formal Concept Analysis: Mathematical Foundations. Springer, 1999

    1999

  14. [14]

    The Zamba2 suite: Technical report.arXiv preprint arXiv:2411.15242, 2024

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, and Beren Millidge. The Zamba2 suite: Technical report.arXiv preprint arXiv:2411.15242, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Zamba: A compact 7B SSM hybrid model.arXiv preprint arXiv:2405.16712, 2024

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7B SSM hybrid model.arXiv preprint arXiv:2405.16712, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Riccardo Grazzi, Julien Siems, Arber Zela, Jörg K. H. Franke, Frank Hutter, and Massimil- iano Pontil. Unlocking state-tracking in linear RNNs through negative eigenvalues.arXiv preprint arXiv:2411.12537, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2022

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces.arXiv preprint arXiv:2111.00396, 2022

    Pith/arXiv arXiv 2022

  19. [19]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Pith/arXiv arXiv 2022

  20. [20]

    RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, et al. RULER: What’s the real context size of your long-context language models?arXiv preprint arXiv:2404.06654, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    The platonic repre- sentation hypothesis

    Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic repre- sentation hypothesis. InInternational Conference on Machine Learning (ICML), 2024. arXiv:2405.07987

    Pith/arXiv arXiv 2024

  22. [22]

    IBM granite 4.0: Hybrid Mamba-2/Transformer open models

    IBM Granite Team. IBM granite 4.0: Hybrid Mamba-2/Transformer open models. IBM technical report, 2025

    2025

  23. [23]

    Kakade, and Eran Malach

    Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying.arXiv preprint arXiv:2402.01032, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Rishi Jha, Collin Zhang, Vitaly Shmatikov, and John X. Morris. Harnessing the universal geometry of embeddings.arXiv preprint arXiv:2505.12540, 2025

    arXiv 2025

  25. [25]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, et al. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  26. [26]

    Transformers are RNNs: Fast autoregressive transformers with linear attention.International Conference on Machine Learning (ICML), 2020

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention.International Conference on Machine Learning (ICML), 2020

    2020

  27. [27]

    Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    Kimi Team. Kimi linear: An expressive, efficient attention architecture.arXiv preprint arXiv:2510.26692, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    Kimi K3: Open frontier intelligence.arXiv preprint arXiv:2607.24653, 2026

    Kimi Team. Kimi K3: Open frontier intelligence.arXiv preprint arXiv:2607.24653, 2026. 40

    arXiv 2026

  29. [29]

    Sophia Koepke, Daniil Zverev, Shiry Ginosar, and Alexei A

    A. Sophia Koepke, Daniil Zverev, Shiry Ginosar, and Alexei A. Efros. Back into Plato’s cave: Examining cross-modal representational convergence at scale.arXiv preprint arXiv:2604.18572, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    On randomized one-round communication complexity.Computational Complexity, 8(1):21–49, 1999

    Ilan Kremer, Noam Nisan, and Dana Ron. On randomized one-round communication complexity.Computational Complexity, 8(1):21–49, 1999

    1999

  31. [31]

    Efficient memory management for large language model serving with PagedAttention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. Efficient memory management for large language model serving with PagedAttention. InSymposium on Operating Systems Principles (SOSP), 2023. arXiv:2309.06180

    Pith/arXiv arXiv 2023

  32. [32]

    Understanding and enhancing mamba-transformer hybrids for memory recall and language modeling.arXiv preprint arXiv:2510.26912, 2025

    Hyunji Lee et al. Understanding and enhancing mamba-transformer hybrids for memory recall and language modeling.arXiv preprint arXiv:2510.26912, 2025

    arXiv 2025

  33. [33]

    Universal intelligence: A definition of machine intelligence

    Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and Machines, 17(4):391–444, 2007

    2007

  34. [34]

    Chain of thought empowers trans- formers to solve inherently serial problems.arXiv preprint arXiv:2402.12875, 2024

    Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers trans- formers to solve inherently serial problems.arXiv preprint arXiv:2402.12875, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Jamba: A hybrid Transformer-Mamba language model.arXiv preprint arXiv:2403.19887, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, et al. Jamba: A hybrid Transformer-Mamba language model.arXiv preprint arXiv:2403.19887, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics (TACL), 2023

    2023

  37. [37]

    Matthew V. Mahoney. Text compression as a test for artificial intelligence. InProceedings of the AAAI Conference on Artificial Intelligence (AAAI), 1999

    1999

  38. [38]

    The parallelism tradeoff: Limitations of log-precision transformers.Transactions of the Association for Computational Linguistics (TACL), 2023

    William Merrill and Ashish Sabharwal. The parallelism tradeoff: Limitations of log-precision transformers.Transactions of the Association for Computational Linguistics (TACL), 2023. arXiv:2207.00729

    Pith/arXiv arXiv 2023

  39. [39]

    The illusion of state in state-space models.arXiv preprint arXiv:2404.08819, 2024

    William Merrill, Jackson Petty, and Ashish Sabharwal. The illusion of state in state-space models.arXiv preprint arXiv:2404.08819, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    Some attention is all you need for retrieval.arXiv preprint arXiv:2510.19861, 2025

    Felix Michalak and Steven Abreu. Some attention is all you need for retrieval.arXiv preprint arXiv:2510.19861, 2025

    arXiv 2025

  41. [41]

    MiniMax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    MiniMax. MiniMax-01: Scaling foundation models with lightning attention.arXiv preprint arXiv:2501.08313, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Expansion span: Combining fading memory and retrieval in hybrid state space models.arXiv preprint arXiv:2412.13328, 2024

    Elvis Nunez, Luca Zancato, Benjamin Bowman, Aditya Golatkar, Wei Xia, and Stefano Soatto. Expansion span: Combining fading memory and retrieval in hybrid state space models.arXiv preprint arXiv:2412.13328, 2024

    Pith/arXiv arXiv 2024

  43. [43]

    Retrievit: In-context retrieval capabilities of transformers, state space models, and hybrid architectures.arXiv preprint arXiv:2603.02874, 2026

    Georgios Pantazopoulos, Malvina Nikandrou, Ioannis Konstas, and Alessandro Suglia. Retrievit: In-context retrieval capabilities of transformers, state space models, and hybrid architectures.arXiv preprint arXiv:2603.02874, 2026

    Pith/arXiv arXiv 2026

  44. [44]

    RWKV: Reinventing RNNs for the transformer era.arXiv preprint arXiv:2305.13048, 2023

    Bo Peng, Eric Alcaide, Quentin Anthony, et al. RWKV: Reinventing RNNs for the transformer era.arXiv preprint arXiv:2305.13048, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Rethinking the role of efficient attention in hybrid architectures.arXiv preprint arXiv:2606.15378, 2026

    Ziqing Qiao et al. Rethinking the role of efficient attention in hybrid architectures.arXiv preprint arXiv:2606.15378, 2026. 41

    arXiv 2026

  46. [46]

    Qwen3-next: Towards ultimate training and inference efficiency

    Qwen Team. Qwen3-next: Towards ultimate training and inference efficiency. Alibaba Cloud technical blog, 2025

    2025

  47. [47]

    Qwen3.8-Max preview

    Qwen Team. Qwen3.8-Max preview. Alibaba Cloud announcement, World AI Conference (WAIC), 2026-07-19, 2026

    2026

  48. [48]

    Hopfield networks is all you need

    Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, et al. Hopfield networks is all you need. International Conference on Learning Representations (ICLR), 2021. arXiv:2008.02217

    Pith/arXiv arXiv 2021

  49. [49]

    Reasoning primitives in hybrid and non-hybrid LLMs: Do architectural differences yield advantages in state-tracking and recall?arXiv preprint arXiv:2604.21454, 2026

    Shivam Rawat, Lucie Flek, Florian Mai, and Nicholas Kluge Corrêa. Reasoning primitives in hybrid and non-hybrid LLMs: Do architectural differences yield advantages in state-tracking and recall?arXiv preprint arXiv:2604.21454, 2026

    Pith/arXiv arXiv 2026

  50. [50]

    The expressive capacity of state space models: A formal language perspective

    Yash Sarrof, Yana Veitsman, and Michael Hahn. The expressive capacity of state space models: A formal language perspective. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2405.17394

    arXiv 2024

  51. [51]

    Claude E. Shannon. A mathematical theory of communication.Bell System Technical Journal, 27(3):379–423, 1948

    1948

  52. [52]

    DeltaProduct: Improving state-tracking in linear RNNs via householder products.arXiv preprint arXiv:2502.10297, 2025

    Julien Siems, Timur Carstensen, Arber Zela, Frank Hutter, Massimiliano Pontil, and Riccardo Grazzi. DeltaProduct: Improving state-tracking in linear RNNs via householder products.arXiv preprint arXiv:2502.10297, 2025

    arXiv 2025

  53. [53]

    Retentive network: A successor to Transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to Transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Pith/arXiv arXiv 2023

  54. [54]

    Convergence without understanding: When language models agree on representations but disagree on reasoning.arXiv preprint arXiv:2605.23315, 2026

    Muhammad Usama and Dong Eui Chang. Convergence without understanding: When language models agree on representations but disagree on reasoning.arXiv preprint arXiv:2605.23315, 2026

    Pith/arXiv arXiv 2026

  55. [55]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), 2017

    2017

  56. [56]

    An empirical study of Mamba-based language models.arXiv preprint arXiv:2406.07887, 2024

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of Mamba-based language models.arXiv preprint arXiv:2406.07887, 2024

    Pith/arXiv arXiv 2024

  57. [57]

    Rush, and Tri Dao

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2408.15237

    Pith/arXiv arXiv 2024

  58. [58]

    Understanding and mitigating bottlenecks of state space models through the lens of recency and over-smoothing.arXiv preprint arXiv:2501.00658, 2025

    Peihao Wang, Ruisi Cai, Yuehao Wang, Jiajun Zhu, Pragya Srivastava, Zhangyang Wang, and Pan Li. Understanding and mitigating bottlenecks of state space models through the lens of recency and over-smoothing.arXiv preprint arXiv:2501.00658, 2025

    Pith/arXiv arXiv 2025

  59. [59]

    Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2024

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training.arXiv preprint arXiv:2312.06635, 2024

    Pith/arXiv arXiv 2024

  60. [60]

    Probabilistic computations: Toward a unified measure of complexity

    Andrew Chi-Chih Yao. Probabilistic computations: Toward a unified measure of complexity. 18th Annual Symposium on Foundations of Computer Science (FOCS), 1977. 42

    1977