Pith. sign in

REVIEW 2 major objections 4 minor 2 cited by

Interleaved Mamba/Transformer hybrids beat pure Transformers and SSMs at dense in-context retrieval, while the locality-structured position embeddings that SSMs learn make them—and their hybrids—slower than Transformers at two-hop position

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 19:15 UTC pith:4JAF3TWJ

load-bearing objection Careful controlled comparison yielding solid empirical findings on hybrid retrieval; the mechanistic spiral story is correlational and needs a same-attention-depth control before it can carry the conclusion. the 2 major comments →

arxiv 2603.02874 v2 pith:4JAF3TWJ submitted 2026-03-03 cs.AI

Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architectures

Georgios Pantazopoulos , Malvina Nikandrou , Ioannis Konstas , Alessandro Suglia This is my paper
classification cs.AI
keywords in-context retrievalstate space modelsMambahybrid architecturestransformersdata efficiencylength generalizationpositional embeddings
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to decide whether hybrid architectures—models that interleave or gate Mamba-style state-space layers with self-attention—can get the best of both worlds: the linear-time processing of SSMs and the precise recall of Transformers. Using two synthetic tasks, n-gram retrieval and position retrieval, it finds that hybrids, especially interleaved ones, are more data-efficient, extrapolate to sequences four times longer than training, and stay robust when queries repeat. Transformers, however, still learn two-hop position retrieval faster, and standalone SSMs essentially fail at it. The paper explains the gap mechanistically: SSM-based models organize their position-token embeddings into a locality-preserving two-dimensional spiral, a structure that helps with simple retrieval but slows precise two-hop associations. If right, the work gives a task-conditional rule for architecture selection and a concrete representational signature to look for in any SSM-based model.

Core claim

Hybrid models combining Mamba and Transformer blocks are more data-efficient, generalize to sequences four times longer, and handle duplicate queries better than either pure family on n-gram retrieval; on position retrieval, Transformers remain fastest and SSMs never solve the task within the training budget. The paper attributes the difference to an emergent representation: models containing SSM blocks learn locality-aware positional embeddings, visualized as a two-dimensional spiral in PCA space, where adjacent positions become neighbors. This property arises from the recurrent hidden-state update rule and, while interpretable, acts as a bottleneck for precise two-hop lookup. The authors a

What carries the argument

The load-bearing object is the locality-preserving two-dimensional spiral in the embedding space of positional tokens, which develops in Mamba, Mamba2, and all hybrid variants but not in Transformers. It explains the serial-position learning dynamics (beginning and end positions are learned first) and is claimed to hinder precise two-hop positional retrieval, since adjacent positions are forced to be neighbors rather than being mapped to arbitrary distinct coordinates.

Load-bearing premise

The load-bearing premise is that the locality-preserving spiral in SSM positional embeddings—not the number of attention layers—is what slows hybrid and SSM models on two-hop position retrieval.

What would settle it

Train an interleaved hybrid in which the position-token embeddings are explicitly prevented from forming a locality structure (e.g., by adding a penalty that spreads neighbors apart) and check whether two-hop position accuracy improves; if it does not, the spiral is a byproduct rather than the cause.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Architecture selection for sequence models can be task-conditional: choose interleaved hybrids for dense retrieval and copying tasks, and pure Transformers with RoPE for tasks requiring precise positional grounding.
  • Interleaved hybrids offer a practical robustness gain for retrieval-augmented generation, since they maintain low error rates when query n-grams appear multiple times, whereas pure SSMs show a strong recency bias.
  • The serial-position effect is a training signature: SSM-based models solve the head and tail of a sequence first, so tasks requiring uniform coverage of all positions will need enough attention layers to overcome this bias.
  • The spiral structure in positional embeddings can serve as a cheap diagnostic: if a hybrid's embeddings form a clear spiral, expect it to lag Transformers on two-hop lookup, regardless of its n-gram retrieval performance.
  • Standalone SSMs are not sufficient for in-context retrieval that requires matching a query to an arbitrary earlier token, even when the state space is scaled up, so pure-SSM systems should be paired with attention when such retrieval matters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the spiral is truly causal rather than a byproduct of attention scarcity, then an intervention that breaks the locality structure—such as a regularizer that pushes adjacent position embeddings apart—should speed up two-hop retrieval in hybrids; the paper does not run this test.
  • The same mechanism likely explains the 'needle in a haystack' brittleness reported for pure Mamba models in other contexts: a locality-preserving state update makes it hard to isolate one specific earlier token, so the spiral may be a general signature of this limitation.
  • The findings imply a concrete engineering rule for multimodal grounding models (e.g., vision-language-action systems built on hybrid backbones): they need a minimum density of attention blocks, since two-hop positional lookup is exactly the operation those models perform.
  • A scaling prediction follows: increasing SSM state size improves n-gram retrieval but may not help two-hop retrieval, and could even make the spiral more rigid; this is testable by comparing Mamba variants with different state dimensions on position retrieval.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper reports a controlled comparison of Transformers, SSMs (Mamba/Mamba2), and hybrid interleaved/two-stream architectures on two synthetic in-context retrieval tasks: n-gram retrieval (reproduce the k tokens following a query n-gram) and position retrieval (two-hop lookup from a query token to its positional index). The authors evaluate data efficiency, length generalization, robustness to duplicate queries, and learned representations. The main empirical findings are: hybrid interleaved models are the most data-efficient on n-gram retrieval (Finding 1), generalize to 4x training length (Finding 2), and are most robust to duplicate queries (Finding 3); Transformers converge fastest on position retrieval (Finding 4); SSMs and hybrids show a serial-position-like learning trajectory (Finding 5); and SSM-based models learn locality-preserving, spiral-shaped positional embeddings, which the paper claims 'ultimately hinders precise two-hop associative lookup' (Finding 6).

Significance. The empirical results are timely and generally well controlled: models are parameter-matched (~150M), each condition is run with 3 data seeds x 3 learning rates (Section A.2), and the authors are transparent about a known confound in the length-generalization experiments, providing ablations in Figure 12. If the main empirical findings hold, the paper provides actionable guidance for architecture selection: interleaved Mamba/Transformer hybrids dominate pure families on dense in-context retrieval, while Transformers remain preferable for two-hop positional lookup. The representation analysis (Figures 9-11) is interesting and potentially interpretable. However, the causal claim that the locality-preserving spiral itself hinders position retrieval is not supported by the current evidence; this is load-bearing because the paper's mechanistic conclusion rests on it.

major comments (2)
  1. [Section 4.2.1 / Conclusion (Finding 6)] The central mechanistic claim--that the locality-preserving spiral in SSM position embeddings 'ultimately hinders precise two-hop associative lookup'--is supported only by correlational evidence. Figures 9-11 show the spiral for one model per family, but nothing varies the spiral while holding attention depth fixed. Table 1 shows pure Transformers have 12 attention layers, whereas the HybridI variants have fewer (e.g., 8 in the 1M2/1T model), and Figure 7c shows that increasing Transformer-block frequency monotonically narrows the gap with Transformers. Thus attention density alone could explain the position-retrieval speed difference, with the spiral as a byproduct. To support the causal interpretation, the authors need either a same-attention-depth control (e.g., a hybrid with as many attention layers as the pure Transformer) or an intervention that breaks/injects the spiral structure
  2. [Section 4.1 / Figure 7c and Table 1] Finding 2 claims hybrid models 'outperform both standalone Transformers... and standalone SSMs' on length generalization, but the paper acknowledges in Section 4.1 that the hybrid models shown in Figure 4 contain the highest number of Transformer layers. Figure 12 provides ablation over hybrid configurations only; it does not include a pure Transformer with a matched number of attention layers. Since Table 1 shows different attention-layer counts across families, the headline comparison could partly reflect model capacity rather than the hybrid architecture per se. The authors should report a pure-Transformer baseline with a number of attention layers matched to the best hybrid to strengthen this claim, or soften the wording to note the confound.
minor comments (4)
  1. [General (plots)] No error bars or confidence bands are shown for the main data-efficiency and duplicate-query curves, despite the paper reporting 9 runs per model. Adding variance information would help the reader judge the reliability of cross-model differences.
  2. [Section 4.2 / Finding 4] Finding 4 says standalone SSMs 'are not able to solve the task within the training budget,' but Figure 7a shows Mamba2 reaching non-negligible accuracy near the end of training. Consider wording it as 'do not reach the accuracy threshold within the budget' to avoid overstatement.
  3. [Appendix B.3, Eq. (1)] The expression l1 = -Σ p(x_i) log p(x_i) denotes negative entropy, which is negative; the subsequent numerical value is consistent with the intended loss formula but the sign convention is confusing. Clarify by writing the loss as -Σ p log p or by defining l1 directly as the cross-entropy term.
  4. [Section 3.1 / Task provenance] The position-retrieval task is attributed to Pantazopoulos et al. (2024), with overlapping authors. Please state explicitly any modifications from the prior task (sequence length, vocabulary, query sampling), so that the self-sourced benchmark is fully transparent.

Circularity Check

0 steps flagged

No significant circularity: the paper's load-bearing claims are empirical measurements and new representation analyses, not derivations from fitted parameters or self-citations.

full rationale

The central findings are controlled experimental comparisons across Transformers, SSMs, and hybrid models on two synthetic retrieval tasks. The n-gram task is taken from external work (Jelassi et al., 2024), and the position-retrieval task is sourced from the authors' own prior work (Pantazopoulos et al., 2024), but that citation is used only to define the task, not as evidence for any result; the reported learning curves, data-efficiency measurements, length-generalization results, and representation analyses are new and are not fitted to the conclusions. Finding 6's causal attribution — that locality-aware spiral embeddings hinder two-hop lookup — is supported by correlational evidence (PCA visualizations, cosine-similarity matrices) and by the observed effect of attention-block frequency, but this is an evidentiary/support concern, not circularity: no equation defines the spiral in terms of task performance, and no fitted parameter is renamed as a prediction. The only self-citation is not load-bearing and does not reduce any claim to its own input.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new entities are postulated: the 'locality spiral' is an observed empirical pattern in trained embeddings (Figures 9-11, 20-22), not a postulated mechanism with a falsifiable out-of-paper handle, so it is treated as an interpretive finding rather than an invented entity. The load-bearing free parameters are the training budget, the S=16 hybrid state dimension, the 95% convergence threshold, the n-gram task shape, and the duplicate-query schedule; all are disclosed but none are justified by sensitivity analysis. The domain assumptions concern task validity, single-configuration generalization, single-run representation analysis, and the self-provenance of the position-retrieval benchmark.

free parameters (5)
  • Position-retrieval training budget = 20x10^6 examples (312,500 steps)
    Finding 4 ('SSMs fail to learn the task entirely within the training budget') is conditional on this horizon; loss curves in Fig. 17 show SSM models still improving at cutoff, so 'cannot solve' is a budget-dependent statement.
  • SSM state dimension for hybrid models = S=16 (pure SSMs run at S=16/32/64)
    Deliberately the minimum capacity tier (Section A.1); governs every hybrid-vs-SSM comparison in Findings 1-3. Conservative for the hybrid claim (a larger state would likely help hybrids further), but the asymmetry is a hand-set design choice.
  • Data-efficiency accuracy threshold = 95%
    Section 4.1: 'reaching near-perfect accuracy (>=95%)'. The ordering of models by data efficiency depends on this threshold, which is chosen without sensitivity analysis across thresholds.
  • N-gram task shape and length caps = n=2, k=3; train length <=100; eval <=400
    The '4x length generalization' headline (Finding 2) and task difficulty are anchored to these constants, chosen without a stated justification or scaling study.
  • Duplicate-query schedule = 2-10 duplicates; segments s={2,3,4,10}
    Duplicate-query robustness (Finding 3) is measured at these densities; error rates grow with duplicates (Fig. 5), so the headline error-rate ordering depends on this evaluation schedule.
axioms (4)
  • domain assumption Synthetic in-context retrieval tasks are valid proxies for real-world capabilities (RAG, instruction following, cross-modal grounding)
    Section 3.1 motivates the tasks as proxies; the authors concede in the Limitations paragraph of Section 5 that transfer to real-world sequence modeling is untested.
  • domain assumption A parameter-matched (~150-160M params, embedding tables excluded) decoder-only comparison isolates architectural family differences
    Section 3.2 and Table 1 fix one configuration per family (state dim, attention density, RoPE/NoPE), so conclusions about 'hybrid architectures in general' extrapolate from a single hyperparameter setting.
  • domain assumption Position embeddings analyzed for 'a model from each family' (single run) are representative of the family
    Sections 4.2.1 and B.5: PCA/cosine-similarity figures (8-11, 20-22) visualize one model instance rather than the 9-run ensemble, so representational conclusions rest on one seed per family.
  • domain assumption The position-retrieval benchmark's construct validity is inherited from the authors' own prior work
    The task was introduced in Pantazopoulos et al. 2024 (three overlapping authors); its suitability as a proxy for two-hop cross-modal grounding is assumed rather than re-established here.

pith-pipeline@v1.3.0-alltime-deepseek · 20529 in / 19677 out tokens · 174475 ms · 2026-08-02T19:15:12.653109+00:00 · methodology

0 comments
read the original abstract

Transformers excel at in-context retrieval but suffer from quadratic complexity with sequence length, while State Space Models (SSMs) offer efficient linear-time processing but have limited retrieval capabilities. We investigate whether hybrid architectures combining Transformers and SSMs can achieve the best of both worlds on two synthetic in-context retrieval tasks. The first task, n-gram retrieval, requires the model to reproduce an n-gram that succeeds the query within the input sequence. The second task, position retrieval, presents the model with a query token and requires it to perform a two-hop lookup: first locating the corresponding element in the sequence, and then outputting its positional index. Under controlled conditions, we assess data efficiency, length generalization, robustness to out of domain training examples, and learned representations across Transformers, SSMs, and hybrid architectures. We find that hybrid models outperform SSMs and match or exceed Transformers in terms of data efficiency and extrapolation for tasks that require precise information retrieval from the input context. However, Transformers maintain superiority in position retrieval tasks. Through representation analysis, we discover that SSM-based models develop locality-aware embeddings where tokens representing adjacent positions become neighbors in embedding space, forming interpretable structures. This property is absent in Transformers as causal attention is sufficient for acquiring positional associations, and the introduction of positional encoding amplifies this behavior, leading to improvement in data efficiency. SSMs on the other hand update their internal representations incrementally and without positional encodings, are required to learn these associations. Our findings reveal fundamental differences in how Transformers and SSMs, and hybrid models learn positional associations.

Figures

Figures reproduced from arXiv: 2603.02874 by Alessandro Suglia, Georgios Pantazopoulos, Ioannis Konstas, Malvina Nikandrou.

Figure 1
Figure 1. Figure 1: Overview of the synthetic in-context retrieval tasks. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of hybrid architectures. Left: Interleaved Mamba and Transformer blocks. A Trans￾former block is inserted every N Mamba blocks. Right: Two stream block with a gate mechanism. The outputs from both streams are fused with a learnable gating mechanism. (Zhang et al., 2023), and robotic manipulation (Kim et al., 2024; Wen et al., 2025a), where the challenge is not only retrieval of a referent but … view at source ↗
Figure 3
Figure 3. Figure 3: (a) N-gram retrieval data efficiency. We train models to retrieve a sequence of k = 3 tokens that follow a randomly selected n-gram (n = 2) in a string of length ≤ 100, and evaluate on strings of length 100. While Transformers train significantly faster than SSMs, hybrid architectures are converging even faster. (b) Effect of state space dimension. We train SSM models with different state space dimension. … view at source ↗
Figure 4
Figure 4. Figure 4: (a) Illustration of the suffix (top) / prefix n-gram retrieval (bottom) variants. In the suffix version the query is given at the end of the input sequence, while in the prefix version the query is provided at the beginning. (b) When training with sequences ≤ 100 Mamba2 exhibits greater generalization than a Transformer with RoPE embeddings, while hybrid models show near-perfect generalization abilities. (… view at source ↗
Figure 5
Figure 5. Figure 5: Error rates with non unique n-gram suffix queries of a model from each family: [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Preference rates with non unique n-gram suffix (top) and prefix (bottom) queries across different [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: (a) Position retrieval data efficiency. We train models to retrieve the position of a token randomly sampled from a sequence of 200 tokens. Transformers converge fastest, while hybrid architectures require more steps and SSMs fail to learn the task entirely within the training budget. (b) Effect of state space dimension. Even SSMs with the largest state space dimensions cannot solve the task. (c) Effect of… view at source ↗
Figure 8
Figure 8. Figure 8: We track the per-token performance of a model from each family: [PITH_FULL_IMAGE:figures/full_fig_p009_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: We track the embeddings of the position tokens in a 2D plane across training for [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: We track the cosine similarities between embeddings of position tokens projected using PCA [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Average absolute distance of K-nearest neighbors of position embeddings. Consequently, we attribute this behavior to the up￾date rule of the hidden state of each model. In prin￾ciple, the Transformer can look at all tokens in the past to update the hidden state at each time step. SSMs and Mamba in particular force a very unique and strict update of the hidden state which is based solely on the state from … view at source ↗
Figure 12
Figure 12. Figure 12: Test-time extrapolation of different hybrid interleaved configurations for the [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: We track the per-token performance of different hybrid interleaved configurations. All hybrid [PITH_FULL_IMAGE:figures/full_fig_p020_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Error rates with non unique n-gram suffix (top) and prefix (bottom) queries of a model from [PITH_FULL_IMAGE:figures/full_fig_p020_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Error rates with non unique n-gram suffix queries of each hybrid interleaved model. All variants [PITH_FULL_IMAGE:figures/full_fig_p021_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Preference rates with non unique n-gram suffix (top) and prefix (bottom) queries across different [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: We track the training loss of Transformers, SSMs and Hybrid models for the position retrieval [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
Figure 19
Figure 19. Figure 19: Left: Data efficiency on the position retrieval. The two stream hybrid with reverse gating (Hybrid2SR), where Mamba output edits the hidden states of a Transformer, converges faster than Hybrid2S and even approximates the performance of the Transformer (RoPE) shown here as a purple dashed line. Right: The per-token performance of Hybrid2SR. 10 5 10 6 10 7 Number of Training Examples 0.00 0.02 0.04 0.06 0.… view at source ↗
Figure 18
Figure 18. Figure 18: Gating of effect of Hybrid2S for the task of position retrieval. Deeper layers have stronger gate activation indicating late fusion. Two-stream hybrid models with reverse gat￾ing We also experiment with dual-stream hybrid models where the Mamba stream edits the hidden states of a Transformer. In this setup, the tanh activation is applied at the outputs of the Mamba model ( [PITH_FULL_IMAGE:figures/full_f… view at source ↗
Figure 20
Figure 20. Figure 20: We track the embeddings of the position tokens in a 2D plane across training for hybrid models. [PITH_FULL_IMAGE:figures/full_fig_p024_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: We plot the cosine similarities between embeddings of position tokens projected using PCA [PITH_FULL_IMAGE:figures/full_fig_p025_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: We plot the cosine similarities between embeddings of position tokens projected using PCA [PITH_FULL_IMAGE:figures/full_fig_p026_22.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

    cs.AI 2026-07 conditional novelty 6.0

    Capability, unlike representation, converges to access-complete hybrid architectures, and a witness task separates pure from hybrid families under fixed budgets.

  2. The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

    cs.AI 2026-07 conditional novelty 6.0

    Long-context capability converges to hybrid architectures pairing O(1) state with a scalable index; pure scaling of SSMs or Transformers alone hits information, horizon, or circuit walls.

Reference graph

Works this paper leans on

3 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [2014]

    Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter

    URLhttps://arxiv.org/abs/1410.5401. Riccardo Grazzi, Julien Siems, Simon Schrodi, Thomas Brox, and Frank Hutter. Is mamba capable of in- context learning?arXiv preprint arXiv:2402.03170, 2024. URLhttps://arxiv.org/abs/2402.03170. Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. ...

    Pith/arXiv arXiv 2024

  2. [2021]

    Liliang Ren, Yang Liu, Yadong Lu, yelong shen, Chen Liang, and Weizhu Chen

    URLhttps://proceedings.mlr.press/v139/radford21a. Liliang Ren, Yang Liu, Yadong Lu, yelong shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. InThe Thirteenth International Conference on Learning Representations, 2025. URLhttps://openreview.net/forum?id=bIlnpVM4bc. Jianlin Su, Murt...

    Pith/arXiv arXiv 2025

  3. [2024]

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy

    URLhttps://openreview.net/forum?id=duRRoGeoQT. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transformers.Advances in Neural Information Processing Systems, 36:24892–24928, 2023. URLscholar.google.com/citations?user=AUFTexwAAAAJ& hl=en. Sahar Ka...

    Pith/arXiv arXiv 2023