Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

A one-layer Mamba can be trained to classify in context accurately even when nearly every prompt example carries an outlier, because its nonlinear gate learns to ignore corrupted inputs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 13:24 UTC pith:4EQTUTSE

load-bearing objection First provable training-dynamics analysis of a gated SSM in ICL; the 'unseen outlier' claim is narrower than advertised (cone restriction), but the core theorem is solid and deserves serious review. the 3 major comments →

arxiv 2510.00399 v2 pith:4EQTUTSE submitted 2025-10-01 cs.LG

How Can Mamba Learn In Context with Outliers and Generalize Provably?

Hongkang Li , Songtao Lu , Xiaodong Cui , Pin-Yu Chen , Meng Wang This is my paper
classification cs.LG
keywords in-context learningMambastate space modelsoutlier robustnesstraining dynamicsgeneralization boundsgating mechanismbinary classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to prove, for the first time, that Mamba—not just Transformers—can acquire in-context learning through ordinary SGD training, and to characterize when that learning survives prompts corrupted by outliers. Its central claim is that a one-layer Mamba trained on binary-classification prompts with an outlier fraction p_a will generalize to unseen tasks with zero-one error at most ε, provided the test prompt's outlier fraction α stays below min{1, p_a·l_tr/l_ts} and the test outlier directions are positive combinations of training outlier patterns. The same training recipe on a linear Transformer only yields such a guarantee for α < 1/2, so the paper identifies nonlinear gating as the mechanism that buys outlier resistance. This matters because real prompts, especially under poisoning attacks, can contain a majority of corrupted examples, and the result says Mamba can still make reliable predictions.

Core claim

The paper's central discovery is a provable separation between Mamba and linear-attention Transformers in outlier-heavy in-context learning. After training on prompts in which a fraction p_a of context examples carry additive outliers, the one-layer Mamba achieves zero-one classification error ≤ ε on unseen test prompts with outlier fraction α < min{1, p_a l_tr/l_ts}, even when the labels of outlier examples are flipped, targeted, or random, and even when the test outlier pattern is a previously unseen positive linear combination of the V training-time outlier patterns. A comparable linear Transformer, with gating removed, is guaranteed only for α < 1/2. The mechanism is the gate itself: the

What carries the argument

The load-bearing object is the closed-form output of a one-layer Mamba, F = Σ_i G_{i,l+1}(w) y_i p_iᵀ W_Bᵀ W_C p_query, where W_B and W_C form linear attention and G_{i,l+1}(w) is a scalar gate: σ(wᵀ p_i) times products of (1 − σ(wᵀ p_j)) for later positions. The gate sums to at most one, which lets the proof split the training dynamics of w into two phases: first w moves negative on outlier-carrying tokens, then a logarithmic phase drives those sigmoids to roughly 0 while keeping clean tokens at Θ(1). That two-phase analysis of the gating parameter is what allows the theorem to bound attention concentration (Corollary 1) and gating suppression plus locality (Corollary 2), and to carry the g

Load-bearing premise

The entire test-time guarantee rests on the assumption that every test outlier direction lies in the positive cone generated by the training outlier patterns (with coefficients summing to at least L > 0 and magnitude within a bounded interval); if an outlier falls outside that cone, the theorem offers no promise.

What would settle it

Train a one-layer Mamba under Theorem 1's conditions with, say, p_a = 0.6 and three training outlier patterns; at test time take outlier fraction α just below min{1, p_a l_tr/l_ts}, magnitude κ'_a at the high end of the allowed range, and an outlier direction v' = 0.7v*_1 + 0.6v*_2 − 0.4v*_3 (coefficient sum 0.9 ≥ L). If the zero-one classification error on such prompts consistently exceeds the predicted ε, the claim that the in-cone condition suffices would be refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Mamba can keep accurate in-context predictions when the fraction of corrupted context examples approaches 1, if training itself contained a similar fraction and test outlier directions lie in the training cone.
  • A linear Transformer, identical except for the gate, fails once the test outlier fraction reaches 1/2; the gate, not larger parameterization, is the source of the difference.
  • Achieving the resistant regime costs roughly Θ(l_tr) more iterations and larger batch sizes than linear Transformers, so the advantage is paid for during training.
  • The trained model's final prediction effectively uses only clean, same-pattern examples close to the query; outliers and far-away clean examples are exponentially downweighted.
  • Setting p_a = 0 recovers a Mamba counterpart of earlier Transformer ICL training analysis, so the framework contains the no-outlier regime as a special case.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The cone condition V′ means 'unseen outliers' are recombinations of known training outlier directions; an adversary using a truly new orthogonal outlier direction falls outside the theorem, so resistance to arbitrary unseen attacks is not established.
  • The exponential locality bias predicts that rearranging a fixed set of clean and outlier examples—putting outliers next to the query or pushing same-pattern clean examples far—should change Mamba's accuracy substantially; this order-sensitivity is a testable signature of the gating mechanism.
  • Because the proof only needs the block to decompose into linear attention plus a per-position gate, the same two-phase argument likely extends to Mamba-2, RetNet, and gated linear attention; a direct extension would sharpen the claim that gating generally, not the specific Mamba parametrization, drives outlier resistance.
  • Training on clean prompts only (p_a = 0) gives no distribution-shift guarantee; the theorem requires poisoning-aware training with moderate-magnitude outliers, so the defensive benefit is contingent on the training distribution matching the attack type.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies a one-layer Mamba model (linear attention followed by a nonlinear gating layer) trained with SGD on binary classification in-context learning tasks, where a fraction of context examples contain additive outliers. It proves a convergence/sample-complexity theorem (Theorem 1), a generalization theorem for distribution-shifted test prompts whose outliers lie in a cone generated by the training outlier directions (Theorem 2), analogous results for a linear Transformer (Theorems 3 and 4), and corollaries characterizing the gating mechanism. The main theoretical claim is that Mamba retains accurate ICL predictions even when the test outlier fraction α approaches 1, under conditions that also require a sufficient proportion of clean examples, whereas a linear Transformer is guaranteed to generalize only for α < 1/2. The paper also reports synthetic experiments, alongside visualizations of attention and gating values, supporting the mechanism.

Significance. If the central theorems hold, this is a valuable first theoretical analysis of the training dynamics of a Mamba-style architecture in an ICL setting, and it offers a concrete explanation for Mamba's empirically observed robustness to outlier-heavy prompts. The proof strategy is a strength: it isolates the role of the nonlinear gating layer and provides explicit concentration-based arguments in the appendix, with conditions that are largely stated in closed form. The comparison to linear Transformers under the same data model is also a useful contribution, and the experimental section tests the stated mechanism directly. However, the generality of the 'unseen outliers' claim is materially narrower than the abstract suggests, and one technical gap in the proof of Theorem 2 needs to be repaired before the result as written can be considered established.

major comments (3)
  1. [Theorem 2 condition (a); Appendix D.2, Eqs. (56)–(59)] The cone V' in Eq. (11) allows coefficients λ_i of mixed sign, requiring only Σ λ_i ≥ L > 0 and ‖v‖=1. The proof of the cross-term bound (56) and the gating-suppression bound (59) for v'_s ∈ V' appears to use Σ λ_i as if it were Σ |λ_i|, or as if each λ_i ≥ 0. Lemma 4 gives only upper bounds of the form w^T v*_i ≤ −c; if some λ_i are negative, the product λ_i (w^T v*_i) can be positive and can dominate the sum, so the claimed suppression w^T v' ≤ negative is not guaranteed. A concrete scenario with V=2, λ_1 ≈ 0.84, λ_2 ≈ −0.54 (which satisfies λ_1+λ_2=0.3, ‖v‖=1) shows that if w^T v*_1 ≈ −1 and w^T v*_2 ≈ −200, then w^T v' ≈ +108, despite v' ∈ V'. Thus the theorem as stated is not justified by the supplied proof. The fix is local: require λ_i ≥ 0 (or impose a separate bound on the negative part, e.g., Σ |λ_i| ≤ C L), and then revise the experimental outlier directions accordingly. As wri
  2. [Section 4.1, Figure 2] The experiments use pa = 0.6 and l_tr = l_ts = 20. Theorem 2 condition (c) permits only α < min{1, pa·l_tr/l_ts} = 0.6. The figure reports correct ICL behavior at α ≈ 0.8, which is outside the theorem's guarantee. The text says this is 'consistent with Remark 5,' but the theorem does not cover that regime. The experiment should be run within the guaranteed region (e.g., pa = 0.9 and α ≤ 0.8, or l_tr/l_ts > α/pa), or the α ≈ 0.8 result should be explicitly labeled as a heuristic extrapolation rather than as a validation of the theorem's quantitative bound. This is not a fatal flaw, but it currently overstates the empirical support for the stated theorem.
  3. [Abstract, Contribution 1, Section 5] The abstract and contribution list say Mamba can generalize 'even when the prompt includes additive outliers' and to 'unseen outliers'. The actual guarantee in Theorem 2 is restricted to outliers in the positive cone V' of training outlier directions; outliers orthogonal to span{v*_1,…,v*_V} or with mixed-sign coefficients are not controlled. The limitations discussion in Section 5 does not mention this restriction. I recommend adding an explicit limitation statement so that the scope of the result is not oversold. This is a presentation issue, but it is important for readers to calibrate the practical meaning of the theorem.
minor comments (5)
  1. [Appendix D.1, Eq. (47)] The summation upper limit 'ltr(1-pa)^{-1}' is likely a typo; the intended sum is over the l_tr context examples. Please correct the notation.
  2. [Notation throughout Appendix E] The symbol ν* is used both for the outlier direction v* and, in places, for the irrelevant pattern ν_k. This makes some inequalities hard to follow. Please use separate symbols, e.g., v* for outliers and ν_k for irrelevant directions.
  3. [Section 4.1, experimental outlier directions] The three test outlier directions v*_1', v*_2', v*_3' include negative coefficients. If condition (a) of Theorem 2 is strengthened in the revision to require λ_i ≥ 0, these directions will no longer satisfy the theorem's assumptions. The experiments should be adjusted to use nonnegative combinations, or the theorem should be generalized in a way that covers the experimental setup.
  4. [Section 4.2] The mechanism experiments use a three-layer Mamba, while the theory is for a one-layer model. This is fine as a heuristic check, but the text should explicitly state that the multi-layer experiments do not directly verify the one-layer theorem.
  5. [Appendix D.2, Eq. (61)] The displayed expression for the lower bound on F(Ψ^(T), g(P')) contains several nested fractions and is hard to parse. Rewriting it with clearer intermediate inequalities would help the reader verify the step.

Circularity Check

0 steps flagged

No material circularity; the only reused prior-work ingredient is a non-load-bearing coverage assumption, and the test-outlier cone restriction narrows but does not circularize the result.

full rationale

The paper's derivation is a sequence of proofs from explicitly stated data distributions, model equations, SGD updates, and sufficient conditions. Theorem 1 proves convergence of training on prompts with outlier fraction p_a; Theorem 2 then proves zero-one error bounds on test prompts whose outliers lie in the cone V' defined in Eq. (11). This restriction is a scope condition, not a circular reduction: the theorem does not claim arbitrary unseen outlier directions, and the proof derives suppression of V'-outliers using the training-dynamics Lemmas 3-6 rather than assuming the conclusion. No fitted constant is renamed as a prediction, and no uniqueness theorem from the authors' prior work is invoked to force the architecture or the result. The main reused prior-work item is Condition 1, restated from (Li et al., 2024a) by overlapping authors; however, it is a task-coverage assumption with a self-contained construction in the appendix, and it does not encode the Mamba generalization or robustness conclusion. The headline 'unseen outliers' is somewhat stronger than the cone condition supports, especially since test directions orthogonal to span{v*_1,...,v*_V} are not controlled, but this is an overstatement/limitation rather than circularity. Overall, the central claim has independent mathematical content and is not forced by self-citation or by definition.

Axiom & Free-Parameter Ledger

0 free parameters · 7 axioms · 0 invented entities

The proof depends on a stylized sparse data model and, crucially, on restricting test outliers to a positive cone of training outliers. No parameters are fit to data, but several theorems state sufficient conditions rather than necessary ones.

axioms (7)
  • domain assumption Inputs are sparse combinations: each token contains exactly one relevant pattern μ_j and one irrelevant pattern ν_k, all mutually orthogonal with norm β; labels are determined by the relevant pattern.
    Section 3.2, Eq. (6); this toy distribution makes the attention/gating computation tractable.
  • domain assumption The training task set T_tr satisfies Condition 1 (uniform coverage of each relevant pattern and label), restated from Li et al. 2024a.
    Section 3.3, Condition 1; needed so that no relevant pattern is biased during SGD.
  • domain assumption Training outlier labels are random, and outlier patterns are orthogonal to relevant/irrelevant patterns. At test time, labels of outlier-containing examples can be arbitrary.
    Definitions 1 and 2; random labels force the model to down-weight outliers through gating.
  • ad hoc to paper Test-time outlier directions are restricted to the cone V' = {Σ λ_i v*_i : Σ λ_i ≥ L > 0, ∥v∥ = 1}.
    Theorem 2 condition (a), Eq. (11); this is the main restriction that turns 'unseen outliers' into combinations of seen outliers.
  • domain assumption The analyzed architecture is a one-layer Mamba with A = -I_m and Δ_{j,i} = softplus(w_j^T u_i), which reduces exactly to linear attention plus scalar gating.
    Eq. (1)–(3) and Appendix E.1; the result is for this simplified Mamba, not the full selective SSM.
  • domain assumption Balanced training labels: ||{n : z_n = +1}| − |{n : z_n = −1}|| = O(√N).
    Section 3.3, following Shi et al. 2021.
  • standard math Standard concentration inequalities (Chernoff, Hoeffding, sub-Gaussian) hold.
    Used throughout the appendix, Lemmas 1–2.

pith-pipeline@v1.3.0-alltime-deepseek · 38499 in / 13861 out tokens · 117280 ms · 2026-08-04T13:24:40.924839+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of How Can Mamba Learn In Context with Outliers and Generalize Provably?." pith.science (2026). https://pith.science/paper/4EQTUTSE

@misc{pith2026251000399,
  author       = {Pith},
  title        = {Pith review of: How Can Mamba Learn In Context with Outliers and Generalize Provably?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4EQTUTSE}},
  note         = {Machine review of arXiv:2510.00399}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The Mamba model has gained significant attention for its computational advantages over Transformer-based models, while achieving comparable performance across a wide range of language tasks. Like Transformers, Mamba exhibits in-context learning (ICL) capabilities, i.e., making predictions for new tasks based on a prompt containing input-label pairs and a query, without requiring fine-tuning. Despite its empirical success, the theoretical understanding of Mamba remains limited, largely due to the nonlinearity introduced by its gating mechanism. To the best of our knowledge, this paper presents the first theoretical analysis of the training dynamics of a one-layer Mamba model, which consists of a linear attention component followed by a nonlinear gating layer, and its ICL generalization on unseen binary classification tasks, even when the prompt includes additive outliers. Our analysis shows that Mamba leverages the linear attention layer to select informative context examples and uses the nonlinear gating layer to suppress the influence of outliers. By establishing and comparing to the analysis of linear Transformers under the same setting, we show that although Mamba may require more training iterations to converge, it maintains accurate predictions even when the proportion of outliers exceeds the threshold that a linear Transformer can tolerate. These theoretical findings are supported by empirical experiments.

Figures

Figures reproduced from arXiv: 2510.00399 by Hongkang Li, Meng Wang, Pin-Yu Chen, Songtao Lu, Xiaodong Cui.

Figure 1
Figure 1. Figure 1: An example of outliers in context inputs. Training Stage: For a given task f, we consider learning with a pa ∈ [0, 1) fraction of examples containing additive outliers {v ∗ r } V r=1 that can affect the label of corresponding examples in each prompt, where v ∗ s ⊥ µj , v ∗ s ⊥ νk for any j ∈ [M1], k ∈ [M2], and s ∈ [V ]. The input of each context example satisfies x =  µj + κνk + κav ∗ s , with a probabil… view at source ↗
Figure 2
Figure 2. Figure 2: ICL classification error of Mamba and linear Transformer against α with different prompt outliers. (A) Label flipping. (B) Targeted labeling. (C) Random labeling. 4.2 The ICL Mechanism of Multi-Layer Mamba The learning model is a three-layer Mamba and a three-layer single-head linear Transformer. pa = 0.4 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: shows the first-layer attention scores in the testing prompt. The sum of attention scores on the examples that share the same pattern as the query is significantly larger than that on examples with other patterns, and this gap increases during training. This verifies Corollary 1 [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: The summation of attention scores in the 2nd and 3rd layers. (A) (B) [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The gating values of examples with or without outliers in the 2nd and 3rd layers. We then introduce other related theoretical works on optimization and generalization of neural networks in this section. Some works Zhong et al. (2017); Fu et al. (2020); Li et al. (2022b); Zhang et al. (2023b); Li et al. (2024d) study the generalization of neural networks using the model recovery framework by investigating t… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Transformers Learn the Optimal DDPM Denoiser for Multi-Token GMMs

    cs.LG 2026-04 unverdicted novelty 8.0

    Transformers converge globally to the optimal DDPM denoiser for multi-token GMMs via self-attention mean denoising, with explicit token and iteration requirements.

Reference graph

Works this paper leans on

32 extracted references · 26 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,

    Pith/arXiv arXiv

  2. [2]

    2: Initialization: W (0) B and W (0) C are initialized such that the first d diagonal entries of W (0) B and W (0) C are set asδ∈(0,0.2].w (0) ∼ N(0,Id+1/(d+ 1))

    Algorithm 1Training with Stochastic Gradient Descent (SGD) 1:Hyperparameters:The step sizeη, the number of iterationsT, batch sizeB. 2: Initialization: W (0) B and W (0) C are initialized such that the first d diagonal entries of W (0) B and W (0) C are set asδ∈(0,0.2].w (0) ∼ N(0,Id+1/(d+ 1)). 3: Training by SGD:For each iteration, we independently sampl...

    2018

  3. [3]

    A. Ali, I. Zimerman, and L. Wolf. The hidden attention of mamba models.arXiv preprint arXiv:2403.01590,

    Pith/arXiv arXiv

  4. [5]

    Y . Bai, F. Chen, H. Wang, C. Xiong, and S. Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection.arXiv preprint arXiv:2306.04637,

    Pith/arXiv arXiv

  5. [6]

    Bondaschi, N

    M. Bondaschi, N. Rajaraman, X. Wei, K. Ramchandran, R. Pascanu, C. Gulcehre, M. Gastpar, and A. V . Makkuva. From markov to laplace: How mamba in-context learns markov chains.arXiv preprint arXiv:2502.10178,

    Pith/arXiv arXiv

  6. [9]

    M. N. R. Chowdhury, M. Wang, K. E. Maghraoui, N. Wang, P.-Y . Chen, and C. Carothers. A provably effective method for pruning experts in fine-tuned sparse mixture-of-experts.arXiv preprint arXiv:2405.16646,

    Pith/arXiv arXiv

  7. [10]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752,

    Pith/arXiv arXiv

  8. [11]

    D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Pith/arXiv arXiv

  9. [12]

    D. Han, Z. Wang, Z. Xia, Y . Han, Y . Pu, C. Ge, J. Song, S. Song, B. Zheng, and G. Huang. Demystify mamba in vision: A linear attention perspective.arXiv preprint arXiv:2405.16605,

    Pith/arXiv arXiv

  10. [13]

    Hatamizadeh and J

    A. Hatamizadeh and J. Kautz. Mambavision: A hybrid mamba-transformer vision backbone.arXiv preprint arXiv:2407.08083,

    Pith/arXiv arXiv

  11. [14]

    11 J. He, X. Pan, S. Chen, and Z. Yang. In-context linear regression demystified: Training dynamics and mechanistic interpretability of multi-head softmax attention.arXiv preprint arXiv:2503.12734,

    Pith/arXiv arXiv

  12. [15]

    P. He, H. Xu, Y . Xing, H. Liu, M. Yamada, and J. Tang. Data poisoning for in-context learning.arXiv preprint arXiv:2402.02160,

    Pith/arXiv arXiv

  13. [16]

    Huang, Y

    Y . Huang, Y . Cheng, and Y . Liang. In-context convergence of transformers. InNeurIPS 2023 Workshop on Mathematics of Modern Machine Learning,

    2023

  14. [17]

    F. A. Joseph, K. K. Haefeli, N. Liniger, and C. Gulcehre. Hippo-prophecy: State-space models can provably learn dynamical systems in context.arXiv preprint arXiv:2407.09375,

    Pith/arXiv arXiv

  15. [18]

    H. Li, M. Wang, S. Liu, P.-Y . Chen, and J. Xiong. Generalization guarantee of training graph convolutional networks with graph topology sampling. InInternational Conference on Machine Learning, pages 13014–13051. PMLR, 2022a. H. Li, S. Zhang, and M. Wang. Learning and generalization of one-hidden-layer neural networks, going beyond standard gaussian data...

    Pith/arXiv arXiv 2023

  16. [19]

    Y . Luo, H. Li, Q. Liu, L. Shi, and X.-M. Wu. Node identifiers: Compact, discrete representations for efficient graph learning.arXiv preprint arXiv:2405.16435, 2024a. Y . Luo, H. Li, L. Shi, and X.-M. Wu. Enhancing graph transformers with hierarchical distance structural encoding.Advances in Neural Information Processing Systems, 37:57150–57182, 2024b. M....

    Pith/arXiv arXiv

  17. [20]

    URLhttps://openreview.net/forum?id=uHcG5Y6fdB. C. Olsson, N. Elhage, N. Nanda, N. Joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y . Bai, A. Chen, et al. In-context learning and induction heads.arXiv preprint arXiv:2209.11895,

    Pith/arXiv arXiv

  18. [21]

    J. Park, J. Park, Z. Xiong, N. Lee, J. Cho, S. Oymak, K. Lee, and D. Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks.arXiv preprint arXiv:2402.04248,

    Pith/arXiv arXiv

  19. [22]

    Qiang, X

    Y . Qiang, X. Zhou, and D. Zhu. Hijacking large language models via adversarial in-context learning. arXiv preprint arXiv:2311.09948,

    Pith/arXiv arXiv

  20. [24]

    Vershynin

    R. Vershynin. Introduction to the non-asymptotic analysis of random matrices.arXiv preprint arXiv:1011.3027,

    Pith/arXiv arXiv

  21. [25]

    Waleffe, W

    R. Waleffe, W. Byeon, D. Riach, B. Norick, V . Korthikanti, T. Dao, A. Gu, A. Hatamizadeh, S. Singh, D. Narayanan, et al. An empirical study of mamba-based language models.arXiv preprint arXiv:2406.07887,

    Pith/arXiv arXiv

  22. [27]

    H. Yang, B. Kailkhura, Z. Wang, and Y . Liang. Training dynamics of transformers to recognize word co-occurrence via gradient flow analysis. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. S. Yang, B. Wang, Y . Shen, R. Panda, and Y . Kim. Gated linear attention transformers with hardware- efficient training. InProce...

    Pith/arXiv arXiv

  23. [28]

    URL https://openreview.net/forum? id=0b328CMwn1. S. Zhao, M. Jia, L. A. Tuan, F. Pan, and J. Wen. Universal vulnerabilities in large language models: Backdoor attacks for in-context learning. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11507–11522,

    2024

  24. [30]

    For the derivation of Theorems 3 and 4, we fix the gating value to 1 and ignore its effect, and then follow the proof strategy of Theorems 1 and 2 accordingly

    Based on these results, we can further derive the classification error in this setting. For the derivation of Theorems 3 and 4, we fix the gating value to 1 and ignore its effect, and then follow the proof strategy of Theorems 1 and 2 accordingly. B Additional Experiments and the Algorithm We first show the visualization result of the second and the third...

    2017

  25. [32]

    37 E.6 Extension to Other SSM/Linear RNN Architectures Our theoretical analysis can be extended to a broader range of SSM or Linear RNN architectures

    ≤ lX i=1 G0 i,l+1(w(t)) + lX i=1 Gl i,l+1(w(t))(1−σ(w (t)⊤ ˆpl 1)) + lX i=1 Gl−1 i,l+1(w(t))(1 −σ(w (t)⊤ ˆpl−1 1 ))(1−σ(w (t)⊤ ˆpl−1 2 )) +· · ·+ lX i=1 G2 i,l+1(w(t)) l−1Y j=1 (1−σ(w (t)⊤ ˆp2 j )) ≤max j∈[l] ( lX i=1 Gj i,l+1(w(t)) ) ·(1 + (1−σ(w (t)⊤ ˆpl 1)) + (1−σ(w (t)⊤ ˆpl−1 1 ))(1 −σ(w (t)⊤ ˆpl−1 2 )) +· · ·+ l−1Y j=1 (1−σ(w (t)⊤ ˆp2 j ))) ≤1 + (1−σ...

    2024

  26. [2010]

    J. Wu, D. Zou, Z. Chen, V . Braverman, Q. Gu, and P. L. Bartlett. How many pretraining tasks are needed for in-context learning of linear regression?arXiv preprint arXiv:2310.08391, 2023a. Z. Wu, Y . Wang, J. Ye, and L. Kong. Self-adaptive in-context learning: An information compression perspective for in-context example selection and ordering.ACL, 2023b....

    Pith/arXiv arXiv

  27. [2017]

    15 A Proof Sketch of Main Theorems The proof idea of main theoretical results is as follows

    URLhttps://arxiv.org/pdf/1706.03175.pdf. 15 A Proof Sketch of Main Theorems The proof idea of main theoretical results is as follows. First, in Lemmas 3, 4, and 5, we depict the growth of WB, WC, and w along the directions of the relevant pattern, the irrelevant pattern, and the outlier pattern, respectively, across different training iterations. This res...

    Pith/arXiv arXiv

  28. [2021]

    13 J. Sun, H. Li, and M. Wang. Theoretical learning performance of graph networks: the impact of jumping connections and layer-wise sparsification.Transactions on Machine Learning Research, 2025a. J. Sun, S. Zhang, H. Li, and M. Wang. Theoretical guarantees and training dynamics of contrastive learning: How misaligned data influence feature purity. InHigh...

    Pith/arXiv arXiv

  29. [2022]

    S. Chen, H. Sheen, T. Wang, and Z. Yang. Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality.arXiv preprint arXiv:2402.19442, 2024a. 10 Y . Chen, C. Zhao, Z. Yu, K. McKeown, and H. He. Parallel structures in pre-training data yield in- context learning. InProceedings of the 62nd Annual Meeting ...

    Pith/arXiv arXiv

  30. [2023]

    K. Ahn, X. Cheng, H. Daneshmand, and S. Sra. Transformers learn to implement preconditioned gradient descent for in-context learning.arXiv preprint arXiv:2306.00297,

    Pith/arXiv arXiv

  31. [2024]

    Allen-Zhu, Y

    Z. Allen-Zhu, Y . Li, and Y . Liang. Learning and generalization in overparameterized neural networks, going beyond two layers. InAdvances in neural information processing systems, pages 6155–6166, 2019a. Z. Allen-Zhu, Y . Li, and Z. Song. A convergence theory for deep learning via over-parameterization. InInternational Conference on Machine Learning, pag...

    Pith/arXiv arXiv

  32. [2025]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners.Advances in Neural Information Processing Systems, 33:1877–1901,

    1901