Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Motif-Mamba: network motif improved mamba for long-range sequence modeling

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A rank-2 motif-constrained coupling added to Mamba's state update yields consistent gains in long-context extrapolation, language modeling, and neural-signal decoding for a negligible parameter cost.

desk verdict A plausible low-rank extension of Mamba whose central claim—that the motif prior matters—is undercut by an under-specified motif loss and an unquantified training control. read the letter →

arxiv 2608.00027 v1 pith:MLY3NSSJ submitted 2026-07-14 cs.AI

Chonghe Hao , Yue Sun , Jian Zhang , Yansong Wang , Wangzi Yao , Yunjie Yao , Tielin Zhang This is my paper
classification cs.AI
keywords statespacemodelsMambanetworkmotifslow-rankrecurrentcouplinglong-sequencemodelinglong-contextextrapolationstructuralpriorsbrain-computerinterfacedecoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Motif-Mamba claims that the diagonal state transitions of Mamba, which let each state dimension evolve mostly independently, can be augmented with a low-rank recurrent coupling that introduces structured cross-dimensional interactions without breaking linear-time inference. The paper adds a rank-r term to the state update and uses a differentiable motif loss to steer the learned coupling matrix toward the connectivity pattern of a chosen three-node network motif. Across long-context extrapolation, language-model benchmarks, and brain-computer interface decoding, the variant beats the corresponding Mamba backbone at every tested scale, with only a few thousand extra parameters. If this is right, it offers a cheap structural prior for turning diagonal state space models into more expressive sequence models while preserving their efficiency.

What carries the argument

The central mechanism is the low-rank recurrent coupling M = MI * MJ, with MI in R^(n x r) and MJ in R^(r x n) for r much smaller than n, inserted into the continuous-time SSM and discretized by exponential Euler. The recurrent update is computed as MI(MJ h_{t-1}), which projects the state into an r-dimensional latent space and back, giving O(nr) cost. A differentiable motif-counting proxy sigma(beta(M⊙M−theta)) supplies a soft adjacency matrix whose three-node motif frequency vector is compared with a target via an R² coefficient; the resulting motif loss L_motif = 1 − R² is added to the task loss. The motif-induced latent flow v(z) = MJ A MJ^+ z − z is used to analyze and visualize how dif

What would settle it

Binarize a trained module's coupling matrix with the threshold used in training and count directed three-node motifs; if the Motif-2 frequency is not elevated relative to an unconstrained low-rank baseline while L_motif is near zero, the regularizer is not enforcing the claimed structure.

Watch

Extended reading notes

Core claim

Motif-Mamba is a state space model that replaces the purely diagonal Mamba state update with a recurrence that includes a low-rank coupling term, so hidden dimensions interact through a rank-r bottleneck. The coupling matrix is trained with an additional loss that compares its soft three-node motif counts, computed through a sigmoid threshold of the squared coupling matrix, against a target motif profile such as Motif-2 (two inputs converging on one node, with no reciprocal edges). The paper reports consistent accuracy gains over Mamba on induction-head extrapolation up to 16,384 tokens, on six language-modeling tasks and LAMBADA perplexity across four model sizes, and on neural decoding. Ab

Load-bearing premise

The whole 'motif-constrained' part depends on comparing raw three-node motif counts computed from an n×n coupling matrix against a target motif vector via an R² score; if those counts are not properly normalized or comparable, the regularizer enforces something other than the stated motif structure.

Editorial extensions

If this is right

  • If the central claim holds, any Mamba-style diagonal state space model can acquire cross-dimensional communication with negligible parameter overhead: the largest reported model gains only 3,120 parameters, about 0.0002% of its size.
  • The fixed Motif-2 constraint, chosen for its convergent non-reciprocal three-node pattern, outperforms the cyclic Motif-12, suggesting that the stability of the local connectivity prior, not its complexity, is what helps long-range memory.
  • Long-context extrapolation beyond the training length improves substantially: on the induction-heads task, accuracy at 1,024 and 4,096 tokens rises from 48.7 to 55.6 and from 20.7 to 23.8, respectively, relative to Mamba.
  • The structural prior transfers to non-text temporal signals: the model decodes neural wrist trajectories better than the same backbone, indicating that the benefit is not language-specific.
  • The gain is not simply from extra training: continued training of the Mamba backbone with the same token budget does not reproduce the improvement, so the coupling structure itself is what matters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same low-rank coupling could be dropped into other diagonal or gated recurrent architectures as a generic cross-channel interaction plug-in, not only Mamba.
  • Inference: the motif loss is effectively a soft structural prior on the coupling graph; one could make the constraint adaptive by learning a distribution over motif targets per layer, which the paper mentions as future work.
  • Inference: the low-rank latent z_t defines a small coordinate system in which the learned dynamics can be inspected, suggesting that motif constraints can serve as an interpretability tool for otherwise opaque state space models.
  • Inference: the R² target-comparison formulation leaves a testable question about normalization, since raw three-node motif counts scale with state size; applying the method at larger d_state would likely require a normalized motif-frequency objective to keep the constraint well posed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. Motif-Mamba augments the diagonal state transition of Mamba with a low-rank recurrent coupling M = M_I M_J, motivated by the dynamics of three-node network motifs. The paper provides a discretization derivation (Lemma 1), a reconstruction property for the latent path (Lemma 2), and a motif-profile regularizer defined through a differentiable adjacency proxy and an R^2 alignment loss. It reports experiments on induction-head extrapolation, LM benchmarks from 130M to 1.4B, ablations, and BCI decoding, claiming consistent improvements over Mamba backbones and attributing them to the motif constraint rather than to the low-rank pathway or extra training.

Significance. If the reported gains are real and reproducible, this would be a useful minimal-cost architectural variant: the added parameters are tiny (Appendix D.2), the recurrence remains linear in sequence length, and the paper attempts to give the low-rank interaction a mechanistic interpretation through network motifs. The proofs in Appendices B.1-B.2 are standard and the complexity/parameter accounting is clear. However, the central attribution to the motif constraint is not yet established because the regularizer is under-specified and the empirical evidence is too weak to exclude continued training or random variation.

major comments (3)
  1. [§2.3, Eqs. (9)-(12), Appendix C] The motif regularizer is not a well-defined objective as written. Eq. (10) uses K, m_{M,k}, m_{target,k}, and m_{target} without ever specifying K, the target connectivity matrix or its size, or whether the counts are normalized to frequencies. The only worked example, c_2(W) in Eq. (25), is a raw sum over ordered triples and scales as O(N^3); in the experiments N = d_state (16 or 8), whereas the target appears to be a three-node motif. If m_M and m_target are raw counts of different orders, the R^2 term is dominated by the total edge density and minimizing L_motif can collapse to shrinking overall connectivity rather than matching motif topology. If a frequency normalization is intended, it is omitted from the method. Since the paper's central claim is that the motif constraint—not merely the low-rank pathway—produces the gains, this gap blocks reproduction and weakens the attribution.
  2. [§3.2, §3.4, D.3] The reported improvements do not distinguish the motif prior from additional training. E.2 states that Motif-Mamba is initialized from a pretrained Mamba checkpoint and continuously trained with 10M tokens, while the Mamba baselines in Table 1 appear to be the released pretrained checkpoints. D.3 asserts that continuing Mamba under the same budget 'does not bring noticeable performance improvement,' but no numbers or curves are provided. Therefore the 0.3–0.9 percentage-point gains in Table 1 and the 0.3 percentage-point Motifblank-vs-Motif2 gain in Table 2 could be due to continued training or to the low-rank pathway alone; there are also no standard deviations or significance tests on any of the LM numbers. The sentence in the conclusion that 'gains mainly come from the motif-based structural constraint' is not supported by the data as reported.
  3. [§3.1 and §3.4] The selection of Motif-2 as the default appears to be based on the same ablation table used to support the conclusion. Section 3.1 fixes 'Motif-Mamba denotes the Motif-2-constrained variant' and motivates it by the motif hierarchy; Section 3.4 then tests Motif2 and Motif12 and reports Motif2 best. If the choice was made after inspecting Table 2, the headline results are selected on the test tasks; if it was made a priori, the paper should state this and should show all constrained variants in the main tables. As written, the main experiments do not provide an independent test of the motif-hierarchy hypothesis.
minor comments (5)
  1. [Eq. (7)] The term 'B_t x_τ' should be 'B_τ x_τ'; the index on the input projection is dropped in the displayed formula.
  2. [Tables 1-2, Figure 2] No error bars, confidence intervals, or numbers of random seeds are reported for the LM benchmarks. For differences of 0.2–1.0 percentage points, the authors should provide at least seed-level variation or significance tests.
  3. [Figure 4 / E.4] BCI decoding results are presented only graphically. Please report the numerical MSE/R^2 values for each condition and day, so readers can compare the magnitude of the claimed improvement.
  4. [Appendix C] The description of Eq. (25) says 'no direct edges between j and k', but the formula excludes both directions. This should be stated explicitly, and the derivation could be shortened by avoiding the repeated index manipulations.
  5. [§2.2.1, Eq. (6)] The complexity is O(Ln) + O(Lnr); since r is a small constant in the experiments, this is still linear, but the paper should state explicitly that r is treated as a constant independent of n and L.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical, measured on held-out benchmarks with controlled ablations, and no predicted quantity reduces to a fitted input by construction.

full rationale

The central claim is empirical: Motif-Mamba improves over Mamba on long-sequence extrapolation, language-modeling benchmarks, and BCI decoding. These improvements are measured against the same Mamba backbones under the same continued-training budget, and the ablation contrast Motifblank-Mamba vs. Motif2-Mamba is meant to isolate the motif constraint from the unconstrained low-rank pathway (Table 2; Appendix D.3). No equation in the paper reduces a predicted benchmark result to an input of the model. Lemma 1 is a standard exponential-Euler discretization and Lemma 2 is an algebraic unrolling of the recurrence, not a derivation of performance. Equations 9-12 define a training regularizer; even if the motif-loss objective is under-specified and potentially ill-posed, an underspecified objective is a reproducibility or confound problem, not circularity. The motif-hierarchy motivation cites [12,14], with [12] overlapping with the present authors, but that citation is not the load-bearing support for the empirical gains: the independent classical reference [14] and the paper's own ablations also underlie the Motif-2 choice. Selecting Motif-2 after inspecting the same benchmark suite is a model-selection/overfitting concern rather than an equation-level reduction. Since the core results are externally falsifiable and not forced by a self-citation chain, the circularity score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central machinery rests on a handful of hand-chosen hyperparameters (rank r=2, λ, β, θ, K, target motif matrix) that are mostly unreported, and on a motif-hierarchy axiom that is self-cited. The math lemmas are standard, but the structural prior is under-determined as written.

free parameters (5)
  • low-rank dimension r = 2
    Chosen so that most three-node motif matrices can be represented (Appendix D.1); no ablation varying r is reported.
  • motif regularization weight λ
    Appears in L = L_task + λ L_motif (Eq. 12); its value is never reported.
  • binarization temperature β and threshold θ
    Used in the differentiable adjacency proxy M̃≈σ(β(M⊙M−θ)); values absent.
  • number of motif categories K and target motif matrix
    The R² alignment in Eq. 10 requires a target motif-frequency vector and a K; neither the motif list nor the target matrix is specified.
  • training tokens (10M) = 10,000,000
    Selected because loss stabilizes after ~2M tokens (Fig. S2); affects the magnitude of continued-training gains.
assumptions (4)
  • domain assumption Three-level dynamical hierarchy of three-node motifs and the stability/flexibility classification (Motif 2 stable, Motif 12 unstable)
    Invoked to justify the default Motif-2 constraint (§3.1, §3.4); grounded in self-cited refs [12,13,14] not proved in this paper.
  • standard math Exponential Euler discretization with first-order Taylor expansion is a valid approximation for the Motif-Mamba update
    Used to derive Lemma 1 (Appendix B.1); standard numerical analysis, though the treatment of the low-rank term as an exogenous input is approximate.
  • domain assumption The low-rank coupling M_I M_J does not destabilize the recurrent system
    No eigenvalue/stability analysis of the combined transition e^{-ΔA}+ΔM_I M_J is provided, yet training relies on stable dynamics.
  • ad hoc to paper The differentiable proxy and motif-count operator yield meaningful gradient signal for the structural prior
    The binarization and counting are described only for motif 2 (Appendix C); the general operator for K categories is asserted without derivation or normalization.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Motif-Mamba: network motif improved mamba for long-range sequence modeling." pith.science (2026). https://pith.science/paper/MLY3NSSJ

@misc{pith2026260800027,
  author       = {Pith},
  title        = {Pith review of: Motif-Mamba: network motif improved mamba for long-range sequence modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MLY3NSSJ}},
  note         = {Machine review of arXiv:2608.00027}
}
read the original abstract

Efficient long-sequence modeling remains a central challenge for large language models, as self-attention scales quadratically with sequence length. Mamba offers a linear-time alternative through selective state space recurrence, but its predominantly diagonal state transitions restrict explicit interactions among state dimensions. We propose Motif-Mamba, a structured state space model that augments Mamba with a motif-constrained low-rank recurrent pathway. Inspired by the dynamics of three-node network motifs, the proposed pathway projects hidden states into a compact dynamical subspace, imposes motif-guided interactions, and maps the resulting dynamics back to the original state space. This design enhances cross-dimensional communication while preserving the linear-time recurrent structure of Mamba. Experiments on long-sequence extrapolation, language modeling benchmarks, and brain--computer interface decoding show consistent improvements over Mamba backbones, suggesting that motif-guided low-rank dynamics provide an effective structural prior for long-range sequence modeling.

Figures

Figures reproduced from arXiv: 2608.00027 by the authors.

Figure 1
Figure 1. Overview of Motif-Mamba. (A) Motif-Mamba adds a motif-constrained coupling term to [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Long-sequence extrapolation on the induction heads task. Models are trained with 256 tokens and tested from 64 to 16384 tokens [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. BCI decoding results on the JangoBCI dataset. The figure compares Motif-Mamba with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Motif constraints induce structured low-rank recurrent dynamics. Projected flow fields and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Population-level comparison of latent interaction magnitudes. Each point represents the average ∥∆znorm∥2 over 200 trials for an individual layer in￾ternal channel (n = 1531). The coordi￾nates compare the interaction strength of Motif12-Mamba (x-axis) against Motif2- M…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 2 canonical work pages

  1. [1]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. InAdvances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020

    1901

  2. [2]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 9

    arXiv 2004

  3. [3]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Informa- tion Processing Systems, 2017

    2017

  4. [4]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    arXiv 1904

  5. [5]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. InAdvances in Neural Information Processing Systems, 2020

    2020

  6. [6]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. InAdvances in Neural Information Processing Systems, 2022

    2022

  7. [7]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations, 2022

    2022

  8. [8]

    Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rnns for the transformer era.arXiv preprint arXiv:2305.13048, 2023

    arXiv 2023

Show all 39 references
  1. [9]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  2. [10]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  3. [11]

    Network motifs: Simple building blocks of complex networks.Science, 298(5594):824–827, 2002

    Ron Milo, Shai Shen-Orr, Shalev Itzkovitz, Nadav Kashtan, Dmitri Chklovskii, and Uri Alon. Network motifs: Simple building blocks of complex networks.Science, 298(5594):824–827, 2002

    2002

  4. [12]

    Functional building blocks of neural networks: From network motifs to collective dynamics

    Zhang Jian, Sun Yue, Yao Wangzi, and Zhang Tielin. Functional building blocks of neural networks: From network motifs to collective dynamics. InInternational Conference on Machine Learning, 2026

    2026

  5. [13]

    Critical-state-accelerated rnn-based reinforce- ment learning.Neurocomputing, 2026

    Wangzi Yao, Yue Sun, Bo Xu, and Tielin Zhang. Critical-state-accelerated rnn-based reinforce- ment learning.Neurocomputing, 2026

    2026

  6. [14]

    Dynamic properties of network motifs contribute to biological network organization.PLoS biology, 3(11):e343, 2005

    Robert J Prill, Pablo A Iglesias, and Andre Levchenko. Dynamic properties of network motifs contribute to biological network organization.PLoS biology, 3(11):e343, 2005

    2005

  7. [15]

    Network motifs: Theory and experimental approaches.Nature Reviews Genetics, 8 (6):450–461, 2007

    Uri Alon. Network motifs: Theory and experimental approaches.Nature Reviews Genetics, 8 (6):450–461, 2007. doi: 10.1038/nrg2102

    2007 doi

  8. [16]

    Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran

    Tara N. Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. InIEEE International Conference on Acoustics, Speech and Signal Processing, pages 6655–...

    2013

  9. [17]

    Linking connectivity, dynamics, and computa- tions in low-rank recurrent neural networks.Neuron, 99(3):609–623, 2018

    Francesca Mastrogiuseppe and Srdjan Ostojic. Linking connectivity, dynamics, and computa- tions in low-rank recurrent neural networks.Neuron, 99(3):609–623, 2018

    2018

  10. [18]

    Long short-term memory.Neural Computation, 9 (8):1735–1780, 1997

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory.Neural Computation, 9 (8):1735–1780, 1997

    1997

  11. [19]

    Language model evaluation harness

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  12. [20]

    The lambada dataset: Word prediction requiring a broad discourse context

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. InProceedings of the 54th Annual Meeting of th...

    2016

  13. [21]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  14. [22]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. InProceedings of the AAAI Conference on Artificial Intelligence, 2020

    2020

  15. [23]

    Think you have solved question answering? try arc, the ai2 reasoning challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. InarXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  16. [24]

    Winogrande: An adversarial winograd schema challenge at scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. InCommunications of the ACM, volume 64, pages 99–106, 2021

    2021

  17. [25]

    Fu, Tri Dao, Khaled K

    Daniel Y . Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry hungry hippos: Towards language modeling with state space models. InInternational Conference on Learning Representations, 2023

    2023

  18. [26]

    Pythia: A suite for analyzing large language models across training and scaling.arXiv preprint arXiv:2304.01373, 2023

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large langu...

    2023 arXiv

  19. [27]

    Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow.Zenodo, 2021

    Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autoregressive language modeling with mesh-tensorflow.Zenodo, 2021

    2021

  20. [28]

    Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models.arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  21. [29]

    Perreault, Lee E

    Xuan Ma, Fabio Rizzoglio, Eric J. Perreault, Lee E. Miller, and Amy Kennedy. Using adversarial networks to extend brain computer interface decoding accuracy over time.eLife, 12:e84296,

  22. [30]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022 arXiv

  23. [31]

    Hirokawa, Aaron Szafer, Susan M

    Quanxin Wang, Song-Lin Ding, Yang Li, Josh Royall, David Feng, Phil Lesnar, Nile Graddis, Murtaza Naeemi, Ben Facer, Anh Ho, Tim Dolbeare, Bob Blanchard, Nick Dee, Wayne Wakeman, Karla E. Hirokawa, Aaron Szafer, Susan M. Sunkin, Seung Wook Oh, Amy Bernard, John W. Phillips, Mi...

    2020 doi

  24. [32]

    Jimmy T. H. Smith, Andrew Warrington, and Scott W. Linderman. Simplified state space layers for sequence modeling. InInternational Conference on Learning Representations, 2023

    2023

  25. [33]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  26. [34]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. 11

    2022

  27. [35]

    Harris, Lydia Ng, Brent Winslow, Nicholas Cain, Stefan Mihalas, Quanxin Wang, Chris Lau, Leonard Kuan, Alex M

    Seung Wook Oh, Julie A. Harris, Lydia Ng, Brent Winslow, Nicholas Cain, Stefan Mihalas, Quanxin Wang, Chris Lau, Leonard Kuan, Alex M. Henry, Marty T. Mortrud, Benjamin Ouellette, Thuc Nghi Nguyen, Staci A. Sorensen, Cliff R. Slaughterbeck, Wayne Wakeman, Yang Li, David Feng, ...

    2014

  28. [36]

    Sadovsky and Jason N

    Alexander J. Sadovsky and Jason N. MacLean. Mouse visual neocortex supports multiple stereotyped patterns of microcircuit connectivity.Journal of Neuroscience, 34(23):7769–7777, 2014

    2014

  29. [37]

    Bronstein

    Federico Monti, Karl Otness, and Michael M. Bronstein. Motifnet: A motif-based graph convolutional network for directed graphs.arXiv preprint arXiv:1802.01572, 2018

    2018 arXiv

  30. [38]

    Motif graph neural network.IEEE Transactions on Neural Networks and Learning Systems, 2023

    Zhengdao Chen, Soledad Villar, Lei Chen, and Joan Bruna. Motif graph neural network.IEEE Transactions on Neural Networks and Learning Systems, 2023. 12 A Notation Table S1: Notation used throughout the paper. Symbol Definition State space model xt,x(t)Discrete and continuous i...

    2023

  31. [2023]

    doi: 10.7554/eLife.84296

    doi

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.