Pith. sign in

REVIEW 3 major objections 4 minor 23 references

With independent forget and input gates, one recurrent state reaches 0.99 on retention and 0.89 on overwriting at four times training length.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:43 UTC pith:UQOG2HIL

load-bearing objection Naju is a well-executed method paper with a genuinely useful architecture and a clean diagnostic result, but the central causal claim—that gate decoupling drives the joint retention/overwrite performance—is never directly tested, and the Transformer baseline looks too weak to trust. the 3 major comments →

arxiv 2607.21000 v1 pith:UQOG2HIL submitted 2026-07-23 cs.AI

Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory

Hyuk Lim , Seunghyun Yoon This is my paper
classification cs.AI
keywords state space modelslong-sequence memoryrecurrent memoryretention and overwritingforget gateinput gateassociative recalllinear-time sequence modeling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that a recurrent sequence model can keep old information and replace stale information in the same fixed-size state if the two operations have separate gates. Its proposed unit, Naju, updates the state as x_n = f_n ⊙ x_{n-1} + i_n ⊙ (B_n h_n), where the forget gate f_n acts directly as the discrete pole controlling retention and an independent input gate i_n controls the strength of new writes. The paper shows that any non-expansive complementary single-gate recurrence ties effective retention r and write gain w through |r| + w ≤ 1, meaning near-lossless retention forces weak writing, and that decoupling f_n from i_n removes this constraint. In the diagnostic suite at four times the training length, Naju is the only evaluated model that stays strong on both retention (0.99) and overwriting (0.89), while keeping linear-time, linear-memory scaling. A sympathetic reader would care because long-session dialogue, repository-level code, and streaming telemetry all ask one fixed memory to hold bindings for a long time and then decisively replace them.

Core claim

The paper's central claim is that long-sequence memory in a recurrent state does not need a detour through continuous-time dynamics: the discrete transition can be parameterized directly, and the right parameterization is to give retention and writing separate gates. In Naju, the forget gate f_n = σ(a_n) is the actual discrete pole of a diagonal state-space recurrence, so each frozen coordinate is Schur-stable by construction, and under uniformly bounded gate logits the time-varying recurrence satisfies a fading-memory/BIBO bound without any stability regularizer. The independent input gate i_n controls the write amplitude, so the model can set f_n near one to protect stored bindings and i_n

What carries the argument

The central mechanism is the affine diagonal recurrence x_n = f_n ⊙ x_{n-1} + i_n ⊙ (B_n h_n). Here f_n = σ(a_n) is the learned discrete pole that sets the retention time scale, i_n = σ(b_n) is an independent write gain, and the token-dependent vectors B_n and C_n define where in state space a write lands and how the state is read out. The memory kernel from a write at position m to an output at position n factors as K_{n,m} = α R_{m→n} i_m C_n^T B_m h_m with survival product R_{m→n} = ∏_{j=m+1}^n f_j, making the separation explicit. The load-bearing structural identity is the complementary-gate inequality |r| + w ≤ 1 for non-expansive single-gate recurrences, which forces near-unit retentio

Load-bearing premise

The claim stands or falls on whether the two synthetic probes—scattered recall for retention and recency-only tracking for overwriting—capture the real memory demands of long-context applications, and on whether the baseline comparisons, including the near-chance results of the attention model, are fair.

What would settle it

Train Naju with the input gate tied to the forget gate as i_n = 1 - f_n, keeping all other components identical; if this variant still reaches roughly 0.99 on T2 and 0.89 on T4 at length 2048, then the independent-gate decoupling is not responsible for the paper's headline result. Alternatively, any coupled-gate or attention-based model that achieves both scores under the same shared protocol would falsify the claimed necessity of independent gates.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single fixed-size recurrent state can preserve bindings over long distances and replace stale ones, removing the need to choose between retention-oriented and overwriting-oriented architectures for long-sequence memory.
  • Complementary single-gate recurrences carry an inherent limit: any configuration approaching lossless retention must attenuate new writes, so such models cannot match decoupled designs on both axes at matched state budgets.
  • Direct discrete-time parameterization makes stability a byproduct: with sigmoid poles and uniformly bounded logits, the time-varying recurrence is BIBO stable with geometrically fading memory, requiring no stability regularizer.
  • The recurrence remains associative-scan-compatible, so the memory gains come at linear time and linear memory in sequence length rather than quadratic attention cost.
  • Competitive results on language modeling, Long Range Arena, and multi-query associative recall indicate that the retention/overwrite decoupling is useful beyond synthetic probes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A sharp test of the paper's causality would tie the input gate to the forget gate inside Naju, e.g. i_n = 1 - f_n; if that variant still reaches roughly 0.99/0.89 at length 2048, the independent-gate decoupling is not what drives the result.
  • The paper's mechanism suggests a natural next step: content-addressed erasure that selectively forgets only the entries for a particular entity. The current channel-wise input gate cannot do selective key-based erasure, so such an extension should further improve recency-only overwriting in the far-distance regime.
  • If the synthetic T2/T4 tasks are a fair proxy for practical workloads, the architectural lesson transfers directly to dialogue systems, code models, and telemetry processing with fixed memory budgets: prefer independent retention and write gains over complementary gating. This transfer is an extrapolation the paper motivates but does not itself prove on real workloads.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Naju, a discrete-time state-space layer whose recurrent update is x_n = f_n ⊙ x_{n-1} + i_n ⊙ (B_n h_n), with a sigmoid forget gate serving directly as the discrete pole and a separately parameterized input gate controlling the write gain. The authors argue that this decoupling removes the complementary-gate constraint |r|+w ≤ 1, give a local pole/memory-kernel analysis, prove a fading-memory/BIBO bound under uniform boundedness assumptions, and provide a stability argument requiring no extra regularizer. Empirically, they report that on a synthetic diagnostic suite (T1–T4), Naju is the only evaluated model that remains strong on both long-range retention (T2: 0.99) and recency-based overwriting (T4: 0.89) at 4× the training length. Additional experiments on MQAR, Long Range Arena, and WikiText-103 are presented to support transfer beyond the diagnostics, along with throughput and memory measurements.

Significance. If the central causal claim is correct, the paper offers a simple and potentially useful design principle for recurrent state-space models: give retention and writing separate gates so that a fixed-size state can both preserve and overwrite bindings. The manuscript is generally careful: it shares training protocols, reports seed-level variability, includes a full factorial robustness check for Mamba, gives a clean parameter-free readout normalization (α=1/√d_state), and provides a correct, if standard, BIBO/stability proof in Appendix A. The external evaluations on WikiText-103, MQAR, and LRA are useful reality checks beyond the synthetic tasks. The main weakness is that the paper's headline mechanistic claim — that decoupling f_n and i_n causes the joint T2/T4 performance — is not directly tested by any tied-gate or coupled-gate ablation, and some baseline results are difficult to interpret.

major comments (3)
  1. [§4.1, Eqs. (46)–(50), and §5.3/Table 4] The central claim is that independently parameterized f_n and i_n remove the complementary-gate constraint and thereby enable simultaneous retention and overwriting. But the experiments never vary the coupling itself. Naju is compared with baselines that differ in memory carrier, state width, normalization, scan implementation, and training recipe. xLSTM also has independent forget and input gates, yet reaches only 0.73 on T4, so the comparison does not isolate the proposed cause. The formal argument in §4.1 bounds a scalar per-step recurrence; it does not rule out a multi-channel or multi-layer network routing retention and overwriting into separate channels or time steps. I request a direct ablation, e.g., a Naju variant with i_n = 1 - f_n, or x_n = (1-g_n)⊙x_{n-1} + g_n⊙(B_n h_n), using the same hyperparameter-selection procedure. If such a coupled variant still achieves high T2 and T
  2. [§5.3, Table 4, and Appendix B (Tables 17–19)] The diagnostic Transformer baseline shows a pathological train/validation pattern: at the training length, train accuracy reaches ≈0.98–1.00 while validation accuracy remains at chance (≈0.13–0.14) on T1, T2, and T4, even after 150 epochs. For such simple associative-recall and state-tracking tasks, a Transformer should be able to learn the underlying rule; the observed memorization pattern is commonly a symptom of an implementation or data-generation issue (e.g., missing positional information, a label/evaluation mismatch, or a non-causal setup) rather than an inherent architectural limitation. The manuscript's own diagnosis as 'task-dependent generalization failure' does not exclude such a flaw. Since the Transformer rows appear in the main comparison table and the paper uses these results to motivate the diagnostic suite, the baseline should be checked and either fixed or reported wit
  3. [§6.1–6.2, Tables 7–9, compared with Table 4] The default hyperparameters b_f=5, b_i=−2, d_state=64 are selected by sweeping on the same T2/T4 length-2048 extrapolation metrics that constitute the headline result. This is a form of selection on the test metric: the reported 0.99/0.89 is the outcome of a configuration chosen to maximize those numbers. The factorial sweep in Table 5 is helpful, but it does not remove the selection problem. I request either a nested validation-based selection procedure, or an explicit statement that the Section 6 sweeps are exploratory and a presentation of the range of Naju's T2/T4 performance across the evaluated configurations, so the reader can judge how much of the advantage is attributable to the decoupled mechanism versus to the selected initialization.
minor comments (4)
  1. [§5.2/Table 4] It would clarify the experiment to state explicitly whether the diagnostic Transformer uses a positional encoding and whether it is causal or bidirectional encoder-style. The table reports 'KV cache O(L)', which is more typical of autoregressive inference, while the diagnostic setup reads the full sequence and classifies at the final EOS.
  2. [§7.3, Table 21] For the Mamba N=64 WikiText-103 runs, the learning rate used is 4×10⁻³, selected from the Mamba N=16 pilot, even though the N=64 pilot's lowest validation perplexity was at 8×10⁻³. This is a deliberate protocol choice, but the state-matched comparison would be more persuasive if the N=64-specific optimal learning rate were also reported or used, or if the sensitivity of the gap to this choice were quantified.
  3. [§4.1–4.2] The notation h_n B_n^⊤ in Eq. (14) and B_n h_n in the scalar analysis is slightly inconsistent. The dimensions are clear from Table 2, but a one-line clarification would help readers.
  4. [Figure 1] Figure 1 would benefit from error bars or shaded regions, since several of the key points (e.g., Mamba, xLSTM, Naju) are reported with non-negligible seed variance in the appendix.

Circularity Check

0 steps flagged

No significant circularity: the Naju recurrence and stability analysis are derived from first principles; the gate-bias initialization is tuned on the diagnostic suite, but external benchmarks give the central claim independent content.

full rationale

The derivation chain is not circular. Naju's core recurrence (Eq. 14) is defined independently of the diagnostic tasks, and Theorem 1 in Section 4.3 derives BIBO/fading-memory stability directly from the sigmoid parameterization f_n = sigma(a_n) with uniformly bounded logits, giving rho = sigma(A_max) < 1. This does not assume the retention/overwriting accuracies it is later used to explain. The complementary-gate bound |r| + w <= 1 in Eqs. (46)-(50) is an algebraic identity for the specific scalar recurrence x_n = (1-g_n)lambda_n x_{n-1} + g_n v_n, and the claim that separate f_n and i_n remove that constraint is a parameterization comparison, not a circular definition. The Mamba 'gated-RNN limit' comparison cites Mamba's own Theorem 1, which is external work, not a self-citation chain. The only borderline issue is hyperparameter selection on the evaluation target: Section 5.2 states that b_f = 5 and b_i = -2 are 'the two knobs swept in Section 6', and Section 6.1/6.2 selects the default by T2/T4 extrapolation accuracy, so Table 4's headline 0.99/0.89 result is a tuned configuration rather than a blind out-of-sample prediction. This is a legitimate experimental-design concern, but it is not circular in the definitional sense: the chosen initializers do not by construction force those accuracies, and the same configuration is subsequently evaluated on independent benchmarks (WikiText-103, LRA, MQAR), which provide external evidence that Naju's behavior is not solely an artifact of the diagnostic-suite tuning. The Transformer baseline's train/val gap is a possible implementation or generalization issue, but it is not a circularity argument.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim relies on the synthetic tasks being representative of real memory workloads, on the applicability of the coupling inequality to the baselines, and on uniform-boundedness assumptions for the stability proof. The gate-bias initializations and state size are fitted to the diagnostic evaluation. No new physical entities or external objects are introduced.

free parameters (4)
  • forget-gate bias initialization b_f = 5
    Chosen by sweeping b_f ∈ {4,5,6} on T4@2048 in Section 6.1; directly controls the initial retention time scale and affects the headline retention/overwrite balance.
  • input-gate bias initialization b_i = -2
    Chosen by sweeping b_i ∈ {-1,-2,-3,-4} on T4@2048 in Section 6.1; controls initial write strength and affects overwrite accuracy.
  • feedthrough initialization D_init = 0.01
    Set so the direct route starts nearly silent and the recurrent memory path dominates early training (Section 5.2). Not tuned, but a hand-chosen constant that affects optimization trajectory.
  • state size d_state = 64
    Selected based on sensitivity analysis (Table 9) and parameter budget trade-offs; larger states improve T4 but increase compute.
axioms (4)
  • domain assumption The synthetic diagnostic suite T2/T4 accurately captures the retention and overwriting demands of real long-sequence workloads.
    The central empirical claim is defined by performance on T2 and T4 (Section 5.1), and the paper generalizes from these tasks to applications like dialogue and code modeling without independent validation of task representativeness.
  • domain assumption The complementary-gate coupling inequality |r|+w≤1 applies to the compared baselines in their relevant form.
    The inequality is proved for the specific complementary-gate recurrence (Eq. 46–48) and for the scalar gated-RNN limit of Mamba (Section 4.1). It is not shown to characterize the full behavior of Mamba, Mamba-2, or GLA in general.
  • domain assumption Forget logits and inputs are uniformly bounded (a_n ≤ A_max, |v_n| bounded) for Theorem 1.
    The fading-memory/BIBO bound requires a uniform upper bound on forget logits and bounded write drives; the model does not enforce this, so the theorem holds only under an unenforced regularity assumption.
  • domain assumption Standard deep-learning assumptions for training (AdamW, cosine schedule, etc.) are sufficient for the empirical comparisons.
    The comparisons assume the shared training protocol is appropriate for all baselines; if some baselines need different optimization (e.g., the Transformer might need positional encoding), the results could be misleading.

pith-pipeline@v1.3.0-alltime-deepseek · 36241 in / 10607 out tokens · 119391 ms · 2026-08-01T08:43:31.932756+00:00 · methodology

0 comments
read the original abstract

Long-sequence memory tracking places two opposing demands on a recurrent state: near-lossless retention of stored bindings over long horizons, and active overwriting of stale ones. In our diagnostic suite, the strongest efficient baselines tend to solve only one side well. Continuous-time-parameterized state-space models (SSMs) such as Mamba obtain their discrete recurrence by zero-order-hold discretization of a continuous-time system; we argue that this detour is unnecessary for memory tracking and parameterize the discrete transition directly. Naju (Native Adaptive Junction Unit) factorizes the recurrent update, schematically $x_n = f_n\odot x_{n-1} + i_n\odot(B_n u_n)$, into an explicit discrete pole (a learned forget gate $f_n$), an independent write gain $i_n$, and input-dependent write/read maps. Since the sigmoid pole satisfies $0<f_n<1$, each frozen local coordinate is Schur-stable by construction, and the full time-varying recurrence satisfies a fading-memory/BIBO bound under uniform boundedness assumptions, with no stability regularizer. We formalize the key structural limitation of coupled designs: any non-expansive complementary single-gate recurrence ties the effective retention $r$ and write gain $w$ through $|r|+w\le 1$, so near-complete retention forces weak writing; decoupling $f_n$ from $i_n$ removes this constraint. Empirically, Naju is the only evaluated model that remains strong on both retention and overwriting at 4x the training length. Beyond the diagnostic suite, we evaluate Naju on WikiText-103 language modeling, Long Range Arena, and multi-query associative recall. Across these settings, Naju consistently combines strong long-range memory with competitive or superior performance, outperforming the Mamba baselines in the principal comparisons while remaining competitive with the Transformer and preserving linear-time, linear-memory scaling.

Figures

Figures reproduced from arXiv: 2607.21000 by Hyuk Lim, Seunghyun Yoon.

Figure 1
Figure 1. Figure 1: Retention and overwriting at length 2048. Extrapolation accuracy on retention (T2, scat￾tered key–value recall; x-axis) and overwriting (T4, hard state tracking; y-axis) under the shared training protocol and the stated model configurations. Among the tested baselines, xLSTM provides strong retention but weaker overwriting, whereas GLA provides strong overwriting but weak long-distance retention. Mamba and… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of the recurrent junctions in LSTM, Mamba, and Naju. Each recurrence contains [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Naju architecture. The full model (top) stacks M Naju blocks between a token embedding and a final prediction head; one block is expanded below. The block input u (l) n is projected into a content branch h (l) n and an output-modulation branch z (l) n . The content branch generates the token-dependent retain/write gates f (l) n and i (l) n and the selective write/read vectors B (l) n and C (l) n , which de… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 18 linked inside Pith

  1. [1]

    Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher R´ e. Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023. URLhttps://arxiv.org/abs/2312.04927

    Pith/arXiv arXiv 2023

  2. [2]

    Simple linear attention language models balance the recall- throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher R´ e. Simple linear attention language models balance the recall- throughput tradeoff. InInternational Conference on Machine Learning (ICML), 2024. URLhttps: //arxiv.org/abs/2402.18668

    Pith/arXiv arXiv 2024

  3. [3]

    xLSTM: Extended long short- term memory.arXiv preprint arXiv:2405.04517, 2024

    Maximilian Beck, Korbinian P¨ oppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, G¨ unter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xLSTM: Extended long short- term memory.arXiv preprint arXiv:2405.04517, 2024. URLhttps://arxiv.org/abs/2405.04517

    Pith/arXiv arXiv 2024

  4. [4]

    Smith, Anushan Fernando, George-Cristian Muraru, et al

    Aleksandar Botev, Soham De, Samuel L. Smith, Anushan Fernando, George-Cristian Muraru, et al. RecurrentGemma: Moving past transformers for efficient open language models.arXiv preprint arXiv:2404.07839, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. InInternational Conference on Machine Learning (ICML), 2024. URL https://arxiv.org/abs/2405.21060

    Pith/arXiv arXiv 2024

  6. [6]

    Language modeling with gated convolutional networks

    Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. InInternational Conference on Machine Learning (ICML), 2017. URLhttps: //arxiv.org/abs/1612.08083

    Pith/arXiv arXiv 2017

  7. [7]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models.arXiv preprint arXiv:2402.19427, 2024

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, Guillaume Desjardins, Arnaud Doucet, David Budden, Yee Whye Teh, Razvan Pascanu, Nando De Freitas, and Caglar Gulcehre. Griffin: Mixing gated linear recurrences with local attention for efficient...

    Pith/arXiv arXiv 2024

  8. [8]

    Fu, Tri Dao, Khaled K

    Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher R´ e. Hungry Hungry Hippos: Towards language modeling with state space models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  9. [9]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023. URLhttps://arxiv.org/abs/2312.00752

    Pith/arXiv arXiv 2023

  10. [10]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher R´ e. Efficiently modeling long sequences with structured state spaces. InInternational Conference on Learning Representations (ICLR), 2022. URLhttps: //arxiv.org/abs/2111.00396

    Pith/arXiv arXiv 2022

  11. [11]

    Jamba: A hybrid transformer-mamba language model.arXiv preprint arXiv:2403.19887, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Erez Shwartz, Mor Zusman, and Yoav Shoham. Jamba: A hybrid transformer-mamba langua...

    Pith/arXiv arXiv 2024

  12. [12]

    Transformers learn shortcuts to automata

    Bingbin Liu, Jordan T Ash, Surbhi Goel, Akshay Krishnamurthy, and Cyril Zhang. Transformers learn shortcuts to automata. InInternational Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2210.10749

    Pith/arXiv arXiv 2023

  13. [13]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. InInternational Conference on Learning Representations, 2017. URLhttps://arxiv.org/abs/1609. 07843. 31

    2017

  14. [14]

    Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De

    Antonio Orvieto, Samuel L. Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. InInternational Conference on Machine Learning (ICML), 2023

    2023

  15. [15]

    Wind, Stanislaw Wozniak, Ruichong Zhang, Zhenyuan Zhang, Qihang Zhao, Peng Zhou, Jian Zhu, and Rui-Jie Zhu

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Ji- aju Lin, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang...

    Pith/arXiv arXiv 2023

  16. [16]

    Eagle and finch: R WKV with matrix-valued states and dynamic recurrence.arXiv preprint arXiv:2404.05892, 2024

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, Kranthi Kiran GV, Jan Kocon, Bartlomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Niklas Muennighoff, Fares Obeid, Atsushi Saito, Guangyu Song, Haoqin Tu, Stanislaw Wozniak, Ruichong...

    Pith/arXiv arXiv 2024

  17. [17]

    Hierarchically gated recurrent neural network for sequence modeling

    Zhen Qin, Songlin Yang, and Yiran Zhong. Hierarchically gated recurrent neural network for sequence modeling. InAdvances in Neural Information Processing Systems (NeurIPS), 2023. URLhttps: //arxiv.org/abs/2311.04823

    Pith/arXiv arXiv 2023

  18. [18]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8), 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.OpenAI blog, 1(8), 2019

    2019

  19. [19]

    GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020

    Noam Shazeer. GLU variants improve transformer.arXiv preprint arXiv:2002.05202, 2020. URL https://arxiv.org/abs/2002.05202

    Pith/arXiv arXiv 2002

  20. [20]

    RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568, 2024

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding.Neurocomputing, 568, 2024. URLhttps://doi.org/10. 1016/j.neucom.2023.127063

    arXiv 2024

  21. [21]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023. URLhttps://arxiv.org/abs/2307.08621

    Pith/arXiv arXiv 2023

  22. [22]

    Long range arena: A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. InInternational Conference on Learning Representations (ICLR), 2021. URLhttps://arxiv.org/abs/ 2011.04006

    Pith/arXiv arXiv 2021

  23. [23]

    Gated linear atten- tion transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear atten- tion transformers with hardware-efficient training. InInternational Conference on Machine Learning (ICML), 2024. URLhttps://arxiv.org/abs/2312.06635. A Proof of the Fading-Memory and BIBO Bounds We prove the bound for one expanded channel with ad state-dimensional st...

    Pith/arXiv arXiv 2024