Pith. sign in

REVIEW 3 major objections 6 minor 31 references

Logarithmic Memory Networks (LMNs): Efficient Long-Range Sequence Modeling for Resource-Constrained Environments

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Log-tree memory cuts attention cost from O(n^2) to O(log n)

desk verdict The O(log n) claim is false on the paper's own tensor shapes; the architecture is a plausible subquadratic hybrid, but the headline doesn't hold and the evidence is too thin. read the letter →

arxiv 2501.07905 v1 pith:QBXGPFTM submitted 2025-01-14 cs.AI cs.LG

Mohamed A. Taha This is my paper · ORCID
classification cs.AIcs.LG
keywords logarithmicmemorynetworkshierarchicaltreesingle-vectorattentionlong-rangesequencemodelingresource-constrainedenvironmentscomputationalcomplexitypositionalencoding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Logarithmic Memory Networks (LMNs), a sequence model that replaces the quadratic self-attention of Transformers with attention over a hierarchical tree of memory summaries. The claim is that storing past context in a logarithmic tree and reading it with a single-vector attention operation reduces computational complexity and memory footprint from O($n^{2}$) to O(log n), while staying competitive with GPT-2 on language modeling. The significance, if the claim holds, is that long-range sequence modeling could run on mobile and edge devices where quadratic attention is infeasible. The model also removes explicit positional encodings by letting tree paths carry position information.

What carries the argument

The central machinery is the logarithmic tree memory plus single-vector attention. The summarizer layer defines the tree: for each pair of nodes at level i, it concatenates the two vectors of dimension E and applies a linear projection back to dimension E, so every parent node is a trained compression of its two children. Single-vector attention reads only the first vector of K (the current token) against every level of the memory, yielding log(L) scores per token, and then weights the value vectors by those scores. A path-through positional encoding emerges because a token's route up the tree—or its binary position in sequential construction—is implicitly stored in the summaries. The expander summarizer optionally adds slots at deeper levels via a 1D transposed convolution to mitigate information loss at high compression.

What would settle it

Train LMNs on a task that requires exact recall of a specific earlier token (e.g., a copy task on a sequence of length 2^k), then probe the memory tree at the highest level with a linear classifier; if the classifier cannot recover which token occupied a given position, the linear summarizer has destroyed information the attention mechanism cannot restore. Alternatively, benchmark memory growth as sequence length doubles: if the tensor is [B, L, log(L), E], measured peak memory will grow faster than O(log L), contradicting the headline claim.

Watch

Extended reading notes

Core claim

Logarithmic Memory Networks are a hierarchical memory architecture. The lowest level of the tree is the token sequence; each higher level is produced by a summarizer layer that takes two adjacent nodes and linearly projects their concatenation into a single node. The resulting memory tensor has shape [B, L, log(L), E]. Attention is then computed only against the first vector of the key tensor—the current token—producing scores of shape [B, L, log(L)], which the paper argues gives O(log n) attention cost. The memory construction runs in parallel during training and sequentially during inference, where the sequential mode acts as a recurrent memory with a small footprint. The paper reports that on the Tiny Shakespeare benchmark, LMNs with roughly GPT-2's parameter count yield lower training and validation loss than GPT-2, and that inference time and memory usage stay flat as sequence length grows, whereas standard attention degrades.

Load-bearing premise

The architecture's usefulness rests on the premise that a single linear projection of two concatenated node vectors can preserve enough task-relevant information at every tree level, so that the hierarchical memory is a faithful summary rather than an uncontrolled compression; the paper offers no bound, ablation, or experiment that directly tests this compression fidelity.

Editorial extensions

If this is right

  • Sequences of tens of thousands of tokens become processable on memory-limited hardware, since the per-token memory and compute grow only with log(L) in the paper's accounting.
  • Positional encodings become unnecessary, simplifying the architecture and removing an extra hyperparameter.
  • The dual-mode execution allows GPU-parallel training while keeping inference as a lightweight recurrent-style memory update.
  • At matched parameter counts, LMNs report better training and validation loss than GPT-2 on the tested small-scale language modeling task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's O(log n) complexity claim does not follow from its own definitions: the memory tensor is [B, L, log(L), E], so storage is O(n log n), and computing scores requires O(n log n) operations; an honest accounting would compare O(n log n) attention against quadratic attention, which is still a large improvement but not the claimed complexity.
  • The summarizer's linear projection from 2E to E is the capacity bottleneck: if two distinct pairs of nodes can map to the same vector, then the memory cannot distinguish certain histories, and attention has no way to recover the lost distinction. A copy-recall probe would reveal how many bits of an original token survive after k levels of summarization.
  • The dual-mode equivalence between parallel and sequential construction is asserted but not proven; checking that both modes produce identical memories for the same input would test whether inference really matches training behavior.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Logarithmic Memory Networks (LMNs), a sequence model that stores past context in a hierarchical tree built by repeatedly summarizing pairs of adjacent node embeddings with a linear projection. It introduces a single-vector attention mechanism that scores the log L memory levels of each position, plus parallel (training) and sequential (inference) construction modes, optional multi-bank memory, and an expander variant. The advertised contribution is reducing attention memory and computation from O(n²) to O(log n) while remaining competitive with GPT-2; the experiments compare loss on Tiny Shakespeare and benchmark inference time and memory against standard attention.

Significance. The central promise—a logarithmic-complexity attention mechanism with competitive accuracy—would be practically significant for resource-constrained long-sequence modeling, and the accompanying MIT-licensed code is a concrete asset. However, the paper's own architectural definitions imply O(n log n) storage and attention cost rather than O(log n), so the headline result is not established. The empirical evaluation is a single run on one small dataset with no error bars, and the quality claims on retrieval and compression rest on unverified assumptions. The architecture is simple and falsifiable, but the paper as written does not substantiate scalable logarithmic behavior.

major comments (3)
  1. [§3.3.1, §3.4, §5.2] The advertised O(log n) complexity is contradicted by the paper's own definitions. Parallel memory construction returns a memory tensor of shape [B, L, log(L), E], so storing the memory is Θ(n log n) entries, not Θ(log n). Single-vector attention computes scores of shape [B, L, log(L)] and then a weighted sum over the log(L) levels, so the attention cost per layer is Θ(n log n) operations. The expander variant in §3.7 is stated as O(k/2 · log² n), and §5.2 itself concedes 'the complexity should ideally be O(log(n)^2).' The argument that focusing on the original token removes one log factor does not remove the L factor from processing every sequence position. The correct characterization is subquadratic (n log n, or n log² n with the expander), not logarithmic, and this affects the validity of the compression ratios reported in §5.2.
  2. [§3.3, §3.4, §6] The retrieval mechanism depends on two unexamined assumptions: that a single linear projection of two concatenated E-dimensional nodes is a sufficiently lossless summary for all future retrievals, and that the first vector of K at each position is an adequate query for all log(L) memory levels. The first assumption is load-bearing because every higher tree level is a further compression of an already lossy summary; the paper offers no information-theoretic bound, no ablation varying the summarizer, and no experiment that isolates compression loss. The second assumption is load-bearing for the 'single-vector attention' claim; without an ablation or analysis, the model's capacity to retrieve long-range dependencies is not established. Section 3.4 also does not specify how the single-vector attention performs cross-position retrieval, since the score tensor has shape [B, L, log(L)] rather than [B, L, L].
  3. [§5.1, §5.3] The empirical support is too thin for the paper's conclusions. Table 1 reports one training run per configuration, with no seeds, no error bars, and a single dataset (Tiny Shakespeare) trained for only 5,000 steps; statements such as 'LMNs ... outperform it' in §5.1 are not justified at this scale, especially since parameter counts are not closely matched (LogMem has 1,072,193 parameters versus GPT-2's 841,281 at embedding size 128). The inference-time and memory benchmarks in §5.3 do not report error bars, the number of repetitions, or the exact attention implementation being compared. The 'compression factor' arithmetic in §5.2 is a restatement of the claimed asymptotic complexity, not a measured overhead reduction, so it cannot compensate for the missing benchmarking details.
minor comments (6)
  1. [§5.2] Equations (1) and (2) are dimensionally confusing: the values are ratios, not percentages, and for n=1024 the ratio is 104,857.6, not 1,048,576%; please state the exact formula and use percentage notation correctly.
  2. [§3.7, §5.2] The notation log2(n) in §3.7 and log(n)^2 in §5.2 is ambiguous: specify whether this means (log n)² or log to base 2, and state whether the expansion factor k and bank count are treated as constants or as terms in the asymptotic expression.
  3. [§3.4] The bullet 'Reduce Parameters' is not part of the attention mechanism and interrupts the description; it should be moved to the experimental or architecture-parameter sections. The claim that it 'outperforms GPT-2' with half the parameters is not supported by Table 1 as presented.
  4. [References] Several references are misattributed or appear unverifiable. For example, [20] is listed as 'Mamba: A memory-augmented model for sequence-to-sequence learning,' but the known Mamba paper is titled 'Mamba: Linear-Time Sequence Modeling with Selective State Spaces'; [3], [4], [21], [22], and [24]–[28] also need verification against actual published venues.
  5. [§3.3.1, §3.4] The paper states the tree depth as log(L) without specifying the base or the handling of non-power-of-two sequence lengths, and the phrase 'the first vector of K which is the current token' is ambiguous in parallel mode where all positions are processed simultaneously; please clarify the indexing and the causal masking applied to the scores.
  6. [§1] There is a grammatical slip in the phrase 'A Logarithmic Memory Networks is proposed'; it should be 'Logarithmic Memory Networks are proposed.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's O(log n) complexity claim is contradicted by its own tensor shapes, but that is an arithmetic/consistency error, not a derivation that reduces to its inputs.

full rationale

I walked the claimed derivation chain. The abstract and Section 5.2 assert a reduction from O(n^2) to O(log(n)), while Section 3.3.1 defines the parallel memory tensor as [B, L, log(L), E] and Section 3.4 defines the score tensor as [B, L, log(L)]. These shapes imply O(n log n) storage and per-layer attention cost, so the advertised O(log n) bound is not entailed by the architecture. Section 5.2 even concedes 'the complexity should ideally be O(log(n)^2)' before dropping a factor on the grounds that 'single-vector attention only needs to focus on the original token'; that reasoning is a non sequitur because the L factor from processing every position remains. However, this is not circularity: no quantity is fitted to a target result, no load-bearing claim is imported from self-citation, and no definition is constructed in terms of the conclusion. The paper contains no self-citations, and the loss comparisons against GPT-2 are empirical measurements of a fixed architecture rather than predictions derived from fitted parameters. The main issues—overclaimed complexity and the unproven compression fidelity of the summarizer—are correctness and validation risks, not circular derivation. Therefore, the score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 3 invented entities

The paper's central claims rest on the unproven fidelity of a linear summarizer, the equivalence of parallel and sequential modes, and the sufficiency of single-vector attention; none are tested in isolation. The complexity claim also relies on treating O(n log n) operations and O(n log n) memory as O(log n).

free parameters (2)
  • memory bank count = 2 in Table 1 (can be set to 1)
    Number of parallel summarizer trees is chosen by hand; it appears in the complexity formula O(k/2 * bank * log^2(n)).
  • expander expansion factor k = default 1
    Controls extra memory slots in the Expander Summarizer; set by hand and enters the complexity formula.
assumptions (4)
  • standard math A complete binary tree built by pairing adjacent nodes has depth O(log n).
    Used implicitly throughout Section 3.3 to justify the logarithmic number of memory levels.
  • ad hoc to paper A single linear projection of the concatenation of two node vectors is a sufficient summary of both nodes for future retrieval.
    Section 3.3 states the summarizer outputs a new node that contains the combined information, but no bound or test is given for information loss.
  • ad hoc to paper The first vector of K, corresponding to the current token, is sufficient as a query key for all memory levels.
    Section 3.4 defines attention scores using only the first vector from K, without a derivation that this key retains the information needed to select the right summary.
  • domain assumption Parallel and sequential memory construction produce equivalent representations.
    Section 3.2 says both construct a similar tree-like memory, which is necessary for the train-parallel/infer-sequential recipe, but no equivalence proof or measurement is provided.
invented entities (3)
  • Hierarchical logarithmic memory tree
    purpose: Store compressed summaries of past tokens at multiple levels.
    The core representation of the paper; no falsifiable prediction outside the paper validates its utility.
  • Single-vector attention
    purpose: Retrieve one summary per level using only the current token key.
    Attention variant introduced here; its sufficiency is asserted, not externally validated.
  • Path-through positional encoding
    purpose: Implicitly encode relative position via tree paths instead of explicit position embeddings.
    The paper claims the tree path encodes position, but provides no experiment isolating position sensitivity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Logarithmic Memory Networks (LMNs): Efficient Long-Range Sequence Modeling for Resource-Constrained Environments." pith.science (2026). https://pith.science/paper/QBXGPFTM

@misc{pith2026250107905,
  author       = {Pith},
  title        = {Pith review of: Logarithmic Memory Networks (LMNs): Efficient Long-Range Sequence Modeling for Resource-Constrained Environments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QBXGPFTM}},
  note         = {Machine review of arXiv:2501.07905}
}
read the original abstract

Long-range sequence modeling is a crucial aspect of natural language processing and time series analysis. However, traditional models like Recurrent Neural Networks (RNNs) and Transformers suffer from computational and memory inefficiencies, especially when dealing with long sequences. This paper introduces Logarithmic Memory Networks (LMNs), a novel architecture that leverages a hierarchical logarithmic tree structure to efficiently store and retrieve past information. LMNs dynamically summarize historical context, significantly reducing the memory footprint and computational complexity of attention mechanisms from O(n2) to O(log(n)). The model employs a single-vector, targeted attention mechanism to access stored information, and the memory block construction worker (summarizer) layer operates in two modes: a parallel execution mode during training for efficient processing of hierarchical tree structures and a sequential execution mode during inference, which acts as a memory management system. It also implicitly encodes positional information, eliminating the need for explicit positional encodings. These features make LMNs a robust and scalable solution for processing long-range sequences in resource-constrained environments, offering practical improvements in efficiency and scalability. The code is publicly available under the MIT License on GitHub: https://github.com/AhmedBoin/LogarithmicMemory.

Figures

Figures reproduced from arXiv: 2501.07905 by the authors.

Figure 1
Figure 1. Logarithmic Memory and its capability to train as a transformers and inference as a [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Visualization of parallel memory construction. The nodes are summarized from the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Visualization of sequential memory construction. The pointer keeps track of the input [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Visualization of the Single Vector Attention mechanism. Q, K and V are generated [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Multi-Memory Banks Visu￾alization. 3.6 Path-Through Positional Encoding Unlike transformers, no explicit positional encoding is added in LMNs. Instead, positional information is implicitly encoded through the hierarchical tree structure. As shown in the figure 7, the s…
Figure 7
Figure 7. Figure 7: Illustration of relative positional encoding in LMNs. The hierarchical tree structure [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Architecture of the Expander Summarizer. The summarizer (top left) condenses [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Inference time comparison for Logarithmic Memory and Attention in Sequential and [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Comparison of memory usage between Logarithmic Memory and Attention. LMNs [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 19 canonical work pages

  1. [20]

    Learning Objective Functions Incrementally by Inverse Optimal Control

    S. Li et al., “Mamba: A memory-augmented model for sequence-to-sequence learning,” arXiv preprint arXiv:2010.15034, 2020

    work page Pith review arXiv 2010

  2. [1]

    Finding structure in time,

    J. L. Elman, “Finding structure in time,” Cognitive Science, vol. 14, no. 2, pp. 179–211, 1990

    work page 1990

  3. [2]

    Recurrent neural network based language model,

    T. Mikolov, M. Karafi´ at, L. Burget, J. Cernock` y, and S. Khudanpur, “Recurrent neural network based language model,” Interspeech, vol. 2, pp. 1045–1048, 2010

    work page 2010

  4. [3]

    Rnn++: A lightweight, optimized recurrent neural network for long sequences,

    H. Tran and K. Zhou, “Rnn++: A lightweight, optimized recurrent neural network for long sequences,” NeurIPS, 2021

    work page 2021

  5. [4]

    Neural memory architectures for sequential decision-making,

    R. Dutta et al., “Neural memory architectures for sequential decision-making,” ICLR, 2023

    work page 2023

  6. [5]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  7. [6]

    Neural machine translation by jointly learning to align and translate,

    D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” International Conference on Learning Representations (ICLR), 2015

    work page 2015

  8. [7]

    Learning phrase representations using rnn encoder-decoder for statistical ma- chine translation,

    K. Cho, B. Van Merri¨ enboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical ma- chine translation,” arXiv preprint arXiv:1406.1078, 2014

    arXiv 2014

Show all 31 references
  1. [8]

    Properties of the lstm and gru networks,

    K. Cho and Y. Bengio, “Properties of the lstm and gru networks,” arXiv preprint arXiv:1412.3555, 2014

    2014 arXiv

  2. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017

    2017

  3. [10]

    Bert: Pre-training of deep bidirec- tional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirec- tional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  4. [11]

    Improving language under- standing by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language under- standing by generative pre-training,” OpenAI Blog, 2018

    2018

  5. [12]

    Transformer-xl: Attentive language models beyond a fixed-length context,

    Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-xl: Attentive language models beyond a fixed-length context,”arXiv preprint arXiv:1901.02860, 2019

    1901 arXiv

  6. [13]

    Longformer: The long-document transformer,

    M. E. Beltagy Iz Peters and C. Arman, “Longformer: The long-document transformer,” arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  7. [14]

    Linformer: Self- attention with linear complexity,

    S. Wang, N. Liu, W. Han, M. Yu, Y. Wang, Y. Yang, and K. Cho, “Linformer: Self- attention with linear complexity,” arXiv preprint arXiv:2006.04768, 2020. 17

    2006 arXiv

  8. [15]

    Neural turing machines,

    A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” arXiv preprint arXiv:1410.5401, 2014

    2014 arXiv

  9. [16]

    Memory networks,

    J. Weston, S. Chopra, and A. Bordes, “Memory networks,” arXiv preprint arXiv:1410.3916, 2014

    2014 arXiv

  10. [17]

    Hybrid computing using a neural network with dynamic external memory,

    A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska-Barwi´ nska, S. G. Colmenarejo, J. ´Swiatkowski, D. Tan, S. Mohamed et al., “Hybrid computing using a neural network with dynamic external memory,” Nature, vol. 538, no. 7626, pp. 471–476, 2016

    2016

  11. [18]

    Hippo: Recurrent memory with optimal polynomial projections,

    A. Gu, T. Dao, S. Ermon, A. Rudra, and C. R´ e, “Hippo: Recurrent memory with optimal polynomial projections,” Advances in Neural Information Processing Systems (NeurIPS), 2020

    2020

  12. [19]

    Efficiently modeling long sequences with structured state spaces,

    A. Gu, K. Goel, and C. R´ e, “Efficiently modeling long sequences with structured state spaces,” arXiv preprint arXiv:2111.00396, 2021

    2021 arXiv

  13. [21]

    Hybrid memory networks for sequential data processing,

    M. Tan and S. Liu, “Hybrid memory networks for sequential data processing,” ICML, 2022

    2022

  14. [22]

    Learnable memory allocation for efficient sequence modeling,

    K. Liu and Y. Huang, “Learnable memory allocation for efficient sequence modeling,” arXiv preprint arXiv:2401.01023, 2024

    2024 arXiv

  15. [23]

    Synthformer: Beyond token-level self-attention,

    Y. Tay, M. Dehghani et al., “Synthformer: Beyond token-level self-attention,” NeurIPS, 2022

    2022

  16. [24]

    Recurrent attention mechanisms for efficient long-sequence process- ing,

    H. Liu and H. Rao, “Recurrent attention mechanisms for efficient long-sequence process- ing,” ICML, 2023

    2023

  17. [25]

    Dynamic memory transformers for efficient long-sequence model- ing,

    Z. Chen and T. Wang, “Dynamic memory transformers for efficient long-sequence model- ing,” ACL, 2021

    2021

  18. [26]

    Coformer: Collaborative memory transformers for efficient long-range processing,

    R. Xu and H. Li, “Coformer: Collaborative memory transformers for efficient long-range processing,” arXiv preprint arXiv:2302.01931, 2023

    2023 arXiv

  19. [27]

    Scaling structured state space models for long sequences,

    A. Gu and T. Dao, “Scaling structured state space models for long sequences,” NeurIPS, 2022

    2022

  20. [28]

    Compact state space models for efficient long-term dependencies,

    Y. Wang et al., “Compact state space models for efficient long-term dependencies,” ICLR, 2024

    2024

  21. [29]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,

    N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” arXiv preprint arXiv:1701.06538, 2017. [Online]. Available: https://arxiv.org/abs/1701.06538

    2017 arXiv

  22. [30]

    Switch transformers: Scaling models with mixture- of-experts layers,

    W. Fedus, B. Zoph, and N. Shazeer, “Switch transformers: Scaling models with mixture- of-experts layers,” Nature Machine Intelligence, 2022

    2022

  23. [31]

    Modular mixture of experts for efficient sequence modeling,

    E. Kim and J. Park, “Modular mixture of experts for efficient sequence modeling,” ICML, 2023. 18

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.