TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms

Zixi Li

REVIEW 5 major objections 5 minor 11 references

TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An attention-free network with 3.16M parameters claims 99% accuracy on the ARC-AGI-2 reasoning benchmark.

desk verdict The headline result—99% on ARC-AGI-2 with 3.16M parameters—is unsupported by the paper's own inconsistent numbers and missing evaluation protocol; this should be desk-rejected. read the letter →

arxiv 2509.05550 v2 pith:Z73VUA34 submitted 2025-09-06 cs.AI

Zixi Li This is my paper · ORCID

classification cs.AI

keywords TreeFFNattention-freearchitectureARC-AGI-2structuredreasoningencoder-decoderadjacentconnectionsparameterefficiencysequencemodeling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a purely attention-free architecture, TreeGPT, built from bidirectional TreeFFN components that only pass information between neighboring sequence positions, can solve ARC-AGI-2 visual reasoning puzzles to 99% validation accuracy. The model uses about 3.16 million parameters and converges in 1,500 training steps, which would put it far ahead of billion-parameter transformers and specialized program-synthesis systems on the same benchmark. If correct, the result would show that attention is not needed for structured reasoning, and that a simple adjacent-connection inductive bias can outperform scale. The paper presents these findings as preliminary and calls for broader evaluation across diverse tasks.

What carries the argument

The key machinery is the pure TreeFFN encoder-decoder. The encoder applies a TreeFFN over adjacent edges $E_{\text{enc}}=\{(i,i+1)\}$ to propagate left-to-right dependencies, while the decoder applies a TreeFFN over reversed edges $E_{\text{dec}}=\{(i,i-1)\}$ for right-to-left pattern generation; the two runs happen in parallel and are combined through additive residuals. Optional edge projection (linearly transforming edge features before message passing) and gated aggregation are included to control information flow. Ablation results single out edge projection as the load-bearing component: without it the model scores 0%, and with it (plus gating) test accuracy reaches 96%.

What would settle it

Download the released implementation and run it on the official ARC-AGI-2 validation set using the official evaluation harness, scoring output grids by exact match to the expected grids. If exact-match accuracy is substantially below 99%, or if the reported accuracy only holds on a hand-picked subset, the central claim collapses; additionally, checking for overlap between training tasks and validation tasks would settle whether the model generalizes or memorizes.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an attention-free encoder-decoder built from pure TreeFFN layers—where the encoder sweeps left-to-right along edges $(i,i+1)$ and the decoder sweeps right-to-left along $(i,i-1)$, both in parallel—can solve the ARC-AGI-2 reasoning benchmark with 99% validation accuracy and 100% token-level accuracy on selected evaluation samples. The authors attribute the result to the combination of bidirectional neighbor-to-neighbor propagation and edge projection, which their ablation identifies as the critical component; configurations without edge projection collapse to 0% accuracy. They report convergence within 1500 training steps using a 3.16M-parameter model, and argue this indicates that attention mechanisms, while powerful, are not necessary for certain structured reasoning tasks.

Load-bearing premise

The central claim depends on the unstated assumption that the ARC-AGI-2 evaluation was run under the official benchmark protocol—including how grid puzzles are tokenized, what the train/validation split is, what 'validation accuracy' counts at task or token level, and how the 'selected evaluation samples' were chosen—so that the 99% figure is directly comparable to the leaderboard baselines in Table 2.

Editorial extensions

If this is right

If the 99% figure holds, attention-free neighbor-connection models would outperform transformer-based and program-synthesis systems by a wide margin on ARC-AGI-2, implying attention is not necessary for this kind of structured reasoning.
A 3.16M-parameter model beating billion-parameter systems on this benchmark would make architectural inductive bias a stronger factor than scale for structured tasks.
Converging in 1,500 steps would indicate that such structured reasoning problems do not need the long optimization typically required by large attention models.
The bidirectional TreeFFN design could be transferred to other sequence tasks with strong local structure, such as grammar-based generation or grid-world navigation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's numbers are not internally consistent—the abstract reports 3.16M parameters and 99% validation accuracy, while several body sections state 1.5M parameters and 96% accuracy—so the abstract's 99% figure is the claim to verify.
A natural testable extension is applying the same two-pass neighbor-connection architecture to ARC-AGI-1 and to synthetic grid-rewriting tasks with known rules, which would show whether the result generalizes beyond the one benchmark.
Because each layer only exchanges information between adjacent positions, the architecture's effective receptive field grows linearly with depth; stress tests that require propagating information across many grid cells would delimit how far attention-free processing can go.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: the central claim is not credible as stated. A 3.16M-parameter attention-free model scoring 99% on ARC-AGI-2 would be a real result, but the paper never defines the evaluation protocol, and its own numbers contradict each other. The architecture itself is not new: Eqs. 4–5 describe bidirectional neighbor-to-neighbor message passing over a linear chain, which is a standard 1D-convolution/graph-message-passing pattern. No formal characterization is offered, and the 'Global Parent-Child Aggregation' named in the intro never appears in the method.

What the paper does well: it makes a genuine attempt at ablation, with a clear component hierarchy (edge projection critical, gating helps, residual can hurt), and it points to a public repository. That is more than many such submissions do. But the ablation table contradicts its own text: the table gives Gating Only 74% test accuracy, while the text says all configurations without edge projection fail completely (0%). Parameter counts flip between 3.16M and 1.5M, and validation accuracy (99%) is inconsistently reported against test accuracy (96%). The abstract's '100% token-level accuracy on selected evaluation samples' admits cherry-picking, yet Table 2 compares this against official leaderboard full accuracy.

Most importantly, the evaluation protocol is absent. There is no tokenization, no input embedding, no output decoding, no train/validation split, no loss definition. Section 4.1 calls the dataset 'grid-based visual puzzles' and stops. Without these details, the 99% figure cannot be checked or compared to the baselines in Table 2. The stress-test note gets this right: the headline comparison is unsupported because the metric is self-defined.

There are also smaller signs of carelessness: the CoT reference is attributed to Vaswani et al., and Section 1 introduces the model as a hybrid with attention before the rest of the paper claims pure attention-free processing. None of these are fatal in isolation, but together they indicate the paper has not been through a serious internal review.

Who would get value from this? Perhaps a reader interested in cataloguing failed or sloppy attempts at attention-free reasoning, but not someone looking for a reliable result. The idea itself—that structured reasoning might not need attention—is worth investigating, but this paper does not advance it. My recommendation: desk reject. There is no point sending referees to work through contradictions and missing protocol details that the authors should have resolved before submission.

Referee Report

5 major / 5 minor

Summary. The paper proposes TreeGPT, an attention-free encoder-decoder architecture built from bidirectional TreeFFN components that propagate information only through adjacent neighbor connections. The authors evaluate the model on the ARC-AGI-2 visual reasoning benchmark and report that it achieves 99% validation accuracy with 3.16M parameters, converges within 1500 training steps, and reaches 100% token-level accuracy on selected evaluation samples. They compare these numbers with the official ARC-AGI-2 leaderboard, claiming substantial improvements over large language models and program-synthesis baselines. The manuscript also includes an ablation study of edge projection, gating, and residual components, and concludes that eliminating attention can be beneficial for structured reasoning.

Significance. If the headline result were established under the official ARC-AGI-2 evaluation protocol, it would be highly significant: a 3.16M-parameter attention-free model outperforming billion-parameter reasoning systems would challenge core assumptions about the necessity of attention and about scaling for abstract reasoning. The paper also advertises a GitHub repository with implementation and training scripts, which would be a useful reproducibility contribution if the architecture and protocol were precisely specified. However, as presented, the empirical claims are not defensible because the evaluation protocol is undefined, the reported metrics are internally inconsistent, and the architecture itself is only partially specified. The paper therefore cannot currently support its central claims, and the manuscript reads as an early-stage report rather than a completed study.

major comments (5)

[Abstract; Sections 4.1 and 4.4; Table 2] The headline claim is not grounded in the official ARC-AGI-2 evaluation protocol. Section 4.1 describes the dataset only as 'grid-based visual puzzles' and never defines tokenization, input embedding, output decoding, training loss, or the train/validation split. The abstract reports '99% validation accuracy', while Table 2 labels the same model as achieving '99% Full Acc' on the ARC-AGI-2 leaderboard, which is a different metric: the leaderboard requires solving held-out evaluation tasks, and a self-defined validation split is not comparable to the official evaluation set. Because the central claim of outperforming DeepSeek-R1-1.5B, Grok-4, SOAR, and Greenblatt relies on this equivalence, the comparison is unsupported.
[Table 1; Sections 4.2 and 4.3] The text and the table contradict each other on a load-bearing point. Section 4.3 states that 'configurations without edge projection fail completely (0% accuracy)', yet Table 1 reports the 'Gating Only' configuration as having 90% validation accuracy and 74% test accuracy. Additionally, the abstract and Section 5.1 give different parameter counts (3.16M vs. 1.5M) and different accuracy numbers (99% validation vs. 96% test), and Section 5.1 states the model achieves 96% accuracy with 1.5M parameters while the abstract claims 99% validation accuracy with 3.16M. These inconsistencies make it impossible to tell which result is claimed and undermine the ablation-based conclusions.
[Sections 3.4 and 3.5; Algorithm 1] The core TreeFFN component is never defined. Algorithm 1 calls TreeFFN(H, E, root) but the manuscript gives no formula for what this function computes: there is no definition of the message-passing update, the MLP in Equation (7) is written with an unclosed bracket, the edge features e_ij in Equations (6)-(7) are never defined, and no loss function or optimization objective is specified. Equations (4)-(13) only describe the edge sets and residual additions, not the actual neural computation. This means the architecture is not reproducible from the manuscript, and the claim that 'the algorithm is well-defined' in the stress-test context cannot be verified.
[Section 1; Section 3.1; Section 5.1] There is a direct internal contradiction about whether attention is used. The title and Section 3.1 claim 'pure TreeFFN' processing with 'complete elimination of attention', while Section 1 states that TreeGPT 'combines Transformer's self-attention mechanism with a Global Parent-Child Aggregation mechanism'. Section 5.1 similarly credits 'the hybrid design' that combines 'attention mechanisms and tree-structured processing'. The 'Global Parent-Child Aggregation mechanism' is mentioned in the introduction and in Figure 5 but is never defined in the method sections. The contribution of the paper is therefore unclear, and the reader cannot determine whether the model truly is attention-free.
[Section 4.4; Abstract] The evaluation figures appearing in Section 5.1 and in the discussion are based on post-hoc selection on the test set. The abstract reports '100% token-level accuracy on selected evaluation samples' without stating the selection criterion, and Section 5.1 reports 96% test accuracy, which corresponds to the best configuration in Table 1 (Edge Proj + Gating). Since Table 1 reports test accuracy for each configuration and the final configuration is chosen from that table, the reported test accuracy is the result of selecting a model on the test set rather than an independent estimate. The 99% validation-accuracy figure in Table 2 is not the same as a held-out full-accuracy score, so the benchmark comparison is not valid.

minor comments (5)

[Section 2] The Chain-of-Thought discussion cites [Vaswani et al.(2017)] as if introducing CoT, but that reference is the original Transformer paper; CoT was introduced by Wei et al. and the citation is incorrect.
[Sections 4.2-4.3 and 4.5-4.6] The subsections are duplicated: '4.2 Ablation Study' and '4.3 Ablation Study' contain overlapping text, and the same duplication occurs for '4.5 Architecture Analysis' and '4.6 Architecture Analysis'. These should be merged into single subsections.
[Equations (6)-(7)] Equation (6) uses e_ij but the edge features are never defined, and the sentence in Section 3.5 says 'where⊕ denotes feature concatenation' but Equation (7) uses the symbol ⊕ in a way that is not mathematically precise. The formula should be written with explicit concatenation brackets.
[Figure captions and cross-references] Section 4.1 says 'Figure 1 demonstrates representative task types' while Figure 1 is the ablation bar chart, and Figure 2 is the task examples; the cross-references appear to be swapped.
[Throughout] There are numerous typos and formatting errors, including 'computationaloverhead' in Section 4.3, 'suppres' in Section 4.5, and an unclosed parenthesis in Equation (7). The text should be carefully copyedited.

Circularity Check

2 steps flagged · score 4.0 of 10

No circularity in the architecture derivation, but the headline 96% test accuracy is selected on the same test set and the 100% token-level claim is defined on a self-selected subset.

fitted input called prediction [Section 4.3 (Ablation Study), Table 1; repeated in Section 5.1]
"The optimal configuration combines edge projection with gated aggregation, achieving 96% test accuracy."

Table 1 reports test accuracy for every component configuration. Section 4.3 declares the edge-projection-plus-gating configuration optimal because it has the highest test accuracy (96%), and Section 5.1 then states 'TreeGPT achieves 96% accuracy using only 1.5M parameters.' The reported accuracy is therefore the selection criterion applied to the same test set, not an estimate on data untouched by model selection. The test set has been used as a fitting signal for the architecture choice, so the 'prediction' of 96% is statistically forced by the selection procedure.
other [Abstract; no evaluation protocol in Sections 4.1 and 4.4]
"The model converges within 1500 training steps and demonstrates 100% token-level accuracy on selected evaluation samples."

The metric is defined only on an unspecified 'selected' subset of evaluation samples. With no sampling rule, the 100% figure can be achieved by post hoc selection of samples the model already solves, making the claim unfalsifiable and self-referential rather than an independent evaluation on the full ARC-AGI-2 suite. This is not a derivation from the model equations but a reported outcome whose definition is under the authors' control.

full rationale

The architecture equations (1)-(13) and Algorithm 1 are self-contained definitions of residual adjacent-edge message passing; no derived quantity is equal to its inputs by construction, and no load-bearing conclusion rests on a self-citation (the references are standard external works). The claimed '99% validation accuracy' is not shown to be circular per se, but it is underspecified: no tokenization, split, loss, or metric definition is provided, so it cannot be compared to leaderboard baselines in Table 2. The two evaluation claims above do reduce to author-controlled choices: the 96% test accuracy is the argmax of the ablation table and the 100% token-level accuracy is defined on a self-selected subset. These are partial, evaluation-level circularities; the architecture derivation itself is independent. Score 4.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The central claim rests on an undefined TreeFFN, an unstated ARC-AGI-2 evaluation protocol, hand-picked hyperparameters, and a model configuration selected on the test set. No formal derivation or external benchmark result is supplied. The ledger entries above are the assumptions the reader must accept to believe the 99% headline.

free parameters (5)

Hidden dimension = 256
Architecture hyperparameter chosen without sensitivity analysis (Section 4.1).
Number of layers and TreeFFN iterations = 2 layers, 2 iterations
Manual choices in Section 4.1; the receptive field of neighbor message passing depends on these, and no ablation over them is reported.
Final model configuration = Edge projection + gating
Selected by test accuracy in Table 1 (96% versus 83%, 92%, 94%); testing on the same set used for model selection makes the reported accuracy partly fitted.
Training steps = 1500
Convergence claim in the abstract; no early-stopping criterion or learning-rate details beyond cosine schedule are given.
Parameter count = 3.16M (elsewhere 1.5M)
The paper gives conflicting numbers (Section 4.1 versus Section 5.1); the discrepancy undermines the claim and the configuration is not fixed.

assumptions (4)

ad hoc to paper A 1D chain of adjacent connections (i,i+1) and (i,i-1) can represent the hierarchical structure needed for ARC-AGI-2 reasoning tasks.
Equations (4)-(5) and Algorithm 1 use only neighbor edges; the paper provides no analysis of how tree or AST structure maps to a line, and nowhere defines a tree. This assumption is introduced to make the architecture work.
domain assumption ARC-AGI-2 grid puzzles can be encoded as token sequences and scored by token-level accuracy.
Section 4.1 describes the dataset but never specifies the tokenization or the mapping from grids to sequences, yet all experiments assume this encoding exists and preserves the task.
domain assumption Validation accuracy on the chosen split is comparable to official leaderboard full accuracy.
Table 2 compares TreeGPT validation accuracy with leaderboard full accuracies from different models without matched protocols (Section 4.4); this comparability is assumed.
ad hoc to paper The 'TreeFFN' component in Algorithm 1 is well-defined and computable.
Algorithm 1 calls TreeFFN(H,E,root) as a black-box; the optional definitions in Section 3.5 leave the base aggregation unspecified, so the algorithm is not actually defined.

invented entities (2)

Pure TreeFFN bidirectional encoder-decoder
purpose: Replace attention with two passes over adjacent connections (Eqs. 1-13, Algorithm 1).
The component is never fully specified (no base TreeFFN definition, no nonlinearity, no normalization); it is introduced purely to support the paper's architecture and has no external falsifiable handle.
Global Parent-Child Aggregation mechanism
purpose: Mentioned in the Introduction as a proposed mechanism for AST processing, but never defined or used in the method or experiments.
Only appears in Sections 1 and 5 as prose; no equation, algorithm, or result uses it. It is an invented concept with no evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms." pith.science (2026). https://pith.science/paper/Z73VUA34

@misc{pith2026250905550,
  author       = {Pith},
  title        = {Pith review of: TreeGPT: Pure TreeFFN Encoder-Decoder Architecture for Structured Reasoning Without Attention Mechanisms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z73VUA34}},
  note         = {Machine review of arXiv:2509.05550}
}

abstract

We present TreeGPT, an attention-free neural architecture that explores the potential of pure TreeFFN encoder-decoder design for structured reasoning tasks. Unlike traditional transformer approaches that rely on attention mechanisms, TreeGPT employs bidirectional TreeFFN components that process sequences through adjacent connections in parallel, aiming to achieve computational efficiency while maintaining reasoning capabilities. Our approach centers on a TreeFFN Encoder-Decoder mechanism: $$\text{Encoder TreeFFN (L} \rightarrow \text{R)} + \text{Decoder TreeFFN (R} \leftarrow \text{L)} \rightarrow \text{Parallel Processing}$$ where the encoder processes left-to-right dependencies while the decoder handles right-to-left patterns, both using simple neighbor-to-neighbor connections. This design eliminates attention computation while maintaining sequence modeling capabilities. We evaluate our approach on the ARC Prize 2025 dataset, where TreeGPT achieves 99\% validation accuracy using 3.16M parameters. The model converges within 1500 training steps and demonstrates 100\% token-level accuracy on selected evaluation samples. Our preliminary results suggest that for certain structured reasoning tasks, specialized TreeFFN architectures may offer advantages over attention-based approaches. While these findings are encouraging, we acknowledge that further investigation across diverse tasks and datasets would be valuable to establish the broader applicability of attention-free designs.

Figures

Figures reproduced from arXiv: 2509.05550 by the authors.

**Figure 2.** ARC-AGI Visual Reasoning Task Examples. The figure shows three representative task types: (top) [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** TreeGPT Training Convergence Analysis. The plots show: (left) accuracy convergence to 99% [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: TreeGPT Pure TreeFFN Encoder-Decoder Architecture. The model processes sequences through [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

**Figure 5.** Figure 5: Component Flow Diagram illustrating the TreeFFN processing pipeline with detailed visualization [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 5 canonical work pages

[1]

Attention is all you need.Advances in Neural Information Processing Systems, 30,

[Vaswani et al.(2017)]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in Neural Information Processing Systems, 30,

work page 2017
[8]

An empirical study of Mamba-based language models.arXiv preprint arXiv:2406.07887,

[Waleffe et al.(2024)]Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, and others. An empirical study of Mamba-based language models.arXiv preprint arXiv:2406.07887,

arXiv 2024
[2015]

Graph convolutional networks for text classification

[Yao et al.(2018)]Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32,

work page 2018
[2017]

Learning program representations with a tree-structured transformer.arXiv preprint arXiv:2208.08643,

[Wang et al.(2022)]Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Learning program representations with a tree-structured transformer.arXiv preprint arXiv:2208.08643,

arXiv 2022
[2018]

Non-local neural networks

[Wang et al.(2018)]Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803,

work page 2018
[2019]

TreeCaps: Tree-Based Capsule Networks for Source Code Processing

[Bui et al.(2020)]Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. TreeCaps: Tree-based capsule networks for source code processing.arXiv preprint arXiv:2009.09777,

work page Pith review arXiv 2020
[2020]

Mamba: Linear-time sequence modeling with selective state spaces

[Gu et al.(2023)]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

arXiv 2023
[2022]

TreeGen: A tree-based transformer architecture for code generation.arXiv preprint arXiv:1911.09983,

[Sun et al.(2019)]Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang. TreeGen: A tree-based transformer architecture for code generation.arXiv preprint arXiv:1911.09983,

arXiv 2019

Show all 11 references

[2023]

Self-improving language models for evolutionary program synthesis: A case study on ARC-AGI.arXiv preprint arXiv:2507.14172,

11 [Pourcel et al.(2025)]Julien Pourcel, Cédric Colas, and Pierre-Yves Oudeyer. Self-improving language models for evolutionary program synthesis: A case study on ARC-AGI.arXiv preprint arXiv:2507.14172,

2025
[2024]

Pointer networks.Advances in Neural Information Processing Systems, 28,

[Vinyals et al.(2015)]Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks.Advances in Neural Information Processing Systems, 28,

2015
[2025]

ARC-AGI-2: A new challenge for frontier AI reasoning systems.arXiv preprint arXiv:2505.11831,

[Chollet et al.(2025)]François Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. ARC-AGI-2: A new challenge for frontier AI reasoning systems.arXiv preprint arXiv:2505.11831,

2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Attention is all you need.Advances in Neural Information Processing Systems, 30,

[Vaswani et al.(2017)]Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in Neural Information Processing Systems, 30,

work page 2017

[2] [8]

An empirical study of Mamba-based language models.arXiv preprint arXiv:2406.07887,

[Waleffe et al.(2024)]Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, and others. An empirical study of Mamba-based language models.arXiv preprint arXiv:2406.07887,

arXiv 2024

[3] [2015]

Graph convolutional networks for text classification

[Yao et al.(2018)]Liang Yao, Chengsheng Mao, and Yuan Luo. Graph convolutional networks for text classification. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32,

work page 2018

[4] [2017]

Learning program representations with a tree-structured transformer.arXiv preprint arXiv:2208.08643,

[Wang et al.(2022)]Wenhan Wang, Ge Li, Bo Ma, Xin Xia, and Zhi Jin. Learning program representations with a tree-structured transformer.arXiv preprint arXiv:2208.08643,

arXiv 2022

[5] [2018]

Non-local neural networks

[Wang et al.(2018)]Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7794–7803,

work page 2018

[6] [2019]

TreeCaps: Tree-Based Capsule Networks for Source Code Processing

[Bui et al.(2020)]Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. TreeCaps: Tree-based capsule networks for source code processing.arXiv preprint arXiv:2009.09777,

work page Pith review arXiv 2020

[7] [2020]

Mamba: Linear-time sequence modeling with selective state spaces

[Gu et al.(2023)]Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752,

arXiv 2023

[8] [2022]

TreeGen: A tree-based transformer architecture for code generation.arXiv preprint arXiv:1911.09983,

[Sun et al.(2019)]Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang. TreeGen: A tree-based transformer architecture for code generation.arXiv preprint arXiv:1911.09983,

arXiv 2019