{"id":"eb27fba7-f0f6-4fa9-aea1-76644ffd43d7","arxiv_id":"2509.05550","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"TreeGPT claims 99% ARC-AGI-2 validation accuracy with a 3.16M-parameter attention-free neighbor-passing TreeFFN, but the protocol and definitions needed to verify it are missing.","lead":"This paper proposes TreeGPT, a neural network that removes attention and relies only on neighbor-to-neighbor connections, claiming 99% validation accuracy on the ARC-AGI-2 reasoning benchmark with 3.16 million parameters. It matters because an attention-free 3-million-parameter model at that accuracy would challenge the scaling and attention paradigm for structured reasoning.","discovery_kind":"unclear","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 99% validation-accuracy figure is not tied to the official ARC-AGI-2 evaluation protocol, so the headline comparison to leaderboard baselines is unsupported.","rationale":"The reader's weakest_assumption correctly identifies the missing evaluation protocol. My stress-test sharpens this into a specific metric mismatch: the paper compares a self-defined validation accuracy against official leaderboard full accuracy (Table 2, Section 4.4). The abstract's 'selected evaluation samples' further weakens the claim, and Section 4.1 gives no details on grid tokenization, train/validation split, or how validation accuracy is computed at task versus token level. This is not a mere stylistic omission; it is the hinge on which the paper's central quantitative claim turns. Without matching the official ARC-AGI-2 protocol, the 99% figure cannot be compared to any baseline in Table 2, and the argument that attention is unnecessary for structured reasoning is unsupported. I therefore agree with the reader's REJECT verdict. I do not raise a separate concern about the unlikelihood of the result per se, because disagreement with current consensus is not by itself a correctness objection; the problem is that the empirical evidence as presented does not establish that the result occurred under the claimed benchmark conditions.","tokens_in":6877,"tokens_out":3173,"duration_ms":30630,"concrete_test":"Obtain the official ARC-AGI-2 evaluation harness described in Chollet et al. (arXiv:2505.11831) and run the released TreeGPT code (or the exact configuration claimed in Section 4.1) on the official held-out evaluation tasks using the same tokenization and split the paper intends. Report task-level and token-level accuracy separately. If the official evaluation accuracy is materially below 99%—or if no tokenization or split can be identified—the central benchmark claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that TreeGPT, a 3.16M-parameter attention-free model, reaches 99% validation accuracy on ARC-AGI-2 and therefore outperforms large models and program-synthesis systems. This claim is load-bearing because every downstream conclusion—parameter efficiency, convergence speed, and the alleged advantage of removing attention—depends on that number. However, the paper never defines the evaluation protocol. Section 4.1 describes the dataset only as 'grid-based visual puzzles' and gives no tokenization, no input embedding, no output decoding, no training loss, and no train/validation split. The abstract introduces '100% token-level accuracy on selected evaluation samples,' admitting non-random sample selection, while Table 2 compares 'TreeGPT 99%' against leaderboard 'Full Acc' values. Those are different metrics: the official ARC-AGI-2 leaderboard requires solving held-out evaluation tasks, whereas 'validation accuracy' on a self-defined split is not comparable to leaderboard full accuracy. If the 99% figure comes from a nonstandard validation split or from token-level accuracy on selected samples, the comparison to DeepSeek-R1-1.5B, Grok-4, SOAR, and Greenblatt is invalid. The architecture's undefined nature independently blocks reproducibility, but the evaluation-protocol mismatch is the most direct reason the headline claim cannot be accepted as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TreeGPT, an attention-free encoder-decoder architecture built from bidirectional TreeFFN components that propagate information only through adjacent neighbor connections. The authors evaluate the model on the ARC-AGI-2 visual reasoning benchmark and report that it achieves 99% validation accuracy with 3.16M parameters, converges within 1500 training steps, and reaches 100% token-level accuracy on selected evaluation samples. They compare these numbers with the official ARC-AGI-2 leaderboard, claiming substantial improvements over large language models and program-synthesis baselines. The manuscript also includes an ablation study of edge projection, gating, and residual components, and concludes that eliminating attention can be beneficial for structured reasoning.","tokens_in":7193,"tokens_out":3312,"duration_ms":28908,"significance":"If the headline result were established under the official ARC-AGI-2 evaluation protocol, it would be highly significant: a 3.16M-parameter attention-free model outperforming billion-parameter reasoning systems would challenge core assumptions about the necessity of attention and about scaling for abstract reasoning. The paper also advertises a GitHub repository with implementation and training scripts, which would be a useful reproducibility contribution if the architecture and protocol were precisely specified. However, as presented, the empirical claims are not defensible because the evaluation protocol is undefined, the reported metrics are internally inconsistent, and the architecture itself is only partially specified. The paper therefore cannot currently support its central claims, and the manuscript reads as an early-stage report rather than a completed study.","major_comments":[{"comment":"The headline claim is not grounded in the official ARC-AGI-2 evaluation protocol. Section 4.1 describes the dataset only as 'grid-based visual puzzles' and never defines tokenization, input embedding, output decoding, training loss, or the train/validation split. The abstract reports '99% validation accuracy', while Table 2 labels the same model as achieving '99% Full Acc' on the ARC-AGI-2 leaderboard, which is a different metric: the leaderboard requires solving held-out evaluation tasks, and a self-defined validation split is not comparable to the official evaluation set. Because the central claim of outperforming DeepSeek-R1-1.5B, Grok-4, SOAR, and Greenblatt relies on this equivalence, the comparison is unsupported.","section":"Abstract; Sections 4.1 and 4.4; Table 2"},{"comment":"The text and the table contradict each other on a load-bearing point. Section 4.3 states that 'configurations without edge projection fail completely (0% accuracy)', yet Table 1 reports the 'Gating Only' configuration as having 90% validation accuracy and 74% test accuracy. Additionally, the abstract and Section 5.1 give different parameter counts (3.16M vs. 1.5M) and different accuracy numbers (99% validation vs. 96% test), and Section 5.1 states the model achieves 96% accuracy with 1.5M parameters while the abstract claims 99% validation accuracy with 3.16M. These inconsistencies make it impossible to tell which result is claimed and undermine the ablation-based conclusions.","section":"Table 1; Sections 4.2 and 4.3"},{"comment":"The core TreeFFN component is never defined. Algorithm 1 calls TreeFFN(H, E, root) but the manuscript gives no formula for what this function computes: there is no definition of the message-passing update, the MLP in Equation (7) is written with an unclosed bracket, the edge features e_ij in Equations (6)-(7) are never defined, and no loss function or optimization objective is specified. Equations (4)-(13) only describe the edge sets and residual additions, not the actual neural computation. This means the architecture is not reproducible from the manuscript, and the claim that 'the algorithm is well-defined' in the stress-test context cannot be verified.","section":"Sections 3.4 and 3.5; Algorithm 1"},{"comment":"There is a direct internal contradiction about whether attention is used. The title and Section 3.1 claim 'pure TreeFFN' processing with 'complete elimination of attention', while Section 1 states that TreeGPT 'combines Transformer's self-attention mechanism with a Global Parent-Child Aggregation mechanism'. Section 5.1 similarly credits 'the hybrid design' that combines 'attention mechanisms and tree-structured processing'. The 'Global Parent-Child Aggregation mechanism' is mentioned in the introduction and in Figure 5 but is never defined in the method sections. The contribution of the paper is therefore unclear, and the reader cannot determine whether the model truly is attention-free.","section":"Section 1; Section 3.1; Section 5.1"},{"comment":"The evaluation figures appearing in Section 5.1 and in the discussion are based on post-hoc selection on the test set. The abstract reports '100% token-level accuracy on selected evaluation samples' without stating the selection criterion, and Section 5.1 reports 96% test accuracy, which corresponds to the best configuration in Table 1 (Edge Proj + Gating). Since Table 1 reports test accuracy for each configuration and the final configuration is chosen from that table, the reported test accuracy is the result of selecting a model on the test set rather than an independent estimate. The 99% validation-accuracy figure in Table 2 is not the same as a held-out full-accuracy score, so the benchmark comparison is not valid.","section":"Section 4.4; Abstract"}],"minor_comments":[{"comment":"The Chain-of-Thought discussion cites [Vaswani et al.(2017)] as if introducing CoT, but that reference is the original Transformer paper; CoT was introduced by Wei et al. and the citation is incorrect.","section":"Section 2"},{"comment":"The subsections are duplicated: '4.2 Ablation Study' and '4.3 Ablation Study' contain overlapping text, and the same duplication occurs for '4.5 Architecture Analysis' and '4.6 Architecture Analysis'. These should be merged into single subsections.","section":"Sections 4.2-4.3 and 4.5-4.6"},{"comment":"Equation (6) uses e_ij but the edge features are never defined, and the sentence in Section 3.5 says 'where⊕ denotes feature concatenation' but Equation (7) uses the symbol ⊕ in a way that is not mathematically precise. The formula should be written with explicit concatenation brackets.","section":"Equations (6)-(7)"},{"comment":"Section 4.1 says 'Figure 1 demonstrates representative task types' while Figure 1 is the ablation bar chart, and Figure 2 is the task examples; the cross-references appear to be swapped.","section":"Figure captions and cross-references"},{"comment":"There are numerous typos and formatting errors, including 'computationaloverhead' in Section 4.3, 'suppres' in Section 4.5, and an unclosed parenthesis in Equation (7). The text should be carefully copyedited.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"This manuscript is not ready for journal publication. The core empirical claim is invalid as presented because the evaluation protocol is undefined, the reported accuracy figures are inconsistent across sections, and the model selection appears to be performed on the test set. The architecture, while conceptually simple, is not specified in enough detail for a reader to implement it, and the self-reported GitHub repository was not attached or verified during review. There is also a serious mismatch between the 'validation accuracy' reported in the abstract and the 'Full Acc' column in Table 2, which would misinform readers comparing against official leaderboard numbers. Given the number of load-bearing inconsistencies, a major revision would effectively require a new paper with a properly defined protocol and independent evaluation. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: the central claim is not credible as stated. A 3.16M-parameter attention-free model scoring 99% on ARC-AGI-2 would be a real result, but the paper never defines the evaluation protocol, and its own numbers contradict each other. The architecture itself is not new: Eqs. 4–5 describe bidirectional neighbor-to-neighbor message passing over a linear chain, which is a standard 1D-convolution/graph-message-passing pattern. No formal characterization is offered, and the 'Global Parent-Child Aggregation' named in the intro never appears in the method.\n\nWhat the paper does well: it makes a genuine attempt at ablation, with a clear component hierarchy (edge projection critical, gating helps, residual can hurt), and it points to a public repository. That is more than many such submissions do. But the ablation table contradicts its own text: the table gives Gating Only 74% test accuracy, while the text says all configurations without edge projection fail completely (0%). Parameter counts flip between 3.16M and 1.5M, and validation accuracy (99%) is inconsistently reported against test accuracy (96%). The abstract's '100% token-level accuracy on selected evaluation samples' admits cherry-picking, yet Table 2 compares this against official leaderboard full accuracy.\n\nMost importantly, the evaluation protocol is absent. There is no tokenization, no input embedding, no output decoding, no train/validation split, no loss definition. Section 4.1 calls the dataset 'grid-based visual puzzles' and stops. Without these details, the 99% figure cannot be checked or compared to the baselines in Table 2. The stress-test note gets this right: the headline comparison is unsupported because the metric is self-defined.\n\nThere are also smaller signs of carelessness: the CoT reference is attributed to Vaswani et al., and Section 1 introduces the model as a hybrid with attention before the rest of the paper claims pure attention-free processing. None of these are fatal in isolation, but together they indicate the paper has not been through a serious internal review.\n\nWho would get value from this? Perhaps a reader interested in cataloguing failed or sloppy attempts at attention-free reasoning, but not someone looking for a reliable result. The idea itself—that structured reasoning might not need attention—is worth investigating, but this paper does not advance it. My recommendation: desk reject. There is no point sending referees to work through contradictions and missing protocol details that the authors should have resolved before submission.","headline":"The headline result—99% on ARC-AGI-2 with 3.16M parameters—is unsupported by the paper's own inconsistent numbers and missing evaluation protocol; this should be desk-rejected.","tokens_in":7689,"tokens_out":1846,"would_cite":false,"duration_ms":17088,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An attention-free network with 3.16M parameters claims 99% accuracy on the ARC-AGI-2 reasoning benchmark.","keywords":["TreeFFN","attention-free architecture","ARC-AGI-2","structured reasoning","encoder-decoder","adjacent connections","parameter efficiency","sequence modeling"],"falsifier":"Download the released implementation and run it on the official ARC-AGI-2 validation set using the official evaluation harness, scoring output grids by exact match to the expected grids. If exact-match accuracy is substantially below 99%, or if the reported accuracy only holds on a hand-picked subset, the central claim collapses; additionally, checking for overlap between training tasks and validation tasks would settle whether the model generalizes or memorizes.","tokens_in":6638,"feed_emoji":"🧩","tokens_out":5270,"duration_ms":40809,"temperature":0.7,"pith_summary":"This paper claims that a purely attention-free architecture, TreeGPT, built from bidirectional TreeFFN components that only pass information between neighboring sequence positions, can solve ARC-AGI-2 visual reasoning puzzles to 99% validation accuracy. The model uses about 3.16 million parameters and converges in 1,500 training steps, which would put it far ahead of billion-parameter transformers and specialized program-synthesis systems on the same benchmark. If correct, the result would show that attention is not needed for structured reasoning, and that a simple adjacent-connection inductive bias can outperform scale. The paper presents these findings as preliminary and calls for broader evaluation across diverse tasks.","feed_headline":"Attention-free model claims 99% on ARC-AGI-2 benchmark","feed_subtitle":"A 3.16M-parameter TreeFFN network reportedly beats billion-parameter systems on structured-reasoning puzzles.","key_machinery":"The key machinery is the pure TreeFFN encoder-decoder. The encoder applies a TreeFFN over adjacent edges $E_{\\text{enc}}=\\{(i,i+1)\\}$ to propagate left-to-right dependencies, while the decoder applies a TreeFFN over reversed edges $E_{\\text{dec}}=\\{(i,i-1)\\}$ for right-to-left pattern generation; the two runs happen in parallel and are combined through additive residuals. Optional edge projection (linearly transforming edge features before message passing) and gated aggregation are included to control information flow. Ablation results single out edge projection as the load-bearing component: without it the model scores 0%, and with it (plus gating) test accuracy reaches 96%.","core_discovery":"On the paper's own terms, the central discovery is that an attention-free encoder-decoder built from pure TreeFFN layers—where the encoder sweeps left-to-right along edges $(i,i+1)$ and the decoder sweeps right-to-left along $(i,i-1)$, both in parallel—can solve the ARC-AGI-2 reasoning benchmark with 99% validation accuracy and 100% token-level accuracy on selected evaluation samples. The authors attribute the result to the combination of bidirectional neighbor-to-neighbor propagation and edge projection, which their ablation identifies as the critical component; configurations without edge projection collapse to 0% accuracy. They report convergence within 1500 training steps using a 3.16M-parameter model, and argue this indicates that attention mechanisms, while powerful, are not necessary for certain structured reasoning tasks.","pith_inferences":["The paper's numbers are not internally consistent—the abstract reports 3.16M parameters and 99% validation accuracy, while several body sections state 1.5M parameters and 96% accuracy—so the abstract's 99% figure is the claim to verify.","A natural testable extension is applying the same two-pass neighbor-connection architecture to ARC-AGI-1 and to synthetic grid-rewriting tasks with known rules, which would show whether the result generalizes beyond the one benchmark.","Because each layer only exchanges information between adjacent positions, the architecture's effective receptive field grows linearly with depth; stress tests that require propagating information across many grid cells would delimit how far attention-free processing can go."],"forward_implications":["If the 99% figure holds, attention-free neighbor-connection models would outperform transformer-based and program-synthesis systems by a wide margin on ARC-AGI-2, implying attention is not necessary for this kind of structured reasoning.","A 3.16M-parameter model beating billion-parameter systems on this benchmark would make architectural inductive bias a stronger factor than scale for structured tasks.","Converging in 1,500 steps would indicate that such structured reasoning problems do not need the long optimization typically required by large attention models.","The bidirectional TreeFFN design could be transferred to other sequence tasks with strong local structure, such as grammar-based generation or grid-world navigation."],"supporting_citations":[{"why":"Supplies the ARC-AGI-2 benchmark dataset and evaluation setup that the accuracy claims reference.","marker":"[Chollet et al.(2025)]"},{"why":"Defines the attention-based Transformer architecture that TreeGPT is contrasted against as the main baseline.","marker":"[Vaswani et al.(2017)]"},{"why":"Introduces Mamba, the selective state-space baseline that TreeGPT compares with and argues lacks tree-structure modeling.","marker":"[Gu et al.(2023)]"},{"why":"Provides program-synthesis results on ARC-AGI that underpin the comparison to search-based methods.","marker":"[Pourcel et al.(2025)]"},{"why":"Presents a tree-structured transformer for program representation, prior work on tree processing that TreeGPT extends.","marker":"[Wang et al.(2022)]"}],"fun_headline_variants":["Attention-free TreeGPT hits 99% on ARC-AGI-2 puzzles","TreeFFN model achieves 99% on ARC-AGI-2 without attention","Small 3.16M-param TreeGPT beats attention models on ARC-AGI-2","No attention, 99% accuracy: TreeGPT's TreeFFN wins on ARC-AGI-2","Bidirectional TreeFFN reaches 99% on ARC-AGI-2 with 3.16M params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on the unstated assumption that the ARC-AGI-2 evaluation was run under the official benchmark protocol—including how grid puzzles are tokenized, what the train/validation split is, what 'validation accuracy' counts at task or token level, and how the 'selected evaluation samples' were chosen—so that the 99% figure is directly comparable to the leaderboard baselines in Table 2.","fun_headline_variants_meta":{"raw":{"variants":["Attention-free TreeGPT hits 99% on ARC-AGI-2 puzzles","TreeFFN model achieves 99% on ARC-AGI-2 without attention","Small 3.16M-param TreeGPT beats attention models on ARC-AGI-2","No attention, 99% accuracy: TreeGPT's TreeFFN wins on ARC-AGI-2","Bidirectional TreeFFN reaches 99% on ARC-AGI-2 with 3.16M params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1435,"prompt_tokens":956,"completion_tokens":479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":355}},"tokens_in":572,"tokens_out":479,"duration_ms":4068,"temperature":1.0,"reasoning_tokens":355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:23:10.060521+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Download the released implementation and run it on the official ARC-AGI-2 validation set using the official evaluation harness, scoring output grids by exact match to the expected grids. If exact-match accuracy is substantially below 99%, or if the reported accuracy only holds on a hand-picked subset, the central claim collapses; additionally, checking for overlap between training tasks and validation tasks would settle whether the model generalizes or memorizes.","supporting_citations":[{"cited_title":"Attention is all you need.Advances in Neural Information Processing Systems, 30,","cited_arxiv_id":null,"evidence_quote":"Defines the attention-based Transformer architecture that TreeGPT is contrasted against as the main baseline."}],"review_version":2}