Learning neuro-symbolic convergent term rewriting systems

Alberto Testolin; Alessandro Sperduti; Flavio Petruzzellis

REVIEW 3 major objections 6 minor 38 references

Learning neuro-symbolic convergent term rewriting systems

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A neuro-symbolic architecture that mirrors the rewrite algorithm can learn to simplify formulas far beyond the depth seen in training, with a faster variant matching or beating strong LLM baselines on the hardest out-of-distribution cases.

desk verdict FastNRS is a solid, efficient extension of the authors' NRS with genuinely strong multi-domain results, but the OOD evaluation protocol needs explicit separation between validation and test, and variance is not reported. read the letter →

arxiv 2507.19372 v1 pith:ITEANM6N submitted 2025-07-25 cs.AI cs.LG

Flavio Petruzzellis , Alberto Testolin , Alessandro Sperduti This is my paper

classification cs.AIcs.LG MSC 68T0768Q42

keywords neuro-symbolicartificialintelligenceconvergenttermrewritingsystemsout-of-distributiongeneralizationlengthtransformerformulasimplificationalgorithmiclearningmulti-task

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that neural networks can learn to execute convergent term rewriting systems—the rule-based simplification procedures that reduce nested mathematical formulas to a single atomic value—if the architecture's modules are wired to mirror the algorithm itself. The authors present two implementations, NRS and FastNRS, both built from three modules: a Selector that finds a solvable leaf sub-expression, a Solver that evaluates it, and a Combiner that splices the value back into the formula. They claim that with label-based positional encodings and a narrow self-attention window in the Selector, both systems generalize to out-of-distribution formulas far deeper than those seen in training, and that FastNRS does so with orders-of-magnitude improvements in memory and speed. The value of showing this is that systematic generalization to novel, longer inputs—a known weakness of standard transformers and large language models—may be obtained by architectural inductive bias rather than by scaling parameters or data.

What carries the argument

The central object is the convergent term rewriting system, a set of rewrite rules that reduce any formula to the same final atomic value regardless of the order in which rules are applied, with no loops. The machinery that carries the argument is the decomposition of that algorithm into three modules: the Selector, which finds a reducible leaf formula (a sub-expression all of whose arguments are atomic values); the Solver, which maps that leaf formula to its equivalent atomic value; and the Combiner, which replaces the leaf with the value in the original formula. Two architectural choices in the Selector do most of the work: label-based positional encodings, in which the positions assigned to tokens are randomly sampled and sorted rather than fixed, acting as a data-augmentation mechanism; and a restricted self-attention receptive field that keeps each token's attention inside a narrow diagonal window (with best results at $k=1$), reflecting the premise that identifying a solvable leaf is a purely local problem. FastNRS further replaces the neural Combiner with a deterministic function and implements selection as a per-token binary classification, which produces its large gains in speed and memory.

What would settle it

Take the arithmetic domain and introduce a rewrite rule that is only applicable when a marker token appears at the far left of the formula (e.g., a flag that toggles whether $(a+b)$ may be simplified). Train the FastNRS Selector with $k=1$ on in-distribution examples with the flag near the leaf, then test on out-of-distribution examples where the flag is many tokens away; if OOD accuracy falls while a full-attention selector retains accuracy, the locality assumption on which the architecture depends is refuted.

Watch

Extended reading notes

Core claim

The central claim is that convergent term rewriting systems can be learned from data by a neuro-symbolic architecture whose computation mirrors the rewrite algorithm itself, rather than by a single end-to-end sequence-to-sequence mapping. The blueprint has three steps—locate a leaf formula (a sub-expression whose arguments are all atomic), reduce that leaf to its atomic value, and substitute the value back—and each step is assigned to a dedicated neural module (Selector, Solver, Combiner). The paper's key design claim is that the Selector determines out-of-distribution generalization: by training it to identify the last solvable leaf formula using label-based positional encodings and a diagonal self-attention window of width $2k+1$ with $k=1$, the system extrapolates to formulas tens of tokens longer than those seen in training. FastNRS re-formulates selection as a per-token binary text-segmentation task, allowing multiple leaf formulas to be selected in parallel and making the Combiner deterministic, at the cost of a small accuracy gap in some domains. On four benchmark domains (logic, ListOps, arithmetic, and algebra), the paper reports that both systems match or exceed the accuracy of the Neural Data Router, GPT-4, and o1-preview on the hardest out-of-distribution formulas.

Load-bearing premise

The load-bearing premise is that finding a solvable sub-expression inside a formula is a purely local problem, so a selector with a narrow diagonal attention window ($k=1$) can identify leaf formulas in inputs far longer than training examples; if any domain required distant context to locate reducible sub-expressions, the selector would fail on out-of-distribution inputs.

Editorial extensions

If this is right

A model trained on nesting levels 1–3 of arithmetic and algebra simplifies formulas of depth 5–6, and logic formulas up to depth 12, with sequence accuracy the paper reports as comparable to or higher than o1-preview on the hardest out-of-distribution splits.
FastNRS cuts inference time by roughly three orders of magnitude on the multi-domain test suite (3m42s vs 51h48m for NRS on the same samples per nesting level), with a small to zero accuracy cost depending on the domain.
The same architecture, without task-specific additions, learns four different term rewriting systems simultaneously in multi-domain training, and multi-domain training often reduces selector errors on arithmetic and algebra.
The comparison with GPT-4 and o1-preview suggests that parameter count alone does not drive systematic out-of-distribution generalization in formula simplification; architectural bias toward local rewrite steps can close the gap.
Selector errors dominate NRS failures at deeper nesting levels (the 'missing' class), while FastNRS never fails to extract at least one valid leaf in any iteration, which the paper credits for its accuracy advantage on deep arithmetic formulas.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the paper tunes hyperparameters, including the self-attention window width, using the OOD validation set (Section 5.2.1), the reported OOD accuracies are selection-filtered estimates; the unbiased expected OOD performance would require fitting hyperparameters on in-distribution data only before evaluation.
The locality premise behind the $k=1$ window suggests a concrete next test: add a rewrite rule whose applicability depends on non-adjacent tokens (e.g., a variable bound at an outer scope) and measure whether OOD accuracy collapses; if it does, the framework would need a new selector design, as the paper's future-work section anticipates.
The paper's stated limitation that the algorithmic blueprint is designed a priori implies the framework is a compiler for a class of algorithms rather than a learner of algorithms; a natural extension is to make the module decomposition itself differentiable so the blueprint is inferred from data, and to benchmark that variant on the same four domains.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

FastNRS is a real step forward from the earlier NRS: the parallel segmentation-based selector is new, and the multi-domain training results are new. The architectural design is clean, the baselines are appropriate (NDR, GPT-4, o1-preview), and the efficiency gains are genuinely impressive—inference time drops by orders of magnitude without a major accuracy cost. The core claim, that a modular neuro-symbolic system can learn local rewrite rules and generalize to deeper formulas, is plausible and supported on the separate test sets.

The soft spot is the OOD evaluation protocol. Hyperparameters, early stopping, and the FastNRS confidence threshold were chosen on an OOD validation set (Section 5.2.1). That is not automatically disqualifying—the final test sets appear disjoint from the validation sets (different sample counts in Tables A.5 and A.3)—but the paper never explicitly says the OOD test set is distinct from the OOD validation set, and Section 6.5 uses the same validation set to claim "indistinguishable convergence trends" on OOD instances. So the Selector's OOD generalization evidence is partly circular as presented. That is fixable: state plainly that a separate OOD test set was used for the headline numbers, and report the convergence plots on that set. As written, the reader cannot tell how much of the OOD performance is genuine generalization versus selection on a proxy.

Second, there are no error bars. The sentence "we observed non-significative variance across runs" is not a substitute for reporting the variance. With only 100 test samples per nesting level, standard errors are meaningful, and the differences between NRS and FastNRS at several depths could easily be within noise. Multiple seeds and confidence intervals are needed.

The limitation section is honest: only local rewriting rules, and the algorithmic blueprint is predefined. That narrows the "general framework" claim but does not undermine what is demonstrated within that scope.

Overall, this is a solid, well-written paper with a useful architecture and convincing single- and multi-domain results, provided the OOD protocol is clarified and variance is reported. I would send it to peer review with a request for revision, not desk-reject it.

Referee Report

3 major / 6 minor

Summary. The paper proposes a neuro-symbolic framework for learning convergent term rewriting systems for mathematical formula simplification. It presents two implementations, NRS and FastNRS, which decompose the rewriting loop into Selector, Solver, and Combiner modules. The Selector uses label-based positional encodings and a restricted diagonal self-attention window to identify leaf formulas; FastNRS recasts selection as text segmentation, enabling parallel leaf extraction and a deterministic conditional replacement step. The models are evaluated on logic, ListOps, arithmetic, and algebra, in both single-domain and multi-domain settings, against the Neural Data Router, GPT-4, and o1-preview. The central claims are strong out-of-distribution generalization, particularly at deeper nesting levels, and large efficiency gains for FastNRS in memory, training time, and inference time.

Significance. If the empirical results are reliable, the work would be significant: it shows that small, purpose-built neural modules with local attention can learn to execute rewrite procedures and generalize to formulas substantially deeper than those seen in training, outperforming large general-purpose LLMs on the hardest tested instances while using far fewer parameters. The architectural blueprint explicitly mirrors the rewriting algorithm, and the modular design supports multi-domain training without task-specific heads. The paper also includes useful ablations of self-attention window width and depth (Appendix B.1.1), detailed hyperparameter tables, and an error-type decomposition. However, the headline OOD generalization claim is currently weakened by the use of the OOD validation set for model selection and threshold tuning, the absence of error bars or seed variance despite test sets of only 100 samples per level, and sensitivity to inference-time thresholds. These issues must be addressed before the OOD claims can be taken at face value.

major comments (3)

[Section 5.2.1, Section 6.5, Tables B.6-B.8] The OOD generalization claim is compromised by model selection on the OOD validation set. Section 5.2.1 states that a 'distinct OOD validation set featuring formulas with higher structural complexity' (nesting levels 4-6) was 'used for model selection,' and Section 6.5 evaluates the Selector's 'out-of-distribution generalization' on that same OOD validation set. Because hyperparameters (Tables B.6-B.8), early stopping, the FastNRS confidence thresholds, and the NRS dynamic windowing threshold T (Section 5.2.1) were chosen with access to OOD-validation performance, the reported OOD test accuracy in Figures 3-4 is a transductively selected estimate rather than an unbiased measure of generalization to an unseen distribution. The Section 6.5 claim of 'almost indistinguishable convergence trends on in- and out-of-distribution instances' is based on the same OOD validation set, so it is not independent evidence. Please re-run the selection procedure using only in-distribution validation data, or report both protocols so the magnitude of the leakage can be assessed.
[Section 6, Figures 3-4] No variance or error bars are reported despite test sets of only 100 formulas per nesting level. The text says 'We observed non-significative variance across runs, which we therefore do not report.' With n=100, a 90% accuracy has a standard error of roughly 3 percentage points, so many observed differences (e.g., FastNRS vs NRS on arithmetic in Figure 3c, and several NRS/o1-preview comparisons in Figure 4) are within the range of sampling noise. The claims of 'significant improvements' and 'consistently achieved equal or higher accuracy' require either multiple seeds with confidence intervals or some other statistical support. Please provide per-seed results or error bars for the main test sets.
[Section 4.2.3, Section 5.2.1, Appendices C-D] The reported accuracy depends on inference-time thresholds that are tuned rather than architectural guarantees. The FastNRS cond_repl confidence threshold is set based on the distribution of Solver confidence on training samples (Section 5.2.1, Appendix D), and the NRS dynamic windowing threshold T is chosen by examining average Selector confidence on both in-distribution and OOD validation sets (Appendix C). These thresholds directly determine whether a candidate leaf is replaced or the computation is aborted, and they are free parameters of the evaluation. The paper should report sensitivity to these thresholds and, ideally, demonstrate that the OOD advantage is not primarily a consequence of threshold selection.

minor comments (6)

[Abstract, Section 7] The abstract's 'general framework' is broader than what is actually implemented: Section 7 acknowledges that rewriting rules must operate on local substrings and that the algorithmic blueprint is predefined by the designer. Please adjust the wording to make the scope explicit in the abstract and title.
[Section 6.2] The text refers to 'the previously proposed Neural Deductive Reasoner (NDR)', but the baseline is the Neural Data Router (Csordás et al., 2022). This appears to be a naming error.
[Section 6.2, Figure 4] Section 6.2 states that o1-preview was benchmarked only on out-of-distribution data splits, but Figure 4 appears to show o1-preview results on both IID and OOD splits. Please clarify the legend or the text so the reader knows exactly which splits each model was evaluated on.
[Abstract, Section 5.2.3, Table E.10] The model is called GPT-4o in the Abstract but GPT-4 in Section 5.2.3 and Table E.10. Please use a single name consistently.
[Section 6, Appendix C] The phrase 'non-significative variance' is nonstandard; use 'no significant variance.' Also, the Appendix C figure captions contain the typo 'Input Lenght' instead of 'Input Length.'
[General] The paper does not state whether the code and data will be released. Given the large number of hyperparameters and the modular pipelines, a public implementation would substantially aid reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the reported results are empirical comparisons on held-out test sets, and no load-bearing claim reduces by construction or self-citation to its inputs.

full rationale

The paper is an empirical evaluation, not a derivation, so there is no definitional equivalence between inputs and outputs. The central claim that NRS and FastNRS generalize out-of-distribution is supported by test-set accuracy reported in Section 6 (Figures 3 and 4), where test sets are described as distinct from training and validation splits (Section 6: 'The test sets on which all models are evaluated are composed of 100 formulas per nesting level'). The use of an OOD validation set for model selection (Section 5.2.1: 'we also developed a distinct OOD validation set featuring formulas with higher structural complexity, using this set for model selection') is a methodological caveat that may inflate OOD estimates, but it is not circular: the reported test accuracy is a different quantity from the OOD validation accuracy, and the comparisons against NDR, GPT-4, and o1-preview are external. Section 7 explicitly disclaims that the system learns the rewriting algorithm itself, which narrows the framework claim but does not make the results equivalent to the input. The only self-citations (e.g., Petruzzellis et al., 2024c, for the earlier NRS) are descriptive rather than load-bearing, as the present paper fully specifies the architecture. No fitted parameter is renamed as a prediction, and no claim reduces by construction to its own evidence.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the convergent rewriting formalization, the locality of leaf selection, and the transfer of Solver rules to OOD formulas. The fitted thresholds and OOD-tuned hyperparameters are the main free parameters. No new physical or conceptual entities are introduced.

free parameters (4)

Dynamic windowing threshold T = 150 (ListOps, Algebra), 125 (Arithmetic), unused (Logic)
Chosen by inspecting average Selector confidence versus input length; changes inference behavior for NRS on long inputs.
Solver confidence threshold for FastNRS cond_repl = -6 (ListOps), -2 (Arithmetic), -3 (Algebra), -0.005 (Logic)
Fit to training-set confidence distributions; controls which Solver outputs are accepted, directly affecting accuracy.
Self-attention window width k = 1 (FastNRS Selector); 2 or 3 (NRS Selector)
Tuned by random search; k=1 gave best OOD accuracy and is the core inductive bias for locality.
Hyperparameters selected on OOD validation = Various (embedding size, layers, learning rate, dropout; see Appendix B tables)
Model selection used the OOD validation set, which biases the reported OOD test numbers.

assumptions (5)

domain assumption Every formula in the problem class is convergent: f = o(a1,...,an) = e with a unique atomic final value.
Section 3 defines formula simplification problems via convergent term rewriting systems; if confluent termination fails, the Selector-Solver-Combiner loop may not terminate.
domain assumption All rewriting rules act on leaf formulas, i.e., formulas whose arguments are all atomic elements.
Section 3 restricts rules to F_L; the architecture assumes the problem decomposes into selecting leaves, solving them, and replacing them.
ad hoc to paper Leaf identification is a local problem solvable with a diagonal attention window.
Section 4.1.1 states that identifying leaf formulas is a local problem that can be solved without integrating distant tokens; this is the basis for the restricted attention that drives OOD generalization.
domain assumption Input formulas are syntactically correct.
Section 4.1.1 states the Selector assumes inputs to always be syntactically correct; the system is not designed to handle malformed inputs.
domain assumption Solver training on leaf formulas from shallow formulas transfers to leaves appearing in deeper formulas.
The Solver is trained on leaf formulas and atomic values only; leaves in OOD formulas are assumed to come from the same distribution of operations and operands.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Learning neuro-symbolic convergent term rewriting systems." pith.science (2026). https://pith.science/paper/ITEANM6N

@misc{pith2026250719372,
  author       = {Pith},
  title        = {Pith review of: Learning neuro-symbolic convergent term rewriting systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ITEANM6N}},
  note         = {Machine review of arXiv:2507.19372}
}

read the original abstract

Building neural systems that can learn to execute symbolic algorithms is a challenging open problem in artificial intelligence, especially when aiming for strong generalization and out-of-distribution performance. In this work, we introduce a general framework for learning convergent term rewriting systems using a neuro-symbolic architecture inspired by the rewriting algorithm itself. We present two modular implementations of such architecture: the Neural Rewriting System (NRS) and the Fast Neural Rewriting System (FastNRS). As a result of algorithmic-inspired design and key architectural elements, both models can generalize to out-of-distribution instances, with FastNRS offering significant improvements in terms of memory efficiency, training speed, and inference time. We evaluate both architectures on four tasks involving the simplification of mathematical formulas and further demonstrate their versatility in a multi-domain learning scenario, where a single model is trained to solve multiple types of problems simultaneously. The proposed system significantly outperforms two strong neural baselines: the Neural Data Router, a recent transformer variant specifically designed to solve algorithmic problems, and GPT-4o, one of the most powerful general-purpose large-language models. Moreover, our system matches or outperforms the latest o1-preview model from OpenAI that excels in reasoning benchmarks.

Figures

Figures reproduced from arXiv: 2507.19372 by the authors.

**Figure 2.** Visual representation of the simplification process of samples from the training [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗

**Figure 3.** Performance of FastNRS, NRS, and NDR on each domain. Sequence accuracy [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Performance of multi-domain models: GPT-4, o1-preview, FastNRS and NRS [PITH_FULL_IMAGE:figures/full_fig_p026_4.png]

**Figure 5.** Figure 5: Breakdown of NRS errors by type in single- and multi-domain settings. [PITH_FULL_IMAGE:figures/full_fig_p029_5.png]

**Figure 6.** Figure 6: Breakdown of FastNRS errors by type in single- and multi-domain settings. [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]

**Figure 7.** Figure 7: FastNRS Selector accuracy during training on the text segmentation task. [PITH_FULL_IMAGE:figures/full_fig_p031_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 25 canonical work pages

[3]

The number of parameters of o1-preview is a ballpark estimate based on the assumption that the model is a fine-tuned version of GPT-4

3The number of parameters of GPT-4 is an independent estimate based on inference speed Schreiner (2023). The number of parameters of o1-preview is a ballpark estimate based on the assumption that the model is a fine-tuned version of GPT-4. 40 Baader, F., Nipkow, T.,

work page 2023
[6]

arXiv preprint arXiv:2107.03374

Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 . Chen, X., Tian, Y.,

arXiv
[8]

Transformers discover an elementary calcu- lation system exploiting local attention and grid-like problem representa- tion, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8. Cormen, T.H., Leiserson, C.E.,

work page 2022
[9]

The devil is in the de- tail: Simple tricks improve systematic generalization of transformers, in: Moens, M., Huang, X., Specia, L., Yih, S.W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, Association for Computational Linguis...

doi:10.18653/v1/2021.emnlp-main.49 2021
[10]

URL: https://openreview.net/forum?id=KBQP4A_J1K

The neural data router: Adaptive control flow in transformers improves systematic generalization, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net. URL: https://openreview.net/forum?id=KBQP4A_J1K. Davis, E.,

work page 2022
[13]

Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. doi: 10.1109/CVPR.2016.90. Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., Gilmer, J.,

doi:10.1109/cvpr.2016.90 2016
[14]

The many faces of robustness: A critical analysis of out-of-distribution generalization, in: 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, IEEE. pp. 8320–8329. URL: https://doi.org/10.1109/ ICCV48922.2021.00823, doi:10.1109/ICCV48922.2021.00823. 42 Hinton, G.E.,

arXiv 2021
[16]

Compositional- ity decomposed: How do neural networks generalise? J. Artif. In- tell. Res. 67, 757–795. URL: https://doi.org/10.1613/jair.1.11674, doi:10.1613/JAIR.1.11674. Kahneman, D.,

doi:10.1613/jair.1.11674

Show all 38 references

[17]

(Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16,

The impact of positional encoding on length generalization in transformers, in: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, ...

2023
[18]

Large language models are zero-shot reasoners, in: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (Eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Infor- mation Processing Systems 2022, NeurIPS 2022, New Orleans, LA...

2022
[21]

ArXiv abs/1912.01412

Deep learning for symbolic mathematics. ArXiv abs/1912.01412. Li, Y., McClelland, J.L.,

1912 arXiv
[23]

Listops: A diagnostic dataset for latent tree learning, in: Cordeiro, S.R., Oraby, S., Pavalanathan, U., Rim, K. (Eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL-HLT 2018, New Orleans, Louisiana, U...

2018 doi
[24]

arXiv:2303.08774

GPT-4 technical report. arXiv:2303.08774. OpenAI,

arXiv
[25]

Accessed: 2024-09-29

Learning to reason with LLMs.https://openai.com/index/ learning-to-reason-with-llms/ . Accessed: 2024-09-29. Petruzzellis, F., Testolin, A., Sperduti, A., 2024a. Assessing the emergent symbolic reasoning abilities of llama large language models. To appear in Proceedings of the...

2024
[26]

CoRR abs/2110.04169

Iterative decoding for compositional generalization in transformers. CoRR abs/2110.04169. URL: https:// arxiv.org/abs/2110.04169, arXiv:2110.04169. Rumelhart, D.E., McClelland, J.L., AU,

arXiv
[27]

Randomized positional encodings boost length generalization of transformers, in: Rogers, A., Boyd-Graber, J.L., Okazaki, N. (Eds.), Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), ACL 2023, Toronto, Canada, Ju...

2023 doi
[28]

URL: https://openreview.net/forum?id= H1gR5iR5FX

Analysing mathemat- ical reasoning abilities of neural models, in: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net. URL: https://openreview.net/forum?id= H1gR5iR5FX. Schreiner, M.,

2019
[29]

https://web.archive

GPT-4 architecture, datasets, costs and more leaked. https://web.archive. org/web/20230712123915/https://the-decoder.com/ gpt-4-architecture-datasets-costs-and-more-leaked/ . Accessed: 2023-07-12. 45 Setzler, M., Howland, S., Phillips, L.A.,

2023
[30]

CoRR abs/2201.11766

Recursive decoding: A situ- ated cognition approach to compositional generation in grounded language understanding. CoRR abs/2201.11766. URL: https://arxiv.org/abs/ 2201.11766, arXiv:2201.11766. Testolin, A.,

arXiv
[32]

Patterns 2, 100273

Neural algorithmic reasoning. Patterns 2, 100273. URL: https://doi.org/10.1016/j.patter.2021.100273, doi:10.1016/J.PATTER.2021.100273. Velickovic, P., Ying, R., Padovano, M., Hadsell, R., Blundell, C.,

2021
[33]

URL: https://openreview.net/forum?id= SkgKO0EtvS

Neu- ral execution of graph algorithms, in: 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26- 30, 2020, OpenReview.net. URL: https://openreview.net/forum?id= SkgKO0EtvS. Vinyals, O., Fortunato, M., Jaitly, N.,

2020
[35]

Self-consistency improves chain of thought reasoning in language models, in: The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

2023
[36]

Chain-of-thought prompting elicits rea- soning in large language models, in: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (Eds.), Advances in Neural Information 46 Processing Systems 35: Annual Conference on Neural Information Pro- cessing Systems 2022, ...

2022
[37]

(Eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp

Towards a theoretical framework of out-of-distribution generalization, in: Ranzato, M., Beygelz- imer, A., Dauphin, Y.N., Liang, P., Vaughan, J.W. (Eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, N...

2021
[38]

URL: https://openreview.net/forum?id= AssIuHnmHX

What algorithms can transformers learn? A study in length generalization, in: The Twelfth International Confer- ence on Learning Representations, ICLR 2024, Vienna, Austria, May 7- 11, 2024, OpenReview.net. URL: https://openreview.net/forum?id= AssIuHnmHX. 47

2024
[1990]

Connectionist symbol processing - preface. Artif. In- tell. 46, 1–4. URL: https://doi.org/10.1016/0004-3702(90)90002-H , doi:10.1016/0004-3702(90)90002-H. Hupkes, D., Dankers, V., Mul, M., Bruni, E.,

doi
[2009]

SciTePress

- ICNC, INSTICC. SciTePress. pp. 452–458. doi:10.5220/0002319704520458. Lake, B.M., Baroni, M.,

doi
[2014]

CoRR abs/1410.5401

Neural turing ma- chines. CoRR abs/1410.5401. URL: http://arxiv.org/abs/1410.5401, arXiv:1410.5401. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska- Barwinska, A., Colmenarejo, S.G., Grefenstette, E., Ramalho, T., Aga- piou, J.P., Badia, A.P., Hermann, ...

arXiv
[2015]

(Eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neu- ral Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp

Pointer networks, in: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neu- ral Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 2692–2700....

2015
[2016]

Hybrid computing using a neural network with dynamic ex- ternal memory. Nat. 538, 471–476. URL: https://doi.org/10.1038/ nature20101, doi:10.1038/NATURE20101. He, K., Zhang, X., Ren, S., Sun, J.,

doi
[2017]

(Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp

Attention is all you need, in: Guyon, I., von Luxburg, U., Bengio, S., Wallach, H.M., Fergus, R., Vishwanathan, S.V.N., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017...

2017
[2018]

(Eds.), Proceedings of the 35th International Con- ference on Machine Learning, ICML 2018, Stockholmsm¨ assan, Stock- holm, Sweden, July 10-15, 2018, PMLR

Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks, in: Dy, J.G., Krause, A. (Eds.), Proceedings of the 35th International Con- ference on Machine Learning, ICML 2018, Stockholmsm¨ assan, Stock- holm, Sweden, July 10-15...

2018
[2019]

Learning to perform local rewriting for combina- torial optimization, in: Wallach, H.M., et al. (Eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Infor- mation Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Ca...

2019
[2020]

(Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual

Language models are few-shot learners, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. C...

2020
[2021]

CoRR abs/2104.14095

Analyzing the nuances of trans- formers’ polynomial simplification abilities. CoRR abs/2104.14095. URL: https://arxiv.org/abs/2104.14095, arXiv:2104.14095. Anil, C., Wu, Y., Andreassen, A., Lewkowycz, A., Misra, V., Ramasesh, V.V., Slone, A., Gur-Ari, G., Dyer, E., Neyshabur, B.,

arXiv
[2022]

Exploring length generalization in large language models, in: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (Eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Infor- mation Processing Systems 2022, NeurIPS 2022, New ...

2022
[2023]

URL: https://arxiv.org/abs/2301.00704, arXiv:2301.00704

Muse: Text-to-image generation via masked generative transformers. URL: https://arxiv.org/abs/2301.00704, arXiv:2301.00704. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.D.O., Kaplan, J., Ed- wards, H., Burda, Y., Joseph, N., Brockman, G., et al.,

arXiv
[2024]

CoRR abs/2410.05229

Gsm-symbolic: Understanding the limitations of mathemat- ical reasoning in large language models. CoRR abs/2410.05229. URL: https://doi.org/10.48550/arXiv.2410.05229, doi: 10.48550/ARXIV. 2410.05229, arXiv:2410.05229. Nangia, N., Bowman, S.R.,

doi arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [3]

The number of parameters of o1-preview is a ballpark estimate based on the assumption that the model is a fine-tuned version of GPT-4

3The number of parameters of GPT-4 is an independent estimate based on inference speed Schreiner (2023). The number of parameters of o1-preview is a ballpark estimate based on the assumption that the model is a fine-tuned version of GPT-4. 40 Baader, F., Nipkow, T.,

work page 2023

[2] [6]

arXiv preprint arXiv:2107.03374

Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 . Chen, X., Tian, Y.,

arXiv

[3] [8]

Transformers discover an elementary calcu- lation system exploiting local attention and grid-like problem representa- tion, in: 2022 International Joint Conference on Neural Networks (IJCNN), IEEE. pp. 1–8. Cormen, T.H., Leiserson, C.E.,

work page 2022

[4] [9]

The devil is in the de- tail: Simple tricks improve systematic generalization of transformers, in: Moens, M., Huang, X., Specia, L., Yih, S.W. (Eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, Association for Computational Linguis...

doi:10.18653/v1/2021.emnlp-main.49 2021

[5] [10]

URL: https://openreview.net/forum?id=KBQP4A_J1K

The neural data router: Adaptive control flow in transformers improves systematic generalization, in: The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, OpenReview.net. URL: https://openreview.net/forum?id=KBQP4A_J1K. Davis, E.,

work page 2022

[6] [13]

Deep residual learning for image recognition, in: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. doi: 10.1109/CVPR.2016.90. Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., Gilmer, J.,

doi:10.1109/cvpr.2016.90 2016

[7] [14]

The many faces of robustness: A critical analysis of out-of-distribution generalization, in: 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, IEEE. pp. 8320–8329. URL: https://doi.org/10.1109/ ICCV48922.2021.00823, doi:10.1109/ICCV48922.2021.00823. 42 Hinton, G.E.,

arXiv 2021

[8] [16]

Compositional- ity decomposed: How do neural networks generalise? J. Artif. In- tell. Res. 67, 757–795. URL: https://doi.org/10.1613/jair.1.11674, doi:10.1613/JAIR.1.11674. Kahneman, D.,

doi:10.1613/jair.1.11674