Pith. sign in

REVIEW 4 major objections 6 minor 5 references

Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees

T0 review · 4 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A GPT-2 transformer, trained on the first 10^11 integers encoded as Dyck-word trees, partially learns the grammar of the arithmetic sequence, reaching word accuracy around 0.4 and stable performance on test blocks at 10^13–10^15.

desk verdict Interesting benchmark idea with a solid modest result, but the abstract promises control and extrapolation experiments that aren't in the body. read the letter →

arxiv 2512.01870 v2 pith:WG52Y4LP submitted 2025-12-01 cs.AI cond-mat.dis-nnmath-phmath.MPmath.NT

Alessandro Breccia , Federica Gerace , Marco Lippi , Gabriele Sicuro , Pierluigi Contucci This is my paper
classification cs.AIcond-mat.dis-nnmath-phmath.MPmath.NT
keywords arithmetictextrootedtreesprimefactorizationDyckwordstransformersnext-wordpredictionmaskedlanguagemodelinglearnability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper attempts to show that a transformer language model can learn the deterministic sequence of rooted trees encoding the iterated prime factorization of the integers. It trains a GPT-2 model on the first 10^11 elements of this arithmetic text and evaluates next-word and masked-word prediction against a Markov baseline. The central claim is that the model reaches about 0.4 word accuracy, well above the baseline, and that its performance remains stable on blocks located at 10^13–10^15, far beyond the training interval. The abstract further claims that the likelihood assigned by the model separates the arithmetic text from control sequences that match its word frequencies but lack its sequential organization. If true, this suggests that a transformer can infer structure beyond frequency statistics, offering a controlled probe of how neural sequence models extract latent rules.

What carries the argument

The central object is the rooted planar tree representation of each integer via iterated prime factorization, mapped to a Dyck word—a balanced binary string—so that the sequence of integers becomes an arithmetic text NT. The learning machinery is a GPT-2-style transformer decoder with 12 layers, 12 attention heads, and an embedding dimension of 768, trained on tokenized Dyck words using Byte-Pair Encoding. The two self-supervised tasks, next-word prediction and masked language modeling, with a temperature-controlled softmax, are used to measure how much of the text's grammar the model captures. A Markov chain transition matrix serves as the minimal baseline.

What would settle it

A concrete way to settle the claim is to evaluate the trained model on integers between 10^13 and 10^15 and compare its word accuracy to the Markov chain baseline; if the accuracy drops to the baseline level, the claimed stable extrapolation is falsified. Similarly, if the reported separation from frequency-matched shuffled controls cannot be reproduced when the experiment is run as described in the abstract, the claim that learning goes beyond frequency statistics collapses.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that a transformer can partially infer the internal syntax of the arithmetic text NT, the sequence of Dyck words representing the rooted-tree structure of each integer's prime factorization. In next-word prediction, the model's word accuracy peaks around 0.4 at low sampling temperatures, outperforming a Markov chain baseline; precision, recall, and F1 for primes are each around 0.3, with higher values for square-free words. The model's errors are systematic, confusing primes with square-free numbers. In masked-language modeling, token accuracy exceeds 0.4 in the low-temperature regime and degrades as masking probability and temperature increase. The abstrac

Load-bearing premise

The load-bearing premise is that the control experiments and the 10^13–10^15 test blocks described in the abstract were actually performed and evaluated as claimed, because Section 5 only reports experiments on data up to 10^11 and does not include those controls or distant test blocks.

Editorial extensions

If this is right

  • If the central claim holds, transformers can partially predict prime factorizations from the preceding sequence, even though they never see the integers themselves.
  • The arithmetic text becomes a controlled testbed for studying how sequence models generalize from a known generative rule, since the data-generating law is exactly known.
  • The reported separation from frequency-matched controls would indicate that the model captures sequential, syntax-like structure rather than mere token statistics.
  • The stable performance on blocks far beyond the training interval would suggest that the learned regularities are not local artifacts but reflect properties of the underlying arithmetic.
  • This opens a path toward probing whether a model's latent representations encode algebraic relationships among numbers, as proposed in the paper's future-work section.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The experimental section as printed describes only the dataset up to n=10^11 with ten chunks and no control experiments, no context-length sweep, and no model-size sweep; the abstract's claims about 10^13–10^15 stability and control separation rest on experiments that are not reported in the body of this version.
  • If those control experiments were actually run, a natural extension would be to verify whether the separation from frequency-matched shuffled sequences grows with context length, as the abstract claims, and whether it holds for other architecture sizes.
  • The finding that the model confuses primes with square-free numbers suggests a testable hypothesis: the transformer may be learning a local rule based on the structure of the Dyck word (e.g., the absence of nested parentheses) rather than a genuine global primality criterion.
  • A direct extension would be to train on a shorter prefix (e.g., 10^9 or 10^10) and measure how accuracy at a fixed distant block scales with training data, which would sharpen the distinction between memorization and genuine rule induction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper trains a GPT-2-scale transformer on the sequence of Dyck-word encodings of rooted trees associated with the prime tower factorization of the first 10^11 integers, and evaluates it on next-word prediction and masked language modelling against a Markov-chain baseline. The body reports loss curves, word accuracy and KL divergence as functions of temperature, precision/recall/F1 for frequent Dyck words, confusion matrices, prime-position error analysis, and a temperature/mask-probability sweep for MLM. The abstract and conclusion, however, advertise two further results that are not described anywhere in the experimental section: stable accuracy on test blocks at 10^13--10^15, and likelihood separation from two control texts (frequency-matched shuffled sequences and sequences violating the 'no four consecutive square-free numbers' rule). The same mismatch applies to the advertised 'scaling analysis over context window, dataset size, vocabulary size and model size', which is not present in the body beyond a vocabulary-size comparison.

Significance. If the absent experiments were actually performed and reported, the paper would make a meaningful empirical contribution: demonstrating that a transformer can partially extrapolate a deterministic arithmetic text beyond its training interval and distinguish structural regularities from frequency-matched controls would be a useful probe of learnability in arithmetic sequences. The reported results that are present -- especially the accuracy above the Markov baseline in Figures 3--6 -- represent a modest but real empirical finding for the Dyck-word representation. The paper would also gain from the clear construction of the arithmetic-text dataset and the use of BPE tokenization. As it stands, however, the strongest advertised conclusions rest on experiments that are not described, so the significance of the work cannot be assessed on the evidence provided.

major comments (4)
  1. [Section 5, Data paragraph] The abstract states that 'performance remains stable on test blocks located at 10^13--10^15, far beyond the training interval.' The experimental section describes only one dataset, NT_n with n=10^11, split into 10 chunks; the test set is the last 25% of the 10th chunk (NT_n^(10,vt)), which lies within n=10^11. No experiment with n=10^13, 10^14, or 10^15 is described, and no figure or table reports such test blocks. This is the load-bearing evidence for the claimed extrapolation capability, and it is missing.
  2. [Abstract and Section 5/6] The abstract claims that 'the likelihood assigned by the model separates the arithmetic text from two controls: synthetic sequences reproducing its word frequencies exactly but carrying no sequential organization... and sequences containing more than three consecutive square-free integers.' No control experiment appears anywhere in Section 5 or in Figures 2--8. Consequently, the Section 6 statement that 'the distinction between statistical correlation and structural dependence emerges clearly' is unsupported by the reported results. This distinction is the central point that would separate 'learning grammar' from 'matching frequency profiles', so the experiment must be present or the claim must be withdrawn.
  3. [Abstract and Section 5] The abstract promises 'a scaling analysis over context window, dataset size, vocabulary size and model size.' The body only varies the tokenizer vocabulary size D (64, 256, 1024 for NWP in Figure 2, and 64, 256 for MLM in Figure 7). There is no context-window sweep, no dataset-size sweep, and no model-size sweep (the architecture is fixed to a 12-layer GPT-2 with 8.7e7 parameters). This advertised scaling analysis is not reported in the experimental section.
  4. [Section 6, Conclusions] The conclusion interprets the model's errors as revealing 'the limit of what can be inferred statistically from local dependencies' and claims that 'the model predicts as far as its structure allows.' This boundary claim is not operationalized: no experiment varies the context length or the model capacity while holding data fixed, so the manuscript does not measure a context or capacity horizon. The conclusion goes beyond what the reported results can support.
minor comments (6)
  1. [Section 5.2, first sentence] Typo: 'model tained' should be 'model trained'.
  2. [Equation (3)] The dimensions of Q and K are written as R^{R x d}; the usual convention is d x R (or d x d_k). Please clarify the notation.
  3. [KL_w definition, Section 5.2] The sum in the definition of KL_w runs over all distinct words in w(t), but the first factor f_w(hat w) is defined on words in the true sentence. The support mismatch makes the definition ambiguous; it should be over the union of the two supports or otherwise specified.
  4. [Figure 3] The text says 'red dotted line' for the Markov baseline, while the figure caption describes a 'green dotted line.' Please make the color reference consistent.
  5. [Figure 4 caption] The caption says the x-axis is 'ordered by words’ frequencies,' but no frequency scale is shown. Please provide the ordering or a labeled axis.
  6. [Section 5, Tokenization] The notation uses |ND_{9·10^10}| before defining the training-set index range; please reconcile this with the definition of ND_n in Section 2.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular steps; the main issue is an abstract/full-text evidence gap, not circular reasoning.

full rationale

This is an empirical benchmark paper, not a derivation chain. The transformer is trained from scratch on NT_n and evaluated on a held-out chunk (NT_(10,vt)), so the reported losses, accuracies, and confusion metrics are direct measurements. The Markov baseline is also fitted to the same data, but it is used as a comparison baseline, not as a claimed first-principles prediction. No equation equates a fitted quantity with the claimed result by construction: Eq. (5) and (7) are standard cross-entropy losses, and Eq. (4), (8a), (8b) are evaluation metrics defined against ground truth. The self-citations (Conti and Contucci 2025; Contucci et al. 2025; etc.) provide background statistical/grammatical observations about NT; they are motivational and are not used to construct the test metrics or to compute any prediction, so they are not load-bearing. I found no self-definitional step, no fitted-input-called-prediction, no imported uniqueness theorem, no ansatz smuggled in via citation, and no renaming of a known result. Explicit non-circular flags: (i) The abstract claims stable performance on test blocks at 10^13–10^15 and separation from two controls, but Section 5 describes a single dataset NT_n with n=10^11, split into 10 chunks, with NT_(10,vt) used as test; no control experiments, no 10^13–10^15 blocks, no context-window sweep, and no model-size sweep are reported in the manuscript. (ii) The Section 6 statement that “the distinction between statistical correlation and structural dependence emerges clearly” is therefore unsupported by the reported experiments as written. These are evidentiary/reproducibility concerns, not circularity: they undermine confidence in the abstract's strongest claims but do not make the derivation self-referential.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The central contribution is experimental, so most free parameters are model/training choices rather than physical constants. The axioms are standard number theory/combinatorics plus two domain assumptions about the encoding and data split. No new physical entities are introduced; the only invented conceptual object is the 'grammar' of NT, which lacks independent evidence in the presented manuscript.

free parameters (7)
  • GPT-2 trainable weights = learned on train split; P=8.7e7
    All reported accuracies are properties of these fitted weights; the paper makes no claim that the architecture alone predicts NT without fitting.
  • BPE vocabulary size D = 64, 256, 1024
    Chosen hyperparameter; training loss and word accuracy depend on it (Figs. 2–3).
  • context window length L = 1024 tokens
    Fixed in all reported runs; the abstract's claim that control separation widens with context would require varying L, which is not shown.
  • temperature T = best Aw at 0.1–0.3; best KL at 0.3–0.7; reported ~0.4 accuracy is at a low-T operating point
    Generation temperature is selected after training; the headline accuracy is not parameter-free.
  • mask probability p_m = varied, best accuracy at low p_m and T
    MLM metric depends on this hyperparameter (Fig. 8).
  • early stopping patience = 6 epochs
    Selects the evaluated checkpoint; no independent justification is given.
  • unspecified training hyperparameters
    Learning rate, optimizer, batch size, schedule, number of test samples m, and seeds are not reported; needed to reproduce exact numbers.
assumptions (5)
  • standard math Unique prime factorization and infinitude of primes
    Used to construct τ(n) and to justify that every Dyck word appears infinitely often (Sec. 2).
  • standard math Every rooted planar tree is representable as a Dyck word by contour traversal
    Invoked in Sec. 2 to encode trees as balanced binary strings; standard enumerative combinatorics (Stanley).
  • standard math Four consecutive square-free integers cannot occur
    Used in the abstract to define a forbidden control; no proof or citation is supplied in the body.
  • domain assumption BPE tokenization on the training corpus is lossless and preserves word structure sufficiently for word-level metrics
    Assumed in Sec. 5 Tokenization; if false, token accuracy and reconstructed word accuracy may diverge.
  • domain assumption Statistical properties of later chunks of NT are sufficiently similar that a model trained on the first 90% can be evaluated on the 10th chunk
    Assumed by the chunked split in Sec. 5; the abstract's much farther 10^13–10^15 test blocks would be a stronger version not documented.
invented entities (1)
  • NT's "internal grammar" (the hypothesized syntactic structure of the rooted-tree sequence)
    purpose: The object the transformer is claimed to partially learn; motivates interpreting accuracy as grammar acquisition rather than frequency matching.
    No independent, falsifiable definition of this grammar is given; the only purported evidence is the control experiments, which are described in the abstract and not reported in the body.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees." pith.science (2026). https://pith.science/paper/WG52Y4LP

@misc{pith2026251201870,
  author       = {Pith},
  title        = {Pith review of: Testing Transformer Learnability on the Arithmetic Sequence of Rooted Trees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WG52Y4LP}},
  note         = {Machine review of arXiv:2512.01870}
}
abstract

We study whether a transformer network can learn the deterministic sequence of trees generated by the iterated prime factorization of the natural numbers. Each integer is mapped into a rooted planar tree and the resulting sequence $\mathbb{N}\mathscr{T}$ defines an arithmetic text with measurable statistical structure. A transformer network (the GPT-2 architecture) is trained from scratch on the first $10^{11}$ elements and evaluated on Next-Token and masked-word prediction tasks, with a Hidden Markov Model as baseline and a scaling analysis over context window, dataset size, vocabulary size and model size. The model reaches a word accuracy of about $0.4$, well above the baseline, and its performance remains stable on test blocks located at $10^{13}$--$10^{15}$, far beyond the training interval. Moreover, the likelihood assigned by the model separates the arithmetic text from two controls: synthetic sequences reproducing its word frequencies exactly but carrying no sequential organization, with a separation that widens as the evaluated context grows; and sequences containing more than three consecutive square-free integers, a configuration that arithmetic forbids. These results indicate that the transformer captures regularities of the arithmetic text that go beyond its frequency profile.

Figures

Figures reproduced from arXiv: 2512.01870 by the authors.

Figure 1
Figure 1. On the left, attention map at single layer level for a single head attention in an [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Loss curves for a model trained via NWP, obtained for different tokenizer’s [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Accuracy for words (Aw, left) and Kullback-Leibler divergence for word distri￾bution (KLw, right) are reported at different temperature values T, averaged over a set of 32 different inputs for each T. The green dotted line represents the Markov Chain (MC) model baseline, while the black dash-dotted line identifies the large temperature limit. 101010 10101010 1010 11001010 1100101010 1010101010 110010101010 110010 10… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Precision, recall and F1 score for a set of L = 1024 generated tokens. The x-axis is ordered by words’ frequencies in the input sentences, namely the true sequence. On the left, result for our LLM model; on the right, corresponding quantities for the Markov baseline. s…
Figure 5
Figure 5. Figure 5: Distributions of the generated Dyck words at real prime positions (left) and of [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Confusion matrix for next word prediction task for the LLM (left) and the [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Loss curves for an architecture trained via masked language modelling. Here it [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Tokens accuracy as function of masking probability and temperature [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

5 extracted references · 1 canonical work pages

  1. [1]

    Practical recommendations for gradient-based training of deep ar- chitectures

    Bengio, Yoshua (2012). “Practical recommendations for gradient-based training of deep ar- chitectures”. In:Neural networks: Tricks of the trade: Second edition. Springer, pp. 437–

    2012

  2. [2]

    Primary cyclotomic units and a proof of Catalans conjecture

    Kemeny, John G. and J. Laurie Snell (1983).Finite Markov Chains. Undergraduate Texts in Mathematics. Springer New York.isbn: 9780387901923. Kolpakov, Alexander and Aidan Rocke (2023).On the Impossibility of Discovering a For- mula for Primes Using AI. arXiv:2308.10817 [cs.CC]. — (2024).Machine Learning of the Prime Distribution. arXiv:2403.12588 [cs.IT]. ...

    arXiv 1983

  3. [9]

    A distributional simplicity bias in the learning dynamics of transformers

    20 Rende, Riccardo, Federica Gerace, et al. (2024a). “A distributional simplicity bias in the learning dynamics of transformers”. In:Advances in Neural Information Processing Systems37, pp. 96207–96228. — (2024b). “Mapping of attention mechanisms to a generalized potts model”. In:Physical Review Research6.2, p. 023057. Rende, Riccardo, Luciano Loris Viter...

    arXiv 2023

  4. [413]

    Algorithms of the M¨ obius Function by Random Forests and Neural Networks

    EDP Sciences, p. 06009.doi:10.1051/e3sconf/ 202341306009. Qin, Huan and Yangbo Ye (2024). “Algorithms of the M¨ obius Function by Random Forests and Neural Networks”. In:Journal of Big Data11.31, pp. 1–16.doi:10.1186/s40537- 024-00889-7. Radford, Alec, Karthik Narasimhan, et al. (2018).Improving language understanding by generative pre-training. Radford, ...

    work page doi:10.1051/e3sconf/ 2024

  5. [478]

    Language models are few-shot learners

    Blake, Sam (2023).Integer Factorisation, Fermat and Machine Learning on a Classical Computer. arXiv:2308.12290 [cs.LG]. Brown, Tom et al. (2020). “Language models are few-shot learners”. In:Advances in neural information processing systems33, pp. 1877–1901. Cagnetta, Francesco et al. (2025).Scaling Laws and Representation Learning in Simple Hierarchical L...

    arXiv 2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.