CALT: A Library for Computer Algebra with Transformer

Hiroshi Kera; Shun Arakawa; Yuta Sato

REVIEW 3 major objections 5 minor 19 references

CALT: A Library for Computer Algebra with Transformer

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CALT is a Python library that makes training Transformers for symbolic computation as easy as writing a small input-output generator; its four showcase tasks succeed at reported rates of 31.7–81.2%.

desk verdict A useful but incremental library with a potentially serious train/test overlap in its flagship factorization demo. read the letter →

arxiv 2506.08600 v1 pith:6KXYVTN4 submitted 2025-06-10 cs.LG cs.SCmath.AC

Hiroshi Kera , Shun Arakawa , Yuta Sato This is my paper

classification cs.LGcs.SCmath.AC

keywords computeralgebraTransformersymboliccomputationsequence-to-sequencelearningchain-of-thoughtpolynomialmultiplicationintegerfactorizationPythonlibrary

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces CALT, a Python library intended to make end-to-end Transformer training for symbolic computation accessible to researchers who are not deep-learning specialists. The central claim is one of packaging: in the minimal use case a user only implements a ProblemGenerator class that returns input–output pairs such as (integer, prime factorization) or (factor tuple, product), and the library manages parallel dataset construction, tokenization, and the training loop. Four showcase tasks are trained on 100,000 generated samples and tested on 1,000, with complete-solution success rates of 74.5% for integer factorization, 81.2% for polynomial multiplication over $\mathbb{Z}$, 31.7% over $\mathbb{F}_7$, and 74.2% over $\mathbb{F}_7$ when the goal is the chain of partial products. The paper argues that these results make the library a feasible entry point and that the interesting open problems shift to generating diverse training instances, such as input polynomials whose Gröbner bases are nontrivial.

What carries the argument

The load-bearing component is the ProblemGenerator interface: a user-defined class whose constructor fixes the task specification and whose call() method returns a single input–output pair $(F, G)$. The library parallelizes calls to this routine to build a dataset, tokenizes pairs with a configurable vocabulary (for example, exponent tokens $E_a$ and coefficient tokens $C_b$ for polynomial rings), and trains an autoregressive Transformer by next-token prediction. The paper's second mechanism is chain-of-thought: in the $\mathbb{F}_7$ task the target is the ordered sequence of partial products $g_1=f_1$, $g_2=f_1f_2$, $\ldots$, $g_s=f_1\cdots f_s$ rather than the final product, which is credited with most of the accuracy gain over the non-CoT version.

What would settle it

Rerun the four showcase tasks with the released code under multiple random seeds and with two different generator settings; if success-rate distributions are wide (say, factorization ranging from 40% to 90% across seeds) or a task fails to reproduce, the paper's evidence for the library's general utility is not stable.

Watch

Extended reading notes

Core claim

The paper's discovery is that the whole training stack for a symbolic computation can be reduced, for the user, to a single instance-generation routine. CALT's pipeline splits into three pieces—instance generator, tokenizer, and Transformer trainer—and the library implements the latter two, so that writing a class whose call() returns one fresh $(F, G)$ pair is the only essential user contribution. The reported experiments are the evidence for this packaging claim: with a standard Transformer, 100,000 samples, and 80k optimization steps, the models learn to factor products of two to five primes up to 100 and to expand sparse bivariate polynomials, with chain-of-thought targets lifting the $\mathbb{F}_7$ success rate from 31.7% to 74.2%. The authors interpret the $\mathbb{F}_7$ gap as consistent with known Transformer difficulty on modular arithmetic and the CoT improvement as evidence that stepwise targets help the model decompose the computation.

Load-bearing premise

The load-bearing premise is that the four reported success rates are stable and representative, because each comes from a single training run on a small synthetic task, with no seeds, confidence intervals, or baseline comparisons.

Editorial extensions

If this is right

For any symbolic task, the research problem becomes designing a generator of input–output pairs; once that class exists the rest of the pipeline is library code.
Chain-of-thought targets offer a practical way to improve accuracy on tasks Transformers find hard, by replacing one difficult prediction with a sequence of simpler ones.
The library's text-file input mode lets users combine hand-written examples with programmatically generated data, which is useful when a task distribution is hard to sample uniformly.
Modular-arithmetic variants remain the weak case, so users should expect lower accuracy there or plan to add intermediate outputs.
Computer algebra researchers can now test whether learned models are fast or reliable enough to assist exact algorithms, without first becoming deep-learning engineers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Going beyond the paper: if the success rates are not stable across seeds, the library can still be useful as a prototyping tool, but the showcase percentages should not be read as benchmark guarantees.
Going beyond the paper: the same interface is a natural testbed for scaling studies—varying factor counts, coefficient ranges, or degrees and watching accuracy decay would make the library a tool for studying what makes symbolic computations learnable.
Going beyond the paper: the CoT result suggests output formatting is a first-class design choice for symbolic learning; rewriting a computation as intermediate normal forms may matter as much as architecture or data size.
Going beyond the paper: the library's generator-first design pairs naturally with backward data generation—sample a desired output and derive an input—which could make expensive forward computations (like Gröbner bases) practical to train on.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is a software contribution: CALT, a Python/SageMath library that wraps PyTorch and HuggingFace to make it easy for computer algebraists to train Transformers on symbolic tasks. The genuinely new piece is the streamlined ProblemGenerator interface plus SageMath integration; the authors themselves say CALT is functionally a subset of PyTorch/HuggingFace. That is honest, but it also means the novelty is incremental.

What the paper does well: it is clearly written, the code is public, the pipeline is standard, and the four showcase tasks cover relevant cases. The CoT result over F7 (74.2% vs 31.7%) is consistent with prior findings and is the kind of observation that could be useful to the community.

Soft spots, in order of importance. First, the integer factorization showcase is likely confounded by train/test overlap. The dataset generation samples t uniform in {2,...,5}, t distinct primes up to 100, product as input. That yields at most 68,380 distinct factorization instances. Training on 100k without saying the test set is disjoint means roughly 81% of test items already appeared in training. The reported 74.5% success rate is below what a memorizing model could hit, so this row does not demonstrate generalization at all. The paper should either generate disjoint splits or drop factorization. Second, all success rates come from single runs: no seeds, no confidence intervals, no baseline comparisons. For a library whose central claim is accessibility for non-experts, that is thin evidence. Third, there is no usability evaluation, so 'user-friendly' is asserted rather than measured.

None of this kills the artifact. The polynomial tasks have huge input spaces and the 81.2% over Z is non-trivial, but without variance or baselines it is anecdotal. The paper would be materially stronger with a few seeds, a simple baseline (e.g., n-gram or LSTM), and a disjoint factorization split.

My take: it is a reasonable system paper for a specialized audience. I would not desk-reject it. It deserves a serious referee, but the referee should insist on fixing the factorization overlap and adding basic robustness evidence. If the authors cannot add seeds, they should soften their claims accordingly.

Referee Report

3 major / 5 minor

Summary. The paper introduces CALT, a Python/SageMath library designed to let researchers in symbolic computation train Transformer models without deep expertise. The user implements a ProblemGenerator class that returns input–output pairs; CALT then handles parallel dataset generation, tokenization, and training. The authors demonstrate the library on four tasks: integer factorization, polynomial multiplication over Z, polynomial multiplication over F7, and polynomial multiplication over F7 with chain-of-thought, reporting success rates between 31.7% and 81.2%. The public repository is linked, and the paper positions CALT as an accessibility layer over PyTorch and HuggingFace Transformers. The evidence is empirical; there is no formal derivation of the library's properties.

Significance. If the library works as described, it is a useful community resource: it addresses a real onboarding problem by packaging data generation, tokenization, and training into a small interface, and the public availability of the code is a concrete strength. The showcases demonstrate that the pipeline can produce non-trivial accuracy on several symbolic tasks, and the observed benefit of chain-of-thought on F7 is a plausible and interesting sanity check. However, the central claim of utility for non-experts is not yet fully supported: the experiments are single runs with no seeds or confidence intervals, there is no comparison with existing frameworks, the factorization showcase is confounded by train/test overlap, and the 'user-friendly' claim is not evaluated directly. The paper is best read as a preliminary library announcement with promising but incomplete evidence.

major comments (3)

[§4, Dataset generation (Prime factorization)] The paper does not state that the 100,000 training samples and 1,000 test samples are disjoint, and for factorization they cannot be: there are only C(25,2)+C(25,3)+C(25,4)+C(25,5)=68,380 distinct instances, so a random test item already appears in the training set with probability about 0.96 (essentially 1 for t=2 and t=3, about 0.85 for t=5). The reported 74.5% success rate is therefore below the memorization ceiling and does not demonstrate that the model computes prime factorizations on unseen inputs. The authors should re-run the factorization showcase on a held-out set that is guaranteed to be disjoint from training, or otherwise report accuracy on genuinely unseen instances separately from memorized ones.
[§4, Training setup and Results (Table 2)] All four reported success rates come from a single training run with no random seeds, confidence intervals, or variance estimates. Since the central claim is that the library reliably enables non-experts to train useful models, the reader cannot tell whether the Table 2 numbers are stable properties of the pipeline or artifacts of one initialization and one data draw. I request multiple seeds (at least three, preferably five) with mean and standard deviation, and a comparison with the closest existing codebase (Int2Int, Ref. [3]) on the same polynomial tasks.
[§1, Abstract and §3] The paper's headline claim is that CALT is 'user-friendly' for non-experts in deep learning, but no usability evaluation is reported: there is no timing, no measure of code length or complexity, no user study, and no comparison with the effort required to train the same models directly in PyTorch/HuggingFace. A minimal end-to-end example with wall-clock time and a short qualitative usability assessment would substantiate this load-bearing claim.

minor comments (5)

[§2.3] There is a typo in 'as folows' which should read 'as follows'.
[§1, Key features list] The text contains 'CAL Toffers' with a missing space; it should be 'CALT offers'.
[§3, Figure 1] The text says 'Figure 1 shows example codes for instance generations', but the figure appears to be a pipeline diagram rather than code. Please make the figure content consistent with the prose, or include the code listing in the figure or an appendix.
[§4, Results (Table 2)] The definition of 'success' is not stated explicitly; please clarify whether a prediction is counted as correct only when the entire output sequence matches the ground-truth token sequence exactly.
[§4, Training setup] The model architecture is described only as 'the standard architecture of Transformer [18]'; please specify the number of layers, hidden dimension, number of attention heads, and parameter count so that the experiments are reproducible.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: CALT is a library paper whose showcased success rates are empirical measurements, not outputs of a fitted or self-referential chain.

full rationale

The paper contains no derivation chain that could collapse into its own inputs. Its central artifacts are (i) a software library that wraps PyTorch/HuggingFace training around a user-supplied instance generator, and (ii) four empirical success-rate measurements reported in Table 2. The training pipeline is not an inference from a model; it is an implementation. The cited prior works of the authors ([10], [9], [11]) are used only as motivation for why symbolic-computation researchers might want a library; they are not parameters, assumptions, or theorems imported to derive the reported results, so they are not load-bearing in a circular sense. The reported percentages are measurements on generated test items, and no fitted parameter is renamed as a prediction. The only substantive validity concern is the reviewer-identified train/test overlap in the prime-factorization showcase: with 100k training samples and roughly 68k distinct products of 2-5 distinct primes below 100, a large fraction of the 1,000 test items probably appears identically in training, so the 74.5% rate is weak evidence of generalization. That is a data-leakage and memorization concern, not circularity under the enumerated patterns: the success rate is not equivalent by construction to the training data, and the paper never claims the factorization result was verified on disjoint inputs. The polynomial tasks have large input spaces and are not affected by exact overlap, though they still rely on single runs without seeds or confidence intervals. Those are correctness-risk and reproducibility issues, not circularity. Honest non-finding is therefore appropriate: score 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new mathematical objects or fitted parameters. It relies on the standard assumption that transformers can learn symbolic mappings and that the underlying computer algebra system is correct.

assumptions (3)

domain assumption Transformer models can learn symbolic computation tasks from input-output example pairs.
The library's whole function is to make such training easier. The learnability claim is supported by cited prior work (Lample and Charton, Kera et al.) but not proven in this paper.
domain assumption SageMath computes the ground-truth products and factorizations used to build training data.
Data generation in Section 3 relies on SageMath; if it produced wrong outputs, the training labels would be wrong.
domain assumption The chosen standard Transformer architecture and hyperparameters (learning rate 5e-5, batch 128, dropout 0.1, 80k steps) are sufficient to learn the four tasks.
No architecture search or justification is given; the paper relies on the reported empirical success.

how reviews work

0 comments

Cite this review

Pith. "Pith review of CALT: A Library for Computer Algebra with Transformer." pith.science (2026). https://pith.science/paper/6KXYVTN4

@misc{pith2026250608600,
  author       = {Pith},
  title        = {Pith review of: CALT: A Library for Computer Algebra with Transformer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6KXYVTN4}},
  note         = {Machine review of arXiv:2506.08600}
}

read the original abstract

Recent advances in artificial intelligence have demonstrated the learnability of symbolic computation through end-to-end deep learning. Given a sufficient number of examples of symbolic expressions before and after the target computation, Transformer models - highly effective learners of sequence-to-sequence functions - can be trained to emulate the computation. This development opens up several intriguing challenges and new research directions, which require active contributions from the symbolic computation community. In this work, we introduce Computer Algebra with Transformer (CALT), a user-friendly Python library designed to help non-experts in deep learning train models for symbolic computation tasks.

Figures

Figures reproduced from arXiv: 2506.08600 by the authors.

**Figure 2.** Loss curves for each task during training. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

[3]

Int2Int : a framework for mathematics with transformers, 2025

Charton, F. Int2Int : a framework for mathematics with transformers, 2025

work page 2025
[1]

Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers, 2024

Alfarano, A., Charton, F., and Hayat, A. Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers, 2024

work page 2024
[2]

Linear algebra with transformers

Charton, F. Linear algebra with transformers. Transactions on Machine Learning Research\/ (2022)

work page 2022
[4]

Decision transformer: Reinforcement learning via sequence modeling

Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems\/ (2021)

work page 2021
[5]

An image is worth 16x16 words: Transformers for image recognition at scale

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations\/ (2021)

work page 2021
[6]

R.-S., Tesileanu, T., Cho, K., and Ho, S

Golkar, S., Pettee, M., Eickenberg, M., Bietti, A., Cranmer, M., Krawezik, G., Lanusse, F., McCabe, M., Ohana, R., Parker, L., Blancard, B. R.-S., Tesileanu, T., Cho, K., and Ho, S. xVal : A continuous number encoding for large language models, 2023

work page 2023
[7]

Deep Learning

Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning . MIT Press, 2016. http://www.deeplearningbook.org

work page 2016
[8]

Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX , 2024

Hugging Face Inc. Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX , 2024. https://huggingface.co/docs/transformers

work page 2024

Show all 19 references

[9]

Geometric generality of transformer-based gr\"obner basis computation, 2025

Kambe, Y., Maeda, Y., and Vaccon, T. Geometric generality of transformer-based gr\"obner basis computation, 2025

2025
[10]

Learning to compute Gr\"obner bases, 2024

Kera, H., Ishihara, Y., Kambe, Y., Vaccon, T., and Yokoyama, K. Learning to compute Gr\"obner bases, 2024

2024
[11]

Computational algebra with attention: Transformer oracles for border basis algorithms, 2025

Kera, H., Pelleriti, N., Ishihara, Y., Zimmer, M., and Pokutta, S. Computational algebra with attention: Transformer oracles for border basis algorithms, 2025

2025
[12]

Transformers provably solve parity efficiently with chain of thought

Kim, J., and Suzuki, T. Transformers provably solve parity efficiently with chain of thought. In The Thirteenth International Conference on Learning Representations\/ (2025)

2025
[13]

Deep learning for symbolic mathematics

Lample, G., and Charton, F. Deep learning for symbolic mathematics. In International Conference on Learning Representations\/ (2020)

2020
[14]

P yTorch: The P yTorch Machine Learning Framework , 2024

PyTorch Core Team . P yTorch: The P yTorch Machine Learning Framework , 2024. https://pytorch.org

2024
[15]

Teaching transformers modular arithmetic at scale, 2024

Saxena, E., Alfarano, A., Wenger, E., and Lauter, K. Teaching transformers modular arithmetic at scale, 2024

2024
[16]

T., Li, Y., and Zhang, Y

Shen, R., Bubeck, S., Eldan, R., Lee, Y. T., Li, Y., and Zhang, Y. Positional description matters for transformers arithmetic, 2023

2023
[17]

S ageMath, the S age M athematics S oftware S ystem ( V ersion 10.0) , 2023

The Sage Developers . S ageMath, the S age M athematics S oftware S ystem ( V ersion 10.0) , 2023. https://www.sagemath.org

2023
[18]

N., Kaiser, L

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems\/ (2017), vol. 30, Curran Associates, Inc

2017
[19]

Wenger, E., Chen, M., Charton, F., and Lauter, K. E. SALSA : Attacking lattice cryptography with Transformers . In Advances in Neural Information Processing Systems\/ (2022), vol. 35, pp. 34981--34994

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

Int2Int : a framework for mathematics with transformers, 2025

Charton, F. Int2Int : a framework for mathematics with transformers, 2025

work page 2025

[2] [1]

Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers, 2024

Alfarano, A., Charton, F., and Hayat, A. Global Lyapunov functions: a long-standing open problem in mathematics, with symbolic transformers, 2024

work page 2024

[3] [2]

Linear algebra with transformers

Charton, F. Linear algebra with transformers. Transactions on Machine Learning Research\/ (2022)

work page 2022

[4] [4]

Decision transformer: Reinforcement learning via sequence modeling

Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In Advances in Neural Information Processing Systems\/ (2021)

work page 2021

[5] [5]

An image is worth 16x16 words: Transformers for image recognition at scale

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations\/ (2021)

work page 2021

[6] [6]

R.-S., Tesileanu, T., Cho, K., and Ho, S

Golkar, S., Pettee, M., Eickenberg, M., Bietti, A., Cranmer, M., Krawezik, G., Lanusse, F., McCabe, M., Ohana, R., Parker, L., Blancard, B. R.-S., Tesileanu, T., Cho, K., and Ho, S. xVal : A continuous number encoding for large language models, 2023

work page 2023

[7] [7]

Deep Learning

Goodfellow, I., Bengio, Y., and Courville, A. Deep Learning . MIT Press, 2016. http://www.deeplearningbook.org

work page 2016

[8] [8]

Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX , 2024

Hugging Face Inc. Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX , 2024. https://huggingface.co/docs/transformers

work page 2024