REVIEW 3 major objections 5 minor 22 references
This paper claims that large language models perform arithmetic on the surface form of numbers: changing only the script or grouping punctuation of otherwise identical expressions drops accuracy by roughly 66–87%, and that few-shot examples
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 08:54 UTC pith:A2FZOQ7Y
load-bearing objection The script-arithmetic claim conflates input reading with output generation; useful benchmark, but the headline needs a redesign. the 3 major comments →
The Effect of Scripts and Formats on LLM Numeracy
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On its own terms, the paper establishes that LLM arithmetic is representation-dependent. Across four large models, moving operands from Hindu-Arabic digits to any of 20 other numeral scripts lowers arithmetic accuracy by roughly 66–87%, and all non-default formatting conventions (F2–F6) show large negative coefficients in a mixed-effects model. Script identification and numeral translation are strongly correlated, and tokens-per-digit is a significant negative factor, indicating that fragmented tokenization contributes to the drop. The paper further shows that prompting with the full expression in the native language plus an explicit digit mapping, or using few-shot formatted examples, subst
What carries the argument
The central analytic tool is a logistic generalized linear mixed-effects model (GLMER) with random intercepts for model and problem index, fitted to accuracy across scripts, formats, and prompting strategies. It isolates the "script tax"—negative coefficients for every non-Hindu-Arabic script relative to the Hindu-Arabic reference—and a tokens-per-digit feature that quantifies tokenization fragmentation. The key intervention is explicit few-shot prompting with numeral mappings, which supplies a reference representation; its coefficient is the largest positive effect in both the script and format models.
Load-bearing premise
The paper's explanation that accuracy loss is caused by underrepresentation in training data rests on estimating script and format frequencies from four public pretraining corpora; the actual pretraining mixes of the closed models are not public, so the causal link between corpus frequency and model behavior is inferred, not observed.
What would settle it
Train or fine-tune a model on data where a rare script, such as Ol Chiki numerals, is artificially overrepresented in arithmetic expressions, then compare accuracy in that script against a matched control script. If accuracy does not improve with increased exposure, the underrepresentation explanation and the "script tax" reading would be wrong, even though raw accuracy differences would remain.
If this is right
- Multilingual applications that feed LLMs numbers in Devanagari, Thai, Arabic-Indic, or other non-Hindu-Arabic scripts will see systematic arithmetic failures even when the model can read the script.
- Arithmetic accuracy in a script broadly tracks how often that script's digits appear in common pretraining corpora; very rare scripts such as Ol Chiki, Adlam, and Osmanya drop to near-zero accuracy.
- Tokenization is a measurable contributor: scripts that fragment into more subword tokens per digit are processed less accurately.
- Few-shot prompting with just two examples in the target format recovers 20–60% of the lost accuracy, and explicit digit mapping helps across scripts, so representation gaps can be partially closed without retraining.
- Standard benchmarks that use only Hindu-Arabic, Western-formatted numerals overstate LLM numerical competence in real-world multilingual settings.
Where Pith is reading between the lines
- If underrepresentation is the true driver, artificially oversampling a rare script in pretraining data should largely close the accuracy gap without architectural changes; the paper's data make this a testable prediction but do not prove it.
- The same methodology could be applied to other representational dimensions—number words in low-resource languages, currency formats, date formats—likely revealing similar representation-conditioned accuracy cliffs.
- The large script penalty suggests arithmetic is performed on surface tokens rather than abstract quantities, which would imply that chain-of-thought or tool-calling may be more robust than direct answer generation; the paper does not test this.
- For production systems, the findings imply a cheap robustness fix: detect the numeral script or format in the input and include a small target-format example in the prompt before asking for computation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how LLM arithmetic accuracy changes when numerals are rendered in non-Hindu-Arabic scripts and in non-default formatting conventions. It builds a dataset of 336 base arithmetic expressions, translates them into 21 numeral scripts and 6 formatting styles, evaluates 9 LLMs on script identification, numeral translation, arithmetic computation, and formatted-arithmetic tasks, and fits GLMER models to quantify the effects of script, format, tokenization, operation, and prompting strategy. The central claim is that LLM numeracy is strongly tied to the surface representation of numerals: shifting from Hindu-Arabic to non-HA scripts causes roughly a 66–87% accuracy drop, and alternative formatting conventions cause large drops as well, but targeted prompting (native-language prompts, digit mapping, few-shot examples) recovers much of the loss.
Significance. If the central claim holds, the paper provides a systematic, controlled demonstration that LLM arithmetic competence is representation-dependent, with practical implications for multilingual evaluation and prompting. Strengths of the work include the direct measurement of accuracy on held-out items, the use of a repeated-measures GLMER that is appropriate for the within-item design, the breadth of scripts and formats considered, and the explicit attempt to quantify corpus prevalence from multiple large corpora. The raw accuracy differences and the effectiveness of prompting interventions are valuable regardless of the causal interpretation. However, the headline causal interpretation—that the drop reflects difficulty in interpreting non-standard input scripts—is weakened by a confound in the task design, and the underrepresentation explanation rests on corpus proxies rather than actual pretraining distributions.
major comments (3)
- [Sec. 3.2.1 and App. C.4] The multiscript arithmetic task requires the model to 'Ensure the answer is in the same script as the numbers in the question.' Thus the non-HA condition differs from the HA baseline in both the input script and the required output script. The reported 66–87% drop therefore conflates input-script comprehension with output-script generation. This is not a pedantic point: Table 3 shows that the same models can translate non-HA digits to HA with 0.77–0.92 average accuracy, so the inputs are often legible. Without an error analysis separating 'correct numeric value, wrong output script' from 'wrong numeric value,' the conclusion in Sec. 6 that failures stem from 'a lack of contextual grounding for interpreting non-standard numeric surface forms' is not supported. The authors should either score outputs in HA, accept any script that represents the correct numeric value, or provide a script-co
- [App. C.1 / D.1 and Sec. 3.3] The claim that accuracy loss is driven by underrepresentation depends on corpus-frequency estimates from PILE, RedPajama, OSCAR, and FineWeb2. The actual pretraining distributions of Claude, GPT-4o, Gemini, and Llama are not public, and the four sampled corpora may not reflect those distributions. This concern is not circularity—the accuracy measurements are independent of the frequency estimates—but it is load-bearing for the 'script tax' interpretation. The paper's raw accuracy results are still meaningful as a behavioral finding. To support the causal story, the authors could report token-frequency statistics in the models' own tokenizers, use controlled perturbation experiments with known exposure, or explicitly soften the causal language. The corpus proxy should be flagged as a limitation in the main text, not only in the appendix.
- [Table 4 and Sec. 3.3] The GLMER that quantifies the 'script tax' excludes Gemini 2.5 Pro because its low accuracy creates imbalance, and excludes five low-resource scripts (Adlam, Osmanya, N'Ko, Balinese, Javanese) because performance was often zero. This is a defensible modeling choice, but it means the estimated script coefficients apply only to the surviving scripts and models. More importantly, for the multiscript arithmetic task, the GLMER includes prompting-condition fixed effects but the paper does not report whether the same-script output requirement was included as a factor or how output-format errors were scored. Since the script coefficients are negative even after controlling for tokens per digit, the authors should explicitly state whether the coefficients reflect a pure input effect or an input-plus-output confound. The current presentation overstates the support for the input-comprehension mech
minor comments (5)
- [App. B] The removal of 109 of 500 expressions because all models failed, and a further 55 because they could not be consistently classified into formatting variants, is reported only in the appendix. This selection may affect the reported accuracy magnitudes. A sentence in Sec. 2 or 3 quantifying the removal and stating that the main comparisons survive on the retained items would improve transparency.
- [Table 1 and Sec. 2.1] The text says 'translated into 20 distinct numeral scripts' but Table 1 lists 21 scripts including Hindu-Arabic. This is likely intentional (20 non-HA scripts plus the HA baseline) but should be stated explicitly to avoid confusion. The same ambiguity appears in the translation task description ('20 scripts').
- [Fig. 2 and Fig. 3] The dots are described as the maximum accuracy among non-HA scripts, but the caption does not indicate whether the bars are averages across all non-HA scripts or across models. Clarify the aggregation level in the captions.
- [App. D.4] The error analysis for formatted numbers is detailed and useful, but the paper does not provide the analogous breakdown for the multiscript arithmetic condition. Adding such a breakdown would directly address the major concern above and would strengthen the paper.
- [Sec. 3.1] The correlation between script identification and translation accuracy is reported as 0.7022 (p = 3.95e−13). It would be helpful to state whether this is Pearson or Spearman and over what units (script-model pairs). Also, the claim that translation errors arise 'from upstream difficulties in script recognition' is stronger than the correlation alone supports; the correlation is suggestive but not direct evidence of a causal chain.
Circularity Check
No significant circularity: the accuracy results are direct measurements and the corpus-frequency estimates come from independent external corpora, not from the model outputs or from a fitted parameter renamed as a prediction.
full rationale
This paper is an empirical evaluation, not a formal derivation. The central quantity—accuracy on arithmetic with non-HA scripts and non-F1 formats—is measured directly on held-out test items generated from 336 base expressions. The 'underrepresented' labels are computed in Appendix C.1 and D.1 by sampling 500k documents from PILE, RedPajama, OSCAR, and FineWeb2; these corpus estimates are independent of the evaluation data and of the model responses. The GLMER models in Sections 3.3 and 4.3 are explanatory regressions fit to the measured accuracies to estimate fixed effects; they do not synthesize the headline drops by construction, because the script and format coefficients are estimated from the observed responses rather than imposed. No fitted parameter is later renamed as a prediction, and no load-bearing result is imported through self-citation or an author-imported uniqueness theorem. The reviewer-flagged 'same script' output requirement is a genuine possible confound for the mechanistic interpretation that failures are due to input interpretation rather than output generation, but a confound is not circularity: the reported accuracy differences remain direct observations. Accordingly, no circular step meets the quoted-reduction standard, and the honest finding is a score of 0.
Axiom & Free-Parameter Ledger
free parameters (2)
- sampling temperature =
0.7 for multiscript tasks; 0.3 for formatted-number tasks
- few-shot example count =
2 examples
axioms (4)
- domain assumption Corpus-frequency estimates from PILE, RedPajama, OSCAR, and FineWeb2 (500k sampled documents each) reflect the relative exposure of the evaluated LLMs to each numeral script and format.
- domain assumption The 336-expression dataset, its script translations, and format variants are correct and preserve the intended arithmetic.
- domain assumption Exact-match answer evaluation is a valid measure of numerical reasoning for these tasks.
- standard math The logistic GLMER with random intercepts for model and problem index yields unbiased coefficient estimates under the stated specification.
Cite this review
Pith. "Pith review of The Effect of Scripts and Formats on LLM Numeracy." pith.science (2026). https://pith.science/paper/A2FZOQ7Y
@misc{pith2026260115251,
author = {Pith},
title = {Pith review of: The Effect of Scripts and Formats on LLM Numeracy},
year = {2026},
howpublished = {\url{https://pith.science/paper/A2FZOQ7Y}},
note = {Machine review of arXiv:2601.15251}
}
read the original abstract
Large language models (LLMs) have achieved impressive proficiency in basic arithmetic, rivaling human-level performance on standard numerical tasks. However, little attention has been given to how these models perform when numerical expressions deviate from the prevailing conventions present in their training corpora. In this work, we investigate numerical reasoning across a wide range of numeral scripts and formats. We show that LLM accuracy drops substantially when numerical inputs are rendered in underrepresented scripts or formats, despite the underlying mathematical reasoning being identical. We further demonstrate that targeted prompting strategies, such as few-shot prompting and explicit numeral mapping, can greatly narrow this gap. Our findings highlight an overlooked challenge in multilingual numerical reasoning and provide actionable insights for working with LLMs to reliably interpret, manipulate, and generate numbers across diverse numeral scripts and formatting styles.
Figures
Reference graph
Works this paper leans on
-
[1]
Tanja Baeumel, Josef Van Genabith, and Simon Ostermann. 2025. https://doi.org/10.18653/v1/2025.blackboxnlp-1.15 The lookahead limitation: Why multi-operand addition is hard for LLM s . In Proceedings of the 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 250--262, Suzhou, China. Association for Computational Linguistics
-
[2]
Douglas Bates, Martin M \"a chler, Ben Bolker, and Steve Walker. 2015. https://doi.org/10.18637/jss.v067.i01 Fitting linear mixed-effects models using lme4 . Journal of Statistical Software, 67(1):1--48
-
[3]
Antara Raaghavi Bhattacharya, Isabel Papadimitriou, Kathryn Davidson, and David Alvarez-Melis. 2025. https://doi.org/10.18653/v1/2025.emnlp-main.1438 Investigating the interaction of linguistic and mathematical reasoning in language models using multilingual number puzzles . In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Pr...
-
[4]
N. E. Breslow and D. G. Clayton. 1993. https://doi.org/10.1080/01621459.1993.10594284 Approximate inference in generalized linear mixed models . Journal of the American Statistical Association, 88(421):9--25
arXiv 1993
-
[5]
Minh Duc Bui, Kyung Eun Park, Goran Glava s , Fabian David Schmidt, and Katharina Von Der Wense. 2025. https://doi.org/10.18653/v1/2025.acl-long.1032 On generalization across measurement systems: LLM s entail more test-time compute for underrepresented cultures . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (V...
-
[6]
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . Preprint, arXiv:2101.00027
Pith/arXiv arXiv 2020
-
[7]
Mayank Jobanputra, Nils Philipp Walter, Maitrey Mehta, Blerta Veseli, Evan Parker Kelly Chapple, Yifan Wang, Sneha Chetani, Ellie Pavlick, Antonio Vergari, and Vera Demberg. 2025. https://arxiv.org/abs/2511.02795 Can llms subtract numbers? Preprint, arXiv:2511.02795
arXiv 2025
-
[8]
Linus Kreitner, Paul Hager, Jonathan Mengedoht, Georgios Kaissis, Daniel Rueckert, and Martin J. Menten. 2025. https://arxiv.org/abs/2510.06824 Efficient numeracy in language models through single-token number embeddings . Preprint, arXiv:2510.06824
Pith/arXiv arXiv 2025
-
[9]
Thomas McCoy, Ellie Pavlick, and Tal Linzen
R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. https://doi.org/10.18653/v1/P19-1334 Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428--3448, Florence, Italy. Association for Computational Linguistics
-
[10]
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2021. https://arxiv.org/abs/2102.13019 Investigating the limitations of transformers with simple arithmetic tasks . Preprint, arXiv:2102.13019
Pith/arXiv arXiv 2021
-
[11]
Pedro Javier Ortiz Su \'a rez, Beno \^i t Sagot, and Laurent Romary. 2019. https://doi.org/10.14618/IDS-PUB-9021 Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures
-
[12]
Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Amir Hossein Kargaran, Colin Raffel, Martin Jaggi, Leandro Von Werra, and Thomas Wolf. 2025. https://arxiv.org/abs/2506.20920 Fineweb2: One pipeline to scale them all -- adapting pre-training data processing to every language . Preprint, arXiv:2506.20920
Pith/arXiv arXiv 2025
-
[13]
Jan-Thorsten Peter, David Vilar, Tobias Domhan, Dan Malkin, and Markus Freitag. 2025. https://arxiv.org/abs/2511.05162 Mind the gap... or not? how translation errors and evaluation details skew multilingual results . Preprint, arXiv:2511.05162
Pith/arXiv arXiv 2025
-
[14]
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. https://arxiv.org/abs/2210.03057 Language models are multilingual chain-of-thought reasoners . Preprint, arXiv:2210.03057
Pith/arXiv arXiv 2022
-
[15]
Aaditya K. Singh and DJ Strouse. 2024. https://arxiv.org/abs/2402.14903 Tokenization counts: the impact of tokenization on arithmetic in frontier llms . Preprint, arXiv:2402.14903
Pith/arXiv arXiv 2024
-
[16]
Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin. 2023. https://arxiv.org/abs/2310.10358 Tabular representation, noisy operators, and impacts on table structure understanding tasks in llms . Preprint, arXiv:2310.10358
Pith/arXiv arXiv 2023
-
[17]
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. https://doi.org/10.18653/v1/D19-1534 Do NLP models know numbers? probing numeracy in embeddings . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pa...
-
[18]
Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. 2024. https://arxiv.org/abs/2411.12372 Redpajama: an open dataset for training large la...
Pith/arXiv arXiv 2024
-
[19]
Bodo Winter. 2013. https://arxiv.org/abs/1308.5499 Linear models and linear mixed effects models in r with linguistic applications . Preprint, arXiv:1308.5499
Pith/arXiv arXiv 2013
-
[20]
Xiang Zhang, Juntai Cao, Jiaqi Wei, Yiwei Xu, and Chenyu You. 2025. https://arxiv.org/abs/2505.14178 Tokenization constraints in llms: A study of symbolic and arithmetic reasoning limits . Preprint, arXiv:2505.14178
Pith/arXiv arXiv 2025
-
[21]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[22]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.