EasyMath: A 0-shot Math Benchmark for SLMs

Angelecia Frey; Drishya Karki; Michiel Kamphuis

REVIEW 4 major objections 7 minor 38 references

EasyMath: A 0-shot Math Benchmark for SLMs

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EasyMath, a 13-category zero-shot benchmark of everyday math with free-form symbolic grading, yields a reliable non-zero measure of practical math reasoning for small language models, tested across 23 models from 14M to 4B parameters.

desk verdict A useful zero-shot math benchmark for small models, but the scoring pipeline is underspecified and the artifacts are not released; the numbers are provisional until that is fixed. read the letter →

arxiv 2505.14852 v1 pith:NEZFSLMX submitted 2025-05-20 cs.CL cs.AIcs.LG

Drishya Karki , Michiel Kamphuis , Angelecia Frey This is my paper

classification cs.CLcs.AIcs.LG

keywords EasyMathsmalllanguagemodelsmathematicalreasoningbenchmarkzero-shotevaluationchain-of-thoughtpromptingfree-formanswergradingsymbolicequivalenceeveryday

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EasyMath, a benchmark of roughly 370 everyday math problems across 13 categories, from basic arithmetic to multi-step word problems, designed for small language models between 14 million and 4 billion parameters. The authors claim it provides a practical, zero-shot measure of real-world math reasoning, scoring free-form answers with exact, numerical, and symbolic equivalence checks rather than multiple choice. They report that accuracy rises with model size and training, chain-of-thought prompting gives modest gains, and run-to-run consistency improves at scale. The benchmark's value is that it yields non-zero, differentiated scores for models that score zero on GSM8K and near-random on MathQA, making it a more useful comparison tool for SLMs.

What carries the argument

The load-bearing component is the EasyMath dataset itself — 13 categories of hand-written everyday problems (basic arithmetic, order of operations, fractions/decimals, large numbers, exponents/roots, percentages, word problems, algebraic expressions, geometry, trigonometry, complex sentences, edge cases, multi-step reasoning) — together with its multi-layer free-form answer grader. The grader extracts all candidate mathematical expressions from a model's response via regex, normalizes notation (e.g., √x → sqrt(x)), and then applies category-dependent matching: strict string matching for exact-answer categories, and for the rest a stack of direct string comparison, numerical evaluation, symbolic equivalence via SymPy, and difference simplification to zero. For edge cases, the question prompt instructs the model to answer 'undefined', and the grader requires that token. This pipeline is what makes the benchmark's central claim — that free-form zero-shot answers can be reliably scored — hang together.

What would settle it

Take the EasyMath question set, write out correct answers in a wide variety of valid equivalent forms (e.g., '1/2', '0.5', '.5', 'one half', '50%', '0.500', '2/4'), feed each through the paper's extraction-and-matching pipeline, and count how many are marked incorrect. If the false-negative rate exceeds a few percent, the reported model accuracies are systematically understated and the benchmark's ability to separate models is weakened.

Watch

Extended reading notes

Core claim

The central claim is that EasyMath is a valid and useful instrument for measuring practical mathematical reasoning in small language models. Across 23 models (14M–4B parameters), the paper finds a clear scaling trend: accuracy increases with parameter count and with instruction or domain training, with sub-200M models scoring below 25% on algebraic expressions and essentially 0% on large-number arithmetic, while 1.5B-parameter models reach near-perfect performance on algebraic expressions and 40–76% on large numbers. Chain-of-thought prompting improves average accuracy for text-completion models (e.g., Pythia-1.4B from 16.77 to 22.92) but does not close the gap with models trained for reasoning, and dedicated reasoning-distilled SLMs (DeepSeek-R1-Distill-Qwen-1.5B, DeepScaleR-1.5B) do not outperform ordinary instruct models of the same size. Consistency analysis shows larger models have lower run-to-run variance (Gemma-3-4B ±0.30% vs SmolLM2-135M ±1.71%), and the paper proposes a probabilistic adjustment for accidental mention of the correct answer, estimating a modest ~1–2 percentage point overstatement of accuracies. Compared to GSM8K, Hendrycks-MATH, Math-500, and MathQA, EasyMath is the only benchmark tested that gave non-zero scores for all six comparison models without using multiple choice.

Load-bearing premise

The whole measurement rests on the assumption that the regex-extraction plus normalization plus SymPy equivalence pipeline correctly recognizes every mathematically correct free-form answer; a correct answer written in an unexpected format, unit, or notation will be counted as wrong and will distort the reported accuracies.

Editorial extensions

If this is right

EasyMath gives small language models a 0-shot benchmark with non-zero, differentiated scores, so it can serve as a practical replacement for GSM8K-style tests when evaluating models below 4 billion parameters.
The reported scaling and training trends imply that pushing an SLM from around 1B to 1.5B parameters, or fine-tuning it on word-problem datasets, yields large gains on everyday math (e.g., SmolLM2-360M improves roughly 12 points after fine-tuning on Orca-Math-Problems).
Chain-of-thought prompting helps text-completion models (e.g., Pythia-1.4B gains 6.15 average points), but the gain is smaller than in large language models, so CoT alone cannot be assumed to rescue small models.
At 1.5B scale, reasoning-distilled models (DeepSeek-R1-Distill-Qwen, DeepScaleR) do not beat ordinary instruct models (Qwen2.5-1.5B, AceMath-1.5B) on EasyMath, suggesting the reasoning capability gap for SLMs remains open.
Consistency data imply that evaluations of models below about 1 billion parameters need multiple runs to be reliable (SmolLM2-135M has ±1.71% standard deviation and 5.32% single-run error), while 4B models can be assessed accurately in a single run.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The accidental-mention probability framework (average 4.19%) suggests reported accuracies are slightly optimistic; a benchmark that required the model to show its work or answer in a canonical form would yield cleaner head-to-head comparisons.
Because EasyMath deliberately avoids specialist topics, it is better suited as a proxy for everyday assistant helpfulness than for academic math ability; the paper does not test this external correlation, so a future study could compare EasyMath scores against human ratings of assistant responses to the same questions.
The evaluation pipeline's reliance on regex and SymPy could be extended to accept natural-language final answers ('10 dollars', '5 miles per hour') and units; such an extension would reduce false negatives and make the benchmark more robust for conversational models.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

EasyMath is a genuinely useful resource in a narrow but real niche: a zero-shot math benchmark that separates small models (14M–4B) where GSM8K and MATH sit at zero. The dataset itself is the contribution—13 everyday-math categories, manually written questions, and a SymPy-based free-form scoring pipeline. The empirical sweep across 23 models is substantial, and the main qualitative findings (size and training help, CoT helps modestly, consistency scales) are credible and consistent with prior work.

The paper does a few things well. The 10-run consistency study on three models is careful, with proper confidence intervals and a bootstrap analysis. The comparison to GSM8K, MATH, Math-500, and MathQA makes the motivation concrete: those benchmarks floor at zero or rely on multiple-choice guessing, while EasyMath produces nonzero scores across the board. The authors are also upfront about the benchmark's scope: not for large models, English-only, manually constructed with possible linguistic biases.

The soft spots are real, and the biggest one is the one the authors concede in the Limitations section. The evaluation pipeline is underspecified and not validated. The regex extraction, normalization rules, and SymPy equivalence checks are described only at a high level. No code, no regex patterns, no normalization table, no SymPy version. The claim that they have not observed false negatives is not backed by any human audit or validation set. Since every reported accuracy passes through that pipeline, this is a load-bearing gap. It is not fatal—the gaps between model families are large enough that a few missed phrasings would not flip the ordering—but it does mean the exact numbers should be treated as provisional until the pipeline is released and audited.

Second, most results come from a single run. The consistency study shows smaller models have ±1.7% or more run-to-run variation, so some of the differences between mid-sized models could be noise. Third, the accidental-mention analysis in Section 4.3 is a heuristic with hand-set priors; the authors are careful to call it a confidence margin rather than a correction, so I don't hold that against them much, but it is not a rigorous contribution.

Overall, this is a solid, useful benchmark paper that fills a real measurement gap, but it is not yet reproducible. The right next step is to release the dataset and evaluation code, and to add a human audit of a sample of scoring decisions.

Who is this for? Anyone evaluating SLMs for practical math, and anyone building benchmarks for small models. It deserves a serious referee, but acceptance should be conditional on releasing the artifacts and documenting the scoring pipeline. I'd send it to review rather than desk-reject.

Referee Report

4 major / 7 minor

Summary. The paper introduces EasyMath, a manually curated 13-category benchmark of 370 everyday mathematics questions designed for small language models. The authors evaluate 23 models from 14M to 4B parameters in a zero-shot setting using an automatic pipeline that extracts expressions from free-form responses and checks exact, numerical, and symbolic equivalence via SymPy. They report that accuracy increases with model size and training, chain-of-thought prompting yields modest gains, and run-to-run consistency improves with scale. They also present fine-tuning experiments on SmolLM2 models, an accidental-mention analysis, and a comparison of EasyMath against GSM8K, Hendrycks-MATH, Math-500, and MathQA.

Significance. If the claims hold, EasyMath could be a useful, practically oriented complement to existing math benchmarks for small models, and the paper's 23-model sweep plus the 10-run consistency study with bootstrap confidence intervals is a step toward more careful SLM evaluation. The qualitative conclusions (larger and better-trained models do better, CoT gives modest gains, consistency improves with scale) are supported by large margins for many comparisons. The dataset itself, however, is the core contribution, and the paper's value depends on releasing the data and the evaluation pipeline, which are currently absent; the central measurement assumption, the scoring pipeline, is also unverified.

major comments (4)

[Section 4.2 and Appendix B] The scoring pipeline is the load-bearing component of every reported accuracy, but it is underspecified and unaudited. The paper gives no regex patterns, no normalization table, no SymPy version or simplification settings, and no evaluation code. The Limitations section concedes that 'some expressions may still evade recognition depending on formatting or notation,' and the claim that this was not observed for the evaluated models is not backed by any described human audit. Concrete format-sensitive cases include the edge-cases category, where the response must contain the literal word 'undefined' (so 'no solution' for 0x=5 is marked wrong), and unit-bearing answers such as '3 hours' versus '3'. Since base/completion models (e.g., Pythia, GPT-Neo) answer in prose more often than instruction-tuned models, false negatives are unlikely to be uniform across model families, which would distort the per-category and aggregate scores in Table 2. The pipeline must be validated on a human-audited sample, and the code and data must be released before the reported accuracies can be taken as measurements of model ability rather than parser behavior.
[Table 2 and Section 5.2] There is a direct internal inconsistency between the main results and the consistency study. Table 2 reports SmolLM2-135M at 28.77% and Llama-3.2-1B at 74.92%, but Section 5.2.1 reports 10-run ranges of 20.81-26.76% and 70.81-74.86% for the same models; the Table 2 values are outside these ranges, and the paper does not explain whether the consistency runs used a different prompt, a different pipeline version, or a different evaluation setting. Moreover, since Section 5.2 reports a 10-run study for only three models, the remaining entries in Table 2 are single-run or run an unspecified number of times, while the paper's own bootstrap analysis estimates single-run mean errors of 5.32% for SmolLM2-135M and 1.01% for Llama-3.2-1B. Small differences in Table 2 (e.g., pythia-70m at 8.31 vs. gpt-neo-125m at 8.15) are therefore within run-to-run noise. The authors should report multiple runs or confidence intervals for all models and reconcile the Table 2 values with the consistency study.
[Section 5.3.1 and Tables 3-6] The fine-tuning experiments are presented as evidence that training on existing datasets improves EasyMath performance, but each entry in Tables 3-6 appears to be a single run with no confidence intervals or significance tests, given the variance documented for 135M-360M models in Section 5.2. The baseline for SmolLM2-135M in Table 3 is 24.05, whereas the same model in Table 2 is 28.77, and a question-weighted average of the category scores in Table 2 gives approximately 25.4; the paper does not explain whether the discrepancy is due to averaging, prompting, or stochasticity. The conclusions that Orca-word-problems improves accuracy by roughly 12% and is 'slightly higher quality' than OpenR1-Math-220k therefore need either repeated runs with error bars or an explicit statement that they are qualitative observations.
[Sections 1 and 4.1.2] The manuscript presents EasyMath as a benchmark and the central artifact of the paper, but the 370-question dataset and the evaluation scripts are not released or even linked to; only a dozen example questions appear in the text. Similarly, Section 4.1.2 asserts that no question overlaps existing benchmarks or training datasets, but gives no method, similarity measure, or list of datasets checked. Without the dataset and scorer, the community cannot reproduce Table 2, audit the pipeline, or use EasyMath as a benchmark, which substantially weakens the paper's contribution as a benchmark paper.

minor comments (7)

[Section 4.3, Eq. (1)] The formula P(Ac in R | Ac not in C) = P(Ac in Q) + P(Ac in I)(1 - P(Ac in Q)) treats Q and I as independent or disjoint without stating an assumption, and the category-specific priors are not reported, so the aggregate value 0.0419 cannot be reproduced from the text.
[Section 5.3.2, Table 7] The statement that 'almost all categories achieve a higher accuracy when using a CoT prompt' is contradicted by Table 7, which shows decreases in basic-arithmetic (60 to 50) and edge-cases (30 to 20) and ties in fractions-and-decimals and trigonometry.
[Table 2] The category abbreviations (B.A., O.O.P., F.A.D., L.N., E.R., P, W.P., A.E., G, T, C.S., E.C., M.S.R.) are not expanded in the table or its caption; a legend should be added.
[Appendices C and E] The averaging convention differs between Appendix C ('averaged over categories') and Appendix E ('Averaged over all questions'); the main text should state this explicitly because it explains why the same model can have different baseline values.
[Section 5.2] The consistency study refers to the model as 'Llama-1B' while Tables 1 and 2 call it 'Llama-3.2-1B'; the names should be consistent throughout.
[Section 5.4] The claim that MathQA scores have an average deviation of only 4.29% from random guessing would benefit from showing the computation, since the four-option baseline is 25%.
[Tables 1 and 2] The Llama citations appear as '(et al., 2024; AI, 2025)'; the first citation should be a named author entry (e.g., Grattafiori et al., 2024) and the reference formatting should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EasyMath is an empirical benchmark study whose claims rest on external model evaluations, not on fitted inputs or self-citations.

full rationale

EasyMath is a benchmark construction and evaluation paper, not a derivation. The benchmark defines correctness by equivalence to author-written SymPy calculations, and all central claims (accuracy scaling with size and training, modest chain-of-thought gains, consistency improving with scale) are contingent empirical measurements of external model outputs. Section 4.3's accidental-mention analysis is the only place where heuristic estimates enter a calculation, but the authors explicitly label the result a confidence margin rather than a strict adjustment, and it is not used to support the main conclusions. There are no load-bearing self-citations: none of the cited prior works are by the present authors, and no uniqueness theorem or prior-ansatz is imported to force a choice. The evaluation pipeline in Section 4.2 and Appendix B is underspecified and unaudited, which is a real correctness and reproducibility risk, but a scoring bug would invalidate the empirical results rather than make them circular: the reported accuracies are not equivalent by construction to any fitted parameter. No pattern of self-definition, fitted-input-called-prediction, or renaming of known results is present, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest on the evaluation pipeline's reliability, the representativeness of the chosen prompts, and the correctness of several hand-set values. The paper introduces no new physical or conceptual entities; the accidental mention probability is a derived metric, not an entity. The largest unverifiable input is the claimed non-overlap with existing benchmarks, which is asserted but not demonstrated.

free parameters (3)

P(A_c in Q) heuristic priors per category = approx. 0.05 for word problems, 0.0 for computation problems
Hand-set priors in Section 4.3 used to compute accidental mention probability and adjusted accuracy; not estimated from data.
P(A_c in I) heuristic priors per category = not specified
Hand-set priors in Section 4.3 for intermediate values; combined with Q priors to compute accidental mention probability.
Fine-tuning hyperparameters (LR, warmup, scheduler, optimizer, epochs) = 2e-5, 500 warmup steps, cosine, 8-bit AdamW, 8 epochs
Chosen by the authors for the SmolLM2 fine-tuning experiments in Section 5.3.1 and Appendix E; these choices affect the reported training curves, though not the central benchmark evaluation.

assumptions (5)

domain assumption Regex extraction + normalization + SymPy equivalence correctly identifies correct free-form answers.
Section 4.2 and Appendix B; if a correct answer is formatted unexpectedly, it is scored wrong. The paper's Limitations acknowledge this risk.
domain assumption Zero-shot prompting with model-specific prompt templates is a fair comparison across models.
Section 5.1; models use different prompt formats, which can affect scores independently of math ability.
ad hoc to paper No EasyMath question overlaps existing math benchmarks or training datasets.
Section 4.1.2; asserted without a described verification procedure. If overlap exists, fine-tuning results and novelty claims are inflated.
domain assumption Equal-weight averaging over 13 categories is the appropriate aggregate metric.
Appendix C; category sizes differ (10 to 50 questions), and equal weighting changes the reported average compared to question-level averaging.
ad hoc to paper The heuristic priors P(A_c in Q) and P(A_c in I) approximate true accidental mention rates.
Section 4.3; used to compute adjusted accuracy, but the priors are hand-set, not fitted or validated against observed model behavior.

how reviews work

0 comments

Cite this review

Pith. "Pith review of EasyMath: A 0-shot Math Benchmark for SLMs." pith.science (2026). https://pith.science/paper/NEZFSLMX

@misc{pith2026250514852,
  author       = {Pith},
  title        = {Pith review of: EasyMath: A 0-shot Math Benchmark for SLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NEZFSLMX}},
  note         = {Machine review of arXiv:2505.14852}
}

read the original abstract

EasyMath is a compact benchmark for practical math reasoning in small language models. It covers thirteen categories, from basic arithmetic and order of operations to word problems, algebraic expressions, edge cases, and omits specialist topics. We tested 23 models (14M to 4B parameters) using exact, numerical, and symbolic checks on free-form answers in a zero-shot setting. Accuracy rises with size and training, chain-of-thought adds modest gains, and consistency improves at scale.

Figures

Figures reproduced from arXiv: 2505.14852 by the authors.

**Figure 1.** SmolLM2 models accuracy 5.3.1 The effect of training on existing datasets We examined the impact of continued finetuning on SmolLM2 models by training them on established mathematics datasets and evaluating their performance on EasyMath. Two separate models were trained for 8 epochs each on subsets of prominent mathematics datasets: 25,000 randomly sampled examples from Orca-word-problems (Mitra et al., 2024) and 2… view at source ↗

**Figure 2.** Distribution of Questions Across Categories [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

**Figure 3.** Flowchart of the EasyMath benchmark evaluation process [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Average accuracy vs. parameter count (log scale) across different model families [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

**Figure 5.** Figure 5: Model consistency comparison [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

**Figure 6.** Figure 6: Model size vs Consistency 14 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]

**Figure 7.** Figure 7: Model reliability comparison [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

**Figure 8.** Figure 8: Category variability heatmap 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

**Figure 9.** Figure 9: Average accuracy (%) over epochs for Orca-Math-Problems and R1-Math on different base-models. [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 9 canonical work pages

[1]

Meta AI. 2025. Llama: Inference code for llama models. https://github.com/meta-llama/llama. Accessed: 2025-03-21

work page 2025
[2]

Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, and 3 others. 2025. https://ar...

arXiv 2025
[3]

Sultan Alrashed. 2024. https://arxiv.org/abs/2412.08347 Smoltulu: Higher learning rate to batch size ratios can lead to better reasoning in slms . Preprint, arXiv:2412.08347

arXiv 2024
[4]

Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. https://doi.org/10.18653/v1/N19-1245 M ath QA : Towards interpretable math word problem solving with operation-based formalisms . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics:...

doi:10.18653/v1/n19-1245 2019
[5]

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. https://arxiv.org/abs/2304.01373 Pythia: A suite for analyzing large language models across training and scaling . Preprint...

arXiv 2023
[6]

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. http://github.com/eleutherai/gpt-neo GPT-Neo : Large scale autoregressive language modeling with mesh-tensorflow

2021
[7]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

arXiv 2021
[8]

DeepSeek-AI. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

arXiv 2025

Show all 38 references

[9]

Aaron Grattafiori et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

2024 arXiv
[10]

Hugging Face. 2025. https://github.com/huggingface/open-r1 Open r1: A fully open reproduction of deepseek-r1

2025
[11]

Google DeepMind Gemma Team. 2025. https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf Gemma 3 technical report . Technical report, Google DeepMind. Accessed: 2025-03-21

2025
[12]

Nathan Godey, Éric de la Clergerie, and Benoît Sagot. 2024. https://arxiv.org/abs/2404.07647 Why do small language models underperform? studying language model saturation via the softmax bottleneck . Preprint, arXiv:2404.07647

2024 arXiv
[13]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the math dataset . Preprint, arXiv:2103.03874

2021 arXiv
[14]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. https://arxiv.org/abs/1503.02531 Distilling the knowledge in a neural network . Preprint, arXiv:1503.02531

2015 arXiv
[15]

Bumjun Kim, Kunha Lee, Juyeon Kim, and Sangam Lee. 2024. https://arxiv.org/abs/2409.12393 Small language models are equation reasoners . Preprint, arXiv:2409.12393

2024 arXiv
[16]

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. https://arxiv.org/abs/2205.11916 Large language models are zero-shot reasoners . Preprint, arXiv:2205.11916

2023 arXiv
[17]

Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. 2025. https://arxiv.org/abs/2502.12143 Small models struggle to learn from strong reasoners . Preprint, arXiv:2502.12143

2025
[18]

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. https://arxiv.org/abs/2305.20050 Let's verify step by step . Preprint, arXiv:2305.20050

2023 arXiv
[19]

Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2025. https://arxiv.org/abs/2412.15084 Acemath: Advancing frontier math reasoning with post-training and reward modeling . Preprint, arXiv:2412.15084

2025 arXiv
[20]

Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica

Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog

2025
[21]

Qing Lyu, Kumar Shridhar, Chaitanya Malaviya, Li Zhang, Yanai Elazar, Niket Tandon, Marianna Apidianaki, Mrinmaya Sachan, and Chris Callison-Burch. 2024. https://arxiv.org/abs/2402.13904 Calibrating large language models with sample consistency . Preprint, arXiv:2402.13904

2024
[22]

Smith, Mateusz Paprocki, Ond r ej C ert\' i k, Sergey B

Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ond r ej C ert\' i k, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam ...

2017 doi
[23]

Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2024. https://arxiv.org/abs/2410.05229 Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models . Preprint, arXiv:2410.05229

2024 arXiv
[24]

Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. 2024. https://arxiv.org/abs/2402.14830 Orca-math: Unlocking the potential of slms in grade school math . Preprint, arXiv:2402.14830

2024 arXiv
[25]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. https://arxiv.org/abs/2412.15115 Qwe...

2025 arXiv
[26]

Open R1. 2025. https://huggingface.co/datasets/open-r1/OpenR1-Math-220k Openr1-math-220k: A large-scale dataset for mathematical reasoning . Accessed: 2025-03-22

2025
[27]

Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay S...

2025 arXiv
[28]

Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. 2025. https://arxiv.org/abs/2502.11569 Towards reasoning ability of small language models . Preprint, arXiv:2502.11569

2025
[29]

Falcon-LLM Team. 2024. https://huggingface.co/blog/falcon3 The falcon 3 family of open models

2024
[30]

Julian Junyan Wang and Victor Xiaoqi Wang. 2025. https://arxiv.org/abs/2503.16974 Assessing consistency and reproducibility in the outputs of large language models: Evidence across diverse finance and accounting tasks . Preprint, arXiv:2503.16974

2025 arXiv
[31]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-thought prompting elicits reasoning in large language models . Preprint, arXiv:2201.11903

2023 arXiv
[32]

Jason Wei and Denny Zhou. 2022. https://research.google/blog/language-models-perform-reasoning-via-chain-of-thought/ Language models perform reasoning via chain of thought . Google Research Blog

2022
[33]

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. https://arxiv.org/abs/2409.12122 Qwen2.5-math technical report: ...

2024 arXiv
[34]

Zishun Yu, Tengyu Xu, Di Jin, Karthik Abinav Sankararaman, Yun He, Wenxuan Zhou, Zhouhao Zeng, Eryk Helenowski, Chen Zhu, Sinong Wang, Hao Ma, and Han Fang. 2025. https://arxiv.org/abs/2501.17974 Think smarter not harder: Adaptive reasoning with inference aware optimization . ...

2025 arXiv
[35]

Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K

Albert S. Yue, Lovish Madaan, Ted Moskovitz, DJ Strouse, and Aaditya K. Singh. 2024. https://github.com/aadityasingh/HARP HARP : A challenging human-annotated math reasoning benchmark

2024
[36]

Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. 2024. https://openreview.net/forum?id=2cczgOfMP4 Chain of preference optimization: Improving chain-of-thought reasoning in LLM s . In The Thirty-eighth Annual Conference on Neural Information Processing Systems

2024
[37]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[38]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Meta AI. 2025. Llama: Inference code for llama models. https://github.com/meta-llama/llama. Accessed: 2025-03-21

work page 2025

[2] [2]

Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, and 3 others. 2025. https://ar...

arXiv 2025

[3] [3]

Sultan Alrashed. 2024. https://arxiv.org/abs/2412.08347 Smoltulu: Higher learning rate to batch size ratios can lead to better reasoning in slms . Preprint, arXiv:2412.08347

arXiv 2024

[4] [4]

Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. https://doi.org/10.18653/v1/N19-1245 M ath QA : Towards interpretable math word problem solving with operation-based formalisms . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics:...

doi:10.18653/v1/n19-1245 2019

[5] [5]

Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. https://arxiv.org/abs/2304.01373 Pythia: A suite for analyzing large language models across training and scaling . Preprint...

arXiv 2023

[6] [6]

Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. 2021. http://github.com/eleutherai/gpt-neo GPT-Neo : Large scale autoregressive language modeling with mesh-tensorflow

2021

[7] [7]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

arXiv 2021

[8] [8]

DeepSeek-AI. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . Preprint, arXiv:2501.12948

arXiv 2025