Pith. sign in

REVIEW 3 major objections 6 minor 26 references

ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ClozeMath shows that equation text-infilling during fine-tuning improves mathematical reasoning.

desk verdict Equation-infilling is a genuinely new fine-tuning objective for math reasoning, but the uncontrolled training budget makes the headline gains over baselines unproven. read the letter →

arxiv 2506.03763 v1 pith:O63FPTGU submitted 2025-06-04 cs.CL

Quang Hieu Pham , Thuy Duong Nguyen , Tung Pham , Anh Tuan Luu , Dat Quoc Nguyen This is my paper
classification cs.CL
keywords ClozeMathtextinfillingequationmaskingmathematicalreasoninglanguagemodelfine-tuningMaskedThoughtGSM8KGSM-Symbolic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes ClozeMath, a fine-tuning objective that adds equation text-infilling to standard next-token prediction. The idea is that a model learns to generalize mathematical reasoning more readily by predicting the masked equations of a solution from the surrounding textual rationale, much as a human first grasps the general approach before filling in step-specific computations. On GSM8K and MATH, ClozeMath improves accuracy over the base language models and over the Masked Thought baseline across Llama-3.2-1B/3B, Llama-3.1-8B, and DeepSeek-Math-7B, and it holds up better on GSM-Symbolic's distribution-varied problems. If correct, the recipe is cheap and general: it needs no new data, no larger model, and only a small change to the fine-tuning loss.

What carries the argument

The mechanism is the equation text-infilling objective: all equation spans in a solution are replaced with mask tokens and the model must predict them from left to right, conditioned on the unmasked textual rationale and the problem statement. It is paired with a PrefixLM attention scheme (bidirectional attention over the prefix, causal attention over the infilling target) and a separator token between the two parts. The argument's load-bearing choice is masking equations, not random spans: random span masking disrupts the logical coherence of the rationale and drops accuracy from 74.22% to 71.19% on GSM8K, while the equation-targeted version surpasses the instruction-tuning baseline.

What would settle it

Fine-tune the pure language-modeling baseline (the W/o-Text-infilling variant) for the same number of total steps as ClozeMath, keeping data and hyperparameters fixed, and compare GSM8K or MATH accuracy; if the gap disappears, ClozeMath's advantage is a training-budget effect rather than equation infilling.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the way the training signal is structured, not the amount of reasoning text, is what improves mathematical reasoning. ClozeMath masks each equation in a solution and asks the model to regenerate the equations from left to right while the textual rationale stays visible, and it combines this with the ordinary language-modeling loss, $\mathcal{L}_{\mathrm{ClozeMath}} = \mathcal{L}_{\mathrm{lm}} + \mathcal{L}_{\mathrm{tf}}$. The authors report that this consistently outperforms Masked Thought, that the gains increase with inference-time compute under CoT decoding, and that replacing equation spans with randomly masked spans removes most of the benefit. They attribute the effect to the equation mask preserving the solution's general plan while forcing the model to produce the formal derivation steps that depend on that plan.

Load-bearing premise

The paper attributes ClozeMath's improvement to the equation-infilling objective, but ClozeMath is trained with roughly twice as many optimization steps per original solution as its pure text baseline, so the comparison assumes the extra training budget is not the real cause.

Editorial extensions

If this is right

  • ClozeMath can be applied to any pretrained decoder-only model with a vocabulary extension for the separator and mask tokens, providing a way to improve math accuracy without new data or a larger model.
  • The benefits transfer to robustness: on GSM-Symbolic's GSM-P1, ClozeMath-trained DeepSeek-Math and Llama-3.1-8B beat Masked Thought by about 5 to 6 percentage points when the problem distribution is shifted by an added constraint.
  • The method scales with inference-time compute: with Chain-of-Thought decoding over 9 samples, ClozeMath outperforms Masked Thought at the same budget on GSM8K.
  • The specific masking target matters: random span masking underperforms equation masking, supporting the paper's explanation that preserving the textual rationale is what makes the infilling objective effective.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same 'keep the prose, mask the formalism' recipe may transfer to other domains where a formal artifact is interleaved with explanation, such as code generation, table manipulation, or structured proofs; the paper only evaluates arithmetic word problems and competition math.
  • Beyond the paper: because the authors duplicate language-model samples to balance the objectives, ClozeMath performs roughly twice as many updates per original solution as its pure LM baseline; an equal-update comparison would isolate whether the gain is the objective itself or the larger training budget.
  • Beyond the paper: the method's left-to-right unmasking implicitly encodes a dependency assumption that each equation depends only on earlier equations, and the paper's 50% masking rule for simultaneous transformation groups is a partial concession; testing other unmasking orders would reveal whether this ordering is doing causal work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ClozeMath, a fine-tuning objective that combines standard next-token language modeling with a text-infilling task in which equation spans in a solution are masked and predicted, sequentially unmasking equations from first to last. Training uses PrefixLM attention, with bidirectional attention on the prompt and causal attention on the target. The method is evaluated by LoRA fine-tuning of Llama-3.2-1B/3B, Llama-3.1-8B, and DeepSeek-Math-7B on GSM8K and MATH, with additional robustness evaluation on GSM-Symbolic and test-time scaling through CoT decoding. Across Tables 1-3, ClozeMath reports higher accuracy than Masked Thought in most settings, and Table 4 ablates the text-infilling objective, PrefixLM, and equation-specific masking.

Significance. If the central claim holds, ClozeMath is a simple and cheap fine-tuning recipe: it only requires that equations be identified in existing solution text, and it improves both accuracy and robustness across model scales and benchmark families. The paper's strengths include evaluations on held-out standard benchmarks as well as the GSM-Symbolic robustness suite, consistent gains across four base models, and an ablation that separates the equation-masking strategy from random span masking. I also see no evidence of evaluation-set fitting; the method is a training objective and all test sets are held out. The main weakness is that the headline comparison does not control for training budget, because duplicated LM samples give ClozeMath more updates per original solution than the pure-LM and Masked Thought baselines; the central claim is therefore conditional on a compute-matched comparison.

major comments (3)
  1. [Section 2 (last paragraph), Table 4, Figure 3] The comparison is confounded by an uncontrolled training budget. To keep L_lm and L_tf at a 50:50 sample ratio, the authors duplicate the L_lm samples, so each original solution contributes a duplicated LM example plus one or more text-infilling examples per optimizer step, while the W/o Text-infilling baseline (71.79%) and the MFT baseline contribute one example. None of Tables 1, 3, or 4 reports a step-matched or compute-matched baseline, and Appendix B omits batch size, epoch count, number of optimizer steps, and total example exposures. Consequently the 2.43-point gain over W/o Text-infilling in Table 4, the gains over MFT in Table 1, and the 'sample efficiency' claim in Figure 3 could in principle be produced by extra gradient updates or by repeated exposure to the same LM data. Please add a control that trains W/o Text-infilling (and ideally MFT) for the same number of optimizer steps with the same number of examples per step, for example by duplicating plain LM examples to match ClozeMath's per-solution sample budget, and report accuracy against both step count and unique-example count.
  2. [Section 2 (definition of F_i)] The procedure for obtaining the equation index sets F_i is not specified. The paper defines F_i abstractly as the set of segments that are 'a mathematical equation' but gives no parser, rule set, annotation protocol, or example of how equations are separated from textual segments. Since the text-infilling objective and all reported results depend on this identification, the method cannot be reproduced or extended to new solution text without this detail. Please describe the segmentation and equation-extraction procedure, or release the code that implements it.
  3. [Section 3.2, Table 3] The robustness claim is partially overstated. While ClozeMath is higher on most GSM-Symbolic cells, several comparisons go the other way or are within noise, e.g., GSM8K-trained DeepSeek-Math on GSM-P2 (20.00 vs 20.20), GSM8K-trained Llama-3.1-8B on GSM-P2 (22.50 vs 22.30), and MATH-trained Llama-3.2-3B on GSM-P2 (1.80 vs 2.30). No variance or repeated-seed information is reported, so the reader cannot tell which differences are reliable. Please add confidence intervals or seed variance, and qualify the robustness claim accordingly.
minor comments (6)
  1. [Section 2, Eqs. (1)-(2)] Equations (1) and (2) use probability expressions where losses should be negative log-likelihoods; please either define P as a log-probability or write log P explicitly.
  2. [Section 2, Eq. (2)] The set comprehension defining M(F_i) is garbled in the typesetting; it should be written as sets of consecutive equation indices, e.g., {{f_t, ..., f_|F_i|} : t = 1, ..., |F_i|}, so that the sequential unmasking is clear.
  3. [Figure 3] The x-axis of Figure 3 is labeled 'training steps' but has no numeric tick labels or step counts in the caption, making the sample-efficiency comparison difficult to evaluate.
  4. [Section 3.3, Table 4] The 'W/o Equation masking' row reports a single number (71.19) but Section 3.3 describes two masking regimes (15% for short spans and 50% for long spans); please state which regime produced the reported number or report both.
  5. [Abstract and Section 1] The abstract lists 'Beam Search and Chain-of-Thought decoding' as 'two test-time scaling decoding algorithms'; beam search is a standard decoding method rather than a test-time scaling algorithm, so please rephrase to avoid overclaiming.
  6. [Appendix B] Appendix B contains a typo ('Apppendix B') and omits batch size, number of epochs, total optimizer steps, and effective batch size; these details are needed for reproducibility and for assessing the training-budget confound.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ClozeMath's training objective and evaluation are distinct; all baselines are external; the training-budget confound is a correctness risk, not a circular reduction.

full rationale

The paper's central claim is that adding an equation text-infilling objective (L_tf) to standard language modeling (L_lm) improves mathematical reasoning on held-out benchmarks. The objective is defined from training solutions, while evaluation is on GSM8K, MATH, and GSM-Symbolic test sets; no evaluation-set labels or benchmark scores are used to construct the method or fit any parameter. The baseline Masked Thought (Chen et al., 2024) is from different authors and is not a self-citation, and no load-bearing argument reduces to the authors' own prior work. The ablation study compares ClozeMath against variants with and without text infilling, PrefixLM, and equation masking, and these are genuine controlled comparisons rather than circular redefinitions. The skeptic's point about duplicated LM samples producing extra optimization steps per solution is a training-budget confound and a validity threat, but it is not circularity under the rubric: the measured advantage is not equivalent to the method's inputs by construction, and no equation in the paper is fitted to the evaluation data. Therefore no circular step can be exhibited with the required specificity, and the appropriate score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No derived constants appear in the method. The only hand-chosen numbers are training ratios and LoRA rank, none fitted to the test sets. The method relies on the domain assumption that equation-level masking is the right unit of reasoning, validated only by the reported ablations.

free parameters (2)
  • LM-infilling mixture ratio = 50:50
    Chosen by hand to balance the two objectives; not tuned on test sets, but affects training dynamics.
  • Equation-group masking rate = 50%
    For simultaneous transformations, each equation is masked with probability 50%; an implementation choice without sensitivity analysis.
assumptions (4)
  • ad hoc to paper Text-infilling with masked equations is a useful training signal for mathematical reasoning beyond standard LM loss
    The core premise of the method; supported by ablations but not derived.
  • ad hoc to paper Equations in solutions can be identified and satisfy a causal order such that masking all and then unmasking from first to last is sensible
    Section 2, Eq. 2; the schedule assumes each equation depends only on previous ones.
  • domain assumption GSM8K and MATH test sets and GSM-Symbolic robustness sets measure the intended reasoning ability
    Standard benchmarks, but limited to arithmetic and school-level math.
  • domain assumption LoRA fine-tuning on base models with rank 32 is representative of full-model behavior
    Used for all experiments; the paper does not verify with full fine-tuning.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations." pith.science (2026). https://pith.science/paper/O63FPTGU

@misc{pith2026250603763,
  author       = {Pith},
  title        = {Pith review of: ClozeMath: Improving Mathematical Reasoning in Language Models by Learning to Fill Equations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/O63FPTGU}},
  note         = {Machine review of arXiv:2506.03763}
}
read the original abstract

The capabilities of large language models (LLMs) have been enhanced by training on data that reflects human thought processes, such as the Chain-of-Thought format. However, evidence suggests that the conventional scheme of next-word prediction may not fully capture how humans learn to think. Inspired by how humans generalize mathematical reasoning, we propose a new approach named ClozeMath to fine-tune LLMs for mathematical reasoning. Our ClozeMath involves a text-infilling task that predicts masked equations from a given solution, analogous to cloze exercises used in human learning. Experiments on GSM8K, MATH, and GSM-Symbolic show that ClozeMath surpasses the strong baseline Masked Thought in performance and robustness, with two test-time scaling decoding algorithms, Beam Search and Chain-of-Thought decoding. Additionally, we conduct an ablation study to analyze the effects of various architectural and implementation choices on our approach.

Figures

Figures reproduced from arXiv: 2506.03763 by the authors.

Figure 1
Figure 1. ClozeMath’s mixture of training objectives. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Implementation of MFT (Chen et al., 2024) and ClozeMath on the same sample. Here, *** denotes the unshown causal attention mask in this figure. MFT aims at learning longer-range dependencies by randomly inputting noisy mask (<M>) tokens into the solution’s parts while keeping the problem definition untouched. MFT is prone to learning spurious correlation; in this example, a derived transformation might not fully per… view at source ↗
Figure 3
Figure 3. GSM8K test accuracies w.r.t. the numbers of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: In this example, the Deepseek-Math model trained with MFT does not understand that the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: In this example, the Deepseek-Math model trained with MFT misunderstood that "Toulouse" has twice [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: In this example, the Deepseek-Math model trained with MFT incorrectly hallucinates that "There are [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

26 extracted references · 19 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Gregor Bachmann and Vaishnavh Nagarajan. 2024. The Pitfalls of Next-Token Prediction . In Proceedings of the 41st International Conference on Machine Learning, pages 2296--2318

    work page 2024

  4. [4]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, page 610–623

    work page 2021

  5. [5]

    Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji-Rong Wen, Rui Yan, and Yongbin Li. 2024. Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5872--5900

    work page 2024

  6. [6]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems . arXiv preprint, arXiv: 2110.14168

    arXiv 2021

  7. [7]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. https...

    doi:10.5281/zenodo.12608602 2024

  8. [8]

    Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models . arXiv preprint, arXiv:2407.21783

    arXiv 2024

Show all 26 references
  1. [9]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring Mathematical Problem Solving With the MATH Dataset . In Proceedings of the Thirty-fifth Conference on Neural Information Processing Systems: Dat...

    2021

  2. [10]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lo RA : Low-Rank Adaptation of Large Language Models . In Proceedings of the Tenth International Conference on Learning Representations

    2022

  3. [11]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners . In Proceedings of the Thirty-sixth Conference on Neural Information Processing Systems, pages 22199--22213

    2022

  4. [12]

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. Let's Verify Step by Step . In Proceedings of the Twelfth International Conference on Learning Representations

    2024

  5. [13]

    Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer

    Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by Summarizing Long Sequences . In Proceedings of The Sixth International Conference on Learning Representations

    2018

  6. [14]

    Mason, L

    J. Mason, L. Burton, and K. Stacey. 2010. Thinking Mathematically . Pearson

    2010

  7. [15]

    Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. GSM -Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models . In Proceedings of the Thirteenth International Conference on Learni...

    2025

  8. [16]

    OpenAI. 2024. Learning to reason with llms. https://openai.com/index/learning-to-reason-with-llms/. Accessed May 18, 2025

    2024

  9. [17]

    Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro

    Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. ArXiv, abs/2303.09014

    2023 arXiv

  10. [18]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research

    2020

  11. [19]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models . arXiv preprint, arXiv:2402.03300

    2024 arXiv

  12. [20]

    Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In Proceedings of The Thirteenth International Conference on Learning Representations

    2025

  13. [21]

    Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler

    Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023 a . UL2: unifying language learning paradigms. In The Eleventh International Confe...

    2023

  14. [22]

    Yi Tay, Jason Wei, Hyung Chung, Vinh Tran, David So, Siamak Shakeri, Xavier Garcia, Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, Denny Zhou, Donald Metzler, Slav Petrov, Neil Houlsby, Quoc Le, and Mostafa Dehghani. 2023 b . Transcending Scaling Laws with 0.1 \ In Proceeding...

    2023

  15. [23]

    Xinyi Wang, Lucas Caccia, Oleksiy Ostapenko, Xingdi Yuan, William Yang Wang, and Alessandro Sordoni. 2024. Guiding Language Model Reasoning with Planning Tokens . In Proceedings of the First Conference on Language Modeling

    2024

  16. [24]

    Xuezhi Wang and Denny Zhou. 2024. Chain-of-thought reasoning without prompting. In Proceedings of The Thirty-eighth Annual Conference on Neural Information Processing Systems

    2024

  17. [25]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models . In Proceedings of the Thirty-sixth Conference on Neural Information Processing Systems, pages 24...

    2022

  18. [26]

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. MA mmo TH : Building Math Generalist Models through Hybrid Instruction Tuning . In Proceedings of the Twelfth International Conference on Learning Representations

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.