Pith. sign in

REVIEW 3 major objections 4 minor 32 references

Enhancing Mathematical Reasoning in LLMs with Background Operators

T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning an 8B LLM to emit Prolog built from 54 standardized math predicates reaches 84.8% accuracy on competition counting and probability problems.

desk verdict New dataset and operator framework, but the reported accuracies are in-sample because the evaluation adds generated solutions from the target set into training. read the letter →

arxiv 2412.04110 v1 pith:324QXPG2 submitted 2024-12-05 cs.AI

Jiajun Chen , Yik-Cheung Tam This is my paper · ORCID
classification cs.AI
keywords mathematicalreasoningPrologbackgroundoperatorsself-traininglargelanguagemodelsMATHcorpusverifiablecountingandprobability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that competition-level math reasoning can be recast as constraint search in Prolog, and that a large language model can be taught to write such programs if it is given a fixed vocabulary of background mathematical predicates. To test this, the authors curate MATH-Prolog, a corpus of 625 hand-corrected Prolog solutions for counting and probability problems from the MATH benchmark, and fine-tune Meta-Llama-3.1-8B-Instruct on it. They then apply a 5-fold cross-validated self-training loop in which the model samples new solutions for the held-out fold, keeps only those whose output matches the reference answer when executed by SWI-Prolog, and adds them to the training data. The claim is that this procedure discovers diverse, correct, fully computable solutions for previously unseen problems, reaching 84.6% cross-validated coverage and 84.8% accuracy on the test set after augmenting the training set with derived test solutions. The reason to care is that, unlike chain-of-thought, the generated reasoning is machine-executable and verifiable, which could make LLM math answers auditable.

What carries the argument

The load-bearing object is the set of 54 hand-crafted background mathematical predicates (e.g., combination(N,R,C), factorial(N,F), findall/3) that standardize every Prolog solution as a composed computation graph. These operators both constrain the model's output space and expose decompositional equivalences (for instance, combination can be rewritten as permutation composed with factorial and division), which the self-training loop exploits to generate diverse but still correct programs. The second piece of machinery is the K-fold cross-validated self-training algorithm: for each fold, the model samples up to J solutions per held-out question, executes each with SWI-Prolog, and adds any solution whose numerical result matches the reference into the training set before the next epoch. This is what turns a static corpus into an expanding one, and it is also the step that makes the reported accuracy scores sensitive to whether generated test-set solutions are reused as training data.

What would settle it

Run the same 5-fold procedure but withhold every generated solution from the training set until after the fold has been scored, and compare the accuracy; if it drops well below 84.6%, the reported gain comes from training on the test problems themselves.

Watch

Extended reading notes

Core claim

The central claim is that declarative Prolog programming with a standardized set of background operators is a better target representation for LLM math reasoning than free-form natural language or procedural Python. The authors define 54 mathematical predicates such as combination/3, factorial/2, findall/3, probability_of_event/4, and cardinality/2, and require every Prolog solution to compose these operators, so a solution is effectively a directed computation graph from problem-specific facts to the answer. Fine-tuning an 8B Llama model on the resulting MATH-Prolog corpus makes it possible to generate such graphs for unseen questions, and the cross-validated self-training algorithm samples multiple candidate programs, executes them with SWI-Prolog, and keeps only those that evaluate to the correct final answer. The paper reports that this yields 84.6% coverage on the 5-fold cross-validation and 84.8% accuracy on the test set, and that including the background operators in the prompt improves the learning trajectory and solution coverage.

Load-bearing premise

The load-bearing premise is that scoring a fold after adding that fold's verified generated solutions to the training set still measures the model's ability to handle previously unseen problems, rather than its ability to reproduce memorized solutions.

Editorial extensions

If this is right

  • If a fine-tuned 8B model can generate executable Prolog at 84.8% coverage, then competition-level counting and probability problems can be delegated to a verifiable search process rather than to uncheckable verbal reasoning.
  • The cross-validated self-training loop removes much of the manual curation burden: correct solutions are discovered and verified automatically by execution, which is why the authors could skip hand-labeling the test set.
  • Because each generated solution is a computation graph over the 54 operators, the set of possible solutions is compositional: decomposing operators (e.g., combination into permutation, factorial, division) yields structurally different but equivalent programs that can serve as diverse training data.
  • Including the background operators in the input prompt accelerates the learning trajectory and improves final coverage, suggesting that exposing the vocabulary is itself an effective inductive bias.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because Algorithm 1 adds verified generated solutions to the training set before scoring, the reported test-set accuracy of 84.8% may partly reflect the model's exposure to the test problems' target programs; an independent held-out evaluation would be needed to separate memorization from generalization.
  • Editorial inference: the background-operator vocabulary functions as a soft grammar constraint that shrinks the space of outputs the model must explore, so part of the gain may come from the restricted output space rather than from Prolog semantics per se.
  • Editorial inference: the decompositional operator structure suggests a cheap route to more training data, namely composing operators into new problems or rewriting existing solutions at different levels of decomposition, without further manual labeling.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes representing mathematical reasoning problems as Prolog programs built from a fixed set of 54 hand-crafted background predicates, and introduces MATH-Prolog, a corpus derived from the counting and probability categories of the MATH dataset. The authors fine-tune Meta-Llama-3.1-8B-Instruct with LoRA and use a 5-fold cross-validated self-training procedure (Algorithm 1) that iteratively generates Prolog solutions for a held-out fold, verifies them with SWI-Prolog, and adds correct ones to the training set. They report 84.6% accuracy on the cross-validated set and 84.8% on the test set, and claim that the approach solves previously unseen problems and that including background operators in prompts improves solution coverage.

Significance. If the evaluation were sound, the paper would make a useful contribution: a structured, mechanically verifiable code-generation approach to competition-level counting and probability problems, together with a curated corpus and an interpretable graph-of-predicates representation. The corpus curation is substantial work, and the diversity examples in Appendix A.1 are illuminating. However, the central experimental claims are not trustworthy as reported. Both headline accuracies are computed after the evaluated data have been added to the training set, so the numbers cannot support the abstract's assertion that the model solves 'previously unseen problems.' With a clean held-out evaluation, the approach might still be valuable as a data-augmentation technique, but that claim is not established by the current manuscript.

major comments (3)
  1. [Section 2.2, Algorithm 1] The pseudocode explicitly adds every verified generated solution from the generation set G into the training set D (line 'Add (q,ˆa) into S and D'), and subsequent epochs fine-tune on D. In the 5-fold protocol, each fold is used as G with the remaining folds as D, so by the time the model is scored on a fold, correct generated solutions from that fold have already been incorporated into training. The 84.6% accuracy reported in Section 3.2 is therefore not an out-of-sample measure, and the claim of solving 'previously unseen problems' is unsupported by this experiment.
  2. [Section 3.2] The test-set result is explicitly in-sample. The text states: 'When we used the derived test solutions from the first-round self-training and augmented them into the training set and rerun self-training again, the test set accuracy was increased to 84.8%.' Augmenting the test solutions into the training set before reporting test accuracy means the test set is no longer held out. The same concern applies to the first-round 75.7% test accuracy: unless evaluation is performed before any test-set sample is added to D, which the manuscript does not state, the model has seen test-set solutions during training. The 84.8% number must be retracted or reframed as an in-sample data-augmentation accuracy.
  3. [Section 3.1, Evaluation] The accuracy metric (unnumbered equation under 'Evaluation') sums over folds and divides by |D|, but it does not specify which model checkpoint is used to produce spred for a fold, nor whether predictions are drawn before or after that fold's correct generated solutions are added to D. Without this specification, the metric cannot be interpreted as measuring the ability to discover solutions to unseen problems. The authors should either (i) evaluate each fold with a model trained only on the other folds and without any fold-generated samples in the training set, or (ii) clearly separate a discovery set S (solutions found) from the training set D and report accuracy computed only on checkpoints before any sample from the evaluated fold is added to D.
minor comments (4)
  1. [Section 3.1] The accuracy equation uses |D| in the denominator but the sum is over folds D_i; if D denotes the union of all folds, the notation should be defined explicitly.
  2. [Section 2 vs. Appendix A.2] The text claims 54 background operators, but Appendix A.2 appears to list fewer; the appendix should enumerate all 54 operators or clarify the count.
  3. [Section 3.2 and Figure 3] Figure 3 is referenced but not shown in the provided text, and its axes and experimental protocol are not described; the reader cannot verify the learning trajectories or the 84.6% value from the figure.
  4. [References] The reference 'Abhimanyu Dubey et. al. 2024' is malformed; the citation should use the standard author–year format with the full author list or 'et al.' as appropriate.

Circularity Check

3 steps flagged · score 7.0 of 10

The headline test-set and cross-validated accuracies are measured after the evaluated questions' verified solutions were added to the training set, so the reported 'unseen problem' generalization numbers reduce to in-sample fit by the paper's own Algorithm 1 and Section 3.2.

  1. fitted input called prediction [Section 2.2, Algorithm 1 (Self-Training Procedure)]
    "for i in I do mi ← update(mi−1,D); for (q,a) in G do ˆa ← sample(mi,q); if eval(ˆa) ==eval(a) and ˆa ̸= a then Add (q,ˆa) into S and D;"

    By construction, every correctly generated solution for the generation set G — whether G is a heldout fold or the official test set — is added to the training set D. Subsequent model updates and the final evaluated model are trained on D, so the questions in G are no longer unseen once their verified solutions enter D. The paper does not specify that accuracy is computed from a checkpoint taken before any G question's solution was added, so the claimed 'previously unseen problems' evaluation is not a held-out measurement.

  2. fitted input called prediction [Section 3.2 (Results)]
    "When we used the derived test solutions from the first-round self-training and augmented them into the training set and rerun self-training again, the test set accuracy was increased to 84.8%."

    This sentence makes the central circularity explicit: the 84.8% 'test set accuracy' is reported after the test set's own derived solutions were placed into the training set and the model was retrained. The test set is therefore not held out; the accuracy measures the model's fit to training examples derived from the test questions themselves. The abstract's claim that the approach 'successfully uncovers new solutions ... for previously unseen problems' is contradicted by this evaluation protocol. The earlier 75.7% test-set figure is similarly suspect because the first-round self-training also used the test set as the generation set, adding verified test solutions into D during the run.

1 more flagged steps
  1. fitted input called prediction [Section 3.1 (Evaluation)]
    "The metric measures the ability of a large language model to discover Prolog solutions in the unseen dataset."

    The cross-validated metric sums over the five folds, but Algorithm 1 has already added verified generated solutions from each generation fold into D during self-training. Unless the reported 84.6% was computed from a model checkpoint taken before any fold question's solution was added — which the paper does not state — the fold questions are not 'unseen' when the final model is evaluated. The term 'unseen dataset' in the metric definition is inconsistent with the augmentation loop, so the 84.6% cross-validated accuracy does not establish generalization to new problems.

full rationale

The paper's own algorithm and results description show that the headline accuracies are computed after the evaluated questions' verified Prolog solutions were added to the training set. In Algorithm 1, every correct generated solution for the generation set is added to D; in Section 3.2, the test set solutions from the first round are 'augmented them into the training set and rerun self-training again' before reporting 84.8% test accuracy. Thus the central claim of solving 'previously unseen problems' is not supported by the reported numbers: the test-set number is in-sample by construction, and the cross-validated number is vulnerable to the same contamination unless an explicit pre-addition checkpoint is used, which the paper does not document. The self-citation to Yang et al. (2024) for the GPT-4 Prolog generation procedure is not load-bearing here, and the prompt-ablation comparison (75.7% vs. 71.1%) retains some independent content, which prevents the maximum score. However, because the headline generalization results reduce to fitting on verified solutions of the evaluated questions, the circularity burden is high; score 7.

Assumptions & free parameters 11 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a hand-curated operator library, a Prolog evaluation oracle, a filtered problem split, and a self-training protocol that folds verified solutions of the evaluation set back into training. The last item is the main circularity source.

free parameters (11)
  • number of cross-validation folds K = 5
    Chosen by hand for the self-training schedule; the reported accuracy depends on the fold split.
  • max solutions per question J = 2
    Stops sampling after two distinct correct solutions per problem (Algorithm 1); affects diversity and amount of extra data.
  • LoRA rank r = 8
    Selected after exploring hyperparameters (Appendix A.6); affects fine-tuning capacity.
  • LoRA scaling alpha = 16
    Selected to trade off performance and compute (Appendix A.6).
  • learning rate = 3e-4
    Fixed for all self-training runs (Appendix A.6).
  • sampling temperature = 0.6
    Sampling hyperparameter for generating diverse solutions (Appendix A.6).
  • top-k = 40
    Sampling hyperparameter (Appendix A.6).
  • top-p = 0.9
    Sampling hyperparameter (Appendix A.6).
  • training epochs = 100
    Self-training runs 100 epochs (Appendix A.6).
  • batch size = 16
    Fixed training batch size (Appendix A.6).
  • token cutoff = 2700
    Samples exceeding 2700 tokens are removed to fit a 4090 GPU (Section 3.1).
assumptions (6)
  • domain assumption The 54 hand-crafted background operators are a correct and sufficiently expressive basis for all retained counting/probability problems.
    Invoked in Section 2: all Prolog solutions are standardized to this operator set; if any operator is wrong or missing, the corpus and generated solutions fail.
  • domain assumption A generated Prolog program whose evaluated output equals the reference answer is a correct solution, even if its reasoning path differs.
    Section 3.1 defines accuracy by equality of Prolog evaluation output with the reference answer, so semantic errors that coincidentally give the right number are treated as correct.
  • ad hoc to paper The 625 retained training problems are representative of the counting/probability distribution after filtering out 146 unsolvable ones.
    Section 3.1 reports filtering without a stated rule; if excluded problems are the hard tail, cross-validated accuracy is optimistic.
  • ad hoc to paper Adding verified generated solutions of the evaluation fold or of the official test set back into training does not invalidate the reported accuracy.
    Algorithm 1 adds correct generated samples to D, and Section 3.2 augments verified test solutions before rerunning self-training; the reported 84.8% assumes this is still a valid held-out measurement.
  • domain assumption SWI-Prolog evaluation is deterministic and complete for the operator set.
    The evaluation metric calls P(a) the SWI-Prolog interpreter; timeouts, nontermination, or missing libraries would change accuracy.
  • domain assumption LoRA fine-tuning on Llama-3.1-8B with the chosen hyperparameters can learn the 54-operator Prolog generation task.
    The whole experimental section depends on the model learning the mapping from problem statement to Prolog code under LoRA.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Mathematical Reasoning in LLMs with Background Operators." pith.science (2026). https://pith.science/paper/324QXPG2

@misc{pith2026241204110,
  author       = {Pith},
  title        = {Pith review of: Enhancing Mathematical Reasoning in LLMs with Background Operators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/324QXPG2}},
  note         = {Machine review of arXiv:2412.04110}
}
read the original abstract

We propose utilizing background operators for mathematical reasoning in large language models (LLMs). To achieve this, we define a set of fundamental mathematical predicates as the basic building blocks. For each mathematical problem, we develop a Prolog solution that includes problem-specific predicates and intermediate predicates derived from these background operators, ensuring that each solution adheres to the defined operator set. We introduce the MATH-Prolog corpus, which is derived from the counting and probability categories of the MATH corpus. For efficient data augmentation, we apply K-fold cross-validated self-training. This method incrementally generates new Prolog solutions for each fold, incorporating those verified as correct into the training set throughout the model training process. Our experimental results demonstrate that 5-fold crossvalidated self-training effectively identifies new, accurate Prolog solutions, achieving an accuracy of 84.6% on the cross-validated set, and 84.8% on the test set during fine-tuning the Meta-Llama-3.1-8B-Instruct model. This approach successfully uncovers new solutions with fully computable inference steps for previously unseen problems. Additionally, incorporating the background mathematical predicates into the prompt enhances solution coverage.

Figures

Figures reproduced from arXiv: 2412.04110 by the authors.

Figure 1
Figure 1. Computation graph of a Prolog solution for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A Prolog code solution for the problem “If [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Accuracy for 5-fold cross-validated self [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Diversity between original solution and augmented solution. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 7 canonical work pages

  1. [1]

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2023. https://arxiv.org/abs/2308.09687 Graph of thoughts: Solving elaborate problems with large language models . Preprint, arXiv:2308.09687

    arXiv 2023

  2. [2]

    Piantadosi

    Nasim Borazjanizadeh and Steven T. Piantadosi. 2024. https://arxiv.org/abs/2407.11373 Reliable reasoning beyond natural language . Preprint, arXiv:2407.11373

    arXiv 2024

  3. [3]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. https://arxiv.org/abs/2211.12588 Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks . Preprint, arXiv:2211.12588

    arXiv 2023

  4. [4]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. https://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . Preprint, arXiv:2110.14168

    arXiv 2021

  5. [5]

    Abhimanyu Dubey et. al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  6. [6]

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. https://arxiv.org/abs/2211.10435 Pal: Program-aided language models . Preprint, arXiv:2211.10435

    arXiv 2023

  7. [7]

    Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. 2023. https://openreview.net/forum?id=KkHY1WGDII Grammar-constrained decoding for structured NLP tasks without finetuning . In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  8. [8]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. https://arxiv.org/abs/2309.17452 Tora: A tool-integrated reasoning agent for mathematical problem solving . Preprint, arXiv:2309.17452

    arXiv 2023

Show all 32 references
  1. [9]

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS

    2021

  2. [10]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  3. [11]

    Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. https://arxiv.org/abs/2210.11610 Large language models can self-improve . Preprint, arXiv:2210.11610

    2022 arXiv

  4. [12]

    Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, and James Kwok. 2024. https://aclanthology.org/2024.findings-acl.397 Forward-backward reasoning in large language models for mathematical verification . In Findings of the Association for Computational Lin...

    2024

  5. [13]

    Thomas Kipf and Max Welling. 2016. https://api.semanticscholar.org/CorpusID:3144218 Semi-supervised classification with graph convolutional networks . ArXiv, abs/1609.02907

    2016 arXiv

  6. [14]

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng Tu, and Shuming Shi. 2023. https://arxiv.org/abs/2305.19118 Encouraging divergent thinking in large language models through multi-agent debate . Preprint, arXiv:2305.19118

    2023 arXiv

  7. [15]

    Smith, and Yejin Choi

    Ximing Lu, Sean Welleck, Peter West, Liwei Jiang, Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lianhui Qin, Youngjae Yu, Rowan Zellers, Noah A. Smith, and Yejin Choi. 2022. https://doi.org/10.18653/v1/2022.naacl-main.57 N euro L ogic a*esque decoding: Constrained text generati...

    2022 doi

  8. [16]

    Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2023. https://arxiv.org/abs/2210.17517 Lila: A unified benchmark for mathematical reasoning . Preprint, arX...

    2023 arXiv

  9. [17]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. https://proceedings.mlr.press/v202/shao23a.html Synthetic prompting: Generating chain-of-thought demonstrations for large language models . In Proceedings of the 40th International Conference...

    2023

  10. [18]

    Xiaoyu Tan, Yongxin Deng, Xihe Qiu, Weidi Xu, Chao Qu, Wei Chu, Yinghui Xu, and Yuan Qi. 2024. https://arxiv.org/abs/2407.14562 Thought-like-pro: Enhancing reasoning of large language models through self-driven prolog-based chain-of-thought . Preprint, arXiv:2407.14562

    2024 arXiv

  11. [19]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca

    2023

  12. [20]

    Trieu Trinh, Yuhuai Wu, Quoc Le, He He, and Thang Luong. 2024. https://doi.org/10.1038/s41586-023-06747-5 Solving olympiad geometry without human demonstrations . Nature

    2024 doi

  13. [21]

    Priyesh Vakharia, Abigail Kufeldt, Max Meyers, Ian Lane, and Leilani H. Gilpin. 2024. Proslm: A prolog synergized language model for explainable domain specific knowledge based question answering. In Neural-Symbolic Learning and Reasoning, pages 291--304, Cham. Springer Nature...

    2024

  14. [22]

    Tianduo Wang, Shichen Li, and Wei Lu. 2024. Self-training with direct preference optimization improves chain-of-thought reasoning. In Proceedings of ACL

    2024

  15. [23]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824--24837

    2022

  16. [24]

    Xiaocheng Yang, Bingsen Chen, and Yik-Cheung Tam. 2024. https://doi.org/10.18653/v1/2024.naacl-short.61 Arithmetic reasoning with LLM : P rolog generation & permutation . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L...

    2024 doi

  17. [25]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models . Preprint, arXiv:2305.10601

    2023 arXiv

  18. [26]

    Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations

    2023

  19. [27]

    Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020. https://doi.org/10.18653/v1/2020.acl-main.362 Graph-to-tree learning for solving math word problems . In Proceedings of the 58th Annual Meeting of the Association for Computational Lingu...

    2020 doi

  20. [28]

    Qihao Zhu Runxin Xu Junxiao Song Mingchuan Zhang Y.K. Li Y. Wu Daya Guo Zhihong Shao, Peiyi Wang. 2024. https://arxiv.org/abs/2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . CoRR, abs/2402.03300

    2024 arXiv

  21. [29]

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. 2023. https://arxiv.org/abs/2205.10625 Least-to-most prompting enables complex reasoning in large language models . Preprint, arXiv...

    2023 arXiv

  22. [30]

    Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang. 2023. https://doi.org/10.18653/v1/2023.acl-long.245 Solving math word problems via cooperative reasoning induced language models . In Proceedings of the 61st Annual Meeti...

    2023 doi

  23. [31]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  24. [32]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.