Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces the Mixture of Refinement Agents (MoRA), an iterative loop in which GPT-4o identifies the first error type in a chain-of-thought physics solution and routes the solution to a specialized refiner, claiming accuracy…

desk verdict A plausible agentic-routing framework for physics QA whose empirical claim is overstated by one contradictory result and an unvalidated critic. read the letter →

arxiv 2412.00821 v1 pith:UI5SMCQI submitted 2024-12-01 cs.AI

Raj Jaiswal , Dhruv Jain , Harsh Parimal Popat , Avinash Anand , Abhishek Dharmadhikari , Atharva Marathe , Rajiv Ratn Shah This is my paper · ORCID
classification cs.AI
keywords physicsreasoningmixtureofrefinementagentserroridentificationchain-of-thoughtcodeexecutiongraphretrievalagentroutingmultiple-choiceQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Open-source language models solve physics problems much less reliably than proprietary ones, mostly because their step-by-step solutions misread the question, apply the wrong formula, or miscalculate. This paper claims that a Mixture of Refinement Agents (MoRA) can close most of that gap without fine-tuning by letting a stronger model, GPT-4o, locate the first error and then handing the solution to a specialist agent designed for that error type. The paper reports that this loop lifts final-answer accuracy by up to 16 percentage points for Llama-3-70B and Gemma-2-27B on a new high-school physics dataset, with smaller but consistent gains on established science benchmarks. If that holds, it offers a cheap, training-free route to better scientific reasoning in open-source models: use a strong critic to point at the mistake, and let a weaker solver repair just that part. The visible exception in the reported tables is Gemma-2-27B on MMLU High School, where MoRA does not beat the chain-of-thought baseline.

What carries the argument

The load-bearing object is the score-and-route loop the paper calls the Mixture of Refinement Agents. GPT-4o inspects a generated solution and returns two binary flags, one for objective alignment and one for variable application, plus two scores that run from 0 to 1 and mark the step where the first conceptual or computational error occurs. The loop activates the miscomprehension agent if either flag is negative, then the concept agent if the concept score falls below a threshold, then the computation agent if the computation score falls below a threshold. This prioritization follows the human solving order—understand the problem, choose the principle, then calculate—and each agent rewrites only from the detected failure step onward. The score-guided localization is what distinguishes MoRA from a generic ask-the-model-to-check-its-answer loop, because it tells the refiner exactly where to start.

What would settle it

Take a random sample of incorrect PhysicsQA solutions, ask human annotators to mark the first error step and type, run GPT-4o's error-identification prompt on the same solutions, and measure agreement; if agreement is close to chance, or if MoRA's accuracy gain disappears when agent routing is randomized while the agents themselves stay the same, the central claim fails.

Watch

Extended reading notes

Core claim

MoRA is an iterative refinement framework built around a simple diagnosis: wrong physics solutions from open-source LLMs fail in one of three ways—miscomprehension, wrong concept or formula, or arithmetic error—and those failure types can be repaired by different tools. At each iteration GPT-4o checks the solution with two flags (objective alignment and variable application) and two scores that localize the first failing conceptual or computational step. The framework then routes the solution, in priority order, to a miscomprehension agent (instruction prompting), a concept agent (retrieval of correct formulae from a physics knowledge base via GraphRAG), or a computation agent (Python code generation and execution). The paper's headline result is that MoRA lifts Gemma-2-27B on PhysicsQA from 59.45% (3-shot) to 70.62% and Llama-3-70B from 59.29% to 70.14%, with similar gains on SciEval-Static and MMLU College; the tables also include one flat result, Gemma-2-27B on MMLU High School at 75.88% versus 77.11% for CoT.

Load-bearing premise

The whole framework depends on GPT-4o being right about where a solution first goes wrong; if the critic mislabels a correct step or misses a real error, the repair agents rewrite the wrong part and the reported accuracy gains could shrink or reverse.

Editorial extensions

If this is right

  • If the reported gains replicate, open-source 27B- and 70B-class models can approach GPT-4o-level physics accuracy on multiple-choice benchmarks without any fine-tuning or task-specific training.
  • The largest gains appear on PhysicsQA and SciEval-Static, where the three targeted error types are most common, so the framework's value is highest on multihop physics problems rather than general knowledge questions.
  • Computational refinement is the strongest single agent, with refinement rates of roughly 73% on PhysicsQA for both models, which points to code execution as the most reliable repair mechanism for arithmetic failures.
  • The priority ordering (miscomprehension, then concept, then computation) is itself part of the method: fixing lower-level errors first prevents the later agents from reworking a solution aimed at the wrong problem.
  • The one reported non-improvement, Gemma-2-27B on MMLU High School, means the framework does not guarantee a gain on every benchmark and that the margin can depend on the baseline chosen.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test would be to swap GPT-4o for an open-source critic of similar size to the solver; if routing quality degrades, the method's gains are tied to the critic's error-location ability rather than to the refinement agents themselves.
  • The error taxonomy (misread, wrong principle, wrong arithmetic) is general enough that the same score-and-route pattern could be applied to other technical reasoning domains, such as organic chemistry or circuits, by swapping the knowledge base and the code-execution tool.
  • Because concept-refinement rates on PhysicsQA are only about 47–49%, the ceiling for that agent appears to be set by the solver's ability to convert the location of a conceptual error into a good retrieval query, not by the framework's routing logic.
  • The reported per-agent refinement rates are computed only on errors that GPT-4o found, so if the critic systematically misses a class of error, the ablations overstate how often that agent succeeds on the true distribution of mistakes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes MoRA, an iterative refinement framework for physics multiple-choice question answering. In each iteration GPT-4o labels an open-source LLM's chain-of-thought solution with three error types (problem miscomprehension, incorrect concept application, computational error) and produces scores; a prioritized routing algorithm then activates one of three refinement agents: instruction-based miscomprehension correction, GraphRAG-based concept retrieval and refinement, or Python code generation/execution for computation. The authors evaluate on SciEval-Static, a newly curated PhysicsQA set, and MMLU High School/College, reporting gains over AO/CoT/3-shot baselines for Llama-3-70B and Gemma-2-27B, including up to 16 percentage points on PhysicsQA.

Significance. If the headline gains are statistically and reproducibly supported, MoRA is a useful, fine-tuning-free recipe for improving open-source LLMs on physics reasoning, and the decomposition into three error types is a sensible organizing principle. The manual error analysis in Table 4 and the per-agent refinement rates in Table 5 are informative for the community's understanding of where open-source models fail. The main significance is conditional, however, on two points that the paper does not yet establish: the reliability of the GPT-4o critic that gates all routing, and the consistency of the improvement claim across all reported benchmarks.

major comments (4)
  1. [Results, Table 3] The sentence in the Results section that MoRA 'delivers marked improvements across all benchmarks' is not supported by Table 3: Gemma-2-27B on MMLU High School drops from 77.11% (CoT) to 75.88% (MoRA). Since the abstract and conclusion claim general improvement, the scope of the claim should be restated or this drop should be explained (for example, as a single stochastic run or as a known routing failure mode).
  2. [Error Identification, Algorithm 1] Algorithm 1 treats (Fobj, Fval, Score_concept, Score_comp) returned by GPT-4o as ground truth and routes refinement on that basis, but the paper gives no evaluation of the accuracy of these error labels. The 'Error Identification' section justifies GPT-4o only with 'our experiments and analysis' and no quantitative agreement study. Since the authors already perform manual error analysis for Table 4, they could report GPT-4o's agreement with human labels, including precision/recall per error type and step-localization accuracy. Without this, the framework's success is not causally explained: the gains could change if the critic misroutes a correct solution to a refinement agent.
  3. [Experiments, Tables 2 and 3] No confidence intervals, significance tests, or multiple-run statistics are reported. Several claimed improvements are small in absolute question counts: for SciEval-Static (n=164), Llama-3-70B improves from 82.23% to 86.58%, approximately 7 questions, and for MMLU College (n=118), from 71.76% to 78.82%, approximately 8 questions. The word 'significant' needs statistical support, for example binomial confidence intervals or a paired test across items, before the 'all benchmarks' improvement claim can be accepted.
  4. [Ablation, Table 5] The ablation narrative is inconsistent with the data in Table 5. The text says that miscomprehension refinement achieves 'a perfect 100% refinement rate' on MMLU datasets for both models, but Table 5 reports 37.5%/33.3% (MMLU College) and 16.7%/37.5% (MMLU High School) for miscomprehension. The values 66.7% and 62.5% for PhysicsQA, attributed in the text to miscomprehension, are the concept-refinement values in the table. The definition of 'refinement rate' and the correct reading of Table 5 need to be clarified and corrected.
minor comments (5)
  1. [Dataset: PhysicsQA] PhysicsQA is not released and no sample questions are provided; the dataset section gives only topic percentages, so the main custom benchmark cannot be inspected or reproduced by other groups.
  2. [Algorithm 1 and Experiments] The hyperparameters in Algorithm 1, namely maximum iterations N, the threshold epsilon, and the computation error tolerance, are never specified in the experiments section.
  3. [Error Identification, score definitions] The formula for Score_concept has no explicit treatment of the case N=1, and the routing condition Score < 1 - epsilon depends on the unspecified value of epsilon.
  4. [Throughout] The manuscript contains numerous grammatical and formatting issues, including 'LLMs struggles', inconsistent 'LLaMa'/'Llama' spelling, and 'Gemma-2-70B' in the Ablation section where Gemma-2-27B is meant; these should be cleaned up.
  5. [Related Works] Several inline citation artifacts, such as '(Anand et al. 2023b)' and '(Anand et al. 2023c)' appearing in the middle of sentences, should be removed or properly integrated into the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MoRA's claimed gains are empirical, measured against external answer keys; GPT-4o's flags are intermediate routing signals, not fitted predictions.

full rationale

The paper's claim is an empirical result, not a derivation: MoRA's accuracy is computed by directly comparing final answers to answer keys, so the headline improvement cannot reduce by construction to the method's own error-identification scores. Algorithm 1 uses GPT-4o's flags and scores as routing intermediates, but these are never fitted to the final accuracy and are not renamed as predictions; they are inputs to a refinement process whose output is judged externally. The self-citations to Anand et al. are scattered and peripheral, none carrying the load of the central framework or the evaluation. The self-curated PhysicsQA dataset is a data-selection choice, not a circular step. The unvalidated assumption that GPT-4o reliably locates errors (stated in 'Error Identification' as 'Our experiments and analysis shows that GPT-4o showcases superior performance') and the internal inconsistency in Table 3 (Gemma-2-27B MMLU High School drops from 77.11% CoT to 75.88% MoRA despite the text claiming improvements across all benchmarks) are correctness and consistency concerns, not circularity. No equation in the paper equates an output to an input, and no fitted parameter is presented as a prediction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The framework's performance rests on the reliability of the GPT-4o critic, the completeness of the three-error taxonomy, the quality of the physics knowledge base and GraphRAG retrieval, the correctness of generated code, and the assumption that multiple-choice accuracy reflects physics reasoning. None of these are validated independently in the paper, and the new PhysicsQA dataset is not released. No free constants are fitted to produce the reported improvements, but several hyperparameters (iteration limit, score threshold, computation tolerance) are unreported.

free parameters (3)
  • maximum_iterations (N)
    Algorithm 1 loops until errors are resolved or a maximum iteration limit N is reached; its value is never reported.
  • score threshold (epsilon)
    Algorithm 1 uses epsilon when comparing concept and computation scores to 1; the threshold is not specified.
  • computation error tolerance = 0.1
    Used in the Computation Verification Score to decide whether arithmetic results match; affects which computations are flagged as errors.
assumptions (5)
  • domain assumption GPT-4o correctly identifies errors and localizes the first erroneous step in open-source LLM solutions.
    The entire routing depends on the flags and scores returned by GPT-4o; no validation or accuracy measurement of this error-identification step is reported.
  • ad hoc to paper The three error categories (problem miscomprehension, incorrect concept application, and computational errors) are exhaustive and workable for routing.
    Introduced from manual observations; no taxonomy validation or inter-annotator agreement is reported.
  • domain assumption The physics knowledge base KP and GraphRAG retrieval return the correct concept or formula at the failure stage.
    KP is not described or released; refinement relies on retrieval quality.
  • domain assumption Generated Python code and its execution correctly compute the needed arithmetic or algebraic value.
    Computational refinement assumes code execution is correct; no checks against ground truth are shown.
  • domain assumption Final-answer accuracy on multiple-choice benchmarks measures physics reasoning ability.
    Used as evaluation; multiple-choice format may reward guessing and does not isolate reasoning quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents." pith.science (2026). https://pith.science/paper/UI5SMCQI

@misc{pith2026241200821,
  author       = {Pith},
  title        = {Pith review of: Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UI5SMCQI}},
  note         = {Machine review of arXiv:2412.00821}
}
read the original abstract

Large Language Models (LLMs) demonstrate remarkable capabilities in various reasoning tasks. However, they encounter significant challenges when it comes to scientific reasoning, particularly in physics, which requires not only mathematical reasoning but also factual and conceptual understanding. When addressing complex physics problems, LLMs typically face three key issues: problem miscomprehension, incorrect concept application, and computational errors. While each of these problems can be addressed individually, there is a need for a generalized approach that can tackle all three issues simultaneously. To address this, we introduce Mixture of Refinement Agents (MoRA), a novel agentic refinement framework that iteratively refines the LLM generated base solution by correcting the aforementioned errors, resulting in a significant performance improvement for open-source LLMs. Our approach aims to bridge the gap between opensource LLMs and GPT-4o by utilizing the latter as error identifier to guide these refinement agents. We evaluate our approach on the SciEval and MMLU subsets along with our own physics dataset (PhysicsQA). MoRA significantly improves the performance of Llama-3-70B and Gemma-2-27B on these datasets, achieving up to a 16% increase in final answer accuracy.

Figures

Figures reproduced from arXiv: 2412.00821 by the authors.

Figure 1
Figure 1. The illustration of three key error observations in the CoT solution of open source LLMs for physics problems. (a) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The illustration of thought generation and concept retrieval for conceptual error refinement in LLM response. Given [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The illustration of code generation and execution for computation error refinement in LLM response. Given the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. UGPhysics: A Comprehensive Benchmark for Undergraduate Physics Reasoning with Large Language Models

    cs.CL 2025-02 conditional novelty 6.0 of 10

    UGPhysics is a new bilingual benchmark of 5,520 undergraduate physics problems; the strongest tested LLM, OpenAI o1-mini, reaches only 49.8% accuracy.

Reference graph

Works this paper leans on

44 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anand, A.; Addala, K.; Baghel, K.; Goel, A.; Hira, M.; Gupta, R.; and Shah, R. R. 2023 a . Revolutionizing High School Physics Education: A Novel Dataset. In Goyal, V.; Kumar, N.; Bhowmick, S. S.; Goyal, P.; Goyal, N.; and Kumar, D., eds., Big Data and Artificial Intelligence, 64--79. Cham: Springer Nature Switzerland. ISBN 978-3-031-49601-1

    work page 2023

  2. [2]

    Anand, A.; Goel, A.; Hira, M.; Buldeo, S.; Kumar, J.; Verma, A.; Gupta, R.; and Shah, R. R. 2023 b . Sciphyrag-retrieval augmentation to improve llms on physics q &a. In International Conference on Big Data Analytics, 50--63. Springer

    work page 2023

  3. [3]

    Anand, A.; Gupta, M.; Prasad, K.; Goel, U.; Lal, N.; Verma, A.; and Shah, R. R. 2023 c . KG-CTG: citation generation through knowledge graph-guided large language models. In International Conference on Big Data Analytics, 37--49. Springer

    work page 2023

  4. [4]

    R.; and Shah, R

    Anand, A.; Gupta, M.; Prasad, K.; Singla, N.; Sanjeev, S.; Kumar, J.; Shivam, A. R.; and Shah, R. R. 2024 a . Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks. arXiv preprint arXiv:2404.13099

    arXiv 2024

  5. [5]

    R.; and Satoh, S

    Anand, A.; Jairath, A.; Lal, N.; Bangar, S.; Sikka, J.; Verma, A.; Shah, R. R.; and Satoh, S. 2023 d . GEC-DCL: Grammatical Error Correction Model with Dynamic Context Learning for Paragraphs and Scholarly Papers. In International Conference on Big Data Analytics, 95--110. Springer

    work page 2023

  6. [6]

    R.; Prasad, K.; Kumar, S.; Verma, A.; et al

    Anand, A.; Jaiswal, R.; Dharmadhikari, A.; Marathe, A.; Popat, H.; Mital, H.; Nair, A. R.; Prasad, K.; Kumar, S.; Verma, A.; et al. 2024 b . GeoVQA: A Comprehensive Multimodal Geometry Dataset for Secondary Education. In 2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR), 102--108. IEEE

    work page 2024

  7. [7]

    MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering

    Anand, A.; Kapuriya, J.; Kirtani, C.; Singh, A.; Saraf, J.; Lal, N.; Kumar, J.; Shivam, A. R.; Verma, A.; Shah, R. R.; et al. 2024 c . MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering. arXiv preprint arXiv:2404.12926

    work page Pith review arXiv 2024

  8. [8]

    Anand, A.; Kapuriya, J.; Singh, A.; Saraf, J.; Lal, N.; Verma, A.; Gupta, R.; and Shah, R. 2024 d . MM-PhyQA: Multimodal Physics Question-Answering with Multi-image CoT Prompting. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, 53--64. Springer

    work page 2024

Show all 44 references
  1. [9]

    Anand, A.; Prasad, K.; Goel, U.; Gupta, M.; Lal, N.; Verma, A.; and Shah, R. R. 2023 e . Context-enhanced language models for generating multi-paper citations. In International Conference on Big Data Analytics, 80--94. Springer

    2023

  2. [10]

    G.; et al

    Arora, D.; Singh, H. G.; et al. 2023. Have llms advanced enough? a challenging problem solving benchmark for large language models. arXiv preprint arXiv:2305.15074

    2023 arXiv

  3. [11]

    D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877--1901

    2020

  4. [12]

    Chen, W.; Ma, X.; Wang, X.; and Cohen, W. W. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588

    2022 arXiv

  5. [13]

    Chern, E.; Zou, H.; Li, X.; Hu, J.; Feng, K.; Li, J.; and Liu, P. 2023. Generative ai for math: Abel. URL https://github. com/GAIR-NLP/abel

    2023

  6. [14]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  7. [15]

    Edge, D.; Trinh, H.; Cheng, N.; Bradley, J.; Chao, A.; Mody, A.; Truitt, S.; and Larson, J. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130

    2024 arXiv

  8. [16]

    Fu, Y.; Peng, H.; Sabharwal, A.; Clark, P.; and Khot, T. 2022. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations

    2022

  9. [17]

    Gao, L.; Madaan, A.; Zhou, S.; Alon, U.; Liu, P.; Yang, Y.; Callan, J.; and Neubig, G. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, 10764--10799. PMLR

    2023

  10. [18]

    L.; Shen, J.; Hu, J.; Han, X.; Huang, Y.; Zhang, Y.; et al

    He, C.; Luo, R.; Bai, Y.; Hu, S.; Thai, Z. L.; Shen, J.; Hu, J.; Han, X.; Huang, Y.; Zhang, Y.; et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008

    2024 arXiv

  11. [19]

    Hendrycks, D.; Burns, C.; Kadavath, S.; Arora, A.; Basart, S.; Tang, E.; Song, D.; and Steinhardt, J. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874

    2021 arXiv

  12. [20]

    u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \

    Lewis, P.; Perez, E.; Piktus, A.; Petroni, F.; Karpukhin, V.; Goyal, N.; K \"u ttler, H.; Lewis, M.; Yih, W.-t.; Rockt \"a schel, T.; et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33: 9459--9474

    2020

  13. [21]

    Li, X.; Wang, W.; Li, M.; Guo, J.; Zhang, Y.; and Feng, F. 2024. Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction. arXiv preprint arXiv:2406.00755

    2024 arXiv

  14. [22]

    Ling, Z.; Fang, Y.; Li, X.; Huang, Z.; Lee, M.; Memisevic, R.; and Su, H. 2024. Deductive verification of chain-of-thought reasoning. Advances in Neural Information Processing Systems, 36

    2024

  15. [23]

    Lu, P.; Mishra, S.; Xia, T.; Qiu, L.; Chang, K.-W.; Zhu, S.-C.; Tafjord, O.; Clark, P.; and Kalyan, A. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507--2521

    2022

  16. [24]

    Luo, H.; Sun, Q.; Xu, C.; Zhao, P.; Lou, J.; Tao, C.; Geng, X.; Lin, Q.; Chen, S.; and Zhang, D. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583

    2023 arXiv

  17. [25]

    Ma, Y.; Gou, Z.; Hao, J.; Xu, R.; Wang, S.; Pan, L.; Yang, Y.; Cao, Y.; and Sun, A. 2024. SciAgent: Tool-augmented Language Models for Scientific Reasoning. arXiv preprint arXiv:2402.11451

    2024 arXiv

  18. [26]

    W.; and Rainforth, T

    Miao, N.; Teh, Y. W.; and Rainforth, T. 2023. Selfcheck: Using llms to zero-shot check their own step-by-step reasoning. arXiv preprint arXiv:2308.00436

    2023 arXiv

  19. [27]

    Mruthyunjaya, V.; Pezeshkpour, P.; Hruschka, E.; and Bhutani, N. 2023. Rethinking language models as symbolic knowledge graphs. arXiv preprint arXiv:2308.13676

    2023 arXiv

  20. [28]

    Ouyang, S.; Zhang, Z.; Yan, B.; Liu, X.; Han, J.; and Qin, L. 2023. Structured chemistry reasoning with large language models. arXiv preprint arXiv:2311.09656

    2023 arXiv

  21. [29]

    Petroni, F.; Piktus, A.; Fan, A.; Lewis, P.; Yazdani, M.; De Cao, N.; Thorne, J.; Jernite, Y.; Karpukhin, V.; Maillard, J.; et al. 2020. KILT: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252

    2020 arXiv

  22. [30]

    Sun, J.; Luo, Y.; Gong, Y.; Lin, C.; Shen, Y.; Guo, J.; and Duan, N. 2023 a . Enhancing chain-of-thoughts prompting with iterative bootstrapping in large language models. arXiv preprint arXiv:2304.11657

    2023 arXiv

  23. [31]

    Sun, L.; Han, Y.; Zhao, Z.; Ma, D.; Shen, Z.; Chen, B.; Chen, L.; and Yu, K. 2023 b . SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research. arXiv preprint arXiv:2308.13149

    2023 arXiv

  24. [32]

    Sun, L.; Han, Y.; Zhao, Z.; Ma, D.; Shen, Z.; Chen, B.; Chen, L.; and Yu, K. 2024. Scieval: A multi-level large language model evaluation benchmark for scientific research. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 19053--19061

    2024

  25. [33]

    Tyen, G.; Mansoor, H.; Carbune, V.; Chen, P.; and Mak, T. 2024. LLM s cannot find reasoning errors, but can correct them given the error location. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics ACL 2024, 13894--1390...

    2024

  26. [34]

    Wang, K.; Ren, H.; Zhou, A.; Lu, Z.; Luo, S.; Shi, W.; Zhang, R.; Song, L.; Zhan, M.; and Li, H. 2023. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. arXiv preprint arXiv:2310.03731

    2023 arXiv

  27. [35]

    V.; Zhou, D.; et al

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824--24837

    2022

  28. [36]

    Xu, B.; Yang, A.; Lin, J.; Wang, Q.; Zhou, C.; Zhang, Y.; and Mao, Z. 2023. Expertprompting: Instructing large language models to be distinguished experts. arXiv preprint arXiv:2305.14688

    2023 arXiv

  29. [37]

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  30. [38]

    H.; and Zhou, D

    Yasunaga, M.; Chen, X.; Li, Y.; Pasupat, P.; Leskovec, J.; Liang, P.; Chi, E. H.; and Zhou, D. 2023. Large language models as analogical reasoners. arXiv preprint arXiv:2310.01714

    2023 arXiv

  31. [39]

    T.; Li, Z.; Weller, A.; and Liu, W

    Yu, L.; Jiang, W.; Shi, H.; Yu, J.; Liu, Z.; Zhang, Y.; Kwok, J. T.; Li, Z.; Weller, A.; and Liu, W. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284

    2023 arXiv

  32. [40]

    Yuan, Z.; Yuan, H.; Li, C.; Dong, G.; Lu, K.; Tan, C.; Zhou, C.; and Zhou, J. 2024. Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

    2024

  33. [41]

    Zhang, Z.; Zhang, A.; Li, M.; and Smola, A. 2022. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493

    2022 arXiv

  34. [42]

    Zhou, A.; Wang, K.; Lu, Z.; Shi, W.; Luo, S.; Qin, Z.; Lu, S.; Jia, A.; Song, L.; Zhan, M.; et al. 2023. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921

    2023 arXiv

  35. [43]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.