REVIEW 36 cited by
MathPrompter: Mathematical Reasoning using Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Large Language Models (LLMs) have limited performance when solving arithmetic reasoning tasks and often provide incorrect answers. Unlike natural language understanding, math problems typically have a single correct answer, making the task of generating accurate solutions more challenging for LLMs. To the best of our knowledge, we are not aware of any LLMs that indicate their level of confidence in their responses which fuels a trust deficit in these models impeding their adoption. To address this deficiency, we propose `MathPrompter', a technique that improves performance of LLMs on arithmetic problems along with increased reliance in the predictions. MathPrompter uses the Zero-shot chain-of-thought prompting technique to generate multiple Algebraic expressions or Python functions to solve the same math problem in different ways and thereby raise the confidence level in the output results. This is in contrast to other prompt based CoT methods, where there is no check on the validity of the intermediate steps followed. Our technique improves over state-of-the-art on the MultiArith dataset ($78.7\%\rightarrow92.5\%$) evaluated using 175B parameter GPT-based LLM.
Forward citations
Cited by 36 Pith papers
-
IGC: Integrating a Gated Calculator into an LLM to Solve Arithmetic Tasks Reliably and Efficiently
A gated calculator module inserted into a frozen Llama 3.1 8B model enables near-perfect arithmetic on BigBench, including multiplication, in a single forward pass without external tools.
-
VLM-R$^3$: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought
VLM-R3 uses reinforcement learning to teach a multimodal model when and where to crop an image during its chain-of-thought, improving scores on several visual reasoning benchmarks.
-
GaussMark: A Practical Approach for Structural Watermarking of Language Models
GaussMark embeds a detectable watermark by adding per-generation Gaussian noise to one weight matrix and detecting gradient alignment with that noise.
-
Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning
A two-stage pattern-aware tool-integrated reasoning method raises code usage and code-plus-correct metrics on math benchmarks, but the paper conflates Code@1 with problem-solving accuracy in its headline claims.
-
SVAgent: AI Agent for Hardware Security Verification Assertion
SVAgent is a prompt-engineering framework that decomposes security requirements into sub-questions to generate SystemVerilog assertions with higher reported accuracy and consistency than direct LLM generation.
-
Direct Behavior Optimization: Unlocking the Potential of Lightweight LLMs
A gradient-free Monte Carlo tree search over JSON key-step plans produces few-shot demonstrations that let LLaMA3-8B and LLaMA3.2-3B outperform GPT-3.5 on most of seven BIG-Bench Hard tasks.
-
Structured Pruning for Diverse Best-of-N Reasoning Optimization
SPRINT learns to select which attention heads to prune per question, improving Pass@N over random head selection and multinomial sampling on MATH500 and GSM8K.
-
More or Less Wrong: A Benchmark for Directional Bias in LLM Comparative Reasoning
Comparative words in prompts can shift LLM answers toward the framed direction in simple arithmetic comparisons, with demographic terms amplifying the effect.
-
Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?
Fine-tuning on data aligned with an LLM's prior knowledge induces overconfidence, and CogCalib mitigates this by gating a calibration loss to known data.
-
CellVerse: Do Large Language Models Really Understand Cell Biology?
CellVerse evaluates 14 LLMs on language-formatted single-cell tasks and finds that generalist models perform poorly, with drug response prediction near random guessing.
-
Evaluating Intermediate Reasoning of Code-Assisted Large Language Models for Mathematics
Code-assisted LLMs often get correct math answers with brute-force or memorized programs, so execution accuracy overstates reasoning quality.
-
Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learning
A frozen large vision-language model feeds text descriptions to a small RL-trained language model, which matches or beats much larger proprietary multimodal models on selected embodied spatial reasoning benchmarks.
-
LLMs can be easily Confused by Instructional Distractions
A new benchmark, DIM-Bench, shows that LLMs frequently follow instructions hidden inside the target input rather than the user's actual instruction, even when explicitly told to ignore them.
-
A Tool for In-depth Analysis of Code Execution Reasoning of Large Language Models
ExeRScope uses static and dynamic program analysis to show that LLMs' code execution reasoning degrades with program complexity, nested structures, longer loops, and non-primitive types.
-
Disentangling Exploration of Large Language Models by Optimal Exploitation
Exploration by LLM agents can be measured separately from exploitation using an optimal exploitation oracle, and most models explore poorly, with exploration performance correlated to reasoning ability.
-
GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training
GeoX achieves state-of-the-art accuracy on GeoQA, UniGeo, Geometry3K, and PGPS9K geometry benchmarks using formalized vision-language pre-training with a Generator-and-Sampler Transformer.
-
Generative Agents for Multi-Agent Autoformalization of Interaction Scenarios
GAMA uses LLM agents to turn natural language game descriptions into validated executable logic programs, reaching about 77% semantic correctness on 110 scenarios from five 2x2 games.
-
Realistic Corner Case Generation for Autonomous Vehicles with Multimodal Large Language Model
AutoScenario translates multimodal real-world driving data into controllable and diverse corner-case scenarios for autonomous vehicle testing using LLMs and SUMO/CARLA simulations.
-
Explaining and Tuning Transformer-based LLMs in Arithmetic Tasks with Human Strategies
A small transformer learns addition, multiplication, and division by mastering simple digit subtasks first, and human teaching strategies lift its arithmetic accuracy to ~100%.
-
A Replicate-and-Quantize Strategy for Plug-and-Play Load Balancing of Sparse Mixture-of-Experts LLMs
Replicate-and-Quantize copies the busiest MoE expert as a quantized duplicate and compresses the least important expert, lowering a new Load-Imbalance Score by up to 1.4x while accuracy varies by roughly -1.2 to +3.0 points.
-
Can Structured Templates Facilitate LLMs in Tackling Harder Tasks? : An Exploration of Scaling Laws by Difficulty
Training on easy synthetic math data lowers accuracy on hard benchmarks, and the proposed SST framework, which teaches explicit procedural chains, aims to reverse that drop.
-
Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuning
CRFT selects critical internal representations via attention and saliency scores and fine-tunes only them, improving GSM8K accuracy over ReFT from 29.0% to 32.8% on LLaMA-2-7B.
-
Active Task Disambiguation with LLMs
Selecting clarifying questions by estimated information gain over sampled solutions outperforms implicit question generation for LLM task disambiguation.
-
CDW-CoT: Clustered Distance-Weighted Chain-of-Thoughts Reasoning
CDW-CoT groups a reasoning dataset into clusters, learns a prompt distribution per cluster, and interpolates these distributions by embedding distance for each new query, reporting higher exact-match accuracy than thr...
-
The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?
A DPO-based alignment method with a balanced mixture of legal and illegal chemistry prompts improves combined safety and utility scores, but its benchmark shares training compounds and its hyperparameters are tuned on...
-
Large Language Models for Interpretable Mental Health Diagnosis
An LLM plus expert review plus a Datalog engine can encode ICD-11 mood disorder criteria into runnable rules that agree with the authors' manual encoding on 30 synthetic patients.
-
GSM-Plus-BN: A Perturbation-Based Benchmark for Bangla Mathematical Reasoning in Large Language Models
The paper releases GSM-Plus-BN, a human-verified Bengali translation of the GSM-Plus perturbed math benchmark, and reports accuracy baselines for six open LLMs under standard and CoT prompting.
-
From Implicit Exploration to Structured Reasoning: Leveraging Guideline and Refinement for LLMs
A guideline-and-refinement prompting framework distills reasoning rules from past successes and failures, applies them stepwise with per-step correction, and reports gains over CoT, ReAct, ToT, Beats, and FoT on selec...
-
We Need Knowledge Distillation for Solving Math Word Problems
A 3-layer student transformer distilled from compressed BERT vectors retains roughly 90 percent of teacher performance on Math23K, according to the authors, though the teacher baseline is not shown.
-
A Large Language Model-Empowered Agent for Reliable and Robust Structural Analysis
An LLM agent that reframes beam analysis as OpenSeesPy code generation reaches over 99 percent reliability on a small benchmark, but chiefly because the prompt contains a near-identical solved example.
-
Generative to Agentic AI: Survey, Conceptualization, and Challenges
Agentic AI is characterized over Generative AI by iterative reasoning, environment interaction, memory, and tool use, with autonomy as the defining difference.
-
A Technical Survey of Reinforcement Learning Techniques for Large Language Models
A survey of RL methods for LLMs that organizes the field by reward modeling, feedback source, and optimization strategy, with benchmark tables favoring a scalar-regression UNA variant over DPO and KTO in offline alignment.
-
Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions
LLM robustness research is organized into adversarial robustness, out-of-distribution robustness, and evaluation, with an accompanying GitHub collection of papers.
-
Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches
This survey argues that embodiment, symbol grounding, causality, and memory are the foundational principles needed to make large language models achieve artificial general intelligence.
-
An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems
A survey and position paper that reviews LLM prompting, RAG, and RL techniques and argues they could support open-ended implementation generation, without presenting new results.
-
Knowledge Augmented Complex Problem Solving with Large Language Models: A Survey
A survey that organizes LLM-based complex problem solving into three components, multi-step reasoning, domain knowledge, and result verification, and reviews methods and challenges across four application domains.
Discussion (0). Continue with ORCID to comment.