REVIEW 7 cited by
Critique Ability of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Critical thinking is essential for rational decision-making and problem-solving. This skill hinges on the ability to provide precise and reasoned critiques and is a hallmark of human intelligence. In the era of large language models (LLMs), this study explores the ability of LLMs to deliver accurate critiques across various tasks. We are interested in this topic as a capable critic model could not only serve as a reliable evaluator, but also as a source of supervised signals for model tuning. Particularly, if a model can self-critique, it has the potential for autonomous self-improvement. To examine this, we introduce a unified evaluation framework for assessing the critique abilities of LLMs. We develop a benchmark called CriticBench, which comprises 3K high-quality natural language queries and corresponding model responses; and annotate the correctness of these responses. The benchmark cover tasks such as math problem-solving, code completion, and question answering. We evaluate multiple LLMs on the collected dataset and our analysis reveals several noteworthy insights: (1) Critique is generally challenging for most LLMs, and this capability often emerges only when models are sufficiently large. (2) In particular, self-critique is especially difficult. Even top-performing LLMs struggle to achieve satisfactory performance. (3) Models tend to have lower critique accuracy on problems where they are most uncertain. To this end, we introduce a simple yet effective baseline named self-check, which leverages self-critique to improve task performance for various models. We hope this study serves as an initial exploration into understanding the critique abilities of LLMs, and aims to inform future research, including the development of more proficient critic models and the application of critiques across diverse tasks.
Forward citations
Cited by 7 Pith papers
-
Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability
Large multimodal models mostly fail to proactively detect flawed textual premises, and their performance depends on error type and on how they weight text versus images.
-
DeepCritic: Deliberate Critique with Large Language Models
A two-stage SFT and RL pipeline turns a 7B instruct model into a deliberate math critic that outperforms GPT-4o and same-size R1-distill models at locating the first erroneous step.
-
A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations
A unified taxonomy and comparative meta-evaluation of automatic evaluation methods across text, vision, and speech generation, concluding that LLM-based evaluators dominate current practice.
-
Can LLMs and humans be friends? Uncovering factors affecting human-AI intimacy formation
Gradual self-disclosure raises perceived intimacy with an LLM chatbot, but the study's design cannot separate this effect from time or question order.
-
Step-Wise Formal Verification for LLM-Based Mathematical Problem Solving
MATH-VF formalizes LLM math solutions into SimpleMath and uses a tool-augmented critic to verify each reasoning step and offer corrective feedback.
-
Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents
Guided by a learned value critic, 1-step lookahead and trajectory selection lift an open-weights SWE agent to 40.8% on SWE-bench Verified, roughly doubling its success rate.
-
MC-NEST: Enhancing Mathematical Reasoning in Large Language Models leveraging a Monte Carlo Self-Refine Tree
MC-NEST adds a constant probability term to MCTSr's node selection and reports improved AIME pass@1 for GPT-4o, but the numbers are weakened by test-set rollout tuning and internal inconsistencies.
Discussion (0). Continue with ORCID to comment.