Recognition: 2 theorem links
· Lean TheoremSelf-critiquing models for assisting human evaluators
Pith reviewed 2026-05-16 20:22 UTC · model grok-4.3
The pith
Fine-tuned language models can generate critiques that help humans identify flaws in summaries they would otherwise overlook.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
We fine-tune large language models to write natural language critiques using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and on most tasks are better at self-critiquing, despite having harder-to-critique outputs. Larger models can also integrate their own self-c
What carries the argument
Behavioral cloning from human critique examples to fine-tune large language models for producing natural language critiques that assist human evaluators on summarization tasks.
If this is right
- Larger models produce more helpful critiques than smaller ones across the studied tasks.
- Larger models are better at self-critiquing even though their generated summaries become harder to critique.
- Models can use their own critiques as feedback to produce improved summaries.
- Measurements indicate that even large models possess relevant knowledge they do not articulate in critique form.
Where Pith is reading between the lines
- The same fine-tuning approach could be applied to other domains such as code review or scientific paper assessment where direct human judgment is costly.
- If critique quality continues to improve with scale, recursive loops of model self-critique and refinement may become feasible for iterative improvement.
- Human evaluators might gain lasting skill improvements from repeated exposure to model-generated critiques on the same task.
- The gap between what models can generate and what they can critique suggests a need for new training objectives that align articulation with internal knowledge.
Load-bearing premise
Training on human critique examples produces critiques that generalize to new summaries without introducing systematic biases or missing flaw types that humans would notice unaided.
What would settle it
An experiment in which humans detect fewer or the same number of summary flaws when given model critiques compared to evaluating the summaries alone, or where the critiques systematically overlook certain flaw categories that unaided humans reliably catch.
read the original abstract
We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and on most tasks, are better at self-critiquing, despite having harder-to-critique outputs. Larger models can also integrate their own self-critiques as feedback, refining their own summaries into better ones. Finally, we motivate and introduce a framework for comparing critiquing ability to generation and discrimination ability. Our measurements suggest that even large models may still have relevant knowledge they cannot or do not articulate as critiques. These results are a proof of concept for using AI-assisted human feedback to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly. We release our training datasets, as well as samples from our critique assistance experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper fine-tunes large language models via behavioral cloning to generate natural-language critiques of summaries. On topic-based summarization, these critiques improve human detection of both naturally occurring flaws (in model- and human-written summaries) and intentionally planted flaws. The work reports scaling trends (larger models produce more helpful critiques and better self-critiques), shows that models can use their own critiques for self-refinement, and introduces a framework comparing critiquing ability to generation and discrimination. Datasets and samples are released as a proof of concept for AI-assisted human feedback on hard-to-evaluate tasks.
Significance. If the results hold after addressing controls, the work supplies a concrete, scalable mechanism for augmenting human oversight of generative models on tasks where direct evaluation is difficult. The empirical scaling observations, self-refinement results, and public release of training data and critique samples constitute tangible strengths that could inform subsequent work on AI-assisted supervision.
major comments (1)
- [Human evaluation protocol] Human evaluation protocol (experiments on topic-based summarization and planted-flaw summaries): the central claim that model critiques cause humans to detect additional real flaws is not isolated from the confound of increased evaluation time or attention. No time-matched or placebo baseline (e.g., a neutral paragraph of equal length) is reported, so measured gains could arise from demand characteristics or extra scrutiny rather than critique content produced by behavioral cloning.
minor comments (2)
- [Framework section] The framework comparing critiquing to generation and discrimination is introduced but would benefit from explicit operational definitions and a table summarizing the three capabilities side-by-side.
- [Scaling and self-refinement experiments] Scaling plots and self-refinement results would be clearer with error bars or confidence intervals and explicit statement of the number of human evaluators per condition.
Simulated Author's Rebuttal
We thank the referee for their positive summary of the work and for identifying a key methodological concern in the human evaluation protocol. We address this point directly below and will revise the manuscript accordingly.
read point-by-point responses
-
Referee: [Human evaluation protocol] Human evaluation protocol (experiments on topic-based summarization and planted-flaw summaries): the central claim that model critiques cause humans to detect additional real flaws is not isolated from the confound of increased evaluation time or attention. No time-matched or placebo baseline (e.g., a neutral paragraph of equal length) is reported, so measured gains could arise from demand characteristics or extra scrutiny rather than critique content produced by behavioral cloning.
Authors: We agree that this is a valid concern. Our current experiments compare the critique condition against a no-critique baseline in which evaluators assess summaries without any additional text, but this does not control for the effects of extra reading time, attention, or demand characteristics induced by any supplementary paragraph. To isolate the contribution of the critique content itself, we will add a placebo control condition in the revised manuscript: evaluators will read a neutral paragraph of matched length (e.g., a generic topic description unrelated to the summary) before performing the flaw-detection task. We will report the results of this additional control alongside the existing conditions. revision: yes
Circularity Check
No circularity: empirical evaluation from external human data
full rationale
The paper is an empirical ML study that fine-tunes models on human-written critiques via behavioral cloning and measures whether the resulting critiques help humans detect additional flaws in summaries. No mathematical derivations, equations, or first-principles results are present that could reduce to the inputs by construction. All reported outcomes (scaling trends, self-critique integration, comparison framework) are measured on held-out summaries and human raters, independent of any fitted-parameter renaming or self-citation load-bearing for uniqueness. The work therefore contains no steps matching the enumerated circularity patterns.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption Behavioral cloning from human critique demonstrations produces useful critique behavior on held-out summaries
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
We fine-tune large language models to write natural language critiques using behavioral cloning... Larger models write more helpful critiques...
-
IndisputableMonolith/Foundation/ArithmeticFromLogic.leanLogicNat recovery theorems unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
We motivate and introduce a framework for comparing critiquing ability to generation and discrimination ability (GDC gaps).
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Forward citations
Cited by 17 Pith papers
-
Self-Correction as Feedback Control: Error Dynamics, Stability Thresholds, and Prompt Interventions in LLMs
Self-correction in LLMs is stable and non-degrading only when ECR/EIR exceeds initial accuracy over (1-accuracy), with EIR below 0.5% cleanly separating helpful from harmful cases across models.
-
Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF
R-CAI inverts constitutional AI to automatically generate diverse toxic data for LLM red teaming, with probability clamping improving output coherence by 15% while preserving adversarial strength.
-
Self-Rewarding Language Models
Iterative self-rewarding via LLM-as-Judge in DPO training on Llama 2 70B improves instruction following and self-evaluation, outperforming GPT-4 on AlpacaEval 2.0.
-
DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
DeltaRubric decomposes multimodal preference evaluation into self-generated planning and verification steps within a single model, producing large accuracy improvements on VL-RewardBench via multi-role reinforcement learning.
-
Building a Precise Video Language with Human-AI Oversight
CHAI framework pairs AI pre-captions with expert human critiques to produce precise video descriptions, enabling open models to outperform closed ones like Gemini-3.1-Pro and improve fine-grained control in video gene...
-
No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning
ECHO jointly optimizes policy and critic via co-evolution, cascaded rollouts, and saturation-aware shaping to deliver non-stale feedback and higher success in open-world LLM agent RL.
-
Towards Understanding Sycophancy in Language Models
Sycophancy is prevalent in state-of-the-art AI assistants and is likely driven in part by human preferences that favor agreement over truthfulness.
-
Cognitive Architectures for Language Agents
CoALA is a modular cognitive architecture for language agents that organizes memory components, action spaces for internal and external interaction, and a generalized decision-making loop to support more systematic de...
-
Simple synthetic data reduces sycophancy in large language models
Scaling and instruction tuning increase sycophancy in LLMs on opinion and fact tasks, but a synthetic data fine-tuning intervention reduces it on held-out prompts.
-
Teaching Large Language Models to Self-Debug
Self-Debugging teaches LLMs to identify and fix their own code errors through rubber-duck-style natural language explanations and execution feedback, delivering 2-12% gains over baselines on Spider, TransCoder, and MBPP.
-
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society
CAMEL proposes a role-playing framework with inception prompting that enables autonomous multi-agent cooperation among LLMs and generates conversational data for studying their behaviors.
-
Language Models (Mostly) Know What They Know
Language models show good calibration when asked to estimate the probability that their own answers are correct, with performance improving as models get larger.
-
Measure Twice, Click Once: Co-evolving Proposer and Visual Critic via Reinforcement Learning for GUI Grounding
A co-evolving proposer-critic RL framework improves GUI grounding accuracy by letting the model critique its own proposals rendered on screenshots.
-
SVSR: A Self-Verification and Self-Rectification Paradigm for Multimodal Reasoning
SVSR trains multimodal models to verify and correct their own reasoning using a preference dataset, supervised fine-tuning, and semi-online DPO with a teacher model.
-
FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
FAITH improves LLM factual accuracy by mapping confidence and semantic entropy into natural-language knowledge-state quadrants for trustworthiness and honestness, then applying PPO with a combined reward and retrieval...
-
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
The paper surveys hallucination in LLMs with an innovative taxonomy, factors, detection methods, benchmarks, mitigation strategies, and open research directions.
-
Self-Refine: Iterative Refinement with Self-Feedback
Self-Refine boosts LLM outputs by ~20% on average across seven tasks by having the same model iteratively generate, critique, and refine its own responses.
Reference graph
Works this paper leans on
-
[1]
We asked for labelers to aim to have answers with different kinds of flaws, e.g
We sometimes also collected misleading answers that should be clearly wrong, but take readers a long time to determine as wrong. We asked for labelers to aim to have answers with different kinds of flaws, e.g. accuracy flaws contradicting part of the text that are hard to find or not stated explicitly and coverage flaws leaving out important details that are ...
-
[2]
We sometimes asked for lists of "quote corroborations". For each quote corroboration, the labeler highlighted a set of spans in the answer, and a set of corroborating spans in the passage A.2.2 Critique task When collecting data for the critique task, we show labelers a passage, multiple questions about the passage, and multiple model-generated answers fo...
-
[3]
Collecting a set of "corroborations" for each answer, of natural language explanations that support the answer
-
[4]
For topic-based summarization, we asked for a category for each critique, one of: • Coverage: summary missing relevant information from passage • Accuracy: summary giving incorrect information • Coherence: summary is poorly written, confusing or nonsensical • Other: a catch-all bucket for everything else
-
[5]
For topic-based summarization, we also explored collecting quotes. For each critique, we asked them to give "passage highlights", required for Coverage critiques, and "answer highlights", required for Accuracy and Coherence critiques. The answer highlights would be spans in either the original answer or a refinement being critiqued. A.2.3 Helpfulness task ...
-
[6]
We sometimes asked labelers to mark when critiques were "clearly helpful", meaning they were unambiguously helpful rather than nit-picky
-
[7]
We sometimes asked labelers to mark severity and category of all model-generated critiques marked as helpful. 26 A.3 Base tasks Early in the project, we asked labelers to create question-answering and summarization tasks. How- ever, we later switched to topic-based summarization and used that for the majority of the project. As noted, our results are repo...
-
[8]
Question creation Our labelers were asked to write 1-8 questions based on a passage and give demonstrations of answers to those questions (topic-based summarization or otherwise) at the same time. During model training, we include the auxiliary task of creating a slate of questions given a passage
-
[9]
Corroborations We explored collecting corroborations of answers, which explain why aspects of an answer are correct. In general, it is most interesting to critique things that are explanation-like, as opposed to short answers with no explanation (e.g. a mathematical proof rather than just a statement). With topic-based summarization, this was less importa...
-
[10]
We also include a variant which conditions on the span of the answer being corroborated
Corroboration quotes We include the task of retrieving relevant quotes from the passage which corroborate an answer. We also include a variant which conditions on the span of the answer being corroborated
-
[11]
At training time, we sometimes convert between various tasks as a form of data augmentation
Question quotes We include the task of retrieving relevant quotes from the passage, based only on the question. At training time, we sometimes convert between various tasks as a form of data augmentation. The most important one is that we convert conditional refinement tasks to direct refinement tasks 50% of the time. We also convert corroboration quotes to...
-
[12]
explaining to humans everything it knows,
The definition of ΣP 2 corresponds precisely to 2-step recursive reward modeling (RRM): we give the verifier access to a model trained with RLHP (analogous to anNP/co-NP oracle). In general, n-step recursive reward modeling corresponds to the nth level of the polynomial hierarchy, just like n-step debate. We can interpret the assistance needed to move up th...
-
[13]
V ocalizing critiques helps a model understand how to discriminate, as a "chain of thought" [WWS+22b, WWS+22a]
-
[14]
More generally, if we do not control for compute. For example, we could search for critiques (see Section D) Recall that in Section 5 we found a negative CD gap for the Addition, Alphabetize, and RACE synthetic tasks. We suspect this is due to C’s usage of the oracle and that we would haveCm < D but do not investigate further in this work. C.3.2 Alternati...
-
[15]
Essentially, we show the human a slate of critiques (like we did in Section 3.4)
C h8: Choose 8 critiques to show to the human, and pick the answer according to validity and severity of the most critical critique. Essentially, we show the human a slate of critiques (like we did in Section 3.4)
-
[16]
This is just like C, but with a better critique
C 8: Choose a single best-of-8 (according to helpfulness score) critique to show to the human, who uses validity and severity in order to judge the answer. This is just like C, but with a better critique
-
[17]
helpfulness score of best-of-N critiques
C m8: Choose a single best-of-8 (according to helpfulness score) critique, and simply use helpfulness score. This cuts out the human from the loop entirely but relies on helpfulness being comparable across different answers (ideally we would use a severity model). Ch8 corresponds to giving humans a chance to review multiple pieces of assistance, similar t...
-
[18]
The best-of-N helpfulness score discriminates better with increasing N 33 Figure 14: Discrimination between misleading and honest answers. Best-of-N according to a helpfulness model appears to asymptote to worse than a direct critiqueability model. This suggests using helpfulness labels directly as a training signal for debate might not be preferred to tr...
-
[19]
The best-of-N helpfulness score never discriminates as well as the critiqueability score
-
[20]
This suggests naive 2-step debate wouldn’t work as well as RLHP for this task
The best-of-N helpfulness score appears to be asymptoting with N→∞ to worse than the critiqueability score. This suggests naive 2-step debate wouldn’t work as well as RLHP for this task. While these are negative results, they don’t imply that debate won’t start working with larger models. Some other important caveats:
-
[21]
a critique-conditional preference-based reward model) would help
Our helpfulness model may primarily prioritize nit-picky critiques rather than severe ones, so we expect training a severity model (or e.g. a critique-conditional preference-based reward model) would help
-
[22]
Furthermore, we are primarily interested in testing in domains where giving critiqueability labels is even more difficult. (a) With human labels, it seems interesting if critiqueability labels come from humans being misled in systematically biased ways. In our case, labels’ performance may simply be poor due to noise from occasional carelessness. This does...
work page 1955
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.