REVIEW 3 major objections 3 minor 1 cited by
Can Language Models Critique Themselves? Investigating Self-Feedback for Retrieval Augmented Generation at BioASQ 2025
T0 review · 3 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Self-feedback in LLMs helps some BioASQ tasks, hurts others.
desk verdict The abstract is too thin to know whether the self-feedback result is real, but the research question is legitimate and worth a peer-review look. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the self-feedback loop: a single LLM acts as generator, evaluator, and refiner of its own output. The loop is applied at two stages—query expansion and final answer generation—and across four answer formats (yes/no, factoid, list, ideal). The mechanism carries the comparison because all models receive the same loop, so any difference in outcome is attributed to the model and the task type.
What would settle it
Run one model on a fixed set of BioASQ questions under two conditions: the real self-feedback loop, and a control loop where the critique step is replaced by a generic instruction of the same length (e.g., 'now revise your answer'). If the control produces the same distribution of score changes, the critique content is not driving the effect. Alternatively, compare self-feedback against human expert revisions; if self-feedback never beats the human baseline, the claim that models can critique themselves into better biomedical answers would fail.
Extended reading notes
Core claim
On its own terms, the paper reports a comparative study of agentic retrieval augmented generation with a self-feedback mechanism. Each model—Gemini-Flash 2.0, o3-mini, o4-mini, and DeepSeek-R1—generated an initial output, evaluated that output, and then refined it, both for expanding queries and for producing four answer types in the BioASQ CLEF 2025 biomedical question-answering challenge. The discovery is negative in shape: preliminary results show varied performance, meaning the same self-feedback loop improves some model-task combinations and degrades others. The paper frames this as evidence that self-correction cannot be assumed to help, and that reasoning models do not consistently ge
Load-bearing premise
The load-bearing premise is that a model's critique of its own output carries a meaningful correction signal; if the critique mostly echoes the model's initial biases or adds prompt noise, the varied performance says little about self-correction.
Editorial extensions
If this is right
- Self-feedback should not be treated as a free improvement in RAG pipelines; deployments need task-level evaluation before enabling it.
- Reasoning-oriented LLMs should not be assumed to give better self-critique; feedback quality has to be checked separately from answer quality.
- For professional search, expert-formulated benchmarks like BioASQ can expose where automated refinement conflicts with expert information needs.
- If self-feedback is applied, its effect on transparency and user involvement should be reported, not just its effect on accuracy.
Reading between the lines
- If the preliminary pattern holds, part of the 'varied performance' could be an artifact of self-agreement bias: a model tends to rate its own first answer highly, so the refinement step may simply confirm the initial output unless the critique is strong enough to override it.
- A stronger test would compare self-feedback with cross-model feedback—one model critiquing another's answer—and with human expert edits. If cross-model feedback outperforms self-feedback, the results would point to critique quality rather than self-correction as the active ingredient.
- In a professional search setting, the paper's implied design could be extended into an interactive loop where model self-feedback is offered as an optional revision rather than applied automatically, preserving user expertise and transparency.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates self-feedback in agentic retrieval-augmented generation for biomedical question answering, using the BioASQ CLEF 2025 benchmark. The authors compare four LLMs (Gemini-Flash 2.0, o3-mini, o4-mini, DeepSeek-R1) with and without a self-feedback loop in which the model generates an output, evaluates it, and refines it, for query expansion and four answer types. The abstract reports 'varied performance' of self-feedback across models and tasks and concludes with insights into LLM self-correction and the need to compare LLM-generated feedback with human expert input.
Significance. If substantiated, the claim that self-feedback has inconsistent or no universal benefit across modern reasoning and non-reasoning models on a professional biomedical search task is valuable: it would temper expectations for a widely used technique and support the case for human-in-the-loop validation. The multi-model, multi-answer-type design is well matched to the BioASQ setting and the study addresses a timely question. However, at the abstract level the evidence is qualitative only: there are no effect sizes, significance tests, error bars, dataset splits, or explicit comparison against a no-feedback baseline. The paper's contribution cannot be assessed from the provided text, and the central empirical claim currently rests on an underspecified result sentence.
major comments (3)
- [Abstract, results sentence] The sentence 'Preliminary results indicate varied performance for the self-feedback strategy across models and tasks' is ambiguous: it could mean that absolute performance varies across model/task combinations, which would be trivially true and would not measure the effect of self-feedback. The central claim requires a per-model, per-answer-type contrast between a self-feedback run and a matched no-feedback run. Please report the deltas, the direction of variation, and the number of test questions per condition. Without this control, the results sentence does not support the stated investigation of whether iterative self-correction improves performance.
- [Abstract, methodology sentence] The self-feedback loop uses the same model for generation, evaluation, and refinement. Because the critic and generator share the same biases and training objective, observed changes may reflect the model's tendency to agree or disagree with its own initial output rather than a general property of critique. To interpret 'varied performance' as evidence about self-correction, the paper should include a baseline with an independent critic (e.g., a different LLM or human feedback) or at least report the content of the critiques and the rate at which the model accepted or rejected its own suggestions. Without such a contrast, the mechanism behind the variation is unidentified.
- [Abstract, overall] The abstract reports no quantitative results: no accuracy, F1, or other BioASQ score is given for any model or condition. Even for a short paper, a compact results table or effect-size summary is needed to support the empirical claim. As written, the reader cannot determine whether the 'varied performance' is statistically or practically meaningful, nor whether the paper's conclusions follow from the data.
minor comments (3)
- [Abstract, first sentence] 'Agentic Retrieval Augmented Generation (RAG) and deep research systems' is a somewhat loose characterization; consider specifying the system types and citing representative works to frame the novelty of self-feedback.
- [Abstract, motivation] The claim that automated systems 'may reduce user involvement and misalign with expert information needs' would benefit from a concrete reference or definitional clarification, since the paper does not appear to measure user involvement directly.
- [Abstract, final sentence] The phrase 'informs future work' is vague; please indicate what specific design or measurement insight the preliminary results provide, even at a qualitative level.
Circularity Check
No circularity found: abstract-only empirical study with no derivation chain to reduce.
full rationale
The paper is an abstract-only report of an empirical study. There is no claimed derivation from first principles, no equations, no self-citations, no uniqueness theorems, and no prediction that is fitted from the data it purports to predict. The central intervention—self-feedback, where an LLM generates, evaluates, and refines its own outputs—is an experimental condition, not a logical construction that defines the outcome in terms of itself. Whether self-feedback improves performance is assessed externally via BioASQ gold answers (standard for the challenge). The abstract does not specify the evaluation metric, but nothing in the text reduces the claimed 'varied performance' to the definition of the intervention. The skeptic's concern that a no-feedback baseline may be missing is a correctness/interpretability threat, not circularity. No passage asserts a circular step, missing support, or self-referential proof. Therefore no significant circularity is present.
Assumptions & free parameters
assumptions (2)
- domain assumption BioASQ expert-formulated questions and gold answers are a valid external benchmark for professional search quality.
- domain assumption The self-feedback loop, in which the same model generates and critiques, produces a feedback signal that can be meaningfully separated from the original generation.
Cite this review
Pith. "Pith review of Can Language Models Critique Themselves? Investigating Self-Feedback for Retrieval Augmented Generation at BioASQ 2025." pith.science (2026). https://pith.science/paper/LDAI6YJO
@misc{pith2026250805366,
author = {Pith},
title = {Pith review of: Can Language Models Critique Themselves? Investigating Self-Feedback for Retrieval Augmented Generation at BioASQ 2025},
year = {2026},
howpublished = {\url{https://pith.science/paper/LDAI6YJO}},
note = {Machine review of arXiv:2508.05366}
}
read the original abstract
Agentic Retrieval Augmented Generation (RAG) and 'deep research' systems aim to enable autonomous search processes where Large Language Models (LLMs) iteratively refine outputs. However, applying these systems to domain-specific professional search, such as biomedical research, presents challenges, as automated systems may reduce user involvement and misalign with expert information needs. Professional search tasks often demand high levels of user expertise and transparency. The BioASQ CLEF 2025 challenge, using expert-formulated questions, can serve as a platform to study these issues. We explored the performance of current reasoning and nonreasoning LLMs like Gemini-Flash 2.0, o3-mini, o4-mini and DeepSeek-R1. A key aspect of our methodology was a self-feedback mechanism where LLMs generated, evaluated, and then refined their outputs for query expansion and for multiple answer types (yes/no, factoid, list, ideal). We investigated whether this iterative self-correction improves performance and if reasoning models are more capable of generating useful feedback. Preliminary results indicate varied performance for the self-feedback strategy across models and tasks. This work offers insights into LLM self-correction and informs future work on comparing the effectiveness of LLM-generated feedback with direct human expert input in these search systems.
Forward citations
Cited by 1 Pith paper
-
Retrieval Augmented Biomedical Question Answering with Weak Question Recovery and Neural Reranking for BioASQ Task 14b
A BioASQ 14b pipeline with weak-question recovery and MiniLM reranking improves MAP@10 by about 28% on Batch 4, but without an isolated component ablation.
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.