Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small language models can learn explicit self-reflection from their own self-generated reflection data and, on BIG-bench, rival open models several times their size.

desk verdict Useful dataset and systematic study, but the BIG-bench headline is compromised by a missing held-out split; send to review but require the split be fixed or the claim re-scoped. read the letter →

arxiv 2505.16475 v1 pith:TLBSRFGX submitted 2025-05-22 cs.AI

Jiaqi Li , Xinyi Dong , Yang Liu , Zhizhuo Yang , Quansen Wang , Xiaobo Wang , SongChun Zhu , Zixia Jia
show 1 more author
Zilong Zheng
This is my paper · ORCID
classification cs.AI
keywords self-reflectionsmalllanguagemodelsself-trainingreflectionlearningdirectpreferenceoptimizationsupervisedfine-tuningBIG-benchmeta-introspection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that small language models can learn effective self-reflection from their own self-generated reflection data, without reasoning traces distilled from a larger model. The ReflectEvo pipeline has the same small model generate an initial answer, receive only a binary correct/incorrect signal, write a reflection that locates errors and plans a fix, and then produce a corrected answer; the reflections themselves are self-generated, with a stronger model used only in one training variant to rank reflections. From 17 source datasets the authors assemble ReflectEvo-460k, a 460k-sample corpus of these reflections, and train small models with SFT and DPO. They report that this reflection learning lifts Llama-3 from 52.4% to 71.2% and Mistral from 44.4% to 71.1% on BIG-bench, rivaling or surpassing open models several times their size. A sympathetic reader would care because the result suggests that an explicit meta-introspection step can be learned and that small models can bootstrap their own reasoning.

What carries the argument

The load-bearing object is the ReflectEvo-460k dataset: tuples of question, initial solution, binary feedback, reflection, and corrected solution, generated by using the same small model as both generator and reflector, with reject sampling and a pool of 32 reflection instructions. The carrying mechanism is reflection learning, which trains the reflector either to produce reflection and correction jointly (one-stage SFT), separately (two-stage SFT), or to prefer reflections that lead to correct answers or that a stronger model selects (DPO on D± and Dpref). The reflection instruction pool is what gives the data its generality: it forces the model to verify the failed solution, locate and diagnose errors across a fixed taxonomy, and outline correction plans.

What would settle it

Re-train Llama-3 with the identical recipe but exclude every BIG-bench-derived question from ReflectEvo-460k and evaluate on the full BIG-bench test set: if accuracy stays near 71.2%, the gain is genuine learned reflection, while a drop toward the untuned 52.4% would indicate the reported gain depends on training/evaluation overlap. A complementary check is to remove the oracle correct/incorrect trigger at inference and use the model's own self-judgment, measuring how much of the gain survives.

Watch

Extended reading notes

Core claim

The central claim is that a small language model can be trained to perform meta introspection: given its own failed solution and nothing but a binary correct/incorrect signal, it can generate a reflection that localizes the error, diagnoses why it happened, and plans a correction, and this learned reflection then raises its reasoning accuracy. Concretely, the paper reports BIG-bench accuracy rising from 52.4% to 71.2% for Llama-3-8B and from 44.4% to 71.1% for Mistral-7B with one reflection turn, matching or beating models up to eight times larger on the same benchmark. The training data are produced by the model itself in a two-turn rollout, filtered to reflections that led to a correct second-turn answer, and supplemented with pairwise preference data; the reflection objective is trained with one- or two-stage SFT and with DPO on either correctness-labeled or teacher-preferred reflection pairs. The paper also argues that the reflections are high quality: error types identified in the reflections align with human-annotated categories, and the semantic correlation between a reflection and the corrected thought tracks task accuracy after training.

Load-bearing premise

The headline results assume that the BIG-bench questions used for evaluation were held out from the examples used to build ReflectEvo-460k, and that binary correct/incorrect feedback is available at inference to decide when to trigger reflection.

Editorial extensions

If this is right

  • Small models can approach or exceed the BIG-bench accuracy of open models several times their size after reflection learning.
  • The learned reflection skill transfers across tasks and across generators: a reflector trained on one model's reflections also corrects answers produced by different models.
  • Multi-turn reflection continues to improve accuracy beyond one turn, with BIG-bench exceeding 80% after six turns.
  • Reflection data generated by one small model can be reused to train another, making the dataset a reusable community resource.
  • Learned reflection outperforms both prompt-based reflection and SFT on direct question-answer pairs for small models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not make: ablate the reflection text itself by training on generic reflections paired with the same corrected answers; if the gain persists, the improvement comes from learning to produce a corrected second pass rather than from the diagnostic content.
  • The oracle correct/incorrect trigger at inference is the main deployment obstacle; the paper's own self-judgment experiment suggests replacing it with a learned verifier, and the few-point drop on LogiQA is an upper bound on what a verifier must recover.
  • Compressing reflection into a single pass, by training on (question, failed solution) to output the corrected answer directly, would remove the extra inference turn while retaining the reflection-guided training signal; the paper does not test this.
  • The small gains on coding and mathematics suggest the missing ingredient is fine-grained step-level critique rather than whole-solution reflection; generating reflections at the granularity of individual reasoning steps is a direct next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ReflectEvo, a self-training pipeline in which a small language model (SLM) generates its own reflections on failed attempts, filters them using ground-truth correctness, and then is fine-tuned with SFT and DPO on the resulting dataset (ReflectEvo-460k) to improve its self-reflection and self-correction abilities. The authors report large gains on LogiQA, MATH, MBPP, and especially BIG-bench, where Llama-3-8B is claimed to rise from 52.4% to 71.2% and Mistral-7B from 44.4% to 71.1%, surpassing larger open-source models. They also provide analyses of error types, correlation between reflection quality and correction success, multi-turn reflection scaling, and generalization across tasks and models.

Significance. If the central claims hold, this would be a meaningful contribution: it would show that small models can bootstrap their own reflection skills without distillation from a stronger model, and it provides a large, reusable dataset. The paper also contains useful systematic comparisons across four training settings and across three base models. However, the headline BIG-bench result is not yet credible because the paper does not establish that the BIG-bench test set is disjoint from the training pool, and because the main inference protocol appears to rely on an oracle ground-truth verifier to decide when to reflect. These issues undermine the central claim that small models rival or surpass much larger models on BIG-bench.

major comments (3)
  1. [Appendix B.1 / Table 10] The paper never specifies how the 500 BIG-bench test examples in Table 10 were selected or whether they are disjoint from the BIG-bench QA pairs used to construct ReflectEvo-460k. Appendix B.1 says that for each BIG-bench source dataset with more than 1000 samples the authors randomly select 1000 QA pairs, and no held-out split is described. If the test set overlaps with the training pool, the BIG-bench gains in Table 2 (e.g., Llama-3 from 38.2% Acc@t1 to 71.2% Acc@t2, Mistral from 36.6% to 71.1%) may be substantially due to memorization, invalidating the comparison against larger models. The authors must state the exact sampling protocol, demonstrate disjointness (e.g., by question ID or hash), and ideally report results on a provably held-out set.
  2. [Sec. 3.2 / Table 7] The main inference protocol uses an oracle ground-truth verifier to decide when to trigger reflection and to select the final answer. Table 7 contrasts 'Oracle Groundtruth' and 'Self-judgement' verifiers on LogiQA, implying that the Table 2 results use the oracle verifier. Since BIG-bench QA tasks do not provide an external correctness signal at deployment, the headline claim that ReflectEvo 'can rival or even surpass' larger models is conditional on access to the reference answers during inference. The authors should report the main tables also with the self-judgement verifier, or clearly qualify the claim to state that the reported numbers assume an oracle correctness signal.
  3. [Sec. 2.3 / Eq. (2)] The curation step in Eq. (2) keeps only reflection–correction pairs where the revised answer equals the ground-truth answer a*. This means that for every retained training example, the model is supervised with the exact correct final answer. While this is a legitimate self-training design, it is a strong form of supervision that should be highlighted in the abstract and conclusions. The statement that the method works 'without distillation from superior models or fine-grained human annotation' is technically true for the D+ setting, but the filtering itself uses benchmark ground truths, which is an important caveat for readers evaluating the claim of fully autonomous self-improvement.
minor comments (4)
  1. [Sec. 2.1] The subsection title contains a typo: 'Prelinminary' should be 'Preliminary'.
  2. [Figure 5] The labels in Figure 5, 'FALSETOTRUE' and 'FALSETOFALSE', lack spaces and appear garbled; they should read 'False to True' and 'False to False'.
  3. [Table 2] Table 2 is difficult to parse because the Acc@t1 values are repeated on multiple rows and the structure of the prompt-based and self-training blocks is not visually clear. A cleaner layout that separates Acc@t1, Acc@t2, and Delta(t1,t2) per method would improve readability.
  4. [Sec. 4.1] The sentence 'It shows significant improvements on Llama-3, exceeding the original base model by 10% on average tasks and outperforming its strongest counterpart with model size ×8' is imprecise; it should specify which counterpart, which evaluation condition, and whether the comparison uses the same inference-time verifier.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the self-training loop is anchored by external correctness labels, and the BIG-bench split ambiguity is an evaluation-validity concern rather than a demonstrated circularity.

full rationale

The claimed derivation chain is not circular at the method level. ReflectEvo-460k is generated by the base SLM itself, but the D+ filter in Eq. (2) requires the revised answer to equal the external ground truth a*, so the supervision signal is not the model's own output; the SFT and DPO objectives in Eqs. (5)-(10) are standard conditional likelihood and preference losses over this externally verified data. No load-bearing claim is justified by self-citation: the same authors' earlier works appear only in related-work or motivation contexts, and no uniqueness theorem or prior ansatz is imported from the authors' own papers to force the present choice of model or objective. The only substantive concern is the BIG-bench evaluation: Appendix B.1 says that for datasets with more than 1000 samples the authors randomly select 1000 QA pairs into ReflectEvo-460k, and Table 10 lists 500 BIG-bench test examples, but the paper does not explicitly state that those 500 are excluded from the training pool. If they are not disjoint, the BIG-bench comparison would be contaminated by memorization. However, the paper's text does not prove overlap, and the hard rule here requires exhibiting a specific reduction rather than a possible leakage. I therefore do not score this as circularity, though it is a legitimate evaluation-validity ambiguity that the authors should clarify.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the availability of oracle feedback, the reliability of GPT-4o as a preference judge, and the disjointness of the BIG-bench training and evaluation sets. The most fragile assumption is the data split, because the paper never specifies a held-out test set for BIG-bench, making the headline results potentially circular.

free parameters (4)
  • Rejection sampling count k = 2
    The number of sampled reflections per question is set to 2 without a sensitivity analysis, affecting the size and diversity of the training set.
  • Number of reflection instructions m = 5 or 6 out of 32
    The paper randomly selects 5 or 6 instructions per dataset, which changes the data distribution and likely affects final performance.
  • LoRA rank r = 8
    LoRA rank is chosen for the one-stage SFT setting; no ablation is reported.
  • DPO beta = 0.01
    The DPO temperature-like parameter is set to 0.01 for both Dpref and D±; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption Binary correctness feedback (correct/incorrect) is available from ground truth during training and inference.
    The pipeline depends on knowing whether the first-turn answer is correct to trigger reflection; this is stated in Section 2.2 and used in the main evaluation.
  • domain assumption GPT-4o can reliably select better reflections for preference data (Dpref).
    The paper uses GPT-4o to choose preferred reflections, assuming its judgments are valid without validating against human preferences.
  • ad hoc to paper The BIG-bench evaluation set is disjoint from the training instances sampled from the same datasets.
    Appendix B.1 samples up to 1000 questions per BIG-bench dataset for training but never describes a held-out split, so the disjointness is assumed implicitly.
  • domain assumption Self-training on filtered self-generated reflections improves reasoning beyond simply training on direct answers.
    This is the central hypothesis of the paper; it is supported by the experiments only if the evaluation is valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection." pith.science (2026). https://pith.science/paper/TLBSRFGX

@misc{pith2026250516475,
  author       = {Pith},
  title        = {Pith review of: ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TLBSRFGX}},
  note         = {Machine review of arXiv:2505.16475}
}
read the original abstract

We present a novel pipeline, ReflectEvo, to demonstrate that small language models (SLMs) can enhance meta introspection through reflection learning. This process iteratively generates self-reflection for self-training, fostering a continuous and self-evolving process. Leveraging this pipeline, we construct ReflectEvo-460k, a large-scale, comprehensive, self-generated reflection dataset with broadened instructions and diverse multi-domain tasks. Building upon this dataset, we demonstrate the effectiveness of reflection learning to improve SLMs' reasoning abilities using SFT and DPO with remarkable performance, substantially boosting Llama-3 from 52.4% to 71.2% and Mistral from 44.4% to 71.1%. It validates that ReflectEvo can rival or even surpass the reasoning capability of the three prominent open-sourced models on BIG-bench without distillation from superior models or fine-grained human annotation. We further conduct a deeper analysis of the high quality of self-generated reflections and their impact on error localization and correction. Our work highlights the potential of continuously enhancing the reasoning performance of SLMs through iterative reflection learning in the long run.

Figures

Figures reproduced from arXiv: 2505.16475 by the authors.

Figure 1
Figure 1. Overview pipeline of ReflectEvo. termediate step-by-step process supervision rather than directly mapping an initial solution to a re￾vised solution. Building on this pipeline, we curate a large￾scale, diverse, and unsupervised reflection learn￾ing dataset ReflectEvo-460k containing 460k re￾flection samples derived from 17 source datasets spanning 10 tasks and domains. We explore the diversity of reflection instruct… view at source ↗
Figure 2
Figure 2. (a) Task-dataset hierarchy distribution of ReflectEvo-460k. (b) Error type distribution of corrected thoughts [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance training with ReflectEvo across [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Performance in multi-turn self-reflection on [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Qualitative examples from the MATH. “False to True” and a “False to False” stand for successful and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Task performance (Acc@t2) versus the corre [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Task performance (Acc@t2) versus the correlation between reflection and second-turn thoughts for [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Correlation of reflection between each pair of tasks. We obtain the semantic representation for all reflections [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Per-bias selection of a cross-family LLM auditor lifts biased-judgment accuracy from 0.805/0.824 baselines to 0.884.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Calculation Error • 1-2

    Mathematical Errors • 1-1. Calculation Error • 1-2. Algorithm Error

    work page

  2. [2]

    Flawed Rationale Error • 2-2

    Logic and Reasoning Errors • 2-1. Flawed Rationale Error • 2-2. Internal Inconsistency

    work page

  3. [3]

    Advances in Neu- ral Information Processing Systems, 36

    Direct preference optimization: Your language model is secretly a reward model. Advances in Neu- ral Information Processing Systems, 36. Matthew Renze and Erhan Guven. 2024. Self-reflection in llm agents: Effects on problem-solving perfor- mance. arXiv preprint arXiv:2405.06682. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu...

    arXiv 2024

  4. [4]

    Factual Errors

    Factual Errors • 4-1. Factual Errors

    work page

  5. [5]

    In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8228–8238

    Diffusion model alignment using direct prefer- ence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8228–8238. Tianduo Wang, Shichen Li, and Wei Lu. 2024a. Self- training with direct preference optimization improves chain-of-thought reasoning. In Proceedings of the 62nd Annual Meeting of the Asso...

    arXiv 2022

  6. [6]

    arXiv preprint arXiv:2308.02151

    Retroformer: Retrospective large language agents with policy gradient optimization. arXiv preprint arXiv:2308.02151. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston

    arXiv

  7. [7]

    Student A

    Self-rewarding language models. arXiv preprint arXiv:2401.10020. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825. Eric Zelikman, Georges Raif Harik, Yijia Shao, Varuna Jayasir...

    arXiv 2023

  8. [10]

    Context Misinterpretation • 3-2

    Instruction Violation • 3-1. Context Misinterpretation • 3-2. Incomplete or Irrelevant Response • 3-3. Format Discrepancy

    work page

Show all 12 references
  1. [12]

    No Errors • 5-1. No Errors Detected # Input - Question: {question} - Thought: {thought} - Reflection: {reflection} # Output - Labels: [Error Type(s) Assigned] - Rationale: [Explanation for label assignment, with specific examples]

  2. [2022]

    Advances in neural in- formation processing systems, 35:27730–27744

    Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong- Hui Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. 2023. Language model self-improve...

    2023 arXiv

  3. [2023]

    arXiv preprint arXiv: 2305.14825

    Large language models are in-context seman- tic reasoners rather than symbolic reasoners. arXiv preprint arXiv: 2305.14825. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, ...

    2024 arXiv

  4. [2024]

    URL https://arxiv

    Training language models to self-correct via reinforcement learning, 2024. URL https://arxiv. org/abs/2409.12917. J Richard Landis and Gary G Koch. 1977. The mea- surement of observer agreement for categorical data. biometrics, pages 159–174. Chankyu Lee, Rajarshi Roy, Mengyao...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.