Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

This paper claims that replacing a single LLM's self-reflection with a structured debate among persona-diverse critics, aggregated by a judge, reduces the repeated-error failure mode known as degeneration of thought and improves reasoning a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:15 UTC pith:ROJBQUG7

load-bearing objection Plausible extension, but the HumanEval gain is confounded by unequal retry budgets and the HotPotQA gain is within noise; worth a revise, not an accept. the 4 major comments →

arxiv 2512.20845 v2 pith:ROJBQUG7 submitted 2025-12-23 cs.AI cs.MA

MAR:Multi-Agent Reflexion Improves Reasoning Abilities in LLMs

Onat Ozer , Yuchen Wang , Grace Wu , Daniel Dosti , Honghao Zhang , Vivi De La Rue This is my paper
classification cs.AI cs.MA
keywords multi-agent debateReflexionself-reflectiondegeneration of thoughtLLM reasoningHotPotQAHumanEvalpersona-based critique
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that single-agent self-reflection, as in the Reflexion framework, tends to repeat its own mistakes because the same model acts, evaluates, and critiques itself. To break this loop, the authors introduce Multi-Agent Reflexion (MAR), where a panel of persona-guided critics debates the failure and a judge synthesizes a consensus reflection into the actor's memory. On HotPotQA, MAR raises exact-match accuracy from 44% to 47%; on HumanEval, it raises pass@1 from 76.4% to 82.6%. If correct, this is a training-free path to more reliable LLM agents, at the cost of roughly three times the API calls.

Core claim

Multi-Agent Reflexion (MAR) consistently outperforms single-agent Reflexion on both multi-hop question answering and program synthesis. The central discovery is that the failure of self-correction comes from a lack of perspective diversity: the same model produces, evaluates, and revises its own reasoning, leading to confirmation bias and mode collapse. By explicitly separating acting, evaluating, critiquing, and aggregating, and by using critics with distinct reasoning personas (Verifier, Skeptic, Logician, Creative, and role-specific programmers), MAR generates higher-quality reflections that escape entrenched errors. Reported gains: HotPotQA exact match improves from 44% to 47%, and Human

What carries the argument

The MAR debate module replaces the single self-reflector. When the actor fails, multiple persona-based critics each write an initial diagnosis; then, for up to two rounds, they agree or disagree with each other's critiques; finally, a judge synthesizes the debate into a single 'Consensus Reflection' that is appended to the actor's memory. The machine's work is to inject structured disagreement into the reflection step, reducing shared blind spots and preventing the repeated reinforcement of earlier mistakes.

Load-bearing premise

The load-bearing premise is that the performance gain on HumanEval comes from the multi-agent debate, but the comparison is not trial-matched: Reflexion gets two attempts per problem while MAR gets three, and the paper never runs Reflexion with three attempts, so the extra retry alone could explain the 6.2-point improvement.

What would settle it

Run Reflexion with three attempts per HumanEval problem (the same budget as MAR) and compare pass@1. If Reflexion at three attempts reaches roughly 82.6%, the multi-agent debate is not the cause of the gain; if MAR still leads by a similar margin, the debate mechanism is confirmed. Similarly, re-score HotPotQA with a semantic-match metric: if the MAR gain disappears under semantic scoring, the reported EM gain may be an artifact of format-strict matching.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If MAR's gains hold, any Reflexion-style self-improvement loop can be upgraded by adding persona-diverse critics and a judge, without retraining the model.
  • The improvement on both QA and code generation suggests the mechanism is general across reasoning domains, not tied to one benchmark.
  • The paper's limitation notes indicate that the benefit of MAR is capped by the quality of the evaluation signal; noisy or format-strict metrics like exact match penalize semantically correct answers and can misdirect even multi-agent reflection.
  • The roughly 3x increase in API calls and latency means that efficiency—such as dynamically selecting personas or reducing debate rounds—is a necessary next step for practical deployment.
  • MAR reduces the stagnation observed in single-agent Reflexion, indicating that structured disagreement helps agents escape local minima in reasoning space.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The HumanEval comparison is not trial-matched: Reflexion is capped at two attempts per problem while MAR gets three, and the paper does not run Reflexion with three attempts, so part of the 6.2-point pass@1 gain could come from the extra retry rather than the debate.
  • A natural testable extension is a matched-budget comparison (give Reflexion the same three attempts) to isolate the debate's contribution; if the gap shrinks to near zero, the multi-agent element is not the driver.
  • Because the paper shows exact-match scoring rejects semantically identical answers, the HotPotQA gain of 3 points is likely an underestimate; re-evaluating both methods with a semantic or F1 metric could widen the demonstrated MAR advantage.
  • A routing policy that triggers the expensive debate module only on hard or repeated failures could preserve most of MAR's gains at a fraction of the cost, making the approach more scalable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper replicates the Reflexion framework on HotPotQA and HumanEval, identifies two failure modes in single-agent self-reflection (confirmation bias and mode collapse), and proposes Multi-Agent Reflexion (MAR), which replaces the single self-reflector with a panel of persona-based critics and a judge that synthesizes their debate into a consensus reflection. The authors report that MAR improves HotPotQA exact-match accuracy from 44% to 47% and HumanEval pass@1 from 76.4% to 82.6% relative to their Reflexion replication, all without parameter updates. The paper includes qualitative failure-case analyses, appendices with prompts and trajectories, and a code repository.

Significance. If the reported gains were cleanly established, the paper would make a useful, training-free contribution: separating acting, evaluation, and critiquing across diverse personas is a plausible mechanism for reducing self-reflection stagnation. The paper also ships a public codebase and detailed logging, which are assets for reproducibility. However, the central empirical claim is currently not supported. The HumanEval comparison is confounded by unequal retry budgets, and the HotPotQA gain is a 3-point difference on 100 questions with no error bars or significance testing. The manuscript also contains an incomplete placeholder in the Results section. The idea is interesting, but the evidence as submitted is not sufficient for publication in a serious journal.

major comments (4)
  1. [HumanEval Setup / Experimental Setup / Appendix D] The claimed HumanEval improvement (76.4 to 82.6 pass@1, Table 4) is confounded by unequal retry budgets. The methodology states that Reflexion receives "one initial attempt followed by a single reflection-guided retry" (max 2 trials), while Appendix D sets MAR's maximum at 3 trials per problem. Because the paper's pass@1 credits a problem if any generated solution passes, the extra retry alone mechanically inflates MAR's score. No trial-matched baseline is reported (e.g., Reflexion with 3 attempts or MAR with 2 attempts). The 6.2-point gain is load-bearing, and without a trial-matched comparison it cannot be attributed to multi-agent debate.
  2. [HotPotQA Results / Table 3] The HotPotQA evidence is statistically weak. Table 3 reports 44% EM for Reflexion and 47% for MAR on 100 curated questions. This is a 3-question difference; with no repeated runs, confidence intervals, or significance tests, the difference is within sampling noise (the approximate standard error of the difference is about 7 percentage points). The paper should report multiple seeds, a larger sample, or a formal test before claiming MAR outperforms Reflexion on HotPotQA.
  3. [Results] The Results section contains a leftover placeholder sentence: "We include the original scores for reference and leave placeholders for our HotPotQA numbers, which will be filled in after full evaluation." The submitted manuscript is therefore incomplete, and the numerical claims cannot be fully audited. This is not an acceptable state for a journal submission.
  4. [Performance Metrics / HumanEval Setup] The term pass@1 is used inconsistently. The metric definition says a model receives credit if its "single sampled solution" passes, but both Reflexion and MAR are allowed multiple attempts (2 and 3, respectively). Standard pass@1 does not permit retries; reporting "pass@1" for best-of-k attempts is misleading. The paper should either use pass@k semantics or explicitly define "pass@1 after retries," especially since the retry budget is the main confound in the HumanEval comparison.
minor comments (4)
  1. [Abstract / Table 4] The abstract states 82.7% on HumanEval, while Table 4 reports 82.6%. Please reconcile the numbers.
  2. [Throughout] Numerous typos and infelicities: "Through out extensive experimentation," "debators" instead of "debaters," "leads to better diversity of in the reflections," and "we designed personas in a systemic way." A careful language edit is needed.
  3. [Personas in Multi-Agent Debate / Appendix C] The Creative persona is described as "low exploit, high exploration, high exploration," presumably a typo for "high strictness" or similar. Please correct.
  4. [HotPotQA Setup] The paper refers to a "curated selection of 100 difficult HotPotQA questions" but does not describe the curation criterion or how it relates to the original Reflexion dataset. This is important for reproducibility and for interpreting the comparison to the original paper's numbers.

Circularity Check

0 steps flagged

No significant circularity: MAR's results are empirical measurements on standard benchmarks; the HumanEval trial-budget mismatch is a confound, not a definitional circularity.

full rationale

The paper does not contain a derivation in which an output is shown to equal an input by construction, nor does it fit a parameter and then relabel that fit as a prediction. Its central claims are direct empirical measurements: HotPotQA EM (32.0 baseline, 44.0 Reflexion, 47.0 MAR) and HumanEval pass@1 (67.1 baseline, 76.4 Reflexion, 82.6 MAR). The MAR method is a new multi-agent reflection procedure compared against re-run Reflexion baselines on standard external benchmarks, so the evaluation is self-contained in the sense required for a non-circular empirical claim. The persona sets, debate rounds, and trial budgets are experimental design choices, and some were made after observing failures on the same benchmarks; that is tuning or adaptation, not definitional circularity. The most serious validity concern is that the HumanEval comparison is not trial-matched: the Methodology says Reflexion receives 'one initial attempt followed by a single reflection-guided retry' (two trials), while Appendix D specifies MAR's 'Maximum trials per problem: 3'. Since pass@1 credits a problem if any attempt passes, the extra retry could explain part of the 6.2-point gap. This is an experimental confound requiring a trial-matched baseline, but it is not circular because the measured scores are not equivalent to the inputs by construction. No load-bearing self-citation or imported uniqueness theorem appears. Therefore, the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 1 invented entities

The paper contributes a prompt-level architecture, not a mathematical derivation. Its quantitative claim depends on hand-chosen personas, debate rounds, and trial budgets, and on assumptions that the 100-question subset and single runs are representative.

free parameters (4)
  • Debate rounds cap = 2
    Chosen 'based on empirical logs and prior MAD work' (Experimental Setup); no ablation shows that 2 is optimal or that round count isn't driving gains.
  • Number of debaters = 4 for HotPotQA, 3 for HumanEval
    Hand-selected per dataset (Appendix D); not varied in experiments.
  • Trial budgets = 5 HotPotQA, 3 HumanEval (vs 2 for Reflexion HumanEval)
    Chosen for cost; HumanEval budget is not matched to Reflexion, confounding the comparison.
  • Persona selection = Verifier/Skeptic/Logician/Creative (HotPotQA), Senior/QA/Algorithm/Code Reviewer (HumanEval)
    Personas 'engineered to disagree' and based on observed failure modes on the same benchmarks; no held-out validation.
axioms (4)
  • domain assumption The 100-question HotPotQA subset is large enough to distinguish methods
    47 vs 44 is a 3-question difference on n=100; no confidence intervals are provided.
  • domain assumption Single runs with GPT-3.5 Turbo are representative
    No seeds, repetitions, or variance reported for any table.
  • domain assumption The original Reflexion protocol was faithfully reproduced
    The paper's comparison relies on their Reflexion numbers being correct; they cite prompt alignment with original codebase but no code hash.
  • domain assumption Persona diversity, not extra tokens/compute, causes observed gains
    MAR uses ~3x API calls; no ablation separates persona diversity from additional critique length or retries.
invented entities (1)
  • Persona critics (Verifier, Skeptic, Logician, Creative, etc.) no independent evidence
    purpose: Generate diverse diagnoses of the Actor's failed reasoning
    No ablation shows these specific roles outperform homogeneous critics or a single longer critique; their effect is conflated with compute and budget differences.

pith-pipeline@v1.3.0-alltime-deepseek · 11689 in / 12512 out tokens · 108608 ms · 2026-08-03T14:15:29.071370+00:00 · methodology

0 comments
read the original abstract

LLMs have shown the capacity to improve their performance on reasoning tasks through reflecting on their mistakes, and acting with these reflections in mind. However, continual reflections of the same LLM onto itself exhibit degeneration of thought, where the LLM continues to repeat the same errors again and again even with the knowledge that its wrong. To address this problem, we instead introduce multi-agent with multi-persona debators as the method to generate reflections. Through out extensive experimentation, we've found that the leads to better diversity of in the reflections generated by the llm agent. We demonstrate an accuracy of 47% EM HotPot QA (question answering) and 82.7% on HumanEval (programming), both performances surpassing reflection with a single llm.

Figures

Figures reproduced from arXiv: 2512.20845 by Daniel Dosti, Grace Wu, Honghao Zhang, Onat Ozer, Vivi De La Rue, Yuchen Wang.

Figure 1
Figure 1. Figure 1: The Reflexion Architecture. Actor performs an action on the environment. The evaluator gives feedback to reflector LLM, with feedback stored in short-term memory. A new action is then performed starting a new iteration. Self-Reflector (Msr). When the Evaluator indicates fail￾ure, the Self-Reflection model converts this outcome into verbal feedback. Given the failed trajectory and its evalu￾ation, Msr produ… view at source ↗
Figure 2
Figure 2. Figure 2: The Multi-Agent Reflexion (MAR) Architec￾ture. This high-level diagram illustrates the extension of the single-agent Reflexion framework. Our extension replaces the single self-reflecting model in Reflexion with a group of LLM agents that each serve as distinct critics. When the Actor produces an incorrect an￾swer, the system does not rely on a single reflection. Instead, it initiates a structured multi-ag… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of HotPotQA performance across trials for ReAct, Reflexion, and Multi-Agent Reflexion (MAR). The baseline gpt-3.5-Turbo (grey) and the Reflex￾ion replication results (blue) are shown. MAR (red) offers the highest EM improvement. while MAR further increases performance to 82.6%. The pattern reflects how each method handles errors: the baseline often repeats the same mistake, Reflexion corrects it… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliability

    cs.LG 2026-05 unverdicted novelty 6.0

    LLM reliability techniques are unified as communication channel operators, with a new cost-aware router achieving superior quality-cost tradeoffs on hard tasks.

  2. TEC: A Collection of Human Trial-and-error Trajectories for Problem Solving

    cs.CL 2026-04 unverdicted novelty 6.0

    TEC is a new public dataset of detailed human trial-and-error trajectories and reflections on web tasks, with humans showing substantially higher accuracy than LLMs.

  3. Security Considerations for Multi-agent Systems

    cs.CR 2026-03 unverdicted novelty 6.0

    No existing AI security framework covers a majority of the 193 identified multi-agent system threats in any category, with OWASP Agentic Security Initiative achieving the highest overall coverage at 65.3%.

Reference graph

Works this paper leans on

15 extracted references · 9 linked inside Pith · cited by 3 Pith papers

  1. [5]

    arXiv preprint arXiv:2303.17760

    CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society. arXiv preprint arXiv:2303.17760. Liang, T.; He, Z.; Jiao, W.; Wang, X.; Wang, Y .; Wang, R.; Yang, Y .; Shi, S.; and Tu, Z

    Pith/arXiv arXiv

  2. [6]

    Liu, Y .; Cao, J.; Li, Z.; He, R.; and Tan, T

    Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate.arXiv preprint arXiv:2305.19118. Liu, Y .; Cao, J.; Li, Z.; He, R.; and Tan, T

    Pith/arXiv arXiv

  3. [8]

    Shinn, N.; Cassano, F.; Berman, E.; Gopinath, A.; Narasimhan, K.; and Yao, S

    Self-Refine: Iterative Refinement with Self- Feedback.arXiv preprint arXiv:2303.17651. Shinn, N.; Cassano, F.; Berman, E.; Gopinath, A.; Narasimhan, K.; and Yao, S

    Pith/arXiv arXiv

  4. [9]

    Reflexion: Language Agents with Verbal Reinforcement Learning.arXiv preprint arXiv:2303.11366. Smit, A. P.; Grinsztajn, N.; Duckworth, P.; Barrett, T. D.; and Pretorius, A

    Pith/arXiv arXiv

  5. [10]

    InProceed- ings of the 41st International Conference on Machine Learn- ing (ICML 2024), volume 235 ofProceedings of Machine Learning Research, 45883–45905

    Should we be going MAD? A Look at Multi-Agent Debate Strategies for LLMs. InProceed- ings of the 41st International Conference on Machine Learn- ing (ICML 2024), volume 235 ofProceedings of Machine Learning Research, 45883–45905. PMLR. Stechly, K.; Marquez, M.; and Kambhampati, S

    2024

  6. [11]

    InFoundations Models for Decision Making Workshop at the Thirty-Seventh Con- ference on Neural Information Processing Systems (NeurIPS 2023)

    GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems. InFoundations Models for Decision Making Workshop at the Thirty-Seventh Con- ference on Neural Information Processing Systems (NeurIPS 2023). New Orleans, LA: NeurIPS Workshop. Sutton, R. S.; and Barto, A. G. 2018.Reinforcement Learn- ing: An Introduction. Adaptive C...

    2023

  7. [12]

    New Orleans, LA: NeurIPS Workshop

    Can Large Language Models Really Improve by Self- critiquing Their Own Plans? InFoundations Models for De- cision Making Workshop at the Thirty-Seventh Conference on Neural Information Processing Systems (NeurIPS 2023). New Orleans, LA: NeurIPS Workshop. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E. H.; Le, Q.; and Zhou, D

    2023

  8. [14]

    Zhang, J.; Yu, S.; Chong, D.; Sicilia, A.; Tomz, M

    ReAct: Synergizing Reason- ing and Acting in Language Models.arXiv preprint arXiv:2210.03629. Zhang, J.; Yu, S.; Chong, D.; Sicilia, A.; Tomz, M. R.; Manning, C. D.; and Shi, W

    Pith/arXiv arXiv

  9. [15]

    double the difference of the elements in the list

    Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diver- sity. arXiv:arXiv:2510.01171. Individual Contributions Onat Ozer:Onat led the replication of HotPotQA, includ- ing prompt alignment, agent scaffolding, and implementa- tion of the CoT and ReAct baselines from the original Re- flexion codebase. He designed and executed detailed loggin...

    Pith/arXiv arXiv

  10. [2020]

    InAdvances in Neural Infor- mation Processing Systems 33 (NeurIPS 2020), 1877–1901

    Language Models are Few-Shot Learners. InAdvances in Neural Infor- mation Processing Systems 33 (NeurIPS 2020), 1877–1901. Vancouver, Canada: Curran Associates, Inc. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y .; Joseph, N.; Brock- man, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G...

    2020

  11. [2021]

    Chen, X.; Song, Z.; Ji, D.; Gao, S.; and Zhu, L

    Evaluating Large Language Models Trained on Code.arXiv preprint arXiv:2107.03374. Chen, X.; Song, Z.; Ji, D.; Gao, S.; and Zhu, L

    Pith/arXiv arXiv

  12. [2022]

    Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y .; Cohen, W.; Salakhut- dinov, R.; and Manning, C

    Chain-of- Thought Prompting Elicits Reasoning in Large Language Models.arXiv preprint arXiv:2201.11903. Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y .; Cohen, W.; Salakhut- dinov, R.; and Manning, C. D

    Pith/arXiv arXiv

  13. [2023]

    Huang, J.; Chen, X.; Mishra, S.; Zheng, H

    Improving Factuality and Reasoning in Lan- guage Models through Multiagent Debate.arXiv preprint arXiv:2305.14325. Huang, J.; Chen, X.; Mishra, S.; Zheng, H. S.; Yu, A. W.; Song, X.; and Zhou, D

    Pith/arXiv arXiv

  14. [2024]

    InProceedings of the Twelfth International Conference on Learning Representa- tions (ICLR 2024)

    Large Language Models Cannot Self-Correct Reasoning Yet. InProceedings of the Twelfth International Conference on Learning Representa- tions (ICLR 2024). ICLR. Kim, G.; Baldi, P.; and McAleer, S

    2024

  15. [2025]

    InProceedings of the International Confer- ence on Learning Representations (ICLR 2025)

    Breaking Mental Set to Improve Reasoning through Diverse Multi- Agent Debate. InProceedings of the International Confer- ence on Learning Representations (ICLR 2025). Singapore: ICLR. Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y .; et al

    2025