When to Trust Context: Self-Reflective Debates for Context Reliability

Amin Saberi; Cheng Meixin; Fang Wu; Haokai Zhao; Shayan Talaei; Tinson Xu; Yejin Choi; Zeqi Zhou

REVIEW 5 major objections 5 minor 33 references

When to Trust Context: Self-Reflective Debates for Context Reliability

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An asymmetric debate among a defender, a context-blind critic, and a judge, gated by token-level confidence, tells an LLM when retrieved context may override memory—and when to answer from memory or abstain.

desk verdict The asymmetric-debate idea is worth a look, but the central 'consistently outperforms' claim is not supported by the reported numbers. read the letter →

arxiv 2506.06020 v1 pith:3GJ36RWJ submitted 2025-06-06 cs.CL cs.AI

Zeqi Zhou , Fang Wu , Shayan Talaei , Haokai Zhao , Cheng Meixin , Tinson Xu , Amin Saberi , Yejin Choi This is my paper

classification cs.CLcs.AI

keywords knowledgeconflictcontextreliabilitymulti-agentdebateself-confidenceretrieval-augmentedgenerationhallucinationquestionansweringClashEval

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models face a fork every time a provided passage disagrees with what they already know: follow the passage and risk parroting a mistake, or follow memory and risk ignoring valid retrieved evidence. This paper argues that the fork can be resolved by splitting one question into two — how confident is the model in its own answer, and is the passage actually trustworthy — and proposes Self-Reflective Debate for Contextual Reliability (SR-DCR) to answer both. A defender argues for the context-based answer, a critic who has never seen the passage argues from prior knowledge, and a judge issues a reliability verdict; a token-level confidence gate then picks the context answer, the prior answer, or abstention. On the ClashEval benchmark across five models and five perturbation strengths, SR-DCR improves accuracy on misleading passages while holding clean-context accuracy near the few-shot baseline, at times matching or exceeding a golden baseline that only ever trusts correct context. The payoff, if the claim holds, is a low-overhead switch that retrieval-augmented systems can use to trust external evidence only when it survives adversarial scrutiny.

What carries the argument

The central object is the Asymmetric Context Verification Debate (ACVD), a three-role exchange in which the defender sees the question and passage and defends the context-derived answer, the critic sees only the question and argues from prior knowledge, and the judge reads the transcript over up to six rounds and returns REASONABLE or UNREASONABLE at the earliest round where the verdict stops changing. The debate is asymmetric on purpose: the context's claims must survive challenge from a participant who cannot see them. The companion mechanism is a self-confidence score — the average log-probability of the answer tokens in a zero-context prediction, collapsed to HIGH or LOW at a 0.90 threshold — combined with a gating rule that emits the context answer on a REASONABLE verdict, the prior answer on an UNREASONABLE verdict with HIGH confidence, and abstention otherwise. Together the two signals separate two facts the model would otherwise conflate: how strongly it believes its prior answer, and whether the external passage deserves belief.

What would settle it

Run SR-DCR with defender passages that are fabricated but internally coherent — a wrong year that sits within a plausible range, not an absurd one — and compare the judge's REASONABLE rate against the ground-truth passage label. If plausibly wrong passages are frequently tagged REASONABLE, the debate is measuring persuasiveness rather than truth, and the reported robustness gains would instead be attributable to the confidence gate alone.

Watch

Extended reading notes

Core claim

The paper claims that the trustworthiness of retrieved context and the model's confidence in its own prior answer are separable signals, and that a judge overseeing an asymmetric debate extracts the former better than either signal alone. In ACVD, a defender who sees the question and passage argues for the context-derived answer, a critic who sees only the question argues from prior knowledge, and a judge reads the transcript over up to six rounds and issues a REASONABLE or UNREASONABLE verdict, taken at the earliest round where the verdict stabilizes. A separate self-confidence score — the average token log-probability of the zero-context answer, thresholded at 0.90 — routes the final answer: trust the context when the verdict is REASONABLE, fall back to the prior answer when the verdict is UNREASONABLE and confidence is HIGH, otherwise abstain. On ClashEval across GPT-3.5 Turbo, GPT-4o, Claude Haiku 3.5, Claude Sonnet 3.7, and Llama 3.3-70B, the authors report that SR-DCR beats confidence-only and debate-only baselines on perturbed contexts while preserving clean-context accuracy, with gains of +7.7 points in robustness to corrupted context and +5.6 points in clean-context recovery.

Load-bearing premise

The load-bearing premise is that the judge can correctly label a passage REASONABLE or UNREASONABLE from a debate in which the critic has never seen the passage, so a persuasive defender arguing for a false but plausible passage could flip the verdict — and with it the final answer — back to the misleading context.

Editorial extensions

If this is right

Retrieval-augmented pipelines can bolt SR-DCR's rule onto any LLM without retraining: emit the context answer only under a REASONABLE verdict, fall back to a high-confidence prior otherwise, and abstain when neither branch applies.
The paper's graded-perturbation analysis shows that Unknown and Weakly Known items are precisely where models keep agreeing with misleading context, which is the population of cases the confidence-gated abstain-and-fallback branch is designed to cover.
Because judges in symmetric debate drift toward trusting context as rounds increase, ACVD's earliest-stabilized verdict removes the round-count tuning that classical multi-agent debate inherits.
The overhead is one extra forward pass for the confidence score plus a debate the context check requires anyway, so the robustness gain does not depend on fine-tuning a dedicated judge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A decisive experiment the paper does not run: give the defender a fabricated but internally coherent passage and count how often the judge calls it REASONABLE; a high rate would mean the debate tracks persuasion rather than truth, and would isolate the confidence gate as the true source of the gain.
The fixed 0.90 threshold invites calibration per domain; treating abstention as a tunable error-cost dial would let SR-DCR set different trust levels for, say, medical versus trivia questions.
The defender–blind-critic–judge–gate pattern generalizes beyond QA: the same structure could arbitrate whether a retrieved document is stale, whether a source snippet supports a claim, or whether a RAG chunk is on-topic, with the retriever's ranking playing the 'prior' role.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Zeqi and colleagues propose SR-DCR, a way to decide when to trust retrieved context over the model's parametric priors. The new piece is the asymmetric debate: the defender argues from the passage, the critic sees only the question, and the judge watches the transcript. That is a genuine twist on symmetric MAD, and the graded perturbation analysis across knowledge tiers is a useful addition. The framework is lightweight, and the gains on GPT-3.5 and Haiku are real.

The central claim, though, overreaches. The abstract promises a comparison against confidence-only baselines, and none is implemented. The baseline list has Few-shot, Self-Ask, RCI, Judge Debate, Naive Debate, and Golden, but no method that routes purely on confidence. That comparison is untested. 'Consistently enhances robustness' is contradicted by the tables: Sonnet's overall accuracy is 69.76 versus Naive Debate's 72.67, and GPT-4o's perturbed accuracy is 54.51 versus Judge Debate R1's 57.33, with similar losses at Offsets 20-100. Losing the overall comparison on one model and the perturbed comparison on another does not constitute 'consistent' improvement.

The confidence threshold tau=0.90 is chosen on the same ClashEval data (Fig 3), and the debate-stabilization rule is built from the Sec 4.3 observation that longer debates shift toward context. That is tuning on the test set, which weakens external validity.

A further soft spot: the paper does not state whether the judge receives the original passage. Sec 3.1 says the judge observes the debate transcript, which contains the defender's description. If the judge never sees the passage, a persuasive but false defender can mislead it, and the entire routing decision depends on that verdict. The authors should clarify and ideally test both settings.

What the paper does well: the context-preference analysis in Fig 2 is clean and informative, the self-consistency proxy for closed models is a reasonable workaround, and the limitations appendix is honest about fixed thresholds and deterministic judge assumptions.

Bottom line: the framework is worth discussing, but the empirical case needs a proper confidence-only baseline, a corrected 'consistently' claim, and out-of-fold threshold selection. This is a good idea in need of revision, not a finished result. I would send it to peer review with clear instructions to the authors to fix the missing baseline and tone down the claim.

Referee Report

5 major / 5 minor

Summary. The paper introduces SR-DCR (Self-Reflective Debate for Contextual Reliability), a framework that combines token-level self-confidence with an asymmetric multi-agent debate to decide whether to trust a retrieved passage over a model's parametric prior. In the proposed ACVD, a context-grounded defender argues against a context-deprived critic, and a judge issues a verdict that is combined with a confidence threshold to select between the context-based answer, the prior answer, or abstention. The authors report experiments on the ClashEval benchmark across five LLMs and claim that SR-DCR 'consistently enhances robustness to misleading context while maintaining accuracy on trustworthy inputs, outperforming both classical debate and confidence-only baselines.' The paper also contributes an analysis of context-preference rates under graded perturbations and of the relationship between self-confidence and knowledge recall.

Significance. If the headline claims were supported, SR-DCR would be a practical, low-overhead mechanism for arbitrating between parametric and contextual knowledge. The core ideas—informational asymmetry in debate and the explicit gating of context use by confidence—are sensible and potentially useful for RAG systems. The paper's empirical analysis of how context reliance varies with perturbation strength and knowledge category (Sections 4.1–4.2) is a strength, as is the explicit release of code. However, the central comparative claims are not supported by the reported evaluation: no confidence-only baseline is implemented, and the results in Tables 1 and 2 contradict the word 'consistently' on several model/perturbation combinations. The framework itself is plausible, but the evidence presented does not yet establish its advertised advantage.

major comments (5)

[Abstract and Section 4 (Baselines)] The abstract and Section 4.4 claim SR-DCR 'outperforms both classical debate and confidence-only baselines,' but no confidence-only baseline is defined or evaluated anywhere in Section 4. The baseline list (Few-shot, Self-Ask, RCI, Judge Debate, Naive Debate, Golden Baseline) contains no method that routes on confidence alone (e.g., answer prior if Conf >= tau, otherwise context). The Golden Baseline is an oracle, not a confidence-based router. Please add a proper confidence-only baseline and report it in Tables 1 and 2, or explicitly temper the claim.
[Table 1 and Section 4.4] The claim that SR-DCR 'consistently enhances robustness ... across models and perturbation settings' is contradicted by the reported numbers. On Claude Sonnet 3.7, SR-DCR's overall accuracy (69.76%) is below Naive Debate (72.67%). On GPT-4o perturbed contexts, SR-DCR (54.51%) is below Judge Debate Round 1 (57.33%). On Llama 3.3-70B perturbed contexts, SR-DCR (28.39%) is below both Judge Debate Round 1 (35.66%) and the Golden Baseline (33.33%). The paper should either provide statistical significance tests or revise the 'consistently' claim to reflect the model-dependent and offset-dependent pattern actually observed.
[Section 3.2 and Figure 3] The confidence threshold tau = 0.90 appears to be selected using the same ClashEval data shown in Figure 3, and the stabilization rule for the debate verdict (Section 3.1) is motivated by the post-hoc observation in Section 4.3 that longer debates shift judgment toward context. The manuscript does not state whether these hyperparameters were tuned on a separate calibration split or locked before evaluation. If tau and the stabilization rule were selected on the same instances used for the final accuracy numbers, the reported gains are optimistic. Please specify the calibration protocol and, if necessary, re-evaluate on a held-out set.
[Section 3.1 (ACVD)] The judge never receives the context passage directly; it only observes the defender's arguments and the critic's challenges. Since the final answer routing depends entirely on the verdict, the paper should validate that the judge's verdict tracks true context reliability (e.g., agreement with ground-truth labels, or human-validated verdicts). Without such a sanity check, a persuasive but incorrect defender could lead the judge to accept false context, and the reported accuracy would depend on a mechanism whose correctness is unverified. Reporting verdict-level accuracy against the known perturbation status would address this concern directly.
[Section 3.2 (ABSTAIN) and Section 4] The decision rule includes an ABSTAIN outcome, but the evaluation section never specifies how abstentions are treated in the exact-match metric or how often they occur. If abstentions are scored as incorrect, that is a legitimate choice; if they are excluded, the comparison with baselines is not apples-to-apples. Please report the abstention rate per model and the scoring rule used, and clarify whether the numbers in Tables 1 and 2 include abstentions.

minor comments (5)

[Appendix C.2] The reference 'extending the analysis shown in Fig. 6' in the first paragraph of C.2 is circular; it should refer to Fig. 3.
[Section 4.4] The listed gains ('+7.7-point improvement in robustness, +5.6-point recovery in accuracy, 2x reduction in prior bias') are not tied to a specific baseline or table; please state the reference method for each number.
[Section 4 (Golden Baseline)] The Golden Baseline for perturbed contexts assumes the model relies entirely on prior knowledge, but priors can be wrong for Weakly-Known and Unknown items (as the paper's own taxonomy shows). Calling it a 'theoretical upper bound' is therefore misleading; it is an oracle only with respect to knowing whether the context is perturbed, not with respect to factual accuracy.
[Section A.3 and Appendix C.1] For closed-source models, self-consistency is approximated with 16 samples, but the paper does not specify how the tau=0.90 threshold is applied to a 16-sample proportion (which can only take values in increments of 1/16). Please define the discretization or use a sample size that allows the threshold to be matched exactly.
[References] The RCI baseline cites Kim et al. (2023), which is a paper about computer-task solving; the authors should either cite the RCI adaptation for QA or clarify how the original method was repurposed.

Circularity Check

1 steps flagged · score 6.0 of 10

The confidence gate is calibrated on the same ClashEval distribution used for the headline evaluation, so the reported robustness is partly in-sample; the abstract's confidence-only comparison is never actually run.

fitted input called prediction [Section 4.2 (threshold calibration), applied in Section 3.2 decision rule, evaluated in Section 4.4 and Tables 1-2]
"Fig. 3 and 7 show that across all models, this conditional probability exceeds 0.88 once tau >= 0.90 and surpasses 0.95 at tau >= 0.95, confirming that high-confidence predictions are strong indicators of prior knowledge. In contrast, confidence below 0.70 yields poor overlap with Highly-Known pairs. These findings validate our key hypothesis: when Conf(q, a) crosses a threshold (e.g., 0.90), the model is highly likely to 'know' the answer. We use this insight to guide dynamic routing in SR-DCR."

The routing threshold tau = 0.90 is selected by inspecting the conditional-confidence curve computed on ClashEval instances (Fig. 3), and the same ClashEval benchmark is then used to report SR-DCR's claimed robustness gains (Tables 1-2). The decision rule in Section 3.2 routes the final answer through this threshold ('Conf(q, a_PRIOR) >= tau'), so the perturbed-context accuracy partly reflects the in-sample effect of a hyperparameter calibrated to maximize overlap with the Highly-Known category on exactly the questions being scored. No held-out validation split for tau is described; the paper adopts the official dev/test splits but does not state that the threshold calibration and the final evaluation use different instances.

full rationale

The judge-debate component of SR-DCR is not circular: the ACVD verdict is produced by zero-shot LLM agents with no access to ground-truth labels, and the final exact-match accuracy is a genuine outcome of running the pipeline. The central circularity is the confidence gate. The paper first establishes on ClashEval that tau = 0.90 separates Highly-Known from Unknown instances, then 'uses this insight' to set the routing threshold, and then evaluates the complete SR-DCR system on the same benchmark. Because the routing decision in Section 3.2 depends directly on this threshold, the reported robustness is partly an in-sample report of the fitted design rather than an out-of-sample prediction. Separately, the abstract claims SR-DCR 'outperforms both classical debate and confidence-only baselines,' but no confidence-only baseline appears in the baseline list of Section 4 or in Tables 1-2; that is an evidentiary gap in the comparative claim rather than a circular reduction. The internal contradiction with the reported tables (e.g., Sonnet 69.76 vs Naive Debate 72.67, GPT-4o perturbed 54.51 vs Judge Debate R1 57.33) is a correctness problem, not a circularity. Overall, the method has independent content in the debate mechanism, so the score is moderate rather than extreme.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new entities. The load-bearing assumptions are about the reliability of the judge's verdict and the validity of the confidence threshold, which is fitted to the benchmark. The free parameters are the confidence threshold, debate rounds, stabilization rule, and self-consistency sample count.

free parameters (4)

Confidence threshold tau = 0.90
Used in the final decision rule to gate between prior answer and abstain. Chosen from the same benchmark (Fig. 3) without a separate validation set.
Debate rounds R = 6
Number of rounds in ACVD; not justified and not swept.
Stabilization criterion = earliest round where V(r) remains constant to R
Design choice for extracting the judge verdict; no analysis of its effect.
Self-consistency sample count = 16
Used to approximate confidence for closed-source models; the threshold tau=0.90 is applied to this proxy without recalibration.

assumptions (4)

domain assumption The judge can accurately assess context reliability from debate transcripts
The entire ACVD rests on the judge's verdict V* being correct. Invoked in Section 3.1; no evidence is given that the judge's verdict correlates with true context reliability beyond the final QA accuracy.
domain assumption A context-deprived critic provides a valid adversarial test
The design assumes that an agent without access to c can meaningfully challenge the context's reliability by arguing from priors. Section 3.1.
domain assumption Exact match on ClashEval is a meaningful measure of factual correctness
Used throughout; no human agreement or alternative metrics reported.
domain assumption The 600-question subset is representative
The subset construction (subsampling and stratification) is not described in detail, so the reader cannot assess selection bias.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When to Trust Context: Self-Reflective Debates for Context Reliability." pith.science (2026). https://pith.science/paper/3GJ36RWJ

@misc{pith2026250606020,
  author       = {Pith},
  title        = {Pith review of: When to Trust Context: Self-Reflective Debates for Context Reliability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3GJ36RWJ}},
  note         = {Machine review of arXiv:2506.06020}
}

read the original abstract

Large language models frequently encounter conflicts between their parametric knowledge and contextual input, often resulting in factual inconsistencies or hallucinations. We propose Self-Reflective Debate for Contextual Reliability (SR-DCR), a lightweight framework that integrates token-level self-confidence with an asymmetric multi-agent debate to adjudicate such conflicts. A critic, deprived of context, challenges a defender who argues from the given passage; a judge model evaluates the debate and determines the context's reliability. The final answer is selected by combining the verdict with model confidence. Experiments on the ClashEval benchmark demonstrate that SR-DCR consistently enhances robustness to misleading context while maintaining accuracy on trustworthy inputs, outperforming both classical debate and confidence-only baselines with minimal computational overhead. The code is available at https://github.com/smiles724/Self-Reflective-Debates.

Figures

Figures reproduced from arXiv: 2506.06020 by the authors.

**Figure 1.** Overview of SR-DCR (Self-Reflective Debate for Contextual Reliability), a debate-driven framework for resolving conflicts between parametric priors and contextual evidence. When the model’s prior belief conflicts with the external passage (left), SR-DCR evaluates two key signals: (a) the self-confidence of the zero-context answer aˆPRIOR; and (b) the context’s reasonableness, determined via an asymmetric debate betw… view at source ↗

**Figure 2.** Context preference rates of the few-shot base [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The distribution of self-confidence scores of [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Prior Knowledge preference rate in judge [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Two illustrative examples of how questions are classified based on the model’s familiarity [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Distribution of self-consistency scores (estimated self-confidence score based on sampling) on our dataset, [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Distribution of self-confidence scores on more models [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

**Figure 8.** Figure 8: Context preference rates of a 4-shot baseline across offset magnitudes and knowledge categories of six [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

**Figure 9.** Figure 9: Prior Knowledge preference rate in judge debate of five models on perturbed context across debate rounds, [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 10 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[3]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[4]

Mahak Agarwal and Divyam Khanna. 2025. When persuasion overrides truth in multi-agent llm debates: Introducing a confidence-weighted persuasion override rate (cw-por). arXiv preprint arXiv:2504.00374

arXiv 2025
[5]

Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

arXiv 2020
[6]

Chi-Min Chan and 1 others. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

arXiv 2023
[7]

Giannis Chatziveroglou, Richard Yun, and Maura Kelleher. 2025. https://arxiv.org/abs/2504.02111 Exploring llm reasoning through controlled prompt variations . Preprint, arXiv:2504.02111

arXiv 2025
[8]

Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. https://doi.org/10.18653/v1/2024.acl-long.381 R e C oncile: Round-table conference improves reasoning via consensus among diverse LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066--7085, Bangkok, Thailand. Association ...

doi:10.18653/v1/2024.acl-long.381 2024

Show all 33 references

[9]

Tenenbaum, and Igor Mordatch

Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325

2023 arXiv
[10]

Zorik Gekhman and 1 others. 2024. Does fine-tuning llms on new knowledge encourage hallucinations? arXiv preprint arXiv:2405.05904

2024 arXiv
[11]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. 2024. https://arxiv.org/abs/2404.14219 The llama 3 herd of models . arXiv preprint arXiv:2404.14219

2024 arXiv
[12]

Taicheng Guo and 1 others. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680

2024 arXiv
[13]

Giwon Hong, Jeonghwan Kim, Junmo Kang, Sung-Hyon Myaeng, and Joyce Whang. 2024. Why so gullible? enhancing the robustness of retrieval-augmented models against counterfactual noise. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2474--2495. Ass...

2024
[14]

Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018. Ai safety via debate. arXiv preprint arXiv:1805.00899

2018 arXiv
[15]

Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Li Qiuxia, and Jun Zhao. 2024. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. In Proceedings of the 2024 Joint International Conference ...

2024
[16]

Zachary Kenton and 1 others. 2024. On scalable oversight with weak llms judging strong llms. In Advances in Neural Information Processing Systems, volume 37, pages 75229--75276

2024
[17]

Bowman, Tim Rockt \"a schel, and Ethan Perez

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rockt \"a schel, and Ethan Perez. 2024. Debating with more persuasive llms leads to more truthful answers. In Proceedings of the 41st Internationa...

2024 arXiv
[18]

Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. https://arxiv.org/abs/2303.17491 Language models can solve computer tasks . In Advances in Neural Information Processing Systems (NeurIPS) 2023, volume 37, pages 21510--21523. ArXiv:2303.17491

2023 arXiv
[19]

Hao Lang, Fei Huang, and Yongbin Li. 2025. Debate helps weak-to-strong generalization. arXiv preprint arXiv:2501.13124

2025 arXiv
[20]

Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2023. Large language models with controllable working memory. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1774--1793. Association...

2023
[21]

Tian Liang and 1 others. 2023. Encouraging divergent thinking in large language models through multi-agent debate. arXiv preprint arXiv:2305.19118

2023 arXiv
[22]

Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of EMNLP

2021
[23]

Julian Michael and 1 others. 2023. Debate helps supervise unreliable experts. arXiv preprint arXiv:2311.08702

2023 arXiv
[24]

Cheng Niu and 1 others. 2023. Ragtruth: A hallucination corpus for developing trustworthy retrieval-augmented language models. arXiv preprint arXiv:2401.00396

2023 arXiv
[25]

OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Technical report, arXiv preprint arXiv:2303.08774

2023 arXiv
[26]

Smith, and Mike Lewis

Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. https://arxiv.org/abs/2210.03350 Measuring and narrowing the compositionality gap in language models . arXiv preprint arXiv:2210.03350

2022 arXiv
[27]

Lei Wang and 1 others. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345

2024
[28]

Xuezhi Wang and 1 others. 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

2022 arXiv
[29]

Kevin Wu, Eric Wu, and James Y. Zou. 2024. Clasheval: Quantifying the tug-of-war between an llm’s internal prior and external evidence. In Advances in Neural Information Processing Systems, volume 37, pages 33402--33422

2024
[30]

Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2024. https://openreview.net/forum?id=auKAUJZMO6 Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts . In The Twelfth International Conference on Learning Representations

2024
[31]

Rongwu Xu, Brian Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2024. The earth is flat because...: Investigating LLM s' belief towards misinformation via persuasive conversation. In Proceedings of the 62nd Annual Meeting of the ...

2024
[32]

Michael Zhang and Eunsol Choi. 2023. Mitigating temporal misalignment by discarding outdated facts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14213--14226, Singapore. Association for Computational Linguistics

2023
[33]

Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[3] [3]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[4] [4]

Mahak Agarwal and Divyam Khanna. 2025. When persuasion overrides truth in multi-agent llm debates: Introducing a confidence-weighted persuasion override rate (cw-por). arXiv preprint arXiv:2504.00374

arXiv 2025

[5] [5]

Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

arXiv 2020

[6] [6]

Chi-Min Chan and 1 others. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201

arXiv 2023

[7] [7]

Giannis Chatziveroglou, Richard Yun, and Maura Kelleher. 2025. https://arxiv.org/abs/2504.02111 Exploring llm reasoning through controlled prompt variations . Preprint, arXiv:2504.02111

arXiv 2025

[8] [8]

Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. https://doi.org/10.18653/v1/2024.acl-long.381 R e C oncile: Round-table conference improves reasoning via consensus among diverse LLM s . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066--7085, Bangkok, Thailand. Association ...

doi:10.18653/v1/2024.acl-long.381 2024