A unified benchmark of 24 black-box UE methods for LLMs finds no universal winner but favors methods that reason over answer candidates and hybrid combinations of signals.
knows for a fact
8 Pith papers cite this work. Polarity classification is still indexing.
8
Pith papers citing it
abstract
Multi-agent debate (MAD) is widely used to improve large language model (LLM) performance through test-time scaling, yet recent work shows that vanilla MAD often underperforms simple majority vote despite higher computational cost. Studies show that, under homogeneous agents and uniform belief updates, debate preserves expected correctness and therefore cannot reliably improve outcomes. Drawing on findings from human deliberation and collective decision-making, we identify two key mechanisms missing from vanilla MAD: (i) diversity of initial viewpoints and (ii) explicit, calibrated confidence communication. We propose two lightweight interventions. First, a diversity-aware initialisation that selects a more diverse pool of candidate answers, increasing the likelihood that a correct hypothesis is present at the start of debate. Second, a confidence-modulated debate protocol in which agents express calibrated confidence and condition their updates on others' confidence. We show theoretically that diversity-aware initialisation improves the prior probability of MAD success without changing the underlying update dynamics, while confidence-modulated updates enable debate to systematically drift to the correct hypothesis. Empirically, across six reasoning-oriented QA benchmarks, our methods consistently outperform vanilla MAD and majority vote. Our results connect human deliberation with LLM-based debate and demonstrate that simple, principled modifications can substantially enhance debate effectiveness.
years
2026 8verdicts
UNVERDICTED 8representative citing papers
Empirical study finds isolation drives gains for weak models in multi-agent RAG while scoring matters for strong ones, enabling MADARA for cost-efficient adaptive assessment.
Introduces protocols to aggregate transformed confidence signals from multiagent debates via soft voting or Bayesian fusion, yielding higher AUARC than single agents or standard baselines while keeping F1 stable across benchmarks.
Multicultural multi-agent LLM systems exhibit substantially lower value diversity than human societies on the World Values Survey, with diversity uncorrelated to per-agent alignment and further reduced by agent interactions.
PEAR is a permutation-equivariant adaptive routing protocol for multi-agent LLM debate that reconfigures sparse topologies each round to improve accuracy over fixed debate baselines.
SVR-MAD treats pre-debate signals as priors and debate results as evidence to build a sparser communication graph, cutting token use by up to 61% while preserving or raising accuracy over prior MAD methods.
Closed-system multi-step LLM reasoning is subject to an information-theoretic bound where mutual information with evidence decreases, preserving accuracy while eroding faithfulness, with EGSR recovering it on SciFact and FEVER.
Generative multi-agent systems exhibit emergent collusion and conformity behaviors that cannot be prevented by existing agent-level safeguards.
citing papers explorer
-
A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models
A unified benchmark of 24 black-box UE methods for LLMs finds no universal winner but favors methods that reason over answer candidates and hybrid combinations of signals.
-
To Isolate or to Score? Model-Adaptive Assessment for Cost-Efficient Multi-Agent RAG
Empirical study finds isolation drives gains for weak models in multi-agent RAG while scoring matters for strong ones, enabling MADARA for cost-efficient adaptive assessment.
-
Multiagent Protocols with Aggregated Confidence Signals
Introduces protocols to aggregate transformed confidence signals from multiagent debates via soft voting or Bayesian fusion, yielding higher AUARC than single agents or standard baselines while keeping F1 stable across benchmarks.
-
Beyond Alignment: Value Diversity as a Collective Property in Multicultural Agent Systems
Multicultural multi-agent LLM systems exhibit substantially lower value diversity than human societies on the World Values Survey, with diversity uncorrelated to per-agent alignment and further reduced by agent interactions.
-
PEAR: Permutation-Equivariant Adaptive Routing Multi-Agent Debate
PEAR is a permutation-equivariant adaptive routing protocol for multi-agent LLM debate that reconfigures sparse topologies each round to improve accuracy over fixed debate baselines.
-
SVR-MAD: A Bayesian-Inspired Framework for Posterior-Guided Multi-Agent Debate
SVR-MAD treats pre-debate signals as priors and debate results as evidence to build a sparser communication graph, cutting token use by up to 61% while preserving or raising accuracy over prior MAD methods.
-
The Reasoning Trap: An Information-Theoretic Bound on Closed-System Multi-Step LLM Reasoning
Closed-system multi-step LLM reasoning is subject to an information-theoretic bound where mutual information with evidence decreases, preserving accuracy while eroding faithfulness, with EGSR recovering it on SciFact and FEVER.
-
Emergent Social Intelligence Risks in Generative Multi-Agent Systems
Generative multi-agent systems exhibit emergent collusion and conformity behaviors that cannot be prevented by existing agent-level safeguards.