{"id":"f2a76f93-a85c-44fc-abdb-29fb963b5c4d","arxiv_id":"2502.08514","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MADISSE assigns LLM evaluators random initial stances (faithful or unfaithful), has them debate in rounds, and adjudicates, improving summary faithfulness evaluation accuracy while introducing an annotated 'ambiguity' dimension.","lead":"This paper proposes MADISSE, a multi-agent LLM debate system for judging whether summaries are faithful to source documents, where agents are randomly assigned to argue 'faithful' or 'unfaithful' before debating. The system beats single-LLM baselines on four faithfulness benchmarks, and the authors add a new 'ambiguity' label plus a 770-sentence annotated dataset for summaries that can be read both ways.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: MADISSE's gains may stem from forced two-sided argumentation, not multi-agent debate dynamics.","rationale":"The paper makes a strong empirical case with consistent gains across four datasets and multiple LLMs, and it releases code and data. However, the central contribution is the multi-agent debate framework, and the current experiments do not isolate the debate component from the forced two-sided argumentation that the stance initialization produces. The reader's weakest assumption identified the same underlying concern: that debate is truth-revealing rather than an artifact of the stance distribution or adjudicator behavior. My stress-test makes this concern concrete by specifying a missing control that would directly test whether the debate dynamics, as opposed to the generation of arguments for both sides, are responsible for the improvements. The proposed control is cheap and would settle the attribution of the gains. The paper's other contributions—the ambiguity taxonomy, the annotated dataset, and the ambiguity detection results—are valuable even if the debate-specific claim is weakened, so the recommended verdict remains CONDITIONAL rather than a full rejection.","tokens_in":36401,"tokens_out":4348,"duration_ms":44544,"concrete_test":"Run the following single-LLM control on all four datasets with Llama3-70B-instruct and the same guidelines as Table 29: prompt the model to (1) generate the strongest arguments that the summary is faithful, (2) generate the strongest arguments that it is unfaithful, (3) then make a final faithfulness judgment based on those arguments and the guidelines, outputting a label. Repeat the procedure 12 times (to match the number of adjudicator votes in the simultaneous setting, k=3 adjudicators × 4 agents) and majority-vote the final labels. Compare BAcc and K-alpha against MADISSE single-session and simultaneous-debates rows in Table 3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that multi-agent debate with random initial stances improves faithfulness evaluation (Table 3). The paper compares against zero-shot, CoT, self-consistency, and MADISSE without initialization, but never ablates the debate mechanism while preserving the forced two-sided argumentation. Each stance-forced agent produces a justification for a randomly assigned label; the adjudicator then reads these justifications. The improvement could therefore come from exposing the adjudicator to arguments on both sides, which a single LLM could do with a prompt that first argues faithful, then unfaithful, then decides. The paper's own Section 3.4 acknowledges 'wrong answer propagation' and 'adjudicator mistake' as failure modes, yet no analysis shows that multi-round interaction (agents responding to each other, changing their minds) improves individual agent accuracy. Table 35 shows the main effect is a recalibration: on MediaSum, zero-shot has FPR ~0 and FNR 86.2%, while MADISSE has FPR 16% and FNR 50.9%; BAcc rises but so does FPR. This is consistent with the forced-stance prompt inducing the model to look for errors, not with debate-specific information exchange. Without a single-LLM 'argue both sides' control, the paper's attribution of the gains to the multi-agent debate component is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MADISSE, a multi-agent debate framework for LLM-based summary faithfulness evaluation. Evaluator agents are randomly assigned initial stances (faithful or unfaithful), argue in up to three debate rounds, and a separate adjudicator makes the final call, optionally with multiple simultaneous sessions aggregated by majority vote. The authors report improved balanced accuracy and Krippendorff alpha over zero-shot, chain-of-thought, self-consistency, and debate-without-initialization baselines on TofuEval MeetingBank/MediaSum and AggreFact CNN/XSum. They also introduce an 'ambiguity' dimension for faithfulness evaluation, provide a 16-type taxonomy, annotate 770 MeetingBank sentences for ambiguity, and report an automatic ambiguity detector that uses the debate arguments (71.4 BAcc). Filtering ambiguous cases from evaluation improves model-human agreement.","tokens_in":36655,"tokens_out":3333,"duration_ms":33281,"significance":"If the reported gains hold, MADISSE is a practical contribution to LLM-based summary faithfulness evaluation, showing consistent improvements across four datasets and two underlying LLMs (Llama3-70B in the main text, GPT-4o-mini and Llama-3-8b in the appendix). The paper also makes a useful conceptual point that some summaries admit multiple plausible interpretations leading to opposing faithfulness judgments, and it contributes a detailed taxonomy and a human-annotated ambiguity dataset on MeetingBank. The release of code and data is a strength. However, the central causal claim—that the gains come from multi-agent debate dynamics rather than from forced two-sided argumentation—is not yet established, and the absence of any variance or significance reporting makes the magnitude of the headline improvements uncertain.","major_comments":[{"comment":"The paper attributes the accuracy gains to multi-agent debate, but MADISSE differs from all baselines in two bundled components: (i) forced two-sided argumentation from random initial stances, and (ii) multi-round interaction among agents. The MADISSE-wo-initialization baseline controls for debate without stances, but there is no control that provides forced two-sided argumentation without debate, e.g., a single-LLM prompt that first argues for faithfulness, then against faithfulness, then adjudicates. This control is necessary because Table 35 shows the main effect is largely a recalibration: on MediaSum, zero-shot has FPR ≈ 0 and FNR 86.2%, while MADISSE has FPR 16% and FNR 50.9%, which is consistent with prompting the model to search for errors rather than with debate-specific information exchange. Without this ablation, the attribution of the gains to the multi-agent debate component is not established.","section":"§3, §5.2, Table 3 and Table 35"},{"comment":"No error bars, confidence intervals, seeds, or significance tests are reported for any of the main results. Many improvements are moderate in absolute terms (e.g., MeetingBank BAcc from 68.2 to 75.1, XSum from 68.1 to 75.1), and the self-consistency baseline on XSum with GPT-4o-mini (74.63) nearly matches MADISSE (72.86), so point estimates alone are insufficient to conclude that MADISSE is systematically better. The authors should report variance across at least several runs and state whether the observed differences are stable.","section":"Table 3 and Appendix D"},{"comment":"The ambiguity detector that uses debate arguments relies on arguments produced by agents with forced opposite stances. Since both faithful and unfaithful arguments are manufactured by construction, the presence of 'sound arguments' on both sides may partly reflect the initialization rather than genuine ambiguity. A control that feeds the same detector arguments produced without stance forcing—or from a single LLM asked to argue both sides—is needed to support the claim that debate arguments help identify ambiguities. Additionally, the filtering experiment in Figure 3 removes instances using gold ambiguity labels, not the proposed automatic detector; as presented, it shows that the ambiguity construct affects evaluator agreement, but it does not validate the automatic detector for filtering in practice.","section":"§4.3, Table 4, Figure 3"},{"comment":"The binary ambiguity annotation has raw inter-annotator agreement of approximately 0.40 before expert adjudication, with final Cohen's Kappa of approximately 0.73 after experts made the final call. The paper should report the proportion of cases where expert adjudication overruled the annotators and discuss whether the 'ambiguity' construct is robust enough to serve as a ground-truth dimension for automatic detection. This is not a fatal issue, but it is load-bearing for the ambiguity claims because the detector is trained and evaluated against this adjudicated label.","section":"Appendix B.2, §4.2"}],"minor_comments":[{"comment":"Figure 6 is captioned 'FPR and FNR results pre and post filtering the ambiguous cases' but the figure panels are labeled BAcc and K-alpha; the caption and figure labels are inconsistent.","section":"Figure 6 caption"},{"comment":"The table caption refers to 'MADISSE wo. random initialization' while the main text and Table 3 use 'MADISSE wo ini- tialization'; please unify terminology.","section":"Table 34"},{"comment":"The sentence 'It can also help with ambiguity detection later discussed in Section 4)' has an unmatched closing parenthesis.","section":"§3.1"},{"comment":"The zero-shot prompt in Table 26 instructs the model to 'go over each sentence of the summary one by one' but the summary-level evaluation uses full summaries; please clarify whether the prompt is the same for summary-level and sentence-level settings.","section":"Appendix C, Table 26"},{"comment":"The evaluation criteria section defines BAcc and Krippendorff alpha but does not define 'K-alpha' abbreviation at first use; also, the relationship between K-alpha and human agreement could be stated more precisely.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a plausible and potentially useful contribution, and the consistent direction of the results across datasets and LLMs gives some confidence in the empirical finding. However, the missing control for forced two-sided argumentation is a central attribution issue that should be resolved before publication, and the lack of any statistical reliability reporting is a concern for a paper whose main evidence is a set of point estimates. I would recommend major revision, with the expectation that the authors add the argued-both-sides control and report variance or significance information. The ambiguity contribution is interesting but would benefit from a more cautious interpretation of the automatic detector's role."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid, useful paper with a real empirical effect, but the headline attribution to multi-agent debate is not fully established. The missing control is a single LLM that is prompted to argue both sides internally before deciding. MADISSE without initialization is not that control; it shows the random stances matter, but not that the multi-agent interaction is what carries the improvement. The FPR/FNR table actually makes this worry concrete: on MediaSum, zero-shot sits at FPR ~0 and FNR 86%, while MADISSE goes to FPR 16% and FNR 51%. The balanced-accuracy gain is a recalibration, not evidence of debate-specific information exchange. That said, this is a missing ablation, not a fatal flaw. The practical method works and practitioners will care about that.\n\nWhat is genuinely new: the uniformly random initial stance assignment, consistently evaluated on four datasets (TofuEval MeetingBank, MediaSum, AggreFact CNN, XSum) with two underlying LLMs in the appendices. The gains are consistent, and the simultaneous-debate variants push them further. The ambiguity taxonomy and the 770-sentence MeetingBank annotation are a real contribution; the adjudicated Cohen’s kappa of 0.73 is decent, and the paper is honest that raw agreement is 0.40. Code and data are released, which I count as real evidence.\n\nSoft spots, in proportion: no seeds, confidence intervals, or significance tests anywhere, so the 5–10 point BAcc gaps have unknown variance. The ambiguity detector has a mild circular flavor—the debate is forced to produce both faithful and unfaithful arguments—though the best variant asks the model to judge which arguments are sound, which is a reasonable way to break the circle. The filtering analysis uses gold ambiguity labels rather than the automatic detector, so the full pipeline benefit remains unquantified. The paper’s own limitations section acknowledges the narrow LLM set and binary-label scope.\n\nWho is this for: anyone building LLM-based summary evaluators and researchers working on ambiguity in faithfulness. The paper deserves a serious referee; I would send it out with a request for the argue-both-sides control, variance estimates, and an end-to-end ambiguity-filtering experiment.","headline":"A useful empirical paper with a real effect, but the missing single-model “argue both sides” control means the gains are not cleanly attributable to multi-agent debate.","tokens_in":37228,"tokens_out":2181,"would_cite":true,"duration_ms":27070,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Assigning LLM summary evaluators randomly chosen \"faithful\" or \"unfaithful\" starting positions and letting them debate improves error detection over standard prompts, while also exposing summaries that are genuinely ambiguous rather than…","keywords":["summary faithfulness","LLM evaluation","multi-agent debate","stance initialization","ambiguity taxonomy","hallucination detection","TofuEval","AggreFact"],"falsifier":"Run MADISSE with the same initial stances but replace each round's counter-arguments with paraphrased repeats of previous arguments: if balanced accuracy stays high, the gain is not coming from genuine evidence exchange. Alternatively, measure how often agents that start on the wrong stance switch to the correct label after reading the other side; if switches are rare and the gain comes only from the seeded unfaithful agents arguing well, the debate is functioning as ensemble sampling rather than persuasion.","tokens_in":36211,"feed_emoji":"⚖️","tokens_out":6213,"duration_ms":56639,"temperature":0.7,"pith_summary":"This paper argues that summary faithfulness evaluation is better when LLM evaluators are forced to argue from a randomly assigned side: half start believing the summary is faithful and half believing it is unfaithful, then debate for up to three rounds before adjudicators vote. Across four datasets this raises balanced accuracy over zero-shot, chain-of-thought, and self-consistency baselines, for example from 68.2 to 75.1 on MeetingBank and from 68.1 to 75.1 on XSum. The paper also argues that a strict faithful/unfaithful label is often not well defined, because some summaries have multiple correct interpretations that yield opposing faithfulness judgments. It introduces an ambiguity dimension with a 16-type taxonomy, shows that debate arguments detect such cases at 71.4 balanced accuracy, and reports that filtering ambiguous summaries raises agreement between model labels and human labels. If correct, this gives an explainable, training-free route to evaluators that catch more hallucination-style errors and that can distinguish model disagreement from legitimate ambiguity.","feed_headline":"Debating LLMs with assigned sides catch more summary errors","feed_subtitle":"On four benchmarks, assigned-side debates lift balanced accuracy by up to 7 points and flag genuinely ambiguous summaries.","key_machinery":"The central object is MADISSE, a three-stage evaluation pipeline: initialization assigns each evaluator agent a fixed stance (uniformly, so half begin with \"faithful\" and half with \"unfaithful\"), debate lets four agents exchange arguments for up to three rounds with shuffled chat histories and shared guidelines, and adjudication has judges read the final arguments and reach a label by majority vote. The paper also introduces a second object, the ambiguity taxonomy, a coarse-to-fine scheme grouping implicit reasoning phenomena, meaning phenomena, and context phenomena into 16 types, which is used both to annotate data and to prompt an LLM to decide whether sound arguments exist on both sides of the faithfulness question.","core_discovery":"The central claim is that a multi-agent debate in which half the evaluator agents start with the imposed stance \"faithful\" and half with \"unfaithful\" produces more diverse arguments, identifies more errors, and yields labels better aligned with human annotation than asking a single LLM or running a debate without forced stances. In the paper's experiments with Llama3-70B-instruct, balanced accuracy rises from 68.2 to 75.1 on MeetingBank and from 68.1 to 75.1 on XSum, with Krippendorff alpha on MeetingBank rising from 0.38 to 0.50; running three simultaneous independent sessions pushes MeetingBank balanced accuracy to 78.1. The paper further claims that many summaries are not cleanly faithful or unfaithful, defines ambiguity as a summary admitting multiple correct interpretations that lead to opposing faithfulness judgments, provides a taxonomy of 16 ambiguity types, and shows that an LLM reading the debate arguments detects these ambiguous cases at 71.4 balanced accuracy. Filtering human-annotated ambiguous sentences out of the evaluation set improves the measured agreement of every evaluator, with the largest gap for the debate approach.","pith_inferences":["A testable consequence the paper leaves implicit is whether the gain comes from genuine evidence exchange or simply from stance diversity: a variant that freezes the initial stances but prevents agents from reading each other's arguments would separate those two sources.","The ambiguity detector could be reused as a general test-time filter for LLM evaluation pipelines: run the debate, keep only instances where the opposing arguments are one-sided, and report faithfulness only on those.","The uniform stance split is a deliberate choice, and the paper's own appendix shows that skewing the split trades false positives against false negatives; a natural extension is to calibrate the assigned stance ratio to a domain's expected error rate.","Since the ambiguity annotations cover only sentence-level MeetingBank instances, an open question is whether ambiguity spans full summaries and whether the taxonomy transfers to other domains such as medical or legal summarization."],"forward_implications":["Automatic summary faithfulness evaluation can be improved without training or fine-tuning by structuring the prompting process as a multi-agent debate with forced opposing initial stances.","Because the debate produces written arguments, the final label comes with explanations, addressing the explainability gap of overlap-based and fine-tuned metrics.","The faithfulness judgment of a summary should be reported with an ambiguity axis: filtering out ambiguous summaries raises model-human agreement, for example Krippendorff alpha on MeetingBank from 0.52 to 0.67 for the zero-shot baseline and from 0.53 to 0.71 for the debate approach.","The debate approach lowers the false negative rate, meaning it catches more errors, at the cost of a higher false positive rate, and the paper attributes a meaningful share of that extra sensitivity to ambiguous summaries rather than to evaluator mistakes.","The approach transfers across underlying LLMs, with the same performance ordering reported for GPT-4o-mini and for a smaller Llama-3-8b model, so the gain is tied to the debate structure rather than to one model's prompting behavior."],"supporting_citations":[{"why":"supplies the TofuEval MeetingBank and MediaSum faithfulness benchmarks, and the MeetingBank sentences that are re-annotated for ambiguity.","marker":"Tang et al. (2024b)"},{"why":"supplies the AggreFact CNN and XSum benchmarks used as the news-domain test bed.","marker":"Tang et al. (2023a)"},{"why":"provides Llama 3, the underlying LLM for the main experimental results.","marker":"AI@Meta (2024)"},{"why":"defines the chain-of-thought baseline that MADISSE is compared against.","marker":"Wei et al. (2022)"},{"why":"defines the self-consistency baseline, which samples multiple reasoning paths and takes a majority vote.","marker":"Wang et al. (2022)"},{"why":"prior multi-agent debate work that assigns affirmative and disagreeing roles, the direct antecedent of stance initialization.","marker":"Liang et al. (2023)"},{"why":"identifies wrong-answer propagation in multi-agent discussion, motivating the simultaneous independent debate sessions in MADISSE.","marker":"Wang et al. (2024)"}],"fun_headline_variants":["Assigned-side debates catch more summary errors","Forced stance debate finds more summary inaccuracies","Ambiguity-aware multi-agent debate boosts faithfulness checking","Side-taking LLM agents flag more summary mistakes","Debate with imposed sides improves summary error detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that forcing an LLM to argue for a randomly assigned side surfaces true evidence about the summary, so the majority vote over the debate is more accurate than an unforced judgment and is not just counting whichever side argued more fluently.","fun_headline_variants_meta":{"raw":{"variants":["Assigned-side debates catch more summary errors","Forced stance debate finds more summary inaccuracies","Ambiguity-aware multi-agent debate boosts faithfulness checking","Side-taking LLM agents flag more summary mistakes","Debate with imposed sides improves summary error detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000254,"raw_usage":{"total_tokens":1570,"prompt_tokens":949,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":565,"tokens_out":621,"duration_ms":6724,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T04:47:59.801559+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MADISSE with the same initial stances but replace each round's counter-arguments with paraphrased repeats of previous arguments: if balanced accuracy stays high, the gain is not coming from genuine evidence exchange. Alternatively, measure how often agents that start on the wrong stance switch to the correct label after reading the other side; if switches are rare and the gain comes only from the seeded unfaithful agents arguing well, the debate is functioning as ensemble sampling rather than persuasion.","supporting_citations":[],"review_version":1}