{"id":"2ef28c4d-1c24-40a5-80ad-3d4f74b23115","arxiv_id":"2502.04790","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"S2-MAD's decision mechanism filters redundant viewpoints and conditionally skips participation, cutting token costs by up to 94.5% versus standard multi-agent debate while keeping accuracy within about 2 points in the headline configuration.","lead":"This paper introduces S2-MAD, a multi-agent debate method that filters redundant agent viewpoints and lets agents skip unproductive discussion rounds, cutting token use by up to 94.5% compared to standard multi-agent debate. It matters because token cost has been the main barrier to scaling LLM debating systems, and the method reportedly keeps accuracy nearly unchanged.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'performance degradation below 2.0%' clause is contradicted by the paper's own Tables 1 and 7, where several configurations show 2.7–3.7 percentage-point accuracy drops relative to MAD.","rationale":"The reader identified the redundancy-filter assumption as the weakest link: treating identical final answers or high cosine similarity as non-informative can discard useful reasoning. I agree that is a real risk, but the most load-bearing problem is more direct: the paper's own empirical tables contradict the headline degradation bound, so the central claim is internally inconsistent as written. This does not require speculation about synonyms or embeddings; it is visible in the reported numbers. The token savings of 80–94% are clearly demonstrated and are the useful part of the paper. Accuracy is often maintained or improved, but in several published cells the loss exceeds 2 percentage points. Since those cells use the same default similarity mechanism described in Section 3.2, the method does not have a guaranteed <2% loss; the claim must be scoped to specific configurations or accompanied by a validation-based threshold-selection protocol. This reinforces rather than overturns the reader's CONDITIONAL verdict: the paper needs revision before the central claim can be accepted, but the empirical contribution is not nullified.","tokens_in":18431,"tokens_out":8555,"duration_ms":86502,"concrete_test":"Recompute, from Tables 1 and 7, the per-configuration accuracy delta Δ = ACC(S2-MAD) − ACC(MAD) for every model/dataset pair, and also compute the token reduction for each row. Then check whether any row with |Δ| > 2.0 percentage points is included in the claim, and whether the configuration attaining the maximum token reduction also satisfies the 2.0% bound. If both are true, report the exact configuration and state that the degradation guarantee is conditional on it; otherwise, replace the blanket 'below 2.0%' language in the abstract and Section 4.2 with per-configuration deltas.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim states that S2-MAD achieves up to 94.5% token-cost reduction 'while maintaining performance degradation below 2.0%.' Because 'up to' qualifies only the token reduction, the degradation bound reads as a property of the method. The paper's own data violate it. In Table 1 (GPT-4-0613, MMLU), MAD accuracy is 90.8 and S2-MAD is 88.1, a 2.7-point drop. In Appendix D Table 7, GPT-4o-mini MMLU drops from 89.5 to 86.1 (3.4 points) and GPT-4o-0806 MATH drops from 79.0 to 75.3 (3.7 points). The Limitation section concedes that the judge can miss synonyms and that token savings depend on answer consistency, but the published numbers already show the 'below 2.0%' clause is not a stable property of the method. If the claim is meant existentially, the paper never identifies the exact configuration that achieves 94.5% savings with <2% degradation, and the five reductions listed in Section 4.2 (94.5%, 84.2%, 92.4%, 83.6%, 88.7%) do not align with any single model row in the tables. The efficiency result is therefore either overgeneralized or under-specified; either way, the strongest claim needs revision or a per-configuration bound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes S2-MAD, a sparsification strategy for multi-agent debate (MAD) in which agents selectively participate based on the similarity of responses to their own, using redundancy filtering, conditional participation, and early stopping. The authors report token-cost reductions of up to 94.5% compared to MAD while claiming 'performance degradation below 2.0%' in the abstract, and they present a theoretical complexity analysis together with experiments on five datasets (GSM8K, MATH, MMLU, GPQA, Arithmetic) using several commercial and open-source LLMs.","tokens_in":18793,"tokens_out":6466,"duration_ms":64743,"significance":"If the reported token reductions hold, S2-MAD addresses a real scalability bottleneck for multi-agent debate. The empirical study is broad: it covers multiple models and datasets, compares against MAD, S-MAD, GroupDebate, CoT, and CoT-SC, and reports both token counts and accuracy with standard deviations. The headline finding of large and consistent token savings is credible and practically useful. However, the abstract's blanket degradation bound is not supported by the paper's own tables, and the theoretical analysis is not a rigorous complexity separation. The practical contribution is still real if the claims are revised to per-configuration statements.","major_comments":[{"comment":"The abstract's claim of 'maintaining performance degradation below 2.0%' is contradicted by the paper's own results. In Table 1, GPT-4-0613 MMLU accuracy drops from 90.8 (MAD) to 88.1 (S2-MAD), a 2.7-point drop; in Table 3, the 2+2+2+2 grouping drops from 86.7 (MAD) to 83.4, a 3.3-point drop; and in Table 7, GPT-4o-mini MMLU drops from 89.5 to 86.1 (3.4 points), while GPT-4o-0806 MATH drops from 79.0 to 75.3 (3.7 points). Since 'up to 94.5%' qualifies only the token reduction, the degradation bound reads as a property of the method. The paper should either report degradation per configuration or replace the blanket claim with a specific list of configurations that satisfy it.","section":"Abstract; Tables 1, 3, 7"},{"comment":"The 'theoretical demonstration' of the token-cost advantage is essentially an accounting identity rather than a proven complexity separation. The final bound, Token = O(MTQ + (M^2 T/N + M S N) C P), is written in terms of P, the upper bound on the average participation probability, which is an unmeasured assumption. The Discussion asserts that agents' answers 'tend to become consistent' and that costs only approach GroupDebate levels when different answers occur with probability only p^{MN}, but no derivation of P from the method's similarity mechanism is given, and no empirical P is reported. As a result, the claimed asymptotic improvement over MAD is conditional on a favorable value of P, and the paper does not establish that this value is achieved. This weakens contribution 2 and should be reframed as an empirical observation.","section":"Section 3.3 and Appendix B"},{"comment":"The similarity threshold tau is a free parameter that materially affects both accuracy and token cost, and Figure 3 shows that the optimal threshold differs by dataset (approximately 0.1 for GSM8K and 0.4 for MATH) and that token cost rises sharply for tau near 0.85. The main results use regular-expression matching, while the VecCS variant is described as an alternative; however, the paper does not provide a principled way to set tau for a new task, so the reported efficiency gains may depend on dataset-specific tuning. The methodology section should state how this parameter would be chosen in practice.","section":"Section 4.3"}],"minor_comments":[{"comment":"The paragraph after the ChatEval discussion contains a long block of '/uni00000018/uni...' artifact text that appears to be a corrupted citation or LaTeX error; it should be removed or repaired.","section":"Section 5.2"},{"comment":"The sentence 'we summarize the outputs from whtnin each group' contains a typo ('whtnin' should be 'within').","section":"Section 3.3"},{"comment":"The definition of D_i^t is difficult to parse: the case conditions are not clearly separated from the summation ranges, and the relationship between D_i^t and the sums in Equations (3) and (4) is not immediately transparent. Please rewrite these equations with clearer notation.","section":"Equation (1) and Equations (3)-(4)"},{"comment":"The list '94.5%, 84.2%, 92.4%, 83.6% and 88.7%' does not specify which model or table row each reduction corresponds to; since Table 1 shows different reductions per model and dataset, please provide an explicit mapping.","section":"Section 4.2"},{"comment":"The caption 'Scaling Study of Token Cost' is uninformative, and the axis labels contain a corrupted Unicode artifact; the figure should be regenerated for clarity.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript contains a corrupted passage in Section 5.2 (the '/uni...' block) that suggests a problem with the PDF generation or the source file; the authors should be asked to fix it. More substantively, the abstract overclaims relative to the paper's own tables on the 'below 2.0% degradation' point, and the theoretical section is presented as a complexity separation when it is actually an assumption-dependent accounting identity. These are fixable with revised claims and clearer reporting. The empirical token-reduction result is strong and reasonably consistent, so a major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the core efficiency result is real: in the (5,4) agent-round configuration, S2-MAD cuts token usage by 75–94% across models and datasets while keeping accuracy within a couple points of MAD in most cells. Second, the abstract's blanket claim—'maintaining performance degradation below 2.0%'—is false as stated. Their own Tables 1 and 7 show drops of 2.7, 3.4, and 3.7 points on MMLU and MATH. The token reduction is 'up to' qualified; the degradation bound is not.\n\nWhat's new: the decision-making mechanism—similarity calculation, redundancy filtering, and conditional participation—is a genuine step beyond S-MAD's fixed sparse edges and GroupDebate's static groups. Agents actively skip redundant exchanges and stop early when views converge. The ablations are informative (filtering matters, early stopping matters, and the 'jump' to inter-group summaries matters), and the scaling study confirms token growth stays sublinear relative to baselines. The limitation section is refreshingly candid: they admit the judge misses synonyms and that savings depend on answer consistency.\n\nSoft spots. The theoretical token-cost 'demonstration' is mostly accounting: total cost is defined with an average participation probability P, but P is never measured, and the claimed asymptotic advantage appears only after choosing N ~ sqrt(MT/S) and assuming P is small. The threshold tau is dataset-dependent (0.1 for GSM8K, 0.4 for MATH), so the method needs per-dataset tuning that the main results don't specify. No code or data are released, so independent replication isn't possible yet. The abstract overclaim is the biggest issue—it's contradicted by the paper's own numbers.\n\nWho this is for: anyone working on multi-agent debate efficiency or token-cost reduction for LLM reasoning systems. The empirical map is useful even if the theory is light. It deserves a serious referee: the core contribution is plausible, the experiments are broad, and the flaws are correctable. I'd send it to review with a note that the abstract claim must be revised to per-configuration bounds, and the theoretical section should be reframed as cost accounting with explicit parameter dependence.","headline":"Real token savings with an overclaimed guarantee—worth reviewing, but the abstract's <2% degradation bound needs to be fixed.","tokens_in":19274,"tokens_out":2115,"would_cite":true,"duration_ms":22128,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"S2-MAD cuts multi-agent debate token costs by up to 94.5% while keeping accuracy loss below 2%.","keywords":["multi-agent debate","token cost reduction","LLM reasoning","redundancy filtering","similarity calculation","sparse communication","group debate","early stopping"],"falsifier":"Construct a dataset of problems where two agents give the same final answer through different derivation paths (for example, same numeric answer via different algebraic routes), run S2-MAD with the redundancy filter enabled, and check whether accuracy falls relative to a version that keeps both derivations; a measurable drop would show the filter is discarding information that identical answers do not capture.","tokens_in":18271,"feed_emoji":"💬","tokens_out":7084,"duration_ms":66202,"temperature":0.7,"pith_summary":"Multi-agent debate improves LLM reasoning, but its token cost grows quickly with the number of agents and rounds. This paper claims most of that cost is wasted: agents in a debate keep repeating similar viewpoints. The authors propose S2-MAD, which measures how similar each incoming response is to what an agent already has, filters out duplicates, and lets agents stay silent when there is nothing new to say. Across five reasoning benchmarks, S2-MAD cuts token use by up to 94.5% versus full debate while keeping accuracy loss below 2%. If that holds, debate-based reasoning becomes far cheaper to scale.","feed_headline":"Filtering repeated answers cuts AI debate token cost by 94%","feed_subtitle":"Selective Sparse MAD keeps accuracy within 2% while letting agents stay silent when they have nothing new to add.","key_machinery":"The Decision-Making Mechanism, composed of three modules: the Similarity Calculation Module, which gauges whether two responses express the same viewpoint by extracting final answers with regular expressions or by computing cosine similarity between embedded responses; the Redundancy Filtering Module, which discards any incoming response similar to the agent's own or previously accepted viewpoints; and the Conditional Participation Module, which keeps an agent silent when all incoming viewpoints match its own. This mechanism converts a fully connected debate graph into a sparse, adaptive one and also enables early termination when agent summaries converge.","core_discovery":"The paper's central claim is that the token cost of multi-agent debate can be reduced by an order of magnitude without meaningful accuracy loss by treating redundant viewpoints as non-information. S2-MAD's Decision-Making Mechanism compares each incoming response with the agent's own through either regular-expression answer matching or embedding-based cosine similarity; responses judged similar are filtered out, and an agent participates only when it holds a viewpoint that differs from what it has already seen. The debate stops early when all group summaries agree, and the final answer comes from majority voting. The authors report token-cost reductions of up to 94.5% against standard MAD, 90.2% against Sparse-MAD, and 87.0% against GroupDebate, with accuracy degradation below 2.0% on GSM8K, MATH, MMLU, GPQA, and an arithmetic task. Their asymptotic analysis gives S2-MAD a token complexity of $O(MTQ + (M^2T/N + MSN)CP)$, which drops below MAD's complexity once the average participation probability $P$ shrinks as answers converge.","pith_inferences":["If the redundancy-is-waste assumption generalizes, other ensemble and collaborative LLM methods (e.g., self-consistency sampling, retrieval-augmented debate) could adopt similarity gating to cut compute, since the paper's mechanism does not depend on the debate format specifically.","The paper's similarity module compares final answers or surface embeddings, not the reasoning traces behind them; a natural next step is measuring similarity on the justification level, which the current regex/embedding judge does not capture.","The reported cost savings are tied to benchmark conditions where agent answers converge quickly; on adversarial or open-ended tasks where diverse answers persist, the paper's own cost analysis implies the savings will shrink, so the 94.5% figure should not be read as a universal bound."],"forward_implications":["Adding agents or debate rounds under S2-MAD costs far fewer tokens than under standard MAD once answers start to converge, so scaling debate becomes affordable in token-limited settings.","The similarity threshold is a direct control on the trade-off: lower thresholds save more tokens but risk discarding useful viewpoints, and the optimal threshold varies by dataset.","Using vectorization with cosine similarity can outperform regex answer matching, yielding both higher accuracy and lower token cost at a tuned threshold.","Early stopping when group summaries align is a cheap source of savings: removing it slightly lowers accuracy and raises tokens in the ablation.","The redundancy filter is the main cost-saving module: removing it raises GSM8K accuracy by about 2 points but nearly triples token usage."],"supporting_citations":[{"why":"Defines the standard Multi-agent Debate framework that S2-MAD sparsifies; serves as the primary baseline for token-cost comparison.","marker":"Liang et al., 2023"},{"why":"Introduces Sparse-MAD, the sparse-communication baseline S2-MAD is compared against.","marker":"Li et al., 2024"},{"why":"Introduces GroupDebate, the grouped-discussion baseline whose token cost S2-MAD reduces further.","marker":"Liu et al., 2024"},{"why":"Provides the self-consistency (CoT-SC) baseline used to show S2-MAD reaches comparable accuracy with fewer tokens.","marker":"Wang et al., 2022"},{"why":"Provides the GSM8K math word problem benchmark used in the main experiments.","marker":"Cobbe et al., 2021"},{"why":"Provides the MATH benchmark and is used for the threshold analysis with GPT-4o-mini.","marker":"Hendrycks et al., 2021"},{"why":"Provides the MMLU benchmark measuring broad multitask accuracy.","marker":"Hendrycks et al., 2020"},{"why":"Provides the GPQA graduate-level question benchmark used for accuracy and token-cost comparison.","marker":"Rein et al., 2023"}],"fun_headline_variants":["S2-MAD slashes debate token costs by 94% with <2% accuracy dip","AI debate's token barrier broken: 94% cost cut, <2% quality loss","Sparse Multi-Agent Debate: 94% token reduction, <2% accuracy drop"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a response similar to what an agent already has—same final answer or high cosine similarity—carries no information worth exchanging; if that fails, the filter can discard a useful justification or keep a redundant one.","fun_headline_variants_meta":{"raw":{"variants":["S2-MAD slashes debate token costs by 94% with <2% accuracy dip","AI debate's token barrier broken: 94% cost cut, <2% quality loss","Sparse Multi-Agent Debate: 94% token reduction, <2% accuracy drop"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000462,"raw_usage":{"total_tokens":2332,"prompt_tokens":986,"completion_tokens":1346,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1271}},"tokens_in":602,"tokens_out":1346,"duration_ms":11481,"temperature":1.0,"reasoning_tokens":1271,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:27:15.117535+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a dataset of problems where two agents give the same final answer through different derivation paths (for example, same numeric answer via different algebraic routes), run S2-MAD with the redundancy filter enabled, and check whether accuracy falls relative to a version that keeps both derivations; a measurable drop would show the filter is discarding information that identical answers do not capture.","supporting_citations":[],"review_version":1}