{"id":"311f7b5c-99d9-4494-bbcf-dee61e346cd6","arxiv_id":"2507.12370","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A leader-follower multi-agent debate protocol improves ambiguity detection for two of three tested LLMs, but the reported results lack error bars, a clear success metric, and contain internal numerical inconsistencies.","lead":"This paper tests whether a debate between three small language models improves their ability to spot ambiguity in robot commands. It finds that debate helps Llama3-8B and Mistral-7B but hurts Gemma2-9B, with the best debate configuration scoring 76.7% success.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline success rates are uninterpretable because 'success' is never operationalized; the scoring rule and treatment of non-consensus debate outcomes are unspecified, so the central single-vs-debate comparison is not computable from the paper.","rationale":"The reader's weakest assumption is on target. I re-read Sections 3.1 through 4.4 and found no operational definition of a successful trial, no dataset size, no gold-standard questions, and no statement of how non-consensus debate outcomes enter the reported rates. The strongest claim depends entirely on these numbers. This is not a disagreement with an external consensus; it is an internal incompleteness in the evaluation protocol. The paper deserves credit for a controlled synthetic dataset and a clear leader-follower protocol, but the central quantitative result is not reproducible as written. The inconsistency between Section 4.3 and Table 1 consensus rates reinforces this. I therefore keep the reader's CONDITIONAL verdict: the finding is plausible but needs the scoring rubric, dataset, and code before acceptance. I do not recommend moving to REJECT because the flaw is fixable by releasing materials and the underlying idea is reasonable.","tokens_in":7628,"tokens_out":7813,"duration_ms":84867,"concrete_test":"Recompute all Section 4.1 and 4.2 success rates from the raw outputs under a fixed, preregistered rubric: for ambiguous items, success requires that the final output flags ambiguity AND proposes a question targeting the injected ambiguity type (numerical, attribute, or spatial); for unambiguous items, success requires the final output to say 'clear.' Apply the same rubric to single-agent and debate outputs, count non-consensus trials as failures, and also report the success rate excluding them. If the ordering 'Debate(Mistral-7B) 76.7% > single Mistral-7B 28.3%' or 'Debate(Gemma2-9B) 48.3% < single Gemma2-9B 80.0%' changes under any reasonable rubric variant, the central claim fails as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines the agent task (state 'clear' or propose one clarifying question) but Section 4 never defines what counts as a successful trial. The reported percentages (13.3%, 28.3%, 80.0%, 76.7%, 40.0%, 48.3%) could be based on (i) binary ambiguity detection only, (ii) detection plus any clarifying question, or (iii) detection plus a question matching a gold-standard or unambiguous counterpart. These criteria can yield very different rates: a model that correctly says 'ambiguous' but proposes an off-target question succeeds under (i) and fails under (iii). Section 3.1 describes only ambiguous/unambiguous instruction pairs, not gold-standard clarification questions or an annotation rubric. Moreover, in the debate condition the final outcome is the Leader's proposal only when consensus is reached; for non-consensus trials Section 3.3.2 says the outcome is 'marked as non-consensus' and the final state is 'recorded,' without stating whether such trials are scored as failures, excluded, or scored using the Leader's last proposal. Single-agent baselines always produce a final proposal, so the two conditions may use different scoring denominators. This makes the central single-vs-debate comparison non-computable from the text. The internal inconsistency between Section 4.3 (Mistral non-consensus 1.7%) and Table 1 (Mistral consensus reach 90%, i.e., non-consensus around 10%) further indicates the evaluation pipeline is not stable. Until the scoring rule and dataset are released, the 76.7% claim cannot be verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes and evaluates a leader-follower multi-agent debate framework for detecting ambiguity in user instructions and generating clarifying questions, using three small open-weight LLMs (Llama3-8B, Gemma2-9B, Mistral-7B) on a programmatically generated dataset of numerical, attribute, and spatial ambiguities. The authors compare single-agent baselines with debates in which one model leads and two others critique, rotating the leader role, and report success rates, consensus rates, and debate latency. The main claim is that debate substantially improves performance for Llama3-8B and Mistral-7B, with Mistral-led debates reaching 76.7% success, while degrading the already strong single-agent Gemma2-9B (80.0% single vs. 48.3% debate).","tokens_in":7993,"tokens_out":3752,"duration_ms":40629,"significance":"If the reported results hold, the paper would provide a useful empirical data point on when multi-agent debate helps or hurts small LLMs for ambiguity detection in human-robot interaction, and its use of a controlled programmatic dataset and role rotation is a reasonable methodological start. However, the central evaluation metric is never operationalized, the dataset size is absent, and the consensus statistics are internally inconsistent, so the headline comparison between single and debate conditions is not currently computable from the manuscript. These are fixable but essential gaps, and the manuscript also lacks any error bars or significance testing, which is especially important given the stochastic nature of LLM decoding and the small reported effect sizes.","major_comments":[{"comment":"The evaluation metric 'success' is never defined. Section 3.3 specifies that an agent must either state that the instruction is clear or propose a single clarifying question, but Section 4.1 reports success rates (e.g., 80.0% for single Gemma2-9B, 76.7% for Debate(Mistral-7B)) without stating whether a successful trial requires correct ambiguity detection only, detection plus any clarifying question, or detection plus a question matching a gold-standard question, and without specifying how unambiguous control items are scored. This makes the central single-versus-debate comparison non-computable from the manuscript.","section":"Section 3.3 and Section 4.1"},{"comment":"The dataset size is not reported. The text says 'a specified number of unique examples' but never gives the total number of instructions or the per-type counts for numerical, attribute, and spatial ambiguity. All percentages in Section 4 therefore lack denominators, and no error bars or confidence intervals are provided anywhere in Section 4, so the reader cannot assess the precision of any reported success rate.","section":"Section 3.1"},{"comment":"The consensus statistics are internally inconsistent. Section 4.3 states that Mistral-7B-led debates had a non-consensus rate of 1.7% (i.e., a consensus rate of 98.3%), while Table 1 reports a consensus reach of 90% for Mistral-7B. Moreover, with a consensus-outcome success rate of 95% from Table 1, neither a 90% nor a 98.3% consensus rate reconciles with the overall Debate(Mistral-7B) success rate of 76.7% under standard assumptions that non-consensus trials are scored as failures. Please state explicitly whether non-consensus trials are counted as failures, excluded from the denominator, or scored using the leader's last proposal, and correct the numerical conflict.","section":"Section 4.3 and Table 1"},{"comment":"The claims of improvement or degradation, such as Mistral-7B's spatial ambiguity detection rising from 10% to 75% under debate, are based on single runs with no variance or significance testing. Given that the models are stochastic and the sample sizes are unreported, these differences may be within noise. The authors should report multiple independent runs or bootstrap confidence intervals for all headline percentages in Section 4.","section":"Section 4.2"}],"minor_comments":[{"comment":"There is a typographical error in the reference list: 'V ol' should be 'Vol' in multiple entries, and the same typo appears in the text.","section":"References"},{"comment":"The text has a spacing typo: 'F ollower' should be 'Follower'.","section":"Section 3.3.2"},{"comment":"The conclusion contains a spacing typo: '13. 3%' should be '13.3%'.","section":"Section 5"},{"comment":"The approximate percentages in the text, such as 'attribute ( 70%), numerical ( 85%), and spatial ( 85%)', should be replaced with exact values or a table, and the spacing should be corrected.","section":"Section 4.2"},{"comment":"The manuscript references Figures 2 through 7 with captions, but the figure images are not visible in the submitted text; please ensure all plots are embedded or provide the data tables underlying them.","section":"Figures 2-7"},{"comment":"It is unclear whether follower responses are shared with the other follower or only with the leader; specify the information flow between followers, since this affects the debate dynamics.","section":"Section 3.3.2"}],"recommendation":"major_revision","confidential_remarks":"I concur with the reader's assessment: the missing success metric is the central blocker. The paper's contribution is modest but potentially suitable for the venue if the evaluation is fully specified, the consensus numbers are corrected, and the dataset size and variance are reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The multi-agent debate setup for ambiguity detection is a reasonable extension of prior work, and the paper deserves credit for testing three small LLMs, rotating leader roles, and acknowledging that debate helps some models and hurts a strong one. The related work is honest about the connection to Madisse and LLM-Consensus. The synthetic dataset generation is described clearly enough for three ambiguity types, though the dataset size is never stated.\n\nThe problem is load-bearing: Section 4 reports success rates but never defines what a successful trial is. Does the model need to flag ambiguity? Does its clarifying question need to match a gold standard? Are “clear” verdicts on unambiguous instructions counted? The paper doesn’t say. Without that, the single-vs-debate comparison is not computable. The stress-test note is right that this is fatal to the empirical claim. A model that correctly says “ambiguous” but asks an off-target question could pass one scoring rule and fail another, so the reported 13.3%, 76.7%, and 80.0% are uninterpretable.\n\nThere’s also a direct numerical contradiction: Section 4.3 puts Mistral’s non-consensus rate at 1.7%, but Table 1 says Mistral reaches consensus 90% of the time, implying a 10% non-consensus rate. Similarly, Llama3-8B text says 31.7% non-consensus while Table 1 implies 15%. That inconsistency suggests the evaluation pipeline was not stable across runs.\n\nNo error bars, no significance tests, and no released data or code further weaken the empirical support. The paper’s own framing—debate helps weaker models, hurts a stronger one—is plausible and consistent with other multi-agent results, but as reported, it’s not backed by a verifiable scoring protocol.\n\nThe paper would be more believable as a workshop contribution than as a citable result. If the authors define the scoring rule, describe how non-consensus trials are treated, release data and code, and fix the numerical inconsistency, it could be a modest but useful application note for HRI researchers using local LLMs. As it stands, a serious referee would need the missing information before trusting any of the percentages. I would not send this to peer review in its current form; I’d desk reject with an invitation to resubmit after major revisions and a complete evaluation description.","headline":"The paper asks a sensible question but never defines what \"success\" means, so the headline 76.7% is not interpretable and the central comparison collapses.","tokens_in":8488,"tokens_out":1756,"would_cite":false,"duration_ms":21660,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A leader-follower debate among three small language models improves ambiguity detection for two of them, lifting Mistral-7B from 28.3% to 76.7% success, while degrading the strongest single model.","keywords":["large language models","ambiguity detection","multi-agent debate","leader-follower protocol","human-robot interaction","clarification questions","small language models"],"falsifier":"Run the same three models on the same dataset with a pre-specified scoring rule that requires the final clarifying question to exactly match the gold-standard question for each ambiguous instruction and requires a 'clear' verdict on the unambiguous counterparts; if Mistral-7B-led debate no longer beats its single-agent baseline under that rule, the paper's central comparison collapses.","tokens_in":7445,"feed_emoji":"💬","tokens_out":8376,"duration_ms":75701,"temperature":0.7,"pith_summary":"The paper asks whether a structured debate among several small language models can catch ambiguities in user instructions that any single model would miss. It reports that a leader-follower debate protocol markedly improves two of the three models tested: Llama3-8B rises from 13.3% to 40.0% success and Mistral-7B from 28.3% to 76.7%, with Mistral-led debates reaching consensus quickly and handling spatial ambiguity especially well. The same protocol hurts the strongest single model: Gemma2-9B alone scores 80.0%, while its debates score 48.3%. The authors conclude that debate is a model-dependent, targeted enhancement rather than a universal fix, and that the temporal cost of roughly 22 to 28 seconds per debate matters for real-time use.","feed_headline":"Debate lifts weak LLM ambiguity detection to 76.7%","feed_subtitle":"Three models debate; Mistral-led talks nearly triple its success, while the strongest solo model gets worse.","key_machinery":"The leader-follower debate protocol is the mechanism that carries the argument. Three agents, one leader and two followers, rotate roles on every instruction; the leader first proposes a verdict of 'clear' or a single clarifying question, each follower independently says 'Agree' or 'Disagree' and may offer an alternative question, and if both followers agree the debate ends. Otherwise the leader integrates the feedback and produces a revised proposal, repeating up to five rounds. This machinery forces the leader to incorporate dissenting views, so the final output represents a consensus among independent agents rather than one model's first guess, which is why the authors attribute the accuracy gains to the structure of the exchange.","core_discovery":"On its own terms, the paper's central claim is that a leader-follower multi-agent debate can extract better ambiguity judgments from small language models than the same models acting alone, and that the gain is conditional on which model leads. The evidence is a programmatically generated set of instructions with numerical, attribute, and spatial ambiguities, each paired with an unambiguous counterpart. Under the paper's reported but unspecified scoring rule, debate raises Llama3-8B from 13.3% to 40.0% and Mistral-7B from 28.3% to 76.7%, with Mistral-7B-led debates reaching consensus in 98.3% of runs and averaging 22.5 seconds; the same protocol drops Gemma2-9B from 80.0% to 48.3%. The authors read this as evidence that debate is a targeted enhancement tool for weaker agents, not a universal improvement.","pith_inferences":["A testable implication beyond the paper is that the debate gain comes mostly from detecting that ambiguity exists, not from generating better clarifying questions, since the protocol forces the leader to acknowledge disagreement before refining a question.","The consensus rate could be reused as a confidence signal: instructions that drive persistent disagreement are the ones most likely to be genuinely ambiguous, so a system could trigger debate only for those cases and save latency on the rest.","The Gemma2-9B degradation suggests a ceiling effect: when the leading model is already right, follower dissent may be noise; a natural extension would let a high-confidence leader override followers instead of always revising."],"forward_implications":["For small open-weights models with low solo accuracy, structured debate can bring ambiguity detection close to or beyond a stronger single model: Mistral-7B-led debate reaches 76.7%, near Gemma2-9B's solo 80.0%.","Debate is not universally beneficial; Gemma2-9B loses accuracy when forced into the protocol, so deployment should be selective and model-aware.","Consensus dynamics carry useful signal: Mistral-7B-led debates reach consensus 98.3% of the time, mostly by round 2, and the paper reports that consensus outcomes are high-quality with 95% success when Mistral-7B leads.","The added latency of roughly 22.5 to 28.1 seconds is a real cost for human-robot interaction, and the paper itself points to adaptive triggers that invoke debate only when needed.","The debate outcome is a direct corollary of the claim that collaborative scrutiny beats individual reasoning for weaker models, which is exactly what a real-time system would rely on."],"supporting_citations":[{"why":"Supplies the spatial-ambiguity phenomena, such as frames of reference like 'to the right of the chair', that motivate the spatial category in the dataset.","marker":"[5]"},{"why":"Provides an automated pipeline for ambiguous image-task datasets and clarification query generation, a comparison target for the debate approach.","marker":"[8]"},{"why":"Argues multi-LLM collaboration and consensus improves reasoning and ambiguity resolution, the conceptual basis for the debate framework.","marker":"[16]"},{"why":"Shows multi-agent debate with assigned initial stances can uncover ambiguity in summary evaluation, the direct methodological antecedent.","marker":"[17]"},{"why":"Contributes the hierarchical leader-follower structure that the debate protocol adapts for pre-execution command analysis.","marker":"[19]"},{"why":"Prior work by the same group on LLM-based ambiguity detection in instructions, establishing the task formulation and prompting style.","marker":"[18]"}],"fun_headline_variants":["Debate raises Mistral ambiguity success from 28% to 77%","LLM debate helps weaker models, hurts strongest","Debate lifts small LLM ambiguity detection, with caveats","Mistral-led debate improves ambiguity detection to 76.7%","Debate boosts weaker LLMs but degrades Gemma's clarity"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central result depends on an unstated scoring rule: the paper never specifies whether a successful trial is one that flags any ambiguity, one whose clarifying question matches a gold-standard reference, or one that correctly declares a clean instruction clear, and the comparison between 76.7% and 28.3% changes meaning with that choice.","fun_headline_variants_meta":{"raw":{"variants":["Debate raises Mistral ambiguity success from 28% to 77%","LLM debate helps weaker models, hurts strongest","Debate lifts small LLM ambiguity detection, with caveats","Mistral-led debate improves ambiguity detection to 76.7%","Debate boosts weaker LLMs but degrades Gemma's clarity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":2993,"prompt_tokens":926,"completion_tokens":2067,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":1978}},"tokens_in":542,"tokens_out":2067,"duration_ms":15147,"temperature":1.0,"reasoning_tokens":1978,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:47:01.223907+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three models on the same dataset with a pre-specified scoring rule that requires the final clarifying question to exactly match the gold-standard question for each ambiguous instruction and requires a 'clear' verdict on the unambiguous counterparts; if Mistral-7B-led debate no longer beats its single-agent baseline under that rule, the paper's central comparison collapses.","supporting_citations":[{"cited_title":"Ambiguities in Spatial Language Understanding in Situated Human Robot Dialogue","cited_arxiv_id":null,"evidence_quote":"Supplies the spatial-ambiguity phenomena, such as frames of reference like 'to the right of the chair', that motivate the spatial category in the dataset."},{"cited_title":"Robotic Task Ambiguity Resolution via Natural Language Interaction","cited_arxiv_id":"2504.17748","evidence_quote":"Provides an automated pipeline for ambiguous image-task datasets and clarification query generation, a comparison target for the debate approach."},{"cited_title":"Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation","cited_arxiv_id":"2502.08514","evidence_quote":"Shows multi-agent debate with assigned initial stances can uncover ambiguity in summary evaluation, the direct methodological antecedent."},{"cited_title":"LLM-based ambiguity detection in natural language instructions for collaborative surgical robots","cited_arxiv_id":null,"evidence_quote":"Prior work by the same group on LLM-based ambiguity detection in instructions, establishing the task formulation and prompting style."}],"review_version":1}