{"id":"51798575-716d-4168-88ca-1864a30e5c49","arxiv_id":"2606.29270","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Minority Sentinel uses a LightGBM model on debate fingerprints to overturn majority votes in LLM debates with 81.2% flip precision and positive net gain on six benchmarks.","lead":"The paper identifies that majority voting in multi-agent LLM debates often overlooks correct answers from minority agents because of correlated errors. It introduces Minority Sentinel, a LightGBM meta-classifier that uses debate logs to decide when to overturn the majority vote, showing positive net gains in accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Training/eval on same six benchmarks risks benchmark-specific overfitting rather than general debate signals","rationale":"Reader's weakest_assumption exactly identifies the same generalization risk. The full-text reference does not alter this because the abstract already flags the same-benchmark training/eval setup; the proposed cross-benchmark check is the minimal concrete test that would either confirm or refute the central claim. Original UNVERDICTED/low-confidence stance should be retained until the check is performed.","tokens_in":1735,"tokens_out":340,"duration_ms":24704,"concrete_test":"Re-run the LightGBM pipeline with strict leave-one-benchmark-out: for each of the six datasets, train exclusively on the other five, then measure flip precision and net gain on the held-out dataset. If average flip precision falls below 70% or net gain turns negative on any held-out set, the generalization assumption fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline performance (81.2% flip precision, positive net gain on all datasets and seeds) is presented as evidence that debate logs contain transferable behavioral signals recoverable by LightGBM. This requires that the multi-dimensional fingerprint and classifier generalize beyond the training distribution. The abstract and reader's note indicate training and evaluation occur on the same set of benchmarks; without explicit leave-one-benchmark-out or external-task testing, the reported metrics could arise from dataset-specific correlations (e.g., task format, agent response styles on those particular problems) rather than robust, domain-agnostic signals. This directly undercuts the claim that the method \"reliably recover[s] suppressed minorities\" in general MAD settings.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper identifies that majority voting in three-agent LLM debates systematically suppresses correct minority answers (Minority Truth) in roughly 25% of divergent cases across six benchmarks, creating a 10pp recovery margin. It introduces Minority Sentinel, a LightGBM meta-classifier trained on multi-dimensional debate fingerprints extracted from logs, which decides when to flip the majority vote. The method reports 81.2% flip precision and positive net gain on all datasets and all 20 random seeds, while an LLM-as-judge baseline yields negative net gain despite higher recall.","tokens_in":1870,"tokens_out":514,"duration_ms":27185,"significance":"If the generalization claim holds, the work supplies concrete evidence that lightweight, non-LLM classifiers can recover suppressed correct answers from behavioral signals in debate logs, improving MAD accuracy without extra LLM inference cost. The consistent positive net gain across seeds and the direct comparison to LLM-as-judge baselines are notable strengths; the approach is falsifiable via the reported flip-precision and net-gain metrics.","major_comments":[{"comment":"Experimental section (implicit in abstract and results): training and test splits are performed within the same six benchmarks without reported leave-one-benchmark-out or external-task validation. This setup risks the LightGBM model capturing benchmark-specific response patterns or task formats rather than domain-agnostic debate signals, which directly weakens the claim that the fingerprint enables reliable recovery 'in general MAD settings.'","section":"Experimental Evaluation / Results"},{"comment":"§ on Net Gain calculation: the definition of net gain and the precise weighting of false-positive flips versus true-positive recoveries are not fully specified, making it impossible to verify that the reported positive net gain is robust to alternative cost assumptions or to confirm it is not an artifact of post-hoc threshold selection on the same data.","section":"Results / Net Gain"}],"minor_comments":[{"comment":"The multi-dimensional fingerprint features are described at a high level; an explicit list or table of the  features used (e.g., token entropy, agreement ratios, response length) would improve reproducibility.","section":"Method"},{"comment":"Figure captions and axis labels for the flip-precision and net-gain plots should include the exact number of trials (20 seeds) and confidence intervals to allow readers to assess stability.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. We address each major comment below and outline the revisions we will make to strengthen the manuscript.","responses":[{"response":"We agree that the current within-benchmark splits limit the strength of the generalization claim. In the revised version we will add leave-one-benchmark-out experiments (training on five benchmarks and evaluating on the held-out benchmark) together with a summary of performance variance across folds. These additional results will be reported in a new subsection of the experimental evaluation.","revision_made":"yes","referee_comment":"Experimental section (implicit in abstract and results): training and test splits are performed within the same six benchmarks without reported leave-one-benchmark-out or external-task validation. This setup risks the LightGBM model capturing benchmark-specific response patterns or task formats rather than domain-agnostic debate signals, which directly weakens the claim that the fingerprint enables reliable recovery 'in general MAD settings.'"},{"response":"We accept that the net-gain definition and weighting require explicit formalization. The revised manuscript will include the exact formula (net gain = TP recoveries \times benefit − FP flips \times cost) with the default 1:1 cost ratio, a sensitivity table for alternative ratios (1:2 and 2:1), and confirmation that the positive net gain remains stable under these weightings. We will also state that the threshold was selected via cross-validation on the training folds only.","revision_made":"yes","referee_comment":"§ on Net Gain calculation: the definition of net gain and the precise weighting of false-positive flips versus true-positive recoveries are not fully specified, making it impossible to verify that the reported positive net gain is robust to alternative cost assumptions or to confirm it is not an artifact of post-hoc threshold selection on the same data."}],"tokens_in":1376,"tokens_out":402,"duration_ms":32800,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that a LightGBM model trained on debate fingerprints can flip majority votes to the minority opinion in multi-agent LLM debates at 81% precision, delivering positive net accuracy gains across the six benchmarks and all seeds tested.\n\nWhat the paper does is introduce this meta-classifier as an alternative to pure majority voting or using an LLM judge. They document that minorities hold the correct answer in roughly one in four disagreement cases, then show their approach recovers many of those flips safely. The results are consistent, and it outperforms the LLM-as-judge baseline on net gain because it prioritizes precision over recall.\n\nThe execution looks reasonable for the reported experiments. Using heterogeneous agents and multiple random seeds adds some robustness to the findings.\n\nThe soft spot is the evaluation setup. Training and testing on the same six benchmarks means the classifier might be capturing dataset-specific correlations in how the agents respond to those particular problems rather than general signals in any debate. Without cross-benchmark validation or tests on unseen tasks, the claim that debate logs contain reliable, transferable behavioral signals for recovering minorities is not fully backed up. That is the main limitation.\n\nThis work is for researchers focused on multi-agent debate systems and ways to make them more accurate. A reader interested in practical fixes for correlated errors in LLMs would get value from the method and the numbers. It deserves a serious referee because it has clear empirical support and addresses a real issue in the MAD paradigm.\n\nI would recommend sending it to peer review.","headline":"A LightGBM meta-classifier on debate logs can flip majority votes with 81% precision and net gain, but same-benchmark training leaves generalization unproven.","tokens_in":2358,"tokens_out":384,"would_cite":false,"duration_ms":44817,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A lightweight classifier can overturn majority votes in LLM debates when the minority answer is correct by reading behavioral signals in the logs.","keywords":["multi-agent debate","majority voting","minority truth","lightgbm classifier","flip precision","llm reasoning","debate logs","condorcet jury theorem"],"falsifier":"Applying the trained Minority Sentinel to a fresh set of benchmarks or different LLMs and observing flip precision below 60 percent or negative net gain would show that the signals are not sufficient or generalizable.","tokens_in":2634,"feed_emoji":"🗳️","tokens_out":701,"duration_ms":36790,"temperature":0.7,"pith_summary":"Multi-agent debate systems use majority voting to boost LLM reasoning under the assumption that agent errors are independent, yet shared pretraining data makes errors correlated so the majority often buries the correct minority view. Experiments with three heterogeneous LLMs across six benchmarks show that the minority holds the right answer in roughly one quarter of divergent cases, leaving a 10-percentage-point recovery margin. Minority Sentinel builds a multi-dimensional fingerprint from the debate logs and trains a LightGBM model to decide when to flip the majority decision. The model reaches 81.2 percent flip precision and positive net gain on every dataset and every random seed, while an LLM-as-Judge baseline produces negative net gain. The result demonstrates that the logs already contain enough behavioral information for a non-LLM classifier to intervene safely.","feed_headline":"LightGBM recovers minority answers in LLM debates at 81% precision","feed_subtitle":"Debate logs contain behavioral signals that let a lightweight model overturn wrong majorities safely across all tested datasets.","key_machinery":"The multi-dimensional debate fingerprint extracted from debate logs, which LightGBM uses to predict when overturning the majority vote will recover a correct minority answer.","core_discovery":"The paper claims that debate logs from three heterogeneous LLM agents contain sufficient behavioral signals for a LightGBM classifier trained on multi-dimensional debate fingerprints to identify cases where the minority answer is correct, achieving a stable 81.2 percent flip precision and positive net gain across all six benchmarks and all 20 random seed trials while avoiding the accuracy degradation seen with LLM-as-Judge baselines.","pith_inferences":["The same fingerprint approach could be tested on debates involving more than three agents to check whether disagreement patterns remain informative.","Explicit logging of agent disagreement trajectories might eventually replace simple majority voting in multi-agent systems.","Extending the method to open-ended or long-form tasks would test whether the behavioral signals generalize beyond the six benchmarks used here."],"forward_implications":["Selective overturns based on the classifier improve overall system accuracy without changing the base LLMs or adding more agents.","Behavioral signals in the logs support safer flips than asking another LLM to judge the debate.","The positive net gain holds across all tested datasets and random seeds, indicating stability of the signals.","Roughly one in four divergent cases offers a recoverable minority truth that majority voting otherwise suppresses."],"fun_headline_variants":["LightGBM flags correct minority answers in LLM debates at 81% precision","LightGBM uses debate fingerprints to flip LLM votes at 81% precision","Multi-dimensional debate logs let LightGBM override majorities at 81% precision","LightGBM classifier achieves stable 81% precision on minority flips"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The behavioral signals recorded in the debate logs are consistent enough that a LightGBM model trained on the six benchmarks will continue to produce high flip precision and positive net gain on new data.","fun_headline_variants_meta":{"raw":{"variants":["LightGBM flags correct minority answers in LLM debates at 81% precision","LightGBM uses debate fingerprints to flip LLM votes at 81% precision","Multi-dimensional debate logs let LightGBM override majorities at 81% precision","LightGBM classifier achieves stable 81% precision on minority flips"]},"model":"grok-4.3","cost_usd":0.007457,"raw_usage":{"total_tokens":3414,"prompt_tokens":647,"num_sources_used":0,"completion_tokens":76,"cost_in_usd_ticks":74574500,"prompt_tokens_details":{"text_tokens":647,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2691,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":647,"tokens_out":76,"duration_ms":35431,"temperature":1.0,"reasoning_tokens":2691,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T02:21:49.652427+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Applying the trained Minority Sentinel to a fresh set of benchmarks or different LLMs and observing flip precision below 60 percent or negative net gain would show that the signals are not sufficient or generalizable.","supporting_citations":[],"review_version":1}