{"id":"82fbabb6-9b74-4553-bc37-536293360123","arxiv_id":"2607.19899","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Correlated agreement blindness: stronger base learners agree more and fail together, so disagreement-based escalation misses up to 90.6% of dangerous under-predictions; ARAT's conservative override and safety flag reduce under-prediction from 4.80% to 1.70% on UNSW-NB15.","lead":"This paper finds that multi-agent systems which use disagreement between models to trigger human review become blind exactly when the models fail together: 90.6% of dangerous under-predictions still slip through. It proposes ARAT, a simple routing layer that cuts under-prediction from 4.80% to 1.70% on a network-intrusion benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 90.6% 'blindness' statistic is computed after the conservative override has already removed most disagreement-based under-predictions; reconstructing the pre-override numbers suggests only ~41% of dangerous under-predictions originally occurred under agreement.","rationale":"The reader identified the two-configuration ablation as the weakest assumption, which concerns the extrapolation that blindness intensifies with base-learner strength. That is a valid concern about generalization, but it is secondary to the concern raised here: the primary quantitative evidence for blindness itself—the 90.6% statistic—appears to be an artifact of conditioning on post-override residuals. Even if the ablation were perfectly clean, the central claim would still rest on a misleading denominator. The pre-override reconstruction from the paper's own numbers implies that disagreement-based escalation would have flagged ~59% of dangerous under-predictions before the override, which contradicts the abstract's assertion that 90.6% evade disagreement-based monitoring. This is more load-bearing because it concerns the headline result, not an auxiliary trend. The appropriate response is to require the authors to report the pre-override contingency table and to soften the 'majority blind' language if the pre-override share is not a majority. Since the reader's verdict was already CONDITIONAL, this concern does not change the overall verdict; it strengthens the condition and redirects it to the most central statistic.","tokens_in":9975,"tokens_out":9099,"duration_ms":92393,"concrete_test":"Using the released code, reproduce ARAT v2 on the 82,332 UNSW-NB15 holdout. Before applying the conservative-override rule, count all true under-predictions (predicted severity < true severity) and split them by whether the two base agents agreed or disagreed. Then apply the override and recompute the same split. If the pre-override under-agreement fraction is ≈41% (or below 50%), the paper's 'majority of dangerous under-predictions evade disagreement-based monitoring' claim fails; report the pre-override contingency table and reframe the abstract accordingly. This is a small diagnostic addition to the existing pipeline, not a reimplementation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline quantitative claim—90.6% of dangerous under-predictions evade disagreement-based monitoring—is conditional on the conservative override having already done its work. From §4 and §6.3: the override eliminates 92.8% of disagreement-based under-predictions, and Table 2 reports 1,625 of 1,793 final under-predictions occur under agreement. Let D be the pre-override count of disagreement-based under-predictions. The 168 residual disagreement under-predictions (1,793 − 1,625) are 7.2% of D, so D ≈ 168/0.072 ≈ 2,333. Pre-override total under-predictions are therefore ≈ 1,625 + 2,333 = 3,958, and the pre-override share occurring under agreement is 1,625/3,958 ≈ 41%, not 90.6%. In other words, disagreement-triggered escalation would have flagged roughly 59% of dangerous under-predictions before the override was applied; the 90.6% figure is a mechanical artifact of conditioning on the residual after most disagreement under-predictions have been corrected. The abstract's phrase '90.6% of dangerous under-predictions evade disagreement-based monitoring' is thus at best ambiguous and at worst reversed: the override is what catches the disagreement cases, and the residual blind spot among agreed cases is a minority of the original dangerous under-predictions. The same conditioning likely inflates the 57.2% errors-under-agreement figure, which is also reported on final ARAT outputs. This directly undermines the paper's central claim that disagreement-based escalation is 'structurally blind to the majority of dangerous failures.'","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces correlated agreement blindness and argues that disagreement-triggered escalation is structurally blind to correlated failures in multi-agent triage. It proposes ARAT, a directed-star routing system with a random-forest agent, a k-NN agent, and a central routing agent that applies soft-disagreement scoring, a conservative severity override, and a safety flag. On 82,332 UNSW-NB15 holdout samples, the paper reports 3.53x joint error inflation, phi = 0.612, 57.2% of errors under agreement, 90.6% of dangerous under-predictions under agreement after conservative override, and an ARAT under-prediction reduction from 4.80% to 1.70% relative to soft voting. Cross-dataset replication on UCI Diabetes and an SVM substitution experiment are presented as supporting evidence. The central thesis is that disagreement signals degrade as base learners improve and correlate, and that ARAT's layered routing recovers part of the resulting safety gap.","tokens_in":1681,"tokens_out":3218,"duration_ms":131755,"significance":"If the empirical characterization holds, the paper provides a useful and practically relevant caution about disagreement-based safety monitoring in multi-agent systems. The paper has strengths: the 3.53x error-inflation statistic and phi coefficient are clearly derived with BCa confidence intervals; the per-layer decomposition is transparent; code and pinned results are public; and the limitation section is candid about the perfect-analyst assumption and scope. The phenomenon is measured on the base agents and is not circularly defined by the architecture. However, the headline blindness statistic is substantially overstated by conditioning on the post-override residual, and the intensification claim rests on a confounded two-point ablation. The contribution would remain significant in the weaker form that a large minority of dangerous under-predictions occur under agreement and disagreement-based escalation is not complete under correlated error.","major_comments":[{"comment":"The headline statistic 90.6% of dangerous under-predictions evade disagreement-based monitoring is an artifact of conditioning on the post-override residual. Section 6.3 states that the conservative override eliminates 92.8% of disagreement-based under-predictions. Table 2 reports 1,625 of 1,793 final under-predictions under agreement, so the residual disagreement under-predictions are 168, which is 7.2% of the pre-override disagreement under-predictions D. Thus D equals about 2,333 and the pre-override total is about 3,958. The pre-override share of dangerous under-predictions under agreement is about 41%, not 90.6%. Equivalently, disagreement-triggered escalation would have flagged or corrected roughly 59% of dangerous under-predictions before the override. Section 7.1's statement that routing can theoretically recover at most 9.4% is the same conditioning artifact. Please report pre-o","section":"Section 4, Section 6.3, Table 2, Abstract"},{"comment":"The claim that agreement blindness intensifies as base learner strength increases is supported only by a two-point ablation per dataset with multiple variables changing simultaneously. On UNSW-NB15, v1-to-v2 changes RF trees from 200 to 500 and features from 37 to 30; on Diabetes, trees change from 100 to 500 with other hyperparameter changes. No confidence intervals are given for the deltas, and the movements are small: on UNSW-NB15, phi changes by +0.028, disagreement by -1.56pp, and errors-under-agreement by +1.73pp; on Diabetes the changes are even smaller. A two-point comparison cannot establish monotonicity, and the extrapolation in Section 7.2 to increasingly capable, correlated agentic pipelines goes beyond the evidence. Please add at least one intermediate configuration and uncertainty intervals for the deltas, or explicitly weaken the conclusion to what this configuration pair","section":"Section 6.5, Table 3, Section 7.2"},{"comment":"The conclusion that ARAT gains are not reproduced by loss-function tuning alone is not fully supported. The text says cost-sensitive LightGBM variants with alpha in {2,3,5,50} were evaluated, but Table 1 reports only the alpha=3 case. Without the full sweep, or the best configuration by under-prediction, the reader cannot assess whether the comparison was selective or whether a larger alpha would close the gap. Please report the complete alpha sweep, state the selection criterion, or limit the claim to the alpha=3 comparison.","section":"Section 5, Table 1, Contribution 2"}],"minor_comments":[{"comment":"The limitation section is candid, but the abstract and first paragraph should carry the perfect-analyst caveat prominently. The conditioning issue described in the major comments should also be added to this section.","section":"Section 8"},{"comment":"The labels ARAT v2 (100%) and ARAT v2 (93.21%) are unclear. Suggest relabeling to ARAT v2 with perfect analyst resolution of escalated cases and ARAT v2 automated-only, conditional on non-escalated cases.","section":"Table 1"},{"comment":"Please clarify whether the MI feature-count sweep and entropy threshold theta were selected using only the training split. If the holdout was used for selection, state this and discuss its effect on the reported intervals.","section":"Section 5 Features"},{"comment":"The escalation meta-model AUROC of 0.924 is reported without specifying whether it was obtained by cross-validation or on the holdout. Since the model is fit on routing-time features and used for queue ordering, please state the evaluation protocol explicitly.","section":"Section 6.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to interest the PAAMS community, but the abstract's 90.6% claim is likely to be repeated and should be corrected before publication. The pre-override reconstruction is straightforward and should be reported in the main text. The intensification claim needs either a denser ablation or a more modest conclusion. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a useful core but the headline statistic is misleading. The 90.6% figure for dangerous under-predictions under agreement is computed after ARAT's conservative override has already removed 92.8% of disagreement-based under-predictions. The stress-test reconstruction checks out: the residual 168 disagreement under-predictions imply roughly 2,333 pre-override disagreement under-predictions, so pre-override only about 41% of under-predictions were under agreement. Disagreement would have flagged 59% of them. That's a very different story than 'structurally blind to the majority of dangerous failures.'\n\nWhat's genuinely good: the joint error inflation (3.53x, phi=0.612) between RF and kNN on 82k holdout samples is a clean, reproducible result, and the code/data are public. The per-layer decomposition of ARAT (override -2.6pp, flag -0.5pp) is transparent and honest about the analyst assumption. The Diabetes replication and SVM substitution showing productive vs. non-productive diversification is a nice boundary-condition experiment.\n\nSoft spots besides the conditioning artifact: the 'intensifies as base learners improve' claim rests on a two-configuration ablation where both tree count and feature count change at once; no third point or statistical test supports monotonicity. Table 1's headline rows don't fully reconcile with perfect analyst resolution (1.75% on 93.21% and zero on 6.79% would give ~1.63% overall, not 1.70%). The entropy threshold and feature count are selected on the same data without nested validation. And the LLM agentic extrapolation is speculative, though flagged as such.\n\nBottom line: this is a solid empirical study of correlated errors in a triage setting, but the central claim needs reframing. A serious referee should ask for pre-override statistics and a more careful use of 'blindness.' With that revision, the paper is a useful contribution. I'd send it to review, not desk reject.\n\nBest.","headline":"The paper has a useful core finding about correlated errors in disagreement-gated triage, but its headline '90.6% blindness' statistic is an artifact of conditioning on post-override residuals and overstates the failure of disagreement-based monitoring.","tokens_in":10894,"tokens_out":4513,"would_cite":true,"duration_ms":46593,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When AI agents agree, 90.6% of dangerous misses evade review.","keywords":["multi-agent systems","disagreement uncertainty","correlated errors","safety-critical triage","network intrusion detection","ensemble arbitration","human-in-the-loop","uncertainty estimation"],"falsifier":"Train the same two-agent system across a continuum of random-forest capacities—for example 50 to 2,000 trees and several feature-set sizes—on the same holdout and plot error correlation and disagreement rate against accuracy. If any accuracy improvement drives error correlation down or disagreement up, meaning more productive disagreement, the paper's extrapolation that stronger agents always deepen correlated agreement blindness fails.","tokens_in":9867,"feed_emoji":"⚠️","tokens_out":5402,"duration_ms":56440,"temperature":0.7,"pith_summary":"This paper argues that using disagreement among AI agents to decide when a human must review a prediction is structurally unsafe: when agents share correlated errors, they often agree on the wrong answer, so the trigger never fires. On 82,332 holdout samples from a network intrusion benchmark, 57.2% of all errors occurred under agreement and 90.6% of dangerous under-predictions escaped disagreement-based monitoring even after a conservative override. The authors introduce ARAT, a three-layer routing system whose conservative override and safety-flag gate reduce under-prediction from 4.80% to 1.70%. A cross-dataset replication on a clinical readmission benchmark and a model-substitution experiment support the mechanism. If the claim holds, any multi-agent system that gates human review on disagreement inherits a blind spot that grows as the agents become more capable.","feed_headline":"Agreeing AI agents hide 90% of dangerous misses","feed_subtitle":"Most dangerous misses happen when agents agree; layered routing shrinks the gap but not the blind spot.","key_machinery":"ARAT is a directed-star arbitration system whose central routing agent computes a soft-disagreement score c = 0.5·H(p_merged) + 0.5·d from the two base agents' probability vectors, then applies two deterministic safety layers: a conservative severity override whenever the agents disagree, and a unanimous-Normal safety flag when both agents predict the lowest-risk class with entropy above a threshold. The safety flag is designed to catch exactly the high-entropy consensus case where disagreement-triggered monitoring is structurally blind. A calibrated logistic-regression escalation model (AUROC 0.924) is used only to rank the analyst review queue.","core_discovery":"The central discovery is 'correlated agreement blindness': in a two-agent triage system (random forest plus k-nearest neighbour), the agents fail together far more often than independence would predict—3.53 times the expected joint error rate, with a phi correlation of 0.612—and the disagreement signal is blind to those failures. After the conservative override, 90.6% of dangerous under-predictions still occur when both agents agree. A matched ablation shows that making the random forest stronger increases error correlation and decreases disagreement, so the blind spot is not a fixed bug but worsens with model improvement.","pith_inferences":["If this generalizes to large language models, agentic systems built from models sharing pretraining or fine-tuning data may show even stronger correlated agreement blindness, since inductive biases converge; this is testable by measuring agreement rates across same-base versus different-base model pairs.","The paper measures the effect with simple tabular learners; a natural extension is to check whether deep classifiers or conformal-prediction uncertainty inherit the same blind spot or avoid it.","A remedy the paper leaves implicit is to route on the calibrated escalation score rather than raw disagreement, since within the unanimous-Normal subset that score ranks dangerous cases with AUROC 0.933.","The safety-flag threshold sets a review budget; in practice the threshold could be tuned against the cost of review versus the cost of a missed dangerous case, which the paper does not explore."],"forward_implications":["Disagreement rate should not be used as a proxy for uncertainty in safety-critical multi-agent triage without first measuring error correlation between agents.","Strengthening base learners can make a system less safe overall, because accuracy gains suppress the disagreement signal that would otherwise trigger human review.","A conservative override and a consensus-on-low-risk safety flag recover a large share of the gap: under-prediction drops from 4.80% to 1.70% on the intrusion benchmark.","Diversifying agent types helps only if the new agent produces productively different errors; the SVM substitution showed it can either reduce under-prediction by 0.64 points or worsen it by 3.1 points depending on the dataset.","A pre-deployment diagnostic based on error correlation and productive-disagreement rate should accompany any disagreement-gated multi-agent system."],"fun_headline_variants":["AI agents that agree hide 90% of dangerous misses","When AI agents agree, 90% of dangerous misses evade safety","Correlated AI errors blind disagreement checks: 90% of misses slip","Improving AI agents worsens their blind spot: 90% of misses","Agreeing AI agents miss 90% of dangerous errors silently"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the assumption that making a base agent more accurate also makes its errors more correlated with the other agent, so that agreement becomes both more common and more misleading; the paper's support for this is a two-configuration comparison, not a measured trend.","fun_headline_variants_meta":{"raw":{"variants":["AI agents that agree hide 90% of dangerous misses","When AI agents agree, 90% of dangerous misses evade safety","Correlated AI errors blind disagreement checks: 90% of misses slip","Improving AI agents worsens their blind spot: 90% of misses","Agreeing AI agents miss 90% of dangerous errors silently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000855,"raw_usage":{"total_tokens":3559,"prompt_tokens":757,"completion_tokens":2802,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2710}},"tokens_in":501,"tokens_out":2802,"duration_ms":19350,"temperature":1.0,"reasoning_tokens":2710,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:22:01.307466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same two-agent system across a continuum of random-forest capacities—for example 50 to 2,000 trees and several feature-set sizes—on the same holdout and plot error correlation and disagreement rate against accuracy. If any accuracy improvement drives error correlation down or disagreement up, meaning more productive disagreement, the paper's extrapolation that stronger agents always deepen correlated agreement blindness fails.","supporting_citations":[],"review_version":1}