REVIEW 3 major objections 4 minor 25 references
Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triage
T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read When AI agents agree, 90.6% of dangerous misses evade review.
desk verdict The paper has a useful core finding about correlated errors in disagreement-gated triage, but its headline '90.6% blindness' statistic is an artifact of conditioning on post-override residuals and overstates the failure of disagreement-based monitoring. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
ARAT is a directed-star arbitration system whose central routing agent computes a soft-disagreement score c = 0.5·H(p_merged) + 0.5·d from the two base agents' probability vectors, then applies two deterministic safety layers: a conservative severity override whenever the agents disagree, and a unanimous-Normal safety flag when both agents predict the lowest-risk class with entropy above a threshold. The safety flag is designed to catch exactly the high-entropy consensus case where disagreement-triggered monitoring is structurally blind. A calibrated logistic-regression escalation model (AUROC 0.924) is used only to rank the analyst review queue.
What would settle it
Train the same two-agent system across a continuum of random-forest capacities—for example 50 to 2,000 trees and several feature-set sizes—on the same holdout and plot error correlation and disagreement rate against accuracy. If any accuracy improvement drives error correlation down or disagreement up, meaning more productive disagreement, the paper's extrapolation that stronger agents always deepen correlated agreement blindness fails.
Extended reading notes
Core claim
The central discovery is 'correlated agreement blindness': in a two-agent triage system (random forest plus k-nearest neighbour), the agents fail together far more often than independence would predict—3.53 times the expected joint error rate, with a phi correlation of 0.612—and the disagreement signal is blind to those failures. After the conservative override, 90.6% of dangerous under-predictions still occur when both agents agree. A matched ablation shows that making the random forest stronger increases error correlation and decreases disagreement, so the blind spot is not a fixed bug but worsens with model improvement.
Load-bearing premise
The central claim rests on the assumption that making a base agent more accurate also makes its errors more correlated with the other agent, so that agreement becomes both more common and more misleading; the paper's support for this is a two-configuration comparison, not a measured trend.
Editorial extensions
If this is right
- Disagreement rate should not be used as a proxy for uncertainty in safety-critical multi-agent triage without first measuring error correlation between agents.
- Strengthening base learners can make a system less safe overall, because accuracy gains suppress the disagreement signal that would otherwise trigger human review.
- A conservative override and a consensus-on-low-risk safety flag recover a large share of the gap: under-prediction drops from 4.80% to 1.70% on the intrusion benchmark.
- Diversifying agent types helps only if the new agent produces productively different errors; the SVM substitution showed it can either reduce under-prediction by 0.64 points or worsen it by 3.1 points depending on the dataset.
- A pre-deployment diagnostic based on error correlation and productive-disagreement rate should accompany any disagreement-gated multi-agent system.
Reading between the lines
- If this generalizes to large language models, agentic systems built from models sharing pretraining or fine-tuning data may show even stronger correlated agreement blindness, since inductive biases converge; this is testable by measuring agreement rates across same-base versus different-base model pairs.
- The paper measures the effect with simple tabular learners; a natural extension is to check whether deep classifiers or conformal-prediction uncertainty inherit the same blind spot or avoid it.
- A remedy the paper leaves implicit is to route on the calibrated escalation score rather than raw disagreement, since within the unanimous-Normal subset that score ranks dangerous cases with AUROC 0.933.
- The safety-flag threshold sets a review budget; in practice the threshold could be tuned against the cost of review versus the cost of a missed dangerous case, which the paper does not explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces correlated agreement blindness and argues that disagreement-triggered escalation is structurally blind to correlated failures in multi-agent triage. It proposes ARAT, a directed-star routing system with a random-forest agent, a k-NN agent, and a central routing agent that applies soft-disagreement scoring, a conservative severity override, and a safety flag. On 82,332 UNSW-NB15 holdout samples, the paper reports 3.53x joint error inflation, phi = 0.612, 57.2% of errors under agreement, 90.6% of dangerous under-predictions under agreement after conservative override, and an ARAT under-prediction reduction from 4.80% to 1.70% relative to soft voting. Cross-dataset replication on UCI Diabetes and an SVM substitution experiment are presented as supporting evidence. The central thesis is that disagreement signals degrade as base learners improve and correlate, and that ARAT's layered routing recovers part of the resulting safety gap.
Significance. If the empirical characterization holds, the paper provides a useful and practically relevant caution about disagreement-based safety monitoring in multi-agent systems. The paper has strengths: the 3.53x error-inflation statistic and phi coefficient are clearly derived with BCa confidence intervals; the per-layer decomposition is transparent; code and pinned results are public; and the limitation section is candid about the perfect-analyst assumption and scope. The phenomenon is measured on the base agents and is not circularly defined by the architecture. However, the headline blindness statistic is substantially overstated by conditioning on the post-override residual, and the intensification claim rests on a confounded two-point ablation. The contribution would remain significant in the weaker form that a large minority of dangerous under-predictions occur under agreement and disagreement-based escalation is not complete under correlated error.
major comments (3)
- [Section 4, Section 6.3, Table 2, Abstract] The headline statistic 90.6% of dangerous under-predictions evade disagreement-based monitoring is an artifact of conditioning on the post-override residual. Section 6.3 states that the conservative override eliminates 92.8% of disagreement-based under-predictions. Table 2 reports 1,625 of 1,793 final under-predictions under agreement, so the residual disagreement under-predictions are 168, which is 7.2% of the pre-override disagreement under-predictions D. Thus D equals about 2,333 and the pre-override total is about 3,958. The pre-override share of dangerous under-predictions under agreement is about 41%, not 90.6%. Equivalently, disagreement-triggered escalation would have flagged or corrected roughly 59% of dangerous under-predictions before the override. Section 7.1's statement that routing can theoretically recover at most 9.4% is the same conditioning artifact. Please report pre-o
- [Section 6.5, Table 3, Section 7.2] The claim that agreement blindness intensifies as base learner strength increases is supported only by a two-point ablation per dataset with multiple variables changing simultaneously. On UNSW-NB15, v1-to-v2 changes RF trees from 200 to 500 and features from 37 to 30; on Diabetes, trees change from 100 to 500 with other hyperparameter changes. No confidence intervals are given for the deltas, and the movements are small: on UNSW-NB15, phi changes by +0.028, disagreement by -1.56pp, and errors-under-agreement by +1.73pp; on Diabetes the changes are even smaller. A two-point comparison cannot establish monotonicity, and the extrapolation in Section 7.2 to increasingly capable, correlated agentic pipelines goes beyond the evidence. Please add at least one intermediate configuration and uncertainty intervals for the deltas, or explicitly weaken the conclusion to what this configuration pair
- [Section 5, Table 1, Contribution 2] The conclusion that ARAT gains are not reproduced by loss-function tuning alone is not fully supported. The text says cost-sensitive LightGBM variants with alpha in {2,3,5,50} were evaluated, but Table 1 reports only the alpha=3 case. Without the full sweep, or the best configuration by under-prediction, the reader cannot assess whether the comparison was selective or whether a larger alpha would close the gap. Please report the complete alpha sweep, state the selection criterion, or limit the claim to the alpha=3 comparison.
minor comments (4)
- [Section 8] The limitation section is candid, but the abstract and first paragraph should carry the perfect-analyst caveat prominently. The conditioning issue described in the major comments should also be added to this section.
- [Table 1] The labels ARAT v2 (100%) and ARAT v2 (93.21%) are unclear. Suggest relabeling to ARAT v2 with perfect analyst resolution of escalated cases and ARAT v2 automated-only, conditional on non-escalated cases.
- [Section 5 Features] Please clarify whether the MI feature-count sweep and entropy threshold theta were selected using only the training split. If the holdout was used for selection, state this and discuss its effect on the reported intervals.
- [Section 6.4] The escalation meta-model AUROC of 0.924 is reported without specifying whether it was obtained by cross-validation or on the holdout. Since the model is fit on routing-time features and used for queue ordering, please state the evaluation protocol explicitly.
Circularity Check
Headline blindness statistics are computed after the conservative override has removed most disagreement under-predictions, making the 90.6% (and 57.2%) figures artifacts of the routing design.
-
self definitional
[Abstract; Section 6.2 (Table 2); Section 6.3]
"Of all dangerous under-predictions, 90.6% occur under agent agreement after conservative override, confirming that disagreement-based monitoring is structurally blind to the majority of dangerous failures. ... The conservative override ... eliminating 92.8% of disagreement-based under-predictions by construction."
The 90.6% denominator is the post-override under-prediction set. The override itself is a disagreement-based mechanism that by construction chooses the more severe prediction on disagreement and eliminates 92.8% of disagreement under-predictions, so the residual set has had most disagreement cases removed. Reconstructing: the 168 residual disagreement under-predictions are 7.2% of the pre-override disagreement under-predictions D, so D ≈ 2,333; the pre-override share of dangerous under-predictions occurring under agreement is 1,625 / (1,625 + 2,333) ≈ 41%, not 90.6%. The same conditioning affects the 57.2% errors-under-agreement figure, which is computed on final ARAT outputs after routing has already corrected many disagreement errors. Thus the headline blindness statistic is generated by
-
self definitional
[Section 6.3]
"the safety flag captures 100% of the dangerous under-predictions that reside within the unanimous-Normal subpopulation, proving it serves as a principled final safety net where disagreement-based monitoring is structurally blind."
The safety flag is defined as a hard gate on exactly that subpopulation: both agents predict Normal and entropy > θ = 0.10. Any dangerous under-prediction inside the unanimous-Normal subpopulation is therefore flagged by definition; the reported 100% capture is a tautology and carries no independent evidential weight for the claim that it is a principled final safety net.
full rationale
ARAT has substantial independent content: the measured error correlation (phi = 0.612), joint-error inflation (3.53x), the v1/v2 mechanism indicators, cross-dataset replication on Diabetes, and the SVM substitution experiment do not reduce to the architecture's design choices, and the paper does not rely on self-citations or imported uniqueness theorems. However, the paper's headline blindness statistics are not independent measurements. The 90.6% figure is computed after the conservative override — itself a disagreement-triggered mechanism — has eliminated 92.8% of disagreement-based under-predictions by construction. Conditioning on the post-override residual makes the agreement share of remaining under-predictions 90.6%, whereas the pre-override share under agreement is approximately 41%, with the disagreement signal having flagged roughly 59% before the override. The 57.2% errors-under-agreement figure is likewise computed on final ARAT outputs after routing has already changed the error set. The safety-flag 100% capture is definitional. These are load-bearing, by-construction artifacts in the central quantitative claims, although the underlying correlated-error phenomenon retains independent support; hence a partial circularity score of 6.
Assumptions & free parameters
free parameters (6)
- Entropy threshold theta =
0.10
- Soft-disagreement mixing weights =
0.5/0.5 for entropy/disagreement
- MI feature count (UNSW-NB15) =
30
- k-NN neighborhood size =
5
- RF hyperparameters (UNSW-NB15 v2) =
500 trees, min leaf 2, balanced weights
- Diabetes RF configuration =
500 trees, depth 15, min leaf 5
assumptions (7)
- domain assumption Disagreement is a practical proxy for uncertainty in multi-agent systems
- domain assumption As base learners improve, they tend to converge (monotonicity premise)
- domain assumption In safety triage, under-prediction is dangerous and over-prediction acceptable
- standard math The expected joint error under independence P(A1 wrong)P(A2 wrong) is the correct null benchmark
- ad hoc to paper Feature-count sweep optimizing High-class F1 on the training data does not materially overfit
- ad hoc to paper The entropy threshold theta=0.10 reflects a realistic analyst review budget
- standard math BCa bootstrap intervals quantify sampling uncertainty correctly
invented entities (1)
-
Correlated agreement blindness
independent evidence
Cite this review
Pith. "Pith review of Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triage." pith.science (2026). https://pith.science/paper/VV4VIHBG
@misc{pith2026260719899,
author = {Pith},
title = {Pith review of: Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triage},
year = {2026},
howpublished = {\url{https://pith.science/paper/VV4VIHBG}},
note = {Machine review of arXiv:2607.19899}
}
read the original abstract
Disagreement-triggered escalation can create a structural blind spot in multi-agent arbitration: as base learners improve, they tend to converge, weakening safety monitoring where correlated failures concentrate. We term this correlated agreement blindness and present ARAT (Arbitrated Reasoning Agents for Alarm Triage), a directed-star system combining an inductive Random Forest (RF) agent, an analogical case-based k-nearest neighbour (k-NN) agent, and a calibrated meta-model to mitigate this effect. On 82,332 holdout samples from the UNSW-NB15 network intrusion detection dataset, 57.2% of errors occur under agreement and 90.6% of dangerous under-predictions evade disagreement-based monitoring even after conservative override; ablation shows that strengthening base learners increases error correlation while reducing disagreement. ARAT reduces under-prediction relative to soft voting from 4.80% to 1.70% via conservative override (-2.6pp) and a safety-flag gate (-0.5pp), demonstrating architectural gains. Cross-dataset validation on clinical readmission supports these indicators, suggesting that diversification improves safety only when it generates productive disagreement rather than convergence. These results indicate that disagreement-triggered escalation can be blind to correlated failure, a risk that may intensify as agentic pipelines deploy increasingly capable, correlated models.
Figures
Reference graph
Works this paper leans on
-
[1]
Information Fusion6(1), 5–20 (2005)
Brown, G., Wyatt, J., Harris, R., Yao, X.: Diversity creation methods: A survey and categorisation. Information Fusion6(1), 5–20 (2005)
2005
-
[2]
Cemri, M., Pan, M.Z., Yang, S., et al.: Why do multi-agent LLM systems fail? In: Advances in Neural Information Processing Systems (2025)
2025
-
[3]
In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
Chen, T., Guestrin, C.: XGBoost: A scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 785–794 (2016)
2016
-
[4]
In: Proceedings of IJCAI
Elkan, C.: The foundations of cost-sensitive learning. In: Proceedings of IJCAI. pp. 973–978 (2001)
2001
-
[5]
In: Proceedings of ECML
Frank, E., Hall, M.: A simple approach to ordinal classification. In: Proceedings of ECML. vol. 2167, pp. 145–156. Springer (2001)
2001
-
[6]
In: International Conference on Machine Learning
Goel, S., Strüber, J., Auzina, I.A., et al.: Great models think alike and this un- dermines AI oversight. In: International Conference on Machine Learning. vol. 267 (2025)
2025
-
[7]
Hammond, L., Chan, A., Clifton, J., et al.: Multi-agent risks from advanced AI. Tech. Rep. 2502.14143, arXiv (2025), https://arxiv.org/abs/2502.14143
arXiv 2025
-
[8]
IEEE Transactions on Big Data7(3), 535–547 (2021)
Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. IEEE Transactions on Big Data7(3), 535–547 (2021). https://doi.org/10.1109/TBDATA. 2019.2921572
arXiv 2021
Show all 25 references
-
[9]
In: Advances in Neural Information Processing Systems
Ke, G., Meng, Q., Finley, T., et al.: LightGBM: A highly efficient gradient boosting decision tree. In: Advances in Neural Information Processing Systems. vol. 30, pp. 3146–3154 (2017)
2017
-
[10]
In: International Conference on Machine Learning
Kim, E.M., Garg, A., Peng, K., Garg, N.: Correlated errors in large language models. In: International Conference on Machine Learning. vol. 267 (2025)
2025
-
[11]
Kuai, C., Jiang, J., Zhu, Z., Wang, H., et al.: How independent are large lan- guage models? A statistical framework for auditing behavioral entanglement and reweighting verifier ensembles (2026), https://arxiv.org/abs/2604.07650
2026 arXiv
-
[12]
In: Data Analytics and Decision Support for Cybersecurity: Trends, Methodologies and Applications, pp
Moustafa, N., Creech, G., Slay, J.: Big data analytics for intrusion detection system: Statistical decision-making using finite Dirichlet mixture models. In: Data Analytics and Decision Support for Cybersecurity: Trends, Methodologies and Applications, pp. 127–156. Springer, C...
2017
-
[13]
In: 2015 Military Communications and Information Systems Conference (MilCIS)
Moustafa, N., Slay, J.: UNSW-NB15: A comprehensive data set for network in- trusion detection systems (UNSW-NB15 network data set). In: 2015 Military Communications and Information Systems Conference (MilCIS). pp. 1–6. IEEE (2015). https://doi.org/10.1109/MilCIS.2015.7348942 1...
2015
-
[14]
Information Security Journal: A Global Perspective25(1–3), 18–31 (2016)
Moustafa, N., Slay, J.: The evaluation of network anomaly detection systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set. Information Security Journal: A Global Perspective25(1–3), 18–31 (2016)
2016
-
[15]
IEEE Transactions on Big Data5(4), 481–494 (2019)
Moustafa, N., Slay, J., Creech, G.: Novel geometric area analysis technique for anomaly detection using trapezoidal area estimation on large-scale networks. IEEE Transactions on Big Data5(4), 481–494 (2019)
2019
-
[16]
Proceedings of the IEEE95(1), 215–233 (2007)
Olfati-Saber, R., Fax, J.A., Murray, R.M.: Consensus and cooperation in networked multi-agent systems. Proceedings of the IEEE95(1), 215–233 (2007). https://doi. org/10.1109/JPROC.2006.887293
2007
-
[17]
IEEE Transactions on Pattern Analysis and Machine Intelligence27(8), 1226–1238 (2005)
Peng, H., Long, F., Ding, C.: Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions on Pattern Analysis and Machine Intelligence27(8), 1226–1238 (2005)
2005
-
[18]
In: Advances in Neural Information Processing Systems
Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: CatBoost: Unbiased boosting with categorical features. In: Advances in Neural Information Processing Systems. vol. 31 (2018)
2018
-
[19]
Reid, A., O’Callaghan, S., Carroll, L., Caetano, T.: Risk analysis techniques for governed LLM-based multi-agent systems (2025), https://arxiv.org/abs/2508.05687
2025 arXiv
-
[20]
In: Big Data Technologies and Applications (BDTA)
Sarhan, M., Layeghy, S., Moustafa, N., Portmann, M.: NetFlow datasets for machine learning-based network intrusion detection systems. In: Big Data Technologies and Applications (BDTA). vol. 371, pp. 117–135. Springer, Cham (2021)
2021
-
[21]
In: Proceedings of COLT 1992
Seung, H.S., Opper, M., Sompolinsky, H.: Query by committee. In: Proceedings of COLT 1992. pp. 287–294. ACM (1992)
1992
-
[22]
BioMed Research International2014, 781670 (2014)
Strack, B., DeShazo, J.P., Gennings, C., Olmo, J.L., Ventura, S., Cios, K.J., Clore, J.N.: Impact of HbA1c measurement on hospital readmission rates: Analysis of 70,000 clinical database patient records. BioMed Research International2014, 781670 (2014). https://doi.org/10.1155...
2014 doi
-
[23]
Turkmen, Y., Buyukates, B., Bastopcu, M.: Don’t always pick the highest- performing model: An information theoretic view of LLM ensemble selection (2026), https://arxiv.org/abs/2602.08003
2026
-
[24]
Journal of Machine Learning Research 24(359), 1–49 (2023)
Wood, D., Mu, T., Webb, A.M., Reeve, H.W.J., Luján, M., Brown, G.: A unified theory of diversity in ensemble learning. Journal of Machine Learning Research 24(359), 1–49 (2023)
2023
-
[25]
Wiley, Chichester, 2 edn
Wooldridge, M.: An Introduction to MultiAgent Systems. Wiley, Chichester, 2 edn. (2009)
2009
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.