{"id":"8e674fa8-06ae-4e0d-a037-34ae624a59c4","arxiv_id":"1908.11091","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Selecting DNN ensemble teams by low Kappa disagreement is presented as a defense against adversarial examples, but the evidence is preliminary and incomplete.","lead":"This paper proposes a framework for building deep neural network ensembles that combine architectural diversity with disagreement-based team selection, and reports preliminary results suggesting such ensembles are hard for adversarial attacks to fool. It is largely a survey of known diversity metrics plus an under-specified experimental sketch, so the practical value depends on future validation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed robustness rests on attacks generated only against the target model; the paper itself concedes in Section V that a fixed ensemble defense is just another target model, so no evidence supports the headline robustness claim.","rationale":"The reader's weakest assumption identifies precisely the load-bearing premise: the paper's robustness numbers are generated only by attacking the target model alone. My review of the full text confirms this. Table 3's caption states 'The adversarial examples are generated from the target model,' and Section III.C describes the attacks the same way. The paper does not generate attacks against the ensemble consensus, nor does it compare against the named defenses (adversarial training, distillation, ensemble transformation) quantitatively. The claimed central result is therefore not established. The paper's own Section V is decisive: a fixed ensemble committee under white-box online attacks is 'just another target model' for which 'existing attack algorithms remain to be effective.' This internal admission applies to the fixed ensembles evaluated in Table 3 and directly undermines the headline claim of stronger robustness than representative defenses. The subsequent discussion of randomized ensemble selection is a forward-looking proposal, not an evaluated defense. I also note secondary issues that strengthen the rejection but are not the primary concern: the attack count is inconsistent (12 attacks in Section III.C versus 15 in Table 3's caption), Table 3 appears not to include actual numerical values in the text, no error bars or attack parameters are given, and no code is released. These omissions make the empirical support impossible to verify. The concern is not merely that the result disagrees with current consensus; it is that the evaluation protocol excludes attacks an adversary would naturally mount, and the manuscript explicitly acknowledges the vulnerability that the evaluation avoids. A single concrete experiment, generating adversarial examples against the ensemble consensus itself, would settle the matter. For these reasons I agree with the reader's REJECT verdict and recommend no change.","tokens_in":11668,"tokens_out":3236,"duration_ms":35603,"concrete_test":"On the CIFAR-10 and ImageNet ensembles from Tables 1-3, run PGD/CW attacks that optimize against the exact consensus function used (weighted average and majority voting) over the full ensemble, with all member parameters known, for the same attack settings. Also include one ensemble-transfer attack from Refs [10-14]. Report robust accuracy of the ensemble under these adaptive attacks alongside the target-only numbers from Table 3. If robust accuracy drops substantially relative to the target-only attack results, the robustness claim is an artifact of the non-adaptive threat model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that type 1 and type 2 diversity ensembles offer stronger robustness than adversarial training, distillation, and ensemble transformation—is supported only by Table 3, whose caption states that adversarial examples are generated from the target model. This evaluates transferability of target-model-specific attacks, not robustness of the ensemble as a defended system. A competent adversary can generate perturbations directly against the ensemble consensus, e.g., the average logits or the voting rule, using the ensemble attacks cited in the paper itself (Refs [10-14]). The paper's own Section V concedes: 'if a fixed ensemble committee is chosen as the defense protection of a target model, then this protected prediction model is just another target model, and existing attack algorithms remain to be effective.' That admission applies exactly to the fixed ensembles evaluated in Table 3. The later suggestion of random per-query ensemble selection is a design proposal, not evaluated data, and is supported only by an unpublished tech report [17]. Therefore the comparative claim of stronger robustness than representative defenses has no supporting evidence under an adaptive threat model; the reported results are consistent with the weaker statement that adversarial examples do not always transfer across the selected pretrained models.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for building deep neural network ensembles that are accurate and robust against adversarial examples. It defines three types of ensemble diversity—structural (type 1), disagreement-based (type 2), and hypothesis-space/training diversity (type 3)—and describes pairwise and non-pairwise diversity metrics, ensemble creation algorithms built on pre-trained model pools, and consensus methods such as majority/plurality voting, averaging, weighted averaging, and supra-Bayesian and stacked generalization. The authors report preliminary experimental results on MNIST, CIFAR, and ImageNet, and claim in Section I that the combination of type 1 and type 2 diversity creates ensembles with 'stronger robustness and higher defensibility' than adversarial training, defense distillation, and ensemble transformation. The paper also discusses offline versus online and black-box versus white-box threat models, and proposes randomized per-query ensemble selection as a defense against white-box online attacks.","tokens_in":11813,"tokens_out":3353,"duration_ms":34948,"significance":"If the central claim were substantiated, the paper would offer an attractive and cheap defense: select diverse pre-trained models by disagreement metrics and use a consensus rule rather than retraining with adversarial training or distillation. The paper has several genuine strengths: it gives a clear taxonomy of diversity types, reviews standard diversity metrics and consensus methods, explicitly distinguishes threat models, and constructs pools of pre-trained models for MNIST, CIFAR-10, and ImageNet. However, the load-bearing empirical claim—that these ensembles are more robust than existing representative defenses—is not supported by the reported experiments. The only comparative table, Table 3, evaluates ImageNet with adversarial examples generated from the target model, which measures cross-model transferability rather than robustness of the ensemble as a defended system, and Section V concedes that a fixed ensemble is just another target model. No adaptive, ensemble-aware attacks are evaluated, no confidence intervals or error bars are provided, and the attack count is inconsistent between the text and the table caption.","major_comments":[{"comment":"The headline claim of 'stronger robustness and higher defensibility than those offered by existing representative defense methods' is not supported by the experimental evidence. Table 3 is the only comparison against adversarial training, distillation, and ensemble transformation, and its caption states that 'the adversarial examples are generated from the target model.' This evaluates whether target-model-specific adversarial examples transfer to the ensemble members; it does not evaluate the robustness of the ensemble as a defended system. A adversary who knows the ensemble can generate perturbations directly against the consensus rule or averaged logits, using the ensemble attack methods cited in the paper itself (Refs. [10-14]). Without such adaptive attacks, the comparative robustness claim is unsupported.","section":"Section I and Section III.C, Table 3"},{"comment":"The paper concedes that 'if a fixed ensemble committee is chosen as the defense protection of a target model, then this protected prediction model is just another target model, and existing attack algorithms remain to be effective.' This admission applies directly to the fixed ensembles evaluated in Table 3, so the reported results are at best evidence of limited transferability, not of robustness against a competent adversary. The subsequent proposal of randomized per-query ensemble selection is a design sketch, not evaluated data, and is supported only by an unpublished technical report [17]. The central robustness claim therefore lacks evidence under the paper's own stated threat-model taxonomy.","section":"Section V"},{"comment":"The experimental reporting is incomplete in ways that undermine the empirical conclusions. The text says the authors evaluate '12 attacks' (Section III.C), while the caption of Table 3 says '15 attacks.' Table 3 reports no confidence intervals, error bars, or repeated runs, and no attack parameters (e.g., perturbation budgets or iteration counts) are given. The table appears to cover only ImageNet, with no comparative table for MNIST or CIFAR-10, despite the paper's broad claims about three benchmark datasets. This makes it impossible to assess the variance or statistical significance of the reported accuracy improvements.","section":"Table 3 and Section III.C"},{"comment":"The paper assumes, rather than establishes, the load-bearing premise that higher disagreement diversity implies lower adversarial transferability and hence higher ensemble robustness. Section II.B states that greater diversity is 'highly correlated' with ensemble accuracy and that negatively correlated classifiers reduce error further, but no formal argument or independent empirical test connects the proposed kappa-based diversity selection to robustness against targeted or untargeted adversarial examples. The authors note in Section VI that several issues 'are not yet well defined with theoretical formulation,' including how to correlate increase in diversity with increase in ensemble robustness. Given that the entire defense proposal rests on this link, it needs either a formal justification or a systematic empirical validation across models, datasets, and attack types.","section":"Section II.B and Section III.C"}],"minor_comments":[{"comment":"The manuscript contains several typographical errors and garbled formulas, e.g., 'parafigm' in Section II.A, 'Carlini & Wagber' in Section III.C, and the equations in Section II.C appear with mangled symbols in the text. A careful proofreading and typesetting pass is needed.","section":"Throughout"},{"comment":"The description of the attack set is confusing: the text lists 'two untargeted attacks: FGSM, BIM, and ten targeted attacks: targeted FGSM, targeted BIM, JSMA, Carlini & Wagner attacks (CW0, CW2, CWy).' The count of ten targeted attacks does not obviously follow from the listed algorithms, and the caption of Table 3 uses a different total of 15. The attack count and the mapping of algorithms to targeted/untargeted categories should be made explicit.","section":"Section III.C"},{"comment":"Tables 1 and 2 are referenced repeatedly (e.g., 'Table 1 shows ten base models for MNIST, eight base models for CIFAR-10 and five models for ImageNet-1000'), but the actual table contents are not included in the manuscript text, only captions. The reader cannot verify the model pools, the kappa values, or the ensemble team compositions.","section":"Section III"},{"comment":"The discussion of the random ensemble results is difficult to follow: the sentence 'the random ensemble of team size 3 in this case outperforms the random Kappa ensemble' appears to compare two similar-sounding baselines without a clear definition of either. Please define RandBase and Randκ explicitly and explain the difference between them.","section":"Section III.C"},{"comment":"The paper uses 'black-box' and 'black box' inconsistently (e.g., 'back-box online attack' appears in Section V). The threat-model terminology should be used uniformly, since the distinctions are central to the paper's argument.","section":"Section V"},{"comment":"Reference [17] is an unpublished technical report that supports the only proposed mechanism for white-box online robustness; the paper should either include its key results or mark the claim as speculative. Also, Reference [32] is cited as 'Patridge' but the standard spelling is 'Partridge.'","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper is best viewed as a conceptual overview of ensemble diversity for adversarial robustness, but it is framed as presenting a comparative defense method. The central empirical claim is not backed by adaptive attacks, and the paper's own threat-model discussion undermines the fixed-ensemble evaluation. A resubmission would need to include ensemble-aware attacks, confidence intervals, consistent attack counts, and either a complete evaluation of the randomized per-query mechanism or a reframing as a survey/position paper. The heavy reliance on the authors' own prior work [2] and unpublished technical report [17] for load-bearing claims is also a concern for a journal submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a survey/position paper with a plausible recipe—select pretrained models by disagreement diversity and combine them—but the headline robustness claim is not backed by the experiments as designed, and the paper's own Section V admits the core limitation.\n\nWhat's new: not much. The three-type diversity taxonomy is a re-labeling of categories already in Brown et al. and Kuncheva and Whitaker. All five metrics are standard. The only genuinely new piece is the idea of applying kappa-ranked team selection to a pool of public pretrained DNNs, and the \"random per-query ensemble selection\" is a design sketch, not an evaluation. The survey parts are competent and would be useful to someone new to the area.\n\nWhat's good: the paper is honest enough to state in Section V that a fixed ensemble defense is \"just another target model\" and that existing attacks remain effective. That admission is the right instinct, and it's exactly what kills the main claim. The preliminary results in Table 3 are at least described with enough detail to see what was done—attacks generated on the target model, then measured against the ensemble. That's a transferability study, not an ensemble robustness study.\n\nWhere it falls down: the abstract and introduction claim \"stronger robustness and higher defensibility\" than adversarial training, distillation, and ensemble transformation, but Table 3 compares only the target model, a baseline defense model, and a few ensemble variants on ImageNet, with no error bars, no attack parameters, no code, and an inconsistency in the number of attacks (12 in text, 15 in caption). No comparison against the named defense baselines appears anywhere. And as noted, the attacks are generated against the target model alone, so they test transfer failure, not resistance to an adversary who can query the ensemble. The paper's own Section V concession applies directly to the evaluated fixed ensembles. The later suggestion of random per-query selection is supported only by an unpublished tech report.\n\nProportion: the central comparative claim has no supporting evidence under a realistic threat model, but the underlying idea—using disagreement diversity as a cheap way to get transfer resistance—is reasonable and worth studying properly.\n\nWho this is for: someone wanting a quick overview of diversity measures and a pointer to a possible defense direction. It is not a completed research result.\n\nRecommendation: if I were the editor, I'd send it to reviewers, not desk-reject, because the topic is relevant and the paper can be revised into an honest survey plus a clearly-labeled preliminary study. But as submitted, it should be rejected or major-revised: the robustness comparisons need adaptive attacks, baselines, and uncertainty estimates.","headline":"A competent survey of ensemble diversity wrapped in an unsupported robustness claim; the paper's own Section V concedes the threat model that would be needed to support it.","tokens_in":12415,"tokens_out":1919,"would_cite":false,"duration_ms":19105,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that diverse DNN ensembles block a dozen adversarial attacks without retraining.","keywords":["ensemble diversity","adversarial robustness","DNN ensembles","kappa statistic","adversarial transferability","ensemble consensus","model diversity","black-box attacks"],"falsifier":"Run the same twelve attacks on the full ensemble—for example, optimizing the perturbation against the committee's averaged loss while treating all members as white-box—and compare accuracy with the target-only attacks. If the examples fool the target and the teammates alike, the disagreement-based consensus cannot repair the damage, and the defense's advantage over baselines should shrink.","tokens_in":11417,"feed_emoji":"🛡️","tokens_out":6958,"duration_ms":67571,"temperature":0.7,"pith_summary":"The paper sets out to show that adversarial robustness can be bought by composition rather than retraining: instead of hardening a single deep network, surround a target model with a team of individually accurate models that disagree on their mistakes, and let a consensus rule decide the final label. It distinguishes three sources of diversity—model construction, output disagreement, and training-time decorrelation—and concentrates on the first two. The proposed recipe is to collect pre-trained models, score every candidate team by pairwise disagreement, rank teams by the kappa statistic, and combine member outputs by majority or weighted averaging. The authors report that such teams outperform representative baselines against a dozen attacks on MNIST, CIFAR-10, and ImageNet. If this is right, a defender can raise robustness at deployment time using off-the-shelf models, without modifying the target's weights.","feed_headline":"Diverse DNN ensembles can block 12 adversarial attacks","feed_subtitle":"Pick pre-trained models that disagree, then vote; the target resists deception.","key_machinery":"The load-bearing object is the diversity-ranked ensemble team. Models are first gathered into a pool by structural variety (architecture, training data, hyperparameters), then every subset of a given size is scored by pairwise disagreement measures—chiefly the kappa statistic, with Q-statistic, rho-statistic, binary disagreement, and entropy as alternatives—and teams with high disagreement are selected. The target model is a member of every team. Consensus methods (majority voting, plurality, simple or confidence-weighted averaging, and supra-Bayesian or stacked combinations) turn the members' outputs into one prediction. The kappa score is doing the work of predicting which teams will be robust: low average kappa means errors are spread across different members, so a consensus can overrule a member fooled by an adversarial example.","core_discovery":"On the paper's own terms, the central discovery is that a diversity-optimized ensemble—built from structurally different pre-trained models selected for pairwise disagreement—can defend a target deep network against a known suite of twelve adversarial attacks more effectively than adversarial training, defense distillation, or input-transformation defenses. The paper defines three types of ensemble diversity, proposes a creation pipeline for type 1 and type 2 diversity teams, and reports that the best kappa-ranked team outperforms random teams and the target on targeted and untargeted attacks, with additional gains when a rotation-denoising input transform is added to the consensus. It also argues that ensemble accuracy is not strongly tied to team size, so small diverse teams can be both accurate and cheap. The robustness argument rests on adversarial examples being inconsistent across models: an example that fools the target need not fool its diverse teammates.","pith_inferences":["Beyond the paper: a decisive evaluation would attack the full committee rather than the target alone; the paper's own results only use attacks generated from the target model, so how much robustness remains against adaptive attacks is untested here.","Beyond the paper: kappa is measured on benign examples, so an adversary aware of the selection rule could try to make the target and its teammates err together, converting low benign disagreement into high attack-time correlation.","Beyond the paper: the proposed per-query random team selection is a moving-target defense in disguise; quantifying how often the ensemble changes per query and how much that raises the attacker's cost could turn the heuristic into a measurable security property."],"forward_implications":["A defender can improve robustness by selecting pre-trained models from a model zoo and combining them, without retraining the target model.","Small, highly diverse teams can deliver the same accuracy as larger teams, reducing test-time cost and storage.","Because diversity is attack-independent, ensembles selected this way may generalize to attacks not present during selection.","Adding an input-transformation member to a kappa-selected team can further improve robustness, but applying the transform before the ensemble can hurt because denoising errors propagate.","Under white- or grey-box online attacks, a fixed committee is just another target, but randomly selecting a defense team per query improves robustness."],"supporting_citations":[{"why":"This work establishes that adversarial examples behave differently across models, which is the premise for disagreement-based ensembles.","marker":"[2]"},{"why":"This work introduces adversarial examples and the observation that they transfer between networks.","marker":"[3]"},{"why":"This work shows that transferability enables black-box attacks through substitute models, defining the threat model the defense must handle.","marker":"[5]"},{"why":"This work supplies the diversity measures and the claimed relationship between diversity and ensemble accuracy.","marker":"[27]"},{"why":"This work defines the kappa statistic used to rank type-2 diversity ensembles.","marker":"[25]"},{"why":"This work provides the ensemble adversarial-training baseline that the paper compares against.","marker":"[19]"},{"why":"This work provides the distillation defense baseline used in the comparison.","marker":"[20]"},{"why":"This work provides the input-randomization baseline the paper compares against.","marker":"[21]"}],"fun_headline_variants":["Diversity in DNN ensembles blocks 12 adversarial attacks","Disagreeing DNN models vote to resist adversarial attacks","Small, diverse DNN teams defeat 12 attacks","Ensemble diversity: robust defense against deception","Vote with contrarian models to block 12 attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported robustness numbers assume that adversarial examples generated over the target model will not transfer to the other ensemble members, so the consensus can correct the target's mistake; the paper only evaluates attacks generated from the target model, never from the ensemble itself.","fun_headline_variants_meta":{"raw":{"variants":["Diversity in DNN ensembles blocks 12 adversarial attacks","Disagreeing DNN models vote to resist adversarial attacks","Small, diverse DNN teams defeat 12 attacks","Ensemble diversity: robust defense against deception","Vote with contrarian models to block 12 attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1734,"prompt_tokens":863,"completion_tokens":871,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":794}},"tokens_in":479,"tokens_out":871,"duration_ms":8383,"temperature":1.0,"reasoning_tokens":794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:24:34.233707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same twelve attacks on the full ensemble—for example, optimizing the perturbation against the committee's averaged loss while treating all members as white-box—and compare accuracy with the target-only attacks. If the examples fool the target and the teammates alike, the disagreement-based consensus cannot repair the damage, and the defense's advantage over baselines should shrink.","supporting_citations":[{"cited_title":"Adversarial Examples in Deep Learning: Characterization and Divergence","cited_arxiv_id":"1807.00051","evidence_quote":"This work establishes that adversarial examples behave differently across models, which is the premise for disagreement-based ensembles."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"This work provides the input-randomization baseline the paper compares against."}],"review_version":1}