{"id":"6925ab2d-3445-45f8-bd67-08c89a3aee66","arxiv_id":"2507.20996","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-teacher adversarial robustness distillation method (MTKD-AR) trains a clean-data student using cosine-similarity-weighted logits from adversarially trained teachers, reporting improved robustness on MNIST and Fashion-MNIST.","lead":"Multiple small models are first hardened against different attacks, then used as teachers to train one student model on clean images. The authors claim this multi-teacher distillation with adaptive weights gives the student broad adversarial robustness without ever seeing attacked images.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Adaptive weighting is the paper's core innovation, but Eq. 11 measures student–teacher logit agreement, not prediction precision; with a randomly initialized student the weights are circular and unvalidated, so the reported robustness may be due to ordinary multi-teacher averaging.","rationale":"The paper is honest in scope: it proposes a specific adaptive multi-teacher distillation mechanism and evaluates it on two datasets with released code. Reading in good faith, the central claim is strong but constrained: a clean-data student can acquire adversarial robustness from multiple adversarially pre-trained teachers, and the adaptive weighting is what makes this work well. For that to be true, the weights in Eq. 13 must track teacher quality in the sense relevant to robustness. The paper never shows this. The prose in Section IV.B.1 calls Eqs. 11–12 'prediction precision,' but the quantity is cosine similarity between student and teacher logits; this is a measure of agreement, and agreement can be high for two non-robust models sharing a wrong boundary. Because the student logits supply the reference vector, the weights are endogenous: early in training v is near-random, and the gradient updates the student toward a target that is itself a function of the current student. This is not necessarily fatal to the robustness result, which can still come from the multi-teacher soft targets, but it means the paper's distinctive adaptive-learning claim is not established by Tables I–VI. A uniform-weight ablation is the decisive experiment because it isolates the mechanism. I therefore endorse the reader's conditional verdict: the central robustness phenomenon is plausible and partly supported by the known viability of adversarial robustness distillation and the released code, but the adaptive contribution needs a direct test before the claims in the abstract and Section IV.B.1 can be accepted. I do not see grounds for rejection based on this concern alone, though the implementation of Eq. 14 should also be clarified in revision.","tokens_in":15264,"tokens_out":6245,"duration_ms":83653,"concrete_test":"Re-run the MTKD-AR pipeline from the released repository on MNIST-Digits with three otherwise identical student trainings: (a) adaptive cosine weights as in Eqs. 11–13; (b) uniform teacher weights; (c) diagnostic oracle weights set to each teacher's measured robust accuracy per attack on a held-out perturbed set. Compare clean accuracy and FGSM, RFGSM, and PGD accuracy at epsilon = 0.1 and epsilon = 0.3 over at least three random seeds. If condition (a) is not clearly better than (b), the adaptive mechanism is not the source of robustness; if (a) does not track (c), the 'prediction precision' interpretation is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.B.1 introduces the adaptive contribution: Eqs. 11–13 and Listing 1 compute teacher weights from cosine similarity between student logits v and teacher logits z_i. The paper calls this 'prediction performance' and 'reliability,' but cosine similarity of logits is agreement, not correctness. During early training the student is randomly initialized, so v is near-random and the cosine similarities are small and noisy; as training proceeds, the weights depend on the very student being trained, creating a feedback loop in which the student can amplify a teacher it already agrees with rather than one that is robust. The mechanism is also evaluated only on clean inputs, whereas the claimed benefit is robustness under perturbation; a teacher with high clean-data agreement need not be a robust teacher under FGSM or PGD. Thus Tables V and VI do not establish that adaptive weighting, rather than simple multi-teacher averaging, produces the reported robustness. The central robustness claim may survive this concern, but the paper's distinctive adaptive-learning claim is load-bearing for its novelty; if uniform weighting performs equally, the method reduces to known multi-teacher ARD with an extra unstable mechanism. Secondary issues, such as the dimension mismatch in Eq. 14 and the inconsistent clean-accuracy baselines across Tables I and V, reinforce the need for a precise, reproducible definition of the distillation loss but are not the primary attack on the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MTKD-AR, a multi-teacher adversarial robustness distillation framework in which several clones of a CNN are adversarially trained on FGSM, FFGSM, RFGSM, and PGD perturbations, and a student model is trained on clean data only using a weighted combination of the teachers' soft targets. The weighting is computed online from the cosine similarity between the student's logits and each teacher's logits. The authors report that the student attains robust accuracy comparable to or better than the individual adversarially trained teachers across the four attack types on MNIST-Digits and Fashion-MNIST, including at perturbation magnitudes larger than those used in teacher training, and they release code at https://github.com/iscaas/MTKD-AR.","tokens_in":15544,"tokens_out":4860,"duration_ms":51109,"significance":"If the central claim holds, the paper offers a practical way to obtain adversarial robustness in a small student model without generating adversarial examples, which would reduce the computational cost of robust training. The multi-teacher adaptive weighting idea is, however, the main novelty, and the paper does not currently demonstrate that this mechanism is what drives the reported gains. The experiments are limited to two low-resolution datasets and four white-box attacks, with no comparison to prior ARD methods or to a uniform-weight multi-teacher baseline. The paper does provide a public implementation and evaluates clean and robust accuracy across multiple epsilon levels, which are positive elements for reproducibility.","major_comments":[{"comment":"Section IV.B.1, Eq. (11): the adaptive weight is computed as the cosine similarity between student logits v and teacher logits z_i, yet the paper repeatedly calls this 'prediction performance' and 'reliability' (e.g., Abstract and Section IV.B.1). Cosine similarity of logits measures student–teacher agreement, not teacher correctness; with a randomly initialized student at the start of distillation the weights are noisy and depend on the very student being trained, creating a feedback loop that can amplify an already-agreeing teacher rather than a robust one. The authors should (a) add an ablation with uniform teacher weighting, (b) show that the weights correlate with teacher accuracy on attacked inputs rather than only clean inputs, and (c) report the stability of the weights at initialization. Without (a), Tables V and VI do not establish that adaptive weighting, rather than ordinary multi-teacher averaging, produces the reported robustness.","section":"IV.B.1, Eq. (11)"},{"comment":"Section IV.B.2, Eq. (14): the distillation loss is written as KLD(p(1,n) * W(1,n), q), but p(1,n) is defined in Eq. (9) as a single summed distribution over all teachers, while W(1,n) is a per-teacher weight vector; the elementwise product is therefore dimensionally mismatched (class dimension versus teacher dimension). Listing 2 correctly computes the weighted prediction as Σ_i teachers_predictions[i] * normalized_teachers_weights[i], but the equation and the listing are inconsistent, and Eq. (14) uses σ (sigmoid) while the text describes a temperature-controlled softmax. The loss must be defined precisely and consistently with the code.","section":"IV.B.2, Eq. (14)"},{"comment":"Section VI (Tables V–VI): the student is evaluated on the same four attack types that were used to build the teacher pool, so the reported robustness is in-sample with respect to attack type; the only held-out dimension is perturbation magnitude (ϵ = 0.2, 0.3), which is a small generalization step. The abstract's claim of withstanding 'different adversarial attacks' and the text's claim of generalization to 'unseen perturbations' require evaluation against held-out attacks such as DeepFool, C&W, or AutoAttack.","section":"VI, Tables V–VI"},{"comment":"Tables I, III, and V: the clean-data accuracy of the same baseline CNNW_OKD on MNIST-Digits is reported as 99.05, 98.89, and 97.66, respectively; Table II is headed 'MNIST-Digits Dataset Results' although the surrounding text and the data (clean accuracy 91.41) describe Fashion-MNIST; Tables V and VI call the proposed method 'MTKD-ADR' instead of 'MTKD-AR'; and the proposed method's PGD accuracy at ϵ = 0.1 is 89.82 in Table IV but 90.82 in Table VI. Numbers obtained under the same protocol should be identical, and the authors must reconcile them and report error bars over multiple runs.","section":"Tables I, II, III, V, VI"},{"comment":"Section VI: the only baselines are the unperturbed-network distillation (CNNW_OKD) and single-teacher distillation (WKD-1T). There is no comparison to prior ARD methods such as Goldblum et al. [19] or Zhu et al. [22], and there is no uniform-multi-teacher baseline. Without these, the claimed advantage of MTKD-AR over the state of the art is not established, and the adaptive mechanism is not isolated.","section":"VI"}],"minor_comments":[{"comment":"The summation notation in tθ(1,n) and tθ'(1,n) is distracting; the equations can be written more simply as tθ_i = 1 + t_similarity_i and tθ'_i = tθ_i / Σ_j tθ_j.","section":"IV.B.1, Eqs. (12)–(13)"},{"comment":"The paper alternates between 'MNIST-Fashion' and 'Fashion-MNIST'; the dataset name should be used consistently throughout.","section":"Abstract and Section V"},{"comment":"The caption of Listing 1 contains the typo 'Pseudocod'; it should be 'Pseudocode'.","section":"Listing 1"},{"comment":"Eq. (14) should specify the KLD direction (forward or reverse) and whether the weighted target distribution is renormalized after multiplication by W.","section":"IV.B.2, Eq. (14)"},{"comment":"The related work discussion is brief and does not mention recent ARD variants such as Robust Soft Label Adversarial Distillation (RSLAD), which would help position the contribution relative to the current literature.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"The paper has several signs of an early-stage submission: inconsistent table headings, mismatched method names, and a loss equation that does not match the code listing. I recommend that the editor require a major revision in which the authors validate the adaptive-weighting claim with a uniform-weight baseline and bring the evaluation up to the current ARD standard, including adaptive attacks and a held-out attack type. The lack of comparison to existing ARD literature is a scope and novelty concern that should be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an incremental but sensible multi-teacher ARD paper with a specific adaptive-weighting twist that is neither well-defined nor properly tested. The central idea—distill robustness from adversarially trained teachers into a clean-data student—is real and relevant, but the paper doesn't yet support its stronger claims.\n\nWhat's actually new: the cosine-similarity-based dynamic teacher weighting (Eqs. 11–13) and the total loss in Eq. 16. That's a minor variant on existing teacher-reliability weighting for ARD (e.g., Zhu et al. 2021), not a new capability. The authors do ship code and describe the pipeline in enough detail to reproduce, which is more than many papers in this space.\n\nWhere it falls down:\n\n1. The adaptive weighting is the paper's load-bearing novelty, and it's not validated. Cosine similarity between student and teacher logits measures agreement, not prediction precision or robustness. With a randomly initialized student, the weights are noisy; as training goes on they become self-referential. The paper never compares to equal-weight multi-teacher distillation, so we have no idea whether the adaptive mechanism does anything. This is the key missing experiment.\n\n2. The evaluation is too narrow. Two small datasets, no comparison to prior ARD methods (Goldblum et al. 2020, Zhu et al. 2021), no adaptive attacks (PGD with many restarts, AutoAttack), no error bars. The robustness numbers are only in-sample with respect to attack type; the held-out dimension is just perturbation magnitude, which is a small step.\n\n3. There are internal inconsistencies that undermine trust in the numbers: the same clean baseline accuracy appears as 99.05, 98.89, and 97.66 across tables; Table II is mislabeled as 'MNIST-Digits' when it's Fashion-MNIST; Eq. 14 mixes sigmoid and softmax and the weighted teacher target is not a valid probability distribution (element-wise product of probabilities and weights needs renormalization). These are fixable, but they need to be fixed.\n\nBottom line: the core idea may survive, but the paper as written doesn't demonstrate its distinctive contribution. I'd send it to a serious referee—it's not a desk reject—but the referee should push for an ablation with uniform weighting, an honest evaluation against existing ARD baselines, and a corrected loss formulation.\n\nMy recommendation: engage in review as a borderline accept-with-major-revisions, not as-is.","headline":"A plausible incremental ARD scheme with unvalidated adaptive weighting and weak evaluation; worth a referee only after major revisions.","tokens_in":16067,"tokens_out":3481,"would_cite":false,"duration_ms":33602,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a student only on clean images can teach it to withstand four different adversarial attacks, by learning from several adversarially trained teachers whose per-image influence is set by cosine similarity.","keywords":["adversarial robustness","knowledge distillation","multi-teacher distillation","adaptive learning","adversarial attacks","convolutional neural networks","image perturbation","clean-data training"],"falsifier":"Evaluate the distilled student against a white-box attack type that is absent from the teacher pool, such as the C&W or DeepFool attack, at the same perturbation budgets used in the paper: if accuracy falls to near-random levels, the claim that robustness generalizes beyond the four training attacks is contradicted.","tokens_in":15068,"feed_emoji":"🛡️","tokens_out":17281,"duration_ms":157409,"temperature":0.7,"pith_summary":"The paper argues that a student CNN trained only on clean images can acquire strong resistance to adversarial attacks when it is supervised by several teacher CNNs that were adversarially trained against different attack types. The proposed MTKD-AR method trains four clones of a lightweight CNN, one against FGSM, one against FFGSM, one against RFGSM, and one against PGD, and then distills their combined knowledge into a single student using a distillation loss in which each teacher's contribution is reweighted per input image. The per-input weights are computed from the cosine similarity between the student's current logits (raw class scores) and each teacher's logits, which the paper treats as a live measure of teacher reliability. If the claim holds, adversarial robustness can be transferred without ever exposing the student to perturbed data, avoiding the main computational cost of adversarial training and making robust models cheaper to deploy on small devices. On MNIST-Digits and Fashion-MNIST, the distilled student keeps accuracy in the high 80s to low 90s across all four attacks at perturbation magnitudes up to 0.3, while each single-teacher model drops sharply on attacks it was not trained to handle.","feed_headline":"Clean-data student inherits attack resistance from four teachers","feed_subtitle":"Each teacher defends against one attack; the student defends against all four using only clean images.","key_machinery":"The load-bearing mechanism is the adaptive teacher-weighting adapter. For each input image, the student logits v (raw class scores) and each teacher's logits z_i are compared by cosine similarity; the score is mapped through 1 + similarity and normalized across teachers, yielding a weight vector that is recomputed at every training step. The distillation target is the element-wise weighted combination of the teachers' soft predictions, produced by a temperature-controlled softmax, and the total loss is a weighted sum of a KL divergence between the student and that target and a cross-entropy loss against the ground-truth label. The weighting is what distinguishes the method from a static ensemble: a teacher that aligns with the current student on a given input dominates that input's distillation signal.","core_discovery":"The central claim is that adversarially pre-trained teachers can transfer robustness to a student trained entirely on clean data, and that the transfer works because the student is not bound to an unweighted average of teacher opinions. Four clones of a lightweight CNN are adversarially trained, one per attack, and then frozen as teachers. For each input image, the adapter computes the cosine similarity between the student's raw output scores and each teacher's raw output scores, shifts the scores by one, normalizes them into a weight vector, and forms a weighted soft target from the teachers' temperature-smoothed probability vectors. The student's total loss is the temperature-scaled KL divergence between its own soft prediction and this weighted target, plus a cross-entropy term against the true label, with α balancing the two. The paper reports that the student retains high accuracy on all four attack types simultaneously, and keeps a large share of that accuracy even at perturbation magnitudes (0.2 and 0.3) beyond the 0.1 used in teacher training; each single-teacher model, in contrast, keeps high accuracy only on the attack it was trained against. In the paper's framing, the adaptive weighting is what prevents inaccurate or irrelevant teacher predictions from misleading the student during distillation.","pith_inferences":["If cosine-similarity weighting truly tracks teacher reliability, the teacher pool could be diversified by architecture or failure mode rather than by attack type, which might generalize to attacks the pool was never designed against.","The self-referential nature of the weights (the student is scored against its own evolving predictions) could create a feedback loop that locks the student onto early agreements; a moving-average or annealed weight would be a direct test of that risk.","Because the reported experiments use 28×28 grayscale datasets with ten classes, a natural next step is to check whether the same weighting mechanism holds on high-resolution, large-vocabulary tasks where logit geometry is very different.","The method could be combined with certified defenses or with teachers trained on different perturbation norms, potentially yielding robustness certificates that cover a wider attack surface than any single teacher."],"forward_implications":["Adversarial robustness can be conferred on a student without adversarial data at training time, shifting the expensive perturbation generation to a one-time teacher-training phase.","Lightweight models, which often benefit little from direct adversarial training, can inherit multi-attack defense from a small pool of specialized teachers.","The reported accuracy under perturbation magnitudes larger than the teachers were trained on suggests the distilled robustness is not brittle to a specific epsilon.","Single-teacher distillation is shown to be insufficient for multi-attack defense, supporting the need for a teacher pool plus input-dependent weighting.","The framework provides a practical path to robustness in resource-constrained environments, where full adversarial training may be infeasible."],"supporting_citations":[{"why":"Introduces the phenomenon of adversarial examples that motivates the defense problem.","marker":"[4]"},{"why":"Supplies the FGSM attack and the adversarial-training formulation used to create the teachers.","marker":"[5]"},{"why":"Provides the PGD attack and the min-max adversarial training framework used for the PGD teacher.","marker":"[11]"},{"why":"Defines adversarial robustness distillation, the single-teacher baseline the paper extends to multiple teachers.","marker":"[19]"},{"why":"Shows that teacher reliability matters in robustness distillation, motivating the adaptive weighting scheme.","marker":"[22]"},{"why":"Introduces ensemble adversarial training, supporting the use of multiple teachers with complementary attack knowledge.","marker":"[23]"}],"fun_headline_variants":["Clean-data student gains all-attack defense via adaptive teachers","Multi-teacher distillation: adaptive weights = robustness transfer","Robustness without adversarial data: adaptive multi-teacher trick","Adaptive teachers distill all-attack robustness to clean student"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the cosine similarity between the student's current output scores and a teacher's output scores measures that teacher's prediction precision for the current image, as the paper calls it; early in training the student is random, no evidence links this agreement to correctness, and if it fails the adaptive weights can amplify an unreliable teacher.","fun_headline_variants_meta":{"raw":{"variants":["Clean-data student gains all-attack defense via adaptive teachers","Multi-teacher distillation: adaptive weights = robustness transfer","Robustness without adversarial data: adaptive multi-teacher trick","Adaptive teachers distill all-attack robustness to clean student"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000383,"raw_usage":{"total_tokens":2068,"prompt_tokens":1025,"completion_tokens":1043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":976}},"tokens_in":641,"tokens_out":1043,"duration_ms":8832,"temperature":1.0,"reasoning_tokens":976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:03:29.154556+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the distilled student against a white-box attack type that is absent from the teacher pool, such as the C&W or DeepFool attack, at the same perturbation budgets used in the paper: if accuracy falls to near-random levels, the claim that robustness generalizes beyond the four training attacks is contradicted.","supporting_citations":[{"cited_title":"Explaining and harnessing adversarial examples. proceedings of the 3rd international conference on learning representations, iclr 2015,","cited_arxiv_id":null,"evidence_quote":"Supplies the FGSM attack and the adversarial-training formulation used to create the teachers."},{"cited_title":"Adversarially robust distillation,","cited_arxiv_id":null,"evidence_quote":"Defines adversarial robustness distillation, the single-teacher baseline the paper extends to multiple teachers."}],"review_version":1}