{"id":"f1521e58-0863-425a-9e6b-1b4bfa1975b1","arxiv_id":"2607.27737","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Adding a clean teacher to Information Bottleneck Distillation raises clean accuracy by about 0.8 points on CIFAR data while holding robust accuracy nearly steady.","lead":"Researchers add a second “clean” teacher to a robust-distillation method, lifting clean-image accuracy by about 0.8 points while keeping adversarial accuracy roughly unchanged. Tests on CIFAR-10 and CIFAR-100 show a slightly better clean/robust balance than the baseline, but the gains are small and significance is not established.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed gains are a few tenths of a harmonic-mean point with no seeds or error bars, and Table 4 contains an unreproducible row; a multi-seed re-run is needed before the central improvement can be trusted.","rationale":"The reader's weakest assumption is about baseline/competitor comparability; that concern is valid but somewhat secondary because the headline over IBD does not require the external SOTA numbers to be fair. My focus is on the lack of repeated runs and the arithmetic inconsistencies in Table 4, which directly threaten the small claimed improvements (e.g., 64.16 vs 63.96 and 38.79 vs 38.76). If a multi-seed rerun shows JD consistently above IBD by more than seed noise, the central claim would stand; if not, the paper's conclusion collapses. I do not see a mathematical flaw in the proposed losses in Eqs. (4) and (5); they are coherent extensions of IBD. The paper also provides an anonymous code link, which makes the proposed concrete test feasible. Since the reader already reached CONDITIONAL and my check would either strengthen or weaken that condition, the verdict should remain unchanged pending the suggested experiment.","tokens_in":10811,"tokens_out":7890,"duration_ms":80139,"concrete_test":"Run the released code for IBD and JD (and DD) on CIFAR-10 and CIFAR-100 with 5 random seeds each, using the paper's stated optimal configurations. Report mean ± std for clean accuracy, AutoAttack accuracy, and harmonic mean, and perform a paired comparison (e.g., paired bootstrap or t-test) between JD and IBD over the same attack batches. Also recompute every harmonic mean in Table 4 from the raw accuracies in Tables 1 and 3; if the selective DD row still cannot be reproduced, correct Table 4 and re-evaluate whether the 'optimal configuration' conclusion survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on very small harmonic-mean differences: JD vs IBD under AutoAttack is 64.16 vs 63.96 on CIFAR-10 (Table 2/4) and 38.79 vs 38.76 on CIFAR-100 (Table 6). No seed count, standard deviation, or significance test is reported, and the paper post-hoc selects each method's best regularization setup (IBD selective; DD/JD global; §4.3, Table 4). Since adversarial-training runs typically vary by more than 0.2–0.8 pp across seeds, the headline advantage could be seed noise. This is compounded by an internal arithmetic inconsistency: Table 4's selective DD row cannot be produced from Table 3. With clean=84.37 and AA=51.40, the harmonic mean is 63.88, not the printed 64.09; the row's clean value 84.06 also does not match Table 3's 84.37, and the PGD harmonic mean is similarly off. The table used to support the 'optimal configuration' claim is therefore not internally reliable, making it unsafe to conclude that the small improvements are real rather than a reporting artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two dual-teacher distillation methods, Double Distillation (DD) and Joint Distillation (JD), that extend the Information Bottleneck Distillation (IBD) framework by adding a clean-teacher stream to the existing robust-teacher stream. DD uses clean-teacher soft labels for clean inputs and robust-teacher soft labels and attention-weighted features for adversarial inputs (Eq. 4); JD additionally distills clean-teacher features on clean inputs via a separate learned attention matrix (Eq. 5). The methods are evaluated on CIFAR-10 and CIFAR-100 with a Pre-Activation ResNet-18 student and WRN teachers, under two regularization/attack-budget configurations. The paper reports improved clean accuracy over IBD at similar robust accuracy, and states that in each method's optimal configuration the proposed methods achieve higher harmonic means than IBD. It also compares against state-of-the-art adversarial training and distillation methods, reports the effect of student size, and analyzes attention-weight behavior under the two training setups.","tokens_in":11121,"tokens_out":3318,"duration_ms":35802,"significance":"If the reported improvements are real, the paper makes a useful empirical contribution: a simple and well-motivated extension of IBD that shifts the clean/robust tradeoff in a favorable direction, particularly for larger student models, and that is competitive with recent dual-teacher frameworks such as B-MTARD. The analysis of attention weight convergence under different regularization regimes is also informative. The claims are, however, entirely empirical and rest on very small differences in harmonic mean (often a few tenths of a percentage point), so the strength of the contribution currently depends on statistical reliability and experimental comparability that the manuscript does not demonstrate. The authors provide anonymous code, which is a positive step, but no proofs or reproducible experimental protocol details are included for the comparison baselines.","major_comments":[{"comment":"The 'optimal configuration' comparison is not internally reproducible. For the selective DD row, Table 3 reports clean=84.37 and AA=51.40; Eq. (6) gives harmonic mean 2*84.37*51.40/(84.37+51.40)=63.88, not the printed 64.09. Similarly, the clean value 84.06 in Table 4 does not match Table 3's 84.37, and the PGD harmonic mean is also inconsistent (66.08 vs. the implied 66.04). Since the conclusion 'each method evaluated in its respective optimal configuration' relies directly on Table 4, this arithmetic inconsistency undermines the central empirical claim until corrected.","section":"Table 4, Section 4.3"},{"comment":"No multiple seeds, standard deviations, or significance tests are reported. The headline advantages over IBD are small (e.g., AA harmonic mean 64.16 vs. 63.96 on CIFAR-10; 38.79 vs. 38.76 on CIFAR-100), and adversarial training runs generally vary by more than these differences across seeds. Without at least three seeds with error bars, or a paired significance test, the claimed improvement could be seed noise. This is load-bearing because the paper's central claim is a modest but consistent harmonic-mean gain.","section":"Tables 1-8, Section 4"},{"comment":"The provenance and comparability of the baseline and comparison numbers are not specified. The IBD baseline is described only as 'a reproduction' with no indication of whether the authors' code or a reimplementation was used. The AT, TRADES, ARD, RSLAD, HBaR, InfoAT, and B-MTARD results in Table 6 are not accompanied by the evaluation protocol (same attack implementations, same student architecture, same training budget, same hyperparameters). If these numbers are copied from prior papers rather than run under identical conditions, the 'competitive with state-of-the-art' claim is not supported. The authors should state this explicitly or rerun the comparisons.","section":"Section 4.1, Table 6"},{"comment":"The selection of each method's 'optimal configuration' is performed post hoc on the evaluation tables (global for DD/JD, selective for IBD). No held-out validation set or model selection procedure is described. Because the harmonic mean differences are within run-to-run variability, selecting the best configuration based on the reported test numbers risks overfitting the conclusion to this particular run. A pre-specified protocol or a validation-based selection would make the 'optimal configuration' claim trustworthy.","section":"Section 4.3, Table 4"}],"minor_comments":[{"comment":"The selective setup changes both the L2 regularization and the number of PGD training steps (10 to 20) relative to the global setup, so the 'regularization setup' comparison is confounded with attack budget. This should be acknowledged in the interpretation.","section":"Section 4.3"},{"comment":"The column header 'Normal Values' is unclear; presumably it refers to the raw accuracy values rather than harmonic means. Please reword.","section":"Table 2"},{"comment":"There are typographical inconsistencies: 'Joint Distillations' should be 'Joint Distillation', and the 'best results' formatting is inconsistent across tables (e.g., Table 5 does not bold the best row).","section":"Tables 4 and 5"},{"comment":"Reference [17] cites 'ResNet-18' as a dataset with a TIB service URL; this should be a proper citation to He et al. (2016) for Pre-Activation ResNet, and the dataset URL is inappropriate.","section":"References"},{"comment":"Figure 5 is referenced in the text but does not appear in the manuscript; the caption mentions a non-uniform y-axis scale, which should be made explicit in the figure if included.","section":"Figure 5"},{"comment":"The notation y_clean and y_rob denotes teacher soft labels, but L_CE usually denotes cross-entropy with hard labels. Please clarify the temperature and soft-target treatment, or rename the loss term.","section":"Eq. (4), Eq. (5)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is reasonable and the experiments point in a plausible direction, but the manuscript is not yet publishable because the central claims rest on very small differences without seed-level statistics, the key comparison table contains arithmetic inconsistencies, and the baseline/comparison provenance is not documented. These are fixable with additional experiments and a corrected table, so I do not recommend rejection, but the revision must include multi-seed results and a clear experimental protocol. I would also ask the editor to ensure the Table 4 discrepancy is resolved in the revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this paper grafts a clean teacher onto the IBD distillation loss, giving two new losses (DD and JD). The idea is simple, the writing is clear, and the code is available. But the reported gains are a few tenths of a harmonic-mean point, there are no error bars, and one of the key tables doesn't add up. I'd send it to review, but only with the expectation that the authors re-run with multiple seeds and fix the arithmetic.\n\nWhat's actually new: the combination of a clean teacher's soft labels (and, in JD, features) with IBD's attention-weighted feature distillation. Equations (4) and (5) are direct extensions of IBD's loss, not a new principle, but that's okay — the paper isn't claiming a new principle. The student-size analysis (Table 5) is genuinely useful: it shows IBD is better for tiny students while the dual-teacher methods catch up as the student grows. The attention analysis under global vs selective regularization is also informative. And they're honest about the cost: ~1.5x training time, nearly double the parameters.\n\nThe soft spots are real and load-bearing. First, no multiple seeds or standard deviations anywhere. Adversarial training runs typically swing by more than 0.5 points, and the headline AA harmonic-mean advantage over IBD is 64.16 vs 63.96 on CIFAR-10 and 38.79 vs 38.76 on CIFAR-100. Those differences could easily be seed noise. Second, the authors pick each method's best setup after seeing the results — global for DD/JD, selective for IBD — which is honest but still post-hoc selection; it inflates the comparison. Third, and most concerning, Table 4 does not match Table 3. The selective DD row in Table 4 reports clean accuracy 84.06 and harmonic means that don't follow from the values in Table 3 (84.37 clean, 51.40 AA; harmonic mean should be 63.88, not 64.09). That inconsistency makes the \"optimal configuration\" table unreliable, and the paper needs to correct it and explain which numbers are right.\n\nAlso worth flagging: the Table 6 baselines don't say whether AT/TRADES/ARD/etc. numbers were run under matched conditions or taken from prior papers. Given the claimed gains are so small, baseline comparability matters.\n\nOverall: the direction of the claim is plausible, and the method is a legitimate incremental extension. But as written, the central conclusion — consistently higher harmonic mean than IBD — is not established. This deserves a serious referee, but with the expectation that the authors add seeds, significance statements, and fix the table. It's a solid workshop-quality contribution in need of stronger evidence.\n\nRecommendation: accept for peer review, not desk reject; require revision before any accept.","headline":"Modest dual-teacher twist on IBD with a plausible claim, but the evidence as reported is too fragile to trust: no seeds, post-hoc setup selection, and an inconsistent table.","tokens_in":11602,"tokens_out":2525,"would_cite":false,"duration_ms":25482,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dual-teacher distillation lifts clean accuracy while preserving adversarial robustness.","keywords":["Information Bottleneck Distillation","adversarial robustness","knowledge distillation","dual-teacher","accuracy-robustness tradeoff","harmonic mean","attention matrix","CIFAR-10"],"falsifier":"Re-run the comparison with the original IBD authors' released code (or a documented re-implementation), identical student architecture, training budget, attack protocol, and the same reported hyperparameters; if IBD reproduces at 84% clean / 52% AutoAttack on CIFAR-10, or if any of the external baseline numbers in Table 6 change by as little as 0.3 points, the claimed harmonic-mean advantage of the dual-teacher methods disappears.","tokens_in":10715,"feed_emoji":"⚖️","tokens_out":7653,"duration_ms":70882,"temperature":0.7,"pith_summary":"This paper tries to establish that the familiar accuracy/robustness tradeoff in adversarially robust distillation can be eased by giving the student a second teacher trained only on clean inputs. It extends Information Bottleneck Distillation (IBD) with two architectures, Double Distillation and Joint Distillation, that align the student's intermediate features with a robust teacher on adversarial inputs and, in the joint variant, with a clean teacher on natural inputs. On CIFAR-10 and CIFAR-100, the new methods hold adversarial accuracy roughly constant while raising clean accuracy, so the harmonic mean of clean and robust accuracy — the paper's headline tradeoff metric — is consistently a few tenths of a point above IBD's. The improvement grows with student model size, and the paper shows the attention-based feature transfer matters even when its learned weights collapse to a uniform pattern. If the results hold, the recipe is simple: a clean teacher plus separate attention matrices for each teacher buys back part of the clean accuracy that adversarial training gives up, at about 1.5 times the training cost.","feed_headline":"Dual-teacher distillation lifts clean accuracy, keeps robustness","feed_subtitle":"A clean-data second teacher improves the accuracy-robustness balance without hurting AutoAttack scores.","key_machinery":"The machinery is the 'clean teacher,' a network trained only on natural images, added alongside the adversarial 'robust teacher' of the original IBD loss. In Joint Distillation, two separate learned cross-layer attention matrices weight the feature-consistency (compression) terms: one matrix aligns the student's features with the clean teacher's on clean inputs, the other with the robust teacher's on adversarial inputs. The same objective mixes clean-teacher soft labels on natural samples with robust-teacher soft labels on perturbed samples. The paper demonstrates the module's function by removing it: in the 'global regularization' setting, where the attention weights converge to a uniform r","core_discovery":"In the authors' terms, their methods 'improve classification accuracy on clean samples compared to the original IBD, while maintaining similar accuracy on adversarial samples,' and under each method's optimal configuration they 'consistently achieve a higher harmonic mean than the baseline IBD.' The strongest evidence is Table 6: on CIFAR-10, Joint Distillation reaches 84.02% clean accuracy and 51.89% AutoAttack accuracy (harmonic mean 64.16), against 83.62%, 51.78%, and 63.96 for IBD; on CIFAR-100 the corresponding numbers are 59.45%, 28.79% (38.79) versus 58.28%, 29.04% (38.76). The gain is real but measured in tenths of a percentage point. The paper's own negative result is that an adapti","pith_inferences":["The reported gains are on the order of 0.2–0.4 points in harmonic mean; if these are replicated by other groups, the method is an incremental improvement, not a step change.","Because the attention module is beneficial even when its weights become uniform, the early training signal of attention may be doing the work; a testable extension would be to freeze or schedule the attention matrix after a few epochs to save compute.","The capacity-dependent gains suggest a hypothesis: below some student capacity, a second teacher's features overload the student; a curriculum that starts with the robust teacher and introduces the clean teacher later might transfer better to small students.","The negative result on adaptive balancing hints that the balancing rule sharpens clean-teacher influence too aggressively; a softer temperature schedule that only lowers the robust teacher's temperature, or a per-layer weighting, might preserve AutoAttack accuracy while recovering clean accuracy."],"forward_implications":["On both CIFAR-10 and CIFAR-100, Joint Distillation attains the highest harmonic mean under AutoAttack among all compared methods (64.16 vs 63.96 for IBD on CIFAR-10; 38.79 vs 38.76 on CIFAR-100).","The clean accuracy gain over IBD appears at every value of the trade-off parameter α tested (0.85–0.95), up to 84.34% for JD vs 83.65% for IBD.","For students at least one-quarter the size of the teacher, the dual-teacher methods match or exceed IBD's robust accuracy; only at the smallest size (1M parameters) does IBD keep a clear edge.","Replacing the fixed tradeoff weight with the adaptive balancing scheme from a recent dual-teacher approach (B-DD/B-JD) raised clean accuracy (up to 86.77%) but hurt AutoAttack robustness, so the authors kept the fixed α.","The proposed methods cost about 1.5× the training time and about 1.7× the parameters of IBD, so the tradeoff gain is bought with extra compute."],"fun_headline_variants":["Dual teachers tip accuracy-robustness tradeoff in your favor","Clean accuracy up, robustness flat with dual-teacher distillation","Two teachers beat one: better clean accuracy, same robustness","Dual-teacher distillation lifts clean accuracy, holds robustness","Small clean-accuracy gain, same robustness: dual-teacher pays off"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The conclusions rest on the trustworthiness of the reproduced IBD baseline and the comparability of all the comparison numbers, because the claimed advantages are differences of a few tenths of a percentage point; the paper does not state whether the IBD baseline and the state-of-the-art results in Table 6 were obtained under identical, reproducible conditions.","fun_headline_variants_meta":{"raw":{"variants":["Dual teachers tip accuracy-robustness tradeoff in your favor","Clean accuracy up, robustness flat with dual-teacher distillation","Two teachers beat one: better clean accuracy, same robustness","Dual-teacher distillation lifts clean accuracy, holds robustness","Small clean-accuracy gain, same robustness: dual-teacher pays off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00153,"raw_usage":{"total_tokens":5974,"prompt_tokens":770,"completion_tokens":5204,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":5131}},"tokens_in":514,"tokens_out":5204,"duration_ms":38678,"temperature":1.0,"reasoning_tokens":5131,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:14:48.469198+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the comparison with the original IBD authors' released code (or a documented re-implementation), identical student architecture, training budget, attack protocol, and the same reported hyperparameters; if IBD reproduces at 84% clean / 52% AutoAttack on CIFAR-10, or if any of the external baseline numbers in Table 6 change by as little as 0.3 points, the claimed harmonic-mean advantage of the dual-teacher methods disappears.","supporting_citations":[],"review_version":1}