{"id":"7bc99e15-cb6d-44b2-ae25-988fb50f5f2d","arxiv_id":"2506.08611","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A label-based adversarial robustness distillation method, ABSLD, re-temperates teacher soft labels per class to shrink the student's class-wise robust error gap and improves worst-class robustness and normalized standard deviation on CIFAR-10, CIFAR-100, and Tiny-ImageNet.","lead":"Adversarial training makes neural nets robust to small image perturbations, but some classes remain much easier to attack than others. This paper shows that adjusting the per-class sharpness of the teacher's soft labels during knowledge distillation, giving harder classes sharper labels, improves the worst-class robustness of small models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorems 1 and 2, the sole theoretical basis for the temperature update in Eq. (10), are not proven in the submitted text; the sign of the temperature-to-gap relationship is assumed, not established.","rationale":"The paper's stated contribution is a theoretical explanation (Theorems 1 and 2) plus a method (ABSLD) whose update rule is directly derived from Theorem 2. The strongest empirical claim—SOTA worst-class robustness and NSD across datasets—is supported by multiple tables. However, the theory is the only justification for the sign of the temperature update; without it, ABSLD is an ad-hoc heuristic that happens to work in the reported settings. The missing appendices are therefore not a cosmetic issue. The teacher-correctness assumption is cited to [41] and is plausible, but even if the teacher is more accurate on average, the relevant condition is per-class: for every class where the student risk is above average, the teacher's softened distribution must be informative enough that lowering temperature increases learning intensity and reduces that class's risk. If the student is already better than the teacher on some hard class, the update could harm. The proposed check—re-deriving the derivative signs—would settle whether the monotonic relationship holds under the paper's stated assumptions or requires extra constraints. The reader's conditional verdict is appropriate; our stress test does not shift it.","tokens_in":23742,"tokens_out":4325,"duration_ms":49057,"concrete_test":"Independently re-derive Theorem 2 from the KL-risk decomposition in Section 3.2, computing the partial derivatives of the student's class-wise error-risk gap (Eq. 7) with respect to τ_c+ and τ_c− for the ABSLD loss (Eq. 11), over the temperature range [0.5, 5] used in the paper. If the derivatives do not have the claimed signs for all feasible teacher distributions satisfying the correctness assumption, or if the proof requires additional conditions not stated in the paper, the central mechanism is not established. Alternatively, run a 2-class synthetic experiment with a fixed teacher and known class-wise correctness to measure the actual sign; if the gap increases in any regime with teacher-correctness as defined, the update rule in Eq. (10) is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that per-class soft-label temperature is a controllable cause of adversarial robust fairness—rests entirely on Theorems 1 and 2 (Section 3.2). Both proofs are deferred to appendices A and B that are not present in the reviewed arXiv v1 text. Theorem 2 is stated only as a qualitative correlation: temperature of hard class positively correlated with the error-risk gap, easy class negatively correlated. The proof would need to show, for the actual KL distillation loss in Eq. (11) and the update in Eq. (10), that ∂(R_c+ - R_c-)/∂τ_c+ > 0 and ∂(R_c+ - R_c-)/∂τ_c- < 0 under the 'teacher more correct than student' assumption borrowed from [41]. This is load-bearing because Eq. (10) applies a sign-based update: any class with above-average risk gets a lower temperature. If the derivative changes sign when the student approaches or exceeds teacher accuracy on a class, or when the teacher itself is weak on the hardest classes, the update would increase the very gap it is designed to reduce. No convergence or stability analysis of the iterative temperature schedule is given, and the only justification for the teacher-correctness premise is a citation to [41]. The empirical results are consistent with the mechanism but do not independently verify it, since the same experiments also include the adaptive update.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies adversarial robust fairness in adversarial training (AT) and adversarial robustness distillation (ARD). It argues that the class-wise smoothness degree of soft labels is an important factor: sharper soft labels for hard classes and smoother soft labels for easy classes can reduce the class-wise error-risk gap. The authors support this with an empirical observation (Fig. 2), a corollary and two theorems (Section 3.2), and then propose ABSLD (Anti-Bias Soft Label Distillation), which adaptively adjusts the temperature of the teacher’s soft labels for each class according to the student’s class-wise error risk (Eq. (10)). They also study combinations with sample-based re-weighting and re-margining. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet report improved worst-class robustness and lower NSD compared with several baselines.","tokens_in":24059,"tokens_out":4421,"duration_ms":54666,"significance":"If the causal claim is established, the label-based perspective on robust fairness is a useful complement to existing sample-based methods. The proposed ABSLD is simple, adaptable, and the paper includes a released codebase and a broad evaluation across datasets, architectures, and attacks. The strongest contribution is the idea that the smoothness of teacher soft labels can be tuned class-wise to steer fairness, supported by the static-label experiment in Fig. 2. However, the theoretical backbone (Theorems 1 and 2) is not verifiable as submitted because the proofs are deferred to absent appendices, and the empirical evidence is weakened by checkpoint selection based on the evaluation metric, lack of error bars, and a partially circular optimization check in Fig. 5.","major_comments":[{"comment":"Theorems 1 and 2 are the theoretical justification for the sign-based temperature update in Eq. (10), but their proofs are deferred to Appendices A and B, neither of which appears in the reviewed text. Theorem 2 is stated only as a qualitative 'positively/negatively correlated' claim; it should be stated as explicit inequalities on the relevant derivatives, e.g., d(R_c+ - R_c-)/dtau_c+ > 0 and d(R_c+ - R_c-)/dtau_c- < 0, for the actual KL distillation loss in Eq. (11). The proof must also address the regime where the student class-wise risk approaches or exceeds the teacher’s, since the premise borrowed from [41] that the teacher is more correct than the student even in worst classes is not verified for the specific teachers used here (WRN-34-10, WRN-70-16, PreActResNet-34). Without these proofs, the central mechanism of ABSLD is unverified as submitted.","section":"Section 3.2; Section 4.2, Eq. (10)"},{"comment":"Model checkpoints are selected based on the highest mean value of all-class average robustness and worst-class robustness, which are the same quantities used to report the fairness results. This selection protocol can inflate the observed improvements because it directly optimizes the evaluation metric. Please report results with a fixed training schedule or a validation-based checkpoint selection, and provide standard deviations over at least three independent runs for the main tables. The reported gains of 0.7 to 2.1 percentage points in worst-class robustness and NSD reductions of 0.01 to 0.08 are otherwise difficult to interpret without variance estimates.","section":"Section 5.1, Tables 1–4"},{"comment":"Figure 5 shows that ABSLD reduces the standard deviation of class-wise optimization error risk during training. Since Eq. (10) is explicitly designed to reduce exactly this gap, the observed decrease is by construction and is not independent evidence that the smoothness of soft labels is the causal mechanism. The static-temperature experiment in Fig. 2 is the more convincing independent probe, but it lacks error bars and statistical comparisons. In Table 5, the adaptive strategy is compared only with a manually chosen extreme temperature setting (0.5 vs. 5) and not with a uniform-temperature baseline; please add that control and report multiple runs so the reader can separate the effect of re-tempering from the effect of the adaptive schedule.","section":"Section 5.2.1, Eq. (10), Fig. 5"}],"minor_comments":[{"comment":"If all class risks are equal to the mean, the denominator max(|R(f_s(x_k)) - average|) is zero; specify the tie-breaking rule, for example by skipping the update when the gap is zero.","section":"Eq. (10)"},{"comment":"The third contribution bullet writes 'ABLSD' instead of 'ABSLD'; please correct the typo.","section":"Section 1, contribution list"},{"comment":"Eq. (9) is written as an equality constraint, but the training procedure does not enforce it exactly; clarify that it is a target condition for the adaptive update rather than a hard constraint in the optimization.","section":"Section 4.1, Eq. (9)"},{"comment":"Please state explicitly that NSD is computed from the standard deviation of class-wise robust accuracies across classes, not from per-sample losses; the current definition (SD/Avg.) does not specify the unit over which SD is taken.","section":"Section 5.1, NSD definition"},{"comment":"Several table entries are missing spaces, for example '0.552.2132.80' in Table 7 and '0.151.7732.80' in Table 8; fix the formatting for readability.","section":"Tables 7 and 8"},{"comment":"The caption reads 'between trainset and testset between sample-based method and label-based method'; please rephrase to clarify the comparison being shown.","section":"Figure 3 caption"}],"recommendation":"major_revision","confidential_remarks":"The main blockers are the missing appendix proofs for the two theorems that motivate Eq. (10), the checkpoint selection protocol, and the lack of variance estimates. If the authors provide complete proofs and rerun the main experiments with fixed checkpoints and multiple seeds, the paper would be suitable for reconsideration. I have no conflicts of interest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing about this one. First, it has a genuinely different angle on robust fairness in adversarial training: instead of re-weighting samples or re-margining perturbation budgets, it adjusts the temperature of the teacher's soft labels per class. That label-based perspective is a real departure from the sample-based methods in the literature, and the empirical evidence that it works is consistent across CIFAR-10, CIFAR-100, and Tiny-ImageNet, with two architectures. Second, the theoretical justification for the central temperature update is not actually in this manuscript. Theorems 1 and 2 are deferred to appendices that aren't present in the arXiv v1, and Theorem 2, which is the load-bearing result for Eq. (10), is stated only as a qualitative correlation between temperature and the class-wise error-risk gap. The sign of that correlation is exactly what the update relies on.\n\nWhat the paper does well: the motivating experiment in Fig. 2 tests class-wise smoothing directly and supports the direction of the claimed effect. The manual re-tempering ablation in Table 5 and the label-smoothing comparison in Table 6 give additional evidence that the mechanism is real, not just an artifact of the adaptive update. The combination with re-weighting and re-margining is a clean way to show the label-based and sample-based ideas don't collide. The overfitting analysis in Fig. 3 is a nice extra. Gains are modest—0.7 to 2.1 points on worst-class robustness, some NSD reduction—but they measure up well against the baselines.\n\nSoft spots in proportion. The missing proofs are the main one. If the derivative in Theorem 2 has the wrong sign when the student approaches teacher accuracy on a class, the update could increase the very gap it targets. The paper borrows the 'teacher is more correct' premise from [41], but that doesn't remove the need to show the derivative's sign for the actual KL loss. I'd also want error bars or multiple seeds; the reported gains are small enough that a single run could change the ranking. Checkpoint selection on the evaluation metric is a known inflationary practice, and the trade-off parameter alpha in Eq. (11) is never reported. None of this is fatal, and the NeurIPS version already established the core method; this journal version's novelty is the combination and the overfitting discussion, which is honest scope.\n\nWho benefits: practitioners and researchers working on robust fairness or adversarial distillation, especially anyone who wants a training-time knob that composes with sample-based methods. It deserves a real referee, but the referee should insist on the deferred proofs, variance estimates, and full hyperparameter reporting before acceptance.","headline":"A useful label-based angle on robust fairness, but the theory is deferred and the gains need variance.","tokens_in":24602,"tokens_out":2905,"would_cite":true,"duration_ms":35515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Class-wise robustness fairness can be steered by the smoothness of teacher soft labels: ABSLD assigns sharper labels to hard classes and smoother labels to easy classes and reports narrower class-wise error-risk gaps and better…","keywords":["adversarial robustness","robust fairness","knowledge distillation","adversarial robustness distillation","soft labels","class-wise error risk","temperature scaling","worst-class robustness"],"falsifier":"Train the student with ABSLD for a setting in which the teacher's class-wise accuracy is known and deliberately lower than the student's on a held-out class (e.g., teacher trained without that class's data). If the per-class temperature update still converges to lower temperatures for that class but worst-class robustness does not improve, or if the improvement tracks the teacher's accuracy rather than the temperature update, then the proposed causal mechanism is falsified. A cheaper check is to measure the empirical sign of the correlation in Theorem 2: for a fixed hard class, raising the teacher temperature should increase the student's error-risk gap; if the opposite sign is observed on any dataset, the theorem's premise fails.","tokens_in":23512,"feed_emoji":"⚖️","tokens_out":8048,"duration_ms":87192,"temperature":0.7,"pith_summary":"Adversarially trained models protect some classes well while leaving intrinsically harder classes vulnerable; this class-wise gap is the robust fairness problem. The paper claims that a controllable cause of that gap is the smoothness of the soft labels used in training: when hard classes receive sharper teacher soft labels and easy classes receive smoother ones, the class-wise error-risk gap narrows. Building on this, the authors propose Anti-Bias Soft Label Distillation (ABSLD), which sets a separate temperature for each class's teacher soft labels and adjusts those temperatures each epoch from the student's own per-class error risk. If the claim holds, adversarial fairness can be improved from the label side alone, without changing sample weights or perturbation budgets, and the method can be combined with sample-based fairness approaches.","feed_headline":"Sharper labels for hard classes shrink the robustness gap","feed_subtitle":"This distillation method tunes teacher soft-label sharpness per class and lifts worst-class robustness under attack.","key_machinery":"The central object is the teacher's per-class temperature pair ($\\tau_t^k$ for clean and $\\tilde{\\tau}_t^k$ for adversarial examples), which controls how sharply the teacher's soft-label distribution is peaked through a tempered softmax. The mechanism that does the work is the online update of Eq. (10): after each epoch, the student's class-wise robust error risk $R(f_s(\\tilde{x}_k))$ is compared with its mean across classes, and classes above the mean are assigned lower temperatures (sharper supervision) while classes below are assigned higher temperatures (smoother supervision). Theorem 2 supplies the claimed sign relationship between these temperatures and the student's error-risk gap, and the KL-based loss of Eq. (11) carries the adjusted teacher labels into the student's optimization.","core_discovery":"The paper's central claim is that class-wise smoothness of teacher soft labels is not a side effect but a steering mechanism for adversarial robust fairness. Corollary 1 and Theorem 1 argue that training with equally smooth labels forces an inherent error-risk gap between hard and easy classes, while a label distribution that is sharper for hard classes and smoother for easy classes strictly reduces that gap. Theorem 2 carries the same logic into knowledge distillation, asserting that the hard class's teacher temperature is positively correlated and the easy class's teacher temperature is negatively correlated with the student's error-risk gap. The resulting per-class temperature update in Eq. (10) lowers the temperature for classes whose student error risk is above average and raises it for classes below the average, and the experiments report that this raises worst-class robustness and lowers Normalized Standard Deviation on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and that adding re-weighting and re-margining improves it further.","pith_inferences":["A testable extension: the same per-class label-tempering rule should apply to plain (non-adversarial) knowledge distillation and long-tailed classification, whenever the teacher is more accurate than the student; running ABSLD on a clean-only task would show whether the worst-class accuracy gap closes there too.","A boundary the paper leaves implicit: if a teacher is not more correct than the student on some class, the optimal temperature sign could reverse; a practical variant could monitor per-class teacher-vs-student agreement and freeze or invert updates for classes where the premise fails.","The online gradient update on temperatures introduces two extra hyperparameters (initial temperature and temperature learning rate); a closed-form schedule computed from error-risk ranks would remove those choices and make the mechanism easier to transfer across datasets."],"forward_implications":["A per-class temperature schedule can be applied to any frozen robust teacher, so fair distillation does not require retraining or modifying the teacher.","Label-based fairness and sample-based fairness are complementary: re-weighting and re-margining can be stacked on ABSLD and yield further gains in worst-class robustness.","Models produced by ABSLD should show lower class-to-class standard deviation of adversarial accuracy and higher worst-class accuracy under FGSM, PGD, CW∞, and AutoAttack, with a modest average-robustness cost.","Because the same class-wise bias appears across training strategies, the label-side fix should transfer to common corruptions as well, which the paper reports for Gaussian noise and channel transforms."],"supporting_citations":[{"why":"defines the class-wise adversarial error-risk formulation and the robustness-fairness trade-off that ABSLD targets.","marker":"[21]"},{"why":"provides the checkpoint-selection rule and the worst-10% class metric used to evaluate fairness fairly.","marker":"[39]"},{"why":"establishes that the teacher's prediction is more correct than the student's even on worst classes, the premise that makes Theorem 2's sign hold.","marker":"[41]"},{"why":"supplies the binary-class error-risk analysis behind Corollary 1 and the claim that clean and adversarial fairness are coupled.","marker":"[42]"},{"why":"provides the robust teacher models and a baseline robust training objective used throughout the experiments.","marker":"[44]"},{"why":"supplies the idea of using temperature to control soft-label smoothness in adversarial distillation, which ABSLD adapts per class.","marker":"[46]"},{"why":"is the RSLAD baseline whose teacher-soft-label loss formulation Eq. (11) extends and re-temperatures.","marker":"[49]"},{"why":"provides the standard adversarial training method used in the motivating empirical observation and as an AT baseline.","marker":"[22]"}],"fun_headline_variants":["Class-wise soft-label sharpness closes adversarial robustness gap","Per-class temperature tuning narrows adversarial fairness gap","Sharper hard-class labels improve worst-class robustness","Soft label sharpness shapes adversarial fairness gap","Distillation sharpens per-class labels to even out robustness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the teacher model is more correct than the student even on the hardest classes, and that sharpening the teacher's labels for a hard class reliably makes the student learn that class better.","fun_headline_variants_meta":{"raw":{"variants":["Class-wise soft-label sharpness closes adversarial robustness gap","Per-class temperature tuning narrows adversarial fairness gap","Sharper hard-class labels improve worst-class robustness","Soft label sharpness shapes adversarial fairness gap","Distillation sharpens per-class labels to even out robustness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1288,"prompt_tokens":961,"completion_tokens":327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":254}},"tokens_in":577,"tokens_out":327,"duration_ms":4375,"temperature":1.0,"reasoning_tokens":254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:06:54.405417+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the student with ABSLD for a setting in which the teacher's class-wise accuracy is known and deliberately lower than the student's on a held-out class (e.g., teacher trained without that class's data). If the per-class temperature update still converges to lower temperatures for that class but worst-class robustness does not improve, or if the improvement tracks the teacher's accuracy rather than the temperature update, then the proposed causal mechanism is falsified. A cheaper check is to measure the empirical sign of the correlation in Theorem 2: for a fixed hard class, raising the teacher temperature should increase the student's error-risk gap; if the opposite sign is observed on any dataset, the theorem's premise fails.","supporting_citations":[{"cited_title":"NeurIPS35, 26230–26241 (2022)","cited_arxiv_id":null,"evidence_quote":"defines the class-wise adversarial error-risk formulation and the robustness-fairness trade-off that ABSLD targets."},{"cited_title":"In: CVPR","cited_arxiv_id":null,"evidence_quote":"provides the checkpoint-selection rule and the worst-10% class metric used to evaluate fairness fairly."},{"cited_title":"NeurIPS (2023)","cited_arxiv_id":null,"evidence_quote":"establishes that the teacher's prediction is more correct than the student's even on worst classes, the premise that makes Theorem 2's sign hold."},{"cited_title":"In: ICML","cited_arxiv_id":null,"evidence_quote":"supplies the binary-class error-risk analysis behind Corollary 1 and the claim that clean and adversarial fairness are coupled."},{"cited_title":"In: ICML","cited_arxiv_id":null,"evidence_quote":"provides the robust teacher models and a baseline robust training objective used throughout the experiments."},{"cited_title":"TPAMI (01), 1–14 (2024)","cited_arxiv_id":null,"evidence_quote":"supplies the idea of using temperature to control soft-label smoothness in adversarial distillation, which ABSLD adapts per class."},{"cited_title":"In: ICCV (2021) Shiji Zhaoreceived his B.S","cited_arxiv_id":null,"evidence_quote":"is the RSLAD baseline whose teacher-soft-label loss formulation Eq. (11) extends and re-temperatures."}],"review_version":1}