Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A confidence-guided diffusion method generates the images a biased student gets wrong and fixes worst-group accuracy under covariate shift.

desk verdict Solid empirical paper on distillation under covariate shift with a genuinely new augmentation idea; the theoretical appendix does not prove what it claims, but the experiments stand on their own. read the letter →

arxiv 2506.02294 v3 pith:METLODOZ submitted 2025-06-02 cs.CV

Niclas Popp , Kevin Alexander Laube , Matthias Hein , Lukas Schott This is my paper
classification cs.CV
keywords knowledgedistillationcovariateshiftspuriousfeaturesdiffusion-baseddataaugmentationconfidenceguidanceworst-groupaccuracygrouprobustnessshortcutlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a student distilled from a robust teacher can be made robust to test groups that are entirely absent from its training data, where spurious features like hair color or background dominate the train set. The authors propose ConfiG, a diffusion-based augmentation method that generates images by maximizing the teacher's confidence in the true class while driving the student's confidence down, so the generated images remove spurious features rather than causal ones. Training on the real images plus these augmentations substantially improves worst-group accuracy on CelebA (from 38.9% to 66.1%) and SpuCo Birds (from 29.4% to 62.7%) compared with CutMix alone, and outperforms all diffusion-based augmentation baselines. The paper also proves a bound under which such augmentations provably reduce the covariate-shift term in the generalization error. A sympathetic reader would care because it shows unknown shortcuts can be addressed without group labels, as long as a robust teacher exists.

What carries the argument

The load-bearing object is the confidence-guided diffusion objective of Equation 7: starting from a real image encoded and inverted into a latent $z_0$ via null-text inversion, the method takes several AdamW steps on $L_{\text{ConfiG}}(z_0) = t(D(M_{\epsilon_\theta}(z_0)))_y^\gamma + (1 - f(D(M_{\epsilon_\theta}(z_0)))_y)^\gamma$ with $\gamma=2$. A preliminary auxiliary student, distilled on real images only, is used as $f$ so that it embodies the spurious-feature shortcut; the final student is then distilled on the union of real and generated images. The theoretical machinery decomposes the generalization error as $\Delta = \Psi + \Omega$, where $\Psi$ is the train-distilled risk gap (controlled by the teacher's approximation of the true conditional distribution) and $\Omega$ is the covariate-shift gap, and Proposition 1 shows ConfiG's augmentations shrink $|\Omega|$ under explicit bounds on teacher accuracy and student-teacher agreement.

What would settle it

Train a student with ConfiG using a deliberately non-robust teacher that is confident on the spurious feature but wrong on a known absent group (e.g., a CLIP teacher with 40.2% worst-group accuracy on SpuCo Birds). If the student's worst-group accuracy still rises substantially, the mechanism does not actually depend on teacher robustness; if it stays low, the teacher-robustness assumption is confirmed as load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that the disagreement between a robust teacher and a biased student is a sufficient guide for generating augmentation images that remove unknown spurious features. Concretely, ConfiG optimizes an inverted latent code from a real image through Stable Diffusion to maximize $t(D(M_{\epsilon_\theta}(z_0)))_y^\gamma + (1 - f(D(M_{\epsilon_\theta}(z_0)))_y)^\gamma$, keeping the teacher's probability for the true class high while forcing the student's probability low. The paper shows empirically that samples produced this way are the hardest for a student trained on real images (37.2% student accuracy on CelebA versus at least 67% for baselines), and that training a fresh student on the augmented set yields worst-group accuracy almost matching the CLIP teacher on CelebA (66.1% vs 66.7%). The theoretical result (Proposition 1) states that under bounded teacher error $\delta_t$ and student-teacher divergence $\delta_s$ on the training support, these augmentations reduce $|\Omega| = |R_{\text{test}}(f) - R_{\text{train}}(f)|$ as long as the confidence gap $\log(\tau/\sigma)$ is large enough and the augmentation weight $\alpha$ is kept small.

Load-bearing premise

The method needs a teacher that is already robust on the test groups that are absent from the student's training data; on SpuCo Birds the authors could only obtain such a teacher by fine-tuning on a balanced dataset that includes those absent groups, which they acknowledge would not be possible in practice.

Editorial extensions

If this is right

  • ConfiG raises worst-group accuracy under covariate shift to 66.1% on CelebA and 62.7% on SpuCo Birds with CutMix plus EDRM, versus 38.9% and 29.4% for CutMix alone.
  • On Spurious ImageNet, ConfiG achieves the highest spurious mAUC among all diffusion-based augmentation methods (81.5% for ViT-T, 86.4% for ViT-S).
  • ConfiG's generated samples are the most challenging to a student trained only on real images: 37.2% accuracy on CelebA and 64.8% on SpuCo Birds, well below every baseline, confirming the samples target the student's spurious-feature reliance.
  • Using only two ConfiG augmentations per real image outperforms all baselines even when they receive five or ten times more synthetic samples, indicating augmentation quality dominates quantity in this setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The confidence-gap signal could be used as a diagnostic: images where the teacher is confident and the student is not form a candidate set of unknown spurious-feature samples, so ConfiG doubles as an automatic bias-discovery tool (the paper's own observation that eye color changed on CelebA supports this).
  • Because the objective only requires differentiability through the generative model, the approach should transfer to other latent-editing models (GAN inversion, image editing) beyond diffusion, with the same teacher-student gap as the editing direction.
  • The theoretical bound predicts a tunable tradeoff: worst-group gain should grow with $\log(\tau/\sigma)$ and fall as the augmentation weight $\alpha$ grows, so varying the number of optimization steps and the number of augmentations per image should trace the bound; the paper's 2x augmentation optimum is consistent with this prediction.
  • In settings where no robust teacher exists (as on SpuCo Birds), one could attempt to synthesize one by ensembling several weak teachers or by using the diffusion model itself as an implicit robust teacher; the paper does not test this, but its assumptions point to it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes ConfiG, a diffusion-based data augmentation method for knowledge distillation under unknown covariate shift. The method generates synthetic training images by jointly maximizing the teacher's confidence in the true class and minimizing the student's confidence, thereby targeting spurious features that the student has learned from a biased training set. The authors introduce a theoretical framework (Proposition 1) intended to show that augmenting the training distribution with such samples provably reduces the distribution-shift gap |Ω|. Empirically, the paper evaluates ConfiG on CelebA, SpuCo Birds, and Spurious ImageNet, reporting improvements in worst-group and mean-group accuracy over several diffusion-based augmentation baselines, with additional experiments on student architectures, augmentation quantity, and computational cost.

Significance. If the empirical results are reliable, ConfiG is a practically motivated contribution to knowledge distillation under unknown covariate shift. The evaluation is broad: three datasets, three student architectures, multiple seeds, ablations on the number of augmentations, and a cost-quality analysis. The paper also explicitly discloses its main limitation regarding the need for a robust teacher. However, the theoretical guarantee in Appendix A.1 is the central claim that the method 'provably decreases |Ω|', and that proof is flawed in a load-bearing way. In addition, the SpuCo Birds experiment relies on a teacher fine-tuned on the groups that are supposed to be absent from the student's data, which weakens the claimed demonstration of transferring robustness in a realistic setting. The overall contribution is promising but needs major revisions to be acceptable.

major comments (3)
  1. [Appendix A.1, Proposition 1] The proof of Proposition 1 does not establish the claimed bound |Ω_aug| ≤ min{Ω−Θ, Ω/2}. From Ω_aug = Ω − α(R_Q(f)−R_train(f)) ≤ Ω − αΘ (Eq. 46) and α ≤ min{1, Ω/(2Θ)}, the paper concludes |Ω_aug| ≤ min{Ω−Θ, Ω/2}. This is algebraically incorrect: to obtain Ω_aug ≤ Ω/2 one needs α ≥ Ω/(2Θ), which is the reverse of the stated upper bound; to obtain Ω_aug ≤ Ω−Θ one needs α ≥ 1, which is not guaranteed. Moreover, the proof only lower-bounds R_Q(f)−R_train(f) by Θ. If R_Q(f)−R_train(f) is large and positive, Ω_aug becomes negative and |Ω_aug| is unbounded; no assumption in Proposition 1 upper-bounds this difference. Consequently, the statement 'provably decreases |Ω|' is unsupported. This is load-bearing because the theoretical motivation is explicitly cited in Section 3.2.
  2. [Appendix A.1, proof of Lemma 1 / Eq. (24)] The proof silently assumes p*(x)_y ≥ 1/L on all relevant supports, though this is not stated among the hypotheses of Proposition 1. Without this assumption, the bound in Eq. (30), log(p*_y/t_y) ≤ log(1/(1−δ_t L)), does not follow because p*_y could be arbitrarily small even when t_y ≤ p*_y + δ_t. The authors should either add this as an explicit assumption and discuss its plausibility, or revise the proof. Additionally, Eq. (42) derives a denominator 1/L − δ_t − δ_s, while the definition of Θ uses 1/L − δ_t − 2δ_s; this inconsistency should be corrected.
  3. [Section 4.1 and Limitations] On SpuCo Birds, the teacher is fine-tuned on the full balanced SpuCo Birds training set, which includes the groups that are absent from the student's training data. As the Limitations state, this is 'not possible in practice.' Consequently, the teacher has access to the very groups the student is missing, so the worst-group improvements on SpuCo Birds (Table 1) do not demonstrate ConfiG's effectiveness under unknown covariate shift where only a pretrained robust teacher is available. The CelebA results with a CLIP teacher are more convincing in this regard. Please either provide a SpuCo Birds result with a teacher that has not seen the absent groups (e.g., a zero-shot or weakly supervised teacher), or clearly frame the SpuCo Birds result as a proof-of-concept / upper bound and discuss what would be needed to obtain such a teacher in practice.
minor comments (4)
  1. [Appendix A.8] In the hyperparameter table, 'Tabel 10' should be 'Table 10'.
  2. [Figure 4 caption] The caption contains grammatical errors: 'ConfiG successfully identifies spurious features and changes removes in the augmentation process' should be rephrased, e.g., 'ConfiG successfully identifies spurious features and removes them in the augmentation process.'
  3. [Appendix A.1, Proposition 1 vs. Lemma 1] Proposition 1 states δ_t + δ_s < 1/L, but the proof of Lemma 1 uses δ_t + δ_s ≤ 1/L; choose one and keep it consistent.
  4. [Section 4.3] The sentence 'ConfiG surpasses all baselines by at least 0.5% mAUC' refers to the spurious score, but ConfiG is only second-best in validation accuracy (after DAFusion) on Spurious ImageNet; the abstract's wording 'best or second-best in sample mean accuracy' is more precise and should be reflected in the main text.

Circularity Check

2 steps flagged · score 2.0 of 10

No definitional circularity: the empirical core is benchmarked externally and the held-out metrics are genuine, but Proposition 1 does not prove the claimed decrease of |Ω| (reversed α-condition, risk gap only lower-bounded, unstated p*_y ≥ 1/L), and the SpuCo Birds teacher is acknowledged to have been fine-tuned on the full balanced set containing the absent groups.

  1. other [Appendix A.1, Proposition 1 proof, Eqs. (43)-(48); claim stated in Section 3.2 and in the contributions.]
    "Thus, Ωaug = Ω − α(RQ(f) − Rtrain(f)) ≤ Ω − αΘ. Since 0 < α ≤ min{1, Ω/(2Θ)} by assumption, we have |Ωaug| ≤ min{Ω − Θ, Ω/2}. (48) ... Under realistic assumptions, using the real training images together with augmentations generated through ConfiG provably decreases |Ω|. ... Under the assumptions stated in Proposition 1, for every x in the support of Ptest, Ptrain or Q one has p∗(x)y ≥ 1/L (24)."

    This is not a definitional reduction but an invalid inference that is flaggable under the reviewing rule because it is load-bearing for the 'provably decreases |Ω|' claim. The bound |Ωaug| ≤ Ω − Θ would require α ≥ 1, and |Ωaug| ≤ Ω/2 would require α(RQ(f) − Rtrain(f)) ≥ Ω/2, i.e., α ≥ Ω/(2(RQ(f) − Rtrain(f))) in the typical case; the stated assumption imposes α ≤ 1 and α ≤ Ω/(2Θ), the opposite directions. The proof only lower-bounds the gap Δ = RQ(f) − Rtrain(f) by Θ and never upper-bounds it, so a large positive Δ makes Ωaug arbitrarily negative and |Ωaug| unbounded. Lemma 1 additionally assumes p∗(x)y ≥ 1/L on all supports (Eq. 24), a hypothesis not stated in Proposition 1.

  2. other [Section 4.1 (Models, SpuCo Birds teacher) and Limitations.]
    "To create a reliable teacher, we use ViT-B [7] pretrained on ImageNet and allow it access to a balanced training set of SpuCo Birds for fine-tuning. ... we decided to obtain a stronger teacher by fine-tuning a ViT-B which would not be possible in practice."

    The method's premise is a teacher robust on the test groups absent from the student's training data, and for SpuCo Birds such a teacher could only be produced by fine-tuning on the full balanced training set that contains the absent groups. The student's worst-group accuracy of 62.7% is therefore partly inherited from a teacher fit on the very groups treated as unknown, so this portion of the demonstration reduces to a known-shift setting for the teacher. The authors disclose this ('not possible in practice'), the comparison with baselines is unaffected because all methods share the identical teacher, the test groups are held out, and the CelebA and Spurious ImageNet results use off-the-shelf teachers, so the circular element is mild and local.

full rationale

The central empirical claim - that ConfiG improves worst-group accuracy under covariate shift relative to diffusion-augmentation baselines - is derived from external benchmarks (CelebA, SpuCo Birds, Spurious ImageNet) with external baselines (GIF, DistDiff, DAFusion, ActGen, Stable Diffusion) under identical teachers, augmentation budgets, and training schedules. Hyperparameters and checkpoints are selected on validation subsets containing only the groups present in the training data, so the reported absent-group accuracies are genuine held-out measurements rather than fitted predictions. The self-citations (DiG-IN [3], DASH [2]) supply an implementation tool (latent-space guidance) that is code-reproduced and does not smuggle in the conclusion; no uniqueness theorem is imported. Two issues are flagged per the reviewing rule, but neither is a definitional reduction. First, Proposition 1's proof is invalid: the α-condition has the wrong direction, the risk gap is only lower-bounded so |Ωaug| is unbounded, and Eq. 24 assumes p∗(x)y ≥ 1/L without being stated; consequently the 'provably decreases |Ω|' guarantee is unsupported, which is a correctness risk for the theoretical motivation rather than a circularity. Second, the SpuCo Birds teacher was fine-tuned on the balanced training set containing the absent groups, an acknowledged 'not possible in practice' setup that lends a mild circular element to that part of the empirical story while leaving the relative comparison to baselines intact. Overall score 2: the evaluation is essentially self-contained against external benchmarks, with one mild disclosed circular element and an unsupported but not self-referential theoretical proof.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method's contribution relies mainly on the availability of a robust teacher and on the diffusion model's editing fidelity, neither of which is provided by the paper with quantitative verification. The theoretical appendix adds an unstated lower-bound assumption on p* and misuses the mixing coefficient alpha, so the proof is not a reliable source of support.

free parameters (5)
  • gamma = 2
    Exponent in Eq. 7 balancing teacher and student confidence terms; chosen empirically ('we find gamma=2 to perform well') with no ablation or sensitivity analysis.
  • number of augmentations per real image = 2
    Selected by ablation on CelebA (Fig. 5); the number changes the effective alpha in the theoretical mixing distribution.
  • latent optimization steps = 5
    Number of AdamW steps in latent space for ConfiG augmentation; fixed without ablation, and it drives the 11.4 sec/image cost.
  • latent optimization learning rate = 0.01
    AdamW learning rate for optimizing z0 in Eq. 7; fixed in Appendix A.10 without sensitivity analysis.
  • per-dataset student learning rate and weight decay = varies (Table 10)
    Tuned for each dataset and training type on a validation subset containing only training-present groups; standard but part of the reported pipeline.
assumptions (5)
  • domain assumption A robust teacher exists that approximates p*(x) on Ptrain and Ptest (and on Q)
    Adopted in Sec. 3.2 and required by Proposition 1. The SpuCo Birds experiment can only satisfy it by fine-tuning the teacher on the full balanced dataset, which is unavailable in the intended scenario.
  • ad hoc to paper p*(x)_y >= 1/L for the true label y on all relevant supports
    Used implicitly in the proof of Lemma 1 and Proposition 1 (Eqs. 24 and 40) but not stated in Proposition 1 and not true in general for ambiguous or noisy labels.
  • ad hoc to paper Optimizing Eq. 7 produces samples in supp(Q) = {x: t_y(x) >= tau, f_y(x) <= sigma}
    Proposition 1 is stated for Q with this support, but the method never enforces the thresholds tau and sigma; the generated samples are only assumed to satisfy them.
  • domain assumption Stable Diffusion edits spurious attributes while preserving the causal label when teacher confidence is high
    The whole augmentation mechanism relies on the diffusion model's ability to change non-causal features without flipping the true class; only qualitative examples are provided.
  • domain assumption Student approximates teacher well on the training support (delta_s small)
    Required by Proposition 1 and generally the goal of EDRM; not verified quantitatively in the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation." pith.science (2026). https://pith.science/paper/METLODOZ

@misc{pith2026250602294,
  author       = {Pith},
  title        = {Pith review of: Improving Knowledge Distillation Under Unknown Covariate Shift Through Confidence-Guided Data Augmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/METLODOZ}},
  note         = {Machine review of arXiv:2506.02294}
}
read the original abstract

Large foundation models trained on extensive datasets demonstrate strong zero-shot capabilities in various domains. Knowledge distillation has become an established tool for transferring knowledge from foundation models to small student networks when data and model size are constrained. However, the efficacy of distillation is often hampered by limited training data coverage. This can result in a covariate shift between training and test data which in turn can lead the student to exploit spurious features or even shortcut learning. We address this problem by introducing a novel diffusion-based data augmentation strategy that generates images by maximizing the disagreement between the teacher and the student, effectively creating challenging samples that the student struggles with, thus mitigating the problem of covariate shift. Experiments demonstrate that, compared to state-of-the-art diffusion-based data augmentation baselines, our approach is best or second-best in sample mean accuracy and improves the worst group and mean group accuracy on CelebA-HQ, SpuCo Birds and BAR as well as the spurious score on Spurious ImageNet under covariate shift.

Figures

Figures reproduced from arXiv: 2506.02294 by the authors.

Figure 1
Figure 1. Knowledge Distillation under Covariate Shift with ConfiG Data Augmentations. Our goal is to maximize student performance even on groups that are fully absent from the training data. Left: Without having seen similar samples, the student is incapable of correctly classifying the unseen groups in the test set. Right: Leveraging the discrepancy between a robust teacher and the biased student, ConfiG generates and adds … view at source ↗
Figure 2
Figure 2. The student relies on spuri￾ous features in the training data and disagrees with the teacher when tested in their absence. Distillation was per￾formed with standard augmentations and EDRM as described in Section 4. We assume access to a teacher that approximates the p ∗ (x) well on both Ptrain and Ptest. In [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of ConfiG and other diffusion-based augmentation methods: Left: the task is gender prediction (male/female) where the men in the training data all have glasses, are old and not blond. Right: Existing methods just produce high confidence for “male” but do not take into account the student. In contrast, ConfiG maximizes the difference between the teacher and the student over the latent space and so target… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Qualitative examples of our method ConfiG (green) and the baselines (red) on CelebA [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Two augmentations per real image maximizes worst and mean group accuracy [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Further examples for diffusion-guided data augmentations for images from CelebA and [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: Further examples for diffusion-guided data augmentations for images from CelebA and [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]
Figure 8
Figure 8. Figure 8: Further examples for diffusion-guided data augmentations for images from ImageNet. [PITH_FULL_IMAGE:figures/full_fig_p025_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 46 canonical work pages

  1. [1]

    Fairness Without Demographic Data: A Survey of Approaches

    Carolyn Ashurst and Adrian Weller. “Fairness Without Demographic Data: A Survey of Approaches”. In:EAAMO. 2023

    work page 2023

  2. [2]

    DASH: Detection and Assessment of Systematic Hallucinations of VLMs

    Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. “DASH: Detection and Assessment of Systematic Hallucinations of VLMs”. In:arxiv2503.23573 (2025)

    arXiv 2025

  3. [3]

    DiG-IN: Diffusion Guidance for Investigating Networks - Uncovering Classifier Differences, Neuron Visualisations, and Visual Counterfactual Explanations

    Maximilian Augustin, Yannic Neuhaus, and Matthias Hein. “DiG-IN: Diffusion Guidance for Investigating Networks - Uncovering Classifier Differences, Neuron Visualisations, and Visual Counterfactual Explanations”. In:CVPR. 2024

    work page 2024

  4. [4]

    Synthetic Data from Diffusion Models Improves ImageNet Classification

    Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. “Synthetic Data from Diffusion Models Improves ImageNet Classification”. In:TMLR(2023)

    work page 2023

  5. [5]

    Knowledge distillation: A good teacher is patient and consistent

    Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. “Knowledge distillation: A good teacher is patient and consistent”. In:CVPR. 2022

    work page 2022

  6. [6]

    AutoAug- ment: Learning Augmentation Strategies From Data

    Ekin D. Cubuk, Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V . Le. “AutoAug- ment: Learning Augmentation Strategies From Data”. In:CVPR. 2019

    work page 2019

  7. [7]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”. In:ICLR. 2021

    work page 2021

  8. [8]

    DataComp: In search of the next generation of multimodal datasets

    Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M. Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Ale...

    work page 2023

Show all 53 references
  1. [9]

    Shortcut learning in deep neural networks

    Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. “Shortcut learning in deep neural networks”. In: Nat. Mach. Intell.2.11 (2020), pp. 665–673

    2020

  2. [10]

    ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations

    Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi, S. Sakshi, Sanjoy Chowdhury, and Dinesh Manocha. “ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations”. In:ACL. 2024, pp. 386–406

    2024

  3. [11]

    Knowledge Distillation: A Survey

    Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. “Knowledge Distillation: A Survey”. In:Int. J. Comput. Vis.129.6 (2021), pp. 1789–1819

    2021

  4. [12]

    Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale

    Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. “Revisit the Power of Vanilla Knowledge Distillation: from Small Scale to Large Scale”. In:NeurIPS. 2023

    2023

  5. [13]

    Distilling the Knowledge in a Neural Network

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. “Distilling the Knowledge in a Neural Network”. In:arXiv1503.02531 (2015)

    2015 arXiv

  6. [14]

    Active Generation for Image Classification

    Tao Huang, Jiaqi Liu, Shan You, and Chang Xu. “Active Generation for Image Classification”. In:ECCV. 2024

    2024

  7. [15]

    Version 0.1

    Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.OpenCLIP. Version 0.1. July 2021.URL: https://doi. org/10.5281/zeno...

    2021 doi

  8. [16]

    Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Features

    Siddharth Joshi, Yu Yang, Yihao Xue, Wenhan Yang, and Baharan Mirzasoleiman. “Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Features”. In:arXiv2306.11957 (2025)

    2025 arXiv

  9. [17]

    Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations

    Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. “Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations”. In:ICLR. 2023

    2023

  10. [18]

    Towards Last-layer Retraining for Group Robustness with Fewer Annotations

    Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. “Towards Last-layer Retraining for Group Robustness with Fewer Annotations”. In:NeurIPS. 2023

    2023

  11. [19]

    BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations

    Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba. “BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations”. In:CVPR. 2022

    2022

  12. [20]

    Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive

    Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. “Adversarial Supervision Makes Layout-to-Image Diffusion Models Thrive”. In:ICCV. 2024

    2024

  13. [21]

    The Genetic Overlap Between Hair and Eye Color

    Bochao Lin, Gonneke Willemsen, Abdel Abdellaoui, Meike Bartels, Erik A Ehli, Gareth E Davies, Dorret I Boomsma, and Jouke-Jan Hottenga. “The Genetic Overlap Between Hair and Eye Color”. In:Twin Research and Human Genetics19.6 (2016), pp. 595–599

    2016

  14. [22]

    Just Train Twice: Improving Group Robustness without Training Group Information

    Evan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. “Just Train Twice: Improving Group Robustness without Training Group Information”. In:ICML. 2021

    2021

  15. [23]

    Deep Learning Face Attributes in the Wild

    Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. “Deep Learning Face Attributes in the Wild”. In:ICCV. 2015

    2015

  16. [24]

    Decoupled Weight Decay Regularization

    Ilya Loshchilov and Frank Hutter. “Decoupled Weight Decay Regularization”. In:ICLR. 2019

    2019

  17. [25]

    A Comprehensive Survey on Knowledge Distilla- tion

    Amir M. Mansourian, Rozhan Ahmadi, Masoud Ghafouri, Amir Mohammad Babaei, Ela- heh Badali Golezani, Zeynab Yasamani Ghamchi, Vida Ramezanian, Alireza Taherian, Kimia Dinashi, Amirali Miri, and Shohreh Kasaei. “A Comprehensive Survey on Knowledge Distilla- tion”. In:arXiv2503.1...

    2025

  18. [26]

    AW ADA: Attention-Weighted Adversarial Domain Adaptation for Object Detection

    Maximilian Menke, Thomas Wenzel, and Andreas Schwung. “AW ADA: Attention-Weighted Adversarial Domain Adaptation for Object Detection”. In:arXiv2208.14662 (2022)

    2022 arXiv

  19. [27]

    A statistical perspective on distillation

    Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Seungyeon Kim, and Sanjiv Kumar. “A statistical perspective on distillation”. In:ICML. Ed. by Marina Meila and Tong Zhang. 2021

    2021

  20. [28]

    Null-text Inversion for Editing Real Images using Guided Diffusion Models

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. “Null-text Inversion for Editing Real Images using Guided Diffusion Models”. In:CVPR. 2023

    2023

  21. [29]

    Spurious Features Everywhere - Large-Scale Detection of Harmful Spurious Features in ImageNet

    Yannic Neuhaus, Maximilian Augustin, Valentyn Boreiko, and Matthias Hein. “Spurious Features Everywhere - Large-Scale Detection of Harmful Spurious Features in ImageNet”. In: ICCV. 2023

    2023

  22. [30]

    BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. “BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers”. In:arXiv2208.06366 (2022)

    2022 arXiv

  23. [31]

    Towards Understanding Knowledge Distillation

    Mary Phuong and Christoph Lampert. “Towards Understanding Knowledge Distillation”. In: ICML. 2019

    2019

  24. [32]

    Feature Distillation Improves Zero-Shot Transfer from Synthetic Images

    Niclas Popp, Jan Hendrik Metzen, and Matthias Hein. “Feature Distillation Improves Zero-Shot Transfer from Synthetic Images”. In:TMLR(2024).ISSN: 2835-8856

    2024

  25. [33]

    Simple and Fast Group Robustness by Automatic Feature Reweighting

    Shikai Qiu, Andres Potapczynski, Pavel Izmailov, and Andrew Gordon Wilson. “Simple and Fast Group Robustness by Automatic Feature Reweighting”. In:ICML. 2023

    2023

  26. [34]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. “Learning Transferable Visual Models From Natural Language Supervision”. In: ICML. Ed. by M...

    2021

  27. [35]

    High-Resolution Image Synthesis with Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. “High-Resolution Image Synthesis with Latent Diffusion Models”. In:CVPR. 2022

    2022

  28. [36]

    ImageNet Large Scale Visual Recognition Challenge

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Li Fei-Fei. “ImageNet Large Scale Visual Recognition Challenge”. In:Int. J. Comput. Vis.115.3 (2015), pp. 211–252

    2015

  29. [37]

    Distributionally Robust Neural Networks

    Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. “Distributionally Robust Neural Networks”. In:ICLR. 2020

    2020

  30. [38]

    Does Knowledge Distillation Really Work?

    Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A. Alemi, and Andrew Gordon Wilson. “Does Knowledge Distillation Really Work?” In:NeurIPS. 2021, pp. 6906–6919. 11

    2021

  31. [39]

    Effective Data Augmentation With Diffusion Models

    Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. “Effective Data Augmentation With Diffusion Models”. In:ICLR. 2024

    2024

  32. [40]

    Group Robust Classification Without Any Group Information

    Christos Tsirigotis, João Monteiro, Pau Rodríguez, David Vázquez, and Aaron C. Courville. “Group Robust Classification Without Any Group Information”. In:NeurIPS. 2023

    2023

  33. [41]

    Efficient Knowledge Distillation from Model Checkpoints

    Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang. “Efficient Knowledge Distillation from Model Checkpoints”. In:NeurIPS. 2022

    2022

  34. [42]

    What Makes a

    Huan Wang, Suhas Lohit, Michael J. Jones, and Yun Fu. “What Makes a "Good" Data Augmentation in Knowledge Distillation - A Statistical Perspective”. In:NeurIPS. Ed. by Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh. 2022

    2022

  35. [43]

    Label-Free Neural Semantic Image Synthesis

    Jiayi Wang, Kevin Alexander Laube, Yumeng Li, Jan Hendrik Metzen, Shin-I Cheng, Julio Borges, and Anna Khoreva. “Label-Free Neural Semantic Image Synthesis”. In:ECCV. Ed. by Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol. 2024

    2024

  36. [44]

    https://github.com/rwightman/pytorch- image-models

    Ross Wightman.PyTorch Image Models. https://github.com/rwightman/pytorch- image-models. 2019.DOI:10.5281/zenodo.4414861

    2019 doi

  37. [45]

    ResNet strikes back: An improved training procedure in timm

    Ross Wightman, Hugo Touvron, and Herve Jegou. “ResNet strikes back: An improved training procedure in timm”. In:NeurIPS 2021 Workshop on ImageNet: Past, Present, and Future. 2021

    2021

  38. [46]

    FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models

    Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. “FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models”. In:NeurIPS. Ed. by Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine. 2023

    2023

  39. [47]

    CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

    Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. “CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features”. In:ICCV. 2019

    2019

  40. [48]

    Efficient Bias Mitigation Without Privileged Information

    Mateo Espinosa Zarlenga, Swami Sankaranarayanan, Jerone T. A. Andrews, Zohreh Shams, Mateja Jamnik, and Alice Xiang. “Efficient Bias Mitigation Without Privileged Information”. In:ECCV. 2024

    2024

  41. [49]

    Mixup: Beyond Empirical Risk Minimization

    Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. “Mixup: Beyond Empirical Risk Minimization”. In:ICLR. 2018

    2018

  42. [50]

    Expanding Small-Scale Datasets with Guided Imagination

    Yifan Zhang, Daquan Zhou, Bryan Hooi, Kai Wang, and Jiashi Feng. “Expanding Small-Scale Datasets with Guided Imagination”. In:NeurIPS. 2023

    2023

  43. [51]

    DatasetGAN: Efficient Labeled Data Factory With Minimal Human Effort

    Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. “DatasetGAN: Efficient Labeled Data Factory With Minimal Human Effort”. In:CVPR. 2021

    2021

  44. [52]

    Distribution-Aware Data Expansion with Diffusion Models

    Haowei Zhu, Ling Yang, Jun-Hai Yong, Hongzhi Yin, Jiawei Jiang, Meng Xiao, Wentao Zhang, and Bin Wang. “Distribution-Aware Data Expansion with Diffusion Models”. In:NeurIPS. 2024. 12 A Appendix A.1 Theoretical Investigation In this section, we show that ConfiG data augmentatio...

    2024

  45. [53]

    a realistic photo of a{classname}

    We use a learning rate of 0.01. A.11 Summary of the baselines DistDiff [52]constructs hierarchical prototypes to approximate the real data distribution. The latent data points within diffusion model are optimized to stay close to these prototypes with hierarchical energy guida...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.