Pith. sign in

REVIEW 5 major objections 5 minor 71 references

PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A proxy model that trains on augmented data and averages parameters with the main model reduces out-of-distribution accuracy swings and improves single-source domain generalization.

desk verdict A simple and promising sDG regularization scheme whose headline SOTA claim is undercut by oracle-tuned hyperparameters and single-run comparisons; worth a serious referee but needs a fair validation protocol. read the letter →

arxiv 2505.12745 v1 pith:AB53BUZW submitted 2025-05-19 cs.LG cs.AI

Dong Kyu Cho , Inwoo Hwang , Sanghack Lee This is my paper · ORCID
classification cs.LGcs.AI
keywords singlesourcedomaingeneralizationdataaugmentationparameteraveragingmodeconnectivitymutualinformationregularizationout-of-distributionperformancefluctuationproxymodelBarlowTwins
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper identifies a problem that augmentation-based single-source domain generalization methods share: during training, their accuracy on unseen target domains swings widely, which makes model selection unreliable. It argues that the swings come from the model failing to accumulate what it learns from diverse augmented views, so previously learned features get distorted. To fix this, PEER trains a proxy model on the augmented data under the guidance of a frozen task model, then periodically merges the proxy's parameters into the task model by averaging. A mutual-information regularizer keeps the two models aligned, so the averaged model behaves like a parameter-space ensemble that stacks the benefits of many augmentations. With only random augmentation, PEER reports state-of-the-art mean target accuracy on PACS and Digits and large gains on Office-Home and VLCS, while reducing the variance of target accuracy during training.

What carries the argument

The load-bearing object is PEER's two-model loop: a frozen task model F that acts as regulator, and a trainable proxy model P that sees augmented views. Every k epochs the proxy's trajectory snapshots Θ are averaged into the task model via $\theta_f \leftarrow \frac{1}{|\Theta|}\sum_{\theta \in \Theta}\theta$, and the augmentation policy is reinitialized. The alignment step is a mutual-information maximization between the projection-headed representations of the two models, implemented as a Barlow-Twins-style feature-decorrelation loss; the paper also tests InfoNCE. The same-initialization requirement and the regularization together keep task and proxy in the same low-loss basin, so parameter averaging acts as a true ensemble (mode connectivity).

What would settle it

On PACS, compute the linear interpolation loss barrier between proxy snapshots $\theta_p^{(0)}$ and $\theta_p^{(100)}$ with and without the PEER regularizer. PEER's claim predicts a large barrier reduction and a corresponding accuracy gain at interpolation weight 0.5; if the barrier stays high or the interpolated accuracy does not beat the best single snapshot, the mode-connectivity story fails.

Watch

Extended reading notes

Core claim

The central discovery is the mid-train OOD fluctuation: augmentation-based sDG methods show large variance in target-domain accuracy across training, and the severity tracks the source-target dataset distance and the complexity of the augmentation. The paper argues the cause is feature distortion - augmented samples that are surprisingly inconsistent with the original source data push the learned representation around, so knowledge from earlier steps is overwritten. PEER counters this by having a proxy model, trained on augmented views with cross-entropy plus a feature-level mutual-information regularizer, and a task model updated by averaging the proxy's parameters along its trajectory. The regularization aligns the proxy with the task model in feature space, which the paper shows lowers the loss barrier between snapshots and preserves learned features (CKA evidence). The result is a single model that behaves like an ensemble of training snapshots without extra inference cost.

Load-bearing premise

The whole gain rests on the premise that the proxy and task models stay in the same low-loss basin, so that averaging their parameters behaves like an ensemble; the paper concedes that the alignment mechanism behind this is only supported empirically (CKA and interpolation), not theoretically.

Editorial extensions

If this is right

  • PEER with simple RandAug beats the strongest prior augmentation-based sDG methods on PACS and Digits mean accuracy, lifting PACS from 55.59 to 59.42 and Digits from 73.98 to 81.06.
  • The variance of target-domain accuracy during training drops on every benchmark, so model selection no longer depends on catching the right epoch; the final averaged task model is the safe choice.
  • On Office-Home and VLCS, where naive RandAug hurts accuracy, PEER still improves over ERM, extending the method's reach beyond the standard sDG benchmarks.
  • The alignment regularizer is load-bearing: parameter averaging without it (P-ENS w/o PEER) fails on PACS, so the mutual-information step, not averaging alone, produces the ensemble gain.
  • Because PEER needs no external pre-trained teacher, it respects the single-source constraint and transfers to tasks where no suitable teacher exists, such as digit classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the fluctuation claim generalizes, a cheap oracle-free selection rule might be to pick the epoch with lowest recent target-variance proxy; the paper does not propose this.
  • The same two-model recipe could be dropped into other augmentation-heavy pipelines (self-supervised pretraining, continual learning with replay) where feature distortion is also a concern; the paper does not test these.
  • The OTDD-based observation that augmented views can be farther from the source than real target domains suggests a principled way to schedule augmentation strength, whereas the paper only reinitializes the augmentation randomly.
  • Replacing random augmentation with a learnable augmentation module inside PEER could stack the two families of gains; the paper only reports random augmentation, so this is untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies single-source domain generalization (sDG) with data augmentation. It first reports an empirical phenomenon, termed mid-train OOD fluctuation: across training, the target-domain accuracy of augmentation-based sDG methods varies substantially. It argues that this fluctuation is caused by the model's inability to accumulate knowledge across diverse augmented views, leading to feature distortion. The authors then propose PEER (Parameter-Space Ensemble with Entropy Regularization), which trains a proxy model under the guidance of a frozen task model via a mutual-information-style regularizer, and periodically updates the task model by averaging parameters along the proxy's trajectory. Experiments on PACS, Digits, Office-Home, and VLCS report improved mean target accuracy and reduced variance of target accuracy during training, and the paper claims state-of-the-art results using only simple random augmentation.

Significance. If the results hold, PEER is an attractive and simple method: it requires no external pretrained teacher, adds a modest computational overhead, and appears to stabilize training while improving target accuracy. The paper also makes a useful empirical observation about the instability of augmentation-based sDG training, and it provides supporting analyses via OTDD distances, CKA similarity, and mode-connectivity interpolation. The main tables are internally consistent with each other in several places (e.g., Tables 2 and 3 match Table 13 for the default backbones), and the ablation study (Table 8) clearly attributes the gain to the combination of augmentation, parameter averaging, and regularization. However, the strength of the headline claims is currently limited by the oracle-based hyperparameter tuning and by the absence of multi-seed statistics, and there is a notable numeric inconsistency between Tables 2 and 9a.

major comments (5)
  1. [Sec. D.3; Tables 2 and 3] The abstract and Sec. 5.2 claim state-of-the-art results (2.30% and 0.96% gains over same-backbone baselines), but Sec. D.3 states that hyperparameters were tuned using the oracle test dataset, and Tables 2 and 3 report single runs with those choices. This makes the headline comparison vulnerable to selection bias. Please provide multi-seed mean and standard deviation for all four benchmarks under both source-validation and crafted-validation selection protocols, and state explicitly which selection protocol was used for each reported number.
  2. [Sec. 5.1; Table 4] The fluctuation metric is the variance of target accuracy across k-th epochs within a single training run. As reported, it does not capture run-to-run seed variance and no confidence intervals are given. Because reducing mid-train OOD fluctuation is a central contribution of the paper, please report the distribution of this variance over at least three independent seeds, and include per-seed target-accuracy curves for the main comparison.
  3. [Table 9a vs. Table 2] For what appears to be the same configuration (w = 2, k = 10), Table 9a reports a PACS average of 57.56, while Table 2 reports 59.42 for PEER (ours). Table 9b gives a variance of 3.59 for the same row, matching Table 4, so this is not simply a mislabeled variance. Please reconcile the two numbers and clarify which protocol, model-selection rule, or training configuration distinguishes them.
  4. [Table 5; Sec. A] In the Digits teacher comparison, the teacher model is pretrained on both the source and target domains of Digits (Sec. A), which violates the single-source protocol and makes the teacher-vs-PEER comparison in Table 5 difficult to interpret. Please replace this oracle teacher with a teacher pretrained only on the source domain or on a standard ImageNet-pretrained model, and also report the teacher's own target accuracy.
  5. [Sec. C.2; Eq. (4)] The mechanism of the parameter-averaging step in Eq. (4) relies on mode connectivity between the task model and the proxy model. Section C.2 explicitly concedes that this alignment has not been analyzed theoretically, and Table 8 shows that parameter averaging without the regularizer is no better than ERM on PACS (53.51 vs. 46.39), while the full method reaches 59.42. The alignment role of the L_PEER term is therefore load-bearing. Please provide loss-barrier or interpolation measurements for PEER with and without regularization, across multiple augmentation policies and at least one additional architecture, to substantiate that the regularizer itself, rather than averaging alone, creates the connectivity.
minor comments (5)
  1. [Abstract; Sec. 3.2] The word 'universally' in the abstract is stronger than the evidence presented; the fluctuation is demonstrated for RandAug and AdvAug on the tested setups. Please soften to something like 'across the augmentation-based methods we examined.'
  2. [Sec. 4] The name 'Entropy Regularization' is potentially misleading because Eq. (2) maximizes mutual information, not entropy. Please clarify the terminology or consider naming it 'mutual-information regularization.'
  3. [Algorithm 1] The ordering of lines 7-10 is ambiguous: the task model update at line 10 appears to occur before the next k-epoch training block, and the snapshot theta_p^(n) saved at line 9 is not defined on the first iteration. Please clarify the epoch-boundary semantics.
  4. [Fig. 3] The axis label 'Augmentation Magnitude 15101520' appears corrupted; the tick labels and colorbar need reformatting.
  5. [Eq. (6)] The balancing coefficient lambda is used in Eq. (6) before it is defined in the text. Please define it at first use or point the reader to Sec. D.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PEER's objectives and updates are defined entirely on source-domain data and are evaluated against external benchmarks; the flagged oracle-tuning issue is an evaluation-fairness concern, not a circular derivation.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. The proposed loss in Eq. (2), L_PEER = -I(R(H_f(x)); R(H_p(x-bar))), and the total proxy objective in Eq. (3) involve only the source sample x and its augmented view x-bar; no target-domain accuracy, fluctuation, or target labels appear in the training objective. The task-model update in Eq. (4) averages proxy snapshots along the source training trajectory, and the reported OOD fluctuation is measured on held-out target domains (PACS, Digits, Office-Home, VLCS) rather than being constructed from the loss. The main empirical claims are benchmark comparisons against external baselines (ERM, ADA, M-ADA, L2D, PDEN, SimDE, AdvST, MetaCNN), not renamed versions of the paper's own fitted quantities. The mode-connectivity mechanism is explicitly acknowledged as lacking a thorough theoretical analysis (Sec. C.2: 'the alignment of models in its loss landscape is a topic that has not yet been thoroughly analyzed from a theoretical perspective'), supported instead by empirical CKA and interpolation evidence, which is a stated limitation rather than a circular dependency. The one flagged issue is Sec. D.3's admission that 'hyperparameters were tuned using the oracle test dataset'; this is a model-selection fairness problem that can inflate reported test accuracy, but it is not a case where an equation or prediction reduces by construction to the paper's inputs. No self-citation chain is load-bearing, and no uniqueness theorem or ansatz is smuggled in via the authors' own prior work. Therefore, the central derivation is not circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

PEER introduces no new entities or forces. It relies on three free hyperparameters, all tuned to the test set, and on two standard background assumptions from previous work plus its own causal hypothesis about fluctuation.

free parameters (3)
  • w = 2.0
    Balancing coefficient for the PEER regularization term in Eq. (3); tuned using the oracle test set (Sec. D.3) and set to 2.0 for all experiments.
  • lambda = 0.005
    Coefficient for the off-diagonal term in the Barlow Twins loss, Eq. (6); inherited from [69] and verified stable in Sec. D.2, set to 0.005.
  • k = 10
    Period for augmentation reinitialization and parameter averaging (Algorithm 1); tuned using the oracle test set (Sec. D.3) and set to 10 for all experiments.
assumptions (3)
  • standard math The InfoNCE/Barlow Twins objective provides a tractable lower bound to mutual information I(R(Hf(x)); R(Hp(x_bar))).
    Invoked in Sec. 4.1 and B.2; relies on prior results [45, 47, 57].
  • domain assumption Parameter averaging of models that share initialization and are trained on the same data yields an ensemble effect (mode connectivity).
    Used to justify Eq. (4) in Sec. 4.2; based on [18, 28, 44]. The paper itself notes in C.2 that this is not theoretically analyzed for deep networks.
  • ad hoc to paper The observed mid-train OOD fluctuation is caused by feature distortion due to data augmentation, not merely stochasticity.
    Central motivation in Sec. 3.2/Fig. 2; supported only by correlational CKA evidence, not by a causal experiment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization." pith.science (2026). https://pith.science/paper/AB53BUZW

@misc{pith2026250512745,
  author       = {Pith},
  title        = {Pith review of: PEER pressure: Model-to-Model Regularization for Single Source Domain Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AB53BUZW}},
  note         = {Machine review of arXiv:2505.12745}
}
read the original abstract

Data augmentation is a popular tool for single source domain generalization, which expands the source domain by generating simulated ones, improving generalization on unseen target domains. In this work, we show that the performance of such augmentation-based methods in the target domains universally fluctuates during training, posing challenges in model selection under realistic scenarios. We argue that the fluctuation stems from the inability of the model to accumulate the knowledge learned from diverse augmentations, exacerbating feature distortion during training. Based on this observation, we propose a novel generalization method, coined Parameter-Space Ensemble with Entropy Regularization (PEER), that uses a proxy model to learn the augmented data on behalf of the main model. The main model is updated by averaging its parameters with the proxy model, progressively accumulating knowledge over the training steps. Maximizing the mutual information between the output representations of the two models guides the learning process of the proxy model, mitigating feature distortion during training. Experimental results demonstrate the effectiveness of PEER in reducing the OOD performance fluctuation and enhancing generalization across various datasets, including PACS, Digits, Office-Home, and VLCS. Notably, our method with simple random augmentation achieves state-of-the-art performance, surpassing prior approaches on sDG that utilize complex data augmentation strategies.

Figures

Figures reproduced from arXiv: 2505.12745 by the authors.

Figure 1
Figure 1. Despite its generalization effect, data augmentation in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of pitfalls of augmentation in generalizing to [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. OTDD distance [2] between the original data (MNIST) and its augmented view. 4. Method We now present a novel generalization method for sDG, coined Parameter-Space Ensemble with Entropy Regulariza￾tion (PEER), that mitigates the augmentation-induced feature distortion and its associated issues (e.g., mid-train OOD fluc￾tuation). Our approach involves two interacting modules with identical architectures: a frozen task… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Mode connectivity in the proxy model’s trajectory. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Layer-wise feature similarity between the fully updated task model and the proxy model at different epochs. The task model [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The PEER framework consists of two interacting modules: a proxy model P and the task model F. During training, the task model retains the knowledge of the proxy model via parameter-averaging. The conventional teacher-student framework consists of a frozen teacher T and…
Figure 7
Figure 7. Figure 7: Layer-wise feature similarity (CKA) between the proxy model after initialization and after training with different epochs. Without [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Layer-wise Feature Similarity (CKA) between the proxy model’s initialization and the trained proxy model (without [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Layer-wise Feature Similarity (CKA) between the proxy model’s initialization and the trained proxy model (with [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 62 canonical work pages

  1. [1]

    Ainsworth, Jonathan Hayase, and Siddhartha Srini- vasa

    Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha Srini- vasa. Git re-basin: Merging models modulo permutation symmetries, 2023. 2, 15

    work page 2023

  2. [2]

    Geometric dataset dis- tances via optimal transport.Advances in Neural Information Processing Systems, 33:21428–21439, 2020

    David Alvarez-Melis and Nicolo Fusi. Geometric dataset dis- tances via optimal transport.Advances in Neural Information Processing Systems, 33:21428–21439, 2020. 3, 4, 7

    work page 2020

  3. [3]

    Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Eric Granger, and Marco Pedersoli

    Masih Aminbeidokhti, Fidel A. Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Eric Granger, and Marco Pedersoli. Domain generalization by rejecting extreme augmentations,

    work page

  4. [4]

    Invariant risk minimization, 2019

    Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization, 2019. 2

    work page 2019

  5. [5]

    Ensemble of averages: Improving model selection and boosting performance in domain generalization.Advances in Neural Information Processing Systems, 35:8265–8277, 2022

    Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization.Advances in Neural Information Processing Systems, 35:8265–8277, 2022. 2, 13

    work page 2022

  6. [6]

    A cookbook of self-supervised learning, 2023

    Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann Le- Cun, and Micah Goldblum. A cookbook of self-supervised learning, 2023. 14

    work page 2023

  7. [7]

    Knowledge distilla- tion: A good teacher is patient and consistent, 2022

    Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov. Knowledge distilla- tion: A good teacher is patient and consistent, 2022. 12

    work page 2022

  8. [8]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023

    Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023. 5, 12

    work page 2023

Show all 71 references
  1. [9]

    Domain Generalization by Mutual-Information Reg- ularization with Pre-trained Models.arXiv e-prints, art

    Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain Generalization by Mutual-Information Reg- ularization with Pre-trained Models.arXiv e-prints, art. arXiv:2203.10789, 2022. 2, 12, 20

    2022 arXiv

  2. [10]

    Fusing finetuned models for better pretraining.arXiv preprint arXiv:2204.03044, 2022

    Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining.arXiv preprint arXiv:2204.03044, 2022. 2, 5, 15

    2022 arXiv

  3. [11]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition work- shops, pages 702–703, 2020. 3, 4, 6, 7, 8, 18, 19

    2020

  4. [12]

    The mnist database of handwritten digit images for machine learning research.IEEE Signal Processing Maga- zine, 29(6):141–142, 2012

    Li Deng. The mnist database of handwritten digit images for machine learning research.IEEE Signal Processing Maga- zine, 29(6):141–142, 2012. 5, 12, 18, 21

    2012

  5. [13]

    Craft- ing distribution shifts for validation and training in single source domain generalization.arXiv:2409.19774, 2024

    Nikos Efthymiadis, Giorgos Tolias, and Ondˇrej Chum. Craft- ing distribution shifts for validation and training in single source domain generalization.arXiv:2409.19774, 2024. 16

    2024 arXiv

  6. [14]

    The role of permutation invariance in lin- ear mode connectivity of neural networks.arXiv preprint arXiv:2110.06296, 2021

    Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in lin- ear mode connectivity of neural networks.arXiv preprint arXiv:2110.06296, 2021. 2

    2021 arXiv

  7. [15]

    https://torchvision.mlverse.org, https://github.com/mlverse/torchvision

    Daniel Falbel.torchvision: Models, Datasets and Transfor- mations for Images, 2023. https://torchvision.mlverse.org, https://github.com/mlverse/torchvision. 20, 21

    2023

  8. [16]

    Adversarially adaptive normal- ization for single domain generalization

    Xinjie Fan, Qifei Wang, Junjie Ke, Feng Yang, Boqing Gong, and Mingyuan Zhou. Adversarially adaptive normal- ization for single domain generalization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8208–8217, 2021. 2, 6

    2021

  9. [17]

    Unbiased met- ric learning: On the utilization of multiple datasets and web images for softening bias

    Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased met- ric learning: On the utilization of multiple datasets and web images for softening bias. InProceedings of the IEEE Inter- national Conference on Computer Vision, pages 1657–1664,

  10. [18]

    Linear mode connectivity and the lottery ticket hypothesis

    Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. InInternational Conference on Machine Learning, pages 3259–3269. PMLR, 2020. 2, 5, 7

    2020

  11. [19]

    Unsupervised domain adaptation by backpropagation

    Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. InInternational Conference on Machine Learning, pages 1180–1189. PMLR, 2015. 5, 18

    2015

  12. [20]

    Domain-adversarial training of neural networks.Journal of Machine Learning Research 17 (2016) 1-35, 2015

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marc- hand, and Victor Lempitsky. Domain-adversarial training of neural networks.Journal of Machine Learning Research 17 (2016) 1-35, 2015. 5, 18

    2016

  13. [21]

    Loss surfaces, mode connectivity, and fast ensembling of dnns.Advances in neural information processing systems, 31, 2018

    Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns.Advances in neural information processing systems, 31, 2018. 2

    2018

  14. [22]

    Maybank, and Dacheng Tao

    Jianping Gou, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. Knowledge distillation: A survey.Interna- tional Journal of Computer Vision, 129(6):1789–1819, 2021. 12

    2021

  15. [23]

    Bootstrap your own latent-a new approach to self-supervised learning.Advances in neural information processing systems, 33:21271–21284, 2020

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doer- sch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Ghesh- laghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning.Advances in neura...

    2020

  16. [24]

    Understanding and improving the role of projection head in self-supervised learning, 2022

    Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hen- gel, and Stephen Gould. Understanding and improving the role of projection head in self-supervised learning, 2022. 4

    2022

  17. [25]

    Distilling the knowledge in a neural network, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. 12

    2015

  18. [26]

    Learning deep representations by mutual information estimation and maximization

    Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. InICLR 2019. ICLR, 2019. 14 9

    2019

  19. [27]

    Towards the generalization of contrastive self-supervised learning, 2021

    Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self-supervised learning, 2021. 14

    2021

  20. [28]

    Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2018

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization.arXiv preprint arXiv:1803.05407, 2018. 2, 3, 7, 15

    2018 arXiv

  21. [29]

    Population parameter averaging (papa).arXiv preprint arXiv:2304.03094, 2023

    Alexia Jolicoeur-Martineau, Emy Gervais, Kilian Fatras, Yan Zhang, and Simon Lacoste-Julien. Population parameter averaging (papa).arXiv preprint arXiv:2304.03094, 2023. 2

    2023 arXiv

  22. [30]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. 20

    2015

  23. [31]

    Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dy- lan Paiton

    David A. Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dy- lan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. InInternational Conference on Learning Representations, 2021. 2

    2021

  24. [32]

    Springer Berlin Heidelberg, 2011

    Vladimir Koltchinskii.Oracle Inequalities in Empirical Risk Minimization and Sparse Recovery Problems: École d’Été de Probabilités de Saint-Flour XXXVIII-2008. Springer Berlin Heidelberg, 2011. 6, 8, 18

    2008

  25. [33]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. InInternational conference on machine learning, pages 3519–3529. PMLR, 2019. 7

    2019

  26. [34]

    Im- agenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Im- agenet classification with deep convolutional neural networks. InAdvances in Neural Information Processing Systems. Cur- ran Associates, Inc., 2012. 19

    2012

  27. [35]

    Fine-tuning can distort pre- trained features and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pre- trained features and underperform out-of-distribution. In International Conference on Learning Representations, 2022. 13

    2022

  28. [36]

    Adver- sarial examples in the physical world

    Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adver- sarial examples in the physical world. InArtificial intelligence safety and security, pages 99–112. Chapman and Hall/CRC,

  29. [37]

    Le Cun, B

    Y . Le Cun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Handwritten digit recognition with a back-propagation network. InProceedings of the 2nd International Conference on Neural Information Processing Systems, page 396–404, Cambridge, MA, USA,

  30. [38]

    Deeper, broader and artier domain generaliza- tion

    Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generaliza- tion. InProceedings of the IEEE international conference on computer vision, pages 5542–5550, 2017. 5, 18, 20, 21

    2017

  31. [39]

    L. Li, K. Gao, J. Cao, Z. Huang, Y . Weng, X. Mi, Z. Yu, X. Li, and B. Xia. Progressive domain expansion network for single domain generalization. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 224–233, Los Alamitos, CA, USA, 2021. IEEE Com...

    2021

  32. [40]

    SIMPLE: Specialized model- sample matching for domain generalization

    Ziyue Li, Kan Ren, XINY ANG JIANG, Yifei Shen, Haipeng Zhang, and Dongsheng Li. SIMPLE: Specialized model- sample matching for domain generalization. InThe Eleventh International Conference on Learning Representations, 2023. 2, 12

    2023

  33. [41]

    Mechanistic mode connectiv- ity

    Ekdeep Singh Lubana, Eric J Bigelow, Robert P Dick, David Krueger, and Hidenori Tanaka. Mechanistic mode connectiv- ity. InInternational Conference on Machine Learning, pages 22965–23004. PMLR, 2023. 2

    2023

  34. [42]

    Weighted ensemble models are strong continual learners.arXiv preprint arXiv:2312.08977, 2023

    Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners.arXiv preprint arXiv:2312.08977, 2023. 3

    2023 arXiv

  35. [43]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y . Ng. Reading digits in natural images with unsupervised feature learning. InNIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011,

    2011

  36. [44]

    What is being transferred in transfer learning?Advances in neural information processing systems, 33:512–523, 2020

    Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning?Advances in neural information processing systems, 33:512–523, 2020. 5, 7

    2020

  37. [45]

    Represen- tation learning with contrastive predictive coding, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Represen- tation learning with contrastive predictive coding, 2018. 3, 4, 5, 13, 15

    2018

  38. [46]

    Estimation of entropy and mutual information

    Liam Paninski. Estimation of entropy and mutual information. Neural Comput., 15(6):1191–1253, 2003. 13

    2003

  39. [47]

    On variational bounds of mutual infor- mation

    Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual infor- mation. InInternational Conference on Machine Learning, pages 5171–5180. PMLR, 2019. 4, 13

    2019

  40. [48]

    Learning to learn single domain generalization

    Fengchun Qiao, Long Zhao, and Xi Peng. Learning to learn single domain generalization. InIEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), pages 12556– 12565, 2020. 2, 6, 18

    2020

  41. [49]

    Designing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaim- ing He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428–10436, 2020. 12, 20

    2020

  42. [50]

    Diverse weight averaging for out-of-distribution generaliza- tion

    Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generaliza- tion. InNeurIPS, 2022. 2, 5

    2022

  43. [51]

    Model ratatouille: Re- cycling diverse models for out-of-distribution generalization

    Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Re- cycling diverse models for out-of-distribution generalization. InInternational Conference on Machine Learning, pages 28656–28679. PMLR, 2023. 2, 5, 15

    2023

  44. [52]

    Rethinking content and style: Exploring bias for unsuper- vised disentanglement

    Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Rethinking content and style: Exploring bias for unsuper- vised disentanglement. In2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 1823–1832, 2021. 2

    2021

  45. [53]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge,

  46. [54]

    A unified approach to domain incremental learning with memory: Theory and algorithm

    Haizhou Shi and Hao Wang. A unified approach to domain incremental learning with memory: Theory and algorithm. 10 Advances in Neural Information Processing Systems, 36, 2024. 13

    2024

  47. [55]

    Esti- mating and maximizing mutual information for knowledge distillation

    Aman Shrivastava, Yanjun Qi, and Vicente Ordonez. Esti- mating and maximizing mutual information for knowledge distillation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 48–57, 2023. 14

    2023

  48. [56]

    C. Tao, H. Wang, X. Zhu, J. Dong, S. Song, G. Huang, and J. Dai. Exploring the equivalence of siamese self-supervised learning via a unified gradient framework. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14411–14420, Los Alamitos, CA, U...

    2022

  49. [57]

    A note on connecting barlow twins with negative-sample-free contrastive learning, 2021

    Yao-Hung Hubert Tsai, Shaojie Bai, Louis-Philippe Morency, and Ruslan Salakhutdinov. A note on connecting barlow twins with negative-sample-free contrastive learning, 2021. 5, 15, 20

    2021

  50. [58]

    Deep hashing network for unsupervised domain adaptation

    Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018–5027, 2017. 5, 18, 21

    2017

  51. [59]

    Generalizing to unseen domains via adversarial data augmentation, 2018

    Riccardo V olpi, Hongseok Namkoong, Ozan Sener, John Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation, 2018. 6

    2018

  52. [60]

    Generalizing to unseen domains via adversarial data augmentation.Advances in neural information processing systems, 31, 2018

    Riccardo V olpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation.Advances in neural information processing systems, 31, 2018. 1, 2

    2018

  53. [61]

    Self-supervised learning with data aug- mentations provably isolates content from style.Advances in neural information processing systems, 34:16451–16467,

    Julius V on Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data aug- mentations provably isolates content from style.Advances in neural information processing systems, 34:16451–16467,

  54. [62]

    Meta convolutional neural networks for single domain generalization

    Chaoqun Wan, Xu Shen, Yonggang Zhang, Zhiheng Yin, Xinmei Tian, Feng Gao, Jianqiang Huang, and Xian-Sheng Hua. Meta convolutional neural networks for single domain generalization. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4672–4681,

  55. [63]

    Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu. Generalizing to unseen domains: A survey on domain generalization, 2021. 1

    2021

  56. [64]

    A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 6

    2024

  57. [65]

    Learning to diversify for single domain generalization

    Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang, and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 834– 843, 2021. 2, 3, 6, 7, 18

    2021

  58. [66]

    Wolpert and W.G

    D.H. Wolpert and W.G. Macready. No free lunch theorems for optimization.IEEE Transactions on Evolutionary Com- putation, 1(1):67–82, 1997. 13, 14

    1997

  59. [67]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy with...

    2022

  60. [68]

    Simde: A simple domain expansion approach for single-source domain generalization

    Qinwei Xu, Ruipeng Zhang, Yi-Yan Wu, Ya Zhang, Ning Liu, and Yanfeng Wang. Simde: A simple domain expansion approach for single-source domain generalization. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4798–4808, 2023. 2, 6

    2023

  61. [69]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. InInternational Conference on Machine Learning, pages 12310–12320. PMLR, 2021. 4, 5, 14, 15, 20

    2021

  62. [70]

    Can parameter-averaging proxy model snapshots without regular- ization create a robust regulator?

    Guangtao Zheng, Mengdi Huai, and Aidong Zhang. Advst: Revisiting data augmentations for single domain generaliza- tion. InProceedings of the AAAI Conference on Artificial Intelligence, pages 21832–21840, 2024. 2, 6, 18 11 A. Study on Model-to-Model Regularization In this secti...

    2024

  63. [71]

    Y" indicates the convolution method, and the

    expand the training domains by progressively learning multiple augmentation modules, each simulating different domain shifts. Alternatively, MetaCNN [ 62] used a meta- convolutional network to learn generalized meta-features from local convolutional features. In contrast, we s...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.