REVIEW 5 major objections 7 minor 15 references
Class-based Subset Selection for Transfer Learning under Extreme Label Shift
T0 review · 5 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A linear program reweighting source classes by Wasserstein distance in a pretrained feature space identifies the classes that best transfer to a target domain, even under disjoint label sets.
desk verdict Simple, plausible class-reweighting method with mostly convincing empirical wins, but the theory section is wrong on a central claim and the appendix has corrupted indices. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the linear program in Eq. (4): a transport-matrix formulation of the Wasserstein distance in which each source class receives a weight $w_i$, the target marginal is uniform, and the source marginal is a class-reweighted mixture. Solving it over the simplex produces the sparse weight vector $w^*$ that selects which source classes to keep and how much to weight each. This class-level reweighting, rather than sample-level transport, is what makes the method computationally practical, and the Sinkhorn algorithm lets it scale to large datasets. Around this optimization the paper builds a theory of induced classifiers and a lifted output space to prove that minimizing this distance is a bound-minimization strategy for target error.
What would settle it
Run WaSS on a target task where ImageNet features are known to conflate semantically different classes, and compare against the ALL baseline; if the Wasserstein-selected subset does not improve target accuracy over uniform source weighting across several splits, the feature-space similarity premise fails.
Extended reading notes
Core claim
The central claim is that solving the linear program that minimizes $W_1(\sum_i w_i D^S_i(Z), D^T(Z))$ in a pretrained feature space yields the source class mixture that best facilitates transfer, and that a classifier trained on that mixture and fine-tuned on limited target labels reaches the highest target accuracy among the compared methods. The paper frames this as bound minimization: a generalization analysis shows the target error of an induced classifier is bounded by the source error plus the Wasserstein distance between reweighted source and target plus a term measuring the change of the final layer after fine-tuning. The method (WaSS) handles the disjoint-label setting by lifting the output space to the union of source and target labels, so the bound does not collapse even when no classes are shared.
Load-bearing premise
The load-bearing premise is that distances between ImageNet-pretrained ResNet-50 feature vectors are a trustworthy measure of how useful a source class is to a target task; the selection is computed once in that fixed space and never revised after the downstream encoder is trained, and the paper itself notes that a pretrained network may not always supply a meaningful embedding.
Editorial extensions
If this is right
- Transfer can work even when the source and target label sets are completely disjoint, because selection is driven by feature-space similarity rather than shared labels.
- The optimal source mixture can be found without re-training the encoder, reducing transfer learning to a linear program plus fine-tuning.
- Class-level reweighting with Sinkhorn scaling keeps the selection step feasible for large source datasets.
- The generalization bound implies that minimizing the Wasserstein term is a principled proxy for target accuracy, not just a heuristic.
Reading between the lines
- Our inference: a direct test of the method's core premise would be to compute the class weights with a randomly initialized encoder; if the selected subset still improves target accuracy, the Wasserstein geometry is not doing the claimed semantic work, and if it fails, the dependence on pretrained features is confirmed.
- Our inference: the paper fixes the pretrained encoder and never re-solves the linear program after fine-tuning, so a natural extension is to iterate selection and fine-tuning; substantial weight shifts across iterations would indicate that the fixed-feature assumption is material.
- Our inference: the same class-reweighting logic should carry to other modalities and to very large, uncurated source pools, where dropping far-away classes could reduce both compute and negative transfer simultaneously.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes WaSS, a two-stage transfer learning method for settings where the source and target label sets need not overlap. In the first stage, a linear program (Eq. 4) reweights source classes so that the reweighted mixture of source class-conditional feature distributions minimizes the empirical 1-Wasserstein distance to the target feature distribution in a fixed pretrained ResNet-50 embedding. In the second stage, a classifier is trained on the reweighted source data and its final layer is fine-tuned on limited target labels. The authors provide a generalization analysis intended to show that this procedure minimizes an upper bound on target error, and they report experiments in disjoint and open-set label shift settings on several datasets.
Significance. If the empirical results are reliable, WaSS is a simple and inexpensive recipe for source class selection under extreme label shift, and the DDA setting (no shared labels) is well motivated. The class-level reweighting by optimal transport in a fixed embedding is a natural idea that could be practically useful. However, the paper's theoretical justification is not currently valid, and the empirical presentation contains inconsistencies that need to be corrected before the claims can be evaluated. The paper does not release code, and the proofs are not machine-checked; the main strengths are the breadth of datasets, the comparison with several baselines, and the clear statement of the intended bound-minimization rationale.
major comments (5)
- [§3.3, Lemma 3.1] Lemma 3.1 as stated is false. Take Z={0,1}, Y={0,1}, d_Z=d_Y=1, P_Z=δ_0, Q_Z=δ_1, P(Y|Z=0)=δ_0 and Q(Y|Z=1)=δ_1. With arbitrary conditional values at null points set to make the conditional expectations vanish, the right-hand side is W1(P_Z,Q_Z)=1, while W1(P,Q)=2 because the single joint atoms are (0,0) and (1,1). The proof fails at Eqs. (18)-(20), where L(y_m,y_t) is set to zero because the two conditionals are 'the same'; the label cost does not vanish when z_m≠z_t. Since this lemma is the main tool connecting feature-space W1 to target error, the proposed upper bound and the 'bound minimization' claim are not established.
- [§3.3, after the upper bound] The statement that 'the conditional distribution DS(Y | Z) is invariant under the reweighting' is only true if the class-conditional feature supports are disjoint. In general, for the reweighted source DS(w)(Z)=Σ_i w_i DS_i(Z), the posterior is DS(w)(Y=y|Z=z)=w_y p_y(z)/(Σ_i w_i p_i(z)), which depends on w whenever the class-conditional densities p_i overlap; ResNet-50 features of natural images are not class-disjoint. Moreover, even if the posterior were fixed, the expectation E_DS(w)(Z)[W1(DS(Y|Z), DT(Y|Z))] depends on w through the outer marginal. Thus Eq. (4) minimizes only the second additive term in the bound, and the claim that it suffices to minimize the first two terms does not follow.
- [§3.3, Theorem 3.1 and Lemma 3.2] Theorem 3.1 is quoted with undefined notation (Ω, X) and an equality that is not stated as a distance or a bound, making the cited result impossible to verify. In the proof of Lemma 3.2, the 1/2 factor from Proposition 3.1 is dropped, the Lipschitz step around Eq. (35) omits the expectation on the right-hand side, and Proposition F.2 is misapplied to a matrix difference although it is stated for vectors. The fine-tuning bound αβσmax(VS−VT) is therefore not rigorously established, which matters because the final stage of WaSS is precisely a fine-tuning step.
- [§4.1, Table 1] The text says WaSS is outperformed in two experiments, but the table shows three losses: PACS [0,1,2] versus OSS, PACS [3,4,5] versus ALL, and C-100 [16,61,9] versus ALL. No standard deviations, confidence intervals, or per-run results are reported for any of the ten iterations, and the caption's paired t-test statement is not substantiated by the visible data; the p-value is only reported for Figure 1, not for Table 1. The 'average 1–2 absolute percentage point increase' claim should also be reconciled with the actual per-split differences.
- [Appendix C, Table 2] The test-class indices are outside the valid label ranges of the datasets. Office-31 has classes 0-30, but the table reports [31,32,33] and [62,63,64]; Office-Home has classes 0-64, but the table reports [65,66,67] and [130,131,132]; VisDA has classes 0-11, but the table reports [12,13,14] and [24,25,26]. If the experimental setup concatenates domains into a larger label set, the paper must state this explicitly; as written, these rows cannot correspond to any class in the datasets, and the results are not reproducible or interpretable.
minor comments (7)
- [§2] The cross-entropy formula is missing a minus sign; as written, ℓCE(g(f(x)), y) is a positive log-likelihood rather than the usual negative log-likelihood.
- [§3.1 and Appendix A] Equation (4) is stated as an exact linear program, but the experiments use Sinkhorn regularization; the entropy regularization parameter is a free parameter, and its effect on the recovered weights is neither discussed nor ablated.
- [§3.1] The claim that an optimal subset is obtained presumes that w* is sparse; the simplex constraint alone does not enforce sparsity, and the paper reports no statistics on the number of nonzero weights produced by the LP.
- [§4.2 and Figure 4] The target class set for Fashion-MNIST is given as {T-shirt/top, Trouser, Pullover} in the text but as {Ankle boot, T-shirt/top, Trousers} in the caption of Figure 4; these should be reconciled.
- [Appendix C, Table 2] The dataset name is spelled 'VizDa' in the table but 'VisDA' in the text; also, the sentence 'three of these datasets are used to emulate distribution shifts for the same set of classes' is unclear.
- [Appendix A and F] There are several typos: 'This allos us' should be 'This allows us', 'Lipchitz' should be 'Lipschitz', and the learning rate '1e4' should probably be '1e-4'.
- [General] The paper states that code will be released upon publication; for a method whose value hinges on solving a numerical LP, a pseudocode description or code release would substantially aid reproducibility.
Circularity Check
No significant circularity: WaSS's W1 objective and its theoretical bound share a term by design, but no fitted input is renamed as a prediction and no load-bearing self-citation exists.
full rationale
The paper's derivation chain is self-contained rather than circular. The linear program in Eq. (4) minimizes the empirical W1 distance between a reweighted source class mixture and the target in a fixed pretrained feature space; the generalization analysis in Section 3.3 then derives an upper bound that contains this same W1 term. That overlap is a constructive design choice—minimizing a term in a proved upper bound—not a case of fitting a parameter to data and then predicting that same quantity. The bound's remaining terms (source error, conditional label divergence, and final-layer discrepancy) are not estimated from or forced by the target accuracies reported in Tables 1 and 2, and the empirical comparison is against independent baselines (ALL, PADA, RND, MN, OSS) on held-out target classes. The paper contains no self-citations by the authors, and no cited theorem is invoked as a uniqueness claim to forbid alternative weightings. The claim in Section 3.3 that DS(Y | Z) is invariant under reweighting is questionable for overlapping feature supports and the 'bound minimization' proof has a gap, but that is a correctness and rigor issue, not circularity: the invariance assertion, even if false, does not make the algorithm's output equivalent to its input by construction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- Sinkhorn entropy regularization =
not reported
- Source training sample size =
not reported
- Target fine-tuning size =
100 per class
assumptions (4)
- domain assumption Pretrained ResNet-50 features make ℓ2 distance in Z a faithful proxy for semantic similarity.
- domain assumption A reweighted mixture of source class conditionals can approximate the target feature distribution closely enough for transfer.
- standard math The joint Wasserstein metric over Z x Y is the additive metric d(z,z')+L(y,y') used in Courty et al.
- ad hoc to paper Softmax is Lipschitz and feature vectors are bounded (Propositions F.1 and F.2).
Cite this review
Pith. "Pith review of Class-based Subset Selection for Transfer Learning under Extreme Label Shift." pith.science (2026). https://pith.science/paper/I54F4C3W
@misc{pith2026250100162,
author = {Pith},
title = {Pith review of: Class-based Subset Selection for Transfer Learning under Extreme Label Shift},
year = {2026},
howpublished = {\url{https://pith.science/paper/I54F4C3W}},
note = {Machine review of arXiv:2501.00162}
}
read the original abstract
Existing work within transfer learning often follows a two-step process -- pre-training over a large-scale source domain and then finetuning over limited samples from the target domain. Yet, despite its popularity, this methodology has been shown to suffer in the presence of distributional shift -- specifically when the output spaces diverge. Previous work has focused on increasing model performance within this setting by identifying and classifying only the shared output classes between distributions. However, these methods are inherently limited as they ignore classes outside the shared class set, disregarding potential information relevant to the model transfer. This paper proposes a new process for few-shot transfer learning that selects and weighs classes from the source domain to optimize the transfer between domains. More concretely, we use Wasserstein distance to choose a set of source classes and their weights that minimize the distance between the source and target domain. To justify our proposed algorithm, we provide a generalization analysis of the performance of the learned classifier over the target domain and show that our method corresponds to a bound minimization algorithm. We empirically demonstrate the effectiveness of our approach (WaSS) by experimenting on several different datasets and presenting superior performance within various label shift settings, including the extreme case where the label spaces are disjoint.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
For CIFAR-100, we maintain the dataset structure by sampling the test domain from a single superclass and using the other superclasses as the source domain. Similarly, for PACS, the three classes composing the target domain are selected from one of the distributions within the dataset while the other3 distributions are used for training. Implementation Al...
work page 2019
-
[2]
The same color corresponds to the same class
A14 (a) WaSS (b) OSS Figure 6: Class distributions for Cifar-10 of test set (inner circle) and class distribution of training set (outer circle) weighted by class selection methods: WaSS and OSS. The same color corresponds to the same class. similar trend to Table 1 appears within Table 2 where WaSS outperforms the baselines. Interestingly, WaSS maintaine...
work page 2018
- [9]
-
[13]
weakly trains a discriminator to iden- tify any target sample as an unknown with probability p. The feature discriminator tries to increase the error of the discriminator by either matching the source and target distribution for large p or making the source and target distribution separate for smallp. Within the paper, Saito et al. use a p of 0.5 such tha...
work page 2017
-
[15]
The following figures 9 10 show the test statistic and p-value results for each pair of methods on Fashion-MNIST and CIFAR-10. E Proof of Lemma 3.1 Lemma 3.1 (Error bound of Wasserstein dis- tance between joint distributions) For any two joint distributionsDS and DT over Z × Y, we have W1(DS, DT ) ≤ W1(DS(Z), DT (Z)) + min EDS (Z)[W1(DS(Y | Z), DT (Y | Z)...
-
[1991]
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Per- ona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Con- ference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer,
work page 2014
-
[2006]
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In 2009 IEEE Conference on Com- puter Vision and Pattern Recognition, pages 248–255, 2009a. doi: 10.1109/CVPR.2009.5206848. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. I...
arXiv 2009
-
[2009]
H. Wang, B. Li, and H. Zhao. Understanding gradual domain adaptation: Improved analysis, optimal path and beyond. arXiv preprint arXiv:2204.08200,
Show all 15 references
-
[2012]
Jiang, Y
J. Jiang, Y. Shu, J. Wang, and M. Long. Transfer- ability in deep learning: A survey.arXiv preprint arXiv:2201.05867,
-
[2015]
Mansour, M
Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms.arXiv preprint arXiv:0902.3430,
-
[2017]
J. Pang, W. Sun, C. Yang, J. Ren, R. Xiao, J. Zeng, and L. Lin. Zoom and learn: Generalizing deep stereo matching to novel domains. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2070–2079,
-
[2018]
Mousavi Kalan, Z
M. Mousavi Kalan, Z. Fabian, S. Avestimehr, and M. Soltanolkotabi. Minimax lower bounds for trans- fer learning with linear and one-hidden layer neural networks. Advances in Neural Information Process- ing Systems, 33:1959–1969,
1959
-
[2019]
Fashion-MNISTisthesimplestdatasetwecon- sider containing 60, 000 train and 10, 000 test grayscale images of ten different clothing items
12 A Additional Details About Experimental Setup Datasets We evaluate the performance of our method in Section 4 on four datasets: Fashion-MNIST [Xiao et al., 2017], CIFAR-10 [Krizhevsky, 2009], PACS [Zhou et al., 2020], and CIFAR-100 [Krizhevsky et al., 2009]. Fashion-MNISTis...
2017
-
[2020]
Norouzi, T
M. Norouzi, T. Mikolov, S. Bengio, Y. Singer, J. Shlens, A. Frome, G. S. Corrado, and J. Dean. Zero-shot learning by convex combination of semantic embed- dings. arXiv preprint arXiv:1312.5650,
-
[2021]
URL http://jmlr.org/ papers/v22/20-451.html. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lem- pitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17 (59):1–35, 2016a. URL http://jmlr.o...
2012
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.