REVIEW 4 major objections 6 minor 37 references
Zero-Shot Feature Selection via Transferring Supervised Knowledge
T0 review · 4 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Guiding feature selection with class attributes instead of labels lets a model trained on seen concepts pick features that also separate unseen concepts.
desk verdict Genuinely new problem and a plausible attribute-transfer idea, but the relaxed objective in Eq. 5 is unbounded below, so the reported feature rankings may be artifacts of stopping time until the constraint is restored. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the joint objective over the feature importance vector $s$ and the linear generating matrix $W$: $$\|Y_s - X\operatorname{diag}(s)W\|$_F^{2}$ + \$\alpha$\|Y_s - \bar{X}\operatorname{diag}(s)W\|$_F^{2}$ + \gamma\|W\|$_F^{2}$,$$ where $Y_s$ holds class attributes, $X$ is the data, and $\bar{X}$ holds class centers. The first term asks selected features to linearly reconstruct attributes, and the second term (the center-characteristic loss) asks them to reconstruct class centers under the same map, which steers selection toward stable central characteristics rather than idiosyncratic or noisy attribute correspondences. Relaxing the 0/1 selection constraint to $s\ge 0$ turns $s$ into a feature importance ranking, and alternating updates converge to a solution.
What would settle it
On a dataset with a known seen/unseen split, randomly permute the attribute values of the seen classes while keeping everything else fixed; if SemFS's unseen-class clustering advantage over label-based baselines does not drop substantially, then the attribute content is not what drives the result. More directly, construct a dataset whose true attributes are nonlinear functions of the features and show that a nonlinear attribute-reconstruction selector outperforms SemFS on unseen classes, which would indicate the linear transfer assumption is the limiting factor.
Extended reading notes
Core claim
The central claim is that the reason supervised feature selection fails on unseen concepts is the supervision itself: 0/1 class labels encode only distinctions among seen classes, so nothing transfers. Replacing labels with class-semantic attributes gives the selector a shared semantic layer, and a linear map from selected features to attributes learned on seen classes keeps working for unseen classes. The paper also claims that simply regressing on attributes is not enough because attributes are noisy, so it adds the center-characteristic loss, which requires the same linear map to reconstruct each seen class's center from its selected features. The joint model, SemFS, produces a non-negative feature importance vector through alternating least squares and projected gradient descent, and ranking by that vector yields the selected features. This, the authors argue, is the first formulation of the zero-shot setting for feature selection.
Load-bearing premise
The method assumes that a linear map from selected features to attributes learned on seen classes remains valid for unseen classes, and that the attributes themselves carry discriminative information rather than noise.
Editorial extensions
If this is right
- A feature selection model trained once on seen concepts can be applied directly to unseen concepts, as long as attribute descriptions for those concepts are available.
- Unlabeled unseen data can be clustered on the selected features, since the paper's evaluation shows K-means on SemFS-selected features yielding higher ACC and NMI than baselines.
- Supervised selectors that use class labels may be actively misleading for unseen concepts; attribute-based supervision avoids this label bias.
- Automatically generated text-derived attributes, not only manually curated ones, are sufficient to transfer feature selection knowledge.
Reading between the lines
- The same swap of labels for attributes could be applied to other supervised feature selection objectives, so SemFS is likely an instance of a general recipe rather than a single algorithm.
- If attribute noise is the main risk, a natural extension is to model attribute uncertainty explicitly, for example by weighting attributes by reliability or adding a per-attribute variance term.
- The paper evaluates transfer only through clustering; a direct test on zero-shot classification with the selected features would be a stronger end-to-end check of the claim.
- On CIFAR10 only two seen classes support the transfer, so the extreme low-resource regime is where the assumption of a linear attribute map is most strained and where nonlinear mappings would be worth testing.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the zero-shot feature selection (ZSFS) problem, in which a feature selection model is trained on seen classes and then applied to select features for unseen classes that share no labels with the training classes. The proposed method, SemFS, uses class-attribute descriptions (manually annotated attributes or automatically generated word vectors) as supervision instead of class labels, and adds a center-characteristic loss that encourages the selected features to preserve the central characteristics of seen classes in the attribute space. The binary cardinality-constrained selection problem is relaxed to a non-negative score vector, and optimization alternates between a closed-form update for a linear mapping W and projected gradient descent on the score vector s. Experiments on SUN, aPY, and CIFAR10 evaluate the selected features by clustering unseen-class data and report ACC and NMI improvements over several unsupervised and supervised feature selection baselines.
Significance. Zero-shot feature selection is a relevant and little-studied problem, and the idea of using attribute embeddings as a transferable supervision signal is natural and potentially impactful. The center-characteristic loss is a sensible adaptation of center loss to feature selection, and the experiments cover both manually annotated and automatically generated attributes. The paper does not provide code, but the pseudocode is clear and the method is simple. However, the technical soundness of the optimization is compromised by a relaxation that yields an objective without a finite minimizer, and the experimental evidence lacks statistical rigor and a full-feature baseline. These issues are load-bearing for the paper's central claims, so the manuscript requires substantial revision before it can be considered for publication.
major comments (4)
- [Section 4.3, Eq. (5)] The relaxation from Eq. (4) to Eq. (5) drops the cardinality constraint s^T 1_d = k as well as the 0/1 constraint, leaving only s ≥ 0. As written, the objective in Eq. (5) has no finite minimizer. Writing U = diag(s)W, the first two terms become ||Y_s - XU||_F^2 + α||Y_s - \bar{X}U||_F^2, which are independent of s, while the regularization term becomes γ||diag(s)^{-1}U||_F^2. For any feasible (s,W) with positive s_i and U nonzero, the scaling (s,W) → (t s, W/t) strictly decreases the objective for t > 1, so the infimum is approached only as t → ∞. Consequently, the 'solution' s is not well-defined, and the ranking produced by Algorithm 1 is an artifact of the initialization, step size, and stopping criterion. Proposition 1 only shows monotone decrease of the objective value; it does not establish convergence of s, and the cited projected-gradient convergence result does not apply because the minimum does not exist. This undermines the central claim that SemFS selects features by optimizing a well-defined criterion. The authors should restore a normalization or sparsity constraint (e.g., s^T 1_d = k or an equivalent penalty that prevents s from diverging), re-derive the updates, and rerun all experiments.
- [Section 6.2, Figures 4 and 5] The text states that SemFS and SemFS/c 'always outperform the others significantly,' but no error bars, confidence intervals, or significance tests are reported. The clustering metrics are averaged over 20 K-means initializations, yet the variance across those runs is not shown, and for CIFAR10 the variance due to the choice of seen/unseen split is also unaccounted for. Without measures of variability or statistical tests, the claim of 'significantly' better performance is unsupported. Please report standard deviations and, where appropriate, paired significance tests across repeated clustering runs and data splits.
- [Section 6.1, Figures 4(c) and 5(c)] The description of the CIFAR10 experiment is ambiguous. The text says 'we randomly adopt two classes as seen' and then notes there are C(10,2) different seen/unseen splits, but Figures 4(c) and 5(c) appear to show a single curve. It is unclear whether the reported result is from one randomly chosen split or averaged over all 45 splits. Since CIFAR10 is the key demonstration for the extreme case with automatically generated attributes, this ambiguity materially affects the claim. Please clarify the protocol and, ideally, report the average and standard deviation over all 45 splits.
- [Section 6.2] No baseline using all 4,096 features is reported. As the paper focuses on feature selection, it is essential to compare the selected features against the full feature set to show that the benefit of SemFS comes from attribute-guided selection rather than from dimensionality reduction alone. A simple K-means clustering result using all 4,096 features for each dataset should be added to Figures 4 and 5, or else the advantage of the method over the full set must be discussed explicitly.
minor comments (6)
- [Section 6.5] In Section 6.5 the text says 'with the fixed regularization parameter λ=1', but Section 6.1.2 reports γ=0.1; please use consistent notation for the regularization parameter.
- [Section 5.1, Algorithm 1] The proof of Proposition 1 relies on an 'appropriate choice of step size' for the projected gradient update, but the algorithm does not specify how the step size is chosen or whether a line search is used. This is a reproducibility issue even apart from the non-existence problem in Eq. (5).
- [Section 2.2] The claim 'to our best knowledge, this is the first study to consider the zero-shot setting in the feature selection problem' is strong and difficult to verify from the provided references; please soften the claim or support it with a more thorough literature review.
- [Figure 7] The axis labels 'Lassoo' and 'Lassos' contain typos and should be corrected to 'Lasso_o' and 'Lasso_s'.
- [Equations (2) and (3)] The transition from Eq. (2) to Eq. (3) replaces Xdiag(s) with Y_s without an explicit derivation; the heuristic of substituting the attribute reconstruction from Eq. (1) should be described more clearly to avoid confusion.
- [Section 3] The definition of the class center \bar{x}_i is ambiguous because y_i is a label vector, not a class index; please clarify, for example by defining a class index c_i and writing \bar{x}_{c_i}.
Circularity Check
No significant circularity: SemFS is trained only on seen-class attributes and evaluated on unseen-class clustering, with hand-fixed hyperparameters.
full rationale
The derivation chain is not circular. The model in Eq. 4 is fit using only seen-concept instances X and their attributes Ys; unseen-class data and attributes are never used to fit s or W. The reported ACC/NMI numbers are produced by running K-means on unseen-class features ranked by the learned s, which is a separate evaluation signal rather than a quantity already encoded in the objective. Hyperparameters α and γ are fixed by hand (α=1, γ=0.1) rather than tuned on the test clusters, and baselines are grid-searched while the proposed method is not, which if anything disadvantages the proposed method. There are no load-bearing self-citations, no imported uniqueness theorems, and no prediction that is definitionally equal to an input. The only concern surfaced by the skeptical analysis is that the relaxed problem in Eq. 5 drops the cardinality constraint and is therefore potentially unbounded under joint scaling of s and W, so the reported ranking may depend on optimization stopping behavior. That is an optimization/reproducibility concern, not circularity: the objective still depends on attributes in the claimed direction, and the evaluation remains external to the training objective. Overall, the central claim is independently supported by the evaluation design, so no circular step is identified.
Assumptions & free parameters
free parameters (4)
- α (center-characteristic loss weight) =
1
- γ (regularization weight) =
0.1
- PGD step size
- Number of iterations
assumptions (5)
- domain assumption A linear mapping W from selected features to attributes captures the relationship for both seen and unseen concepts.
- domain assumption Attributes are shared between seen and unseen concepts and contain discriminative information about unseen classes.
- domain assumption Class centers Xbar (mean feature vectors) are stable enough to guide feature selection.
- ad hoc to paper The relaxed non-negative score s, after ranking, yields the optimal top-k feature subset of the original 0/1 problem.
- ad hoc to paper Projected gradient descent converges to the minimum of the non-convex joint objective.
Cite this review
Pith. "Pith review of Zero-Shot Feature Selection via Transferring Supervised Knowledge." pith.science (2026). https://pith.science/paper/RIAB35CT
@misc{pith2026190803464,
author = {Pith},
title = {Pith review of: Zero-Shot Feature Selection via Transferring Supervised Knowledge},
year = {2026},
howpublished = {\url{https://pith.science/paper/RIAB35CT}},
note = {Machine review of arXiv:1908.03464}
}
read the original abstract
Feature selection, an effective technique for dimensionality reduction, plays an important role in many machine learning systems. Supervised knowledge can significantly improve the performance. However, faced with the rapid growth of newly emerging concepts, existing supervised methods might easily suffer from the scarcity and validity of labeled data for training. In this paper, the authors study the problem of zero-shot feature selection (i.e., building a feature selection model that generalizes well to "unseen" concepts with limited training data of "seen" concepts). Specifically, they adopt class-semantic descriptions (i.e., attributes) as supervision for feature selection, so as to utilize the supervised knowledge transferred from the seen concepts. For more reliable discriminative features, they further propose the center-characteristic loss which encourages the selected features to capture the central characteristics of seen concepts. Extensive experiments conducted on various real-world datasets demonstrate the effectiveness of the method.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Multi-cue zero-shot learning with strong supervision
Akata, Z., Malinowski, M., Fritz, M., Schiele, B., 2016. Multi-cue zero-shot learning with strong supervision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 59–68
work page 2016
-
[2]
Pca disjoint models for mul- ticlass cancer analysis using gene expression data
Bicciato, S., Luchini, A., Di Bello, C., 2003. Pca disjoint models for mul- ticlass cancer analysis using gene expression data. Bioinformatics 19 (5), 571–578
work page 2003
-
[3]
Weighted nearest neighbors feature selection
Bugata, P., Drot´ ar, P., 2019. Weighted nearest neighbors feature selection. Knowledge-Based Systems 163, 749–761
work page 2019
-
[4]
Unsupervised feature selection for multi- cluster data
Cai, D., Zhang, C., He, X., 2010. Unsupervised feature selection for multi- cluster data. In: Proceedings of the 16th ACM SIGKDD international con- ference on Knowledge discovery and data mining. ACM, pp. 333–342. 20
work page 2010
-
[5]
Exact top-k feature selection via l2, 0-norm constraint
Cai, X., Nie, F., Huang, H., 2013. Exact top-k feature selection via l2, 0-norm constraint. In: International Joint Conference on Artificial Intelli- gence (IJCAI). pp. 1240–1246
work page 2013
-
[6]
Calamai, P. H., Mor´ e, J. J., 1987. Projected gradient methods for linearly constrained problems. Mathematical programming 39 (1), 93–116
work page 1987
-
[7]
A survey on feature selection methods
Chandrashekar, G., Sahin, F., 2014. A survey on feature selection methods. Computers & Electrical Engineering 40 (1), 16–28
work page 2014
-
[8]
Bidirectional feature selection with global and local structure preservation for small size samples
Chu, Y., Zhao, Y., 2018. Bidirectional feature selection with global and local structure preservation for small size samples. Cognitive Systems Re- search 52, 756–764
work page 2018
Show all 37 references
-
[9]
Feature selection for clustering
Dash, M., Liu, H., 2000. Feature selection for clustering. In: Pacific-Asia Conference on knowledge discovery and data mining. Springer, pp. 110–121
2000
-
[10]
Unsupervised feature selection with adaptive structure learning
Du, L., Shen, Y.-D., 2015. Unsupervised feature selection with adaptive structure learning. In: Proceedings of the 21th ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining. ACM, pp. 209–218
2015
-
[11]
O., Hart, P
Duda, R. O., Hart, P. E., Stork, D. G., 1995. Pattern classification and scene analysis 2nd ed. ed: Wiley Interscience
1995
-
[12]
O., Hart, P
Duda, R. O., Hart, P. E., Stork, D. G., 2012. Pattern classification. John Wiley & Sons
2012
-
[13]
Describing objects by their attributes
Farhadi, A., Endres, I., Hoiem, D., Forsyth, D., 2009. Describing objects by their attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 1778–1785
2009
-
[14]
Semi-supervised active learn- ing with cross-class sample transfer
Guo, Y., Ding, G., Gao, Y., Wang, J., 2016. Semi-supervised active learn- ing with cross-class sample transfer. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence. pp. 1526–1532. 21
2016
-
[15]
An introduction to variable and feature se- lection
Guyon, I., Elisseeff, A., 2003. An introduction to variable and feature se- lection. Journal of machine learning research 3 (Mar), 1157–1182
2003
-
[16]
H., Socher, R., Manning, C
Huang, E. H., Socher, R., Manning, C. D., Ng, A. Y., 2012. Improving word representations via global context and multiple word prototypes. In: Pro- ceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. pp. 873–882
2012
-
[17]
Zero-shot recognition with unreliable attributes
Jayaraman, D., Grauman, K., 2014. Zero-shot recognition with unreliable attributes. In: Advances in Neural Information Processing Systems. pp. 3464–3472
2014
-
[18]
Manifold embedding for zero-shot recognition
Ji, Z., Yu, X., Yu, Y., He, Y., 2019. Manifold embedding for zero-shot recognition. Cognitive Systems Research 55, 34–43
2019
-
[19]
S., 2010
Kong, X., Yu, P. S., 2010. Semi-supervised feature selection for graph clas- sification. In: Proceedings of the 16th ACM SIGKDD international confer- ence on Knowledge discovery and data mining. ACM, pp. 793–802
2010
-
[20]
Learning multiple layers of features from tiny images
Krizhevsky, A., Hinton, G., 2009. Learning multiple layers of features from tiny images
2009
-
[21]
H., Nickisch, H., Harmeling, S., 2009
Lampert, C. H., Nickisch, H., Harmeling, S., 2009. Learning to detect un- seen object classes by between-class attribute transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 951–958
2009
-
[22]
A., 1988
Lee Rodgers, J., Nicewander, W. A., 1988. Thirteen ways to look at the correlation coefficient. The American Statistician 42 (1), 59–66
1988
-
[23]
A sharing multi-view feature selection method via alternating direction method of multipliers
Lin, Q., Xue, Y., Wen, J., Zhong, P., 2018. A sharing multi-view feature selection method via alternating direction method of multipliers. Neuro- computing
2018
-
[24]
G., Ye, Y., 2015
Luenberger, D. G., Ye, Y., 2015. Linear and nonlinear programming. Vol
2015
-
[25]
H., 2010
Nie, F., Huang, H., Cai, X., Ding, C. H., 2010. Efficient and robust fea- ture selection via joint l2, 1-norms minimization. In: Advances in neural information processing systems. pp. 1813–1821
2010
-
[26]
Trace ratio criterion for feature selection
Nie, F., Xiang, S., Jia, Y., Zhang, C., Yan, S., 2008. Trace ratio criterion for feature selection. In: AAAI. Vol. 2. pp. 671–676
2008
-
[27]
Sun attribute database: Discovering, an- notating, and recognizing scene attributes
Patterson, G., Hays, J., 2012. Sun attribute database: Discovering, an- notating, and recognizing scene attributes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 2751– 2758
2012
-
[28]
Qiao, R., Liu, L., Shen, C., Hengel, A. v. d., 2016. Less is more: zero- shot learning from online textual documents with noise suppression. arXiv preprint arXiv:1604.01146
2016 arXiv
-
[29]
Joint image-text rep- resentation by gaussian visual-semantic embedding
Ren, Z., Jin, H., Lin, Z., Fang, C., Yuille, A., 2016. Joint image-text rep- resentation by gaussian visual-semantic embedding. In: Proceedings of the 2016 ACM on Multimedia Conference. ACM, pp. 207–211
2016
-
[30]
What helps where–and why? semantic relatedness for knowledge transfer
Rohrbach, M., Stark, M., Szarvas, G., Gurevych, I., Schiele, B., 2010. What helps where–and why? semantic relatedness for knowledge transfer. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 910–917
2010
-
[31]
The westbury lab wikipedia corpus
Shaoul, C., 2010. The westbury lab wikipedia corpus. Edmonton, AB: Uni- versity of Alberta
2010
-
[32]
Unsupervised feature selection for linked social media data
Tang, J., Liu, H., 2012. Unsupervised feature selection for linked social media data. In: Proceedings of the 18th ACM SIGKDD international con- ference on Knowledge discovery and data mining. ACM, pp. 904–912
2012
-
[33]
Regression shrinkage and selection via the lasso
Tibshirani, R., 1996. Regression shrinkage and selection via the lasso. Jour- nal of the Royal Statistical Society. Series B (Methodological), 267–288. 23
1996
-
[34]
A discriminative feature learning approach for deep face recognition
Wen, Y., Zhang, K., Li, Z., Qiao, Y., 2016. A discriminative feature learning approach for deep face recognition. In: European Conference on Computer Vision. Springer, pp. 499–515
2016
-
[35]
R.-T., Jin, R., 2010
Xu, Z., King, I., Lyu, M. R.-T., Jin, R., 2010. Discriminative semi- supervised feature selection via manifold regularization. IEEE Transactions on Neural networks 21 (7), 1033–1047
2010
-
[36]
Semi-supervised feature selection based on local discriminative information
Zeng, Z., Wang, X., Zhang, J., Wu, Q., 2016. Semi-supervised feature selection based on local discriminative information. Neurocomputing 173, 102–109
2016
-
[37]
Trace ratio criterion based discriminative feature selection via l2, p-norm regularization for supervised learning
Zhao, M., Lin, M., Chiu, B., Zhang, Z., Tang, X.-s., 2018. Trace ratio criterion based discriminative feature selection via l2, p-norm regularization for supervised learning. Neurocomputing 321, 1–16. 24
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.