REVIEW 3 major objections 4 minor 56 references
Multiple Riemannian Manifold-valued Descriptors based Image Set Classification with Multi-Kernel Metric Learning
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper argues that modeling image sets with covariance, subspace, and Gaussian descriptors at once, fused through Riemannian kernels and multi-kernel metric learning, improves classification over single-geometry and equal-weight…
desk verdict The multi-kernel fusion idea is a real but modest extension; the paper's blanket superiority claim is refuted by its own YTC table, so it needs major revision before I'd trust the empirical story. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the multi-kernel metric learning objective with adaptive gating. Equation (18) maximizes the trace ratio $R_b(W)/R_w(W)$ under a learned projection $W$, with three Riemannian kernels supplying Gram matrices $K^q$; the gating model of Eq. (28) is a softmax over per-descriptor scores, making each local kernel region's weight data-dependent. The optimization alternates an iterative trace-ratio solver for the projection $E$ with gradient ascent for the gating parameters $\delta_q$ and $\rho_q$.
What would settle it
Compute the combined kernel matrix $K = \sum_{q=1}^{3} \xi_q K^q$ on held-out pairs and test positive semidefiniteness; negative eigenvalues would show the three RKHS embeddings cannot act as one common feature space, undercutting the projection step.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a fusion recipe: encode each image set with three Riemannian manifold-valued descriptors, map each into a reproducing kernel Hilbert space using the Log-Euclidean kernel for covariance matrices and Gaussian-embedded SPD matrices and the Projection Metric kernel for linear subspaces, and then learn a projection into a common lower-dimensional subspace by maximizing the trace ratio of between-class to within-class scatter while a softmax gating model assigns per-descriptor weights. The paper claims this leads to improved image set classification; in the reported tables the proposed method has the highest accuracy on ETH-80, AFEW, and MDSD, while on YTC the earlier MMML method is listed at 76.70 percent against the proposed 74.82 percent.
Load-bearing premise
The derivation assumes the three kernel feature maps all live in a single common Hilbert space, so that one projection matrix can act on them together; the paper does not state or prove that common-space condition.
Editorial extensions
If this is right
- On the reported evidence, fusing the three descriptors beats each single descriptor on all four datasets, which is the paper's main argument for their complementarity.
- The adaptive per-descriptor weights are claimed to give an advantage over equal-weight fusion methods such as HERML and MMML, with the ETH-80, AFEW, and MDSD results consistent with that advantage.
- Because the same framework is applied to four very different tasks, the fusion recipe is presented as task-independent rather than tuned specifically to face recognition.
- The iterative trace-ratio and gradient-ascent procedure is reported to stabilize after a small number of iterations, so the method is usable without deep networks or manifold backpropagation.
Reading between the lines
- A testable extension is to examine whether the learned gating weights concentrate on one descriptor per dataset; the ablation results suggest Grassmann features matter most for faces and Gaussian features for scenes, implying a dataset-adaptive attention pattern.
- The paper does not exploit temporal order; injecting frame order into the set descriptors is a natural extension that could strengthen video-based tasks.
- The common-space assumption behind the derivation could be tested by checking positive semidefiniteness of the weighted sum of the three kernel matrices, since an indefinite sum would mean the projection step has no clean RKHS interpretation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an image-set classification framework that represents each image set simultaneously by a covariance matrix, a linear subspace, and a Gaussian distribution, which live on three different Riemannian manifolds. These heterogeneous descriptors are mapped into reproducing kernel Hilbert spaces via three Riemannian kernels, and a multi-kernel metric learning objective with adaptive gating weights is used to learn a common low-dimensional subspace for classification. The method is evaluated on four tasks: video-based face recognition (YTC), set-based object categorization (ETH-80), video-based emotion recognition (AFEW), and dynamic scene classification (MDSD). The Abstract, Section V, and the Conclusion claim that the reported results justify superiority over state-of-the-art methods.
Significance. If the empirical claim and the derivation were both sound, the work would be a useful contribution to image-set classification: it combines three complementary geometric descriptors, provides a kernel-based fusion mechanism, and includes ablations and a complexity analysis. The paper is best understood as an extension of the authors' earlier MMML framework, and it is commendable that the comparison includes that earlier method. However, the central headline claim is directly contradicted by the paper's own Table I, and the kernelized derivation has a load-bearing gap concerning the common Hilbert space assumption. These issues are central rather than cosmetic, so the paper in its current form does not support its main conclusion.
major comments (3)
- [V-B, Table I] The paper's central claim, stated in the Abstract and repeated in the Conclusion, is that extensive experimental results justify superiority over the state of the art. Table I reports 74.82% for the proposed method on YTC, while MMML [49], listed as a compared state-of-the-art method and the authors' own earlier work, achieves 76.70%. The text in Section V-B also states that 'the proposed algorithm shows better classification performance than them on this data.' This is a direct internal contradiction of the blanket-superiority claim on one of the four evaluation datasets. The issue is not a minor wording problem, because the claimed universal superiority is the headline result of the paper; the reported evidence refutes the claim as stated, and the authors would need to revise the claim and explain why the proposed extension underperforms its predecessor on YTC.
- [IV-B, Eqs. (23)-(26)] The kernelization step is not rigorously justified. Equations (16)-(17) use a single projection matrix W acting on feature maps φ_i^q, which requires all three descriptor-dependent feature maps to lie in a single common Hilbert space F. However, each Riemannian kernel function induces its own RKHS, and the paper does not state or prove a common-codomain assumption. In Eq. (23), the basis vector w_h is expanded as a linear combination of training samples φ_i^q with a q-dependent index, and Eq. (24) sums over q as if all φ_i^q belong to the same space in which the inner product defining K^q is valid. Without a precise statement of the shared Hilbert space, or a block-diagonal construction of U over the direct sum of the three RKHSs, the projection learned in Eqs. (25)-(27) has no well-defined action across the heterogeneous feature spaces. This is a load-bearing gap because the entire method depends on this derivation.
- [IV-C and V-G] The joint optimization of the projection E and the gating parameters (δ_q, ρ_q) is not shown to converge. Section IV-C alternates between solving a trace-ratio problem for E and gradient-ascent updates for the gating parameters, but the combined objective is nonconvex and no convergence guarantee is provided. Section V-G explicitly admits that 'it is hard for us to provide a systematic theoretical proof of convergence behavior,' and only reports objective values on AFEW and YTC. Since the optimization is central to the method, the paper should either provide a convergence argument or demonstrate stability across all four datasets and multiple random initializations; otherwise the reported results may depend on the initialization choices.
minor comments (4)
- [V-B] The numerical discrepancy between the proposed method and MMML on YTC should also be discussed in the text; currently MMML is not mentioned in the paragraph comparing the proposed method with LMKML and HERML, even though MMML is the strongest competitor on that dataset.
- [V-B through V-E] All experimental tables report only average rates without standard deviations or significance tests. Given that several comparisons are separated by less than one percentage point, error bars are needed to support claims of improvement.
- [IV-A, Eq. (13)] The regularization parameter α is fixed to 10^3 in all experiments, but no sensitivity analysis is provided; since this parameter affects the SPD regularization for both the covariance and Gaussian descriptors, its influence on the results should be reported.
- [Abstract and Introduction] The phrase 'monotonically increasing' in the Abstract is awkward, and the term 'well-equipped' is used repeatedly to describe Riemannian kernels; the wording should be revised for clarity.
Circularity Check
No circularity found; the method is a standard supervised multi-kernel metric-learning pipeline evaluated on external benchmark data.
full rationale
The paper's claimed contribution is an empirical image-set classification method that fuses three Riemannian descriptors with learned kernel weights and a projection. The derivation chain in Section IV-B uses standard RKHS kernel constructions: the kernels in Eqs. (5), (8), and (11) are cited from prior external works, and the representer-style expansion in Eq. (23) is the usual kernel trick, although the common-Hilbert-space assumption it relies on is a correctness concern rather than a circular one. The projection E and gating parameters (delta_q, rho_q) are optimized on training data via Eqs. (27)-(42), and classification is then evaluated on held-out test splits (YTC, ETH-80, AFEW, MDSD) against independent baselines; this is standard supervised learning, not a fitted quantity renamed as a prediction. The authors' own prior method MMML [49] appears as a baseline and in Section IV-F as an explicit comparison of technical differences; it is not invoked to justify the method's validity or to forbid alternatives. No self-definitional, fitted-input-as-prediction, or self-citation-load-bearing step is present. The internal Table I inconsistency (Proposed 74.82 vs MMML 76.70 on YTC) undermines the blanket superiority claim in the Abstract, but that is a factual/correctness issue and is outside the scope of circularity analysis.
Assumptions & free parameters
free parameters (5)
- alpha (regularization constant) =
10^3
- d_w (target subspace dimensionality) =
YTC: 70, MDSD: 25, AFEW: 70, ETH-80: 8
- gamma (learning rate) =
10^-4
- B (iteration count) and epsilon (convergence error) =
Not specified numerically
- delta_q, rho_q (gating model parameters) =
Learned from training data (not reported)
assumptions (4)
- domain assumption The three Riemannian kernel functions (LEM, projection kernel, Gaussian-embedded LEM) are positive definite and preserve the relevant manifold structure.
- standard math A d-dimensional Gaussian distribution can be represented uniquely as a point in Sym+_{d+1} via the information-geometric embedding of Eq. (10).
- ad hoc to paper There exists a single common Hilbert space F shared by the feature maps of all three kernels, so a single projection matrix W (or E) can be applied across the heterogeneous descriptor spaces.
- domain assumption The trace ratio optimizer [48] converges to the global optimum, and this convergence survives interleaving with gradient ascent on the gating parameters.
Cite this review
Pith. "Pith review of Multiple Riemannian Manifold-valued Descriptors based Image Set Classification with Multi-Kernel Metric Learning." pith.science (2026). https://pith.science/paper/GXNKM2QL
@misc{pith2026190801950,
author = {Pith},
title = {Pith review of: Multiple Riemannian Manifold-valued Descriptors based Image Set Classification with Multi-Kernel Metric Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/GXNKM2QL}},
note = {Machine review of arXiv:1908.01950}
}
read the original abstract
The importance of wild video based image set recognition is becoming monotonically increasing. However, the contents of these collected videos are often complicated, and how to efficiently perform set modeling and feature extraction is a big challenge for set-based classification algorithms. In recent years, some proposed image set classification methods have made a considerable advance by modeling the original image set with covariance matrix, linear subspace, or Gaussian distribution. As a matter of fact, most of them just adopt a single geometric model to describe each given image set, which may lose some other useful information for classification. To tackle this problem, we propose a novel algorithm to model each image set from a multi-geometric perspective. Specifically, the covariance matrix, linear subspace, and Gaussian distribution are applied for set representation simultaneously. In order to fuse these multiple heterogeneous Riemannian manifoldvalued features, the well-equipped Riemannian kernel functions are first utilized to map them into high dimensional Hilbert spaces. Then, a multi-kernel metric learning framework is devised to embed the learned hybrid kernels into a lower dimensional common subspace for classification. We conduct experiments on four widely used datasets corresponding to four different classification tasks: video-based face recognition, set-based object categorization, video-based emotion recognition, and dynamic scene classification, to evaluate the classification performance of the proposed algorithm. Extensive experimental results justify its superiority over the state-of-the-art.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[49]
Wang R, Wu X J, Chen K X, Kittler J, Multiple Manifolds Metric Learning with Application to Image Set Classification, in: ICPR (2018) 627−632
work page 2018
-
[1]
J. R. Barr, K. W. Bowyer, P. J. Flynn, S. Biswas, Face recognition from video: A review, International Journal of Pattern Recognition and Artificial Intelligence 26 (05) (2012) 1266002 −1266002−53
work page 2012
-
[2]
Z. Shaohua Kevin, C. Rama, M. Baback, Visual tracking and recognition using appearance-adaptive models in particle filters, IEEE TIP 13 (11) (2004) 1491−1506
work page 2004
-
[3]
N. Ye, Y . Ning, T. Sim, Towards general motion-based face recognition, in: CVPR (2010) 2598−2650
work page 2010
-
[4]
M. Kim, S. Kumar, V . Pavlovic, H. Rowley, Face tracking and recognition with visual constraints in real-world videos, in: CVPR (2008) 1 −8
work page 2008
-
[5]
Chen Z, Wu X J, Kittler J, A sparse regularized nuclear norm based matrix regression for face recognition with contiguous occlusion, Pattern Recognition Letters, 2019
work page 2019
-
[6]
Yu D, Wu X J, 2DPCANet: a deep leaning network for face recognition, Multimedia Tools and Applications, 77(10) (2018) 12919−12934
work page 2018
-
[7]
O. Arandjelovic, G. Shakhnarovich, J. Fisher, R. Cipolla, T. Darrell, Face recognition with image sets using manifold density divergence, in: CVPR (2005) 581−588
work page 2005
Show all 56 references
-
[8]
Cevikalp, B
H. Cevikalp, B. Triggs, Face recognition based on image sets, in: CVPR (2010) 2567−2573
2010
-
[9]
Y . Hu, A. S. Mian, R. Owens, Sparse approximated nearest points for image set classification, in: CVPR (2011) 121 −128
2011
-
[10]
W. Wang, R. Wang, Z. Huang, S. Shan, X. Chen, Discriminant analysis on riemannian manifold of gaussian distributions for face recognition with image sets, in: CVPR (2015) 2048−2057
2015
-
[11]
Yamaguchi, K
O. Yamaguchi, K. Fukui, K.-i. Maeda, Face recognition using temporal image sequence, in: FG (1998) 318−323
1998
-
[12]
J. Hamm, D. D. Lee, Grassmann discriminant analysis: a unifying view on subspace-based learning, in: ICML (2008) 376 −383
2008
-
[13]
Z. Wu, Y . Huang, L. Wang, Learning representative deep features for image set analysis, IEEE TMM 17 (11) (2015) 1960 −1968
2015
-
[14]
R. Wang, H. Guo, L. S. Davis, Q. Dai, Covariance discriminative learning: A natural and efficient approach to image set classification, in: CVPR (2012) 2496−2503
2012
-
[15]
Huang, R
Z. Huang, R. Wang, S. Shan, X. Li, X. Chen, Log-euclidean metric learning on symmetric positive definite manifold with application to image set classification, in: ICML (2015) 720 −729
2015
-
[16]
Harandi, M
M. Harandi, M. Salzmann, R. Hartley, Dimensionality reduction on spd manifolds: The emergence of geometry-aware methods, IEEE TPAMI (2018) 48−62
2018
-
[17]
Tuzel, F
O. Tuzel, F. Porikli, P. Meer, Region covariance: A fast descriptor for detection and classification (2006) 589−600
2006
-
[18]
Tuzel, F
O. Tuzel, F. Porikli, P. Meer, Pedestrian detection via classification on riemannian manifolds, IEEE TPAMI (2008) 1713−1727
2008
-
[19]
Turaga, A
P. Turaga, A. Veeraraghavan, A. Srivastava, R. Chellappa, Statistical computations on grassmann and stiefel manifolds for image and video- based recognition, IEEE TPAMI (2011) 2273−2286
2011
-
[20]
Pennec, P
X. Pennec, P. Fillard, N. Ayache, A riemannian framework for tensor computing, IJCV (2006) 41−66
2006
-
[21]
Arsigny, P
V . Arsigny, P. Fillard, X. Pennec, N. Ayache, Geometric means in a novel vector space structure on symmetric positive-definite matrices, SIAM journal on matrix analysis and applications (2007) 328 −347
2007
-
[22]
Sra, Positive definite matrices and the s-divergence, in: Proceedings of the American Mathematical Society (2016) 2787 −2797
S. Sra, Positive definite matrices and the s-divergence, in: Proceedings of the American Mathematical Society (2016) 2787 −2797
2016
-
[23]
Huang, R
Z. Huang, R. Wang, S. Shan, X. Chen, Projection metric learning on grassmann manifold with application to video based face recognition, in: CVPR(2015) 140−149
2015
-
[24]
M. T. Harandi, C. Sanderson, R. Hartley, B. C. Lovell, Sparse coding and dictionary learning for symmetric positive definite matrices: A kernel approach, in: ECCV (2012) 216−229
2012
-
[25]
Jayasumana, R
S. Jayasumana, R. Hartley, M. Salzmann, H. Li, M. Harandi, Kernel methods on the riemannian manifold of symmetric positive definite matrices, in: CVPR (2013) 73−80
2013
-
[26]
Huang, L
Z. Huang, L. J. Van Gool, A riemannian network for spd matrix learning,in: AAAI (2017) 3
2017
-
[27]
Huang, J
Z. Huang, J. Wu, L. Van Gool, Building deep networks on grassmann manifolds, in: AAAI (2018)
2018
-
[28]
Huang, C
Z. Huang, C. Wan, T. Probst, L. Van Gool, Deep learning on lie groups for skeleton-based action recognition, in: CVPR (2017) 1243 −1252
2017
-
[29]
Shakhnarovich, J
G. Shakhnarovich, J. W. Fisher, T. Darrell, Face recognition from long- term observations, in: ECCV (2002) 851−865
2002
-
[30]
T. M. Cover, J. A. Thomas, Elements of information theory, John Wiley & Sons, 2012
2012
-
[31]
Amari, H
S.-i. Amari, H. Nagaoka, Methods of information geometry, American Mathematical Soc., 2007
2007
-
[32]
Lovric, M
M. Lovric, M. Min-Oo, E. A. Ruh, et al., Multivariate normal dis- tributions parametrized as a riemannian symmetric space, Journal of Multivariate Analysis (2000) 36−48
2000
-
[33]
Huang, R
Z. Huang, R. Wang, S. Shan, X. Chen, Face recognition on large-scale video in the wild with hybrid euclidean-and-riemannian metric learning, Pattern Recognition (2015) 3113−3124
2015
-
[34]
W. Wang, R. Wang, S. Shan, X. Chen, Discriminative covariance oriented representation learning for face recognition with image sets, in: CVPR(2017) 5599−5608
2017
-
[35]
M. H. Quang, M. San Biagio, V . Murino, Log-hilbert-schmidt metric between positive definite operators on hilbert spaces, in: NIPS (2014) 388−396
2014
-
[36]
D. J. Hand, Kernel discriminant analysis, New York: Research studies press (1982)
1982
-
[37]
H. Sun, X. Zhen, Y . Zheng, G. Yang, Y . Yin, S. Li, Learning deep match kernels for image-set classification, in: CVPR (2017) 3307 −3316
2017
-
[38]
J. Lu, G. Wang, W. Deng, P. Moulin, J. Zhou, Multi-manifold deep metric learning for image set classification, in: CVPR (2015) 1137−1145
2015
-
[39]
J. Lu, G. Wang, P. Moulin, Image set classification using holistic mul- tiple order statistics features and localized multi-kernel metric learning, in: ICCV (2013) 329−336
2013
-
[40]
Pennec, P
X. Pennec, P. Fillard, N. Ayache, A riemannian framework for tensor computing, International Journal of computer vision (2006) 41 −66
2006
-
[41]
Absil, R
P.-A. Absil, R. Mahony, R. Sepulchre, Optimization algorithms on matrix manifolds, Princeton University Press, 2009
2009
-
[42]
Edelman, T
A. Edelman, T. A. Arias, S. T. Smith, The geometry of algorithms with orthogonality constraints, SIAM journal on Matrix Analysis and Applications (1998) 303−353
1998
-
[43]
M and Alpaydin
Gonen. M and Alpaydin. E, Localized multiple kernel learning, in: ICML (2008) 352−353
2008
-
[44]
Y , Methodologies for cross-domain data fusion: An overview, IEEE Transactions on Big Data, 1 (1) (2015) 16 −34
Zheng. Y , Methodologies for cross-domain data fusion: An overview, IEEE Transactions on Big Data, 1 (1) (2015) 16 −34
2015
-
[45]
Zhang. Z, Jia. L, Zhao. M, Liu. G, Wang. M, Yan. S, Kernel-induced label propagation by mapping for semi-supervised classification. IEEE Transactions on Big Data, 2018
2018
-
[46]
Baudat, F
G. Baudat, F. Anouar, Generalized discriminant analysis using a kernel approach, Neural computation (2000) 2385−2404
2000
-
[47]
Fukunaga, Introduction to statistical pattern recognition, Elsevier, 2013
K. Fukunaga, Introduction to statistical pattern recognition, Elsevier, 2013
2013
-
[48]
H. Wang, S. Yan, D. Xu, X. Tang, T. Huang, Trace ratio vs. ratio trace for dimensionality reduction, in: CVPR (2007) 1 −8
2007
-
[50]
Dhall, R
A. Dhall, R. Goecke, J. Joshi, K. Sikka, T. Gedeon, Emotion recognition in the wild challenge 2014: Baseline, data and protocol, in: ICMI (2014) 461−466
2014
-
[51]
Davis. J. V , Kulis. B, Jain. P, Sra. S, Dhillon. I. S, Information-theoretic metric learning, in: ICML (2007)
2007
-
[52]
Y , Zenios
Censor. Y , Zenios. S, Parallel optimization: Theory, algorithms, and applications, Oxford University Press (1997)
1997
-
[53]
M. T. Harandi, C. Sanderson, S. Shirazi, B. C. Lovell, Graph embedding discriminant analysis on grassmannian manifolds for improved image setmatching, in: CVPR (2011) 2705−2712
2011
-
[54]
Kim. T. K, Kittler. J, Cipolla. R, Discriminative learning and recognition of image set classes using canonical correlations, IEEE TPAMI (2007) 1005−1018
2007
-
[55]
R. Wang, S. Shan, X. Chen, W. Gao, Manifold-manifold distance with application to face recognition based on image set (2008) 1 −8
2008
-
[56]
Harandi, M
M. Harandi, M. Salzmann, M. Baktashmotlagh, Beyond gauss: Image- set matching on the riemannian manifold of pdfs, in: ICCV (2015) 4112−4120
2015
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.