REVIEW 2 major objections 5 minor 37 references
ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read ViSiL claims video similarity should be computed from refined frame-to-frame similarity matrices built on regional CNN features, not from global video embeddings, and reports large gains across four retrieval tasks.
desk verdict ViSiL is a genuinely new and mostly well-evaluated video similarity architecture, with one benchmark comparison (EVVE) that doesn't support the headline claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the frame-to-frame similarity matrix between all pairs of frames of two videos. It is built by decomposing each frame into $N \times N$ region vectors from intermediate CNN layers, weighting regions by a learned attention vector, computing the Tensor Dot product between region vectors of every frame pair, and applying Chamfer Similarity (the average, over one frame's regions, of each region's maximum similarity to the other frame's regions). A four-layer CNN then processes the matrix and a final Chamfer Similarity yields the video-to-video score, trained with triplet loss plus a regularization that keeps pre-activation values inside the hard-tanh range. The matrix is the carrier of the spatio-temporal information: regional matching supplies spatial invariance, while the CNN learns temporal patterns such as diagonals and segment-level blocks.
What would settle it
If an independent re-annotation of the top-ranked supposedly false positives in FIVR-200K showed they are genuinely irrelevant, or if rerunning the comparison on the original uncorrected CC_WEB_VIDEO labels let an existing method such as the circulant temporal encoding beat ViSiL, the claimed superiority would be in doubt.
Extended reading notes
Core claim
ViSiL's central claim is that comparison before aggregation is what matters: regional CNN features from intermediate layers are whitened, attention-weighted, and matched with Tensor Dot plus Chamfer Similarity to form a frame-to-frame similarity matrix; a four-layer CNN then reads that matrix and Chamfer Similarity reduces it to a video-to-video score. Because no global pooling collapses the frame or video into a vector before matching, the network can exploit diagonal and block-like temporal structures that indicate shared segments, and suppress isolated spurious frame matches. The paper supports the claim with ablations showing each component adds mAP on FIVR-5K, and with comparisons on CC_WEB_VIDEO, FIVR-200K, EVVE, and ActivityNet.
Load-bearing premise
The load-bearing premise is that the evaluation benchmarks' ground-truth labels are trustworthy enough that reported gains reflect true retrieval quality; the paper itself notes that CC_WEB_VIDEO contains many positives mislabeled as negatives and shows unlabeled related videos in FIVR-200K.
Editorial extensions
If this is right
- On fine-grained incident retrieval (FIVR-200K), ViSiLv reports mAP of 0.892 on duplicate-scene retrieval, higher than the compared dynamic-programming and temporal-network reimplementations (0.775 and 0.724).
- On activity retrieval with 3D convolutional features, the symmetric variant ViSiLsym reaches 0.745 mAP, outperforming the deep metric learning baseline's 0.705.
- On the cleaned near-duplicate benchmark, ViSiLv approaches perfect ranking (0.993 mAP over the whole dataset), implying that the remaining errors are few once label noise is removed.
- Because the method never collapses frames or videos into a single descriptor before matching, it can report a video-to-video score while retaining which segments matched, which is directly useful for localization-style retrieval tasks.
Reading between the lines
- Inference: the two-stage design is a template: any region-level or spatio-temporal frame feature could be dropped into the TD+CS front end, so the method's gains may transfer to other backbones or modalities beyond the two feature types tested.
- Inference: the asymmetric Chamfer Similarity outperforming its symmetric variant suggests that retrieval direction matters—query-to-database matching may benefit from treating the query as the reference—an effect worth testing on ranking fairness and duplicate-direction analysis.
- Inference: the learned CNN on the similarity matrix could be applied to partial-copy detection and video re-localization directly by scanning submatrices of the frame-to-frame matrix, a direction the paper lists as future work.
- Inference: the dependence on cleaned labels means a shared, independently verified annotation protocol across near-duplicate benchmarks would be needed to confirm the margin over prior methods.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ViSiL, a video similarity learning architecture that computes frame-to-frame similarity from regional CNN features via Tensor Dot and Chamfer Similarity, and then learns a video-to-video similarity score by passing the resulting frame-similarity matrix through a small CNN. The network is trained with a triplet loss on the VCDB dataset, using whitened and attention-weighted features, and is evaluated on four video retrieval tasks: near-duplicate video retrieval (CC_WEB_VIDEO), fine-grained incident video retrieval (FIVR-200K), event video retrieval (EVVE), and action video retrieval (ActivityNet). The authors report consistent improvements over the compared baselines and release the implementation publicly.
Significance. If the reported results hold, the paper makes a useful contribution by demonstrating that fine-grained spatial and temporal relations, captured through regional frame descriptors and a learned interaction on the frame-similarity matrix, are beneficial for video retrieval beyond global video embeddings. The method is well motivated and the design is technically sound in its components. The public release of the implementation is a concrete strength. However, the strength of the empirical claim is weakened by two evaluation issues that affect the interpretation of the state-of-the-art comparisons.
major comments (2)
- [§6.3.3, Table 7] The EVVE comparison is not a valid head-to-head comparison with LAMV. The paper states that ViSiL results are reported on roughly 80% of the EVVE dataset (1,897 of 2,375 videos and 503 of 620 queries), while the LAMV baseline numbers (0.536 and 0.587) are from the full dataset. Mean Average Precision is not invariant to subsetting the database and queries: the set of ground-truth positives per query changes, the ranking changes, and per-event mAP is computed over different query sets. Therefore the reported difference between ViSiLv (0.631, subset) and LAMV+QE (0.587, full) does not establish that ViSiL outperforms LAMV on EVR. The per-event comparisons in the same table suffer from the same problem. To support the stated 'large improvements' claim on EVR, the authors should either evaluate LAMV on the exact same subset, report results on the full EVVE dataset, or explicitly retract the state-of-the-art claim for EVR and present the results only as indicative on the available subset.
- [§5.1, Table 5] The near-perfect results on CC_WEB_VIDEO are reported on a 'cleaned' version of the annotations that the authors created, in addition to the original annotations. Because the cleaning is a post hoc modification of a public benchmark's ground truth, the numbers on the cleaned version are not directly comparable with previously published results on the original dataset. The paper does report original-annotation results as well, which is good, but the abstract's claim of 'large improvements' on NDVR relies substantially on the cleaned-annotation results (e.g., 0.996 vs. 0.985 on CC_WEBVIDEO_c). The authors should make the cleaning procedure and the list of changed labels publicly available and should clearly separate the two evaluations in all claims, rather than presenting the cleaned results as the primary evidence of superiority.
minor comments (5)
- [Section 2] The text refers to 'Goa et al.'; the correct name is Gao et al. (the reference is correctly listed as [11] and [35] in the bibliography).
- [Section 4.2, Eq. (4)] The notation d^T_{ij} b_{kl} is clear, but it may help to explicitly state that d and b are unit-norm region vectors so that the dot product is a cosine similarity, which is assumed in the attention weighting of Section 4.1.
- [Figure 2] The caption says 'The double arrows indicate shared weights,' but the figure contains only single arrows in the provided text version; please ensure the figure and caption are consistent.
- [Section 6.3.3, Table 7] The per-event class columns in Table 7 are presented without event names, only an ordering note referring to the original EVVE paper. It would improve reproducibility to print the event names in the table or in a supplementary file.
- [Section 5.2] The paper reports no variance or multiple-seed statistics for the training procedure. Since model selection is performed on a validation set (with mAP), reporting the mean and standard deviation over several training runs would strengthen the reliability of the comparisons, especially where margins are thin (e.g., ViSiLv vs. ViSiLsym on ActivityNet).
Circularity Check
No derivation-level circularity: ViSiL is trained on VCDB and evaluated on external benchmarks, but the EVVE comparison is benchmark-mismatched and the CC_WEB annotations are manually cleaned; these are evaluation-validity issues, not circular reductions.
full rationale
ViSiL's derivation chain is self-contained: frame features are extracted by a fixed CNN with PCA whitening learned from VCDB, frame similarity is Chamfer Similarity on Tensor Dot products of regional features, video similarity is the output of a CNN trained with triplet and regularization losses, and the trained model is evaluated on CC_WEB_VIDEO, FIVR-200K, EVVE, and ActivityNet. No test-set labels enter training, and no fitted parameter is renamed as a prediction. The attention vector u and the video-similarity CNN weights are learned from Equations 7-9, not set to reproduce benchmark scores. The use of the authors' prior BoW method [20] for triplet mining and for defining FIVR-5K is a training/subset-construction choice, not a reduction of the reported mAP values. The main caveats are evaluation-validity issues rather than circularity: (i) the EVVE comparison in Table 7 is reported on about 80% of the dataset while the LAMV baselines are from the full dataset, so the claimed EVR improvement is not established by that table; (ii) CC_WEB_VIDEO results are reported on the authors' manually 'cleaned' annotations, so near-perfect scores on the cleaned labels are not fully independent of that relabeling. Neither issue makes the architecture's output equivalent by definition to its training input or to a self-citation. The central result has independent grounding in the FIVR-200K and ActivityNet evaluations, and the self-citations (DML, LBoW, FIVR dataset) are not load-bearing in the derivation.
Assumptions & free parameters
free parameters (5)
- triplet margin gamma =
0.5
- regularization weight r =
0.1
- training snippet size W =
64
- region granularity N =
3 (L3-iMAC)
- PCA whitening dimensionality =
256 (L3-iMAC)
assumptions (4)
- domain assumption Pretrained CNN features (ResNet-50, I3D) provide representations that preserve enough visual information for similarity comparison.
- domain assumption Chamfer similarity over regional features is an appropriate frame-to-frame similarity function, with max-pooling over regions giving spatial invariance.
- domain assumption The VCDB training set and artificially transformed videos provide a training distribution that generalizes to the evaluation tasks (incident, event, action retrieval).
- ad hoc to paper Hard tanh clipping plus a regularization loss keeps outputs bounded and improves training.
invented entities (1)
-
Visual context vector u
Cite this review
Pith. "Pith review of ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning." pith.science (2026). https://pith.science/paper/GH4QI74V
@misc{pith2026190807410,
author = {Pith},
title = {Pith review of: ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/GH4QI74V}},
note = {Machine review of arXiv:1908.07410}
}
read the original abstract
In this paper we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained Spatio-Temporal relations between pairs of videos -- such relations are typically lost in previous video retrieval approaches that embed the whole frame or even the whole video into a vector descriptor before the similarity estimation. By contrast, our Convolutional Neural Network (CNN)-based approach is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra- and inter-frame relations. In the proposed method, pairwise frame similarity is estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on regional CNN frame features - this avoids feature aggregation before the similarity calculation between frames. Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score -- this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences. We train the proposed network using a triplet loss scheme and evaluate it on five public benchmark datasets on four different video retrieval problems where we demonstrate large improvements in comparison to the state of the art. The implementation of ViSiL is publicly available.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Aggregating local deep features for image retrieval
Artem Babenko and Victor Lempitsky. Aggregating local deep features for image retrieval. InProceedings of the IEEE International Conference on Computer Vision, pages 1269– 1277, 2015
work page 2015
-
[2]
LAMV: Learning to align and match videos with ker- nelized temporal layers
Lorenzo Baraldi, Matthijs Douze, Rita Cucchiara, and Herv ´e J´egou. LAMV: Learning to align and match videos with ker- nelized temporal layers. In Proceedings of the IEEE con- ference on Computer Vision and Pattern Recognition, pages 7804–7813, 2018
work page 2018
-
[3]
Parametric correspondence and chamfer matching: Two new techniques for image matching
Harry G Barrow, Jay M Tenenbaum, Robert C Bolles, and Helen C Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching. Techni- cal report, SRI AI Center, 1977
work page 1977
-
[4]
Activitynet: A large-scale video benchmark for human activity understanding
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceed- ings of the IEEE conference on Computer Vision and Pattern Recognition, pages 961–970, 2015
work page 2015
-
[5]
Million-scale near-duplicate video retrieval system
Yang Cai, Linjun Yang, Wei Ping, Fei Wang, Tao Mei, Xian- Sheng Hua, and Shipeng Li. Million-scale near-duplicate video retrieval system. In Proceedings of the 19th ACM international conference on Multimedia , pages 837–838. ACM, 2011
work page 2011
-
[6]
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset
Joao Carreira and Andrew Zisserman. Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 4724–4733. IEEE, 2017
work page 2017
-
[7]
Pattern- based near-duplicate video retrieval and localization on web- scale videos
Chien-Li Chou, Hua-Tsung Chen, and Suh-Yin Lee. Pattern- based near-duplicate video retrieval and localization on web- scale videos. IEEE Transactions on Multimedia, 17(3):382– 395, 2015
work page 2015
-
[8]
An image-based approach to video copy detection with spatio- temporal post-filtering
Matthijs Douze, Herv ´e J ´egou, and Cordelia Schmid. An image-based approach to video copy detection with spatio- temporal post-filtering. IEEE Transactions on Multimedia , 12(4):257–266, 2010
work page 2010
Show all 37 references
-
[9]
Stable hyper-pooling and query expansion for event detection
Matthijs Douze, J ´erˆome Revaud, Cordelia Schmid, and Herv´e J´egou. Stable hyper-pooling and query expansion for event detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 1825–1832, 2013
2013
-
[10]
Video re-localization
Yang Feng, Lin Ma, Wei Liu, Tong Zhang, and Jiebo Luo. Video re-localization. In Proceedings of the European Con- ference on Computer Vision, pages 51–66, 2018
2018
-
[11]
ER3: A unified framework for event retrieval, recognition and recounting
Zhanning Gao, Gang Hua, Dongqing Zhang, Nebojsa Jojic, Le Wang, Jianru Xue, and Nanning Zheng. ER3: A unified framework for event retrieval, recognition and recounting. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2253–2262, 2017
2017
-
[12]
Unsupervised t-distributed video hashing and its deep hashing extension
Yanbin Hao, Tingting Mu, John Y Goulermas, Jianguo Jiang, Richang Hong, and Meng Wang. Unsupervised t-distributed video hashing and its deep hashing extension. IEEE Trans- actions on Image Processing, 26(11):5531–5544, 2017
2017
-
[13]
Deep residual learning for image recognition
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on Computer Vision and Pattern Recognition, pages 770–778, 2016
2016
-
[14]
Learning spatial-temporal fea- tures for video copy detection by the combination of cnn and rnn
Yaocong Hu and Xiaobo Lu. Learning spatial-temporal fea- tures for video copy detection by the combination of cnn and rnn. Journal of Visual Communication and Image Represen- tation, 55:21–29, 2018
2018
-
[15]
Negative evidences and co- occurences in image retrieval: The benefit of pca and whiten- ing
Herv ´e J´egou and Ondˇrej Chum. Negative evidences and co- occurences in image retrieval: The benefit of pca and whiten- ing. In Proceedings of the European Conference on Com- puter Vision, pages 774–787. Springer, 2012
2012
-
[16]
VCDB: a large-scale database for partial copy detection in videos
Yu-Gang Jiang, Yudong Jiang, and Jiajun Wang. VCDB: a large-scale database for partial copy detection in videos. In Proceedings of the European Conference on Computer Vi- sion, pages 357–371. Springer, 2014
2014
-
[17]
Partial copy detection in videos: A benchmark and an evaluation of popular methods
Yu-Gang Jiang and Jiajun Wang. Partial copy detection in videos: A benchmark and an evaluation of popular methods. IEEE Transactions on Big Data, 2(1):32–42, 2016
2016
-
[18]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014
2014 arXiv
-
[19]
FIVR: Fine-grained In- cident Video Retrieval
Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, and Ioannis Kompatsiaris. FIVR: Fine-grained In- cident Video Retrieval. arXiv preprint arXiv:1809.04094 , 2018
2018 arXiv
-
[20]
Near-duplicate video re- trieval by aggregating intermediate cnn layers
Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, and Yiannis Kompatsiaris. Near-duplicate video re- trieval by aggregating intermediate cnn layers. In Interna- tional conference on Multimedia Modeling, pages 251–263. Springer, 2017
2017
-
[21]
Near-duplicate video re- trieval with deep metric learning
Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, and Yiannis Kompatsiaris. Near-duplicate video re- trieval with deep metric learning. InProceedings of the IEEE International Conference on Computer Vision , pages 347– 356, 2017
2017
-
[22]
IR feature embedded bof indexing method for near-duplicate video re- trieval
Kaiyang Liao, Hao Lei, Yuanlin Zheng, Guangfeng Lin, Congjun Cao, Mingzhu Zhang, and Jie Ding. IR feature embedded bof indexing method for near-duplicate video re- trieval. IEEE Transactions on Circuits and Systems for Video Technology, 2018
2018
-
[23]
Deep video hashing
Venice Erin Liong, Jiwen Lu, Yap-Peng Tan, and Jie Zhou. Deep video hashing. IEEE Transactions on Multimedia , 19(6):1209–1219, 2017
2017
-
[24]
An image-based near-duplicate video retrieval and localization using improved edit distance
Hao Liu, Qingjie Zhao, Hao Wang, Peng Lv, and Yanming Chen. An image-based near-duplicate video retrieval and localization using improved edit distance. Multimedia Tools and Applications, 76(22):24435–24456, 2017
2017
-
[25]
Near-duplicate video re- trieval: Current research and future trends
Jiajun Liu, Zi Huang, Hongyun Cai, Heng Tao Shen, Chong Wah Ngo, and Wei Wang. Near-duplicate video re- trieval: Current research and future trends. ACM Computing Surveys (CSUR), 45(4):44, 2013
2013
-
[26]
Temporal matching kernel with explicit feature maps
S ´ebastien Poullot, Shunsuke Tsukatani, Anh Phuong Nguyen, Herv ´e J ´egou, and Shin’Ichi Satoh. Temporal matching kernel with explicit feature maps. In Proceedings of the 23rd ACM international conference on Multimedia, pages 381–390. ACM, 2015
2015
-
[27]
Fine- tuning CNN image retrieval with no human annotation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
Filip Radenovi ´c, Giorgos Tolias, and Ondrej Chum. Fine- tuning CNN image retrieval with no human annotation.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
2018
-
[28]
Event retrieval in large video collections with circulant temporal encoding
J ´erˆome Revaud, Matthijs Douze, Cordelia Schmid, and Herv´e J´egou. Event retrieval in large video collections with circulant temporal encoding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2459–2466, 2013
2013
-
[29]
Very deep convo- lutional networks for large-scale image recognition
Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014
2014 arXiv
-
[30]
Multiple feature hashing for real-time large scale near-duplicate video retrieval
Jingkuan Song, Yi Yang, Zi Huang, Heng Tao Shen, and Richang Hong. Multiple feature hashing for real-time large scale near-duplicate video retrieval. In Proceedings of the 19th ACM international conference on Multimedia , pages 423–432. ACM, 2011
2011
-
[31]
Self-supervised video hashing with hierarchical binary auto-encoder
Jingkuan Song, Hanwang Zhang, Xiangpeng Li, Lianli Gao, Meng Wang, and Richang Hong. Self-supervised video hashing with hierarchical binary auto-encoder. IEEE Trans- actions on Image Processing, 27(7):3210–3221, 2018
2018
-
[32]
Scalable detection of partial near-duplicate videos by visual-temporal consistency
Hung-Khoon Tan, Chong-Wah Ngo, Richard Hong, and Tat- Seng Chua. Scalable detection of partial near-duplicate videos by visual-temporal consistency. InProceedings of the 17th ACM international conference on Multimedia , pages 145–154. ACM, 2009
2009
-
[33]
Particular object retrieval with integral max-pooling of cnn activations
Giorgos Tolias, Ronan Sicre, and Herv ´e J ´egou. Particular object retrieval with integral max-pooling of cnn activations. arXiv preprint arXiv:1511.05879, 2015
2015 arXiv
-
[34]
Learning spatiotemporal features with 3d convolutional networks
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. InProceedings of the IEEE Inter- national Conference on Computer Vision, pages 4489–4497, 2015
2015
-
[35]
Practical elimination of near-duplicates from web video search
Xiao Wu, Alexander G Hauptmann, and Chong-Wah Ngo. Practical elimination of near-duplicates from web video search. In Proceedings of the 15th ACM international con- ference on Multimedia, pages 218–227. ACM, 2007
2007
-
[36]
Deep multi-task representation learning: A tensor factorisation approach
Yongxin Yang and Timothy Hospedales. Deep multi-task representation learning: A tensor factorisation approach. In International Conference on Learning Representations , 2017
2017
-
[37]
Hierarchical attention networks for document classification
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech...
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.