A Comparison of CNN and Classic Features for Image Retrieval

Michael Lew; Theodoros Georgiou; Umut \"Ozayd{\i}n

REVIEW 3 major objections 5 minor 23 references

A Comparison of CNN and Classic Features for Image Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CNN-based local feature detectors do not uniformly beat handcrafted ones: in this paper's image retrieval benchmark, SURF leads with 60.16% top-1 accuracy and the best average rank, while LIFT wins under blur and noise and AGAST under…

desk verdict A legitimate but statistically thin benchmark: the claim that handcrafted SURF still wins retrieval and that the best detector depends on context is plausible, but the reported single-run numbers cannot carry the specific SURF-over-Quad ranking. read the letter →

arxiv 1908.09300 v1 pith:B3L2XZHG submitted 2019-08-25 cs.CV

Umut Özayd{i}n , Theodoros Georgiou , Michael Lew This is my paper · ORCID

classification cs.CV

keywords imageretrievallocalfeatureskeypointdetectionfeaturedescriptorsconvolutionalneuralnetworksbag-of-wordsrepeatabilitybenchmarkcomparison

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether replacing handcrafted keypoint detectors with convolutional neural networks actually improves local feature quality. It benchmarks three CNN-based detectors—Quad, LIFT, and an unsuccessfully implemented covariant detector—against SIFT, SURF, CenSurE, and AGAST, using repeatability, matching, coverage, and a bag-of-words retrieval task. The central result is that no family dominates: SURF has the highest retrieval accuracy (60.16% top-1) and lowest average rank, LIFT is best under blur and noise, and AGAST is best under rotation and scaling. The practical upshot is that handcrafted features remain a strong baseline and the best detector depends on the context.

What carries the argument

The evaluation machinery is a bag-of-words image retrieval pipeline built on a 250K-image database with an 8K visual-word dictionary, together with repeatability, matching precision/recall, non-redundancy, and coverage metrics measured under rotation, scaling, blur, Gaussian noise, and illumination changes. This combination lets the authors assess detector and descriptor quality both in isolation and through an end-to-end application, carrying the argument that performance is context-dependent rather than uniformly favoring one family.

What would settle it

Re-running the same 250K-image retrieval benchmark with official implementations of every detector and a fixed descriptor across all detectors would settle the claim; if a CNN detector then matches or exceeds SURF's 60.16% top-1 accuracy, the paper's conclusion that handcrafted features remain the retrieval baseline would no longer hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that CNN-based local feature detectors do not clearly outperform classical ones; instead, each type of feature is best in different contexts. On a 250,000-image bag-of-words retrieval benchmark with 2,500 transformed queries, SURF achieves the highest top-1 accuracy and lowest average rank, the learned Quad detector comes close, and the SIFT-supervised LIFT pipeline performs worst in retrieval. In repeatability tests, LIFT excels under blur and Gaussian noise, AGAST excels under rotation and scaling, and SURF is consistently strong. The authors conclude that the default assumption that deep learned features will greatly outperform handcrafted features is not supported for local feature detection, and that more research is needed before abandoning handcrafted features.

Load-bearing premise

The comparison assumes that the re-implemented Quad detector and the parameter settings for every method faithfully represent the published methods, so the context-dependent rankings are not artifacts of unfair configuration.

Editorial extensions

If this is right

Designers of image retrieval systems can expect handcrafted detectors such as SURF to remain competitive baselines that learned methods must explicitly beat.
Choosing a detector by context—SURF for general retrieval, LIFT for noisy or blurred imagery, AGAST for rotated or scaled scenes—should outperform any single default choice.
Replacing handcrafted descriptors with pre-trained VGG activations lowered retrieval accuracy, so CNN descriptors are not automatically better in this pipeline.
Training a detector on SIFT keypoints may inherit the limitations of SIFT, which helps explain why LIFT's retrieval performance was weak.
CNN-based detectors underperformed under rotation, so rotation invariance remains a specific weakness that future learned methods would need to address.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is a hybrid pipeline that selects the detector based on the predicted distortion type, which could beat any single method on the same benchmark.
The authors only tested one way of using pre-trained VGG features, so other CNN descriptor architectures or layers might close the gap without changing the paper's core comparison.
Because the Quad detector was re-implemented and trained only on illumination changes, its strong retrieval performance may be tied to that training distribution and could shift under other transformations.
One could probe when handcrafted features lose their edge by increasing the severity of non-linear photometric distortions, which are common failure modes not fully explored here.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a legitimate benchmark paper, not a technical breakthrough. It compares seven detector/descriptor combinations on repeatability, matching, non-redundancy, and bag-of-words image retrieval, and the headline result is that handcrafted SURF still gets the best retrieval accuracy while the best detector depends on the distortion. That conclusion is plausible, but the specific quantitative ranking used to support it is statistically weak as reported.

What is new: the particular combination of methods in one pipeline. I do not know of another paper that puts SIFT, SURF, CenSurE, AGAST, Quad, LIFT, and VGG descriptors in the same repeatability and retrieval evaluation. That makes it useful for practitioners who want a sanity check on learned versus handcrafted features. The authors also deserve credit for being explicit about limitations: Quad is a re-implementation by the authors trained only on illumination changes; the covariant detector was omitted after it failed to produce stable keypoints; and the paper says more experiments are needed before drawing conclusive remarks.

Soft spots are real and concentrated in the retrieval table. All numbers come from one run. The gap between SURF (60.16%) and Quad (59.52%) is 0.64 percentage points on 2500 queries, which is inside one binomial standard error; that gap cannot establish that SURF is better. The Quad re-implementation is the more important issue: published Quad-Networks learned from ranking constraints over quadruples with specified transformations, and this version used only illumination changes, which changes the detector's invariance properties. So the central line in the abstract and conclusion, that CNN detectors do not dominate and each method is best in different contexts, is supported only if a faithful Quad baseline behaves the same way. Also, LIFT's orientation estimator differs between the repeatability and retrieval experiments, so comparing those two sets of LIFT results is not apples-to-apples. AGAST's repeatability advantage under rotation and scaling is a qualitative finding that likely holds, but it too lacks error bars.

The citation pattern looks clean. No code or models are promised, so the numbers cannot be independently checked without contacting the authors.

Who this is for: someone making practical baseline choices in local feature work, or a reading group wanting a quick comparative snapshot. It deserves a serious referee at a workshop or short-paper track; for a rigorous journal, I would want multiple runs with error bars, a verified or clearly labeled Quad baseline, a unified LIFT configuration, and ideally released code and models. With those revisions, the main qualitative claim would likely survive.

Recommendation: send it to peer review, but with revision rather than acceptance as-is.

Referee Report

3 major / 5 minor

Summary. The paper compares four hand-crafted local feature pipelines (SIFT, SURF, CenSurE, AGAST) with two learned detectors (Quad, LIFT) and with VGG16 descriptor variants, using repeatability, matching, non-redundancy/coverage, and a bag-of-words image-retrieval benchmark on 250K database images and 2.5K transformed queries. The main empirical findings are that SURF achieves the highest retrieval accuracy (60.16%) and lowest average rank (1180), LIFT has the best repeatability under blur and noise, AGAST under rotation and scaling, and that CNN-based detectors do not dominate. The paper concludes that the best feature type depends on the task and that hand-crafted features remain competitive.

Significance. The paper addresses a timely and practical question, and its qualitative message—that learned detectors are not uniformly better than hand-crafted ones—is plausible and consistent with the reported plots. Its strengths include the use of an established retrieval benchmark (BoW/ILSVRC), a clear enumeration of detector/descriptor/orientation combinations in Table II, and the explicit admission of the covariant detector's failure in pilot tests. If the described configurations faithfully represent the published methods, the comparison is a useful reference for practitioners. However, the quantitative ordering is not supported with uncertainty estimates or repeated trials, and the Quad entry is an author re-implementation trained under a simplified correspondence setting, so the precise rankings and the implied practical recommendations should be treated as provisional.

major comments (3)

[Section V, Table III] The headline result that SURF is the best IR method rests on a single run. With n=2,500 binary top-1 queries and p≈0.60, the standard error is about 0.98 percentage points, so the 0.64-point gap between SURF (60.16%) and Quad (59.52%) is within one standard error; the gap between SURF and CenSurE (48.48%) is larger but still needs standard errors or confidence intervals because the queries are generated with random transformations. Figures 1–3 likewise show single-run averages with no error bars. Please report multiple trials with different random-transformation seeds, or at least compute binomial confidence intervals, and qualify the 'SURF best' claim accordingly.
[Section V] The Quad detector is re-implemented by the authors and trained on DTU with only illumination changes as correspondences. The original Quad-Networks learns from quadruples of patches under specified transformations, including scale and rotation; limiting training correspondences to illumination changes plausibly changes the detector's invariance and therefore its retrieval accuracy. Since Quad is the strongest CNN in Table III, the comparison does not currently establish how the published Quad-Networks would rank. Please either use the original implementation or a pretrained model, document the reimplementation's fidelity by reproducing Quad-Networks' own results on a standard benchmark, or clearly present the entry as a variant and weaken the CNN comparison claim.
[Sections IV–V] The LIFT results mix two configurations: the repeatability experiment uses SIFT orientation assignment because the rotation estimator was not trained on large rotations, while the IR experiment uses the version with the rotation estimator trained on large rotations. The context-dependent conclusion (LIFT best under blur/noise, weaker under rotation) therefore confounds the detector with the orientation estimator. Report the two configurations separately in each experiment, or justify why comparing across these configurations is valid for the conclusion.

minor comments (5)

[Sections I and V] There are typographical errors: 'perfrormance' should be 'performance' and 'They key difference' should be 'The key difference'.
[Table II] The 'SIFT/LIFT' entry for LIFT orientation is ambiguous; the table should specify which experiment uses which orientation estimator, matching the two-version description in Section V.
[Figure 1 caption] The phrase 'manual transformations' appears to mean 'applied transformations'; please clarify the wording.
[Section IV] The ranges and random generation process for scale, rotation, blur, noise, contrast, and brightness in the 2.5K query images are not specified. State the parameter ranges and, ideally, the random seed or number of repeated trials for reproducibility.
[Section V] The statement that parameters were set as close as possible to the original papers is not operational; include a table of the actual parameter values used for each detector or cite the exact OpenCV defaults applied.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical comparison with no circular derivation; conclusions follow directly from the experiments.

full rationale

This paper is an experimental comparison of handcrafted and CNN-based local feature detectors. It does not derive any result from an assumed model, fit parameters to a target quantity and then predict that same quantity, or import a load-bearing conclusion from the authors' prior work. The abstract and Section VI state that different feature types perform best in different contexts, and this claim is supported by directly measured repeatability, matching, and image retrieval numbers (e.g., Table III). The Quad detector was re-implemented by the authors, but this is an implementation choice affecting experimental validity, not a circular step: the retrieval accuracy of Quad is measured, not assumed. Likewise, the omission of the covariant feature detector and the statement that more experiments would be required are explicit limitations, not circular reasoning. There is no self-citation chain used to justify the paper's conclusions, and no fitted value is renamed as a prediction. The skeptical concerns about the small accuracy margin, lack of error bars, and faithful reimplementation are threats to the strength of the conclusion, but they do not make the argument circular. Therefore the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on several hand-chosen benchmark parameters and on the assumption that the implementations and re-implementations used in the experiments faithfully represent the published methods. No new entities are introduced.

free parameters (3)

Number of keypoints per image for BoW dictionary = 10
Hand-chosen experimental setting; dictionary built from 10 most prominent keypoints per image affects retrieval accuracy.
Number of clusters for BoW = 8000
Hand-chosen; affects histogram granularity and retrieval results.
Number of keypoints per image for histogram = 200
Hand-chosen; controls the amount of information per image in the retrieval experiment.

assumptions (3)

domain assumption OpenCV 3 implementations and default parameters faithfully represent the original detectors/descriptors
Section V states OpenCV 3 implementations were used with parameters close to original papers; if implementations differ, results may not reflect the methods as published.
domain assumption ImageNet test and train sets provide a representative distribution for evaluating local feature detectors and retrieval
Experiments in Sections IV and V use ImageNet; the conclusions are assumed to generalize beyond this dataset.
ad hoc to paper The Quad detector re-implementation and training on DTU with illumination changes is a valid instance of the Quad method
Section V describes the re-implementation; no code or weights released, so the fidelity to the original method is unverified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Comparison of CNN and Classic Features for Image Retrieval." pith.science (2026). https://pith.science/paper/B3L2XZHG

@misc{pith2026190809300,
  author       = {Pith},
  title        = {Pith review of: A Comparison of CNN and Classic Features for Image Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B3L2XZHG}},
  note         = {Machine review of arXiv:1908.09300}
}

read the original abstract

Feature detectors and descriptors have been successfully used for various computer vision tasks, such as video object tracking and content-based image retrieval. Many methods use image gradients in different stages of the detection-description pipeline to describe local image structures. Recently, some, or all, of these stages have been replaced by convolutional neural networks (CNNs), in order to increase their performance. A detector is defined as a selection problem, which makes it more challenging to implement as a CNN. They are therefore generally defined as regressors, converting input images to score maps and keypoints can be selected with non-maximum suppression. This paper discusses and compares several recent methods that use CNNs for keypoint detection. Experiments are performed both on the CNN based approaches, as well as a selection of conventional methods. In addition to qualitative measures defined on keypoints and descriptors, the bag-of-words (BoW) model is used to implement an image retrieval application, in order to determine how the methods perform in practice. The results show that each type of features are best in different contexts.

Figures

Figures reproduced from arXiv: 1908.09300 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 17 canonical work pages

[1]

Recent advances in features extraction and description algorithms: A comprehensive survey,

E. Salahat and M. Qasaimeh, “Recent advances in features extraction and description algorithms: A comprehensive survey,” in 2017 IEEE International Conference on Industrial Technology (ICIT) . IEEE, 2017, pp. 1059–1063

work page 2017
[2]

A survey of recent advances in visual feature detection,

Y . Li, S. Wang, Q. Tian, and X. Ding, “A survey of recent advances in visual feature detection,” Neurocomputing, vol. 149, pp. 736–751, 2015

work page 2015
[3]

Distinctive image features from scale-invariant keypoints,

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision , vol. 60, no. 2, pp. 91–110, 2004

2004
[4]

Speeded-up robust features (surf),

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (surf),” Computer vision and image understanding , vol. 110, no. 3, pp. 346–359, 2008

2008
[6]

Brisk: Binary robust invariant scalable keypoints,

S. Leutenegger, M. Chli, and R. Y . Siegwart, “Brisk: Binary robust invariant scalable keypoints,” in Proceedings of the 2011 International Conference on Computer Vision , ser. ICCV ’11. Washington, DC, USA: IEEE Computer Society, 2011, pp. 2548–2555. [Online]. Available: http://dx.doi.org/10.1109/ICCV .2011.6126542

arXiv 2011
[7]

Freak: Fast retina keypoint,

A. Alahi, R. Ortiz, and P. Vandergheynst, “Freak: Fast retina keypoint,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition . Ieee, 2012, pp. 510–517

work page 2012
[8]

Learning covariant feature detectors,

K. Lenc and A. Vedaldi, “Learning covariant feature detectors,” in European Conference on Computer Vision . Springer, 2016, pp. 100– 117

work page 2016
[9]

Tilde: A temporally invariant learned detector,

Y . Verdie, K. M. Yi, P. Fua, and V . Lepetit, “Tilde: A temporally invariant learned detector,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015, pp. 5279–5288

work page 2015

Show all 23 references

[10]

Quad- networks: unsupervised learning to rank for interest point detection,

N. Savinov, A. Seki, L. Ladicky, T. Sattler, and M. Pollefeys, “Quad- networks: unsupervised learning to rank for interest point detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1822–1830

2017
[11]

LIFT: Learned Invariant Feature Transform,

K. M. Yi, E. Trulls, V . Lepetit, and P. Fua, “LIFT: Learned Invariant Feature Transform,” in Proceedings of the European Conference on Computer Vision, 2016

2016
[12]

Learning to assign orientations to feature points,

K. Moo Yi, Y . Verdie, P. Fua, and V . Lepetit, “Learning to assign orientations to feature points,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2016, pp. 107–116

2016
[13]

Extracting semantic information from visual data: A survey,

Q. Liu, R. Li, H. Hu, and D. Gu, “Extracting semantic information from visual data: A survey,” Robotics, vol. 5, no. 1, p. 8, 2016

2016
[14]

Agrawal, K

M. Agrawal, K. Konolige, and M. R. Blas, CenSurE: Center Surround Extremas for Realtime Feature Detection and Matching . Berlin, Heidelberg: Springer Berlin Heidelberg, 2008, pp. 102–115

2008
[15]

Adaptive and generic corner detection based on the accelerated segment test,

E. Mair, G. D. Hager, D. Burschka, M. Suppa, and G. Hirzinger, “Adaptive and generic corner detection based on the accelerated segment test,” in Proceedings of the 11th European Conference on Computer Vision: Part II , ser. ECCV’10. Berlin, Heidelberg: Springer-Verlag, 2010, p...
[16]

Machine learning for high-speed corner detection,

E. Rosten and T. Drummond, “Machine learning for high-speed corner detection,” in Proceedings of the 9th European Conference on Computer Vision - V olume Part I , ser. ECCV’06. Berlin, Heidelberg: Springer- Verlag, 2006, pp. 430–443

2006
[18]

Mea- suring the coverage of interest point detectors,

S. Ehsan, N. Kanwal, A. F. Clark, and K. D. McDonald-Maier, “Mea- suring the coverage of interest point detectors,” in Image Analysis and Recognition, M. Kamel and A. Campilho, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 253–261

2011
[19]

Is repeatability an unbiased criterion for ranking feature detectors?

I. Rey-Otero and M. Delbracio, “Is repeatability an unbiased criterion for ranking feature detectors?” SIAM Journal on Imaging Sciences, vol. 8, no. 4, pp. 2558–2580, 2015. [Online]. Available: https://doi.org/10.1137/15M1007732

2015 doi
[20]

Very deep convolutional networks for large-scale image recognition,

K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556 , 2014

2014 arXiv
[21]

Sift meets cnn: A decade survey of instance retrieval,

L. Zheng, Y . Yang, and Q. Tian, “Sift meets cnn: A decade survey of instance retrieval,” IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 5, pp. 1224–1244, 2018

2018
[22]

Video google: a text retrieval approach to object matching in videos,

Sivic and Zisserman, “Video google: a text retrieval approach to object matching in videos,” in Proceedings Ninth IEEE International Confer- ence on Computer Vision , Oct 2003, pp. 1470–1477 vol.2

2003
[23]

Interesting interest points,

H. Aanæs, A. Dahl, and K. Steenstrup Pedersen, “Interesting interest points,” International Journal of Computer Vision , pp. 1–18, 2011

2011
[24]

TensorFlow: Large-scale machine learning on heterogeneous systems,

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, and et al., “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015. [Online]. Available: http://tensorﬂow.org/

2015
[2015]

Available: http://arxiv.org/abs/1511.05879

[Online]. Available: http://arxiv.org/abs/1511.05879

arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Recent advances in features extraction and description algorithms: A comprehensive survey,

E. Salahat and M. Qasaimeh, “Recent advances in features extraction and description algorithms: A comprehensive survey,” in 2017 IEEE International Conference on Industrial Technology (ICIT) . IEEE, 2017, pp. 1059–1063

work page 2017

[2] [2]

A survey of recent advances in visual feature detection,

Y . Li, S. Wang, Q. Tian, and X. Ding, “A survey of recent advances in visual feature detection,” Neurocomputing, vol. 149, pp. 736–751, 2015

work page 2015

[3] [3]

Distinctive image features from scale-invariant keypoints,

D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision , vol. 60, no. 2, pp. 91–110, 2004

2004

[4] [4]

Speeded-up robust features (surf),

H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (surf),” Computer vision and image understanding , vol. 110, no. 3, pp. 346–359, 2008

2008

[5] [6]

Brisk: Binary robust invariant scalable keypoints,

S. Leutenegger, M. Chli, and R. Y . Siegwart, “Brisk: Binary robust invariant scalable keypoints,” in Proceedings of the 2011 International Conference on Computer Vision , ser. ICCV ’11. Washington, DC, USA: IEEE Computer Society, 2011, pp. 2548–2555. [Online]. Available: http://dx.doi.org/10.1109/ICCV .2011.6126542

arXiv 2011

[6] [7]

Freak: Fast retina keypoint,

A. Alahi, R. Ortiz, and P. Vandergheynst, “Freak: Fast retina keypoint,” in 2012 IEEE Conference on Computer Vision and Pattern Recognition . Ieee, 2012, pp. 510–517

work page 2012

[7] [8]

Learning covariant feature detectors,

K. Lenc and A. Vedaldi, “Learning covariant feature detectors,” in European Conference on Computer Vision . Springer, 2016, pp. 100– 117

work page 2016

[8] [9]

Tilde: A temporally invariant learned detector,

Y . Verdie, K. M. Yi, P. Fua, and V . Lepetit, “Tilde: A temporally invariant learned detector,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2015, pp. 5279–5288

work page 2015