{"id":"24bf5dfb-ed58-4004-b5c9-182f5f3d4a13","arxiv_id":"1908.09300","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In a bag-of-words image retrieval benchmark on ImageNet, handcrafted SURF outperforms CNN-based local feature detectors overall, while CNN methods like LIFT excel under blur and noise.","lead":"This paper compares classic handcrafted image feature detectors against CNN-based ones on keypoint quality and image retrieval. The experiments show that handcrafted SURF still wins overall, and that no single feature type dominates all contexts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IR ranking that places SURF over Quad rests on an unverified Quad reimplementation and a 0.64-point margin with no error bars; the context-dependent conclusion could flip under a faithful baseline.","rationale":"The paper's central claim is modest: different feature types win in different contexts. The strongest quantitative support is Table III, where SURF (60.16%) beats Quad (59.52%) by 0.64 points and has a better average rank. This is a small margin on a single benchmark run, and the Quad detector is an author reimplementation trained only on illumination changes, not the published Quad-Networks model. The paper itself flags this and also reports that the covariant detector was omitted and that LIFT used different orientation estimators in different experiments. No error bars or significance tests are reported, and no code or models are released. Thus the finding that handcrafted features are competitive, and that the top CNN method does not overtake SURF, is not yet robust. The repeatability-based parts of the 'different contexts' claim (AGAST under rotation/scaling, LIFT under blur/noise) are qualitative plot readings without uncertainty quantification. None of this is evidence of unfairness or error; it simply means the headline comparison should be treated as provisional. The reader's conditional verdict remains appropriate.","tokens_in":6171,"tokens_out":4146,"duration_ms":40397,"concrete_test":"Obtain the original Quad-Networks code or pretrained weights, or retrain with the original transformation-augmented quadruple ranking loss on the original training data, and rerun the Section V IR benchmark with at least 10 independently sampled query sets (2.5K each) for both SURF and Quad. Report mean and 95% confidence intervals for top-1 accuracy and average rank. If the confidence interval for Quad overlaps or exceeds SURF, the paper's conclusion that classic features are best is unsupported; if SURF remains clearly ahead, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that handcrafted features remain competitive and that the best detector depends on context rests heavily on Table III: SURF 60.16% vs Quad 59.52% top-1 accuracy, and average ranks 1180 vs 1544. This margin is small, and the Quad entry is not the published Quad-Network: the authors re-implemented it in PyTorch and trained it on the DTU dataset using only illumination changes as correspondences (Section V). The original Quad-Networks learning uses ranking constraints over quadruples with specified transformations; a reimplementation that only uses illumination changes may change the detector's scale and rotation behavior and therefore its retrieval accuracy. Additionally, all numbers come from a single run: the 2.5K query images are constructed with random transformations, so the 0.64-point gap is within roughly one binomial standard error (about 0.98 percentage points for n=2500 at p=0.6). The conclusion that SURF beats Quad, which is then used to argue that CNNs do not dominate, is therefore not established. The separate repeatability claims for LIFT and AGAST also have no error bars, and LIFT's orientation estimator differs between the repeatability and IR experiments, so the 'best in different contexts' claim is partly comparing different configurations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four hand-crafted local feature pipelines (SIFT, SURF, CenSurE, AGAST) with two learned detectors (Quad, LIFT) and with VGG16 descriptor variants, using repeatability, matching, non-redundancy/coverage, and a bag-of-words image-retrieval benchmark on 250K database images and 2.5K transformed queries. The main empirical findings are that SURF achieves the highest retrieval accuracy (60.16%) and lowest average rank (1180), LIFT has the best repeatability under blur and noise, AGAST under rotation and scaling, and that CNN-based detectors do not dominate. The paper concludes that the best feature type depends on the task and that hand-crafted features remain competitive.","tokens_in":6421,"tokens_out":5179,"duration_ms":53501,"significance":"The paper addresses a timely and practical question, and its qualitative message—that learned detectors are not uniformly better than hand-crafted ones—is plausible and consistent with the reported plots. Its strengths include the use of an established retrieval benchmark (BoW/ILSVRC), a clear enumeration of detector/descriptor/orientation combinations in Table II, and the explicit admission of the covariant detector's failure in pilot tests. If the described configurations faithfully represent the published methods, the comparison is a useful reference for practitioners. However, the quantitative ordering is not supported with uncertainty estimates or repeated trials, and the Quad entry is an author re-implementation trained under a simplified correspondence setting, so the precise rankings and the implied practical recommendations should be treated as provisional.","major_comments":[{"comment":"The headline result that SURF is the best IR method rests on a single run. With n=2,500 binary top-1 queries and p≈0.60, the standard error is about 0.98 percentage points, so the 0.64-point gap between SURF (60.16%) and Quad (59.52%) is within one standard error; the gap between SURF and CenSurE (48.48%) is larger but still needs standard errors or confidence intervals because the queries are generated with random transformations. Figures 1–3 likewise show single-run averages with no error bars. Please report multiple trials with different random-transformation seeds, or at least compute binomial confidence intervals, and qualify the 'SURF best' claim accordingly.","section":"Section V, Table III"},{"comment":"The Quad detector is re-implemented by the authors and trained on DTU with only illumination changes as correspondences. The original Quad-Networks learns from quadruples of patches under specified transformations, including scale and rotation; limiting training correspondences to illumination changes plausibly changes the detector's invariance and therefore its retrieval accuracy. Since Quad is the strongest CNN in Table III, the comparison does not currently establish how the published Quad-Networks would rank. Please either use the original implementation or a pretrained model, document the reimplementation's fidelity by reproducing Quad-Networks' own results on a standard benchmark, or clearly present the entry as a variant and weaken the CNN comparison claim.","section":"Section V"},{"comment":"The LIFT results mix two configurations: the repeatability experiment uses SIFT orientation assignment because the rotation estimator was not trained on large rotations, while the IR experiment uses the version with the rotation estimator trained on large rotations. The context-dependent conclusion (LIFT best under blur/noise, weaker under rotation) therefore confounds the detector with the orientation estimator. Report the two configurations separately in each experiment, or justify why comparing across these configurations is valid for the conclusion.","section":"Sections IV–V"}],"minor_comments":[{"comment":"There are typographical errors: 'perfrormance' should be 'performance' and 'They key difference' should be 'The key difference'.","section":"Sections I and V"},{"comment":"The 'SIFT/LIFT' entry for LIFT orientation is ambiguous; the table should specify which experiment uses which orientation estimator, matching the two-version description in Section V.","section":"Table II"},{"comment":"The phrase 'manual transformations' appears to mean 'applied transformations'; please clarify the wording.","section":"Figure 1 caption"},{"comment":"The ranges and random generation process for scale, rotation, blur, noise, contrast, and brightness in the 2.5K query images are not specified. State the parameter ranges and, ideally, the random seed or number of repeated trials for reproducibility.","section":"Section IV"},{"comment":"The statement that parameters were set as close as possible to the original papers is not operational; include a table of the actual parameter values used for each detector or cite the exact OpenCV defaults applied.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the Quad reimplementation: because the published model is not used, the comparison's central ranking could be an artifact of the authors' training setup. The manuscript is otherwise within scope, and the empirical design can be strengthened with error bars and configurational transparency. I would not reject it outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate benchmark paper, not a technical breakthrough. It compares seven detector/descriptor combinations on repeatability, matching, non-redundancy, and bag-of-words image retrieval, and the headline result is that handcrafted SURF still gets the best retrieval accuracy while the best detector depends on the distortion. That conclusion is plausible, but the specific quantitative ranking used to support it is statistically weak as reported.\n\nWhat is new: the particular combination of methods in one pipeline. I do not know of another paper that puts SIFT, SURF, CenSurE, AGAST, Quad, LIFT, and VGG descriptors in the same repeatability and retrieval evaluation. That makes it useful for practitioners who want a sanity check on learned versus handcrafted features. The authors also deserve credit for being explicit about limitations: Quad is a re-implementation by the authors trained only on illumination changes; the covariant detector was omitted after it failed to produce stable keypoints; and the paper says more experiments are needed before drawing conclusive remarks.\n\nSoft spots are real and concentrated in the retrieval table. All numbers come from one run. The gap between SURF (60.16%) and Quad (59.52%) is 0.64 percentage points on 2500 queries, which is inside one binomial standard error; that gap cannot establish that SURF is better. The Quad re-implementation is the more important issue: published Quad-Networks learned from ranking constraints over quadruples with specified transformations, and this version used only illumination changes, which changes the detector's invariance properties. So the central line in the abstract and conclusion, that CNN detectors do not dominate and each method is best in different contexts, is supported only if a faithful Quad baseline behaves the same way. Also, LIFT's orientation estimator differs between the repeatability and retrieval experiments, so comparing those two sets of LIFT results is not apples-to-apples. AGAST's repeatability advantage under rotation and scaling is a qualitative finding that likely holds, but it too lacks error bars.\n\nThe citation pattern looks clean. No code or models are promised, so the numbers cannot be independently checked without contacting the authors.\n\nWho this is for: someone making practical baseline choices in local feature work, or a reading group wanting a quick comparative snapshot. It deserves a serious referee at a workshop or short-paper track; for a rigorous journal, I would want multiple runs with error bars, a verified or clearly labeled Quad baseline, a unified LIFT configuration, and ideally released code and models. With those revisions, the main qualitative claim would likely survive.\n\nRecommendation: send it to peer review, but with revision rather than acceptance as-is.","headline":"A legitimate but statistically thin benchmark: the claim that handcrafted SURF still wins retrieval and that the best detector depends on context is plausible, but the reported single-run numbers cannot carry the specific SURF-over-Quad ranking.","tokens_in":6925,"tokens_out":3873,"would_cite":false,"duration_ms":37821,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CNN-based local feature detectors do not uniformly beat handcrafted ones: in this paper's image retrieval benchmark, SURF leads with 60.16% top-1 accuracy and the best average rank, while LIFT wins under blur and noise and AGAST under…","keywords":["image retrieval","local features","keypoint detection","feature descriptors","convolutional neural networks","bag-of-words","repeatability","benchmark comparison"],"falsifier":"Re-running the same 250K-image retrieval benchmark with official implementations of every detector and a fixed descriptor across all detectors would settle the claim; if a CNN detector then matches or exceeds SURF's 60.16% top-1 accuracy, the paper's conclusion that handcrafted features remain the retrieval baseline would no longer hold.","tokens_in":1488,"feed_emoji":"🖼️","tokens_out":1613,"duration_ms":59303,"temperature":0.7,"pith_summary":"This paper asks whether replacing handcrafted keypoint detectors with convolutional neural networks actually improves local feature quality. It benchmarks three CNN-based detectors—Quad, LIFT, and an unsuccessfully implemented covariant detector—against SIFT, SURF, CenSurE, and AGAST, using repeatability, matching, coverage, and a bag-of-words retrieval task. The central result is that no family dominates: SURF has the highest retrieval accuracy (60.16% top-1) and lowest average rank, LIFT is best under blur and noise, and AGAST is best under rotation and scaling. The practical upshot is that handcrafted features remain a strong baseline and the best detector depends on the context.","feed_headline":"SURF tops CNN detectors at 60.16% accuracy in retrieval","feed_subtitle":"A 250K-image benchmark shows handcrafted features still lead overall, while learned detectors win under blur, noise, and rotation.","key_machinery":"The evaluation machinery is a bag-of-words image retrieval pipeline built on a 250K-image database with an 8K visual-word dictionary, together with repeatability, matching precision/recall, non-redundancy, and coverage metrics measured under rotation, scaling, blur, Gaussian noise, and illumination changes. This combination lets the authors assess detector and descriptor quality both in isolation and through an end-to-end application, carrying the argument that performance is context-dependent rather than uniformly favoring one family.","core_discovery":"The paper's central claim is that CNN-based local feature detectors do not clearly outperform classical ones; instead, each type of feature is best in different contexts. On a 250,000-image bag-of-words retrieval benchmark with 2,500 transformed queries, SURF achieves the highest top-1 accuracy and lowest average rank, the learned Quad detector comes close, and the SIFT-supervised LIFT pipeline performs worst in retrieval. In repeatability tests, LIFT excels under blur and Gaussian noise, AGAST excels under rotation and scaling, and SURF is consistently strong. The authors conclude that the default assumption that deep learned features will greatly outperform handcrafted features is not supported for local feature detection, and that more research is needed before abandoning handcrafted features.","pith_inferences":["A testable extension is a hybrid pipeline that selects the detector based on the predicted distortion type, which could beat any single method on the same benchmark.","The authors only tested one way of using pre-trained VGG features, so other CNN descriptor architectures or layers might close the gap without changing the paper's core comparison.","Because the Quad detector was re-implemented and trained only on illumination changes, its strong retrieval performance may be tied to that training distribution and could shift under other transformations.","One could probe when handcrafted features lose their edge by increasing the severity of non-linear photometric distortions, which are common failure modes not fully explored here."],"forward_implications":["Designers of image retrieval systems can expect handcrafted detectors such as SURF to remain competitive baselines that learned methods must explicitly beat.","Choosing a detector by context—SURF for general retrieval, LIFT for noisy or blurred imagery, AGAST for rotated or scaled scenes—should outperform any single default choice.","Replacing handcrafted descriptors with pre-trained VGG activations lowered retrieval accuracy, so CNN descriptors are not automatically better in this pipeline.","Training a detector on SIFT keypoints may inherit the limitations of SIFT, which helps explain why LIFT's retrieval performance was weak.","CNN-based detectors underperformed under rotation, so rotation invariance remains a specific weakness that future learned methods would need to address."],"supporting_citations":[{"why":"Supplies the SIFT detector/descriptor baseline and the keypoints used to supervise LIFT.","marker":"[3]"},{"why":"SURF is the top-performing handcrafted detector in the retrieval benchmark.","marker":"[4]"},{"why":"Quad is the learned detector with the best CNN retrieval results and was re-implemented for this comparison.","marker":"[10]"},{"why":"LIFT is the combined learned detector and descriptor whose weak retrieval accuracy is a key data point.","marker":"[11]"},{"why":"CenSurE is a handcrafted detector baseline with mid-range retrieval accuracy.","marker":"[14]"},{"why":"AGAST is a handcrafted detector baseline that wins repeatability under rotation and scaling.","marker":"[15]"},{"why":"VGG-16 activations are used as a CNN descriptor in place of handcrafted descriptors.","marker":"[20]"},{"why":"The DTU dataset is used to train the authors' re-implementation of the Quad detector.","marker":"[23]"}],"fun_headline_variants":["Handcrafted SURF beats deep CNN detectors in retrieval","CNN detectors don't outclass SURF in image retrieval","Deep features lose to classic SURF in retrieval test","No clear win for CNN detectors over classic features"],"cache_read_input_tokens":9088,"weakest_assumption_plain":"The comparison assumes that the re-implemented Quad detector and the parameter settings for every method faithfully represent the published methods, so the context-dependent rankings are not artifacts of unfair configuration.","fun_headline_variants_meta":{"raw":{"variants":["Handcrafted SURF beats deep CNN detectors in retrieval","CNN detectors don't outclass SURF in image retrieval","Deep features lose to classic SURF in retrieval test","No clear win for CNN detectors over classic features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":2078,"prompt_tokens":884,"completion_tokens":1194,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1130}},"tokens_in":500,"tokens_out":1194,"duration_ms":9201,"temperature":1.0,"reasoning_tokens":1130,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:15:23.854167+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the same 250K-image retrieval benchmark with official implementations of every detector and a fixed descriptor across all detectors would settle the claim; if a CNN detector then matches or exceeds SURF's 60.16% top-1 accuracy, the paper's conclusion that handcrafted features remain the retrieval baseline would no longer hold.","supporting_citations":[{"cited_title":"Quad- networks: unsupervised learning to rank for interest point detection,","cited_arxiv_id":null,"evidence_quote":"Quad is the learned detector with the best CNN retrieval results and was re-implemented for this comparison."},{"cited_title":"LIFT: Learned Invariant Feature Transform,","cited_arxiv_id":null,"evidence_quote":"LIFT is the combined learned detector and descriptor whose weak retrieval accuracy is a key data point."},{"cited_title":"Agrawal, K","cited_arxiv_id":null,"evidence_quote":"CenSurE is a handcrafted detector baseline with mid-range retrieval accuracy."},{"cited_title":"Adaptive and generic corner detection based on the accelerated segment test,","cited_arxiv_id":null,"evidence_quote":"AGAST is a handcrafted detector baseline that wins repeatability under rotation and scaling."},{"cited_title":"Interesting interest points,","cited_arxiv_id":null,"evidence_quote":"The DTU dataset is used to train the authors' re-implementation of the Quad detector."}],"review_version":1}