REVIEW 4 major objections 5 minor 1 cited by
Global-to-Local or Local-to-Global? Enhancing Image Retrieval with Efficient Local Search and Effective Global Re-ranking
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A local-first retrieval pipeline with per-query MDS embeddings sets new state of the art on Oxford and Paris.
desk verdict Real new pipeline and plausible mechanism, but the headline SOTA claim is over-scoped and the unablated missing-distance imputation leaves the core MDS result under-supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Multidimensional scaling (MDS), specifically the SMACOF variant, is the load-bearing mechanism. MDS takes a matrix of pairwise dissimilarities and returns points whose Euclidean distances approximate those dissimilarities; SMACOF does this by iteratively minimizing a stress function and can handle non-metric and incomplete inputs. Here it converts Chamfer-based local retrieval rankings, which are non-metric and known only for near neighbors, into a query-specific Euclidean embedding of the query and its top-k candidates. That embedding is then usable by a metric-space global re-ranking method, letting local information flow into what is normally a global-feature pipeline.
What would settle it
Compute true Chamfer distances for all pairs among the top-k images, run the same MDS re-ranking on the dense exact matrix, and compare to the sparse matrix with missing entries set to 1; if the sparse version's gain vanishes, the reported improvement rests on the imputation rather than on the local-to-global structure.
Extended reading notes
Core claim
The paper's central claim is that local-to-global retrieval—efficient local-feature search followed by global-feature re-ranking—outperforms the conventional global-to-local design. On a query, CANN with FIRE local features retrieves the top-k database images using Chamfer similarity; this stage is what finds partial matches. The authors then build a small (k+1)-point dissimilarity matrix from those pairwise local similarities, fill unknown pairwise distances with the maximum distance 1, and run SMACOF/MDS to produce an embedding that approximately respects the local distances. That embedding serves as an on-the-fly global feature, fused with SuperGlobal features and passed through the globa
Load-bearing premise
The load-bearing premise is that filling the unknown pairwise distances with 1, and then running SMACOF on this partially synthetic matrix, yields an embedding whose ordering still reflects true local similarities well enough to improve re-ranking; the paper neither ablate nor prove this.
Editorial extensions
If this is right
- Large-scale retrieval can be built around a local search stage without losing speed: the whole query pipeline runs in about 0.7 s on a 24-core CPU.
- Partial matches and viewpoint or occlusion cases that global features miss become findable, because local similarities drive the initial ranking.
- Any non-metric, sparse similarity measure—not just FIRE Chamfer distance—can be turned into a metric re-ranking input by per-query MDS; the authors show swapping in AMES improves the same framework.
- On-the-fly embeddings make re-ranking query-specific, so a single global feature model need not be re-trained to benefit.
Reading between the lines
- The paper leaves the missing-distance imputation untested: setting absent pairwise distances to 1 makes the synthetic matrix drive part of the embedding, and an exact-distance ablation would show how much of the gain depends on that choice.
- The same per-query MDS trick should generalize beyond images to any domain with an expensive, non-metric pairwise similarity and a cheap top-k index—document, product, or molecular retrieval, for instance.
- The framework suggests global embeddings could be synthesized on demand rather than pretrained, which would let retrieval systems adapt to new similarity measures without retraining feature extractors.
- Because MDS is recomputed for k+1 points per query, the approach could be combined with fast approximate MDS to scale k substantially, potentially improving recall on very hard cases.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a 'local-to-global' (L2G) image retrieval pipeline: an efficient local-feature search stage (CANN with FIRE features) is followed by a global-feature re-ranking stage in which query-specific 'global' embeddings are computed on-the-fly using multidimensional scaling (SMACOF) over sparse pairwise Chamfer dissimilarities among the query and its top-k neighbours; these MDS embeddings are then mixed with SuperGlobal embeddings and re-ranked using the method of Shao et al. [26]. Experiments on Revisited Oxford and Paris claim state-of-the-art results, especially on the +1M Hard variants. The main novelty is the use of MDS to convert non-metric local similarities into an embedding that can be plugged into a metric-space global re-ranker.
Significance. If the claims are supported, the paper makes a useful conceptual inversion of the dominant global-to-local paradigm, and the on-the-fly MDS embedding is a plausible way to inject non-metric local matching information into a global re-ranking framework. The ablation study (Table 2) and the 0.7s per-query timing are helpful. However, the headline SOTA claim is currently undermined by three load-bearing issues: hyperparameters are tuned on the ROxf benchmark used for evaluation, the missing-distance imputation in the MDS matrix is unablated, and the SOTA statement is contradicted by the paper's own Table 1 on base Oxford. The MDS-based re-ranking is not circular in the formal sense because the final mAP is computed against external ground truth, but the MDS features are derived from the same local similarities that produced the initial ranking, so the improvement should be interpreted as re-encoding the retrieval signal rather than adding a fully independent source of information.
major comments (4)
- [§4.1, Table 1] Hyperparameters w, p, epsilon, k and the CANN implementation are tuned on the ROxf benchmark ('All hyperparameters were tuned on a small sample (1000 images) of Oxford only', 'tune it on the ROxford dataset'), and ROxf is also one of the evaluation datasets in Table 1. This means the ROxf and ROxf+1M results are not a clean test of the method. The authors should report results with hyperparameters selected on a genuine held-out set (e.g., RPar or a non-overlapping validation split of ROxf) and/or provide a sensitivity analysis; otherwise the headline 79.8% ROxf+1M Hard result may be inflated by test-set tuning.
- [§3.3, Table 2] The MDS matrix is formed by setting every missing pairwise distance to 1, the maximum possible distance. With k=700 and database sizes of roughly 5,000 (ROxf) or 6,300 (RPar), each index image stores only its top-k neighbours, so the majority of off-diagonal entries in the 701×701 query-specific matrix are this fabricated constant rather than measured similarity. The paper does not report the fraction of known entries, and no ablation varies the imputation value or scheme. Table 2 shows the MDS stage is essential (91.7 vs 86.4 on RPar Hard), but it does not separate the effect of the MDS mechanism from the effect of the imputation heuristic. This is load-bearing: the claim that the embedding 'respects the local similarities' is not established if the embedding is dominated by a constant. Please provide imputation sensitivity experiments and coverage statistics.
- [Abstract, §4.2, Table 1] The abstract and Section 4.2 claim 'setting new state-of-the-art results on the Revisited Oxford and Paris datasets' and 'beating the best AMES results by 2.1% and 1.4%'. In Table 1, on base ROxf, AMES achieves higher mAP than the proposed method on both Medium (93.6 vs 92.9) and Hard (84.8 vs 83.0). The 2.1%/1.4% gains apply only to the +1M Hard columns. The claims should be qualified to the large-scale Hard setting, or the base results should be discussed honestly.
- [§3.2 vs §3.3, Table 2] There is an internal inconsistency: Section 3.2 states that SMACOF can 'infer the missing entries to construct a complete distance matrix', but Section 3.3 replaces missing entries with the constant 1 instead of performing inference. This distinction matters because the ablation 'Replace FIRE local similarity by AMES' yields 93.6/85.2 (RPar/ROxf Hard), which is better than the full model's 91.7/83.0. The paper should reconcile why the advertised inference capability is not used, and why the best configuration in Table 2 is not the one evaluated against the state of the art in Table 1.
minor comments (5)
- [§3.3] 'Standard MDS' is ambiguous: earlier text describes SMACOF, but Section 3.3 says 'standard MDS' and gives O(k^2) time. Please specify the exact MDS variant (classical, metric, or SMACOF), the number of iterations, and the stress convergence criterion used.
- [Figure 4] The axes are not labeled; please state what 'topK' means and whether the curves show counts of correct images or mAP at a cutoff.
- [Table 2] Please specify for each row whether the reported numbers are Medium/Hard and base/+1M. The column headers 'RParis' and 'ROxford' alone are insufficient, especially because Table 1 has four variants per dataset.
- [§4.1] The symbol k is used both for the number of top-ranked images fed to MDS and for the neighbourhood size in the SuperGlobal re-ranker. This is confusing; consider renaming one of them.
- [References] Reference [26] (Shao et al.) has no venue or publication year; please update. Also, the paper links to the official CANN implementation but not to the L2G/MDS code; releasing the re-ranking implementation would improve reproducibility.
Circularity Check
No significant circularity: all load-bearing claims are empirical and evaluated on external ground truth.
full rationale
The L2G pipeline is an empirical system: CANN local retrieval produces an initial ranking; MDS embeds the query and top-k images from pairwise dissimilarities (with missing entries imputed to 1); the embedding is combined with SuperGlobal features and re-ranked following the external method of Shao et al. [26]. The final mAP is computed against ground-truth annotations, which are outside the system. MDS is defined to preserve the input dissimilarities, so saying the embedding 'respects local similarities' is a property of MDS, not a circular prediction of retrieval quality. Re-ranking does not reduce to the initial ranking: the ablation shows substantial differences (RPar Hard 91.7 with MDS vs 86.4 when MDS is replaced by SuperGlobal), so the transformation adds information. Self-citations to CANN [1] and SuperGlobal [26] are to prior published systems with public implementations and are used as building blocks, not as unverified justifications. The imputation of missing distances to 1 is an unablated heuristic—a correctness/robustness concern, not a circularity. No equation or definition makes a claimed output equivalent to an input.
Assumptions & free parameters
free parameters (6)
- w =
0.19
- p =
0.01
- epsilon =
0.1
- k =
700
- M =
1600
- missing_distance_imputation =
1
assumptions (5)
- standard math SMACOF stress minimization yields a Euclidean configuration that preserves the ordering of non-metric dissimilarities and can complete missing entries.
- domain assumption CANN Chamfer similarity is a valid retrieval signal and the top-k candidates contain enough relevant images.
- ad hoc to paper Setting missing distances to 1 and applying MDS to the sparse matrix preserves ranking information rather than corrupting it.
- domain assumption The Shao et al. re-ranking algorithm remains effective when fed MDS embeddings instead of trained global features.
- ad hoc to paper Weighted averaging of SuperGlobal and MDS embeddings with w=0.19 is complementary and beneficial.
Cite this review
Pith. "Pith review of Global-to-Local or Local-to-Global? Enhancing Image Retrieval with Efficient Local Search and Effective Global Re-ranking." pith.science (2026). https://pith.science/paper/LUCYSFV5
@misc{pith2026250904351,
author = {Pith},
title = {Pith review of: Global-to-Local or Local-to-Global? Enhancing Image Retrieval with Efficient Local Search and Effective Global Re-ranking},
year = {2026},
howpublished = {\url{https://pith.science/paper/LUCYSFV5}},
note = {Machine review of arXiv:2509.04351}
}
read the original abstract
The dominant paradigm in image retrieval systems today is to search large databases using global image features, and re-rank those initial results with local image feature matching techniques. This design, dubbed global-to-local, stems from the computational cost of local matching approaches, which can only be afforded for a small number of retrieved images. However, emerging efficient local feature search approaches have opened up new possibilities, in particular enabling detailed retrieval at large scale, to find partial matches which are often missed by global feature search. In parallel, global feature-based re-ranking has shown promising results with high computational efficiency. In this work, we leverage these building blocks to introduce a local-to-global retrieval paradigm, where efficient local feature search meets effective global feature re-ranking. Critically, we propose a re-ranking method where global features are computed on-the-fly, based on the local feature retrieval similarities. Such re-ranking-only global features leverage multidimensional scaling techniques to create embeddings which respect the local similarities obtained during search, enabling a significant re-ranking boost. Experimentally, we demonstrate solid retrieval performance, setting new state-of-the-art results on the Revisited Oxford and Paris datasets.
Figures
Forward citations
Cited by 1 Pith paper
-
LARE: Low-Attention Region Encoding for Text-Image Retrieval
LARE uses parallel encoding of full images and low-attention regions to improve text-image retrieval, shown on a new Dense-Set subset of COCO and Flickr30K with re-captioned overlooked areas.
Reference graph
Works this paper leans on
-
[26]
S. Shao, K. Chen, A. Karpur, Q. Cui, A. Araujo, and B. Cao. Global Features are All You Need for Image Retrieval and Reranking. 2023. 2, 3, 4, 5, 6, 7
work page 2023
- [1]
-
[2]
A. Babenko, A. Slesarev, A. Chigorin, and V . Lempitsky. Neural Codes for Image Retrieval. InProc. ECCV, 2014. 2
work page 2014
-
[3]
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-Up Robust Features (SURF).CVIU, 2008. 2
work page 2008
-
[4]
B. Cao, A. Araujo, and J. Sim. Unifying Deep Local and Global Features for Image Search. InProc. ECCV, 2020. 1, 2, 6
work page 2020
-
[5]
Y . Chen, H. Hu, Y . Luan, H. Sun, S. Changpinyo, A. Ritter, and M.-W. Chang. Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions? In Proc. EMNLP, 2023. 1
work page 2023
-
[6]
Applications of convex analysis to multidi- mensional scaling
Jan De Leeuw. Applications of convex analysis to multidi- mensional scaling. In J. R. Barra, F. Brodeau, G. Romier, and B. Van Cutsem, editors,Recent Developments in Statistics, pages 133–145. North-Holland Publishing Company, 1977. 4, 5
work page 1977
-
[7]
Muvera: Multi-vector retrieval via fixed dimensional encodings
Laxman Dhulipala, Majid Hadian, Rajesh Jayaram, Jason Lee, and Vahab Mirrokni. Muvera: Multi-vector retrieval via fixed dimensional encodings. InAdvances in Neural In- formation Processing Systems, 2023. 3
work page 2023
Show all 39 references
-
[8]
Fastmap: A fast algo- rithm for indexing, data-mining and visualization of tradi- tional and multimedia datasets
Christos Faloutsos and King-Ip Lin. Fastmap: A fast algo- rithm for indexing, data-mining and visualization of tradi- tional and multimedia datasets. InProceedings of the 1995 ACM SIGMOD international conference on Management of data, pages 163–174, 1995. 5
1995
-
[9]
Gordo, J
A. Gordo, J. Almazan, J. Revaud, and D. Larlus. End-to- end Learning of Deep Visual Representations for Image Re- trieval.IJCV, 2017. 2
2017
-
[10]
Z. Hu, A. Iscen, C. Sun, Z. Wang, K.-W. Chang, Y . Sun, C. Schmid, D. Ross, and A. Fathi. REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory. InProc. CVPR, 2023. 1
2023
-
[11]
J ´egou, F
H. J ´egou, F. Perronnin, M. Douze, J. Sanchez, P. Perez, and C. Schmid. Aggregating Local Image Descriptors into Com- pact Codes.PAMI, 2012. 2
2012
-
[12]
Krause, M
J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3D Object Representations for Fine-Grained Categorization. InProc. ICCV Workshops, 2013. 1
2013
-
[13]
S. Lee, H. Seong, S. Lee, and E. Kim. Correlation Verifica- tion for Image Retrieval. InProc. CVPR, 2022. 1, 2, 6
2022
-
[14]
Z. Liu, P. Luo, S. Qiu, X. Wang, and X. Tang. Deepfash- ion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations. InProc. CVPR, 2016. 1
2016
-
[15]
D. Lowe. Distinctive Image Features from Scale-Invariant Keypoints.IJCV, 2004. 2
2004
-
[16]
Mensink, J
T. Mensink, J. Uijlings, L. Castrejon, A. Goel, F. Cadar, H. Zhou, F. Sha, A. Araujo, and V . Ferrari. Encyclopedic VQA: Visual Questions About Detailed Properties of Fine-Grained Categories. InProc. ICCV, 2023. 1
2023
-
[17]
T. Ng, V . Balntas, Y . Tian, and K. Mikolajczyk. SOLAR: Second-Order Loss and Attention for Image Retrieval. In Proc. ECCV, 2020. 2
2020
-
[18]
Nist ´er and H
D. Nist ´er and H. Stewenius. Scalable Recognition with a V ocabulary Tree. InProc. CVPR, 2006. 2
2006
-
[19]
H. Noh, A. Araujo, J. Sim, T. Weyand, and B. Han. Large- Scale Image Retrieval with Attentive Deep Local Features. InProc. ICCV, 2017. 2
2017
-
[20]
J. Peng, C. Xiao, and Y . Li. RP2K: A Large-Scale Re- tail Product Dataset for Fine-Grained Image Classification. arXiv:2006.12634, 2021. 1
2006 arXiv
-
[21]
Philbin, O
J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object Retrieval with Large V ocabularies and Fast Spatial Matching. InProc. CVPR, 2007. 2, 5
2007
-
[22]
Philbin, O
J. Philbin, O. Chum, M. Isard, J. Sivic, and A. Zisserman. Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases. InProc. CVPR, 2008. 5
2008
-
[23]
Radenovi ´c, A
F. Radenovi ´c, A. Iscen, G. Tolias, Y . Avrithis, and O. Chum. Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking. InProc. CVPR, 2018. 2, 5
2018
-
[24]
Revaud, J
J. Revaud, J. Almazan, R. S. Rezende, and C. R. Souza. Learning With Average Precision: Training Image Retrieval With a Listwise Loss. InProc. ICCV, October 2019. 2
2019
-
[25]
Saeed, H
N. Saeed, H. Nam, M. Haq, and D. Saqib. A Survey on Multidimensional Scaling.ACM Comput. Surv., 2018. 2, 4
2018
-
[27]
V . d. Silva and J. B. Tenenbaum. Sparse multidimensional scaling using landmark points.Technical Report (Stanford University), 2004. 4
2004
-
[28]
Sivic and A
J. Sivic and A. Zisserman. Video Google: A Text Retrieval Approach to Object Matching in Videos. InICCV, 2003. 2
2003
-
[29]
H. Song, Y . Xiang, S. Jegelka, and S. Savarese. Deep Metric Learning via Lifted Structured Feature Embedding. InProc. CVPR, 2016. 1
2016
-
[30]
P. Suma, G. Kordopatis-Zilos, A. Iscen, and G. Tolias. AMES: Asymmetric and Memory-Efficient Similarity Esti- mation for Instance-level Retrieval. InProc. ECCV, 2024. 1, 2, 6, 8
2024
-
[31]
F. Tan, J. Yuan, and V . Ordonez. Instance-level Image Re- trieval using Reranking Transformers. InProc. ICCV, 2021. 1, 2
2021
-
[32]
Teichmann, A
M. Teichmann, A. Araujo, M. Zhu, and J. Sim. Detect-to- Retrieve: Efficient Regional Aggregation for Image Search. InCVPR, 2019. 2
2019
-
[33]
Tolias, Y
G. Tolias, Y . Avrithis, and H. Jegou. Image Search with Se- lective Match Kernels: Aggregation Across Single and Mul- tiple Images.IJCV, 2015. 2
2015
-
[34]
Tolias, T
G. Tolias, T. Jenicek, and O. Chum. Learning and Aggregat- ing Deep Local Descriptors for Instance-Level Recognition. InECCV, 2020. 2
2020
-
[35]
Global versus local methods in nonlinear dimensionality reduction.Neural Networks, 23(1):125–136, 2010
Jarkko Venna, Jaakko Peltonen, Kristian Nybo, Helena Ai- dos, and Samuel Kaski. Global versus local methods in nonlinear dimensionality reduction.Neural Networks, 23(1):125–136, 2010. 4
2010
-
[36]
Weinzaepfel, T
P. Weinzaepfel, T. Lucas, D. Larlus, and Y . Kalantidis. Learning Super-Features for Image Retrieval. InICLR, 2022. 2, 5, 7
2022
-
[37]
M. Yang, D. He, M. Fan, B. Shi, X. Xue, F. Li, E. Ding, and J. Huang. DOLG: Single-Stage Image Retrieval with Deep Orthogonal Fusion of Local and Global Features. InProc. ICCV, 2021. 2, 6
2021
-
[38]
Ypsilantis, K
N.-A. Ypsilantis, K. Chen, B. Cao, M. Lipovsk ´y, P. Dogan- Schonberger, G. Makosa, B. Bluntschli, M. Seyedhosseini, O. Chum, and A. Araujo. Towards Universal Image Em- beddings: A Large-Scale Dataset and Challenge for Generic Image Representations. InProc. ICCV, 2023. 1
2023
-
[39]
Ypsilantis, N
N.-A. Ypsilantis, N. Garcia, G. Han, S. Ibrahimi, N. Van No- ord, and G. Tolias. The Met Dataset: Instance-level Recog- nition for Artworks. InProc. NeurIPS Datasets and Bench- marks Track, 2021. 1
2021
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.