REVIEW 4 major objections 5 minor 1 cited by
A frozen vision-language model, prompted with retrieved coordinates of similar and dissimilar images from a ten-million-image gallery, localizes street photos within 1 km more accurately than published trained baselines on three benchmarks,
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Using SigLIP retrieval to feed a Qwen2-VL or InternVL2 model with similar and dissimilar coordinates yields reported street-level accuracies of 23.2%, 17.1%, and 24.3% on IM2GPS, IM2GPS3k, and YFCC4k.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection A clean engineering write-up of RAG+MLLM geolocation whose headline numbers are undermined by a missing overlap check and no retrieval-only baseline. the 4 major comments →
Street-Level Geolocalization Using Multimodal Large Language Models and Retrieval-Augmented Generation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper's central claim is that retrieval-augmented generation by itself—without any task-specific training—is enough to push street-level geolocation past previous published results. The system stores embeddings and coordinates for roughly 10 million images from the EMP-16 and OSV-5M collections. A query image is embedded with SigLIP, and Faiss returns the 16 nearest and 16 farthest neighbours by Euclidean distance; those coordinates, together with the query image, form the prompt for Qwen2-VL-72B-Instruct or InternVL2-76B. The model's coordinate output is scored by geodesic distance against ground truth. The paper reports 1 km accuracy of 23.2% on IM2GPS, 17.1% on IM2GPS3k, and 24.3% on
What carries the argument
The machinery is a hybrid retrieval gallery plus contrastive prompting. Roughly ten million geotagged images—everyday user photos from EMP-16 and road-level street view from OSV-5M—are embedded with SigLIP so that visual similarity is measured as Euclidean distance. For a new photo, Faiss retrieves the 16 closest and 16 farthest images; their GPS coordinates are written into the prompt as positive and negative context. The multimodal language model, which is frozen and quantized, must reconcile the query image with these coordinate hints and output a location. The contrast between near and far neighbours is the load-bearing design choice: it gives the model both evidence for where the scene
Load-bearing premise
The benchmark images (or near-duplicates of them) are not sitting in the retrieval gallery with their correct coordinates; if they were, the nearest-neighbour retrieval could hand the model the answer and the accuracy would not be true localization.
What would settle it
Embed every image in IM2GPS, IM2GPS3k, and YFCC4k, search the ten-million-image gallery, and list any benchmark query whose nearest neighbour is itself or an obvious near-duplicate. Then recompute the 1 km accuracies after deleting those neighbours from the gallery; if the reported margins shrink or vanish, the result was answer-copying rather than geolocation.
If this is right
- Adding new imagery to the retrieval gallery improves or extends geolocation coverage without retraining any model, so the system can track new cities or eras by ingestion alone.
- Because the encoder and the language model are swappable, improvements from either component should transfer directly to the pipeline without task-specific training.
- YFCC4k gains of roughly ten percentage points at street level and five at city level show the largest wins occur at fine granularity, where specialized classifiers previously struggled.
- Quantized large models are enough; the method does not require full-precision weights, lowering the hardware barrier to deployment.
Where Pith is reading between the lines
- A deciding test the paper does not report is gallery-benchmark overlap: if a nontrivial share of correct 1 km predictions correspond to near-duplicates already in EMP-16 or OSV-5M, the method is partly reading coordinates from memory. A deduplication pass should be run before relying on the headline numbers.
- The 169 YFCC4k images that could not be downloaded (3.7% of the dataset) were excluded from the authors' run but not from the baselines; how that missing slice shifts scores is unquantified.
- The method's ceiling is set by gallery coverage: a photo from a region with no similar stored images will have only distant neighbours as context, so the model must fall back on its own geographic priors. A held-out-region evaluation would reveal how much of the accuracy is retrieval versus model knowledge.
- Ablating the 'dissimilar' half of the prompt (16 similar only versus 16 similar plus 16 dissimilar) would isolate whether negative coordinates actually add signal or mostly lengthen the prompt.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a retrieval-augmented generation (RAG) approach to image geolocation. The authors build a Faiss index over SigLIP embeddings of roughly 10 million images from two sources (EMP-16 and OSV-5M). For each query image, they retrieve the 16 nearest and 16 farthest image coordinates and feed those coordinates together with the query image into open-weight multimodal LLMs (Qwen2-VL-72B-Instruct and InternVL2-Llama3-76B), which are asked to output a geolocation. They report street-level (1 km) accuracies of 23.2% on IM2GPS, 17.1% on IM2GPS3k, and 24.3% on YFCC4k, claiming state-of-the-art results without fine-tuning. The core evaluation, however, has a potentially circular component: the RAG gallery is drawn from Flickr/YFCC-derived data that overlaps in provenance with the benchmark query sets, and the paper does not analyze whether query images or near-duplicates are present in the gallery.
Significance. If the reported numbers survive a clean evaluation, this would be a practically useful result: frozen open-weight MLLMs prompted with retrieved coordinates can outperform specialized geolocation models at street level while avoiding fine-tuning and allowing incremental database expansion. The hybrid gallery of user-generated images and street-view imagery, the use of both similar and dissimilar retrieved coordinates, and the open-weight reproducibility are all strengths. The paper does not provide code or a retrieval-only baseline, but the general recipe is simple enough to re-implement. The significance is conditional on resolving the overlap and baseline concerns; without that, the claimed state-of-the-art result cannot be attributed to the proposed method.
major comments (4)
- [§III-A, Table I] The RAG gallery is constructed from EMP-16 plus OSV-5M, and the benchmarks IM2GPS, IM2GPS3k, and YFCC4k are all Flickr/YFCC-derived collections. The paper never reports an overlap or deduplication analysis between the gallery and the query sets. If a query image or a near-duplicate is present in the gallery, the top SigLIP neighbor can be the query itself; the retrieved coordinates are then the ground-truth answer, and the reported prediction is label copying rather than geolocation. This is especially relevant to the +9.9 point gain on YFCC4k. The authors must either (i) demonstrate, via photo IDs or perceptual near-duplicate detection, that the gallery and benchmark queries are disjoint, or (ii) rerun after excluding all overlapping and near-duplicate items. Without this, the central SOTA claim in Table I is unsupported.
- [§III-C and Fig. 2] The prompt contains the query image and the retrieved (x, y) coordinates, but not the retrieved images. Consequently, a system that simply returns the top-1 retrieved coordinate is a natural and much simpler baseline, and it is absent from Table I. The comparisons against PlaNet, GeoDecoder, Img2Loc, etc. do not isolate the MLLM's contribution, because none of these is the retriever alone. If the nearest-neighbor coordinate already achieves the reported accuracies, the MLLM is only reciting the retrieved label. Add retrieval-only columns (e.g., top-1 and majority-vote over the 16 retrieved coordinates) for all three benchmarks.
- [§III-C] The hyperparameters — 16 similar / 16 dissimilar neighbors, temperature 0.1, top-p 0.1, max length 6,000, max tokens 512 — are described as empirically determined, but no validation split or cross-validation is reported. Since the same benchmark sets are used both for this selection and for final evaluation, the reported accuracies can be optimistically biased. This is especially important with only 237–4,536 test images. Specify the validation protocol, or re-run the model selection on a separate split and report the test-set numbers.
- [§IV and §V (YFCC4k)] For YFCC4k, the evaluation drops 169 of 4,536 images (3.7%) because they were unavailable, while prior methods in Table I were evaluated on the full set. The Discussion asserts this 'did not significantly affect overall results', but no evidence is provided. The percentages in Table I are therefore not directly comparable. The exact subset used should be released, the full set should be re-run, or prior methods should be evaluated on the same 4,367-image subset. This is load-bearing for the YFCC4k SOTA claim.
minor comments (5)
- [Abstract, §I, §IV] The abstract and conclusion claim the method achieves 'higher accuracy' on all three benchmarks, but at 200 km and 750 km the method is worse than the previous best on IM2GPS and IM2GPS3k by 6+ percentage points. The claims should be restricted to the specific thresholds where the improvement holds, or the wording should be qualified.
- [§IV] The text says the IM2GPS3k continent-level improvement is '+0.1%', while Table I shows 85.6% vs. 84.7% (a +0.9 percentage-point difference). The same paragraph also refers to a '+0.1% improvement at continent level' in the summary list. Correct this inconsistency.
- [§II, reference [66]] The paper refers to [66] as PIGEOTTO, but the reference title is 'Pigeon: Predicting image geolocations' (CVPR 2024). Either the model name or the citation is wrong; please verify the correct model and cite accordingly.
- [Throughout] Notation and copy-editing: 'Img2GPS' / 'IM2GPS' are used inconsistently; 'succesful' appears in the introduction; and some sentences are incomplete (e.g., 'current methods still grapple with issues such as ... sparsely labeled data [11] that generalize effectively'). A careful proofread is needed.
- [Fig. 1, §III-A] Figure 1 is credited as 'adapted from [45]', and the method is closely related to Img2Loc. The paper should clearly state the concrete differences beyond the database size, image encoder, and model choice, so that the novelty is unambiguous.
Circularity Check
RAG gallery may contain benchmark queries and k/model selection appear tuned on the same benchmarks, making the reported SOTA partly a lookup/test-set-selection artifact.
specific steps
-
fitted input called prediction
[Section III-C (Implementation) and Table I]
"We empirically determined that retrieving the 16 most similar and 16 most dissimilar embeddings yielded the best performance. Fewer neighbors (1, 5, or 10) provided less contrastive information, while larger sets did not yield additional improvements and increased computational overhead."
The neighbor count k=16/16 is selected by 'best performance' without specifying a held-out split. If that performance was measured on IM2GPS/IM2GPS3k/YFCC4k, then the accuracy reported in Table I is the very quantity used to choose the hyperparameter; the 'prediction' is partly forced by the selection criterion rather than being an independent evaluation. The paper gives no evidence of a separate validation set.
-
fitted input called prediction
[Section III-B (Model Selection) and Table I]
"The preliminary test results are consistent with previous research [75, 18]. This can promise a higher performance for MLLMs with a higher number of parameters, in geolocation estimation tasks, similar to other tasks [76, 77, 78]. Consequently, certain models mentioned above were excluded from the final analysis to ensure fairness in the comparison. Ultimately, two of the most successful of our tasks MLLMs —Qwen2-VL-72B-Instruct and InternVL2-Llama3-76B— are utilized."
The two reported models were selected as 'the most successful' on the basis of 'preliminary test results.' If those tests used the same benchmark datasets, the Table I results are conditional on having chosen the best-scoring models on those very benchmarks; the SOTA numbers then partly measure the selection process, not an out-of-sample prediction. No separate validation split is described.
-
other
[Section III-A (RAG Database Construction and Image Query) and Table I]
"After constructing the RAG database, we use the SigLIP image encoder to identify the most similar and most dissimilar neighbors for the incoming image. By ”most similar” and ”most dissimilar,” we refer to finding the nearest and farthest embeddings within the vector space generated by the SigLIP image encoder. In our case, the similarity and dissimilarity are determined based on the L2 distance (Euclidean distance) between vector spaces. The location information from the most similar and most dissimilar images is then used to augment the prompt, which serves as input to the MLLMs system."
This construction closes a loop if a benchmark query is in the gallery: the query's embedding is distance 0 from itself, so the 'most similar' retrieved location is the ground-truth coordinate, and that coordinate is inserted into the prompt. The MLLM output is then a restatement of the retrieved answer, not an independent geolocation prediction. The EMP-16 portion of the gallery is a 4.6M-image geo-tagged collection of the same kind as the Flickr/YFCC-derived benchmarks, and the paper reports no deduplication or overlap analysis; the Table I SOTA numbers therefore cannot rule out this by-construction equivalence.
full rationale
The paper's derivation is empirical rather than equation-based, but the central SOTA claim has two circularity burdens. First, the RAG construction retrieves the nearest embedding and feeds its coordinates into the MLLM prompt. Because EMP-16 is a 4.6M image collection drawn from the same public Flickr/YFCC pool as IM2GPS, IM2GPS3k and YFCC4k, and because no deduplication or overlap check is reported, any benchmark query that is present in the gallery makes the 'similar' location equal to the ground-truth label by L2 identity. The 'prediction' then reduces to reading the retrieved answer back out of the prompt. Second, the method's free parameters—the number of retrieved neighbors and even the MLLM itself—were selected based on 'best performance'/'preliminary test results' without a stated validation split; if those results used the same benchmarks, the reported Table I numbers are the optimization target, not an independent evaluation. The absence of a retrieval-only baseline further blurs whether the MLLM adds any geolocation ability beyond copying the retrieved coordinate. These issues are conditional (they depend on unseen overlap and on how 'preliminary' tests were split), so the paper is not definitionally circular in every case; but the burden is on the authors to show the gallery excludes the test images and that the configuration was not tuned on the test set. No load-bearing self-citation chain is present; the circularity is in the empirical construction, not in the references. Hence score 6.
Axiom & Free-Parameter Ledger
free parameters (6)
- similar_neighbors_k =
16
- dissimilar_neighbors_k =
16
- temperature =
0.1
- top_p =
0.1
- max_tokens =
512
- max_model_length =
6000
axioms (4)
- domain assumption Benchmark ground-truth GPS coordinates are correct
- domain assumption No overlap between RAG gallery (EMP-16 and OSV-5M) and benchmark query sets
- domain assumption SigLIP L2 distance ranks images by geographic relevance
- domain assumption MLLMs can fuse image content with textual coordinates
Cite this review
Pith. "Pith review of Street-Level Geolocalization Using Multimodal Large Language Models and Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/PJDMJA2M
@misc{pith2026250901341,
author = {Pith},
title = {Pith review of: Street-Level Geolocalization Using Multimodal Large Language Models and Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/PJDMJA2M}},
note = {Machine review of arXiv:2509.01341}
}
read the original abstract
Street-level geolocalization from images is crucial for a wide range of essential applications and services, such as navigation, location-based recommendations, and urban planning. With the growing popularity of social media data and cameras embedded in smartphones, applying traditional computer vision techniques to localize images has become increasingly challenging, yet highly valuable. This paper introduces a novel approach that integrates open-weight and publicly accessible multimodal large language models with retrieval-augmented generation. The method constructs a vector database using the SigLIP encoder on two large-scale datasets (EMP-16 and OSV-5M). Query images are augmented with prompts containing both similar and dissimilar geolocation information retrieved from this database before being processed by the multimodal large language models. Our approach has demonstrated state-of-the-art performance, achieving higher accuracy compared against three widely used benchmark datasets (IM2GPS, IM2GPS3k, and YFCC4k). Importantly, our solution eliminates the need for expensive fine-tuning or retraining and scales seamlessly to incorporate new data sources. The effectiveness of retrieval-augmented generation-based multimodal large language models in geolocation estimation demonstrated by this paper suggests an alternative path to the traditional methods which rely on the training models from scratch, opening new possibilities for more accessible and scalable solutions in GeoAI.
Figures
Forward citations
Cited by 1 Pith paper
-
DisasterTD: Disaster Toponym Disambiguation Using Multimodal LLMs and Cross-View Geolocalization
Combining MLLM candidate place extraction with DINOv2 cross-view matching of social-media, street-view, and satellite images raises disaster photo geolocation accuracy to 71.6% within 1 km on Hurricane Harvey data.
Reference graph
Works this paper leans on
-
[1]
Revisiting im2gps in the deep learning era,
N. V o, N. Jacobs, and J. Hays, “Revisiting im2gps in the deep learning era,” in Proceedings of the IEEE international conference on computer vision , 2017, pp. 2621–2630
2017
-
[2]
Using twitter data to monitor natural disaster social dynamics: A recurrent neural net- work approach with word embeddings and kernel density estimation,
A. Hernandez-Suarez, G. Sanchez-Perez, K. Toscano- Medina, H. Perez-Meana, J. Portillo-Portillo, V . Sanchez, and L. J. Garc ´ıa Villalba, “Using twitter data to monitor natural disaster social dynamics: A recurrent neural net- work approach with word embeddings and kernel density estimation,” Sensors, vol. 19, no. 7, p. 1746, 2019
2019
-
[3]
Suwaileh, T
R. Suwaileh, T. Elsayed, and M. Imran, Role of Geolocation Prediction in Disaster Management . Singapore: Springer Nature Singapore, 2022, pp. 1–31. [Online]. Available: https://doi.org/10.1007/ 978-981-16-8800-3 176-2
2022
-
[4]
True lies in geospatial big data: detecting location spoofing in social media,
B. Zhao and D. Z. Sui, “True lies in geospatial big data: detecting location spoofing in social media,” Annals of GIS, vol. 23, no. 1, pp. 1–14, 2017
2017
-
[5]
Geotagging text content with language models and feature mining,
G. Kordopatis-Zilos, S. Papadopoulos, and I. Kompat- siaris, “Geotagging text content with language models and feature mining,” Proceedings of the IEEE , vol. 105, no. 10, pp. 1971–1986, 2017
1971
-
[6]
Geolocalization and navigation by visible light communication to ad- dress automated logistics control,
P. Louro, M. Vieira, and M. A. Vieira, “Geolocalization and navigation by visible light communication to ad- dress automated logistics control,” Optical Engineering, vol. 61, no. 1, pp. 016 104–016 104, 2022
2022
-
[7]
Ubiquitous real-time geo- spatial localization,
A. Gupta and A. Yilmaz, “Ubiquitous real-time geo- spatial localization,” in Proceedings of the Eighth ACM 8 SIGSPATIAL International Workshop on Indoor Spatial Awareness, 2016, pp. 1–10
2016
-
[8]
Single-image localisation using 3d models: Combining hierarchical edge maps and semantic segmentation for domain adap- tation,
D. Acharya, R. Tennakoon, S. Muthu, K. Khoshelham, R. Hoseinnezhad, and A. Bab-Hadiashar, “Single-image localisation using 3d models: Combining hierarchical edge maps and semantic segmentation for domain adap- tation,” Automation in Construction, vol. 136, p. 104152,
-
[9]
Analysing gender differences in the perceived safety from street view imagery,
Q. Cui, Y . Zhang, G. Yang, Y . Huang, and Y . Chen, “Analysing gender differences in the perceived safety from street view imagery,” International Journal of Ap- plied Earth Observation and Geoinformation , vol. 124, p. 103537, 2023
2023
-
[10]
Multi-level urban street representation with street-view imagery and hybrid se- mantic graph,
Y . Zhang, Y . Li, and F. Zhang, “Multi-level urban street representation with street-view imagery and hybrid se- mantic graph,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 218, pp. 19–32, 2024
2024
-
[11]
Street view imagery in urban analytics and gis: A review,
F. Biljecki and K. Ito, “Street view imagery in urban analytics and gis: A review,” Landscape and Urban Planning, vol. 215, p. 104217, 2021
2021
-
[12]
Global streetscapes – a comprehensive dataset of 10 million street-level images across 688 cities for urban science and analytics,
Y . Hou, M. Quintana, M. Khomiakov, W. Yap, J. Ouyang, K. Ito, Z. Wang, T. Zhao, and F. Biljecki, “Global streetscapes – a comprehensive dataset of 10 million street-level images across 688 cities for urban science and analytics,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 215, pp. 216–238, 2024
2024
-
[13]
OpenStreetView-5M: The Many Roads to Global Visual Geolocation
G. Astruc, N. Dufour, I. Siglidis, C. Aronssohn, N. Bouia, S. Fu, R. Loiseau, V . N. Nguyen, C. Raude, E. Vincent, L. XU, H. Zhou, and L. Landrieu, “Openstreetview-5m: The many roads to global visual geolocation,” 2024. [Online]. Available: https://arxiv.org/abs/2404.18873
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[14]
Are these from the same place? seeing the unseen in cross-view image geo- localization,
R. Rodrigues and M. Tani, “Are these from the same place? seeing the unseen in cross-view image geo- localization,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , 2021, pp. 3753–3761
work page 2021
-
[15]
Geolocation by light: accuracy and precision affected by environmen- tal factors,
S. Lisovski, C. M. Hewson, R. H. Klaassen, F. Korner- Nievergelt, M. W. Kristensen, and S. Hahn, “Geolocation by light: accuracy and precision affected by environmen- tal factors,” Methods in Ecology and Evolution , vol. 3, no. 3, pp. 603–612, 2012
work page 2012
-
[16]
Season-invariant gnss-denied visual localization for uavs,
J. Kinnari, F. Verdoja, and V . Kyrki, “Season-invariant gnss-denied visual localization for uavs,” IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 10 232–10 239, 2022
work page 2022
-
[17]
Language models are few-shot learners,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amo...
Pith/arXiv arXiv 2020
-
[18]
A survey on multimodal large language models,
S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” National Science Review , vol. 11, no. 12, Nov
-
[19]
Retrieval- augmented generation for knowledge-intensive nlp tasks,
P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W. tau Yih, T. Rockt ¨aschel, S. Riedel, and D. Kiela, “Retrieval- augmented generation for knowledge-intensive nlp tasks,” 2021. [Online]. Available: https://arxiv.org/abs/ 2005.11401
Pith/arXiv arXiv 2021
-
[20]
Retrieval-augmented generation for large language models: A survey,
Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, M. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” 2024. [Online]. Available: https://arxiv.org/abs/2312.10997
Pith/arXiv arXiv 2024
-
[21]
M. A. K. Raiaan, M. S. H. Mukta, K. Fatema, N. M. Fahad, S. Sakib, M. M. J. Mim, J. Ahmad, M. E. Ali, and S. Azam, “A review on large language models: Architectures, applications, taxonomies, open issues and challenges,” IEEE Access , vol. 12, pp. 26 839–26 874, 2024
work page 2024
-
[22]
H. Xu, J. Yuan, A. Zhou, G. Xu, W. Li, X. Ban, and X. Ye, “Genai-powered multi-agent paradigm for smart urban mobility: Opportunities and challenges for integrating large language models (llms) and retrieval-augmented generation (rag) with intelligent transportation systems,” 2024. [Online]. Available: https://arxiv.org/abs/2409.00494
Pith/arXiv arXiv 2024
-
[23]
Im2gps: estimating geographic information from a single image,
J. Hays and A. A. Efros, “Im2gps: estimating geographic information from a single image,” in 2008 ieee confer- ence on computer vision and pattern recognition. IEEE, 2008, pp. 1–8
work page 2008
-
[24]
Planet-photo ge- olocation with convolutional neural networks,
T. Weyand, I. Kostrikov, and J. Philbin, “Planet-photo ge- olocation with convolutional neural networks,” in Com- puter Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Pro- ceedings, Part VIII 14 . Springer, 2016, pp. 37–55
work page 2016
-
[25]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning. PMLR, 2021, pp. 8748–8763
2021
-
[26]
Sigmoid loss for language image pre-training,
X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 11 975–11 986
work page 2023
-
[27]
J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Boot- strapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742
work page 2023
-
[28]
Dinov2: Learning robust visual features without supervision,
M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023
Pith/arXiv arXiv 2023
-
[29]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” CoRR, vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762 9
Pith/arXiv arXiv 2017
-
[30]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019. [Online]. Available: https://arxiv.org/abs/1810.04805
Pith/arXiv arXiv 2019
-
[31]
Roberta: A robustly optimized bert pretraining approach,
Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019. [Online]. Available: https://arxiv.org/abs/1907.11692
Pith/arXiv arXiv 2019
-
[32]
Palm: Scaling language modeling with pathways,
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y . Tay, N. Shazeer, V . Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev,...
Pith/arXiv arXiv 2022
-
[33]
Llama: Open and efficient founda- tion language models,
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient founda- tion language models,” arXiv preprint arXiv:2302.13971, 2023
Pith/arXiv arXiv 2023
-
[34]
OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V . Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brunda...
Pith/arXiv arXiv 2024
-
[35]
H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023. [Online]. Available: https: //arxiv.org/abs/2304.08485
Pith/arXiv arXiv 2023
-
[36]
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,
Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y . Qiao, and J. Dai, “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” 2024. [Online]. Available: https://arxiv.org/abs/2312.14238
Pith/arXiv arXiv 2024
-
[37]
Minicpm-v: A gpt-4v level mllm on your phone,
Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He, Q. Chen, H. Zhou, Z. Zou, H. Zhang, S. Hu, Z. Zheng, J. Zhou, J. Cai, X. Han, G. Zeng, D. Li, Z. Liu, and M. Sun, “Minicpm-v: A gpt-4v level mllm on your phone,” 2024. [Online]. Available: https://arxiv.org/abs/2408.01800
Pith/arXiv arXiv 2024
-
[38]
J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” 2023. [Online]. Available: https://arxiv.org/abs/2308.12966
Pith/arXiv arXiv 2023
-
[39]
A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al- Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, 10 A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang, B. Chern, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Kel...
Pith/arXiv arXiv 2024
-
[40]
Show 11 and tell: A neural image caption generator,
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show 11 and tell: A neural image caption generator,” in Proceed- ings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3156–3164
work page 2015
-
[41]
Deep visual-semantic align- ments for generating image descriptions,
A. Karpathy and L. Fei-Fei, “Deep visual-semantic align- ments for generating image descriptions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3128–3137
work page 2015
-
[42]
Vqa: Visual question answering,
S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh, “Vqa: Visual question answering,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2425–2433
work page 2015
-
[43]
Vse++: Improving visual-semantic embeddings with hard nega- tives,
F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler, “Vse++: Improving visual-semantic embeddings with hard nega- tives,” arXiv preprint arXiv:1707.05612 , 2017
Pith/arXiv arXiv 2017
-
[44]
Vilbert: Pre- training task-agnostic visiolinguistic representations for vision-and-language tasks,
J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pre- training task-agnostic visiolinguistic representations for vision-and-language tasks,” Advances in neural informa- tion processing systems , vol. 32, 2019
work page 2019
-
[45]
Z. Zhou, J. Zhang, Z. Guan, M. Hu, N. Lao, L. Mu, S. Li, and G. Mai, “Img2loc: Revisiting image geolocaliza- tion using multi-modality foundation models and image- based retrieval-augmented generation,” in Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , 2024, pp. 2749–2754
work page 2024
-
[46]
Large-scale image geolocal- ization,
J. Hays and A. A. Efros, “Large-scale image geolocal- ization,” Multimodal location estimation of videos and images, pp. 41–62, 2015
work page 2015
-
[47]
What makes paris look like paris?
C. Doersch, S. Singh, A. Gupta, J. Sivic, and A. A. Efros, “What makes paris look like paris?” Communications of the ACM, vol. 58, no. 12, pp. 103–110, 2015
work page 2015
-
[48]
Ge- olocation estimation of photos using a hierarchical model and scene classification,
E. Muller-Budack, K. Pustu-Iren, and R. Ewerth, “Ge- olocation estimation of photos using a hierarchical model and scene classification,” in Proceedings of the European conference on computer vision (ECCV) , 2018, pp. 563– 579
work page 2018
-
[49]
Cplanet: Enhancing image geolocalization by combinatorial par- titioning of maps,
P. H. Seo, T. Weyand, J. Sim, and B. Han, “Cplanet: Enhancing image geolocalization by combinatorial par- titioning of maps,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 536– 551
work page 2018
-
[50]
Where in the world is this image? transformer-based geo-localization in the wild,
S. Pramanick, E. M. Nowara, J. Gleason, C. D. Castillo, and R. Chellappa, “Where in the world is this image? transformer-based geo-localization in the wild,” in Euro- pean Conference on Computer Vision . Springer, 2022, pp. 196–215
work page 2022
-
[51]
B. Clark, A. Kerrigan, P. P. Kulkarni, V . V . Cepeda, and M. Shah, “Where we are and what we’re looking at: Query based worldwide image geo-localization using hierarchies and scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 23 182–23 190
work page 2023
-
[52]
Transgeo: Transformer is all you need for cross-view image geo-localization,
S. Zhu, M. Shah, and C. Chen, “Transgeo: Transformer is all you need for cross-view image geo-localization,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 1162–1171
work page 2022
-
[53]
Joint representation learning and keypoint detection for cross-view geo-localization,
J. Lin, Z. Zheng, Z. Zhong, Z. Luo, S. Li, Y . Yang, and N. Sebe, “Joint representation learning and keypoint detection for cross-view geo-localization,” IEEE Trans- actions on Image Processing , vol. 31, pp. 3780–3792, 2022
work page 2022
-
[54]
Cross-view geo-localization via learning disentangled geometric layout correspondence,
X. Zhang, X. Li, W. Sultani, Y . Zhou, and S. Wshah, “Cross-view geo-localization via learning disentangled geometric layout correspondence,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 3, 2023, pp. 3480–3488
work page 2023
-
[55]
The benchmarking initiative for multimedia evaluation: Mediaeval 2016,
M. Larson, M. Soleymani, G. Gravier, B. Ionescu, and G. J. Jones, “The benchmarking initiative for multimedia evaluation: Mediaeval 2016,” IEEE MultiMedia, vol. 24, no. 1, pp. 93–96, 2017
work page 2016
-
[56]
Inter- pretable semantic photo geolocation,
J. Theiner, E. M ¨uller-Budack, and R. Ewerth, “Inter- pretable semantic photo geolocation,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), January 2022, pp. 750–760
work page 2022
-
[57]
The faiss library,
M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazar ´e, M. Lomeli, L. Hosseini, and H. J ´egou, “The faiss library,” 2024
2024
-
[58]
Billion-scale similarity search with gpus,
J. Johnson, M. Douze, and H. J ´egou, “Billion-scale similarity search with gpus,” IEEE Transactions on Big Data, vol. 7, no. 3, pp. 535–547, 2019
work page 2019
-
[59]
Yfcc100m: The new data in multimedia research,
B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li, “Yfcc100m: The new data in multimedia research,” Communications of the ACM , vol. 59, no. 2, pp. 64–73, 2016
work page 2016
-
[60]
Geopositioning accuracy assess- ment of geoeye-1 panchromatic and multispectral im- agery,
M. A. Aguilar, F. J. Aguilar, M. d. Mar Salda ˜na, I. Fern ´andez et al. , “Geopositioning accuracy assess- ment of geoeye-1 panchromatic and multispectral im- agery,” Photogrammetric Engineering & Remote Sens- ing, vol. 78, no. 3, pp. 247–257, 2012
work page 2012
-
[61]
Y . Zhuang, Z. Syed, J. Georgy, and N. El-Sheimy, “Autonomous smartphone-based wifi positioning system by using access points localization and crowdsourcing,” Pervasive and mobile computing , vol. 18, pp. 118–136, 2015
work page 2015
-
[62]
J. Zhuang, M. Dai, X. Chen, and E. Zheng, “A faster and more effective cross-view matching method of uav and satellite images for uav geolocalization,” Remote Sensing, vol. 13, no. 19, 2021. [Online]. Available: https://www.mdpi.com/2072-4292/13/19/3979
work page 2021
-
[63]
Efficient localisation using images and open- streetmaps,
M. Zhou, X. Chen, N. Samano, C. Stachniss, and A. Calway, “Efficient localisation using images and open- streetmaps,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2021, pp. 5507–5513
work page 2021
-
[64]
Automatic discovery and geotagging of objects from street view imagery,
V . A. Krylov, E. Kenny, and R. Dahyot, “Automatic discovery and geotagging of objects from street view imagery,” Remote Sensing, vol. 10, no. 5, 2018. [Online]. Available: https://www.mdpi.com/2072-4292/10/5/661
work page 2018
-
[65]
Swin transformer: Hierarchical vi- sion transformer using shifted windows,
Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vi- sion transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 10 012–10 022
work page 2021
-
[66]
Pigeon: Predicting image geolocations,
L. Haas, M. Skreta, S. Alberti, and C. Finn, “Pigeon: Predicting image geolocations,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 12 Recognition (CVPR), June 2024, pp. 12 893–12 902
work page 2024
-
[67]
V . Vivanco Cepeda, G. K. Nayak, and M. Shah, “Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization,” Advances in Neural Information Processing Systems , vol. 36, 2024
work page 2024
-
[68]
G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V . Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt, “Openclip,” Jul. 2021, if you use this software, please cite it as below. [Online]. Available: https: //doi.org/10.5281/zenodo.5143773
-
[69]
Learning generalized zero-shot learners for open-domain image geolocaliza- tion,
L. Haas, S. Alberti, and M. Skreta, “Learning generalized zero-shot learners for open-domain image geolocaliza- tion,” 2023
work page 2023
-
[70]
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,
P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y . Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin, “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,” 2024. [Online]. Available: https://arxiv.org/abs/2409.12191
Pith/arXiv arXiv 2024
-
[71]
How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites,
Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma et al. , “How far are we to gpt-4v? closing the gap to commercial multi- modal models with open-source suites,” arXiv preprint arXiv:2404.16821, 2024
Pith/arXiv arXiv 2024
-
[72]
P. Agrawal, S. Antoniak, E. B. Hanna, B. Bout, D. Chaplot, J. Chudnovsky, D. Costa, B. D. Monicault, S. Garg, T. Gervet, S. Ghosh, A. H ´eliou, P. Jacob, A. Q. Jiang, K. Khandelwal, T. Lacroix, G. Lample, D. L. Casas, T. Lavril, T. L. Scao, A. Lo, W. Marshall, L. Martin, A. Mensch, P. Muddireddy, V . Nemychnikova, M. Pellat, P. V . Platen, N. Raghuraman, ...
Pith/arXiv arXiv 2024
-
[73]
Aria: An open multimodal native mixture-of-experts model,
D. Li, Y . Liu, H. Wu, Y . Wang, Z. Shen, B. Qu, X. Niu, G. Wang, B. Chen, and J. Li, “Aria: An open multimodal native mixture-of-experts model,” 2024. [Online]. Available: https://arxiv.org/abs/2410.05993
Pith/arXiv arXiv 2024
- [74]
-
[75]
A survey on evaluation of multimodal large language models,
J. Huang and J. Zhang, “A survey on evaluation of multimodal large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2408.15769
Pith/arXiv arXiv 2024
-
[76]
M3exam: A multilingual, multimodal, multilevel bench- mark for examining large language models,
W. Zhang, M. Aljunied, C. Gao, Y . K. Chia, and L. Bing, “M3exam: A multilingual, multimodal, multilevel bench- mark for examining large language models,” Advances in Neural Information Processing Systems , vol. 36, pp. 5484–5505, 2023
work page 2023
-
[77]
Lvlm-ehub: A comprehensive evaluation benchmark for large vision- language models,
P. Xu, W. Shao, K. Zhang, P. Gao, S. Liu, M. Lei, F. Meng, S. Huang, Y . Qiao, and P. Luo, “Lvlm-ehub: A comprehensive evaluation benchmark for large vision- language models,” IEEE Transactions on Pattern Analy- sis and Machine Intelligence , pp. 1–18, 2024
work page 2024
-
[78]
Seed-bench: Benchmarking multimodal large language models,
B. Li, Y . Ge, Y . Ge, G. Wang, R. Wang, R. Zhang, and Y . Shan, “Seed-bench: Benchmarking multimodal large language models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024, pp. 13 299–13 308
work page 2024
-
[79]
Awq: Activation-aware weight quantization for llm compression and acceleration,
J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quantization for llm compression and acceleration,” 2024. [Online]. Available: https://arxiv.org/abs/2306.00978
Pith/arXiv arXiv 2024
-
[80]
Gptq: Accurate post-training quantization for generative pre-trained transformers,
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Accurate post-training quantization for generative pre-trained transformers,” 2023. [Online]. Available: https://arxiv.org/abs/2210.17323
Pith/arXiv arXiv 2023
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.