{"id":"9ce83746-a9aa-47dd-822e-f3be2b114d10","arxiv_id":"2506.03371","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fine-grained Korean street-view benchmark shows that adding captions with place names to photos lets vision-language models pinpoint locations at high rates, highlighting privacy exposure.","lead":"KoreaGEO Bench is a new benchmark of 1,080 Korean street-view images with social-media-style captions, used to test ten vision-language models on fine-grained geolocation. It finds that captions revealing place names sharply boost localization accuracy, exposing a concrete privacy risk in everyday photo posts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'modality beats scale' claim is confounded: high-risk captions are generated to include the exact place name, so the large accuracy gain may reflect answer leakage rather than multimodal inference.","rationale":"The reader correctly identified the high-risk caption leakage as an issue in the rationale ('the 'modality beats scale' conclusion is partly an artifact of the high-risk captions containing the answer'), but their formally stated weakest assumption concerned dataset filtering and exact reproducibility. In my judgment, the more load-bearing concern is the answer-leakage confound because it directly undermines the paper's headline empirical conclusion. The benchmark itself and the descriptive accuracy tables retain value: the image-only results, the functional-caption comparison, the spatial bias analysis (confusion toward Seoul), and the privacy-risk observation that captions with explicit place names amplify localization are all useful and likely robust. However, the strong causal statement 'input modality has a greater influence than model scale or architecture' cannot be drawn from a design where one modality path supplies the ground-truth answer. The concrete test proposed would settle the issue: if masking the place names collapses the effect, the claim must be reworded to 'explicit location text in captions amplifies localization precision,' which is a different and more modest statement. Since the reader already issued a conditional verdict, and the fix is to reanalyze and soften the claim rather than to reject the resource, the verdict remains CONDITIONAL (unchanged in category). The rationale in the paper's Limitations section does not flag this confound, so it is an omitted check that should be made explicit.","tokens_in":17649,"tokens_out":2632,"duration_ms":31182,"concrete_test":"Run the high-risk caption condition with the place name masked or replaced (e.g., replacing the proper noun with 'OO' or a different city's name) while preserving the caption's style and emotional tone. If 1km accuracy drops from the reported ~46% to near the functional-caption level (~5-16%), the effect is driven by answer leakage, not by modality. As a second decisive check, run a text-only condition with the original high-risk caption and no image; if text-only accuracy matches the image+text accuracy, the image contributes nothing and 'modality' language is misleading. Both checks directly test whether the strong claim in Section 5.1.1 survives when information content is held constant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that 'input modality has a greater influence on localization performance and privacy exposure than model scale or architecture' (Section 5.1.1 and Conclusion). This claim rests on the large accuracy jump in the 'Visual with High-risk Caption' condition. However, the high-risk caption generation prompt (Figure 13) explicitly requires that 'proper nouns must be naturally included in the caption' — i.e., the ground-truth place name or address is inserted into the text. The model therefore receives the answer as text and the task reduces to text-based geocoding of a known location name. The image becomes largely superfluous, and the condition is not a fair test of 'modality' at constant information content. The appropriate modality comparison is the functional-caption path, which prohibits proper nouns; there the improvements over image-only are modest and inconsistent ('others show inconsistent or even degraded results'). Thus the conclusion that modality dominates model scale/architecture is not supported by the experimental design. The privacy-risk observation that captions revealing place names amplify localization is plausible and separately valuable, but attributing this to 'input modality' conflates answer leakage with multimodal reasoning. The paper does not report a text-only baseline for high-risk captions, so it cannot distinguish leakage from genuine visual-text integration.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces KoreaGEO Bench, a Korean street-view geolocation benchmark of 1,080 images sampled across four socio-spatial clusters and nine place types, with manual contextual annotations and two styles of Korean social-media-style captions (functional and high-risk). Ten vision-language models are evaluated under three input conditions—image only, image with functional caption, and image with high-risk caption—using distance-threshold accuracy, cluster/city confusion matrices, and reasoning-output analysis. The authors report that input modality has a greater effect on localization performance and privacy exposure than model scale or architecture, and that predictions are biased toward Seoul and the core cluster.","tokens_in":17938,"tokens_out":4706,"duration_ms":52120,"significance":"If the central claims were supported, KoreaGEO Bench would fill a clear gap: existing geolocation benchmarks are coarse-grained, English-centric, and rarely consider multimodal social-media captions or fine-grained intra-country bias. The dataset construction is thoughtful in its cluster stratification, place-type taxonomy, and contextual annotation, and the three-path evaluation protocol is a useful template for studying privacy leakage. The bias analysis via confusion matrices is a concrete contribution. However, the paper's headline claim about modality dominance rests on an experimental condition that leaks the ground-truth location through the caption, and the dataset filtering procedure is under-specified. These issues are fixable, but they currently prevent the results from supporting the stated conclusion.","major_comments":[{"comment":"The central claim that input modality has a greater influence on localization and privacy exposure than model scale or architecture is not supported by the experimental design. The high-risk caption prompt (Figure 13) explicitly instructs the generator to include the place name or address in the caption, so the VLM receives the ground-truth location as text; the 'Visual with High-risk Caption' condition therefore measures text-based geocoding of a revealed proper noun rather than multimodal inference at constant information content. No text-only baseline (high-risk caption without the image) is reported, so the large accuracy jumps in Table 2 (e.g., o3 from 9.91% to 46.48% at 1km and Gemini from 15.93% to 56.48% at 1km) cannot be attributed to the visual modality or to visual-textual fusion. The functional-caption condition, which removes proper nouns, shows only modest and inconsistent gains—for instance, GPT-4o's 1km accuracy drops from 7.59% to 5.83%—which is consistent with answer leakage rather than a general modality advantage. The authors should add a text-only high-risk-caption baseline and/or a fourth condition in which proper nouns are redacted from high-risk captions, and should re-state the conclusion in terms of privacy leakage from explicit textual cues rather than 'input modality'.","section":"§5.1.1, Fig. 13, Table 2"},{"comment":"The filtering from 7,200 collected images to 1,080 retained samples is described only as removing 'low-quality or semantically irrelevant samples' and applying 'the spatial and contextual coverage strategy,' with no operational definition of these criteria. If the filtering is non-random, the resulting benchmark may not represent the population of Korean street scenes, which undermines the generalizability of both the accuracy figures and the spatial-bias analysis. The authors should specify the exclusion criteria, report exclusion counts by cluster and place type, provide inter-annotator agreement for the quality filtering, and document the coverage-matching algorithm used to retain 30 images per place type per cluster.","section":"§3.4"},{"comment":"Because the released dataset contains only coordinates and the paper states that no raw images are stored, the benchmark's reproducibility depends on the Google Street View API returning the same imagery at those coordinates at evaluation time. Street View imagery is updated over time, so the exact images used for the reported results may not be recoverable. The authors should either release image hashes or thumbnails if the terms of use permit, document capture dates, or provide an archiving mechanism; otherwise the benchmark should be treated as a dynamic rather than static resource, and the reported numbers may not be exactly reproducible.","section":"§3.2, Ethics Statement"},{"comment":"The claim that modality is more influential than model scale or architecture is a comparative ranking claim, but the paper reports no statistical significance tests, confidence intervals, or repeated runs. With 1,080 images, differences of a few percentage points at the coarser thresholds (e.g., 65.00% vs. 63.52% at 100km for GPT-4o vs. o3 under image-only input) may be within sampling error. The 'greater influence' conclusion should be supported by paired significance tests or effect-size estimates across the ten models.","section":"§5.1.1, Table 2"}],"minor_comments":[{"comment":"The phrase 'we employe both the elbow method' contains a typo; it should read 'we employ both the elbow method.'","section":"Appendix A.1"},{"comment":"The claim that KoreaGEO Bench is 'the first fine-grained and multimodal geolocation benchmark specifically designed for a single country' should be qualified, because existing city-level benchmarks such as SF-XL already provide fine-grained single-city evaluation; the claim should be narrowed to Korea or to the specific combination of fine-grained granularity, multimodal captions, and a single-country scope.","section":"§1"},{"comment":"The modality-attribution analysis uses Gemini as an external judge; while the self-preference mitigation is cited, the analysis would be more robust with a second judge or with human-annotated modality labels on a sample of outputs.","section":"§5.3.2"},{"comment":"The model name 'GPT-o3' appears in Section 3.3 but the model is elsewhere called 'o3'; the naming should be made consistent throughout the paper.","section":"§3.3"},{"comment":"The modality attribution is reported only for the functional-caption setting; reporting the analogous attribution for high-risk captions would help clarify whether the large accuracy gains are actually driven by visual-textual integration or by text-only location lookup.","section":"Fig. 9"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern is well-founded: the 'modality beats scale' result depends on a high-risk caption condition that explicitly reveals the ground-truth place name, and without a text-only baseline the headline claim cannot stand. The dataset and the bias analysis are potentially valuable, but I would not accept the paper in its current form. The authors should be asked to add the missing baseline, tighten the filtering documentation, and re-frame the claims around privacy leakage from explicit textual cues. I would also ask the handling editor to ensure that the Google Street View terms-of-use and data-release claims are checked carefully, since the paper simultaneously states that no raw images are released and that the benchmark consists of 1,080 images accessible via coordinates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the resource is real and worth engaging; the central interpretive claim is not supported as stated. KoreaGEO Bench is a first: a fine-grained, single-country multimodal geolocation benchmark with 1,080 Korean street-view images stratified over four socio-spatial clusters and nine place types, plus two caption styles and rich context annotations. That fills a genuine gap. The sampling design (GRDP/density clustering, place-type taxonomy, contextual labels) is thoughtful, and the Seoul-centric bias analysis via confusion matrices is a useful diagnostic. Running ten VLMs across three input paths is a reasonable evaluation protocol. I would not desk-reject this.\n\nThe big soft spot is the 'input modality beats model scale/architecture' conclusion. In the high-risk caption condition, the generation prompt (Figure 13) explicitly requires the place name to appear in the caption. So the model is handed the answer in text; the image is almost superfluous. That condition is answer leakage, not multimodal inference. The appropriate contrast is the functional-caption path, where proper nouns are banned, and there the gains over image-only are modest and inconsistent across models. The paper does not report a text-only baseline for high-risk captions, so it cannot separate leakage from genuine visual-text integration. The privacy-risk observation itself—captions revealing place names make localization dramatically easier—is plausible and valuable, but that is a statement about text leakage, not about modality dominating scale.\n\nOther soft spots, in descending order: the filtering from 7,200 to 1,080 images is described only as 'low-quality or semantically irrelevant' (Section 3.4), so representativeness is not established; no raw images are released and the exact API parameters (heading, pitch, date, zoom) are missing, so exact reproducibility is not currently possible; accuracy numbers come without confidence intervals or repeated-run variance, which matters for small per-cell samples. The human caption evaluation covers only 120 captions, but that is a peripheral validation issue, not a load-bearing flaw. The limitations and ethics statements are honest and should be taken at face value.\n\nNet: cite it as a dataset/resource, but not for the 'modality beats scale' claim until the authors add a no-leakage control or a text-only high-risk baseline and release code. It deserves a serious referee. I would send it out with a request for major revision.","headline":"A genuinely new Korean street-view geolocation benchmark, but the headline claim that modality beats scale rests on a leakage confound: high-risk captions contain the answer.","tokens_in":18419,"tokens_out":2161,"would_cite":true,"duration_ms":23512,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"KoreaGEO Bench: input modality — especially a place-naming caption — drives VLM geolocation accuracy and privacy leakage more than model scale or architecture, with prediction errors converging on Seoul.","keywords":["KoreaGEO Bench","geolocation benchmark","Korean street views","vision-language models","location privacy","multimodal evaluation","spatial bias","privacy exposure"],"falsifier":"Re-run the three-path protocol on freshly fetched Street View images from the released coordinates with independently regenerated captions; if the high-risk-caption lift at 1 km does not reproduce at roughly the reported magnitudes (o3: 9.91% to 46.48%; Gemini: 15.93% to 56.48%), the modality-dominance claim fails.","tokens_in":17358,"feed_emoji":"📍","tokens_out":10371,"duration_ms":101707,"temperature":0.7,"pith_summary":"KoreaGEO Bench is a new benchmark for asking whether vision-language models can pinpoint where a Korean street photo was taken, and what leaks that ability. The paper collects 1,080 street-view images across four urban clusters and nine place types, attaches two styles of Korean social-media captions, and runs ten models through image-only, caption-without-place-name, and caption-with-place-name conditions. Its central claim is that input modality decides geolocation accuracy and privacy exposure more than model scale or architecture: once a caption names the place, even small or open-source models localize precisely, while images alone are weak and vague captions can mislead. The authors also document a structural spatial bias: mispredictions flow disproportionately into Seoul, so accuracy declines along the urban hierarchy. If the claim is right, location-privacy defense should center on text redaction, and fine-grained benchmarks are needed to expose spatial bias.","feed_headline":"Captions, not model size, decide how precisely VLMs geolocate","feed_subtitle":"On a 1,080-image Korean street benchmark, input modality outweighs model scale, and errors converge on Seoul.","key_machinery":"The load-bearing object is the KoreaGEO Bench dataset itself: 1,080 Google Street View images, sampled from four KMeans-derived urban clusters and nine place types, with eleven manually annotated contextual dimensions and two Korean caption styles — functional captions that suppress place names and high-risk captions that include them. The evaluation protocol runs each image through three paths — image-only, image plus functional caption, and image plus high-risk caption — and scores predictions by Haversine distance at 0.1 km, 1 km, 20 km, and 100 km thresholds. The mechanism behind the central result is the contrast between these paths: a caption containing a place name supplies the model with explicit location tokens, overcoming visual ambiguity, while a vague caption without proper nouns can mislead the model into matching irrelevant cues.","core_discovery":"The paper's discovery is a modality-driven shift in localization precision. On image-only input, the best model reaches 3.52% at 100 m and 15.93% at 1 km (Gemini 2.5 Pro), and open-source models are near zero at fine scales. Functional captions — natural Korean captions without proper place names — do not reliably help and sometimes reduce accuracy. High-risk captions that name the place change the picture: the three leading closed models jump to roughly 46–56% at 1 km and above 95% at 20 km, and open-source LLaMA reaches 95% at 100 km while Qwen reaches 67% at 20 km. Because the same pattern appears across architectures and scales, the paper concludes that modality sensitivity matters more than model size or provider. A second finding is spatial: accuracy is highest in the Seoul cluster, confusion matrices show errors from growth hubs and satellites flowing into Seoul, and city-level mistakes converge on Seoul.","pith_inferences":["Editorial extension: the modality-dominance result suggests current VLM geolocation is partly text-reading; cropping or masking storefront text and signs in the images should lower image-only accuracy while leaving caption-driven accuracy unchanged, a direct test of that reading.","Editorial extension: Seoul-convergence bias may be a general property of monocentric countries; replicating the protocol in another single-dominant-city country would show whether the centralization pattern is structural rather than Korean-specific.","Editorial extension: because the release contains coordinates rather than images, scores have a shelf life — Street View imagery updates over time, so future re-runs may measure different scenes; pinning image hashes or capture timestamps would make the benchmark repeatable."],"forward_implications":["Location-bearing captions are a concrete geolocation vulnerability: at the 1 km threshold, high-risk captions lift leading models to roughly 46–56% accuracy, compared with under 16% for image-only input.","Privacy protections that strip EXIF metadata or blur faces are insufficient; text that names a place is itself a high-precision leak channel.","Coarse country- or city-level scores hide the reliability gap: at 100 m most models are near zero, and only landmarks and a few functional zones are reliably identifiable.","Model rankings depend on input modality, so an image-only leaderboard does not predict privacy risk once social-media captions are present.","Spatial-bias reporting, not just aggregate accuracy, is needed because errors systematically converge on the capital cluster."],"supporting_citations":[{"why":"Supplies the prior finding that combining text with images amplifies location leakage, which the paper's risk framing builds on.","marker":"(Tömekçe et al., 2024)"},{"why":"Provides the Google Street View global geolocation benchmark that the paper contrasts as coarse-grained and Western-focused.","marker":"(Jay et al., 2025)"},{"why":"A privacy-aware geolocation dialogue benchmark lacking real social-media-style captions, motivating the multimodal caption design.","marker":"(Mendes et al., 2024)"},{"why":"GeoDE contributes the geographically diverse place-type taxonomy used in the scene-validation stage.","marker":"(Ramaswamy et al., 2023)"},{"why":"PlaNet anchors the coarse-grained geolocation baseline and the high-level place-type scheme for street-view imagery.","marker":"(Weyand et al., 2016)"},{"why":"Supplies the method for mitigating self-preference bias when Gemini judges modality attribution.","marker":"(Panickssery et al., 2024)"}],"fun_headline_variants":["Modality, not model size, drives VLM geolocation on Korean streets","Korean street benchmark shows captions beat scale for VLM geolocation","Input type shifts VLM precision; Seoul dominates spatial errors","Captions outweigh model size in Korean street-view geolocation","VLM geolocation on Korean streets: modality matters, Seoul skews"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity depends on the retained 1,080 images being a representative sample of Korean street scenes after the 7,200-to-1,080 filter, and on the released coordinates allowing anyone to recover exactly the same Street View images.","fun_headline_variants_meta":{"raw":{"variants":["Modality, not model size, drives VLM geolocation on Korean streets","Korean street benchmark shows captions beat scale for VLM geolocation","Input type shifts VLM precision; Seoul dominates spatial errors","Captions outweigh model size in Korean street-view geolocation","VLM geolocation on Korean streets: modality matters, Seoul skews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00012,"raw_usage":{"total_tokens":1067,"prompt_tokens":900,"completion_tokens":167,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":73}},"tokens_in":516,"tokens_out":167,"duration_ms":2704,"temperature":1.0,"reasoning_tokens":73,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:04:01.770731+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the three-path protocol on freshly fetched Street View images from the released coordinates with independently regenerated captions; if the high-risk-caption lift at 1 km does not reproduce at roughly the reported magnitudes (o3: 9.91% to 46.48%; Gemini: 15.93% to 56.48%), the modality-dominance claim fails.","supporting_citations":[],"review_version":1}