REVIEW 4 major objections 4 minor 32 references
Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Map-grounded LLM captions for satellite imagery reduce hallucination and improve few-shot detection.
desk verdict Useful new dataset and map-grounded captioning idea, but the hallucination metric is circular—OSM serves as both prompt and ground truth—and the few-shot gains are small without significance tests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is a two-part pipeline: first, retrieving OpenStreetMap tiles via the Mapbox API from fMoW bounding-box coordinates and feeding them, along with the satellite image and metadata, into GPT-4o to generate captions; second, a false discovery rate (FDR) metric, computed as one minus the fraction of proper nouns in the generated caption that appear in the reference list of proper nouns on the corresponding OSM tile. The FDR metric supplies the measure by which the paper chooses map resolution, label configuration, and prompt count, while the map-conditioned generation supplies the captions whose semantic density the few-shot detection experiments use for continual pretraining of CLIP.
What would settle it
Take a set of fMoW images whose captions pass the FDR test, have human annotators mark each proper noun as visible in the satellite image or absent, and check whether a substantial share of map-referenced landmarks are not actually visible, or whether images that predate OSM label updates show captions copying outdated map names. Alternatively, scramble the map labels before prompting by swapping tiles with random other tiles and check whether FDR stays artificially low while captions become more inaccurate.
Extended reading notes
Core claim
The central claim is that grounding LLM caption generation in map context simultaneously reduces hallucination and improves downstream utility. For the fMoW-mm dataset, continually pretraining CLIP with these captions gives mAP50 of 0.1574, 0.1902, and 0.1972 at 5, 10, and 30 shots on DIOR, surpassing CLIP, OpenCLIP, GeoRSCLIP, and RemoteCLIP baselines despite fMoW-mm being far smaller than the datasets used for GeoRSCLIP and RemoteCLIP. Ablations show higher map resolution lowers the false discovery rate of proper nouns, landmarks-only labels strike the best balance between specificity and hallucination, and prompt ensembling does not help. The paper presents this as evidence that richer, map-grounded captions are more valuable than raw dataset size for scarce-label remote sensing tasks.
Load-bearing premise
The claim assumes OpenStreetMap tags are a complete and accurate reference for what a caption should mention, even though the same OSM tile is what GPT-4o is told to base its caption on; captions that simply repeat map labels can therefore look hallucination-free even if the labels are wrong, stale, or not visible in the image.
Editorial extensions
If this is right
- fMoW-mm, with 83,412 tuples of {satellite, map, metadata, caption}, provides a large public multimodal resource for remote sensing vision-language research.
- Map-grounded caption generation yields lower FDR and lower uncertain-word rates than unimodal captioning, pointing to external data augmentation as a hallucination mitigation strategy that does not require model access.
- Continual pretraining on fMoW-mm improves few-shot object detection on DIOR at 5, 10, and 30 shots relative to CLIP-based baselines, including much larger datasets, which suggests semantic density of captions matters more than raw dataset size.
- The FDR metric offers a simple, statistical, reference-based way to audit LLM captions in geospatial contexts where OSM provides a source of proper nouns.
Reading between the lines
- The FDR metric likely rewards parroting map labels: a caption that copies many OSM names verbatim will score well even if those objects are not visible in the imagery, so the reported hallucination rates are only as reliable as OSM's completeness and currency; a testable check would compare FDR scores against human-verified landmark visibility.
- Because fMoW imagery is historical (the sample is dated 2016) and OSM reflects current map state, stale or newly built landmarks may be systematically mislabeled in the captions; measuring FDR on a temporally matched OSM snapshot would separate this effect.
- The approach should transfer to any domain with authoritative geospatial or schematic side information, such as floor plans in indoor robotics, nautical charts, or cadastral maps, and the paper's ablations suggest the label style of the side information is the main tuning knob.
- A strict test of the 'semantic density' hypothesis would be to train on fMoW-mm captions with map labels stripped versus retained, holding image and model constant; the paper does not isolate the map input's contribution from the richer instruction prompt.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a method for generating vision-language captions for remote sensing imagery by prompting GPT-4o with satellite images, metadata, and OpenStreetMap tiles. It introduces fMoW-mm, a dataset of 83,412 image-map-metadata-caption tuples built on fMoW-rgb, and defines a false discovery rate (FDR) metric over proper nouns to quantify hallucinations. The authors report ablations on map resolution, map label configuration, and prompt ensembling, and evaluate the dataset by continually pretraining CLIP and testing few-shot object detection on DIOR, claiming superior performance over existing remote sensing vision-language datasets.
Significance. The dataset and pipeline are concrete and the authors make the data publicly available, which is a useful community resource. The idea of using maps as external context for caption generation is interesting and potentially practical. However, the two central claims—that map-grounded captions reduce hallucinations and that fMoW-mm improves few-shot detection—are not yet established: the FDR metric is unvalidated and references the same map tile shown to the model, and the reported detection margins over RemoteCLIP are tiny with no error bars or significance tests. If these issues were addressed with human evaluation and proper statistics, the contribution would be solid.
major comments (4)
- [Hallucination Metric / OSM Tile Retrieval] Eq. (1) defines FDR against a reference list R of proper nouns extracted from the same OSM tile that is provided as input to GPT-4o during caption generation (see 'OSM Tile Retrieval' and 'Caption Generation with GPT-4o'). A caption that copies map labels achieves low FDR even when those labels are not visible in the satellite image, while a visually accurate detail absent from OSM is counted as a hallucination. Since FDR is used to validate the method and to select map resolution, map type, and prompt count in Figure 4, this circularity undermines the central claim that map context reduces hallucination. The paper does not validate FDR against human judgment or an independent reference.
- [Table 2, Few-Shot Object Detection with CLIP] The reported margins over RemoteCLIP at 5-shot and 10-shot are 0.0003 and 0.0009, respectively (0.1574 vs 0.1571 and 0.1902 vs 0.1893). Although the text states that results are averaged over 5 splits, no variance, error bars, or significance tests are reported. The claim of 'superior performance compared to other vision-language remote sensing datasets' is therefore not statistically supported, particularly at the lower shot counts.
- [Few-Shot Object Detection with CLIP] The paper attributes the detection improvement to 'increased semantic density in the generated captions' but does not ablate the map input. Without a comparison to captions generated by GPT-4o from the same image and metadata but without the OSM tile (the unimodal LLM baseline illustrated in Figure 1), the improvement cannot be attributed to map grounding; it could instead be caused by text style, metadata injection, or dataset size. This missing baseline is load-bearing for the causal claim that map-grounded captions improve visual representations.
- [Hallucination Metric, Eq. (1)] FDR is computed only over proper nouns and treats OSM as a complete and correct reference. It does not capture false attributes, quantities, spatial relations, or other hallucination types, and it penalizes correct visual details that are absent from OSM. Because OSM completeness varies by region and fMoW has global coverage, the FDR values in Figure 4 may partly reflect regional OSM tag density rather than caption truthfulness.
minor comments (4)
- [Ablations, Figure 4] The 'uncertain word percentage' metric used in Figure 4 and the Ablations section is never defined; please specify how it is computed and which words are counted as uncertain.
- [Ablations] The Ablations section states that 'Prompt ensembling did not result in noticeable improvements' but then says 'For fMoW-mm, we aggregate responses from three prompts, yielding the lowest FDR.' Please reconcile these statements and clarify the selection criterion.
- [Few-Shot Object Detection with CLIP] The continual pretraining details are incomplete: the optimizer, learning rate, learning rate schedule, and any regularization are not reported, which limits reproducibility.
- [Multimodal Functional Map of the World (fMoW-mm)] The dataset link is a bit.ly URL; a persistent identifier such as a DOI or a stable repository link would be preferable.
Circularity Check
The hallucination metric is self-referential: OSM tiles are both the prompt input and the reference list for FDR, so low hallucination partially reduces to map-label copying; the few-shot detection benchmark is external and not circular.
-
self definitional
[Hallucination Metric, Eq. (1); Caption Generation with GPT-4o]
"To address the lack of suitable metrics, we propose a statistical metric inspired by BLEU precision that uses OSM tags as source text to measure hallucination rates. ... where the candidate list C = [c1, c2, ..., cK] is an array of K proper nouns, and the reference list R = [r1, r2, ..., rM ] is an array of M proper nouns. ... For each sample, we input the fMoW-rgb satellite image, metadata and OSM tile."
Eq. (1) defines FDR as 1 - (1/K) * sum_c 1_R(c), so a generated proper noun counts as correct exactly when it appears in the reference list R. The paper states that the metric 'uses OSM tags as source text' for R, and the same OSM tile is one of the inputs to GPT-4o when the caption is generated. Thus a caption that simply recites map labels achieves a low FDR by construction, while a true visual detail absent from OSM is counted as a hallucination. The reported 'hallucination rate' is therefore a measure of overlap between the generated text and the input map, not of factual correctness relative to the satellite image. The central claim that map context reduces hallucination is partly equivalent to this definition.
-
self definitional
[Experiments, Ablations: Map Types]
"Adding text labels, such as landmarks and street names, predictably increases hallucination rates. ... For the fMoW-mm dataset, we selected the landmarks-only configuration as it strikes a good balance, minimizing hallucinations while maintaining reasonable specificity."
The map-style ablation is scored using the same FDR metric whose reference list consists of OSM tags from the map shown to the model. Selecting the landmarks-only configuration 'to minimize hallucinations' therefore optimizes the caption's agreement with the map text, not its agreement with the satellite image. The design choice is an artifact of the self-referential metric: changing the map style changes both the prompt and the reference list, so the measured 'hallucination' differences partly reflect how much map text the model copies.
full rationale
The paper's few-shot object detection evaluation on DIOR is an external benchmark and is not circular: the mAP50 numbers compare visual backbones on a fixed task, independent of the FDR definition. However, the paper's first central claim, that integrating maps reduces hallucination in generated captions, rests on the FDR metric of Eq. (1). That metric defines a 'false positive' as a proper noun not present in R, and the paper explicitly describes R as built from OSM tags, while the same OSM tile is given to GPT-4o as input. Consequently, low FDR can be achieved by copying map labels, and high FDR can result from mentioning visually true details that OSM omits. The ablations and dataset-format choices (map resolution, map type, prompt ensembling) are all selected using this self-referential score, so the mitigation conclusions are partially forced by construction. The detection gains, though small and reported without error bars, are not circular because they are measured on an independent object detection benchmark. Overall, the central hallucination-reduction claim reduces in part to the definition of the metric, while the transfer-learning claim retains independent content; this is partial circularity rather than full equivalence.
Assumptions & free parameters
free parameters (4)
- Map resolution =
1024x1024
- Map label configuration =
Landmarks-only
- Prompt ensemble count =
3
- Continual pretraining schedule =
50 epochs, batch size 125
assumptions (4)
- domain assumption OSM tags are accurate and comprehensive as ground truth for landmarks.
- domain assumption Proper nouns in captions can be reliably extracted and matched to OSM tags.
- domain assumption The Mapbox tile retrieved from the fMoW bounding box aligns with the satellite image footprint.
- domain assumption Continual pretraining of CLIP on fMoW-mm captions transfers to DIOR few-shot object detection.
Cite this review
Pith. "Pith review of Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing." pith.science (2026). https://pith.science/paper/HZFBGQRI
@misc{pith2026250114905,
author = {Pith},
title = {Pith review of: Measuring and Mitigating Hallucinations in Vision-Language Dataset Generation for Remote Sensing},
year = {2026},
howpublished = {\url{https://pith.science/paper/HZFBGQRI}},
note = {Machine review of arXiv:2501.14905}
}
read the original abstract
Vision language models have achieved impressive results across various fields. However, adoption in remote sensing remains limited, largely due to the scarcity of paired image-text data. To bridge this gap, synthetic caption generation has gained interest, traditionally relying on rule-based methods that use metadata or bounding boxes. While these approaches provide some description, they often lack the depth needed to capture complex wide-area scenes. Large language models (LLMs) offer a promising alternative for generating more descriptive captions, yet they can produce generic outputs and are prone to hallucination. In this paper, we propose a new method to enhance vision-language datasets for remote sensing by integrating maps as external data sources, enabling the generation of detailed, context-rich captions. Additionally, we present methods to measure and mitigate hallucinations in LLM-generated text. We introduce fMoW-mm, a multimodal dataset incorporating satellite imagery, maps, metadata, and text annotations. We demonstrate its effectiveness for automatic target recognition in few-shot settings, achieving superior performance compared to other vision-language remote sensing datasets.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
Exploring Robust Features for Few-Shot Object Detection in Satellite Imagery
Bou, X.; Facciolo, G.; von Gioi, R. G.; Morel, J.-M.; and Ehret, T. 2024. Exploring Robust Features for Few-Shot Object Detection in Satellite Imagery. arXiv:2403.05381
work page Pith review arXiv 2024
-
[2]
Cherti, M.; Beaumont, R.; Wightman, R.; Wortsman, M.; Ilharco, G.; Gordon, C.; Schuhmann, C.; Schmidt, L.; and Jitsev, J. 2023. Reproducible Scaling Laws for Contrastive Language-Image Learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2818–2829. IEEE
work page 2023
-
[3]
Christie, G.; Fendley, N.; Wilson, J.; and Mukherjee, R. 2018. Functional Map of the World. arXiv:1711.07846
arXiv 2018
-
[4]
Deutsch, D.; Bedrax-Weiss, T.; and Roth, D. 2021. Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary. arXiv:2010.00490
work page Pith review arXiv 2021
-
[5]
Dhingra, B.; Faruqui, M.; Parikh, A.; Chang, M.-W.; Das, D.; and Cohen, W. W. 2019. Handling Divergent Reference Texts when Evaluating Table-to-Text Generation. arXiv:1906.01081
work page Pith review arXiv 2019
-
[6]
Du, Y.; Liu, Z.; Li, J.; and Zhao, W. X. 2022. A Survey of Vision-Language Pre-Trained Models. arXiv:2202.10936
arXiv 2022
-
[7]
Du s ek, O.; and Kasner, Z. 2020. Evaluating Semantic Accuracy of Data-to-Text Generation with Natural Language Inference. In Davis, B.; Graham, Y.; Kelleher, J.; and Sripada, Y., eds., Proceedings of the 13th International Conference on Natural Language Generation
work page 2020
-
[8]
Filippova, K. 2020. Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data. In Cohn, T.; He, Y.; and Liu, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2020
work page 2020
Show all 32 references
-
[9]
J.; Madotto, A.; and Fung, P
Ji, Z.; Lee, N.; Frieske, R.; Yu, T.; Su, D.; Xu, Y.; Ishii, E.; Bang, Y. J.; Madotto, A.; and Fung, P. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12): 1–38
2023
-
[10]
Khanna, S.; Liu, P.; Zhou, L.; Meng, C.; Rombach, R.; Burke, M.; Lobell, D.; and Ermon, S. 2024. DiffusionSat: A Generative Foundation Model for Satellite Imagery. arXiv:2312.03606
2024 arXiv
-
[11]
Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597
2023 arXiv
-
[12]
Li, K.; Wan, G.; Cheng, G.; Meng, L.; and Han, J. 2020. Object detection in optical remote sensing images: A survey and a new benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 159
2020
-
[13]
Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out
2004
-
[14]
Liu, F.; Chen, D.; Guan, Z.; Zhou, X.; Zhu, J.; Ye, Q.; Fu, L.; and Zhou, J. 2024. RemoteCLIP: A Vision Language Foundation Model for Remote Sensing. arXiv:2306.11029
2024 arXiv
-
[15]
Liu, T.; Zhang, Y.; Brockett, C.; Mao, Y.; Sui, Z.; Chen, W.; and Dolan, B. 2022. A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation. arXiv:2104.08704
2022 arXiv
-
[16]
Lobry, S.; Marcos, D.; Murray, J.; and Tuia, D. 2020. RSVQA: Visual Question Answering for Remote Sensing Data. IEEE Transactions on Geoscience and Remote Sensing
2020
-
[17]
C.; and Yang, H
Long, S.; Cao, F.; Han, S. C.; and Yang, H. 2022. Vision-and-Language Pretrained Models: A Survey. arXiv:2204.07356
2022 arXiv
-
[18]
Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311--318
2002
-
[19]
W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00020
2021 arXiv
-
[20]
S.; and Das, D
Rashkin, H.; Reitter, D.; Tomar, G. S.; and Das, D. 2021. Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features. arXiv:2107.06963
2021 arXiv
-
[21]
A.; Burns, K.; Darrell, T.; and Saenko, K
Rohrbach, A.; Hendricks, L. A.; Burns, K.; Darrell, T.; and Saenko, K. 2019. Object Hallucination in Image Captioning. arXiv:1809.02156
2019 arXiv
-
[22]
Singh, S. 2018. Natural Language Processing for Information Extraction. arXiv:1807.02383
2018 arXiv
-
[23]
Uc-Cetina, V.; Navarro-Guerrero, N.; Martin-Gonzalez, A.; Weber, C.; and Wermter, S. 2022. Survey on reinforcement learning for language processing. Artificial Intelligence Review, 56(2): 1543–1575
2022
-
[24]
Wang, P.; Lin, J.; Yang, A.; Zhou, C.; Zhang, Y.; Zhou, J.; and Yang, H. 2021. Sketch and Refine: Towards Faithful and Informative Table-to-Text Generation. In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021
2021
-
[25]
Wang, Z.; Prabha, R.; Huang, T.; Wu, J.; and Rajagopal, R. 2023. SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing. arXiv:2312.12856
2023 arXiv
-
[26]
Weng, R.; Yu, H.; Wei, X.; and Luo, W. 2020. Towards Enhancing Faithfulness for Neural Machine Translation. In Webber, B.; Cohn, T.; He, Y.; and Liu, Y., eds., Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2675--2684. Online: A...
2020
-
[27]
Wu, Z.; Galley, M.; Brockett, C.; Zhang, Y.; Gao, X.; Quirk, C.; Koncel-Kedziorski, R.; Gao, J.; Hajishirzi, H.; Ostendorf, M.; and Dolan, B. 2021. A Controllable Model of Grounded Response Generation. arXiv:2005.00613
2021 arXiv
-
[28]
Yuan, Z.; Xiong, Z.; Mou, L.; and Zhu, X. X. 2024. ChatEarthNet: A Global-Scale Image-Text Dataset Empowering Vision-Language Geo-Foundation Models. arXiv:2402.11325
2024 arXiv
-
[29]
Zhang, J.; Huang, J.; Jin, S.; and Lu, S. 2024 a . Vision-Language Models for Vision Tasks: A Survey. arXiv:2304.00685
2024 arXiv
-
[30]
Zhang, Z.; Zhao, T.; Guo, Y.; and Yin, J. 2024 b . RS5M and GeoRSCLIP: A Large-Scale Vision- Language Dataset and a Large Vision-Language Model for Remote Sensing. IEEE Transactions on Geoscience and Remote Sensing, 62: 1–23
2024
-
[31]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...
-
[32]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.