REVIEW 3 major objections 4 minor 46 references
Understand Before Detect: Vision--Language Learning for Omni-Domain Infrared Small Target Detection
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that infrared small-target detection should begin with language-supervised scene understanding, and that its two-stage vision-language detector, JinSight, raises WideIRSTD IoU by over 14% relative to prior methods.
desk verdict The OmniIRST-VL dataset and JinSight framework are real contributions, but the headline gains are mostly backbone scale; the language-supervision effect is modest and cross-dataset claims lack a same-capacity control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the two-stage "understand before detect" pipeline. In Stage I, the paper takes the InternVL2.5-1B generative vision-language model and fine-tunes its ViT, projector, and language model on the OmniIRST-VL instruction mixture with the autoregressive objective $\mathcal{L}_{\text{inst}} = -\sum_{t=1}^{|y|} \log p(y_t \mid I, q, y_{<t})$, then keeps the instruction-tuned ViT as an IRST-aware backbone. In Stage II, the language projector and decoder are dropped, and Latent Semantic Interaction (LSI) modules are interleaved with the ViT blocks: each LSI block runs bidirectional cross-attention in a latent space of dimension $r \ll D$, transferring local spatial details from multi-scale features into ViT tokens and propagating global semantics back, with a low-rank FFN refining the latent tokens. OmniIRST-VL supplies 39,701 instruction-response pairs across six tasks (detailed captioning, physical VQA, image- and region-level counting, image- and region-level grounding), organized into scene-oriented and target-centric groups. The design's purpose is to make the encoder retain scene context and target-context relations that binary masks throw away, so that downstream dense prediction sees domain-invariant semantics.
What would settle it
Train the same InternVL ViT, initialized from the same pretrained weights, directly into the same dense-prediction decoder on the same training split without Stage I language tuning and with a matching training budget; if its IoU on WideIRSTD approaches or exceeds 42.94, the paradigm's claimed contribution is not supported.
Extended reading notes
Core claim
The paper's central claim is that generative language supervision, applied before any dense-prediction training, reshapes an infrared visual encoder so that it carries domain-invariant semantics rather than domain-specific low-level statistics. Under this claim, the autoregressive objective forces the vision transformer to keep spatial and semantic evidence about few-pixel targets, while language ties heterogeneous imaging conditions (NIR/SWIR/LWIR, land/aerial/space platforms) to a shared vocabulary; afterwards, the same backbone can be dropped into a standard dense-prediction head and outperform task-specific detectors. On the WideIRSTD benchmark the paper reports an IoU of 42.94 against 36.62 for the strongest visual baseline, Text-IRSTD, a relative improvement it summarizes as over 14%, and on leave-one-dataset-out tests it reports the highest detection probability and IoU on each of three held-out datasets.
Load-bearing premise
The load-bearing premise is that language-guided understanding, not the much larger pretrained backbone, is what lets a single detector generalize across infrared domains; the paper reports no same-capacity visual-only control.
Editorial extensions
If this is right
- A single JinSight model can be deployed across land, aerial, and space imaging platforms and across NIR, SWIR, and LWIR bands without per-domain retraining.
- Text is used only as training-time supervision: at inference the language branch is removed, and detection runs on infrared images alone.
- The low-rank interaction is also an efficiency win: in the paper's latent-dimension ablation, setting $r=128$ roughly halves latency relative to the no-LSI baseline while raising IoU.
- Both instruction groups contribute to the gain: removing either scene-oriented or target-centric instructions lowers IoU and detection probability, supporting the claim that holistic understanding, not just localization text, drives the improvement.
- OmniIRST-VL provides a shared evaluation ground for large vision-language models on infrared understanding, with a four-option single-choice format for discriminative tracks and open-ended caption generation.
Reading between the lines
- Because the encoder is grounded in language, a natural extension is zero-shot target discovery, where a user describes a new target type in text and the model segments it without mask supervision for that type.
- The same two-stage recipe could transfer to other sparse-object settings where scene context matters, such as tiny lesions in medical images or small vessels in satellite imagery.
- The four-option benchmark format could be complemented by open-ended evaluation of the same grounding and counting questions, since a model may score well on option selection while failing to generate a correct coordinate or count.
- The task ordering in OmniIRST-VL suggests a curriculum variant: train on scene-oriented tasks first (captioning, physical VQA), then on target-centric tasks (counting, grounding); the paper trains all six tasks as one shuffled mixture.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. JinSight is a two-stage framework for omni-domain infrared small target detection. Stage I instruction-tunes a generative VLM (InternVL2.5-1B) on OmniIRST-VL, a new dataset of 39.7k image-instruction-response pairs derived from WideIRSTD, covering captioning, physical VQA, counting, and grounding. Stage II transfers the tuned ViT to dense prediction via UperNet and a low-rank Latent Semantic Interaction (LSI) module that exchanges global semantics with multi-scale spatial features. The paper claims consistent state-of-the-art detection performance, including over 14% IoU improvement on WideIRSTD, and attributes the gains to language supervision that learns domain-invariant semantics.
Significance. The OmniIRST-VL dataset is a substantial new resource: it spans six instruction tasks, multiple platforms and spectral bands, includes expert verification, and will be released with code and weights. The internal ablation in Table 4 is well designed and shows that instruction tuning and LSI add consistent, if modest, gains over a same-backbone visual baseline. The leave-one-dataset-out evaluation protocol is a good methodological choice. However, the paper's central claim of consistent state-of-the-art performance is not yet established because the main comparisons in Table 2 are confounded by model capacity: the same backbone without any language supervision already beats all published baselines. The cross-dataset generalization claim lacks a same-capacity visual-only control. If the authors can address this capacity confound, the work would be a solid contribution to the IRST community.
major comments (3)
- [§5.2 Table 2; §5.3 Table 4] The main comparison is confounded by model capacity. JinSight uses a 314M-parameter InternVL ViT backbone, whereas the best visual baseline (Text-IRSTD) has 181M parameters and most baselines have under 15M. Table 4's 'w/o IVIT, w/o LSI' row shows that the same 314M ViT plus UperNet, with no language supervision and no LSI, already reaches 40.72 IoU and 70.28 Pd on WideIRSTD, exceeding every method in Table 2 (best SOTA IoU: Text-IRSTD at 36.62). The full JinSight reaches 42.94 IoU, so the gain attributable to the proposed paradigm is about 2.2 IoU (5.5% relative), not the 'over 14%' claimed in the abstract. For the three cross-dataset generalization tasks in Table 2, no same-capacity visual-only control is reported, so the claim that JinSight 'consistently transfers across heterogeneous domains' is not established against the most plausible alternative explanation. Please add a visual-only baseline using the same InternVL ViT backbone (with and without LSI) on all four evaluation settings.
- [§9.1 vs §5.3 Table 5] The implementation details state that the LSI latent dimension is set to r=128, but Table 5 reports r=256 as giving the highest IoU (42.16 vs 41.79 for r=128). If the main results use r=128, the paper should justify that choice explicitly; if r=256 is optimal, the main experimental results in Table 2 should be updated accordingly. This inconsistency matters for reproducibility and for the reported SOTA numbers.
- [Table 3] The comparison of JinSight-1B with generic VLMs is not a controlled test of the 'understand before detect' paradigm. JinSight-1B is fine-tuned on OmniIRST-Train, while InternVL2.5-1B, Qwen3-VL-2B, and LLaVA-NeXT-7B are evaluated zero-shot; the improvements could simply reflect in-distribution training. A same-initialization baseline fine-tuned on generic visual instruction data (or on OmniIRST-VL with a subset of instructions) would be needed to attribute the gains to the proposed instruction set.
minor comments (4)
- [Abstract and Conclusion] The phrase 'over 14% IoU improvement' is ambiguous (relative vs absolute) and is misleading in light of the capacity confound; please state the baseline and the relative/absolute interpretation explicitly.
- [Figure 3] Figure 3 contains typographical errors ('Descirbe' and the malformed closing tag '<\point>' in the example); please proofread the figure text.
- [Table 5] The memory usage with LSI (e.g., 9.43 GB at r=64) is lower than without LSI (16.92 GB), which is counterintuitive since LSI adds parameters; please explain this result.
- [References] The reference for EarthDial (Sagar Soni et al., 2025) has a garbled author list; please correct the citation.
Circularity Check
No circularity found: the method is evaluated on external held-out benchmarks, the ablations are internal, and the central claims do not reduce to the training inputs by construction.
full rationale
I walked the paper's derivation chain. Stage I optimizes the autoregressive objective in Eq. (1) on OmniIRST-VL, and Stage II's LSI is the bidirectional cross-attention fusion in Eqs. (2)-(3); neither defines the reported IoU or Pd in terms of itself. The headline WideIRSTD result and the cross-dataset results are obtained against external baselines retrained on the same split, with held-out test images, and the cross-dataset test sets are explicitly excluded from both stages of training. Table 4's 'w/o IVIT, w/o LSI' row is an internal baseline, not a circular fit. The OmniIRST-VL dataset is constructed from WideIRSTD, but the benchmark split is image-disjoint from the training split, and the language tasks are not the quantities being predicted in the detection claims. Self-citations, such as DNANet, appear only as baseline comparisons and are not load-bearing premises. The capacity confound raised by the skeptic is a legitimate external-validity and attribution concern, but it is not a case where a prediction is equivalent to its input by definition or where a fitted parameter is renamed as a prediction. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (2)
- LSI latent dimension r =
256 in main text (optimal in Table 5), 128 in supplementary implementation details
- Number of instruction-tuning epochs =
3
assumptions (4)
- domain assumption Language instruction tuning on the same training images used for detection provides transferable semantics that improve dense prediction.
- domain assumption The ViT backbone retains sufficient spatial detail for few-pixel target localization after instruction tuning.
- domain assumption Metadata-derived language annotations (band, platform, scene) are correct and learnable from image content.
- domain assumption The multiple-choice benchmark format measures understanding rather than exploitable distractor patterns.
Cite this review
Pith. "Pith review of Understand Before Detect: Vision--Language Learning for Omni-Domain Infrared Small Target Detection." pith.science (2026). https://pith.science/paper/VIJZUD5O
@misc{pith2026260807015,
author = {Pith},
title = {Pith review of: Understand Before Detect: Vision--Language Learning for Omni-Domain Infrared Small Target Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/VIJZUD5O}},
note = {Machine review of arXiv:2608.07015}
}
read the original abstract
Omni-domain infrared small target (IRST) detection is crucial for infrared surveillance, yet remains challenging due to heterogeneous imaging domains and inconsistent target characteristics. Previous deep learning-based methods have been developed for visual-only paradigms and achieved promising performance on domain-specific tasks. However, existing methods follow the task-specific supervised learning paradigm. This paradigm simplifies the full-scene infrared observations to sparse target supervision, discarding the semantics that remain invariant across heterogeneous domains. Consequently, detection performance suffers substantially under domain shifts. To handle this issue, we introduce \textbf{``understand before detect''}, a paradigm that formulates omni-domain IRST detection as an understanding-driven process, where holistic infrared target understanding precedes precise detection. Building on this paradigm, we propose \textbf{JinSight}, which first develops holistic IRST understanding through language supervision and then transfers the learned cross-domain representations to precise small-target detection. By grounding infrared representations in language semantics, JinSight enables a single model to generalize across heterogeneous infrared domains. We then introduce Latent Semantic Interaction (LSI), which exchanges language-aligned global semantics with fine-grained spatial features in a compact low-rank space. To address the lack of multimodal omni-domain IRST benchmarks, we build \textbf{OmniIRST-VL}, the first large-scale, highly diverse vision--language dataset for omni-domain IRST detection. It comprises over 39k annotations across six complementary instruction tasks covering both scene-level understanding and target-centric reasoning.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Aibibu, T.; Lan, J.; Zeng, Y.; Lu, W.; and Gu, N. 2024. Feature-enhanced attention and dual-gelan net (feadg-net) for uav infrared small object detection in traffic surveillance. Drones, 8(7): 304
work page 2024
-
[2]
An, X.; Sun, J.; Gui, Z.; and He, W. 2025. Choice: benchmarking the remote sensing capabilities of large vision-language models. In NeurIPS, volume 38
work page 2025
-
[3]
Bai, S.; Cai, Y.; Chen, R.; Chen, K.; Chen, X.; Cheng, Z.; Deng, L.; Ding, W.; Gao, C.; Ge, C.; et al. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631
arXiv 2025
-
[4]
Chen, S.; Ji, L.; Peng, S.; Zhu, S.; Ye, M.; and Sang, Y. 2025. Language-driven motion prior knowledge learning for moving infrared small target detection. IEEE Transactions on Geoscience and Remote Sensing
work page 2025
-
[5]
Chen, T.; Ye, Z.; Tan, Z.; Gong, T.; Wu, Y.; Chu, Q.; Liu, B.; Yu, N.; and Ye, J. 2024 a . MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small-Target Detection. IEEE Transactions on Geoscience and Remote Sensing, 62: 1--13
work page 2024
-
[6]
Chen, Z.; Duan, Y.; Wang, W.; He, J.; Lu, T.; Dai, J.; and Qiao, Y. 2022. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.08534
arXiv 2022
-
[7]
Chen, Z.; Wu, J.; Wang, W.; Su, W.; Chen, G.; Xing, S.; Zhong, M.; Zhang, Q.; Zhu, X.; Lu, L.; et al. 2024 b . InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In CVPR
work page 2024
-
[8]
Chung, W. Y.; Lee, I. H.; and Park, C. G. 2023. Lightweight infrared small target detection network using full-scale skip connection U-Net. IEEE Geoscience and Remote Sensing Letters, 20: 1--5
work page 2023
Show all 46 references
-
[9]
Dai, Y.; Wu, Y.; Zhou, F.; and Barnard, K. 2021 a . Asymmetric contextual modulation for infrared small target detection. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 950--959
2021
-
[10]
Dai, Y.; Wu, Y.; Zhou, F.; and Barnard, K. 2021 b . Attentional local contrast networks for infrared small target detection. IEEE Transactions on Geoscience and Remote Sensing, 59(11): 9813--9824
2021
-
[11]
A.; Shah, S
Danish, M.; Munir, M. A.; Shah, S. R. A.; Kuckreja, K.; Khan, F. S.; Fraccaro, P.; Lacoste, A.; and Khan, S. 2025. Geobench-vlm: Benchmarking vision-language models for geospatial tasks. In ICCV, 7132--7142
2025
-
[12]
I.; Waldner, F.; Caccetta, P.; and Wu, C
Diakogiannis, F. I.; Waldner, F.; Caccetta, P.; and Wu, C. 2020. ResUNet-a: A deep learning framework for semantic segmentation of remotely sensed data. ISPRS Journal of Photogrammetry and Remote Sensing, 162: 94--114
2020
-
[13]
Guo, X.; Lao, J.; Dang, B.; Zhang, Y.; Yu, L.; Ru, L.; Zhong, L.; Huang, Z.; Wu, K.; Hu, D.; et al. 2024. Skysense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery. In CVPR, 27672--27683
2024
-
[14]
Huang, B.; Li, J.; Chen, J.; Wang, G.; Zhao, J.; and Xu, T. 2023. Anti-UAV410: A thermal infrared benchmark and customized scheme for tracking drones in the wild. IEEE TPAMI, 46(5): 2852--2865
2023
-
[15]
Huang, F.; Zheng, S.; Qiu, Z.; Liu, H.; Bai, H.; and Chen, L. 2025. Text-IRSTD: Leveraging Semantic Text to Promote Infrared Small Target Detection in Complex Scenes. In ICCV, 10635--10644
2025
-
[16]
Li, B.; Xiao, C.; Wang, L.; Wang, Y.; Lin, Z.; Li, M.; An, W.; and Guo, Y. 2022. Dense nested attention network for infrared small target detection. IEEE TIP, 32: 1745--1758
2022
-
[17]
Li, B.; Ying, X.; Li, R.; Liu, Y.; Shi, Y.; and Li, M. 2024. The first competition on resource-limited infrared small target detection challenge: Methods and results. arXiv preprint arXiv:2408.09615
2024 arXiv
-
[18]
Li, R.; An, W.; Wang, Y.; Ying, X.; Dai, Y.; Wang, L.; Li, M.; Guo, Y.; and Liu, L. 2025. Probing deep into temporal profile makes the infrared small target detector much better. arXiv preprint arXiv:2506.12766
2025
-
[19]
Li, R.; An, W.; Xiao, C.; Li, B.; Wang, Y.; Li, M.; and Guo, Y. 2023. Direction-coded temporal U-shape module for multiframe infrared small target detection. IEEE Transactions on Neural Networks and Learning Systems
2023
-
[20]
Lin, C.-Y. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 74--81
2004
-
[21]
Lin, T.-Y.; Doll \'a r, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. 2017. Feature pyramid networks for object detection. In CVPR, 2117--2125
2017
-
[22]
Liu, H.; Li, C.; Li, Y.; and Lee, Y. J. 2024 a . Improved Baselines with Visual Instruction Tuning. In CVPR, 26286--26296
2024
-
[23]
Liu, P.; Zhang, Y.; Guo, G.; and Ding, J. 2024 b . Enhanced detection and recognition of road objects in infrared imaging using multi-scale self-attention. Sensors, 24(16): 5404
2024
-
[24]
Liu, Q.; Liu, R.; Zheng, B.; Wang, H.; and Fu, Y. 2024 c . Infrared small target detection with scale and location sensitivity. In CVPR, 17490--17499
2024
-
[25]
Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101
2017 arXiv
-
[26]
Muhtar, D.; Li, Z.; Gu, F.; Zhang, X.; and Xiao, P. 2024. Lhrs-bot: Empowering remote sensing with vgi-enhanced large multimodal language model. In ECCV, 440--457
2024
-
[27]
Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL, 311--318
2002
-
[28]
W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 8748--8763
2021
-
[29]
Ronneberger, O.; Fischer, P.; and Brox, T. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, 234--241
2015
-
[30]
Sagar Soni, H. D. M. F. M. A. M. M. S. D. P. F. C. W. L. J. K. S. K. F. K., Akshay Dudhane. 2025. EarthDial: Turning Multi-sensory Earth Observations to Interactive Dialogues. ArXiv
2025
-
[31]
Shi, Y.; He, R.; Hui, L.; Li, X.; Yang, J.; Cheng, M.-M.; and Dai, Y. 2025. AuxDet: Auxiliary Metadata Matters for Omni-Domain Infrared Small Target Detection. arXiv e-prints, arXiv--2505
2025
-
[32]
Singh, P.; and Singh, P. 2025. Leveraging Language Prior for Infrared Small Target Detection. arXiv preprint arXiv:2507.13113
2025 arXiv
-
[33]
Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015. Cider: Consensus-based image description evaluation. In CVPR, 4566--4575
2015
-
[34]
Wu, T.; Li, B.; Luo, Y.; Wang, Y.; Xiao, C.; Liu, T.; Yang, J.; An, W.; and Guo, Y. 2023. MTU-Net: Multilevel TransUNet for Space-Based Infrared Tiny Ship Detection. IEEE Transactions on Geoscience and Remote Sensing, 61: 1--15
2023
-
[35]
Wu, X.; Hong, D.; and Chanussot, J. 2022. UIU-Net: U-Net in U-Net for infrared small object detection. IEEE TIP, 32: 364--376
2022
-
[36]
Xiao, T.; Liu, Y.; Zhou, B.; Jiang, Y.; and Sun, J. 2018. Unified perceptual parsing for scene understanding. In ECCV, 418--434
2018
-
[37]
M.; and Luo, P
Xie, E.; Wang, W.; Yu, Z.; Anandkumar, A.; Alvarez, J. M.; and Luo, P. 2021. SegFormer: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 34: 12077--12090
2021
-
[38]
Xu, W.; Zheng, S.; Wang, C.; Zhang, Z.; Ren, C.; Xu, R.; and Xu, S. 2025. SAMamba: Adaptive state space modeling with hierarchical vision for infrared small target detection. Information Fusion, 124: 103338
2025
-
[39]
Yang, J.; Liu, S.; Wu, J.; Su, X.; Hai, N.; and Huang, X. 2025. Pinwheel-shaped convolution and scale-based dynamic loss for infrared small target detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 9202--9210
2025
-
[40]
Yuan, M.; Meng, D.; Xi, Z.; Zhao, T.; Zhao, S.; Dai, Y.; and Wei, X. 2026. Seeing Through the Noise: Improving Infrared Small Target Detection and Segmentation from Noise Suppression Perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...
2026
-
[41]
Yuan, S.; Qin, H.; Yan, X.; Akhtar, N.; and Mian, A. 2024. Sctransnet: Spatial-channel cross transformer network for infrared small target detection. IEEE Transactions on Geoscience and Remote Sensing, 62: 1--15
2024
-
[42]
Yuksekgonul, M.; Bianchi, F.; Kalluri, P.; Jurafsky, D.; and Zou, J. 2023. When and why Vision-Language Models behave like Bags-of-Words, and what to do about it? In International Conference on Learning Representations
2023
-
[43]
Zhang, M.; Li, X.; Gao, F.; Guo, J.; Gao, X.; and Zhang, J. 2025. SAIST: Segment any infrared small target model guided by contrastive language-image pretraining. In CVPR, 9549--9558
2025
-
[44]
Zhang, M.; Zhang, R.; Yang, Y.; Bai, H.; Zhang, J.; and Guo, J. 2022. ISNet: Shape matters for infrared small target detection. In CVPR, 877--886
2022
-
[45]
Zhang, W.; Cai, M.; Zhang, T.; Zhuang, Y.; and Mao, X. 2024. EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain. IEEE Transactions on Geoscience and Remote Sensing, 62: 1--27
2024
-
[46]
Zhao, M.; Li, W.; Li, L.; Hu, J.; Ma, P.; and Tao, R. 2022. Single-Frame Infrared Small-Target Detection: A survey. IEEE Geoscience and Remote Sensing Magazine, 10(2): 87--119
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.