Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

Da Yu: Towards USV-Based Image Captioning for Waterway Surveillance and Scene Understanding

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces WaterCaption, the first image-captioning dataset for waterway surveillance, and Da Yu, an edge-deployable multimodal language model whose Nano Transformer Adaptor produces state-of-the-art long captions at lower…

desk verdict A genuinely useful waterway captioning dataset, wrapped in a SOTA claim that the evidence doesn't yet support. read the letter →

arxiv 2506.19288 v2 pith:HPN22PYW submitted 2025-06-24 cs.CV cs.RO

Runwei Guan , Ningwei Ouyang , Tianhao Xu , Shaofeng Liang , Wei Dai , Yafeng Sun , Shang Gao , Songning Lai
show 5 more authors
Shanliang Yao Xuming Hu Ryan Wen Liu Yutao Yue Hui Xiong
This is my paper · ORCID
classification cs.CVcs.RO
keywords imagecaptioningwaterwaysurveillanceunmannedsurfacevehiclemultimodallargelanguagemodelvision-languageprojectorlong-textcaptiongenerationedgedeploymentWaterdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that waterway perception for unmanned surface vessels has stalled at object-level tasks such as detection and segmentation, which cannot produce the global, semantic situational picture that navigation logs require. To close that gap, it introduces WaterCaption, the first image-captioning dataset built specifically for waterway scenes, with 20.2k image-text pairs of fine-grained, multi-region, long-text descriptions averaging 88.6 words. It then proposes Da Yu, an edge-deployable multimodal large language model whose vision-to-language projector, the Nano Transformer Adaptor, combines pooled global attention with grouped dilated convolution at linear complexity. The paper claims that Da Yu, especially the 3.2B-parameter Da Yu-L, achieves the best captioning quality on WaterCaption while matching or improving benchmark scores on COCO, and that the NTA is both cheaper and more accurate than existing adaptors such as LDPv2. The practical payoff would be real-time, cloud-free surveillance log generation on a USV within the collision-avoidance response window.

What carries the argument

The load-bearing component is the Nano Transformer Adaptor (NTA), a vision-to-language projector designed for linear-complexity visual token fusion. Its Multi-Head Pooling Attention first pools the query sequence down to a smaller number of tokens, computes attention against the full keys and values, and then expands the pooled context back onto the original queries with a softmax attention step, while a grouped dilated convolution residual path restores high-frequency detail. This structure gives a global receptive field with O(N n d) complexity rather than O($N^{2}$ d), and the paper's ablations show that removing the dilated convolution, shrinking the pooled query size, or replacing pooling attention with vanilla cross-attention all hurt the balanced BLEU/METEOR/GPT-Score profile. Around the NTA, pixel-shuffle compresses visual tokens from MobileCLIP-B, and a two-stage training procedure first aligns modalities with a frozen visual encoder and frozen Qwen 2.5 LLM, then unfreezes the last visual block and the LLM for captioning.

What would settle it

Take 500 fresh waterway images from sources other than WaterScenes, such as new USV recordings with independent ground-truth scene checklists, and have human experts score Da Yu's captions against the seven key regions. If Da Yu-L does not outperform MobileVLM v2 and InternVL 2.5 on factual correctness, or if the WaterCaption ranking changes when GPT-Score is replaced by human verdicts, the claimed balance of performance and efficiency is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a long-form image-captioning task can serve as the perceptual backbone for automated waterway monitoring, replacing fragmented detection outputs with structured, natural-language navigation logs. To make that claim testable, the authors construct WaterCaption, whose captions are organized around seven key semantic regions—waterway objects, buildings, weather, wave amplitude, shore condition, cleanliness, and ego-USV position—and are annotated by seven experts with LLM-assisted rewriting. The supporting engineering claim is that the Nano Transformer Adaptor eliminates the usual trade-off between global image context and local detail, delivering linear-complexity visual fusion with 4.81M parameters and 2.46 GFLOPs, and improves long-caption generation relative to MLP, LDP, LDPv2, and MWA adaptors. On the WaterCaption benchmark Da Yu-L reaches the highest scores across ROUGE, BLEU, METEOR, CIDEr, and GPT-Score, and the NTA also lifts MobileVLM v2 when dropped into that model, which the paper takes as evidence of portability. The inference claim is that the model runs on an NVIDIA Jetson Orin edge device, with Da Yu-S at 4.35 seconds per caption, inside the 8-second planning window.

Load-bearing premise

The load-bearing premise is that WaterCaption's human- and LLM-corrected captions are factually accurate enough to serve as ground truth, so the performance gap between Da Yu and other models is a real gain in scene understanding rather than an artifact of the benchmark or of the GPT-Score judge.

Editorial extensions

If this is right

  • If the WaterCaption benchmark is adopted, long-text captioning quality becomes a measurable target for waterway perception rather than detection scores.
  • An NTA-style projector could be dropped into other small MLLMs to get better long captions at lower parameter and FLOP counts, as the MobileVLM v2 porting experiment suggests.
  • A USV carrying Da Yu could generate structured monitoring logs onboard within the 8-second response window required by inland navigation rules, without cloud round trips.
  • The dataset forces models to describe seven distinct scene regions, so future waterway captioning systems can be audited region by region instead of by a single sentence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to convert each caption into a structured checklist of the seven key elements and score factual agreement with the WaterScenes annotations, testing whether semantic quality goes beyond text overlap.
  • Because GPT-Score is itself an LLM, the authors' reported margins between Da Yu and rivals may partly measure which model produces output that another LLM prefers; a human or rule-based factual-consistency judge could change the ranking.
  • The NTA's pooling of queries to a fixed small set suggests a natural scaling path: keeping the pooled query size constant while increasing image resolution should keep adaptor cost bounded, which is exactly the regime USV cameras operate in.
  • If WaterCaption's captions are treated as ground truth for training, any systematic annotation bias, such as under-reporting small floating debris in glare conditions, will be inherited by trained models; collecting independent images without WaterScenes labels would test how well the dataset transfers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces WaterCaption, a new image-captioning dataset for waterway surveillance built on the WaterScenes dataset, with 20.2k/20,193 image-text pairs and long, multi-region captions covering seven domain key elements (waterway objects, weather, wave amplitude, shore condition, cleanliness, buildings, and ego-USV position). The paper also proposes Da Yu, a lightweight MLLM using a MobileCLIP-B visual encoder, a Qwen 2.5 language model, and a novel Nano Transformer Adaptor (NTA) that combines pooled-query attention with grouped dilated convolution. Experiments compare Da Yu with 15 models on WaterCaption, ablate the adaptor, report COCO generalization, and measure latency on a Jetson Orin platform. The central claims are that Da Yu achieves state-of-the-art captioning quality on WaterCaption and efficient edge deployment, and that NTA significantly improves long-form caption generation while using fewer parameters and FLOPs than compared adaptors.

Significance. If validated, WaterCaption would be a useful new benchmark for long-text, domain-specific image captioning, and NTA's efficiency results (4.81M parameters, 2.46G FLOPs, Table VII) are a plausible contribution to lightweight MLLM design. The paper honestly discloses several limitations in Section VI-A, and the qualitative case in Fig. 13 is a valuable stress test. The authors also release a project page. However, the headline claims are not yet supported by the evidence as presented: the benchmark is self-constructed with LLM-assisted annotations, the main semantic metric is an LLM judge whose reliability is acknowledged to be open, and no statistical uncertainty is reported for any experimental table. The dataset contribution itself may be valuable regardless of the model claims, but the current manuscript does not establish the claimed superiority of Da Yu or NTA.

major comments (5)
  1. [§III-A, §V-A, Table III] The state-of-the-art claim rests on a self-constructed benchmark whose reference captions were produced by seven annotators with ChatGPT-4o/DeepSeek-R1 rewriting and no independent factual verification against the WaterScenes structured labels (bounding boxes, drivable area, shoreline). Since every model is scored against these captions, any systematic annotation bias is inherited by all metrics. Please add a human-verified subset, a factual-consistency check against the WaterScenes labels, and inter-annotator agreement or verification statistics. This is essential to support the claim of 'surpassing state-of-the-art' on WaterCaption.
  2. [§V-B, Tables V and VI] The claimed advantage of NTA over MLP and LDPv2 is within likely noise. In Table V, NTA vs MLP shows BLEU-3 0.28 vs 0.29 (lower), ROUGH-L 0.38 vs 0.36, METEOR 0.46 vs 0.43, and GPT-Score 0.83 vs 0.81; NTA vs LDPv2 shows a METEOR tie at 0.46 and a GPT-Score tie at 0.83 vs 0.82. No confidence intervals, multiple seeds, or significance tests are reported in Tables III, V, VI, X, or XII. Please report at least 3 seeds with mean and standard deviation, and a paired significance test for the headline comparisons, before claiming that NTA 'significantly' improves long-form generation.
  3. [§VI-A, Fig. 13] GPT-Score is the main semantic metric, yet the authors state that 'the reliability of large language models in evaluation remains an open question,' and Fig. 13 shows a prediction receiving GPT-Score 0.92 while BLEU-3, CIDEr, and SPICE are near zero. The paper does not specify the GPT-Score prompt, the LLM version, the scoring scale, or the number of evaluation samples. Please provide a precise evaluation protocol, a correlation study with human judgments on a sample (ideally using the same expert annotators who built the dataset), and a separate report of cases where GPT-Score and lexical metrics diverge.
  4. [§V-B, Table X] The COCO generalization claim is not supported by the reported numbers. Da Yu-B (1.7B) matches InternVL 2.5 (1.9B) on BLEU-4 (0.43 vs 0.43) and edges CIDEr (1.41 vs 1.38) and SPICE (0.33 vs 0.32), a tie within likely noise at single-run precision. The COCO training/evaluation protocol is also underspecified: it is unclear whether all comparison models were trained under identical data splits, initialization, and checkpoint selection. Please clarify the protocol and report multi-run statistics before claiming robust generalization.
  5. [§III, Table I, Abstract] The dataset statistics are internally inconsistent. The Introduction and Abstract state 20,193 and '20.2k', while Section III states 20,247, and the sum of the reported train/validation/test splits (14,135 + 2,019 + 4,039) is 20,193. Additionally, 'Vocab Size 1.8M' in Table I cannot be a unique-vocabulary count for 20k captions of about 88 words each; it is likely the total number of tokens, in which case the column is mislabeled. Please correct these numbers and clarify the vocabulary definition.
minor comments (5)
  1. [Throughout] Typographical errors should be fixed: 'ROUGH' should be 'ROUGE' in Tables III–IV and the text, 'METROR' should be 'METEOR' in Table IV, 'MW A' should be 'MWA' in Tables V and VII, and Fig. 7's caption refers to 'NTP' where it should say 'NTA'.
  2. [§V-B] Several table/figure cross-references are wrong: the text says 'Table II' when referring to scenario results (should be Table IV), 'Table VII' when referring to adaptor performance (should be Table V), and 'Figure XI'/'Figure X' when referring to Tables XI/X. Please correct these references.
  3. [Eqs. (5)–(6)] The notation for the attention matrices is ambiguous: S is used as both q·K^T (an n×N matrix) and as the matrix in S/√d, and the sentence 'd represents the dimensionality of the attention components S, Q, and q' is incorrect because S has dimensions n×N, not d. Please introduce distinct symbols and clarify dimensions.
  4. [§III-A, Step 4] The annotation pipeline uses LLM rewriting (ChatGPT-4o and DeepSeek-R1) but it is not stated whether the LLMs had access to the WaterScenes structured labels or only the image. Please clarify, because this affects the factual-verification claim made in Step 5.
  5. [References] The GPT-Score metric is only cited to the DriveLM paper [55]; please provide a more direct reference for LLM-based caption evaluation or describe the scoring rubric in the paper.

Circularity Check

0 steps flagged · score 2.0 of 10

No equation-level circularity; the central claims rest on a self-constructed benchmark and an LLM-judge metric whose reliability the paper itself leaves open, but no prediction is fitted to its own target by construction.

full rationale

The paper does not derive any headline result from an equation that is defined in terms of that result, and it does not fit parameters to the test set and then rename them as predictions. The WaterCaption benchmark is constructed by the authors, and the main semantic metric, GPT-Score, is an LLM judge whose reliability is explicitly left open in Section VI-A: 'the reliability of large language models in evaluation remains an open question.' These are validity and generalizability concerns rather than circular reductions: model outputs are scored against held-out references, and GPT-Score is not a function of the proposed model's parameters. The dependency on WaterScenes [18], a dataset co-authored by members of this team, is a provenance dependency rather than a load-bearing self-citation used to forbid alternatives. External COCO evaluation (Table X) and standard n-gram metrics (Tables III and V) provide independent, if partial, grounding. Because no specific equation, fitted parameter, or uniqueness theorem is shown to reduce to its own input, the circularity score is low.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claims rest on standard deep learning components and on untested domain assumptions about annotation correctness and evaluation validity. The free parameters listed are the most consequential hand-set choices in the pipeline; no new physical or unobservable entities are introduced.

free parameters (3)
  • Pooled query size n in NTA = 12x12, i.e., n = 144
    A user-defined hyperparameter in Equation 6; the ablation shows 8x8 degrades metrics, so 12x12 is chosen by hand rather than derived.
  • Pixel shuffle rate r = 2
    Set in Section IV-B to balance token compression and spatial fidelity; no derivation or sensitivity analysis is given.
  • Training schedule = WaterCaption: 10 epochs, lr 4e-5, batch 32; COCO: 30 epochs, lr 1e-6 to 5e-7, batch 128
    These fine-tuning hyperparameters are chosen by hand and no sensitivity analysis is reported, yet they affect all reported quality and latency numbers.
assumptions (6)
  • standard math Standard attention, layer norm, and convolution primitives behave as specified in deep learning frameworks.
    Used in Equations 4 to 8; assumed as framework-correct background machinery.
  • domain assumption WaterScenes annotations are accurate enough to serve as reliable references for captions.
    Section III-A1 relies on WaterScenes 2D boxes, drivable area, and shoreline labels without independent verification.
  • domain assumption The seven key semantic regions are a complete and necessary description of a waterway scene.
    Section III-A2 defines the caption protocol around these seven elements; completeness is asserted, not demonstrated.
  • domain assumption LLM-assisted calibration plus expert consensus yields factually correct ground-truth captions.
    Sections III-A4 and III-A5 describe the hybrid pipeline; no inter-annotator agreement or external check is reported.
  • domain assumption GPT-Score is a reliable proxy for long-caption quality.
    GPT-Score is used throughout Tables III to XII, and Section VI-A states that the reliability of LLM evaluation remains an open question.
  • domain assumption Pretrained MobileCLIP and Qwen 2.5 provide suitable priors for waterway captioning.
    Section IV adopts these backbones and only fine-tunes the adaptor and the last visual block; if the priors lack maritime knowledge, performance would degrade.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Da Yu: Towards USV-Based Image Captioning for Waterway Surveillance and Scene Understanding." pith.science (2026). https://pith.science/paper/HPN22PYW

@misc{pith2026250619288,
  author       = {Pith},
  title        = {Pith review of: Da Yu: Towards USV-Based Image Captioning for Waterway Surveillance and Scene Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HPN22PYW}},
  note         = {Machine review of arXiv:2506.19288}
}
read the original abstract

Automated waterway environment perception is crucial for enabling unmanned surface vessels (USVs) to understand their surroundings and make informed decisions. Most existing waterway perception models primarily focus on instance-level object perception paradigms (e.g., detection, segmentation). However, due to the complexity of waterway environments, current perception datasets and models fail to achieve global semantic understanding of waterways, limiting large-scale monitoring and structured log generation. With the advancement of vision-language models (VLMs), we leverage image captioning to introduce WaterCaption, the first captioning dataset specifically designed for waterway environments. WaterCaption focuses on fine-grained, multi-region long-text descriptions, providing a new research direction for visual geo-understanding and spatial scene cognition. Exactly, it includes 20.2k image-text pair data with 1.8 million vocabulary size. Additionally, we propose Da Yu, an edge-deployable multi-modal large language model for USVs, where we propose a novel vision-to-language projector called Nano Transformer Adaptor (NTA). NTA effectively balances computational efficiency with the capacity for both global and fine-grained local modeling of visual features, thereby significantly enhancing the model's ability to generate long-form textual outputs. Da Yu achieves an optimal balance between performance and efficiency, surpassing state-of-the-art models on WaterCaption and several other captioning benchmarks.

Figures

Figures reproduced from arXiv: 2506.19288 by the authors.

Figure 1
Figure 1. Overview of the USV-oriented waterway automatic captioning. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The process of captioning annotation for the WaterCaption dataset, which mainly contains four steps (A, B, C, D and E). [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Samples in the WaterCaption dataset. The rightmost sample shows the caption text corresponding to different regions. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Pairwise statistics of environmental variables (time, waterway, weather [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 4
Figure 4. Figure 4: Statistics of captioning corpus in WaterCaption dataset. (a) presents [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 7
Figure 7. Figure 7: The architecture of our proposed multi-modal large language model: Da Yu [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: The structure of Pooling Attention in Nano Transformer Adaptor. [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 9
Figure 9. Figure 9: The overview of progressive two-stage training strategy. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Prediction results of various multi-modal large language models under different waterway scenarios. [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 13
Figure 13. Figure 13: The limitations in WaterCaption dataset. The text is the predicted cap [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 12
Figure 12. Figure 12: Comparison between different models on waterway image captioning [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey

    cs.RO 2025-09 conditional novelty 4.0 of 10

    A structured survey of LLM-based trajectory prediction methods, organized into trajectory-language mapping, multimodal fusion, and constraint-based reasoning, with benchmarks, metrics, and future directions.

Reference graph

Works this paper leans on

57 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    A data set for airborne maritime surveillance environments,

    R. Ribeiro, G. Cruz, J. Matos, and A. Bernardino, “A data set for airborne maritime surveillance environments,” IEEE Transactions on Circuits and Systems for Video Technology , vol. 29, no. 9, pp. 2720– 2732, 2017

    work page 2017

  2. [2]

    Asy-vrnet: Waterway panoptic driving perception model based on asymmetric fair fusion of vision and 4d mmwave radar,

    R. Guan, S. Yao, K. L. Man, X. Zhu, Y . Yue, J. Smith, E. G. Lim, and Y . Yue, “Asy-vrnet: Waterway panoptic driving perception model based on asymmetric fair fusion of vision and 4d mmwave radar,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 12 479–12 486

    work page 2024

  3. [3]

    Usvtrack: Usv-based 4d radar-camera tracking dataset for autonomous driving in inland waterways,

    S. Yao, R. Guan, Y . Ni, Y . Yue, X. Zhu, and R. W. Liu, “Usvtrack: Usv-based 4d radar-camera tracking dataset for autonomous driving in inland waterways,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025 , 2025

    work page 2025

  4. [4]

    Are we ready for unmanned surface vehicles in inland waterways? the usvinland multisensor dataset and benchmark,

    Y . Cheng, M. Jiang, J. Zhu, and Y . Liu, “Are we ready for unmanned surface vehicles in inland waterways? the usvinland multisensor dataset and benchmark,” IEEE Robotics and Automation Letters , vol. 6, no. 2, pp. 3964–3970, 2021

    work page 2021

  5. [5]

    Achelous: A fast unified water-surface panoptic perception framework based on fusion of monocular camera and 4d mmwave radar,

    R. Guan, S. Yao, X. Zhu, K. L. Man, E. G. Lim, J. Smith, Y . Yue, and Y . Yue, “Achelous: A fast unified water-surface panoptic perception framework based on fusion of monocular camera and 4d mmwave radar,” in 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC) . IEEE, 2023, pp. 182–188

    work page 2023

  6. [6]

    Achelous++: Power-oriented water- surface panoptic perception framework on edge devices based on vision- radar fusion and pruning of heterogeneous modalities,

    R. Guan, H. Zhao, S. Yao, K. L. Man, X. Zhu, L. Yu, Y . Yue, J. Smith, E. G. Lim, W. Ding et al., “Achelous++: Power-oriented water- surface panoptic perception framework on edge devices based on vision- radar fusion and pruning of heterogeneous modalities,” arXiv preprint arXiv:2312.08851, 2023

    arXiv 2023

  7. [7]

    Watervg: Waterway visual grounding based on text-guided vision and mmwave radar,

    R. Guan, L. Jia, S. Yao, F. Yang, S. Xu, E. Purwanto, X. Zhu, K. L. Man, E. G. Lim, J. Smith et al., “Watervg: Waterway visual grounding based on text-guided vision and mmwave radar,” IEEE Transactions on Intelligent Transportation Systems, 2025

    work page 2025

  8. [8]

    Microsoft coco captions: Data collection and evaluation server,

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325 , 2015

    arXiv 2015

Show all 57 references
  1. [9]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, and J. Hocken- maier, “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 2641–2649

    2015

  2. [10]

    Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,

    S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, “Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 3558–3568

    2021

  3. [11]

    Im2text: Describing images using 1 million captioned photographs,

    V . Ordonez, G. Kulkarni, and T. Berg, “Im2text: Describing images using 1 million captioned photographs,” Advances in neural information processing systems, vol. 24, 2011

    2011

  4. [12]

    Vizwiz grand challenge: Answering visual questions from blind people,

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham, “Vizwiz grand challenge: Answering visual questions from blind people,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 3608–3617

    2018

  5. [13]

    Learning deep represen- tations of fine-grained visual descriptions,

    S. Reed, Z. Akata, H. Lee, and B. Schiele, “Learning deep represen- tations of fine-grained visual descriptions,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 49–58

    2016

  6. [14]

    Fashion captioning: Towards generating accurate descrip- tions with semantic rewards,

    X. Yang, H. Zhang, D. Jin, Y . Liu, C.-H. Wu, J. Tan, D. Xie, J. Wang, and X. Wang, “Fashion captioning: Towards generating accurate descrip- tions with semantic rewards,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Par...

    2020

  7. [15]

    Break- ingnews: Article annotation by image and text processing,

    A. Ramisa, F. Yan, F. Moreno-Noguer, and K. Mikolajczyk, “Break- ingnews: Article annotation by image and text processing,” IEEE trans- actions on pattern analysis and machine intelligence , vol. 40, no. 5, pp. 1072–1085, 2017

    2017

  8. [16]

    Textcaps: a dataset for image captioning with reading comprehension,

    O. Sidorov, R. Hu, M. Rohrbach, and A. Singh, “Textcaps: a dataset for image captioning with reading comprehension,” in Computer Vision– ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16 . Springer, 2020, pp. 742–758

    2020

  9. [17]

    Deep learning approaches on image captioning: A review,

    T. Ghandi, H. Pourreza, and H. Mahyar, “Deep learning approaches on image captioning: A review,” ACM Computing Surveys , vol. 56, no. 3, pp. 1–39, 2023

    2023

  10. [18]

    Waterscenes: A multi-task 4d radar-camera fusion dataset and benchmarks for autonomous driving on water surfaces,

    S. Yao, R. Guan, Z. Wu, Y . Ni, Z. Huang, R. W. Liu, Y . Yue, W. Ding, E. G. Lim, H. Seo et al. , “Waterscenes: A multi-task 4d radar-camera fusion dataset and benchmarks for autonomous driving on water surfaces,” IEEE Transactions on Intelligent Transportation Systems, 2024

    2024

  11. [19]

    U. C. Guard, Navigation Rules and Regulations Handbook: Interna- tional—Inland. Simon and Schuster, 2018

    2018

  12. [20]

    Mobilevlm v2: Faster and stronger baseline for vision language model,

    X. Chu, L. Qiao, X. Zhang, S. Xu, F. Wei, Y . Yang, X. Sun, Y . Hu, X. Lin, B. Zhang et al. , “Mobilevlm v2: Faster and stronger baseline for vision language model,” arXiv preprint arXiv:2402.03766 , 2024

    2024 arXiv

  13. [21]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PmLR, 2021, pp. 8748–8763

    2021

  14. [22]

    Align before fuse: Vision and language representation learning with momentum distillation,

    J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” Advances in neural information processing systems, vol. 34, pp. 9694–9705, 2021

    2021

  15. [23]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in International conference on machine learning . PMLR, 2022, pp. 12 888–12 900

    2022

  16. [24]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” in International conference on machine learning . PMLR, 2023, pp. 19 730–19 742

    2023

  17. [25]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892– 34 916, 2023

    2023

  18. [26]

    Qwen2. 5 technical report,

    A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al., “Qwen2. 5 technical report,” arXiv e-prints, pp. arXiv–2412, 2024

    2024

  19. [27]

    Task-adaptive attention for image captioning,

    C. Yan, Y . Hao, L. Li, J. Yin, A. Liu, Z. Mao, Z. Chen, and X. Gao, “Task-adaptive attention for image captioning,” IEEE Transactions on Circuits and Systems for Video technology , vol. 32, no. 1, pp. 43–51, 2021

    2021

  20. [28]

    Vision-enhanced and consensus- aware transformer for image captioning,

    S. Cao, G. An, Z. Zheng, and Z. Wang, “Vision-enhanced and consensus- aware transformer for image captioning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 10, pp. 7005–7018, 2022

    2022

  21. [29]

    Adaptive path selection for dynamic image captioning,

    T. Xian, Z. Li, Z. Tang, and H. Ma, “Adaptive path selection for dynamic image captioning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 9, pp. 5762–5775, 2022

    2022

  22. [30]

    Double-stream position learning trans- JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 14 former network for image captioning,

    W. Jiang, W. Zhou, and H. Hu, “Double-stream position learning trans- JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 14 former network for image captioning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 11, pp. 7706–7718, 2022

    2015

  23. [31]

    A comprehen- sive survey of 3d dense captioning: Localizing and describing objects in 3d scenes,

    T. Yu, X. Lin, S. Wang, W. Sheng, Q. Huang, and J. Yu, “A comprehen- sive survey of 3d dense captioning: Localizing and describing objects in 3d scenes,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 3, pp. 1322–1338, 2023

    2023

  24. [32]

    Spt: Spatial pyramid transformer for image captioning,

    H. Zhang, P. Zeng, L. Gao, X. Lyu, J. Song, and H. T. Shen, “Spt: Spatial pyramid transformer for image captioning,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 6, pp. 4829– 4842, 2023

    2023

  25. [33]

    Multimodal transformer with multi- view visual representation for image captioning,

    J. Yu, J. Li, Z. Yu, and Q. Huang, “Multimodal transformer with multi- view visual representation for image captioning,” IEEE transactions on circuits and systems for video technology , vol. 30, no. 12, pp. 4467– 4480, 2019

    2019

  26. [34]

    Mobilevlm: A vision-language model for better intra-and inter-ui understanding,

    Q. Wu, W. Xu, W. Liu, T. Tan, J. Liu, A. Li, J. Luan, B. Wang, and S. Shang, “Mobilevlm: A vision-language model for better intra-and inter-ui understanding,” arXiv preprint arXiv:2409.14818 , 2024

    2024 arXiv

  27. [35]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu et al. , “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, ...

    2024

  28. [36]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He et al., “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  29. [37]

    Tinyllava: A framework of small-scale large multimodal models,

    B. Zhou, Y . Hu, X. Weng, J. Jia, J. Luo, X. Liu, J. Wu, and L. Huang, “Tinyllava: A framework of small-scale large multimodal models,”arXiv preprint arXiv:2402.14289, 2024

    2024 arXiv

  30. [38]

    Mask-vrdet: A robust riverway panoptic perception model based on dual graph fusion of vision and 4d mmwave radar,

    R. Guan, S. Yao, L. Liu, X. Zhu, K. L. Man, Y . Yue, J. Smith, E. G. Lim, and Y . Yue, “Mask-vrdet: A robust riverway panoptic perception model based on dual graph fusion of vision and 4d mmwave radar,” Robotics and Autonomous Systems , vol. 171, p. 104572, 2024

    2024

  31. [39]

    Nanomvg: Usv-centric low-power multi- task visual grounding based on prompt-guided camera and 4d mmwave radar,

    R. Guan, J. Liu, L. Jia, H. Zhao, S. Yao, X. Zhu, K. L. Man, E. G. Lim, J. Smith, and Y . Yue, “Nanomvg: Usv-centric low-power multi- task visual grounding based on prompt-guided camera and 4d mmwave radar,” arXiv preprint arXiv:2408.17207 , 2024

    2024 arXiv

  32. [40]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  33. [41]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  34. [42]

    Mobileclip: Fast image-text models through multi-modal reinforced training,

    P. K. A. Vasu, H. Pouransari, F. Faghri, R. Vemulapalli, and O. Tuzel, “Mobileclip: Fast image-text models through multi-modal reinforced training,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 15 963–15 974

    2024

  35. [43]

    Coca: Contrastive captioners are image-text foundation models,

    J. Yu, Z. Wang, V . Vasudevan, L. Yeung, M. Seyedhosseini, and Y . Wu, “Coca: Contrastive captioners are image-text foundation models,” Transactions on Machine Learning Research

  36. [44]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” in Proceedings of the IEEE/CVF international conference on computer vision , 2023, pp. 11 975–11 986

    2023

  37. [45]

    Agent attention: On the integration of softmax and linear attention,

    D. Han, T. Ye, Y . Han, Z. Xia, S. Pan, P. Wan, S. Song, and G. Huang, “Agent attention: On the integration of softmax and linear attention,” in European Conference on Computer Vision . Springer, 2024, pp. 124– 140

    2024

  38. [46]

    Dual-level collaborative transformer for image captioning,

    Y . Luo, J. Ji, X. Sun, L. Cao, Y . Wu, F. Huang, C.-W. Lin, and R. Ji, “Dual-level collaborative transformer for image captioning,” in Proceedings of the AAAI conference on artificial intelligence , vol. 35, no. 3, 2021, pp. 2286–2293

    2021

  39. [47]

    Comprehending and ordering seman- tics for image captioning,

    Y . Li, Y . Pan, T. Yao, and T. Mei, “Comprehending and ordering seman- tics for image captioning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 17 990–17 999

    2022

  40. [48]

    Oscar: Object-semantics aligned pre-training for vision-language tasks,

    X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei et al., “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX

    2020

  41. [49]

    Springer, 2020, pp. 121–137

    2020

  42. [50]

    Qwen2. 5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang et al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  43. [51]

    Multiway- adapater: Adapting large-scale multi-modal models for scalable image- text retrieval,

    Z. Long, G. Killick, R. McCreadie, and G. A. Camarasa, “Multiway- adapater: Adapting large-scale multi-modal models for scalable image- text retrieval,” arXiv preprint arXiv:2309.01516 , 2023

    2023 arXiv

  44. [52]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out , 2004, pp. 74–81

    2004

  45. [53]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  46. [54]

    Lavie and A

    A. Lavie and A. Agarwal, “Meteor,” 01 2007, pp. 228–231

    2007

  47. [55]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proceedings of the IEEE confer- ence on computer vision and pattern recognition , 2015, pp. 4566–4575

    2015

  48. [56]

    Drivelm: Driving with graph visual question answering,

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual question answering,” in European Conference on Computer Vision . Springer, 2024, pp. 256–274. Runwei Guan (Member, IEEE) is currently a rese...

    2024

  49. [96]

    His accolades include Fellow of the AAAI, AAAS, IEEE, and CAAI, the Distinguished Scientist of the Association for Computing Machinery (ACM), and received highly recognized awards such as the 2023 ACM SIGKDD Service Award, the 2021 AAAI Best Paper Award and IEEE ICDM Best Pape...

    2011

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.