MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning

Duc Thanh Nguyen; Quang-Trung Truong; Rinaldi Gotama; Sai-Kit Yeung; Vo Hoang Kim Tuyen Dang; Yuk-Kwan Wong

REVIEW 4 major objections 4 minor 48 references

MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper presents MSC, a dataset pairing 24.8 hours of underwater footage with clip-level captions and per-object masks for six marine categories, to ground language in what is visible and reduce hallucination in marine video understanding

desk verdict Useful dataset idea, but the paper's own numbers don't agree on its size, and the clip-level captioning benchmark is circular — the contribution as reported doesn't hold up. read the letter →

arxiv 2508.04549 v3 pith:4PQEEMLX submitted 2025-08-06 cs.CV cs.AIcs.MM

Quang-Trung Truong , Yuk-Kwan Wong , Vo Hoang Kim Tuyen Dang , Rinaldi Gotama , Duc Thanh Nguyen , Sai-Kit Yeung This is my paper

classification cs.CVcs.AIcs.MM

keywords marinevideodatasetcaptioningvisualgroundinginstancesegmentationvideo-textbenchmarkwildlifetext-to-videogeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents MSC, a public dataset of real underwater diving footage organized as video, text, and segmentation-mask triplets. It claims to be the first large-scale marine video dataset in which every short clip carries both pixel-level instance masks, for six categories (fish, reefs, aquatic plants, wrecks, human divers, sea floor), and one to four expert-refined natural-language captions. The intended payoff is that captioning, visual grounding, and text-to-video generation models can be developed and tested against marine scenes that generic video datasets handle poorly. The construction uses masks to anchor captions to visible objects and video splitting to isolate single-event clips, and the benchmark reports that current open and commercial models leave substantial room for improvement on this domain.

What carries the argument

The load-bearing object is the video-segmentation mask-text triplet: each clip is stored with pixel-wise instance masks and a grounded caption that names only what the masks verify. Three mechanisms carry the dataset: (1) a SAM-initialized annotation tool refined by professionals to get masks; (2) clip splitting, which turns long, event-diluted videos into single-event segments whose captions are more concrete; and (3) generation of draft captions by three LLMs followed by biologist refinement, which is the step that binds text to the masked objects and is meant to suppress hallucination.

What would settle it

Measure the edit distance or n-gram overlap between the raw LLM drafts and the released human-refined captions: if the refined captions are nearly identical to the GPT-4.1 drafts, Table 3 is a self-comparison. A second check is to have fresh human annotators write clip captions from scratch and test whether GPT-4.1 still outperforms Gemini by the same margin on those references.

Watch

Extended reading notes

Core claim

The central claim is that marine video understanding needs grounded triplets rather than caption-only data. To build MSC, the authors recorded GoPro footage during 71 diving visits across 20 sites and selected videos for clarity, complexity, and diversity; annotators then produced iterative SAM-initialized pixel masks for six marine object categories, videos were split into short semantically coherent clips, and GPT-4.1, Gemini-2.0 Flash-Lite, and Qwen-VL generated draft captions that 18 marine biologists refined and aggregated into a per-video summary. The benchmark then evaluates video-level captioning, clip-level captioning, visual grounding, and text-to-video generation. On captioning, G

Load-bearing premise

The captioning benchmark treats the biologists' edited LLM drafts as independent ground truth, so if the human edits were light, the clip-level scores mostly measure how well a model reproduces GPT-4.1's own wording.

Editorial extensions

If this is right

MSC enables joint evaluation of captioning, grounding, and generation on identical marine clips, so progress in one task can be checked against the others.
Because captions are anchored to masks, model outputs can be tested for hallucination: a caption mentioning an object that has no segmented instance is a detectable error.
The reported grounding results imply that relying on COCO-style categories leaves marine visual grounding under-solved, with natural captions causing the biggest drop for open-world detectors.
Weak FID/FVD scores for text-to-video models on marine prompts indicate the domain is underrepresented in training data and that MSC's captioned clips can serve as domain-specific supervision or evaluation.
The two-stage pipeline itself is a recipe for other expert domains: masks constrain what synthetic captions may say, and a small expert panel corrects the residue — a cheaper path to grounded video-text data than full manual captioning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial: the clip-level captioning benchmark may partly measure self-agreement, because the reference captions began as GPT-4.1 drafts; a held-out set of from-scratch human captions would reveal how much of GPT-4.1's lead is genuine quality.
Editorial: the dataset's mask-caption pairing permits an automatic truthfulness audit — checking that every grounded noun phrase in a caption has a corresponding mask — which would be a novel validity ceiling on video caption generation models.
Editorial: the text reports 396 selected videos in Section 3.1 while Table 1 and Figure 1 describe 2,743 videos and 24.8 hours; the released archive should be checked to determine which count is the actual dataset size.
Editorial: the six chosen categories are coarse; applying the same mask-then-caption pipeline at species level would make the dataset directly usable for ecology and conservation monitoring, where fine-grained identity and behavior are what matter.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hi — quick take on MSC. The useful part is the dataset itself: a marine video collection with per-frame segmentation masks and clip-level captions, plus human-in-the-loop annotation using SAM and biologists. That triple combination appears to be new, and if the released data is as described, it could be a genuine resource for marine video understanding. Credit where due: the annotation process is described concretely, the data and code are promised to be released, and the authors brought in domain biologists rather than trusting LLM output blindly. That is the right instinct.

The problems are not small. The paper reports 396 selected videos in §3.1 but Table 1 lists 2,743 videos. The total duration arithmetic makes sense only if the 2,743 number counts clips, not videos; same for the 20 countries/regions in Table 1 versus 13 in Figure 1. The paper never reconciles these, so the headline 'large-scale' claim is unverifiable as written. A dataset paper has to be precise about what is in the download; this ambiguity undermines every downstream statistic.

Second, the clip-level captioning benchmark is effectively circular. GPT-4.1 generated the synthetic captions that humans then refined, and Table 3 shows GPT-4.1 outperforming everything by a wide margin. The paper doesn't report how much the biologists changed the synthetic text, so we can't tell whether the score measures caption quality or agreement with the model's own style. That is a load-bearing flaw for the benchmark contribution.

Third, the claim that clip splitting improves caption semantics is asserted but never ablated. It might be true, but there's no evidence for it in the paper.

On balance, the dataset idea is valuable and the annotation pipeline is plausible, but the manuscript as written doesn't support the claims. The internal inconsistencies should have been caught in any careful proofread, and the circularity needs a human-agreement or edit-distance analysis to be meaningful. If the authors fix the numbers and provide a proper human-refinement measurement, this could be a solid dataset paper. As it stands, I would not rely on the benchmark results.

Who is this for? Researchers who want a marine video-text-segmentation resource and are willing to inspect the released data themselves. It deserves referee attention because the underlying resource is potentially useful; it just needs a major revision. Recommend: send to peer review, but expect heavy revision.

Referee Report

4 major / 4 minor

Summary. The paper introduces MSC, a claimed first large-scale marine wildlife video dataset with per-frame instance segmentation masks, clip-level captions, synthetic captions refined by biologists, and video-level summaries. The authors describe a two-stage annotation pipeline and report benchmark results for video-level and clip-level captioning, visual grounding, and text-to-video generation, comparing open-source and commercial models.

Significance. A publicly released marine video dataset with grounded segmentation and expert-refined captions would fill a real gap, since existing marine datasets are mostly image-centric and existing video-text datasets are generic or human-centric. The paper's strengths are the domain-specific annotation effort, the involvement of biologists, and the breadth of benchmark tasks. However, the central contribution cannot be evaluated as written because the dataset size and identity are internally contradictory, and the headline clip-captioning benchmark is circular. If these issues were resolved, the resource could still be useful, but the current manuscript does not support its main claims.

major comments (4)

[§3.1, Table 1, Fig. 1] The dataset scale is irreconcilable. §3.1 states 'we finally selected 396 videos out of 2,743 videos,' but Table 1 lists '#Video 2743' for Our MSC. Figure 1's caption says the dataset 'is recorded from 13 different countries,' while Table 1 reports 20 countries/regions and §3.3 says '20 distinct diving sites.' The arithmetic suggests Table 1's #Video counts clips, not videos: 2,743 × 32.8 s ≈ 24.98 h, matching the 24.8 h total, whereas 396 × 32.8 s ≈ 3.6 h. If #Video counts clips, then every per-video statistic, annotation coverage figure, and benchmark denomination must be redefined, and the 'large-scale video dataset' claim is substantially weakened. If #Video truly counts videos, the 396-video selection in §3.1 is unexplained. This must be resolved before any benchmark result can be interpreted.
[§3.2, Step 2; Table 3] The clip-level captioning benchmark is circular. Section 3.2 states that GPT-4.1, Gemini-2.0 Flash-Lite, and Qwen-VL generated the clip descriptions, which were then refined by biologists. Section 3.3 says each clip has 1 to 4 descriptions, comprising 3 synthetic and 1 human-written caption. Table 3 then reports GPT-4.1 as the best clip-level captioner, with BLEU 0.7196, METEOR 0.5186, ROUGE-L 0.7844, CIDEr 4.9314, and SPICE 0.6139. Unless the reference text used in Table 3 is exclusively the independently human-written caption (not the GPT-4.1-generated or GPT-4.1-seeded captions), these scores largely measure agreement with GPT-4.1's own output. The paper does not specify which reference is used. This makes the strongest result in Table 3 uninterpretable as a measure of general captioning quality.
[§3.2, §4.2] The claimed benefit of clip splitting is asserted without quantitative support. Section 3.2 says splitting long videos into short clips 'helps enhance the semantics of the generated captions,' and the abstract and conclusion repeat this claim. However, the clip-level benchmark in Table 3 is evaluated only on clip-level references and is not compared against the video-level results in Table 2 under a matched protocol. The paper provides no experiment showing that captions produced from clips are semantically better than captions produced from whole videos, nor any human evaluation of caption quality. The claim is therefore not supported by the presented data.
[§4.4, Table 5] The text-to-video benchmark uses only 50 clip-level caption prompts, with no confidence intervals or statistical tests. The conclusions in §4.4, e.g., that 'commercial models often perform well' and that 'pre-trained T2V models exhibit suboptimal performance in FID and FVD metrics due to insufficient diversity,' are drawn from a very small sample. This is a secondary contribution, but the stated conclusions should be scaled to the evidence.

minor comments (4)

[§2] The sentence 'they rely on LLMs that often produce hallucinated outcomes human interventions' appears to be missing a phrase (e.g., 'without human interventions').
[Table 2 vs Table 3] LLaVA is dated 2024 in Table 2 and 2023 in Table 3; Qwen-VL-Chat is dated 2023 in both but the year is inconsistent in style. Please standardize the year attributions.
[§4.4] Typo: 'temporal consistancy' should be 'temporal consistency.'
[Fig. 4(c)] The region label 'Eqypt' should be 'Egypt.'

Circularity Check

2 steps flagged · score 7.0 of 10

Table 3's GPT-4.1 clip-captioning scores are circular: the reference captions are GPT-4.1's own synthetic captions (human-refined), so the high scores measure self-agreement, not independent captioning quality.

self definitional [Section 3.2 Step 2 and Section 3.3; Table 3]
"After splitting long-time videos into short-time clips, we used GPT-4.1 [2], Gemini-2.0 Flash-Lite [1], Qwen-VL [7] to generate a textual description for each clip. The generated descriptions were then refined by biologists ... Each short-time clip in our dataset was annotated with 1 to 4 text descriptions, comprising 3 synthetic and 1 human-written caption."

The Table 3 clip-level captioning benchmark scores GPT-4.1 against MSC reference captions. By the paper's own pipeline, those references are either the GPT-4.1 synthetic captions directly (one of the '3 synthetic' descriptions) or the human-written captions produced by refining those GPT-4.1-generated descriptions. The high BLEU/CIDEr/SPICE scores for GPT-4.1 therefore reflect agreement with its own output, not independent captioning quality. The reference is defined in terms of the model being scored; the human refinement does not break the dependency unless the captions were fully rewritten, which the paper does not claim.
self definitional [Section 4.1 and Table 2]
"MovieBench is based on GPT-4.1 ... Specifically, we input a list of keyframes from a video sequence, along with the images of target objects, and use GPT-4.1 to describe the observable features and behaviors of the target objects in no more than 75 words."

The video-level reference captions are the biologists' aggregation of refined clip-level descriptions, where the clip-level descriptions were originally generated by GPT-4.1 (among other LLMs) and only refined. MovieBench is therefore evaluated against a reference that is derived from GPT-4.1's own text, making the comparison non-independent in the same way as Table 3.

full rationale

The dataset construction itself — manual video filtering, segmentation masks, and clip splitting — is not circular; those are independent physical annotations. Circularity arises in the captioning benchmarks: reference captions are produced from LLM-generated text, including GPT-4.1, and GPT-4.1 (directly in Table 3, via MovieBench in Table 2) is then scored against that same text. The paper states that synthetic captions 'are employed to reduce the workload for manual captioning,' implying light refinement rather than independent human authorship. I assign score 7 because the headline benchmark claim (GPT-4.1's near-perfect scores) reduces by construction, while the dataset resource itself remains independently meaningful. I also note internal numerical inconsistencies (Section 3.1 says '396 videos' while Table 1 lists #Video 2743; Figure 1 says '13 countries' while Table 1 says '20'), but those are correctness/consistency concerns, not circularity, and do not affect this score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

This is a dataset paper; the main inputs are annotation choices and assumptions rather than fitted physical parameters. No new physical entities are introduced.

free parameters (5)

object category set = fish, reefs, aquatic plants, wrecks, human divers, sea floor
The six categories are chosen by the authors and bound the dataset's scope; grounding and captioning benchmarks depend on this label space.
video filtering criteria = clarity, complexity, diversity (subjective thresholds)
Selection of 396 of 2,743 videos relies on annotator judgment, not operationalized thresholds. This affects dataset representativeness.
frames per clip in evaluation = 10
Clip-level captioning benchmarks sample 10 frames per clip; the reported scores depend on this choice.
prompt template = maximum 75 words, background about 10 words, omitted words list
The instruction prompt is a hand-designed constraint applied to all captioning models and influences all benchmark numbers.
T2V prompt count = 50
Text-to-video generation benchmark uses 50 clip-level captions; the metrics in Table 5 are based on a small sample.

assumptions (4)

domain assumption Biologists' refinements of LLM captions constitute valid ground-truth descriptions.
Section 3.2 states biologists refine synthetic captions; the benchmark treats these as reference. No inter-annotator agreement or refinement extent is reported.
domain assumption SAM-based pseudo-masks refined by annotators are accurate enough for segmentation and grounding.
Step 1 in Section 3.2 relies on the annotation tool inheriting from SAM and [40] without quality metrics or error analysis.
ad hoc to paper The six object categories fully cover marine objects relevant to the downstream tasks.
No justification is given for excluding other classes (e.g., turtles, cephalopods), though captions may mention them.
standard math Standard captioning and grounding metrics (BLEU, METEOR, ROUGE, CIDEr, SPICE, mIoU, Recall) are meaningful for this domain.
Metrics are standard, but their validity for domain-specific captions is assumed without domain adaptation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning." pith.science (2026). https://pith.science/paper/4PQEEMLX

@misc{pith2026250804549,
  author       = {Pith},
  title        = {Pith review of: MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PQEEMLX}},
  note         = {Machine review of arXiv:2508.04549}
}

read the original abstract

Marine videos present significant challenges for video understanding due to the dynamics of marine objects and the surrounding environment, camera motion, and the complexity of underwater scenes. Existing video captioning datasets, typically focused on generic or human-centric domains, often fail to generalize to the complexities of the marine environment and gain insights about marine life. To address these limitations, we propose a two-stage marine object-oriented video captioning pipeline. We introduce a comprehensive video understanding benchmark that leverages the triplets of video, text, and segmentation masks to facilitate visual grounding and captioning, leading to improved marine video understanding and analysis, and marine video generation. Additionally, we highlight the effectiveness of video splitting in order to detect salient object transitions in scene changes, which significantly enrich the semantics of captioning content. Our dataset and code have been released at https://msc.hkustvgd.com.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 31 canonical work pages

[1]

https://gemini.google.com/

Gemini. https://gemini.google.com/

work page
[2]

https://openai.com/index/gpt-4-1/

GPT-4.1. https://openai.com/index/gpt-4-1/

work page
[3]

https://hailuoai.video

Hailuo. https://hailuoai.video

work page
[4]

https://klingai.com/

Kling 1.5. https://klingai.com/

work page
[5]

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 382–398

work page 2016
[6]

Ali Athar, Xueqing Deng, and Liang-Chieh Chen. 2024. ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation. arXiv preprint arXiv:2412.09754 (2024)

work page Pith review arXiv 2024
[7]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966 (2023)

arXiv 2023
[8]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

2005

Show all 48 references

[9]

Otto Brookes, Maksim Kukushkin, Majid Mirmehdi, Colleen Stephens, Paula Dieguez, Thurston C Hicks, Sorrel Jones, Kevin Lee, Maureen S McCarthy, Amelia Meier, et al . 2025. The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognition. arXiv ...

2025 arXiv
[10]

Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy
[11]

Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek. 2018. Ac- tor and action video segmentation from a sentence. In Proceedings of the IEEE conference on computer vision and pattern recognition . 5958–5966

2018
[12]

Shuting He and Henghui Ding. 2024. Decoupling static and hierarchical motion perception for referring video segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 13332–13341

2024
[13]

Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. 2024. Miradata: A large-scale video dataset with long durations and structured captions. Advances in Neural Information Processing Systems 37 (2024), 48955–48970

2024
[14]

Anna Khoreva, Anna Rohrbach, and Bernt Schiele. 2019. Video object segmenta- tion with language referring expressions. In Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14 . Springer, 123–141

2019
[15]

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al
[16]

Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, et al. 2021. Value: A multi-task benchmark for video-and-language understanding evaluation. arXiv preprint arXiv:2106.04632 (2021)

2021 arXiv
[17]

In Proceedings of the IEEE/CVF international conference on computer vision

Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision. 4015–4026
[18]

Shijie Lian, Ziyi Zhang, Hua Li, Wenjie Li, Laurence Tianruo Yang, Sam Kwong, and Runmin Cong. 2024. Diving into underwater: Segment anything model guided underwater salient instance segmentation and a large-scale dataset. In Proceedings of the 41st International Conference on...

2024
[19]

Shijie Lian, Hua Li, Runmin Cong, Suqi Li, Wei Zhang, and Sam Kwong. 2023. Watermask: Instance segmentation for underwater imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 1305–1315

2023
[20]

Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proceedings of the the European Conference on Computer Vision. 740–755

2014
[21]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

2004
[22]

Kun Liu, Qi Liu, Xinchen Liu, Jie Li, Yongdong Zhang, Jiebo Luo, Xiaodong He, and Wu Liu. 2025. HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation. arXiv preprint arXiv:2503.23715 (2025)

2025 arXiv
[23]

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 26296–26306

2024
[24]

Xin Ma, Yaohui Wang, Xinyuan Chen, Gengyun Jia, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. 2025. Latte: Latent diffusion transformer for video generation. Transactions on Machine Learning Research (2025)

2025
[25]

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al . 2024. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In arXiv preprint arXiv:2303.05499. 38–55

2024 arXiv
[26]

Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. 2024. VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos. arXiv preprint arXiv:2411.04923 (2024)

2024 arXiv
[27]

Kazi Sajeed Mehrab, M Maruf, Arka Daw, Abhilash Neog, Harish Babu Manoga- ran, Mridul Khurana, Zhenyang Feng, Bahadir Altintas, Yasin Bakis, Elizabeth G Campolongo, et al. 2024. Fish-vista: A multi-purpose dataset for understanding & identification of traits from images. arXiv...

2024 arXiv
[28]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics . 311–318

2002
[29]

Yulu Pan, Ce Zhang, and Gedas Bertasius. 2025. BASKET: A Large-Scale Video Dataset for Fine-Grained Skill Estimation. arXiv preprint arXiv:2503.20781 (2025)

2025 arXiv
[30]

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

2024 arXiv
[31]

Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. 2024. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pa...

2024
[32]

Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loïc Barrault, Lucia Specia, and Florian Metze. 2018. How2: a large-scale dataset for multimodal language understanding. arXiv preprint arXiv:1811.00347 (2018)

2018 arXiv
[33]

Luca Rossetto, Klaus Schoeffmann, Cathal Gurrin, Jakub Lokoč, and Werner Bailer. 2024. Results of the 2024 Video Browser Showdown. arXiv preprint arXiv:2502.15683 (2024)

2024 arXiv
[34]

Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, et al. 2024. Bioclip: A vision foundation model for the tree of life. In Proceedings of the IEEE/CVF confere...

2024
[35]

Seonguk Seo, Joon-Young Lee, and Bohyung Han. 2020. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16 . Springer, 208–223

2020
[36]

Devis Tuia. 2025. MammAlps: A multi-view video behavior monitoring dataset of wild mammals in the Swiss Alps. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition 2025

2025
[37]

Quang-Trung Truong, Tuan-Anh Vu, Tan-Sang Ha, Jakub Lokoč, Yue-Him Wong, Ajay Joneja, and Sai-Kit Yeung. 2023. Marine Video Kit: A new marine video dataset for content-based analysis and retrieval. In International Conference on Multimedia Modeling. Springer, 539–550

2023
[38]

Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition . 4566–4575

2015
[39]

Lucia Vadicamo, Rahel Arnold, Werner Bailer, Fabio Carrara, Cathal Gurrin, Nico Hezel, Xinghan Li, Jakub Lokoc, Sebastian Lubos, Zhixin Ma, et al . 2024. Evaluating performance and trends in interactive video retrieval: Insights from the 12th vbs competition. IEEE Access (2024)

2024
[40]

Yuk-Kwan Wong, Ziqiang Zheng, Mingzhe Zhang, David Suggett, and Sai-Kit Yeung. 2024. CoralSCOP-LAT: Labeling and Analyzing Tool for Coral Reef Images �� with Dense Mask. arXiv:2410.20436 [cs.CV] https://arxiv....

2024
[41]

Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, et al. 2024. Koala-36m: A large- scale video dataset improving consistency between fine-grained conditions and video content. arXiv preprint arXiv:2410.08260 (2024)

2024 arXiv
[42]

Weijia Wu, Mingyu Liu, Zeyu Zhu, Xi Xia, Haoen Feng, Wen Wang, Kevin Qinghong Lin, Chunhua Shen, and Mike Zheng Shou. 2024. MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation. arXiv preprint arXiv:2411.15262 (2024)

2024 arXiv
[43]

Weijia Wu, Yuanqiang Cai, Debing Zhang, Sibo Wang, Zhuang Li, Jiahong Li, Yejun Tang, and Hong Zhou. 2021. A bilingual, openworld video text dataset and end-to-end video text spotter with transformer. arXiv preprint arXiv:2112.04888 (2021)

2021 arXiv
[44]

Ziqiang Zheng, Yiwei Chen, Huimin Zeng, Tuan-Anh Vu, Binh-Son Hua, and Sai-Kit Yeung. 2024. Marineinst: A foundation model for marine image analysis with instance visual description. In European Conference on Computer Vision . Springer, 239–257

2024
[45]

Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994 (2024)

2024 arXiv
[46]

Ziqiang Zheng, Haixin Liang, Binh-Son Hua, Yue Him Wong, Put Ang, Apple Pui Yi Chui, and Sai-Kit Yeung. 2024. CoralSCOP: segment any coral image on this planet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 28170–28180

2024
[47]

Ziqiang Zheng, Yiwe Chen, Huimin Zeng, Tuan-Anh Vu, Binh-Son Hua, and Sai-Kit Yeung. 2024. MarineInst: A Foundation Model for Marine Image Analysis with Instance Visual Description. In Proceedings of the European Conference on Computer Vision. Springer

2024
[2023]

In Proceedings of the IEEE/CVF international conference on computer vision

MeViS: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF international conference on computer vision. 2694–2703

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

https://gemini.google.com/

Gemini. https://gemini.google.com/

work page

[2] [2]

https://openai.com/index/gpt-4-1/

GPT-4.1. https://openai.com/index/gpt-4-1/

work page

[3] [3]

https://hailuoai.video

Hailuo. https://hailuoai.video

work page

[4] [4]

https://klingai.com/

Kling 1.5. https://klingai.com/

work page

[5] [5]

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 382–398

work page 2016

[6] [6]

Ali Athar, Xueqing Deng, and Liang-Chieh Chen. 2024. ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation. arXiv preprint arXiv:2412.09754 (2024)

work page Pith review arXiv 2024

[7] [7]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.arXiv preprint arXiv:2308.12966 (2023)

arXiv 2023

[8] [8]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

2005