Pith. sign in

REVIEW 3 major objections 3 minor 2 cited by

AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs

T0 review · 3 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a GRPO-trained model, AV-Reasoner, can transfer counting ability from audio-visual QA and grounding tasks to counting benchmarks, more than doubling the base model's exact-count accuracy on DVD-Counting.

desk verdict A genuinely useful benchmark, undermined by train/eval leakage that guts the model's headline claim; worth a serious referee but needs a clean redo. read the letter →

arxiv 2506.05328 v2 pith:VLWNTSML submitted 2025-06-05 cs.CV

Lidong Lu , Guo Chen , Zhiqi Li , Yicheng Liu , Tong Lu This is my paper
classification cs.CV
keywords audio-visualcountingmultimodallargelanguagemodelsreinforcementlearningGRPOcurriculumvideobenchmarkclue-groundedreasoningtemporalgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish two things: that current multimodal LLMs cannot yet do clue-grounded audio-visual counting in long videos, and that a training recipe built on reinforcement learning can substantially close the gap. To test the first claim it builds CG-AV-Counting, 1,027 manually annotated questions with 5,845 counting clues over 497 videos longer than ten minutes, covering event, object, and attribute counting with audio-only, visual-only, and cross-modal queries. To test the second it trains AV-Reasoner, starting from the Ola-Omni-7B base model, using GRPO with curriculum learning on related tasks: audio-visual QA, temporal and spatial grounding, then counting. The result is a rise in exact-count accuracy on DVD-Counting from 16.50 to 44.00 and on CG-AV-Counting Long Acc from 17.92 to 22.30 over the base model. The paper also reports that forcing the model to emit an explicit thinking block at inference does not help and can hurt out-of-domain performance.

What carries the argument

The machinery is GRPO reinforcement learning with verifiable rewards, organized by a curriculum. Each task has a rule-based reward—format regularity for QA and counting, JSON validity plus IoU for grounding, and relative MAE for counting—so the model is optimized by environment checks rather than human preference labels. The curriculum orders tasks by difficulty (QA to grounding to counting), and the Stage Review Mechanism plus a final full-task RL stage address forgetting across stages. What carries the argument is the reward design: because every task can be scored automatically, the pipeline needs no manual reasoning annotations.

What would settle it

Retrain AV-Reasoner without DVD-Counting in any training stage, or use a clean held-out split of DVD-Counting unseen during training, and rerun the black-box evaluation. If the accuracy gain over Ola-Omni drops sharply or vanishes relative to the reported 16.50 to 44.00, the claimed generalization comes from training on the test distribution; if the gain persists, the counting transfer claim is supported.

Watch

Extended reading notes

Core claim

The central discovery is that counting ability can be bootstrapped from related groundable tasks instead of large-scale counting annotations. The model is trained in three escalating stages—question answering, grounding, and counting—with a verifiable reward for each, and a Stage Review Mechanism that mixes 20% of earlier samples to limit forgetting. This yields state-of-the-art results on MusicAVQA, LLP, UnAV, ARIG, and DVD-Counting, and the paper's ablations show that grounding-based RL, not QA RL, is the largest contributor to counting gains. A second, negative discovery is that explicit verbal reasoning in the output can hurt out-of-domain benchmarks: on the WorldSense hallucination subset accuracy drops from 45.56 to 35.56 when thinking is turned on.

Load-bearing premise

The reported headline gains assume DVD-Counting is a valid test benchmark even though its samples appear in the training set, so no evidence separates generalizable counting skill from memorization of the test distribution.

Editorial extensions

If this is right

  • If the transfer claim is right, counting ability in MLLMs can be improved without collecting large counting-specific datasets; related grounding tasks supply the needed supervision.
  • CG-AV-Counting gives the community a way to separate short-horizon counting from long-range temporal localization, since Long Acc and Ref Acc are measured separately.
  • The white-box WCS results imply that even accurate counts are rarely backed by correctly localized evidence, so model outputs that look reasoned may not be trustworthy.
  • The thinking-versus-hallucination finding implies that sampling or post-hoc selection of reasoning traces is not automatically beneficial; reasoning supervision must be tied to answer correctness.
  • Other audio-visual tasks should expect gains from the same recipe, since AV-Reasoner also reports SOTA on AVQA, LLP, UnAV, and ARIG.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If a clean DVD-Counting split were released, the +27.5 point headline would test whether the gain is memorization or transfer; the paper trains on DVD-Counting and does not provide such a split, so this is the first thing I would check.
  • Because the curriculum's ordering (QA to grounding to counting) is the load-bearing design choice, a cheaper replication could drop the final full-task RL stage while keeping the grounding stage; if grounding alone explains most of the gain, the explicit counting stage may be unnecessary.
  • The reasoning-consistency limitation the authors note suggests a concrete next experiment: train with a reward that penalizes cases where the thinking trace and final count disagree, and measure out-of-domain hallucination.
  • The near-zero white-box WCS scores across all models suggest WCS itself could serve as a diagnostic for whether future models are doing evidence-grounded counting rather than number guessing.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper introduces CG-AV-Counting, a manually annotated clue-grounded audio-visual counting benchmark with 1,027 multimodal questions and 5,845 clues over 497 long videos, together with black-box and white-box evaluation protocols. It also proposes AV-Reasoner, a model built on Ola-Omni-7B and trained with cold-start SFT, curriculum-based GRPO reinforcement learning with a stage review mechanism, and full-task RL on audio-visual QA, grounding, and counting datasets. The authors report state-of-the-art results on several audio-visual benchmarks, improved counting accuracy over the base model, and an analysis showing that explicit thinking output can hurt out-of-domain performance.

Significance. If the benchmark construction and the evaluation of the method were fully validated, the contribution would be useful: CG-AV-Counting fills a real gap by providing long-video, multimodal, clue-grounded counting evaluation with both black-box and white-box settings, and the public release of the code and benchmark supports reproducibility. The paper also contains a valuable negative result on explicit reasoning output in out-of-domain settings. However, the headline claim that AV-Reasoner achieves state-of-the-art counting results is currently not supported as presented, because its largest counting improvement is measured on DVD-Counting, which is also used as a training set. The benchmark itself may stand on its own, but the method's evidence requires re-evaluation on a clean split or on held-out data.

major comments (3)
  1. [§4.2/Table 3 vs §5.1/Table 4 and §5.2/Table 5] DVD-Counting is listed as GRPO training data in the Counting row of Table 3, but Tables 4 and 5 report DVD-Counting as an evaluation benchmark, where AV-Reasoner improves from 16.50 to 44.00 over the base model. No train/test split is described anywhere in the manuscript, and Section 5.3/Table 6 explicitly labels DVD-Counting as 'in-domain'. The +27.50 gain is therefore not evidence of generalizable counting ability; it can be explained by optimizing the rMAE reward on the test distribution. Please either enforce and document a strict split of DVD-Counting and RepCount into train and test portions, or remove DVD-Counting from the evaluation claims and re-run the comparison using only held-out benchmarks.
  2. [§3.1 and Table 2] The benchmark is described as manually annotated, but no inter-annotator agreement, number of annotators, or acceptance/refinement/rejection statistics from the three-stage pipeline are reported. Similarly, the Human (full-video) row in Table 2 is presented without any protocol description, such as whether the same annotators answered the 1,027 questions and whether the human white-box WCS score was computed with the same metric as for models. Without these details, the reliability of the 5,845 clue annotations and the claim that models 'significantly underperform human performance' cannot be independently assessed. Please add annotation agreement statistics and a clear human evaluation protocol.
  3. [§5.2/Table 5 and Abstract] Once DVD-Counting is set aside, the held-out counting evidence is limited to CG-AV-Counting, where AV-Reasoner improves Long Acc by +4.38, Ref Acc by +10.50, and WCS by +0.27 relative to Ola-Omni. These gains are much smaller than the headline DVD-Counting improvement, and the paper also shows that explicit thinking hurts several other benchmarks. The claims that the method 'generalizes counting ability from related tasks' and 'achieves state-of-the-art results across multiple benchmarks' should be restricted to what the held-out evidence supports, and additional held-out counting evaluations should be added if such claims are to be maintained.
minor comments (3)
  1. [Throughout] There are numerous typos and formatting artifacts, including 'Expirements' (Section 5), 'Modaility' and 'Conuting' (Table 1), 'coutning' (Figure 2), 'reasoing' (Figure 1), and inconsistent spacing in 'A V-Reasoner' and 'CG-A V-Counting'; a careful proofreading pass is needed.
  2. [§3.3, Eq. (1)] The WCS definition should explicitly state that for object and attribute counting the localization accuracy is computed on predefined clue frames, since this dependence is only acknowledged later in the Limitations section and affects the interpretation of all white-box object/attribute scores.
  3. [§4.4 and Table 9] The offline data filtering thresholds (five rollouts per sample, discarding samples with all-correct rollouts, discarding grounding samples with average IoU above 0.9) are presented without sensitivity analysis, and the cold-start SFT hyperparameters such as learning rate, number of epochs, and optimizer settings are not reported; these details are needed for reproducibility.

Circularity Check

2 steps flagged · score 7.0 of 10

The headline SOTA claim relies on evaluating benchmarks that are also used as training data; the +27.5 DVD-Counting gain is an in-domain training outcome, not a prediction.

  1. fitted input called prediction [Table 3, Table 5, Section 5.2, and Table 6 discussion]
    "Table 3: The datasets used during training. ... Counting DVD-Counting [16], RepCount [19]. ... Tab. 5 highlights the effectiveness of GRPO in improving counting capabilities. ... Ola-Omni (Base Model) 16.50 ... AV-Reasoner-Thinking (Ours) 44.00 ... (+27.50). ... SFT improves in-domain DVD-Counting [16] accuracy but suffers from overfitting, leading to degraded out-of-domain performance."

    DVD-Counting is explicitly listed as the Counting training dataset for the cold-start SFT, curriculum-based RL, and full-task RL pipeline, with the rMAE reward (Eq. 2) computed against DVD-Counting ground truth. Tables 4 and 5 then report DVD-Counting accuracy as a benchmark, and the +27.50 improvement over Ola-Omni is used as evidence that GRPO generalization improves counting. The paper never documents a train/test split for DVD-Counting, and the ablation text itself calls DVD-Counting 'in-domain.' The reported gain is therefore a training-set outcome, making the abstract's 'state-of-the-art results across multiple benchmarks' and 'effectiveness of reinforcement learning' claims substantially circular as written.

  2. fitted input called prediction [Table 3, Table 4, Section 4.2, Section 5.1]
    "we use original annotations from datasets like AVQA [49], MUSIC-AVQA [50], AVE [51], UnAV [52], DVD-Counting [16], and RepCount [19]. ... AV-Reasoner-Thinking achieves an accuracy of 85.01 on MusicAVQA [50] ... 64.70 (segment-level) and 62.00 (event-level) on LLP [53], and 65.18 mAP on UnAV [52] ... 46.73 cIoU and 0.49 AUC on ARIG [54]."

    The same named datasets used for training (Music-AVQA, LLP, UnAV, AVSS-ARIG, AVE) are then reported as SOTA evaluation benchmarks in Table 4 and Table 10. No train/test split is stated for any of these datasets, so the reported accuracy, mAP, cIoU, and AUC numbers are not demonstrably independent of the data whose GRPO rewards were optimized. These 'state-of-the-art' claims are thus consistent with scoring the model on its own training distribution, further weakening the paper's central claim that the method generalizes counting ability.

full rationale

Two concrete train/eval overlaps make the central method claim circular in its reported form. First, DVD-Counting appears both in Table 3 as the Counting training set and in Tables 4 and 5 as a benchmark, with the +27.5 gain over Ola-Omni presented as the flagship improvement; the ablation even labels DVD-Counting 'in-domain.' Second, Music-AVQA, LLP, UnAV, and ARIG are trained on and then reported as SOTA benchmarks without any stated split. The CG-AV-Counting benchmark itself, with its manual clue annotations and white-box protocol, is an independent contribution and is not in the training set; WorldSense, AV-Odyssey, and OmniBench also provide some held-out signal. However, the abstract's headline claims of state-of-the-art performance 'across multiple benchmarks' and the effectiveness of reinforcement learning rest substantially on the contaminated training benchmarks, so the central claim is not supported as written. The benchmark contribution may stand on its own, but the method's headline effectiveness claim requires a documented clean split to be credible.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim rests on annotation quality, a transfer assumption, and valid evaluation. The DVD-Counting leakage is the largest burden: the same dataset is used for training and reported as a benchmark result.

free parameters (5)
  • KL penalty coefficient beta = 0.1
    Section B.1 sets the KL divergence penalty beta to 0.1 in GRPO; the balance between exploration and staying close to the base policy depends on this choice.
  • Rollout count per sample = 8
    Section B.1 states each sample is expanded into 8 rollout trajectories; this affects reward estimation, data filtering, and difficulty-based sampling.
  • Stage Review Mechanism mixing ratio = 20%
    Section 4.4 mixes 20% of previously seen samples into later curriculum stages to reduce forgetting; Table 7 shows this ratio changes results.
  • Offline data filtering thresholds = Discard all-correct QA/counting rollouts; discard grounding rollouts with average IoU above 0.9
    Section 4.4 describes filtering based on five rollouts from the reference model; these thresholds shape the training distribution and are not varied in the paper.
  • Curriculum epochs per level = 2 per level
    Section 4.4 trains each curriculum level for two epochs; no sensitivity analysis is provided for this choice.
assumptions (3)
  • domain assumption Benchmark annotations (answers and clues) are correct.
    The paper relies on human annotators and Gemini-generated proposals with no inter-annotator agreement or third-party audit (Section 3.1); if ground truth is wrong, all measured accuracies change.
  • domain assumption Training on related audio-visual tasks transfers to counting.
    Section 4.1 states the model acquires counting ability through task-related exploration; this transfer is assumed and only weakly tested on the held-out CG-AV-Counting benchmark.
  • ad hoc to paper DVD-Counting accuracy is a valid out-of-sample evaluation.
    Tables 3 and 4 use DVD-Counting in both training and evaluation with no stated split; the evaluation assumes no leakage, but the paper provides no evidence for this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs." pith.science (2026). https://pith.science/paper/VLWNTSML

@misc{pith2026250605328,
  author       = {Pith},
  title        = {Pith review of: AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLWNTSML}},
  note         = {Machine review of arXiv:2506.05328}
}
read the original abstract

Despite progress in video understanding, current MLLMs struggle with counting tasks. Existing benchmarks are limited by short videos, close-set queries, lack of clue annotations, and weak multimodal coverage. In this paper, we introduce CG-AV-Counting, a manually-annotated clue-grounded counting benchmark with 1,027 multimodal questions and 5,845 annotated clues over 497 long videos. It supports both black-box and white-box evaluation, serving as a comprehensive testbed for both end-to-end and reasoning-based counting. To explore ways to improve model's counting capability, we propose AV-Reasoner, a model trained with GRPO and curriculum learning to generalize counting ability from related tasks. AV-Reasoner achieves state-of-the-art results across multiple benchmarks, demonstrating the effectiveness of reinforcement learning. However, experiments show that on out-of-domain benchmarks, reasoning in the language space fails to bring performance gains. The code and benchmark have been released on https://av-reasoner.github.io.

Figures

Figures reproduced from arXiv: 2506.05328 by the authors.

Figure 1
Figure 1. Overview of CG-AV-Counting benchmark and AV-Reasoner baseline. Our benchmark covers three types of counting targets, providing end-to-end counting evaluation based on black-box testing and reasoing counting evaluation based on white-box testing, which can comprehensively evaluate the accuracy and interpretability of the model counts.AV-Reasoner uses GRPO algorithms based on course learning for intensive fine-tuning,… view at source ↗
Figure 2
Figure 2. A Three-Stage Data Annotation Pipeline for CG-AV-Counting. It is designed to improve both annotation efficiency and accuracy. V (637) A (13) V2A (100) A2V (134) AV 143 Question (1027) Object (768) Event (245) Attribute (14) Question (1027) 0 20 40 60 80 100 120 140 160 180 200 1 3 5 7 9 11 13 15 17 19 21 23 25 N u m ber of Q uestio ns Count Number ≥25 0 50 100 150 200 250 5 10 15 20 25 30 35 40 45 50 55 60 N u m b e… view at source ↗
Figure 3
Figure 3. Statistics of Our Proposed Benchmark. Our benchmark features diverse query modalities and a wide range of counting targets. The count distribution follows a long-tail pattern, providing a challenging testbed for evaluating models’ generalization and compositional reasoning abilities. 3.2 Dataset Statistics As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Overview of Our Training Strategy. Our strategy consists of three stages. During the curriculum-based RL stage, we introduce a Stage Review Mechanism, which mitigates forgetting of previously trained tasks by mixing in a proportion of earlier samples. answers in a pred…
Figure 5
Figure 5. Figure 5: Example Ouputs on Various Tasks. AV-Reasoner not noly outputs the correct answer, but also give the thinking process. 5.2 Comparison with Base Model on Counting Tasks Tab. 5 highlights the effectiveness of GRPO in improving counting capabilities. Compared to the base m…
Figure 6
Figure 6. Figure 6: Statistics of Video Content Categories. 0 5 10 15 20 25 30 10 12 14 16 18 2 0 2 2 2 4 2 6 2 8 3 0 3 2 3 4 3 6 3 8 4 0 4 2 4 4 4 6 4 8 5 0 5 2 5 5 5 7 5 9 N u m b e r o f Vid e os Duration (minutes) [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Statistics of Video Duration. - **A2V (Audio to Visual)**: The question is triggered by an **audio cue** (e.g., a sound, noise, or dialogue) which helps the model **identify the relevant time segment** in the video. Once the time segment is located using the audio cue,…
Figure 8
Figure 8. Figure 8: Model accuracy across different count targets. [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Model accuracy across different count ranges. [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Model accuracy across different query modalities. [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Model’s White-box Evaluation Performance across Different Counting Targets. [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Empowering Long-form Omni-modal Understanding with Robust Audio Perception

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Decoupled audio-visual caption and CoT-QA datasets plus two-stage fine-tuning measurably strengthen auditory perception and cross-modal reasoning in a 7B omni-modal LLM.

  2. SVCBench: A Streaming Video Counting Benchmark for Spatial-Temporal State Maintenance

    cs.CV 2026-03 accept novelty 6.0 of 10

    Streaming multi-point counting on 406 videos with three trajectory metrics reveals large human-model gaps in spatial-temporal state maintenance, worst on periodic events.

Reference graph

Works this paper leans on

71 extracted references · 24 canonical work pages · cited by 2 Pith papers

  1. [1]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  2. [2]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    arXiv 2025

  3. [3]

    Eagle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    Guo Chen, Zhiqi Li, Shihao Wang, Jindong Jiang, Yicheng Liu, Lidong Lu, De-An Huang, Wonmin Byeon, Matthieu Le, Tuomas Rintamaki, et al. Eagle 2.5: Boosting long-context post-training for frontier vision-language models.arXiv preprint arXiv:2504.15271, 2025

    arXiv 2025

  4. [4]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    arXiv 2025

  5. [5]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 - System Demonstrations, Singapore, December 6-10, 2023, pages 543–553. Association for Computational Linguistics, 2023

    work page 2023

  6. [6]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.arXiv preprint arXiv:2406.07476, 2024

    arXiv 2024

  7. [7]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    arXiv 2023

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023

    arXiv 2023

Show all 71 references
  1. [9]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites.arXiv preprint arXiv:2404.16821, 2024

    2024 arXiv

  2. [10]

    Internvideo: General video foundation models via generative and discriminative learning.CoRR, abs/2212.03191, 2022

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative lear...

    2022 arXiv

  3. [11]

    Internvideo2: Scaling video foundation models for multimodal video understanding.arXiv preprint arXiv:2403.15377, 2024

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding.arXiv preprint arXiv:2403.15377, 2024

    2024 arXiv

  4. [12]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499, 2024

    2024

  5. [13]

    Cogvlm2: Visual language models for image and video understanding.arXiv preprint arXiv:2408.16500, 2024

    Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. Cogvlm2: Visual language models for image and video understanding.arXiv preprint arXiv:2408.16500, 2024

    2024 arXiv

  6. [14]

    Videollm: Modeling video sequence with large language models

    Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305.13292, 2023

    2023 arXiv

  7. [15]

    Ovr: A dataset for open vocabulary temporal repetition counting in videos.arXiv preprint arXiv:2407.17085, 2024

    Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, and Andrew Zisserman. Ovr: A dataset for open vocabulary temporal repetition counting in videos.arXiv preprint arXiv:2407.17085, 2024

    2024 arXiv

  8. [16]

    Dvd-counting, 2025

    Video-R1. Dvd-counting, 2025. URL https://huggingface.co/datasets/Video-R1/ DVD-counting. 12

    2025

  9. [17]

    Counting out time: Class agnostic video repetition counting in the wild

    Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Counting out time: Class agnostic video repetition counting in the wild. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10387–10396, 2020

    2020

  10. [18]

    Repetitive activity counting by sight and sound

    Yunhua Zhang, Ling Shao, and Cees GM Snoek. Repetitive activity counting by sight and sound. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14070–14079, 2021

    2021

  11. [19]

    Transrac: Encoding multi-scale temporal correlation with transformers for repetitive action counting.arXiv preprint arXiv:2204.01018, 2022

    Huazhang Hu, Sixun Dong, Yiqun Zhao, Dongze Lian, Zhengxin Li, and Shenghua Gao. Transrac: Encoding multi-scale temporal correlation with transformers for repetitive action counting.arXiv preprint arXiv:2204.01018, 2022

    2022 arXiv

  12. [20]

    Cg-bench: Clue-grounded question answering benchmark for long video understanding.arXiv preprint arXiv:2412.12075, 2024

    Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang, Tong Lu, and Limin Wang. Cg-bench: Clue-grounded question answering benchmark for long video understanding.arXiv preprint arXiv:2412.12075, 2024

    2024 arXiv

  13. [21]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04328, 2025

    2025 arXiv

  14. [22]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  15. [23]

    Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020

    Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey.Journal of Machine Learning Research, 21(181):1–50, 2020

    2020

  16. [24]

    Anygpt: Unified multimodal llm with discrete sequence modeling.arXiv preprint arXiv:2402.12226, 2024

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with discrete sequence modeling.arXiv preprint arXiv:2402.12226, 2024

    2024 arXiv

  17. [25]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. InProceedings of the IEEE/CVF Conference on Computer Vision and...

    2024

  18. [26]

    Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    Zhifei Xie and Changqiao Wu. Mini-omni2: Towards open-source gpt-4o with vision, speech and duplex capabilities.arXiv preprint arXiv:2410.11190, 2024

    2024 arXiv

  19. [27]

    Omcat: Omni context aware transformer.arXiv preprint arXiv:2410.12109, 2024

    Arushi Goel, Karan Sapra, Matthieu Le, Rafael Valle, Andrew Tao, and Bryan Catanzaro. Omcat: Omni context aware transformer.arXiv preprint arXiv:2410.12109, 2024

    2024 arXiv

  20. [28]

    Baichuan-omni-1.5 technical report.arXiv preprint arXiv:2501.15368, 2025

    Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al. Baichuan-omni-1.5 technical report.arXiv preprint arXiv:2501.15368, 2025

    2025

  21. [29]

    Crab: A unified audio-visual scene understanding model with explicit cooperation.arXiv preprint arXiv:2503.13068, 2025

    Henghui Du, Guangyao Li, Chang Zhou, Chunjie Zhang, Alan Zhao, and Di Hu. Crab: A unified audio-visual scene understanding model with explicit cooperation.arXiv preprint arXiv:2503.13068, 2025

    2025 arXiv

  22. [30]

    video-salmonn: Speech-enhanced audio-visual large language models.arXiv preprint arXiv:2406.15704, 2024

    Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models.arXiv preprint arXiv:2406.15704, 2024

    2024 arXiv

  23. [31]

    Meerkat: Audio-visual large language model for grounding in space and time

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. InEuropean Conference on Computer Vision, pages 52–70. Springer, 2024

    2024

  24. [32]

    Pave: Patching and adapting video large language models, 2025

    Zhuoming Liu, Yiquan Li, Khoi Duc Nguyen, Yiwu Zhong, and Yin Li. Pave: Patching and adapting video large language models, 2025. URLhttps://arxiv.org/abs/2503.19794

    2025 arXiv

  25. [33]

    Needle in a video haystack: A scalable synthetic evaluator for video mllms.arXiv preprint arXiv:2406.09367, 2024

    Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in a video haystack: A scalable synthetic evaluator for video mllms.arXiv preprint arXiv:2406.09367, 2024

    2024 arXiv

  26. [34]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–222...

    2024

  27. [35]

    Worldsense: Evaluat- ing real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluat- ing real-world omnimodal understanding for multimodal llms.arXiv preprint arXiv:2502.04326, 2025

    2025 arXiv

  28. [36]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  29. [37]

    Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning.arXiv preprint arXiv:2503.01785, 2025

    2025 arXiv

  30. [38]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  31. [39]

    Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  32. [40]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  33. [41]

    Introducing gpt-4.1 in the api, 2025

    OpenAI. Introducing gpt-4.1 in the api, 2025. URL https://openai.com/index/gpt-4-1 /

    2025

  34. [42]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  35. [43]

    Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Jiawei Wang, et al. Seed1. 5-vl technical report.arXiv preprint arXiv:2505.07062, 2025

    2025 arXiv

  36. [44]

    Eagle 2: Building post-training data strategies from scratch for frontier vision-language models.arXiv preprint arXiv:2501.14818, 2025

    Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, et al. Eagle 2: Building post-training data strategies from scratch for frontier vision-language models.arXiv preprint arXiv:2501.14818, 2025

    2025 arXiv

  37. [46]

    Internvideo2.5: Empowering video mllms with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. Internvideo2.5: Empowering video mllms with long and rich context modeling.arXiv prepri...

    2025 arXiv

  38. [47]

    Videochat-flash: Hierarchical compression for long-context video modeling.arXiv preprint arXiv:2501.00574, 2024

    Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flash: Hierarchical compression for long-context video modeling.arXiv preprint arXiv:2501.00574, 2024

    2024 arXiv

  39. [48]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215, 2025

    2025 arXiv

  40. [49]

    Avqa: A dataset for audio-visual question answering on videos

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. InProceedings of the 30th ACM international conference on multimedia, pages 3480–3491, 2022

    2022

  41. [50]

    Learning to answer questions in dynamic audio-visual scenarios

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108–19118, 2022

    2022

  42. [51]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. InECCV, 2018

    2018

  43. [52]

    Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline

    Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22942–22951, 2023. 14

    2023

  44. [53]

    Cross-modal learning for audio-visual video parsing

    Jatin Lamba, Abhishek, Jayaprakash Akula, Rishabh Dabral, Preethi Jyothi, and Ganesh Ramakrishnan. Cross-modal learning for audio-visual video parsing. 2021. URL https: //arxiv.org/abs/2104.04598

    2021 arXiv

  45. [54]

    Audio-visual segmentation with semantics.International Journal of Computer Vision, pages 1–21, 2024

    Jinxing Zhou, Xuyang Shen, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong. Audio-visual segmentation with semantics.International Journal of Computer Vision, pages 1–21, 2024

    2024

  46. [55]

    Enhancing geometric factors in model learning and inference for object detection and instance segmentation.IEEE transactions on cybernetics, 52(8):8574–8586, 2021

    Zhaohui Zheng, Ping Wang, Dongwei Ren, Wei Liu, Rongguang Ye, Qinghua Hu, and Wang- meng Zuo. Enhancing geometric factors in model learning and inference for object detection and instance segmentation.IEEE transactions on cybernetics, 52(8):8574–8586, 2021

    2021

  47. [56]

    URL https: //www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf

    Anthropic.The Claude 3 Model Family: Opus, Sonnet, Haiku Anthropic. URL https: //www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf

  48. [57]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26584–26...

    2024

  49. [58]

    Groundinggpt: Language enhanced multi-modal grounding model.arXiv preprint arXiv:2401.06071, 2024

    Zhaowei Li, Qi Xu, Dong Zhang, Hang Song, Yiqing Cai, Qi Qi, Ran Zhou, Junting Pan, Zefeng Li, Van Tu Vu, et al. Groundinggpt: Language enhanced multi-modal grounding model.arXiv preprint arXiv:2401.06071, 2024

    2024 arXiv

  50. [59]

    Valor: Vision-audio-language omni-perception pretraining model and dataset.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, and Jinhui Tang. Valor: Vision-audio-language omni-perception pretraining model and dataset.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  51. [60]

    X-instructblip: A framework for aligning image, 3d, audio, video to llms and its emergent cross-modal reasoning

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning image, 3d, audio, video to llms and its emergent cross-modal reasoning. InEuropean Conference on ...

    2024

  52. [61]

    Avicuna: Audio-visual llm with interleaver and context-boundary alignment for temporal referential dialogue.arXiv e-prints, pages arXiv–2403, 2024

    Yunlong Tang, Daiki Shimada, Jing Bi, and Chenliang Xu. Avicuna: Audio-visual llm with interleaver and context-boundary alignment for temporal referential dialogue.arXiv e-prints, pages arXiv–2403, 2024

    2024

  53. [62]

    Av-odyssey bench: Can your multimodal llms really understand audio-visual information?arXiv preprint arXiv:2412.02611, 2024

    Kaixiong Gong, Kaituo Feng, Bohao Li, Yibing Wang, Mofan Cheng, Shijia Yang, Jiaming Han, Benyou Wang, Yutong Bai, Zhuoran Yang, et al. Av-odyssey bench: Can your multimodal llms really understand audio-visual information?arXiv preprint arXiv:2412.02611, 2024

    2024 arXiv

  54. [63]

    Omnibench: Towards the future of universal omni-language models.arXiv preprint arXiv:2409.15272, 2024

    Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Zekun Wang, Jian Yang, et al. Omnibench: Towards the future of universal omni-language models.arXiv preprint arXiv:2409.15272, 2024

    2024

  55. [64]

    Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone.arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  56. [65]

    How many people spoke in the scene showing the conference table?

    Jiashuo Yu, Ying Cheng, Rui-Wei Zhao, Rui Feng, and Yuejie Zhang. Mm-pyramid: Multi- modal pyramid attentional network for audio-visual event localization and video parsing. In Proceedings of the 30th ACM international conference on multimedia, pages 6241–6249, 2022. 15 A Benc...

    2022

  57. [66]

    Each question must correspond to a **specific and bounded video segment**

  58. [67]

    The **answer must be a count** (e.g., number of visual objects, number of auditory events, number of multimodal occurrences, etc.)

  59. [68]

    The answer must be **clearly determined** and not ambiguous within the video span

  60. [69]

    The difficulty of the question should be specified based on perceptual complexity (e.g., occlusion, background noise, overlapping motion/sound)

  61. [70]

    Include a **diverse mix** of A2V, V2A, AV, A, and V questions in the output

  62. [71]

    Avoid redundant or trivial questions

  63. [72]

    question

    The question should not include specific timepoints. ### **Output Format** Return your result as a **JSON array**, where each entry is a dictionary with the following fields: 17 - ‘"question"‘: A clear and specific counting question. - ‘"type"‘: One of ‘"A2V"‘, ‘"V2A"‘, ‘"AV"‘...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.