Pith. sign in

REVIEW 4 major objections 5 minor 75 references

AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dense question-answer rewards sharpen audio-video captions to match proprietary models.

desk verdict A solid dataset-and-RL package with real gains on external benchmarks, but the headline AVCap-Score metric is not yet validated independently, so the 'strict atomic-level' claims should be taken as provisional. read the letter →

arxiv 2608.06930 v1 pith:F25O6XWA submitted 2026-08-07 cs.CV

Mingyang Wu , Kaituo Feng , Bohao Li , Kaixiong Gong , Zihao Yin , Xiangyu Yue This is my paper · ORCID
classification cs.CV
keywords audio-videocaptioningdetail-awarerewardGRPOatomic-levelevaluationvideodatasetmultimodallargelanguagemodelsaudiovisualalignmentcaptionqualitymetric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AVCap argues that fine-grained audio-video captioning is limited by reward and evaluation granularity as much as by data scarcity. To break that limit, it builds AVCap-100K, a 100K-video dataset of temporally aligned, detail-rich captions, and trains captioners with a new reward, Da-GRPO, that grades captions on atomic visual, audio, and joint facts instead of a single holistic score. It also proposes AVCap-Bench and AVCap-Score, a 1,000-video benchmark and a probe-based metric that measures whether atomic facts survive in a generated caption. The paper reports that the trained AVCap-30B reaches 56.94 on AVCap-Bench, 85.1 on UGC-VideoCap, and 32.7 on Video-SALMONN-2, matching or surpassing Gemini-2.5-Pro on several evaluations. If right, it gives the community a dataset, reward design, and evaluation standard for detailed captioning.

What carries the argument

The load-bearing mechanism is the Da-GRPO reward: a judge model raises N atomic probes from the ground-truth caption, answers each probe from the candidate caption alone, and scores semantic similarity against ground-truth answers; the normalized average becomes the reward. This turns caption quality into a dense verification problem rather than a single holistic judgment. The same probe-and-score protocol is reused as AVCap-Score, making the training reward and the evaluation metric share the same atomic fact representation.

What would settle it

Inject controlled factual errors into captions, such as changing an instrument, swapping a color, or shifting an audio-visual timing, and compare AVCap-Score deltas with human error counts; if score changes do not monotonically track the number and severity of injected errors, the atomic-factuality claim of the metric fails.

Watch

Extended reading notes

Core claim

The central claim is that replacing coarse, event-level reward signals with a dense, atomic-level verification signal materially improves factual detail in audio-video captioning. Da-GRPO works by having a judge model raise a set of granular questions about the ground-truth caption, split into visual, audio, and audio-visual joint facts, answer those questions using only the generated caption as context, and then grade the semantic similarity of the answers. The averaged similarity across the probe set is the reward fed into group-relative policy optimization. The paper contends that this Raise-Answer-Check reward is what lets a 7B model surpass existing open-source baselines and a 30B model match or beat Gemini-2.5-Pro on several evaluations, and that the same probe protocol, applied as AVCap-Score, gives the first atomic-level evaluation standard for audiovisual captions.

Load-bearing premise

The metric's validity rests on the judge model scoring semantic similarity in a way that tracks factual alignment rather than stylistic similarity to the judge's own preferred captions, and the paper does not validate this assumption against human judgments of the metric itself.

Editorial extensions

If this is right

  • A 7B audio-video model trained on AVCap-100K with SFT alone surpasses prior open-source baselines, separating the dataset's contribution from the RL stage's.
  • Adding Da-GRPO to the SFT checkpoint improves all reported benchmarks at both 7B and 30B scales, showing that reward granularity, not just data volume, drives fine-grained caption quality.
  • Because AVCap-Score grades recoverable atomic facts, captioners trained with it should show fewer hallucinations and omissions on details such as colors, instrument names, accents, and audio-visual synchronization.
  • The released dataset and metric give future work a shared target: a 100K dense-caption resource and a reproducible judge-based evaluation protocol.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the judge model both produced the training captions and grades the benchmark answers, part of the measured gain may reflect style matching to the judge's preferred caption form; a human study comparing AVCap-Score against human factual-error counts would tell how much of the gain is truly factual.
  • The atomic QA reward is a general recipe: any generator whose output can be probed by question-answer fact checks could be trained with the same Raise-Answer-Check loop, including image captioning, long-form video grounding, or fact-checking pipelines.
  • The 100K captions, with mean length around 416 words and explicit audio-visual alignment, could double as conditioning data for text-to-video generation, where dense joint descriptions are known to help controllability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AVCap, a framework for detailed audio-video joint captioning, comprising three contributions: (1) AVCap-100K, a 100K-video dataset of temporally aligned, detail-rich captions generated via a multi-stage pipeline with explicit visual/audio disentanglement and rating-based filtering; (2) Da-GRPO, a GRPO variant whose reward is built from atomic QA probes derived from ground-truth captions, with the judge model answering each probe from the candidate caption and scoring semantic similarity to the gold answer; and (3) AVCap-Bench and AVCap-Score, a benchmark and metric that evaluate atomic-level factual preservation using the same probe-and-verify paradigm. The authors claim that AVCap-7B-SFT outperforms open-source baselines of similar scale and that AVCap-30B reaches 56.94 on AVCap-Bench, 85.1 on UGC-VideoCap, and 32.7 on Video-SALMONN-2, remaining competitive with Gemini-2.5-Pro on several evaluations.

Significance. If the core claims hold, the framework is valuable to the community: it releases a large-scale fine-grained audio-visual caption dataset, a dense reward design that appears to improve factual detail over coarse rewards, and an atomic-level evaluation metric. The paper is commendably reproducible, with code, models, and datasets released, and it provides controlled ablations (Tables 3 and 4) showing consistent gains on both a 7B dense model and a 30B-A3B MoE model. The external benchmarks (Video-SALMONN-2, UGC-VideoCap, DailyOmni, WorldSense) and the human evaluation in Appendix B.5 offer independent support for the model's overall quality. The main unresolved issue is the validity of AVCap-Score as an independent measure of fine-grained factual fidelity, which is load-bearing for the paper's central claim that Da-GRPO 'strictly measures' atomic-level detail preservation.

major comments (4)
  1. [Section 3.3 and Eq. (3)] AVCap-Score is both the training reward (Eq. 3) and the headline evaluation metric (Section 3.3), and both use judge models from the Qwen3 family that also generated the ground-truth captions in AVCap-100K. The paper does not provide a negative control showing that the judge's answer to a probe is actually grounded in the provided caption. The audit in Table A6 checks 100 judge answers where the fact is present (1% error rate), but it does not test the case where the caption omits or contradicts the fact, which is exactly the case where a prior-driven judge would still produce a high score. The claim in Section 3.2 that the reward 'strictly measures whether the predicted caption accurately preserves the fine-grained visual and auditory details' is therefore not yet established; a style-matching or judge-prior explanation remains plausible. I recommend adding a controlled perturbation study (e.g., removing or altering the fact-bearing sentence in the caption and measuring whether the probe score drops) and, if possible, correlating AVCap-Score with human judgments of atomic fact preservation on a held-out subset.
  2. [Appendix C.2, Table A9] The alternative-judge experiment swaps only the scoring model (Gemini-3.1-Pro and GPT-5.4), but question generation and answer extraction still use Qwen3-30B-A3B. The probe questions are therefore generated by a model from the same family as the trained model and as the data-generation model, and the ranking of Qwen-family models may still be inflated by shared stylistic priors. While the relative ranking of AVCap-30B over baselines is preserved under the alternative judges, this experiment does not address the core circularity concern because the full probe-generation pipeline is unchanged. To properly test for judge-family bias, the question generation and answer-extraction steps should also be swapped, or the metric should be validated against human-annotated atomic facts on the same 1,000-video benchmark.
  3. [Section 3.2, Eq. (1)-(3) and Appendix A.4, Table A4] There is an inconsistency between the description of the reward model and the evaluation judge. Section 3.2 refers generically to the 'Judge Model Mjudge' for reward computation, while Appendix A.4, Table A4 specifies that the reward node in Da-GRPO uses Qwen3-14B with thinking disabled. Evaluation in Section 3.3, in contrast, uses Qwen3-30B-A3B-Instruct as the judge. The paper does not discuss whether this reward/evaluation judge mismatch affects the alignment between the training target and the evaluation metric. This is a concrete reproducibility and interpretation issue, and the authors should either use the same judge for both or report ablations showing that the choice of reward judge does not materially change the results.
  4. [Section 3.3, Question Generation Principles and Score Calculation] The paper states that AVCap-Score 'directly measures whether the atomic fact queried by qk is recoverable from the generated caption,' but no experiment directly tests recoverability. The current evidence consists of a manual audit of 100 generated probe-answer pairs (Table A6) and the human evaluation (Appendix B.5), but the latter scores agreement with the ground-truth caption, not the metric's sensitivity to fact omission. I recommend adding a sensitivity analysis where candidate captions are systematically perturbed (e.g., deleting audio facts, visual facts, or joint claims) and showing that AVCap-Score drops accordingly. This would substantiate the 'strict' claim and would also make the metric more useful to the community.
minor comments (5)
  1. [Section 1] Typo: 'exsisting' should be 'existing' in the third paragraph of the Introduction.
  2. [Appendix B.6.1] Typo: 'insted' should be 'instead' in the Visual Filtering User Prompt.
  3. [Table 2 caption] The caption says 'accuracy of atomic fact verification,' but the scores are semantic-similarity scores, not binary accuracy. Please rephrase to avoid confusion.
  4. [Section 3.3] The notation 'Qv', 'Qa', 'Qav' is used for question subsets, but later 'Q' is reused for the full probe set. Consider using distinct symbols (e.g., 'Q_v', 'Q_a', 'Q_av', and 'Q_total') to reduce ambiguity.
  5. [Appendix C.3.5, Eq. (6)] The equation writes 'AVCap-Score = Σ Score_i = 87.0', but the example in Table A10 sums the 20 per-question scores. The relationship between these per-question scores and the modality-normalized sub-scores defined in Section 3.3 is not explicitly shown; adding a one-line derivation would improve reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

AVCap-Bench is the Da-GRPO training objective relabeled as an evaluation metric; external benchmarks and the human study carry the independent evidence.

  1. fitted input called prediction [Section 3.3 ('AVCap-Bench and AVCap-Score'), compared with Section 3.2 Eqs. (1)-(3); results in Tables 2 and 4.]
    "AVCap-Score decomposes the evaluation into specific atomic facts, mirroring the granularity of our training objective, following protocols of multimodal post-training works that directly optimize benchmark-aligned core metrics as rewards [45–47]. ... For a candidate caption Cˆ, the same judge predicts aˆk = Mjudge(Answer|q_k, Cˆ), and then assigns a semantic similarity score sk ∈[0,5] by comparing aˆk against agt_k with the Judge Model."

    In Da-GRPO the per-caption reward is r_i = (1/(N·Smax)) Σ_k s_i,k, where s_i,k is the judge's semantic similarity between the answer extracted from the candidate caption (Eq. 1) and the answer extracted from the ground truth. AVCap-Score is Σ_k s_k computed with the same judge (Qwen3-30B-A3B), the same ground-truth-derived QA set, the same caption-only answer step, and the same 0–5 rubric. Thus AVCap-Bench scores in Tables 2–4 are the training reward up to N normalization: the 56.94 vs 53.03 Da-GRPO-over-SFT gain measures how well the policy learned to satisfy this exact judge, not an independent assessment of caption detail. The paper itself states the metric 'mirrors' the training objective and cites works that directly optimize benchmark-aligned metrics as rewards.

full rationale

The clearest circular step is that AVCap-Score, the paper's proposed atomic-level metric, is the same function Da-GRPO maximizes during RL: both use Qwen3-30B-A3B to generate questions from the ground-truth caption, answer them from the candidate caption as sole context, and score the predicted answer against the ground-truth answer. The paper explicitly says the metric 'mirrors' the training objective and cites prior works that optimize benchmark-aligned core metrics as rewards, so the AVCap-Bench improvements are largely a measure of reward optimization rather than independent validation. This does not invalidate the whole paper: gains on Video-SALMONN-2, UGC-VideoCap, DailyOmni, WorldSense, and the 100-sample human evaluation in Appendix B.5 are external to the fitted reward and provide independent support. I did not find load-bearing self-citations or imported uniqueness theorems; references [45–47] are not the authors' own work. The separate concern that the judge may answer from priors rather than from the caption is a construct-validity risk for the 'strictly measures' claim, but it is not a construction-level equation reduction, so it is not scored as a second circular step. Overall, one central home-benchmark prediction reduces by construction, while the framework retains independent content elsewhere, giving a partial circularity score of 6.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework's load-bearing assumptions are model-fidelity and judge-validity assumptions: the same AI family generates the training captions, the reward signal, the benchmark ground truths, and the evaluation grades. The GRPO optimizer itself is standard.

free parameters (4)
  • Reward/eval probe count N = 20
    Selected as 'empirical optimal' from Figure 6, which plots AVCap-Score against N; this tunes the training reward to the paper's own metric.
  • Question distribution 5 visual / 5 audio / 10 joint = 5/5/10
    Hand-chosen in AVCap-Score question generation (Appendix C.3.1) and mirrored in Da-GRPO probes; affects balance of sub-scores and reward.
  • Semantic similarity scale Smax = 5
    Scale choice for judge scores in Eq. 2; arbitrary, affects reward magnitude but not ranking much.
  • Rating-based filtering thresholds = unspecified
    Section 3.1 says only samples achieving high confidence scores are retained, but thresholds for visual/audio/joint filter scores are not given; this is a hand-chosen quality bar.
assumptions (5)
  • domain assumption Qwen3-Omni-Thinking produces accurate visual, vocal, BGM, and joint captions used as ground truth for training and evaluation.
    The entire AVCap-100K and AVCap-Bench rest on this generator's fidelity; only 1,000 samples are manually audited, with hallucination rate below 4%.
  • domain assumption A judge model's 0-5 semantic similarity score between predicted and ground-truth answers is a valid proxy for factual correctness of a caption.
    Eq. 2 and the AVCap-Score grading prompt assume semantic similarity tracks truth; no validation against human judgments is reported for the score itself.
  • domain assumption Ground-truth captions in AVCap-Bench, checked by five annotators, are complete and free of cross-modal errors.
    Appendix B.4 says captions with unresolved conflicts are removed, but completeness is not independently measured.
  • standard math GRPO's group-relative advantage estimator with KL constraint improves policy under the dense reward.
    Adopted from DeepSeekMath [16] and prior RL works; treated as given in Eq. 5.
  • domain assumption The 5/5/10 question split covers the space of atomic facts representatively.
    Question generation principles in Section 3.3 are qualitative; no analysis that 20 questions per caption saturate information content beyond the N-scaling curve.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward." pith.science (2026). https://pith.science/paper/F25O6XWA

@misc{pith2026260806930,
  author       = {Pith},
  title        = {Pith review of: AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F25O6XWA}},
  note         = {Machine review of arXiv:2608.06930}
}
read the original abstract

Detailed audio-video joint captioning is essential for multimodal video understanding and generation. However, prior works are constrained by three main limitations: (1) the scarcity of high-quality public datasets with fine-grained audio-visual joint captions; (2) reinforcement-learning methods that rely on coarse reward signals; and (3) the lack of a benchmark and metric for evaluating detailed audiovisual captions at the atomic level. To address these challenges, we propose: (1) AVCap-100K, a high-quality dataset of 100K temporally aligned, detail-rich audio-video captions; (2) AVCap, a model optimized via Detail-Aware GRPO (Da-GRPO) that achieves state-of-the-art performance among open-source models and matches or surpasses proprietary models on several evaluations; and (3) AVCap-Bench and AVCap-Score, a specialized benchmark and metric for evaluating atomic-level details in audiovisual captions. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avcap.

Figures

Figures reproduced from arXiv: 2608.06930 by the authors.

Figure 1
Figure 1. Annotation Pipeline of AVCap-100K. The pipeline first performs Dynamic Segmentation, then extracts unimodal priors through a Visual Branch and a hierarchical Audio Branch with Demucs-based source separation, ASR, Vocal Captioning, and BGM Captioning. These priors are fused in Joint Reasoning to produce the final audio-visual caption, followed by rating-based quality filtering. coarse holistic rewards limited to shor… view at source ↗
Figure 2
Figure 2. Distribution of Video Categories in AVCap-100K [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Overview of the Da-GRPO Reward calculation. The workflow follows a Raise-Answer￾Check paradigm: (1) Raise: A set of fine-grained audio-visual probes is retrieved from the dataset; (2) Answer: The generated caption acts as the sole context for a Judge Model to answer these probes; (3) Score: The predicted answers are semantically verified against ground-truth answers to compute a dense, atomic-level reward. For a can… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Evaluation Pipeline of AVCap-Score. The metric compares informational density through a QA proxy: (1) the top branch derives 20 atomic QA pairs from the human-checked ground-truth caption; (2) the bottom branch answers the same questions using the generated caption as …
Figure 6
Figure 6. Figure 6: Impact of Probe Question Quantity (N). Second, we propose the AVCap Model, which utilizes Da-GRPO to incentivize factually grounded descriptions. Its consistent gains on a dense 7B backbone and a substantially different MoE 30B-A3B backbone demonstrate effectiveness ac…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

75 extracted references · 31 canonical work pages

  1. [1]

    Avocado: An audiovisual video captioner driven by temporal orchestration.arXiv preprint arXiv:2510.10395, 2025

    Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, et al. Avocado: An audiovisual video captioner driven by temporal orchestration.arXiv preprint arXiv:2510.10395, 2025

    arXiv

  2. [2]

    Ugc-videocaptioner: An omni ugc video detail caption model and new benchmarks.arXiv preprint arXiv:2507.11336, 2025

    Peiran Wu, Yunze Liu, Zhengdong Zhu, Enmin Zhou, and Junxiao Shen. Ugc-videocaptioner: An omni ugc video detail caption model and new benchmarks.arXiv preprint arXiv:2507.11336, 2025

    arXiv 2025

  3. [3]

    Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities.arXiv preprint arXiv:2505.17862, 2025

    Ziwei Zhou, Rui Wang, and Zuxuan Wu. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities.arXiv preprint arXiv:2505.17862, 2025

    arXiv 2025

  4. [4]

    Vidcapbench: A comprehensive benchmark of video captioning for controllable text-to-video generation.arXiv preprint arXiv:2502.12782, 2025

    Xinlong Chen, Yuanxing Zhang, Chongling Rao, Yushuo Guan, Jiaheng Liu, Fuzheng Zhang, Chengru Song, Qiang Liu, Di Zhang, and Tieniu Tan. Vidcapbench: A comprehensive benchmark of video captioning for controllable text-to-video generation.arXiv preprint arXiv:2502.12782, 2025

    arXiv 2025

  5. [5]

    Sharegpt4video: Improving video understanding and generation with better captions.Advances in Neural Information Processing Systems, 37: 19472–19495, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions.Advances in Neural Information Processing Systems, 37: 19472–19495, 2024

    work page 2024

  6. [6]

    Mavors: Multi-granularity video representation for multimodal large language model

    Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, et al. Mavors: Multi-granularity video representation for multimodal large language model. InProceedings of the 33rd ACM International Conference on Multimedia, pages 10994–11003, 2025

    2025

  7. [7]

    Timechat: A time-sensitive multimodal large language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313–14323, 2024

    2024

  8. [8]

    Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024

    Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models.arXiv preprint arXiv:2407.00634, 2024. 9

    arXiv 2024

Show all 75 references
  1. [9]

    Tarsier2: Advanc- ing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025

    Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, and Yuan Lin. Tarsier2: Advanc- ing large vision-language models from detailed video description to comprehensive video understanding.arXiv preprint arXiv:2501.07888, 2025

    2025 arXiv

  2. [10]

    Panda-70m: Captioning 70m videos with multiple cross-modality teachers

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. InProceedings of the IEEE/CVF Conferen...

    2024

  3. [11]

    Advancing high-resolution video-language representation with large-scale video transcriptions

    Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. Advancing high-resolution video-language representation with large-scale video transcriptions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogn...

    2022

  4. [12]

    video-salmonn 2: Captioning-enhanced audio-visual large language models.arXiv preprint arXiv:2506.15220, 2025

    Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: Captioning-enhanced audio-visual large language models.arXiv preprint arXiv:2506.15220, 2025

    2025

  5. [13]

    Avhbench: A cross-modal hallucination benchmark for audio-visual large language models

    Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. Avhbench: A cross-modal hallucination benchmark for audio-visual large language models. arXiv preprint arXiv:2410.18325, 2025

    2025 arXiv

  6. [14]

    Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos

    Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18959–18969, 2025

    2025

  7. [15]

    Miti- gating audiovisual mismatch in visual-guide audio captioning.arXiv preprint arXiv:2505.22045, 2025

    Le Xu, Chenxing Li, Yong Ren, Yujie Chen, Yu Gu, Ruibo Fu, Shan Yang, and Dong Yu. Miti- gating audiovisual mismatch in visual-guide audio captioning.arXiv preprint arXiv:2505.22045, 2025

    2025 arXiv

  8. [16]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  9. [17]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabil...

    2025 arXiv

  10. [18]

    Watch, listen, and describe: Globally and locally aligned cross-modal attentions for video captioning

    Xin Wang, Yuan-Fang Wang, and William Yang Wang. Watch, listen, and describe: Globally and locally aligned cross-modal attentions for video captioning. In Marilyn Walker, Heng Ji, and Amanda Stent, editors,Proceedings of the 2018 Conference of the North American Chapter of the...

    2018 doi

  11. [19]

    Videoclip: Contrastive pre-training for zero- shot video-text understanding.arXiv preprint arXiv:2109.14084, 2021

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero- shot video-text understanding.arXiv preprint arXiv:2109.14084, 2021

    2021 arXiv

  12. [20]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  13. [21]

    Glave-cap: Global-local aligned video captioning with vision expert integration.arXiv preprint arXiv:2509.11360, 2025

    Wan Xu, Feng Zhu, Yihan Zeng, Yuanfan Guo, Ming Liu, Hang Xu, and Wangmeng Zuo. Glave-cap: Global-local aligned video captioning with vision expert integration.arXiv preprint arXiv:2509.11360, 2025

    2025

  14. [22]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report.arXiv preprint arXiv:2503.20215, 2025. 10

    2025 arXiv

  15. [23]

    Omnicaptioner: One captioner to rule them all.arXiv preprint arXiv:2504.07089, 2025

    Yiting Lu, Jiakang Yuan, Zhen Li, Shitian Zhao, Qi Qin, Xinyue Li, Le Zhuo, Licheng Wen, Dongyang Liu, Yuewen Cao, et al. Omnicaptioner: One captioner to rule them all.arXiv preprint arXiv:2504.07089, 2025

    2025 arXiv

  16. [24]

    Omni-captioner: Data pipeline, models, and benchmark for omni detailed perception.arXiv preprint arXiv:2510.12720, 2025

    Ziyang Ma, Ruiyang Xu, Zhenghao Xing, Yunfei Chu, Yuxuan Wang, Jinzheng He, Jin Xu, Pheng-Ann Heng, Kai Yu, Junyang Lin, et al. Omni-captioner: Data pipeline, models, and benchmark for omni detailed perception.arXiv preprint arXiv:2510.12720, 2025

  17. [25]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  18. [26]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Junfei Wu, Xiaoying Zhang, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  19. [27]

    Adatooler-v: Adaptive tool-use for images and videos.arXiv preprint arXiv:2512.16918, 2025

    Chaoyang Wang, Kaituo Feng, Dongyang Chen, Zhongyu Wang, Zhixun Li, Sicheng Gao, Meng Meng, Xu Zhou, Manyuan Zhang, Yuzhang Shang, et al. Adatooler-v: Adaptive tool-use for images and videos.arXiv preprint arXiv:2512.16918, 2025

    2025 arXiv

  20. [28]

    Deepvideo-r1: Video rein- forcement fine-tuning via difficulty-aware regressive grpo.arXiv preprint arXiv:2506.07464, 2025

    Jinyoung Park, Jeehye Na, Jinyoung Kim, and Hyunwoo J Kim. Deepvideo-r1: Video rein- forcement fine-tuning via difficulty-aware regressive grpo.arXiv preprint arXiv:2506.07464, 2025

    2025

  21. [29]

    Editthinker: Unlocking iterative reasoning for any image editor

    Hongyu Li, Manyuan Zhang, Dian Zheng, Ziyu Guo, Yimeng Jia, Kaituo Feng, Hao Yu, Yexin Liu, Yan Feng, Peng Pei, et al. Editthinker: Unlocking iterative reasoning for any image editor. arXiv preprint arXiv:2512.05965, 2025

    2025

  22. [30]

    Sophiavl-r1: Reinforcing mllms reasoning with thinking reward.arXiv preprint arXiv:2505.17018, 2025

    Kaixuan Fan, Kaituo Feng, Haoming Lyu, Dongzhan Zhou, and Xiangyu Yue. Sophiavl-r1: Reinforcing mllms reasoning with thinking reward.arXiv preprint arXiv:2505.17018, 2025

    2025

  23. [31]

    Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing.arXiv preprint arXiv:2506.09965, 2025

    Junfei Wu, Jian Guan, Kaituo Feng, Qiang Liu, Shu Wu, Liang Wang, Wei Wu, and Tieniu Tan. Reinforcing spatial reasoning in vision-language models with interwoven thinking and visual drawing.arXiv preprint arXiv:2506.09965, 2025

    2025 arXiv

  24. [32]

    Time-r1: Post-training large vision language model for temporal video grounding.arXiv preprint arXiv:2503.13377, 2025

    Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al. Time-r1: Post-training large vision language model for temporal video grounding.arXiv preprint arXiv:2503.13377, 2025

    2025 arXiv

  25. [33]

    Onethinker: All-in-one reasoning model for image and video.arXiv preprint arXiv:2512.03043, 2025

    Kaituo Feng, Manyuan Zhang, Hongyu Li, Kaixuan Fan, Shuang Chen, Yilei Jiang, Dian Zheng, Peiwen Sun, Yiyuan Zhang, Haoze Sun, et al. Onethinker: All-in-one reasoning model for image and video.arXiv preprint arXiv:2512.03043, 2025

    2025 arXiv

  26. [34]

    Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. Videochat-r1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning.arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  27. [35]

    Videocap-r1: Enhancing mllms for video captioning via structured thinking.arXiv preprint arXiv:2506.01725, 2025

    Desen Meng, Rui Huang, Zhilin Dai, Xinhao Li, Yifan Xu, Jun Zhang, Zhenpeng Huang, Meng Zhang, Lingshu Zhang, Yi Liu, et al. Videocap-r1: Enhancing mllms for video captioning via structured thinking.arXiv preprint arXiv:2506.01725, 2025

    2025 arXiv

  28. [36]

    Exploring the role of audio in video captioning

    Yuhan Shen, Linjie Yang, Longyin Wen, Haichao Yu, Ehsan Elhamifar, and Heng Wang. Exploring the role of audio in video captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2090–2100, 2024

    2024

  29. [37]

    Hybrid transformers for music source separation

    Simon Rouard, Francisco Massa, and Alexandre D ´efossez. Hybrid transformers for music source separation. InICASSP, 2023

    2023

  30. [38]

    Audio-visual event localization in unconstrained videos

    Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. InECCV, 2018

    2018

  31. [39]

    Vggsound: A large-scale audio-visual dataset

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-visual dataset. InICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721–725. IEEE, 2020. 11

    2020

  32. [40]

    Condensed movies: Story based retrieval with contextual embeddings

    Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zisserman. Condensed movies: Story based retrieval with contextual embeddings. InProceedings of the Asian Conference on Computer Vision, 2020

    2020

  33. [41]

    Avqa: A dataset for audio-visual question answering on videos

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. InProceedings of the 30th ACM international conference on multimedia, pages 3480–3491, 2022

    2022

  34. [42]

    Movienet: A holistic dataset for movie understanding

    Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. InEuropean conference on computer vision, pages 709–727. Springer, 2020

    2020

  35. [43]

    A dataset for movie description

    Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3202–3212, 2015

    2015

  36. [44]

    Auroracap: Efficient, performant video detailed captioning and a new benchmark

    Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jenq- Neng Hwang, Saining Xie, and Christopher D Manning. Auroracap: Efficient, performant video detailed captioning and a new benchmark. InThe Thirteenth International Conference on Learning Rep...

    2024

  37. [45]

    Time-r1: Post-training large vision language model for temporal video grounding

    Ye Wang, Ziheng Wang, Boshen Xu, Yang Du, Kejun Lin, Zihan Xiao, Zihao Yue, Jianzhong Ju, Liang Zhang, Dingyi Yang, et al. Time-r1: Post-training large vision language model for temporal video grounding. InThe Thirty-ninth Annual Conference on Neural Information Processing Sys...

    2025

  38. [46]

    Mixed-r1: Unified reward perspective for reasoning capability in multimodal large language models.arXiv preprint arXiv:2505.24164, 2025

    Shilin Xu, Yanwei Li, Rui Yang, Tao Zhang, Yueyi Sun, Wei Chow, Linfeng Li, Hang Song, Qi Xu, Yunhai Tong, et al. Mixed-r1: Unified reward perspective for reasoning capability in multimodal large language models.arXiv preprint arXiv:2505.24164, 2025

    2025 arXiv

  39. [47]

    Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-r1: A stable and generalizable r1-style large vision-language model.arXiv preprint arXiv:2504.07615, 2025

    2025 arXiv

  40. [48]

    Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1 (2):3, 2022

    2022

  41. [49]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, pa...

    2023

  42. [51]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. InProceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  43. [52]

    Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness.arXiv preprint arXiv:2503.21755, 2025

    Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, et al. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness.arXiv preprint arXiv:2503.21755, 2025

    2025 arXiv

  44. [53]

    Humanomniv2: From understanding to omni-modal reasoning with context.arXiv preprint arXiv:2506.21277, 2025

    Qize Yang, Shimin Yao, Weixuan Chen, Shenghao Fu, Detao Bai, Jiaxing Zhao, Boyuan Sun, Bowen Yin, Xihan Wei, and Jingren Zhou. Humanomniv2: From understanding to omni-modal reasoning with context.arXiv preprint arXiv:2506.21277, 2025

    2025 arXiv

  45. [54]

    Arc-hunyuan-video-7b: Structured video comprehension of real-world shorts.arXiv preprint arXiv:2507.20939, 2025

    Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, et al. Arc-hunyuan-video-7b: Structured video comprehension of real-world shorts.arXiv preprint arXiv:2507.20939, 2025. 12

    2025 arXiv

  46. [55]

    Minicpm-o 2.6: A gpt-4o level mllm for vision, speech, and multimodal live stream- ing on your phone.https://github.com/OpenBMB/MiniCPM-V, 2025

    OpenBMB. Minicpm-o 2.6: A gpt-4o level mllm for vision, speech, and multimodal live stream- ing on your phone.https://github.com/OpenBMB/MiniCPM-V, 2025. GitHub repository

    2025

  47. [56]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    2025 arXiv

  48. [57]

    Self- critical sequence training for image captioning

    Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 7008–7024, 2017

    2017

  49. [58]

    id": "sample id

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. 13 A Implementation Details A.1 Training...

    2002

  50. [60]

    Use the exact terminology found in the text (e.g., if the caption says ”shatters”, use ”shatters”, not ”breaks”)

    Source of Truth: Base the answer solely on the caption. Use the exact terminology found in the text (e.g., if the caption says ”shatters”, use ”shatters”, not ”breaks”)

  51. [61]

    Conciseness: Provide a descriptive phrase or sentence fragment (under 20 words)

  52. [62]

    Start directly with the answer content

    Directness: Do NOT use filler words like ”The answer is...” or ”According to the caption...”. Start directly with the answer content

  53. [63]

    strengths

    Format: Escape internal double quotes with a backslash (\”) if necessary. Output Requirement: Return only the raw answer text string. Do not use quotes around the answer unless they are part of the content. QA answering chat template.System: {system instruction above} User: Ca...

  54. [64]

    **Hallucinations:** Identify if there are any audio events or sounds mentioned in the caption that are **not present** in the original audio. 2. **Omissions:** Identify if there are any significant audio events or sounds present in the original audio that are **missing** from ...

  55. [65]

    evaluation

    **Inaccuracies:** Identify if any audio events or sounds in the caption are **incorrectly described** compared to the original audio. This includes misidentifications or wrong descriptions of sounds. 4. **Granularity & Detail:** Evaluate the level of detail in the audio descri...

  56. [66]

    Do not add any artistic interpretation, subjective analysis, or infer any character’s internal 23 thoughts, emotions, or intentions beyond what is explicitly visible or audible

    **Objective Narration**: Describe events exactly as they occur. Do not add any artistic interpretation, subjective analysis, or infer any character’s internal 23 thoughts, emotions, or intentions beyond what is explicitly visible or audible. Your description must be purely factual

  57. [67]

    While Character A is speaking, Character B simultaneously turns their head

    **Strict Chronology and Granularity**: The narrative must follow the video’s timeline with extreme precision. Describe events, actions, and sounds on a moment-by-moment basis. Capture simultaneous actions by clearly stating them (e.g., "While Character A is speaking, Character...

  58. [68]

    He closed the door,

    **Multimodal Synthesis**: You must seamlessly integrate all provided information. When describing a visual action, you must also describe its corresponding sound effect. For instance, instead of "He closed the door," write "As he pushes the door shut, a loud ’click’ is heard f...

  59. [69]

    The camera cuts to a close-up of her face,

    **Detailed Audio-Visual Description**: * **Visuals**: Detail all character actions, movements, gestures, facial expressions, and interactions with objects. Describe camera work, such as cuts, zooms, pans, or changes in shot composition (e.g., "The camera cuts to a close-up of ...

  60. [70]

    **Accuracy and Completeness Check**: Before outputting the final description, you must perform a rigorous self-correction cycle. You will verify that your description: * **Contains No Hallucinations**: Does not describe any event, object, sound, or dialogue that is not present...

  61. [71]

    Do not use bullet points, headings, or line breaks within the narrative

    **Final Output Format**: The entire output must be a single, long, and cohesive paragraph in English. Do not use bullet points, headings, or line breaks within the narrative. Joint Captioning - User Prompt Analyze the following multimodal video data and generate a single, comp...

  62. [72]

    His voice, thick with sarcasm, says

    **Ultra-Fine Temporal Detail:** * Describe events and their timing with precise granularity. * If a person speaks, use direct quotes for dialogue to restore the original lines, avoiding indirect reporting. * Describe the speaker’s tone of voice and emotional inflection (e.g., ...

  63. [73]

    thud" as a book hits the table, a

    **Audio-Visual Interplay:** * Integrate visual and auditory elements seamlessly into a single narrative. * Detail the background music, noting its shifts in tempo, volume, and mood. * Describe actions and the specific sounds they produce, using onomatopoeia where appropriate (...

  64. [74]

    While Person A’s hand reaches for the glass, Person B’s eyes dart to the side, and the music swells

    **Simultaneous Actions:** * Describe multiple concurrent actions within the frame. For example, "While Person A’s hand reaches for the glass, Person B’s eyes dart to the side, and the music swells."

  65. [75]

    Avoid summarizing

    **Comprehensive and Long-Form:** * The final output should be a detailed, long-form narrative. Avoid summarizing. Your goal is to recreate the video experience with words, capturing every nuance as if you were a human observer with a stopwatch. But don’t make your caption too ...

  66. [76]

    strengths

    **Language:** * The entire description must be in English. Return the caption content ONLY, no other replies required. Don’t make your reply too prolix, fine-detailed is enough. 24 Joint Filtering - System Prompt You are a senior reviewer validating long-form temporal captions...

  67. [2025]

    URLhttps://arxiv.org/abs/2502.04326

    arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.