Pith. sign in

REVIEW 2 major objections 4 minor 80 references

Dense video captions can be generated in parallel without quality loss by first planning events as latent tokens and then decoding each event branch at once.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:47 UTC pith:LN4D2RQJ

load-bearing objection Clean engineering win: latent event planning + factorized attention gives real wall-clock speedup and better DVC metrics, not just a re-packaging of parallel decoding. the 2 major comments →

arxiv 2607.02963 v1 pith:LN4D2RQJ submitted 2026-07-03 cs.CV cs.AIcs.MM

Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning

Wenzheng Zeng , Siyi Jiao , Chen Gao , Hwee Tou Ng , Mike Zheng Shou This is my paper
classification cs.CV cs.AIcs.MM
keywords dense video captioningparallel autoregressive decodinglatent event planningomni-modal video understandingevent groundingdependency graph restructuringvideo large language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Autoregressive video language models write dense captions one token after another. That serial chain becomes painfully slow once a long video contains many events, each needing a multi-sentence description. This paper argues that the slowdown is largely unnecessary: local tokens belonging to different events are only weakly dependent on one another. The authors therefore restructure the causal graph. First the model emits a short sequence of compact latent “global event tokens” that discover the video’s event structure, encode inter-event causality, and pull in the relevant audio-visual features. Guided by those anchors, the model then decodes every event’s caption tokens in parallel while still generating left-to-right inside each event. The resulting method, PadCaptioner, simultaneously raises event-grounding and caption quality and delivers roughly 3.7–3.8 imes wall-clock speedup on standard omni-modal benchmarks, showing that efficiency and accuracy need not trade off.

Core claim

Weak local dependencies across temporally distinct events can be exploited to restructure the autoregressive dependency graph so that event-conditioned caption sub-chains become conditionally independent given shared global event tokens, enabling lossless parallel decoding that improves both speed and temporally grounded caption quality.

What carries the argument

Latent global planning followed by event-factorized parallel decoding: the model first autoregressively produces compact global event tokens that ground and aggregate event-level audio-visual semantics; it then decodes the local caption tokens of all events simultaneously under a factorized attention mask that keeps intra-event sequential order while blocking cross-event local attention, yet still lets every branch see the full set of global tokens.

Load-bearing premise

Local tokens of different events really are only weakly dependent, so any remaining inter-event information can be carried entirely by the compact global event tokens produced in the planning stage.

What would settle it

If a controlled ablation that restores full causal attention among local tokens of different events (while keeping the same global planning) measurably improves grounding F1 or caption similarity, the claim that the masked cross-event local dependencies are dispensable is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dense-caption inference cost scales with the length of the longest event rather than with the total number of tokens, making hour-scale videos practical.
  • The same latent-event planning plus factorized decoding can be applied to other temporally grounded omni-modal tasks (temporal grounding, segment captioning, cross-modal retrieval) without task-specific redesign.
  • Because the global event tokens already carry grounding information, event localization can be performed by simple feature matching at inference time, removing the need for separate proposal networks.
  • Wall-clock gains of roughly 3–4 imes become available to any video-LLM that can be fine-tuned with the two-stage planning-plus-parallel recipe.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same planning-plus-factorization pattern could accelerate multi-sentence generation in long-document or multi-turn dialogue models whenever content naturally factors into loosely coupled segments.
  • If the latent event tokens prove sufficiently rich, they may serve as a compact intermediate representation for controllable multi-event video generation rather than only for captioning.
  • Custom attention kernels that support the event-factorized mask would remove the remaining training-time overhead and push the speedup closer to the theoretical parallel limit set by average event count.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes PadCaptioner, a parallelized autoregressive framework for omni-modal dense video captioning. It first autoregressively generates a variable number of latent global event tokens that are explicitly grounded (via a BCE similarity loss) and enriched by adaptive audio-visual aggregation; these tokens restructure the causal dependency graph so that event-conditioned caption subchains can be decoded in parallel under an event-factorized attention mask, while each subchain remains sequential and retains access to all global tokens. Built on a 3B Video-SALMONN 2+ backbone and fine-tuned with LoRA, the method reports simultaneous gains in event grounding/caption metrics (e.g., F1 +6.7 %, Sim +6.1 % on LongVALE) and wall-clock decoding speed (≈3.7–3.8× vs. the prior 7B SOTA) on LongVALE, ChronusAV, zero-shot YouCook2, and several related temporally grounded tasks, with ablations isolating planning, attention factorization, and aggregation.

Significance. If the claims hold, the work supplies a practical, MLLM-native route to accelerate dense multi-event captioning without the quality–efficiency trade-off typical of diffusion LLMs or non-AR proposal-based DVC models. The combination of model-inferred latent planning, explicit grounding, adaptive aggregation, and factorized attention is a clean architectural contribution that also generalizes to other time-sensitive omni-modal tasks. Strengths include systematic ablations (Tables 6–9), wall-clock timing on the same GPU, length-controlled comparisons, and a public project page. The result is of clear interest to the video-language community.

major comments (2)
  1. The central “lossless” claim rests on residual cross-event local token dependencies being weak enough to mask once the global event tokens carry inter-event structure (Sec. 3.2–3.4, Fig. 3). Tables 6 and 8 already show that event-factorized parallel decoding preserves (and slightly improves) F1/Sim relative to sequential latent planning, while standard causal attention under the same parallel schedule collapses performance. These ablations make the assumption internally consistent for the reported regimes; no further load-bearing experiment is required for acceptance, though a short qualitative discussion of residual failure modes (already partially present in the appendix) would strengthen the claim.
  2. Inference grounding relies on a free similarity threshold (Appendix Tabs. A1–A2). The reported operating points (0.5 / 0.7) are stable within a narrow band, yet the main tables do not state which threshold was used for each benchmark. A single sentence in Sec. 4.1 clarifying the chosen thresholds (and that they were not tuned on the test sets) would remove any residual ambiguity about free parameters.
minor comments (4)
  1. Fig. 1 right panel claims “3.8× / 3.1× speed-up” while Table 2 reports ≈3.7× wall-clock and ≈3× per-token; a brief note reconciling the two numbers would avoid reader confusion.
  2. Sec. 3.4: the padding-with-EOS training scheme and shared position indices across subchains are important for variable-length parallel decoding; a short pseudocode block or expanded caption for Fig. 3 would make the implementation fully self-contained.
  3. Appendix B.4 notes that the modified attention is currently incompatible with FlashAttention-2, inflating training cost. Mentioning this limitation (and the planned kernel work) already in the main-text limitations paragraph would be useful for practitioners.
  4. A few typographical inconsistencies remain (e.g., “PadCaptioner−” vs. “PadCaptioner-”, occasional missing spaces around citations). These are purely cosmetic.

Circularity Check

0 steps flagged

No significant circularity: parallel decoding is a restructured causal factorization trained with ordinary SFT + grounding BCE, not a quantity forced by construction or self-citation.

full rationale

The paper's central claim (lossless parallel AR decoding via event-level dependency restructuring) is realized by (i) autoregressive generation of latent global event tokens G_i under an explicit BCE grounding loss against annotated intervals (Eq. 2) plus adaptive AV aggregation, followed by (ii) an event-factorized attention mask that keeps intra-event sequential AR while allowing cross-event parallel steps conditioned only on the shared G tokens (Eq. 3 and Fig. 3). Both stages are trained with standard token-level cross-entropy on the reordered sequence; no free parameter is fitted to a target metric and then re-presented as a prediction, and no uniqueness theorem or ansatz is imported from overlapping-author work to force the design. Ablations (Tabs. 6–9) show that removing planning, aggregation, or the factorized mask measurably degrades F1/Sim, confirming the performance gains are empirical rather than definitional. Self-citations (e.g., [70] for feature-matching grounding) supply only an implementation detail and are not load-bearing for the parallel-decoding claim. The derivation is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 3 invented entities

The central claim rests on one domain modeling assumption (weak cross-event local dependence), a small set of engineering free parameters (thresholds, frame budget, LoRA), and three invented architectural tokens/masks that have no independent existence outside the paper. No deep mathematical axioms beyond standard transformer attention are required.

free parameters (3)
  • similarity threshold for event grounding = 0.5 / 0.7
    Chosen per dataset (0.5 LongVALE, 0.7 ChronusAV) to decide which time segments belong to a global event token; directly affects reported F1.
  • max frames / sampling rate = 256 frames @ 0.5 FPS
    Hard limit of 256 frames at 0.5 FPS; longer videos are uniformly subsampled, affecting both grounding and caption content.
  • LoRA rank and training subset size
    Fine-tuning hyper-parameters and the 12 k / 18 k video subset of ChronusAV are chosen by the authors; performance numbers depend on them.
axioms (3)
  • domain assumption Local token-level dependencies across temporally distinct events are weak; inter-event information can be adequately summarized by compact global tokens.
    Stated as the key insight in Sec. 1 and 3.2; if false, the parallel scheme loses information.
  • standard math Standard causal transformer attention and next-token prediction remain valid inside each event sub-chain.
    Used throughout Sec. 3.4; no new proof required.
  • domain assumption Ground-truth event intervals in the training annotations form a valid semantic partition that can supervise latent event tokens via similarity matching.
    Eq. (2) and surrounding text; the quality of the learned G_i tokens depends on this annotation assumption.
invented entities (3)
  • latent global event tokens G_i (special <G> vocabulary items) no independent evidence
    purpose: Serve as both structural anchors for dependency restructuring and semantic aggregators of event-level audio-visual features.
    Introduced in Sec. 3.3; no independent existence outside the model.
  • switch token S no independent evidence
    purpose: Marks the end of the planning stage and the start of parallel local decoding.
    Architectural control token defined by the authors.
  • event-factorized attention mask no independent evidence
    purpose: Allows each local token to see the shared prefix, all G tokens, and its own history while blocking other events’ local tokens.
    Defined in Sec. 3.4 and Fig. 3; the core mechanism that makes parallel decoding possible.

pith-pipeline@v1.1.0-grok45 · 28300 in / 2733 out tokens · 28372 ms · 2026-07-12T05:47:40.097306+00:00 · methodology

0 comments
read the original abstract

Dense video captioning aims to generate temporally grounded descriptions of video events, benefiting both event-level video understanding and generation. In this domain, autoregressive video large language models have emerged as a prevalent paradigm due to their strong generative and cross-modal modeling capacity. However, generating dense captions under the token-by-token paradigm severely limits inference efficiency and hinders scalability as video length and event density increase. In this work, we propose a parallelized autoregressive framework that not only improves generation efficiency but also enhances temporally grounded captioning performance. Our key insight is to exploit the weak local dependencies across temporally distinct events to restructure the causal dependency graph, thereby enabling lossless parallel generation. Specifically, tokens with weak cross-event dependencies can be decoded in parallel, while tightly coupled tokens within each event retain sequential decoding to preserve local semantic coherence. To realize this insight, we introduce two key components for lossless parallel decoding: (1) a latent global planning mechanism that automatically learns the event-level structure and produces compact tokens encoding global inter-event causality while adaptively aggregating event-level audio-visual semantics, guiding subsequent dependency restructuring and parallel decoding; and (2) an event-factorized parallel decoding mechanism that effectively balances local focus with global inter-event awareness. Experiments on various benchmarks demonstrate the clear advantage of our approach in both efficiency and performance in omni-modal event grounding and captioning. Project website: https://github.com/showlab/PadCaptioner.

Figures

Figures reproduced from arXiv: 2607.02963 by Chen Gao, Hwee Tou Ng, Mike Zheng Shou, Siyi Jiao, Wenzheng Zeng.

Figure 1
Figure 1. Figure 1: Left: Comparison of token dependency modeling and decoding strategies between existing Video-LLM-based captioning models and ours. We achieve lossless parallel autoregressive decoding under a model-inferred restructured dependency graph. Right: Our method surpasses the previous SOTA [8] in both grounded captioning accuracy and decoding efficiency on the LongVALE benchmark [13]. substantially more challengi… view at source ↗
Figure 2
Figure 2. Figure 2: The overall pipeline of the proposed parallelized autoregressive decoding frame￾work, where we set the model-inferred event count K = 3 for illustration. within event boundaries, while cross-event relations are largely governed by higher-level global semantics. Consequently, enforcing detailed token-level causal dependencies across events introduces redundant serialization, where such cross￾event interacti… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of the different decoding strategies and attention mechanisms. preserves the autoregressive property within each subchain to maintain strong local semantic coherence. Based on the reformulated dependency graph, we derive a modified causal attention scheme that enables structured parallel decoding. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative example of a video caption generated by PadCaptioner. Effect of adaptive semantic aggregation. (1) As shown in Tab. 9, removing audio-visual semantic aggregation into global event tokens (Row-1 vs. the remain￾ing rows) leads to a substantial performance drop. This highlights the benefit of enriching global event tokens with event-focused semantics, enabling them to serve as expressive anchors f… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

80 extracted references · 17 linked inside Pith

  1. [1]

    In: The Thirteenth International Conference on Learning Representations (2025)

    Arriola,M.,Gokaslan,A.,Chiu,J.T.,Yang,Z.,Qi,Z.,Han,J.,Sahoo,S.S.,Kuleshov, V.: Block diffusion: Interpolating between autoregressive and diffusion language models. In: The Thirteenth International Conference on Learning Representations (2025)

    2025

  2. [2]

    arXiv preprint arXiv:2511.21631 (2025)

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al.: Qwen3-vl technical report. arXiv preprint arXiv:2511.21631 (2025)

    Pith/arXiv arXiv 2025

  3. [4]

    In: The Thirteenth International Conference on Learning Representations (2025) 16 W

    Chai, W., Song, E., Du, Y., Meng, C., Madhavan, V., Bar-Tal, O., Hwang, J.N., Xie, S., Manning, C.D.: Auroracap: Efficient, performant video detailed captioning and a new benchmark. In: The Thirteenth International Conference on Learning Representations (2025) 16 W. Zeng et al

    2025

  4. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, J., Lv, Z., Wu, S., Lin, K.Q., Song, C., Gao, D., Liu, J.W., Gao, Z., Mao, D., Shou, M.Z.: Videollm-online: Online video large language model for streaming video. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18407–18418 (2024)

    2024

  5. [6]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Chen, J., Zeng, Z., Lin, Y., Li, W., Ma, Z., Shou, M.Z.: Livecc: Learning video llm with streaming speech transcription at scale. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 29083–29095 (2025)

    2025

  6. [7]

    In: International Conference on Learning Representations (2026)

    Chen, X., Ding, Y., Lin, W., Hua, J., Yao, L., Shi, Y., Li, B., Zhang, Y., Liu, Q., Wan, P., et al.: Avocado: An audiovisual video captioner driven by temporal orchestration. In: International Conference on Learning Representations (2026)

    2026

  7. [9]

    arXiv preprint arXiv:2510.23397 (2025)

    Dong, L., Zhang, H., Lin, H., Yan, Z., Zeng, X., Zhang, H., Huang, Y., Wang, Y., Ling, Z.H., Wang, L., et al.: Videotg-r1: Boosting video temporal grounding via curriculum reinforcement learning on reflected boundary annotations. arXiv preprint arXiv:2510.23397 (2025)

    arXiv 2025

  8. [10]

    arXiv preprint arXiv:2512.05665 (2025)

    Dong, S., Wang, S., Liu, X., Li, C., Hou, H., Wei, Z.: Interleaved latent visual reasoning with selective perceptual modeling. arXiv preprint arXiv:2512.05665 (2025)

    arXiv 2025

  9. [11]

    In: Proceedings of the 40th International Conference on Machine Learning

    Driess, D., Xia, F., Sajjadi, M.S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al.: Palm-e: an embodied multimodal language model. In: Proceedings of the 40th International Conference on Machine Learning. pp. 8469–8488 (2023)

    2023

  10. [12]

    arXiv preprint arXiv:2507.20939 (2025)

    Ge, Y., Ge, Y., Li, C., Wang, T., Pu, J., Li, Y., Qiu, L., Ma, J., Duan, L., Zuo, X., et al.: Arc-hunyuan-video-7b: Structured video comprehension of real-world shorts. arXiv preprint arXiv:2507.20939 (2025)

    Pith/arXiv arXiv 2025

  11. [13]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Geng, T., Zhang, J., Wang, Q., Wang, T., Duan, J., Zheng, F.: Longvale: Vision- audio-language-event benchmark towards time-aware omni-modal perception of long videos. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18959–18969 (2025)

    2025

  12. [14]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al.: Ego4d: Around the world in 3,000 hours of egocentric video. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 18995–19012 (2022)

    2022

  13. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Grauman, K., Westbury, A., Torresani, L., Kitani, K., Malik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al.: Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19383–19400 (2024)

    2024

  14. [16]

    arXiv preprint arXiv:2405.13382 (2024)

    Guo, Y., Liu, J., Li, M., Cheng, D., Tang, X., Sui, D., Liu, Q., Chen, X., Zhao, K.: Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding. arXiv preprint arXiv:2405.13382 (2024)

    Pith/arXiv arXiv 2024

  15. [17]

    In: International Conference on Learning Representations (2025)

    Guo, Y., Liu, J., Li, M., Tang, X., Liu, Q., Chen, X.: Trace: Temporal grounding video llm via causal event modeling. In: International Conference on Learning Representations (2025)

    2025

  16. [18]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    He, Y., He, Y., He, S., Chen, F., Zhou, H., Zhang, K., Zhuang, B.: Neighboring autoregressive modeling for efficient visual generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19000–19010 (2025) PadCaptioner 17

    2025

  17. [19]

    arXiv preprint arXiv:2601.17868 (2026)

    He, Z., Chen, T., Wang, K., Qin, Z., Shao, Y., Gan, C., Li, S., Wu, Z., Lin, W.: Vid- lada: Bidirectional diffusion large language models for efficient video understanding. arXiv preprint arXiv:2601.17868 (2026)

    arXiv 2026

  18. [20]

    ICLR1(2), 3 (2022)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al.: Lora: Low-rank adaptation of large language models. ICLR1(2), 3 (2022)

    2022

  19. [21]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Huang, B., Wang, X., Chen, H., Song, Z., Zhu, W.: Vtimellm: Empower llm to grasp video moments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14271–14280 (2024)

    2024

  20. [22]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Kim, M., Kim, H.B., Moon, J., Choi, J., Kim, S.T.: Do you remember? dense video captioning with cross-modal memory retrieval. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13894–13904 (2024)

    2024

  21. [23]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Kim, M., Kim, H.B., Moon, J., Choi, J., Kim, S.T.: Hicm2: Hierarchical com- pact memory modeling for dense video captioning. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 4293–4301 (2025)

    2025

  22. [24]

    In: Proceedings of the IEEE international conference on computer vision

    Krishna, R., Hata, K., Ren, F., Fei-Fei, L., Carlos Niebles, J.: Dense-captioning events in videos. In: Proceedings of the IEEE international conference on computer vision. pp. 706–715 (2017)

    2017

  23. [25]

    In: International Conference on Learning Representations (2026)

    Li, A., Wang, C., Fu, D., Yue, K., Cai, Z., Zhu, W.B., Liu, O., Guo, P., Neiswanger, W., Huang, F., et al.: Zebra-cot: A dataset for interleaved vision language reasoning. In: International Conference on Learning Representations (2026)

    2026

  24. [26]

    In: The Fourteenth International Conference on Learning Representations (2026)

    Li, H., Yang, J., Li, G., Wang, H.: Autoregressive image generation with random- ized parallel decoding. In: The Fourteenth International Conference on Learning Representations (2026)

    2026

  25. [27]

    arXiv preprint arXiv:2602.02994 (2026)

    Li, J., Yin, H., Xu, H., Xu, B., Tan, W., He, Z., Ju, J., Luo, Z., Luan, J.: Video-opd: Efficient post-training of multimodal large language models for temporal video grounding via on-policy distillation. arXiv preprint arXiv:2602.02994 (2026)

    Pith/arXiv arXiv 2026

  26. [28]

    Science China Information Sciences 68(10), 200102 (2025)

    Li, K., He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., Qiao, Y.: Videochat: Chat-centric video understanding. Science China Information Sciences 68(10), 200102 (2025)

    2025

  27. [29]

    arXiv preprint arXiv:2501.00574 (2025)

    Li, X., Wang, Y., Yu, J., Zeng, X., Zhu, Y., Huang, H., Gao, J., Li, K., He, Y., Wang, C., et al.: Videochat-flash: Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00574 (2025)

    Pith/arXiv arXiv 2025

  28. [30]

    arXiv preprint arXiv:2504.06958 (2025)

    Li, X., Yan, Z., Meng, D., Dong, L., Zeng, X., He, Y., Wang, Y., Qiao, Y., Wang, Y., Wang, L.: Videochat-r1: Enhancing spatio-temporal perception via reinforcement fine-tuning. arXiv preprint arXiv:2504.06958 (2025)

    Pith/arXiv arXiv 2025

  29. [31]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Li, Y., Yao, T., Pan, Y., Chao, H., Mei, T.: Jointly localizing and describing events for dense video captioning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7492–7500 (2018)

    2018

  30. [32]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Li, Z., Yang, B., Liu, Q., Zhang, S., Ma, Z., Yin, L., Deng, L., Sun, Y., Liu, Y., Bai, X.: Lira: Inferring segmentation in large multi-modal models with local interleaved region assistance. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 24056–24067 (2025)

    2025

  31. [33]

    In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    Li, Z., Zhang, X., Guo, Y., Bennamoun, M., Boussaid, F., Dwivedi, G., Gong, L., Ke, Q.: Watch and listen: Understanding audio-visual-speech moments with multimodal llm. In: The Thirty-ninth Annual Conference on Neural Information Processing Systems (2025)

    2025

  32. [34]

    Bench: Towards open- ended event-level video-language understanding

    Liu, Y., Ma, Z., Qi, Z., Wu, Y., Shan, Y., Chen, C.W.: E.T. Bench: Towards open- ended event-level video-language understanding. In: The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024) 18 W. Zeng et al

    2024

  33. [35]

    In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Liu, Z., Zhang, X., Liu, J.: Task-specific information decomposition for end-to- end dense video captioning. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 16524– 16536 (2025)

    2025

  34. [36]

    arXiv preprint arXiv:2502.04328 (2025)

    Liu, Z., Dong, Y., Wang, J., Liu, Z., Hu, W., Lu, J., Rao, Y.: Ola: Pushing the frontiers of omni-modal language model. arXiv preprint arXiv:2502.04328 (2025)

    Pith/arXiv arXiv 2025

  35. [37]

    In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Maaz, M., Rasheed, H., Khan, S., Khan, F.: Video-chatgpt: Towards detailed video understanding via large vision and language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12585–12602 (2024)

    2024

  36. [38]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Mun, J., Yang, L., Ren, Z., Xu, N., Han, B.: Streamlined dense video captioning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6588–6597 (2019)

    2019

  37. [39]

    arXiv preprint arXiv:2502.09992 (2025)

    Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.R., Li, C.: Large language diffusion models. arXiv preprint arXiv:2502.09992 (2025)

    Pith/arXiv arXiv 2025

  38. [40]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Pang, Z., Zhang, T., Luan, F., Man, Y., Tan, H., Zhang, K., Freeman, W.T., Wang, Y.X.: Randar: Decoder-only autoregressive visual generation in random orders. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 45–55 (2025)

    2025

  39. [41]

    In: International Conference on Learning Representations (2026)

    Pang, Z., Otani, M., Nakashima, Y.: Measure twice, cut once: Grasping video struc- tures and event semantics with llms for video temporal localization. In: International Conference on Learning Representations (2026)

    2026

  40. [42]

    arXiv preprint arXiv:2511.14349 (2025)

    Pu, J., Wang, T., Ge, Y., Ge, Y., Li, C., Shan, Y.: Arc-chapter: Structuring hour- long videos into navigable chapters and hierarchical summaries. arXiv preprint arXiv:2511.14349 (2025)

    arXiv 2025

  41. [43]

    arXiv preprint arXiv:2402.11435 (2024)

    Qian, L., Li, J., Wu, Y., Ye, Y., Fei, H., Chua, T.S., Zhuang, Y., Tang, S.: Momentor: Advancing video large language model with fine-grained temporal reasoning. arXiv preprint arXiv:2402.11435 (2024)

    Pith/arXiv arXiv 2024

  42. [44]

    arXiv preprint arXiv:2601.07568 (2026)

    Qian, Y.Y., Su, J., Hu, L., Zhang, P., Deng, Z., Zhao, P., Zhang, H.: d3llm: Ultra-fast diffusion llm using pseudo-trajectory distillation. arXiv preprint arXiv:2601.07568 (2026)

    arXiv 2026

  43. [45]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Xu, T., Brockman, G., McLeavey, C., Sutskever, I.: Robust speech recognition via large-scale weak supervision. In: International conference on machine learning. pp. 28492–28518. PMLR (2023)

    2023

  44. [46]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Ren, S., Yao, L., Li, S., Sun, X., Hou, L.: Timechat: A time-sensitive multi- modal large language model for long video understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14313–14323 (2024)

    2024

  45. [47]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Shao, Z., Wang, M., Yu, Z., Pan, W., Yang, Y., Wei, T., Zhang, H., Mao, N., Chen, W., Yu, J.: Growing a twig to accelerate large vision-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20064–20074 (2025)

    2025

  46. [48]

    In: Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants! pp

    Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., Cai, D.: Pandagpt: One model to instruction-follow them all. In: Proceedings of the 1st Workshop on Taming Large Language Models: Controllability in the era of Interactive Assistants! pp. 11–23 (2023)

    2023

  47. [49]

    In: International Conference on Learning Representations (2026) PadCaptioner 19

    Su, Y., Zhang, H., Li, S., Liu, N., Liao, J., Pan, J., Liu, Y., Xing, X., Sun, C., Li, C., et al.: Patch-as-decodable-token: Towards unified multi-modal vision tasks in mllms. In: International Conference on Learning Representations (2026) PadCaptioner 19

    2026

  48. [51]

    In: ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

    Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., Zhang, C.: Extending large language models for speech and audio captioning. In: ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 11236–11240. IEEE (2024)

    2024

  49. [52]

    In: Proceedings of the AAAI Conference on Artificial Intelligence

    Tang, Y., Shimada, D., Bi, J., Feng, M., Hua, H., Xu, C.: Empowering llms with pseudo-untrimmed videos for audio-visual temporal understanding. In: Proceedings of the AAAI Conference on Artificial Intelligence. pp. 7293–7301 (2025)

    2025

  50. [53]

    arXiv preprint arXiv:2512.16584 (2025)

    Tong, J., Gu, J., Lou, Y., Fan, L., Zou, Y., Wu, Y., Ye, J., Li, R.: Sketch-in-latents: Eliciting unified reasoning in mllms. arXiv preprint arXiv:2512.16584 (2025)

    arXiv 2025

  51. [54]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Wang, T., Zhang, R., Lu, Z., Zheng, F., Cheng, R., Luo, P.: End-to-end dense video captioning with parallel decoding. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6847–6857 (2021)

    2021

  52. [55]

    arXiv preprint arXiv:2508.09192 (2025)

    Wang, X., Xu, C., Jin, Y., Jin, J., Zhang, H., Deng, Z.: Diffusion llms can do faster- than-ar inference via discrete diffusion forcing. arXiv preprint arXiv:2508.09192 (2025)

    Pith/arXiv arXiv 2025

  53. [56]

    arXiv preprint arXiv:2503.13377 (2025)

    Wang, Y., Wang, Z., Xu, B., Du, Y., Lin, K., Xiao, Z., Yue, Z., Ju, J., Zhang, L., Yang, D., et al.: Time-r1: Post-training large vision language model for temporal video grounding. arXiv preprint arXiv:2503.13377 (2025)

    Pith/arXiv arXiv 2025

  54. [57]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, Y., Ren, S., Lin, Z., Han, Y., Guo, H., Yang, Z., Zou, D., Feng, J., Liu, X.: Parallelized autoregressive visual generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12955–12965 (2025)

    2025

  55. [58]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wu, K., Li, P., Fu, J., Li, Y., Wu, Y., Liu, Y., Wang, J., Zhou, S.: Event-equalized dense video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8417–8427 (2025)

    2025

  56. [59]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Wu, W., Liu, M., Zhu, Z., Xia, X., Feng, H., Wang, W., Lin, K.Q., Shen, C., Shou, M.Z.: Moviebench: A hierarchical movie level dataset for long video generation. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 28984–28994 (2025)

    2025

  57. [60]

    arXiv preprint arXiv:2503.07314 (2025)

    Wu, W., Zhu, Z., Shou, M.Z.: Automated movie generation via multi-agent cot planning. arXiv preprint arXiv:2503.07314 (2025)

    Pith/arXiv arXiv 2025

  58. [63]

    arXiv preprint arXiv:2510.09608 (2025)

    Xu, R., Xiao, G., Chen, Y., He, L., Peng, K., Lu, Y., Han, S.: Streamingvlm: Real-time understanding for infinite video streams. arXiv preprint arXiv:2510.09608 (2025)

    Pith/arXiv arXiv 2025

  59. [64]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Yang, A., Nagrani, A., Seo, P.H., Miech, A., Pont-Tuset, J., Laptev, I., Sivic, J., Schmid, C.: Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10714–10726 (2023)

    2023

  60. [65]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yang, Z., Yu, X., Chen, D., Shen, M., Gan, C.: Machine mental imagery: Empower multimodal reasoning with latent visual tokens. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 33510–33520 (2026)

    2026

  61. [66]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Yang, Z., Yu, Y., Zhao, Y., Lu, S., Bai, S.: Timeexpert: An expert-guided video llm for video temporal grounding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 24286–24296 (2025) 20 W. Zeng et al

    2025

  62. [67]

    arXiv preprint arXiv:2512.22615 (2025)

    Ye, J., Gong, S., Gao, J., Fan, J., Wu, S., Bi, W., Bai, H., Shang, L., Kong, L.: Dream-vl & dream-vla: Open vision-language and vision-language-action models with diffusion language model backbone. arXiv preprint arXiv:2512.22615 (2025)

    arXiv 2025

  63. [68]

    arXiv preprint arXiv:2505.16933 (2025)

    You, Z., Nie, S., Zhang, X., Hu, J., Zhou, J., Lu, Z., Wen, J.R., Li, C.: Llada-v: Large language diffusion models with visual instruction tuning. arXiv preprint arXiv:2505.16933 (2025)

    Pith/arXiv arXiv 2025

  64. [69]

    arXiv preprint arXiv:2506.13759 (2025)

    Yu, R., Li, Q., Wang, X.: Discrete diffusion in large language and multimodal models: A survey. arXiv preprint arXiv:2506.13759 (2025)

    arXiv 2025

  65. [70]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zeng, W., Gao, D., Shou, M.Z., Ng, H.T.: Factorized learning for temporally grounded video-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20683–20693 (2025)

    2025

  66. [71]

    In: International Conference on Learning Representations (2025)

    Zeng, X., Li, K., Wang, C., Li, X., Jiang, T., Yan, Z., Li, S., Shi, Y., Yue, Z., Wang, Y., et al.: Timesuite: Improving mllms for long video understanding via grounded tuning. In: International Conference on Learning Representations (2025)

    2025

  67. [72]

    In: Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations

    Zhang, H., Li, X., Bing, L.: Video-llama: An instruction-tuned audio-visual language model for video understanding. In: Proceedings of the 2023 conference on empirical methods in natural language processing: system demonstrations. pp. 543–553 (2023)

    2023

  68. [73]

    arXiv preprint arXiv:2512.14698 (2025)

    Zhang,J.,Wang,T.,Ge,Y.,Ge,Y.,Li,X.,Shan,Y.,Wang,L.:Timelens:Rethinking video temporal grounding with multimodal llms. arXiv preprint arXiv:2512.14698 (2025)

    arXiv 2025

  69. [74]

    In: International Conference on Learning Representations (2026)

    Zhang, Z., Huang, L.J., Wu, C., Yang, S., Peng, K., Lu, Y., Han, S.: Locality-aware parallel decoding for efficient autoregressive image generation. In: International Conference on Learning Representations (2026)

    2026

  70. [75]

    In: Proceedings of the AAAI conference on artificial intelligence (2018)

    Zhou, L., Xu, C., Corso, J.: Towards automatic learning of procedures from web instructional videos. In: Proceedings of the AAAI conference on artificial intelligence (2018)

    2018

  71. [76]

    In: Proceedings of the IEEE conference on computer vision and pattern recognition

    Zhou, L., Zhou, Y., Corso, J.J., Socher, R., Xiong, C.: End-to-end dense video captioning with masked transformer. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8739–8748 (2018) Parallelized Autoregressive Decoding for Omni-Modal Dense Video Captioning Wenzheng Zeng, Siyi Jiao, Chen Gao, Hwee Tou Ng†, Mike Zheng Sh...

    2018

  72. [77]

    5-vl technical report

    Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

    Pith/arXiv arXiv 2025

  73. [78]

    arXiv preprint arXiv:2512.09841 (2025)

    Chen, Y., Wu, Y., Guan, K., Ren, Y., Wang, Y., Song, R., Ru, L.: Chronu- somni: Improving time awareness of omni large language models. arXiv preprint arXiv:2512.09841 (2025)

    arXiv 2025

  74. [79]

    In: International Conference on Learning Representations

    Dao, T.: Flashattention-2: Faster attention with better parallelism and work par- titioning. In: International Conference on Learning Representations. vol. 2024, pp. 35549–35562 (2024)

    2024

  75. [80]

    In: Proceedings of the Computer Vision and Pattern Recognition Conference

    Geng, T., Zhang, J., Wang, Q., Wang, T., Duan, J., Zheng, F.: Longvale: Vision- audio-language-event benchmark towards time-aware omni-modal perception of long videos. In: Proceedings of the Computer Vision and Pattern Recognition Conference. pp. 18959–18969 (2025) 26 W. Zeng et al

    2025

  76. [81]

    Bench: Towards open- ended event-level video-language understanding

    Liu, Y., Ma, Z., Qi, Z., Wu, Y., Shan, Y., Chen, C.W.: E.T. Bench: Towards open- ended event-level video-language understanding. In: The Thirty-eighth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (2024)

    2024

  77. [82]

    arXiv preprint arXiv:2506.15220 (2025)

    Tang, C., Li, Y., Yang, Y., Zhuang, J., Sun, G., Li, W., Ma, Z., Zhang, C.: video- salmonn 2: Captioning-enhanced audio-visual large language models. arXiv preprint arXiv:2506.15220 (2025)

    arXiv 2025

  78. [83]

    5-omni technical report

    Xu, J., Guo, Z., He, J., Hu, H., He, T., Bai, S., Chen, K., Wang, J., Fan, Y., Dang, K., et al.: Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215 (2025)

    Pith/arXiv arXiv 2025

  79. [84]

    arXiv preprint arXiv:2509.17765 (2025)

    Xu, J., Guo, Z., Hu, H., Chu, Y., Wang, X., He, J., Wang, Y., Shi, X., He, T., Zhu, X., et al.: Qwen3-omni technical report. arXiv preprint arXiv:2509.17765 (2025)

    Pith/arXiv arXiv 2025

  80. [85]

    Sugarella Sweets

    Zeng, W., Gao, D., Shou, M.Z., Ng, H.T.: Factorized learning for temporally grounded video-language models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 20683–20693 (2025) PadCaptioner 27 1 2 3 4 5 1 2 3 4 5 6 7 1 2 3 4 5 Ground truth ChronusOmni 7B Ours 3B [0s–14s] Visual: A brand logo is shown, followed by a monkey ca...

    2025