Pith. sign in

REVIEW 4 major objections 5 minor 77 references

Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a factorized autoregressive decoder, invoked per video segment with a shared text decoder and causal memory of prior segments, produces denser and more localized captions than global decoders while using about 20…

desk verdict A solid factorized-decoding architecture with honest internal ablations, but the SOTA claims outrun the evidence and the 'local content' framing is overstated. read the letter →

arxiv 2411.14688 v1 pith:PQ3O2MLL submitted 2024-11-22 cs.CV cs.CLcs.LG

AJ Piergiovanni , Dahun Kim , Michael S. Ryoo , Isaac Noble , Anelia Angelova This is my paper · ORCID
classification cs.CVcs.CLcs.LG
keywords factorizeddecodingdensevideocaptioningonlineunderstandingautoregressivememorytemporaleventlocalizationlong-videomodelingcross-segmentmasking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that dense video captioning does not need one global decoder that watches the entire video. Instead, the video is cut into fixed segments, and a single shared text decoder captions each segment independently while a compressed autoregressive memory carries information from all earlier segments. The authors report that this online, factorized design generates more frequent, more detailed captions than both global-decoder baselines and the sparse human ground truth, and it uses about 20 percent less compute. If correct, the method makes live captioning of long, streaming videos practical and could automate the dense annotation of large video collections.

What carries the argument

The central mechanism is the factorized text decoder: a single decoder run once per video segment, with its cross-attention restricted to that segment's features, while a causally masked autoregressive transformer over segment tokens provides memory of all earlier segments. This arrangement turns the quadratic cost of global self-attention in video length into a cost that is linear in the number of segments, and it biases generation toward short, local captions rather than one global summary. A cross-segment masking scheme lets the decoder be trained once over all segments jointly, sharing weights and saving memory.

What would settle it

Compare F1 and CIDEr on videos whose ground-truth events are artificially shifted to straddle the fixed segment boundaries; if scores drop markedly relative to a global-decoder model on the same videos, the end-time alignment rule is the bottleneck.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that factorizing the decoder by video segment is the right inductive bias for dense, temporally aligned captioning. A lightweight autoregressive transformer over segment-level features supplies memory of prior context, and each segment's caption is generated locally by a weight-shared text decoder whose cross-attention is masked to that segment. Training labels are assigned to the segment containing each caption's end time, so the model learns to emit localized captions as it streams. The paper reports state-of-the-art SODA, CIDEr, METEOR, and F1 scores on ViTT and YouCook2, and SODA and METEOR on ActivityNet, with 18 to 20 percent lower FLOPs than a global decoder of the same size.

Load-bearing premise

The model assumes every caption can be learned from one fixed segment's visual features plus compressed memory of earlier segments, because each ground-truth caption is assigned only to the segment containing its end time; events that span several segments or lose detail in the compressed memory are never seen in full by the decoder.

Editorial extensions

If this is right

  • Because each segment is decoded without future frames, the same architecture can caption a video as it streams, removing the need to buffer the whole video.
  • Because attention cost is linear in the number of segments rather than quadratic in total length, the reported 18-20 percent FLOP saving grows as videos and captions get longer.
  • The model emits roughly 12 to 15 captions per video on ViTT and YouCook2, versus 7 to 8 in the ground truth, so it can produce denser annotations than humans do, even when standard metrics penalize the extra outputs.
  • Matching the number of segments to the expected number of events and adding a time prefix to each segment measurably improve SODA, CIDEr, and METEOR.
  • Because the decoder outputs localized captions with start and end times, the model also performs event localization, not just caption generation, in a single pass.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the dense outputs are as accurate as they appear qualitatively, the model could serve as an automatic annotator that expands sparse human labels into training data for other video-language systems.
  • The same per-segment factorization could transfer to other long-form video tasks with local structure, such as step-by-step instruction comprehension or event-level video question answering, where a single global summary is known to lose detail.
  • The paper's label assignment anchors each caption to the segment containing its end time, so a useful stress test would be to perturb segment boundaries and measure how quickly localization F1 degrades; this would show how much the method depends on the fixed grid.
  • Because current captioning metrics compare against sparse ground truth, the paper's density claim would be better adjudicated by a human study on whether the extra captions are correct and non-redundant, which benchmark numbers alone cannot settle.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an online dense video captioning model that splits a video into T segments, encodes each segment with a shared TubeViT vision encoder, reduces dimensionality with a transformer, and passes the resulting features through a causal autoregressive transformer that acts as compressed memory. A shared text decoder is then applied independently to each segment (factorized decoding), with a cross-segment masking scheme during training that restricts cross-attention to the current segment's features while allowing the autoregressive memory to carry prior context. Captions are generated in the Vid2Seq token format with start and end times, and training labels are assigned to the segment containing each caption's end time. Experiments on ViTT, YouCook2, and ActivityNet report dense captioning and localization metrics; the paper claims state-of-the-art results on most metrics, denser and more detailed outputs than ground truth, and 18-20% FLOP savings relative to a global decoder. Internal ablations and output statistics support the factorized-decoder design.

Significance. If the claimed results hold, factorized autoregressive decoding with cross-segment memory is a practical and credible way to make dense video captioning online and scalable to long videos. The paper's strengths include a clean internal ablation (Table 4) that isolates the factorized decoder from the global decoder under the same components and pretraining, an explicit FLOP comparison (Table 2), and a candid limitations section (Appendix D) that acknowledges the metric mismatch caused by denser outputs. The proposed architecture goes beyond the current consensus that global decoders are necessary for dense video captioning and is internally coherent. However, the state-of-the-art claims are currently supported mainly by uncontrolled external comparisons and by a caption-to-segment alignment rule whose failure modes are not quantified; these need to be addressed before the empirical headline can be accepted.

major comments (4)
  1. [Section 3.3] The assignment of each ground-truth caption to the single segment containing its end time is load-bearing for the central claim that captions reflect 'actual local content' (Abstract and Section 3.2.1). A caption whose event spans multiple segments is never supervised from the visual features of the frames in which the event actually occurs; the model must recover that information from the compressed autoregressive memory. The manuscript acknowledges that this alignment 'can be noisy if segments do not align well to the captions' but provides no estimate of how often captions cross segment boundaries on ViTT, YouCook2, or ActivityNet, and no analysis of whether performance degrades on such captions. I request a quantitative breakdown (e.g., the fraction of captions whose annotated intervals are fully contained in one segment versus spanning multiple segments) and a controlled experiment on cross-boundary events showing that the model's captions remain grounded in the relevant visual features rather than inferred from language priors. Without this, the localized-grounding advantage over global decoders is not established.
  2. [Table 1 and Section 4.1] The headline state-of-the-art claim rests on Table 1, in which the baselines use substantially different pretraining data (e.g., Vid2Seq on YT-Temporal-1B, DIBS on a custom re-captioned HowTo100M, Zhou et al. on WebLI, and the proposed model on the original HowTo100M). No error bars, significance tests, or per-seed variances are reported, and the text itself warns that CIDEr 'is generally more unstable and may not be fully reliable.' Because Table 4 provides a proper controlled comparison (same components, same pretraining, only the decoder differs), the paper should either base its state-of-the-art claim on matched-pretraining experiments or add significance testing and interval estimates to the external comparisons. The ActivityNet CIDEr result (18.4 vs. 37.8 for Zhou et al.) also needs explanation beyond the generic instability caveat.
  3. [Section 4.2 and Appendix C] The abstract and Section 3.2.2 state that the model 'uses 20% less compute,' but Table 2 reports only theoretical GFLOP counts for the model at the 8/16-segment settings, not end-to-end training or inference time. Section 3.2.2 asserts 'realized compute savings of 20% during inference, as we observed in our experiments,' but no wall-clock time, peak memory, or throughput measurement is reported, and there is no efficiency comparison with the other streaming method [75]. Please clarify the scope of the compute claim and provide actual runtime and memory measurements for online decoding, including how the savings scale with video length.
  4. [Appendix D] The limitations section candidly states that the model's outputs are more numerous and detailed than the ground truth and that 'even if the model is correct, the metrics will penalize the model.' This is an important caveat for interpreting Table 1, because dense captioning metrics are known to be sensitive to the exact evaluation implementation (e.g., how SODA aligns predicted and ground-truth events, and how temporal NMS is applied). The paper should specify the exact metric computation, including the NMS threshold and the handling of captions with no matching ground truth, so that the comparison to prior published numbers is meaningful.
minor comments (5)
  1. [Section 3.2.2] The statement that the decoder for segment s 'can only see the associated video features for that segment' appears to conflict with Section 3.2.1's claim that the cross-attention inputs have access to all prior video features; please clarify whether prior context is delivered only through the autoregressive memory representation and not through direct cross-attention to earlier segment features.
  2. [Figure 4] The three masks shown in Figure 4 are not described in enough detail in the caption; please add a legend or a sentence explaining the difference between 'Standard Masking,' 'Causal Cross-Segment Masking,' and 'Chunkwise Cross-Segment Masking,' and which one is used in the final model.
  3. [Tables 3 and 5] The ablation differences in Tables 3 and 5 are small (e.g., SODA 4.0 to 4.2), and it is not stated whether the trends are stable across multiple training seeds; please report variance or conduct a paired significance test for the component-wise ablations.
  4. [Section A.1] The text in Section A.1 contains a typo ('couterpart'), and the dataset name is spelled inconsistently as 'VITT' and 'ViTT' in several places; please proofread.
  5. [Reproducibility] The paper does not state whether the code and trained models will be released; for reproducibility, please add an availability statement.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claims are external benchmark results and a measured compute comparison, with only minor non-load-bearing self-citations.

full rationale

The paper's central claims are empirical: the factorized autoregressive decoder improves dense captioning metrics on ViTT, YouCook2, and ActivityNet, and uses 18-20% less compute than a matched global-decoder baseline (Table 2). These are benchmark results against external datasets and a direct FLOPs comparison, not quantities derived from fitted parameters or defined into existence. The per-segment training-label assignment in Section 3.3 ('we make the labels for a segment be the text with an end time within the segments temporal interval') is a training-choice assumption about caption alignment, and the paper itself acknowledges 'this alignment can be noisy if segments do not align well to the captions.' That is a robustness caveat, not a circular reduction: the reported SODA/CIDEr/METEOR/F1 numbers are computed against held-out ground truth, not against the segment-assignment rule. Self-citations to TubeViT (ref 39), MaMMUT (ref 27), and Mirasol3B (ref 41) provide a pretrained vision encoder, initialization, and a dimensionality-reduction inspiration, but the factorized-decoding contribution is not defined by those references, and no load-bearing uniqueness theorem or unverified prior claim is invoked to forbid alternatives. The only same-paper baseline is the global decoder used for the efficiency comparison, which is placed on equal footing rather than used as a fitted input. Therefore no step meets the bar of reducing by construction to its own inputs, and the correct finding is no significant circularity.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The paper makes no pretense of a closed-form derivation; its claims are architectural and empirical. The listed free parameters are the hand-tuned configuration whose sensitivity is documented in the paper. The domain assumptions capture the unproven sufficiency of compressed memory, the segment-aligned labeling rule, and the use of sparse ground truth as the evaluation reference. No new physical or ontological entities are introduced.

free parameters (7)
  • Number of video segments T = 16 for main results
    Controls caption granularity and decode count; Table 5 shows matching segment count to ground truth captions improves metrics and the value is chosen by hand.
  • Input video length and resolution = 512 frames, 448x448, 16 segments
    Main experiments use 512 frames at 448x448; ablations test 32 to 512 frames and this choice affects results.
  • Output tokens per segment l = 32
    Fixed padding length per segment used in the cross-segment mask; it determines the text sequence length and interacts with training and inference.
  • Number of time bins = 32
    Discretization resolution for start and end times; Table 8 shows CIDEr 22.4 at 32 bins versus 12.0 at 128 bins, so the result is very sensitive to this choice.
  • Temporal NMS IoU threshold = 0.7
    Post-processing removes overlapping predictions; the paper reports 0.7 worked best, indicating it was tuned on evaluation data.
  • Decoding settings = 18 samples, temperature 1 (Section 4.3); beam search with 24 outputs (Appendix C)
    The two sections report inconsistent decoding settings, and both were selected during experiments rather than fixed by the method.
  • Optimization hyperparameters = Learning rate 1e-4, batch size 16, label smoothing 0.1, dropout 0.1, weight decay 1e-5
    Training details in Appendix C; no sensitivity analysis is provided for these values.
assumptions (4)
  • domain assumption The compressed autoregressive memory retains enough information from prior segments for the current segment's caption.
    Section 3.2 states the autoregressive transformer functions as memory and allows 'the dense decoders to integrate information from the earlier parts of the video'; the sufficiency of this compression is not proven.
  • domain assumption Assigning each caption to the segment containing its end time yields a useful training signal.
    Section 3.3 defines this labeling rule and acknowledges it 'can be noisy if segments do not align well to the captions,' but assumes the model handles the noise.
  • domain assumption TubeViT features plus dimensionality reduction preserve the visual detail needed for local captions.
    Section 3.2 relies on TubeViT and a transformer reducing K tokens to N; no reconstruction loss or probe verifies that detail is retained.
  • domain assumption Sparse ground truth captions on ViTT, YouCook2, and ActivityNet are the appropriate external reference for dense caption quality.
    The paper uses these benchmarks as ground truth while also arguing the metrics penalize dense outputs, so the validity of the evaluation reference is contested within the paper itself.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning." pith.science (2026). https://pith.science/paper/PQ3O2MLL

@misc{pith2026241114688,
  author       = {Pith},
  title        = {Pith review of: Whats in a Video: Factorized Autoregressive Decoding for Online Dense Video Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQ3O2MLL}},
  note         = {Machine review of arXiv:2411.14688}
}
read the original abstract

Generating automatic dense captions for videos that accurately describe their contents remains a challenging area of research. Most current models require processing the entire video at once. Instead, we propose an efficient, online approach which outputs frequent, detailed and temporally aligned captions, without access to future frames. Our model uses a novel autoregressive factorized decoding architecture, which models the sequence of visual features for each time segment, outputting localized descriptions and efficiently leverages the context from the previous video segments. This allows the model to output frequent, detailed captions to more comprehensively describe the video, according to its actual local content, rather than mimic the training data. Second, we propose an optimization for efficient training and inference, which enables scaling to longer videos. Our approach shows excellent performance compared to both offline and online methods, and uses 20\% less compute. The annotations produced are much more comprehensive and frequent, and can further be utilized in automatic video tagging and in large-scale video data harvesting.

Figures

Figures reproduced from arXiv: 2411.14688 by the authors.

Figure 1
Figure 1. Our online dense video captioning and event localiza [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The model outputs dense captions for long videos, which are generated much more frequently than the ground truth and are more [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Model architecture overview. The model consists of multiple decoders which are responsible for captioning video segments. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Example of standard, global cross-segment masks, and [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Visualizations of the model outputs. Top: samples from the video sequence. Middle: The ground truth. Bottom: Our model [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 52 canonical work pages

  1. [75]

    Streaming dense video captioning

    Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In CVPR, 2024. 3, 6, 7

    work page 2024

  2. [1]

    Flamingo: a visual language model for few-shot learning,

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Bink...

  3. [2]

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and # Predicted Captions # Pred Words # GT Captions # GT Words ViTT Single, global decoder 5.5 25.2 7.1 22.0 Dense Decoder (Ours) 12.4 52.4 7.1 22.0 YouCook2 Single, global decoder 6.8 67.5 8 70.4 Dense Decoder (Ours) 15.4 103.5 8 70.4 Table 10. Comparison of the average number of captions and words per ca...

    work page

  4. [3]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 6

    2005

  5. [4]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 ,

    arXiv

  6. [5]

    Recur- rent memory transformer

    Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. Recur- rent memory transformer. Advances in Neural Information Processing Systems, 35:11079–11091, 2022. 4

    work page 2022

  7. [6]

    Quo vadis, action recognition? a new model and the kinetics dataset

    Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR,

  8. [7]

    PaLI-X: On scaling up a multilingual vision and language model

    Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Xiao Wang Sebastian Goodman, Yi Tay, Daniel Salz Sia- mak Shakeri, Mostafa Dehghani, Mario Lucic, Michael Tschannen, Arsha Nagrani, Hexiang Hu, Mandar Joshi, Bo Pang, Ceslee Montgomery, Paulina Pietrzyk, Marvin Ritter, AJ Piergiovanni, Matthias Mi...

    arXiv 2023

Show all 77 references
  1. [8]

    PaLI: A jointly-scaled multilingual language- image model

    Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergio- vanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, , Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Has- san Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, J...

    2023

  2. [9]

    Videoofa: Two- stage pre-training for video-to-text generation

    Xilun Chen, Lili Yu, Wenhan Xiong, Barlas Oguz, Yashar Mehdad, and Wen-Tau Yih. Videoofa: Two- stage pre-training for video-to-text generation. In arXiv:abs/2305.03204, 2023. 1, 3

    2023 arXiv

  3. [10]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV,

  4. [11]

    Tallformer: Tempo- ral action localization with long-memory transformer

    Feng Cheng and Gedas Bertasius. Tallformer: Tempo- ral action localization with long-memory transformer. In ArXiv:2204.01680, 2022, 2022. 3

    2022 arXiv

  5. [12]

    Monotonic chunkwise attention

    Chung-Cheng Chiu and Colin Raffel. Monotonic chunkwise attention. In ICLR, 2018. 3

    2018

  6. [13]

    Vision transformers need registers

    Timoth ´ee Darcet, Maxime Oquab, Julien Mairal, and Pi- otr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023. 4

    2023 arXiv

  7. [14]

    An empirical study of training end-to-end vision-and-language transformers

    Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An empirical study of training end-to-end vision-and-language transformers. In CVPR, 2022. 3

    2022

  8. [15]

    Violet: End-to-end video-language transformers with masked visual-token mod- eling

    Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End-to-end video-language transformers with masked visual-token mod- eling. In arXiv:2111.1268, 2021. 1, 3

    2021

  9. [16]

    Soda: Story oriented dense video captioning evaluation framework

    Soichiro Fujita, Tsutomu Hirao, Hidetaka Kamigaito, Man- abu Okumura, and Masaaki Nagata. Soda: Story oriented dense video captioning evaluation framework. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16 , pages ...

    2020

  10. [17]

    Mist: Multi-modal iterative spatial- temporal transformer for long-form video question answer- ing

    Difei Gao, Luowei Zhou, Lei Ji, Linchao Zhu, Yi Yang, and Mike Zheng Shou. Mist: Multi-modal iterative spatial- temporal transformer for long-form video question answer- ing. In CVPR, pages 14773–14783, 2023. 3

    2023

  11. [18]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ...

    2017

  12. [19]

    Videollm: Modeling video sequence with large language models

    Jiahao Wang Jilan Xu Yifei Huang Junting Pan Yi Wang Yali Wang Yu Qiao Tong Lu Guo Chen, Yin-Dong Zheng and Limin Wang. Videollm: Modeling video sequence with large language models. https://arxiv.org/abs/2305. 13292), year=2023. 3

    2023

  13. [20]

    Multimodal pretraining for dense video cap- tioning

    Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. Multimodal pretraining for dense video cap- tioning. In AACL-IJCNLP, 2020. 3, 6

    2020

  14. [21]

    A better use of audio-visual cues: Dense video captioning with bi-modal transformer

    Vladimir Iashin and Esa Rahtu. A better use of audio-visual cues: Dense video captioning with bi-modal transformer. In BMVC, 2020. 3

    2020

  15. [22]

    Long movie clip classification with state-space video models

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In ECCV,

  16. [23]

    Perceiver: General perception with iterative attention, 2021

    Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zis- serman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention, 2021. 3, 4

    2021

  17. [24]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021. 9

    2021

  18. [25]

    The ki- netics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The ki- netics human action video dataset. In arXiv preprint arXiv:1705.06950, 2017. 1

    2017 arXiv

  19. [26]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In International Conference on Computer Vision (ICCV), 2017. 3, 6

    2017

  20. [27]

    MaMMUT: A simple architecture for joint learning for mul- timodal tasks

    Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Andrew Dai Lu- owei Zhou, Zhifeng Chen, Claire Cui, and Anelia Angelova. MaMMUT: A simple architecture for joint learning for mul- timodal tasks. In Transactions on Machine Learning Re- sea...

    2023

  21. [28]

    Selvaraju, Akhilesh D

    Junnan Li, Ramprasaath R. Selvaraju, Akhilesh D. Gotmare, Shafiq Joty, Caiming Xiong, and Steven C.H. Hoi. Align be- fore fuse: Vision and language representation learning with momentum distillation. In NeurIPS, 2021. 3

    2021

  22. [29]

    Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. arXiv preprint arXiv:2201.12086, 2022. 3

    2022 arXiv

  23. [30]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. In ICCV, 2023. 1

    2023

  24. [31]

    Oscar: Object-semantics aligned pre-training for vision-language tasks

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV,

  25. [32]

    Eclipse: Efficient long-range video retrieval using sight and sound

    Yan-Bo Lin, Jie Lei, Mohit Bansal, and Gedas Bertasius. Eclipse: Efficient long-range video retrieval using sight and sound. In ECCV, 2022. 3

    2022

  26. [33]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In CVPR, 2019. 3

    2019

  27. [34]

    Unified-io: A unified model for vision, language, and multi-modal tasks

    Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mot- taghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.08916, 2022. 3

    2022 arXiv

  28. [35]

    Univl: A unified video and language pre-training model for multimodal understanding and generation

    Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353, 2020. 1

    2002 arXiv

  29. [36]

    Clip4clip: An empirical study of clip for end to end video clip retrieval

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval. In arXiv: https://arxiv.org/abs/2104.08860, 2021. 3

    2021 arXiv

  30. [37]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019. 6

    2019

  31. [38]

    Moments in time dataset: one million videos for event understanding

    Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ra- makrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl V ondrick, et al. Moments in time dataset: one million videos for event understanding. In PAMI, 2019. 1

    2019

  32. [39]

    Re- thinking video vits: Sparse video tubes for joint image and video learning

    AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Re- thinking video vits: Sparse video tubes for joint image and video learning. CVPR, 2023. 3, 4

    2023

  33. [40]

    Dynamic pretraining of vision-language models

    AJ Piergiovanni, Weicheng Kuo, Wei Li, and Anelia An- gelova. Dynamic pretraining of vision-language models. In First workshop on Multimodal Representation Learning, In- ternational Conference on Learning Representations (ICLR),

  34. [41]

    Mirasol3B: A multi- modal autoregressive model for time-aligned and contextual modalities

    AJ Piergiovanni, Isaac Noble, Dahun Kim, Michael Ryoo, Victor Gomes, and Anelia Angelova. Mirasol3B: A multi- modal autoregressive model for time-aligned and contextual modalities. In CVPR, 2024. 4

    2024

  35. [42]

    Timechat: A time-sensitive multimodallarge language model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodallarge language model for long video understanding. In CVPR, 2024. 7

    2024

  36. [43]

    Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova

    Michael S. Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: Adaptive space-time tokenization for videos. 2021. 4

    2021

  37. [44]

    Ryoo, Keerthana Gopalakrishnan, Kumara Ka- hatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, and Anurag Arnab

    Michael S. Ryoo, Keerthana Gopalakrishnan, Kumara Ka- hatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, and Anurag Arnab. Token turing machines. In CVPR, 2023. 4

    2023

  38. [45]

    Tridet: Temporal action detection with relative boundary modeling

    Dingfeng Shi, Yujie Zhong, Qiong Cao, Lin Ma, Jia Li, and Dacheng Tao. Tridet: Temporal action detection with relative boundary modeling. In CVPR, 2023. 3

    2023

  39. [46]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guil- laume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In CVPR, pages 15638–15650, 2022. 3

    2022

  40. [47]

    Ucf101: A dataset of 101 human action classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human action classes from videos in the wild. In CRCV-TR-12-01, 2012. 1, 3

    2012

  41. [48]

    Long-form video-language pre- training with multimodal temporal contrastive learning

    Yuchong Sun, Hongwei Xue, Ruihua Song, Bei Liu, Huan Yang, and Jianlong Fu. Long-form video-language pre- training with multimodal temporal contrastive learning

  42. [49]

    Lxmert: Learning cross- modality encoder representations from transformers

    Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transformers. In EMNLP, 2019. 3

    2019

  43. [50]

    Clip4caption: Clip for video caption

    Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. Clip4caption: Clip for video caption. In arXiv: 2110.06615, 2021. 3

    2021 arXiv

  44. [51]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 6

    2015

  45. [52]

    Bidirectional attentive fusion with context gating for dense video captioning

    Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR, 2018. 3

    2018

  46. [53]

    Git: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. 1, 3, 7

    2022 arXiv

  47. [54]

    Omnivid: A generative framework for universal video understanding

    Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. Omnivid: A generative framework for universal video understanding. In CVPR,

  48. [55]

    Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052, 2022. 3

    2022 arXiv

  49. [56]

    End-to-end dense video captioning with parallel decoding

    Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In ICCV, 2021. 3, 7

    2021

  50. [57]

    Image as a foreign language: Beit pretraining for all vision and vision- language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision- language tasks. arXiv preprint arXiv:2208.10442, 2022. 3

    2022 arXiv

  51. [58]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He1, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, Sen Xing, Guo Chen, Junting Pan, Jiashuo Yu, Yali Wang, Limin Wang, and Yu Qiao. Internvideo: General video foundation models via generative and discriminative lea...

    2022 arXiv

  52. [59]

    Simvlm: Simple visual language model pretraining with weak supervision

    Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021. 3

    2021 arXiv

  53. [60]

    Vl-bert: Pre-training of generic visual- linguistic representations

    Yue Cao Bin Li Lewei Lu Furu Wei Jifeng Dai Wei- jie Su, Xizhou Zhu. Vl-bert: Pre-training of generic visual- linguistic representations. In ICLR, 2020. 3

    2020

  54. [61]

    Towards long-form video understanding

    Chao-Yuan Wu and Philipp Krahenbuhl. Towards long-form video understanding. In CVPR, 2021. 3

    2021

  55. [62]

    Dibs: Enhanc- ing dense video captioning with unlabeled videos via pseudo boundary enrichment and online refinement

    Hao Wu, Huabin Liu, Yu Qiao, and Xiao Sun. Dibs: Enhanc- ing dense video captioning with unlabeled videos via pseudo boundary enrichment and online refinement. InCVPR, 2024. 7

    2024

  56. [63]

    mplug-2: A modularized multi-modal founda- tion model across text, image and video

    Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, Guohai Xu, Ji Zhang, Songfang Huang, Fei Huang, and Jin- gren Zhou. mplug-2: A modularized multi-modal founda- tion model across text, image and video. In arXiv preprint...

    2023 arXiv

  57. [64]

    Videococa: Video- text modeling with zero-shot transfer from contrastive cap- tioners

    Shen Yan, Tao Zhu, ZiRui Wang, Yuan Cao, Mi Zhang, So- ham Ghosh, Yonghui Wu, and Jiahui Yu. Videococa: Video- text modeling with zero-shot transfer from contrastive cap- tioners. In ArXiV:2212.04979, 2022. 3

    2022 arXiv

  58. [65]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. CVPR,

  59. [66]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mo- jtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022. 3

    2022

  60. [67]

    Hierarchical video-moment retrieval and step-captioning

    Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Bar- las Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 23056–23065, 2023. 3

    2023

  61. [68]

    Mer- lot: Multimodal neural script knowledge models

    Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Mer- lot: Multimodal neural script knowledge models. 2021. 1, 3

    2021

  62. [69]

    Actionformer: Lo- calizing moments of actions with transformers

    Chenlin Zhang, Jianxin Wu, and Yin Li. Actionformer: Lo- calizing moments of actions with transformers. In ECCV,

  63. [70]

    Vinvl: Revisiting visual representations in vision-language models

    Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. arXiv preprint arXiv:2101.00529, 2021. 3

    2021 arXiv

  64. [71]

    Unifying event detec- tion and captioning as sequence generation via pre-training

    Qi Zhang, Yuqing Song, and Qin Jin. Unifying event detec- tion and captioning as sequence generation via pre-training. In ECCV, 2022. 3

    2022

  65. [72]

    Open-ended long-form video question answering via hierarchical convolutional self-attention networks

    Zhu Zhang, Zhou Zhao, Zhijie Lin, Jingkuan Song, and Xi- aofei He. Open-ended long-form video question answering via hierarchical convolutional self-attention networks. In IJ- CAI, 2019. 3

    2019

  66. [73]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 3, 6

    2018

  67. [74]

    End-to-end dense video captioning with masked transformer

    Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. CVPR, 2018. 3, 7

    2018

  68. [76]

    Towards un- derstanding sample variance in visually grounded language generation: Evaluations and observations

    Wanrong Zhu, Xin Eric Wang, Pradyumna Narayana, Kazoo Sone, Sugato Basu, and William Yang Wang. Towards un- derstanding sample variance in visually grounded language generation: Evaluations and observations. arXiv preprint arXiv:2010.03644, 2020. 7

    2010 arXiv

  69. [77]

    Thapliyal, William Yang Wang, and Radu Soricut

    Wanrong Zhu, Bo Pang, Ashish V . Thapliyal, William Yang Wang, and Radu Soricut. End-to-end dense video captioning as sequence generation. In COLING, 2022. 3, 7

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.