Pith. sign in

REVIEW 2 major objections 5 minor 88 references

DistinctAD: Distinctive Audio Description Generation in Contexts

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DistinctAD: Generating movie audio descriptions that avoid repeating the same words across similar consecutive scenes.

desk verdict Solid empirical paper on distinctive AD generation, but Eq. (12) is misprinted or wrong and must be fixed before the method is reproducible. read the letter →

arxiv 2411.18180 v1 pith:KNR7U5OM submitted 2024-11-27 cs.CV

Bo Fang , Wenhao Wu , Qiangqiang Wu , Yuxin Song , Antoni B. Chan This is my paper · ORCID
classification cs.CV
keywords audiodescriptionmovienarrationcontextualredundancydistinctivecaptioningExpectation-MaximizationAttentionCLIPadaptationRecall@k/Nvision-languagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Audio description (AD) for movies and TV narrates what happens visually between lines of dialogue. Generating ADs automatically is hard because (i) pretrained vision-language models were trained on web images, not movies, and (ii) consecutive clips in a movie often look alike, so a model that sees only one clip at a time tends to repeat the same description over and over. This paper proposes a two-stage system, DistinctAD, that first adapts a CLIP vision encoder to the movie-AD domain without needing extra AD text corpora, then adds a module that strips common visual bases from a block of consecutive clips and an extra loss that rewards predicting clip-specific words. The paper claims that on MAD-Eval, CMD-AD, and TV-AD benchmarks, DistinctAD beats prior automatic AD systems, with the largest gains on Recall@k/N, a retrieval-style metric that directly measures whether a generated AD picks out the right clip among temporally neighboring ones. If correct, the method is a step toward ADs that are not just competent captions but actually distinctive, which matters for blind and low-vision audiences who rely on AD to follow what is happening.

What carries the argument

The central objects are (a) a CLIP-AD adaptation stage, which fine-tunes only the CLIP vision encoder using two contrastive losses (global video-to-AD matching and frame-to-word multi-instance loss) while keeping the CLIP text encoder frozen, and (b) a Contextual Expectation-Maximization Attention (EMA) module operating on N consecutive video clips, which iteratively estimates responsibilities, updates K compact bases, reconstructs denoised features $\hat{H}$, and additionally attends from raw features to bases to produce $\hat{e}_H$ features. A distinctive word loss $L_{\text{dist}}$ over a word set created by filtering out duplicates from the N-context ADs is added to the standard auto-regressive loss.

What would settle it

Run the Stage-II training with the printed $L_{\text{dist}}$ term replaced by a loss that, for each token position, sums over all non-repeated distinctive words as in Eq. 12, and observe whether the model's generated ADs become incoherent or the Recall@5/16 gain over the baseline disappears; conversely, if the paper were to release the exact training code, comparing the loss computation in the released code against Eq. 12 would settle whether the reported gain is attributable to $L_{\text{dist}}$ as described.

Watch

Extended reading notes

Core claim

The central claim is that the two failure modes of automatic audio description, a vision-side domain gap and contextual redundancy between neighboring clips, can be addressed by (1) fine-tuning only the CLIP vision encoder with a global video-AD contrastive loss plus a parameter-free fine-grained frame-word multi-instance loss, and (2) adding a Contextual Expectation-Maximization Attention module that compresses and re-attends N consecutive clips' features through K learned bases, together with a distinctive word prediction loss that explicitly pushes the frozen LLM to generate words unique to the current AD rather than words repeated in nearby ADs. The paper reports that this pipeline, with the adapted CLIPAD-B16 encoder and LLaMA3-8B, reaches CIDEr 27.3 and Recall@5/16 56.0 on MAD-Eval and beats AutoAD-III's CIDEr and Recall@1/5 on CMD-AD and TV-AD. The qualitative examples show that prior systems tend to output generic actions like 'looks at something' while DistinctAD tends to name specific objects and actions that identify the current clip.

Load-bearing premise

The load-bearing premise is that the distinctive-word prediction loss, as implemented, truly improves distinctiveness without corrupting normal sentence generation, but the paper only prints a loss formula that appears to conflict with the auto-regressive loss and gives no implementation detail to resolve the discrepancy.

Editorial extensions

If this is right

  • If the central claim holds, automatic AD can move from generic clip-level captions to context-aware narration that flags what changes from one scene to the next, improving accessibility for blind and low-vision viewers and also supporting eye-free watching.
  • The adapted CLIP vision encoder (CLIPAD) is a drop-in replacement that can replace CLIP in other CLIP-based partial-fine-tuning AD systems, so the Stage-I adaptation should transfer to other captioning or narration pipelines.
  • The Contextual EMA and distinctive-word loss are generic enough that the same redundancy-removal recipe could be applied to dense video captioning and to any long-video-to-text task where adjacent segments are semantically redundant.
  • The distinctive-word loss could be made even more explicit by decoding with a constraint that actively avoids the words already present in the neighboring ADs.
  • If the evaluation metric Recall@k/N is trusted as a proxy for distinctiveness, the gains reported here (e.g., 56.0 vs 52.8 for AutoAD-III on MAD-Eval) suggest that the method's main contribution is real, measurable improvement in clip-specificity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own experiments include a non-consecutive-clips control (Table 6) showing that R@5/16 *improves* when the N clips are not consecutive; the paper attributes this to more diverse visual content, which would suggest distinctiveness and local accuracy are at least partially in tension.
  • The distinctive-word loss, as printed in Eq. 12, sums log-probabilities of every distinctive word at every token position, which as written would fight the auto-regressive loss at each position, so the published equation probably does not mirror the actual training objective and would need clarification before the result can be reproduced.
  • Because the paper downloads raw movies and re-aligns timestamps, the reported results implicitly depend on its own re-sorted MAD split (482 movies, 326,632 ADs rather than the original 488 movies and 334,296 ADs), so direct comparisons with numbers reported by other papers on MAD-Eval may not all be on the same training data.
  • The method still leaves a large gap to human annotations (CIDEr ~27 vs human-like reconstruction CIDEr 612 from the paper's own Appendix A), so the practical outcome is better *relative* AD quality within a still-far-from-human regime.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper presents DistinctAD, a two-stage framework for automatic audio description (AD) generation for movies. Stage-I adapts a CLIP vision encoder to movie-AD data via a global video-AD contrastive loss and a fine-grained frame-AD multi-instance matching loss, claiming to close the domain gap without requiring extra AD text corpora. Stage-II processes N consecutive video clips with a Contextual Expectation-Maximization Attention (EMA) module and an explicit distinctive word prediction loss, using a frozen LLM (GPT-2 or LLaMA) to generate ADs. The authors report state-of-the-art results on MAD-Eval, CMD-AD, and TV-AD, with particular gains on the Recall@k/N metric.

Significance. If the proposed training scheme is sound, the paper makes a meaningful contribution to AD generation: the Stage-I adaptation is a lightweight, reusable way to reduce the movie-AD domain gap, and the Contextual EMA offers a principled mechanism to compact redundant visual information across clips. The empirical study is extensive, covering three benchmarks, multiple backbones, and component ablations, and the reported gains over strong baselines such as AutoAD-II/III are substantial. The paper also includes qualitative comparisons and a motivating analysis of the CLIP embedding space. However, the central distinctiveness mechanism is not reproducible from the published equations, and the paper does not address the significant overlap between the proposed loss and the headline evaluation metric.

major comments (2)
  1. [Section 3.2, Eq. (12)] The distinctive-word loss as printed is not a valid autoregressive training objective. At each token position n, the double sum over i=1..u of log P(w_n = w_i^d | prompt, w_<n) asks the model to maximize the probability of all u distinctive words simultaneously, but a softmax output layer normalizes over the vocabulary, so these probabilities compete. This conflicts with the standard autoregressive loss Lauto in Eq. (11), and since u can be large, Ldist can dominate the total loss, potentially destroying next-token generation. Moreover, no masking, position alignment, indicator function, or reweighting is specified to connect the set w_d to the actual target tokens, and the term "word" is not mapped to the LLM's subword vocabulary. The paper must provide a corrected equation and exact implementation details (how w_d is filtered, how words are tokenized, and how the loss is applied in training). Without this, the gains attributed to Ldist in Table 5 (e.g., C3 vs B3: 25.5 vs 23.3 CIDEr and 51.7 vs 48.0 R@5/16) cannot be traced to the stated mechanism.
  2. [Sections 3.2 and 4.1] The distinctive-word loss and the R@k/N metric are closely aligned by design: Ldist explicitly trains the model to produce words that are unique to the current AD among its N neighboring context ADs, while R@k/N evaluates retrieval of the current ground-truth AD among those same N neighboring GT texts using token-overlap-based BERTScore. This overlap means that gains on R@k/N are partly a direct consequence of optimizing for the same notion of distinctiveness, and the paper does not discuss this alignment. To make the distinctiveness claim more convincing, the authors should report an independent evaluation (e.g., human ratings of distinctiveness, or a diversity/novelty metric computed on the generated outputs) and should analyze how much of the R@k/N improvement is due to the explicit loss versus the contextual EMA module, ideally with a version of the model trained without Ldist evaluated on a distinctiveness metric not aligned with the training signal.
minor comments (5)
  1. [Abstract and Table 2] The abstract states that DistinctAD "consistently outperforms baselines," but in Table 2 on CMD-AD, AutoAD-III† reports a higher CIDEr (25.0 vs 22.7). The authors acknowledge this in Section 4.2, but the wording in the abstract should be qualified, e.g., "outperforms previous methods that do not use large-scale external pre-training."
  2. [Section 4.1, Metrics] The definition of R@k/N is incomplete: the paper says it "calculates the average value of Recall@k for each AD with its N temporally adjacent GT texts" using BERTScore, but does not specify the retrieval procedure (e.g., how candidates are formed, whether the current GT is included, and how ties are broken). Please provide an exact formula or a more detailed description, since this is a headline metric.
  3. [Appendix A] The motivating analysis states that ADs reconstructed from CLIP visual features are "often of poor quality," but only the text-reconstruction results are shown in Table A.1. To support the claim that the vision encoder is the bottleneck, the authors should include the corresponding visual-reconstruction experiment.
  4. [Section 3.1, Eq. (3)] The fine-grained frame-AD matching loss samples a single frame f from Vi; please clarify how many frames are sampled per clip and whether this is done repeatedly per batch, as this affects the gradient estimate and the choice of negative set.
  5. [Section 3.2 and Fig. 4] The hyperparameter settings for the Contextual EMA are not fully explained: the temperatures τ in Eqs. (2) and (6) are not given numerical values, and the x-axis labels in Fig. 4 are not shown for all panels. Please specify the used values in the implementation details and ensure the plots are self-contained.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; all central claims are evaluated against held-out benchmarks and no load-bearing step reduces to its own inputs by construction.

full rationale

DistinctAD's three contributions are an empirical system: fine-tuning a CLIP vision encoder on movie-AD pairs (Stage-I), a Contextual EMA module that summarizes N consecutive clips, and a distinctive-word prediction loss added to the autoregressive LLM objective (Stage-II). None of these defines its headline quantities in terms of the evaluation metric. The R@k/N metric ranks generated ADs against neighboring ground-truth ADs using BERTScore on held-out test splits; it is not the training objective, and no test-set parameter is fitted to it. The distinctive-word loss is applied to training examples and the reported gains are measured on unseen benchmarks, so the claim of improved distinctiveness is an empirical result rather than a tautology. The paper does import previous work (EMA, Perceiver, CLIP, LLaMA, AutoAD), but these are standard components and are not invoked to forbid alternatives or to justify a conclusion solely by author self-citation. The only closely related self-citations are Chan's earlier distinctive-captioning papers (refs. 75-77), which appear only in related work and are not load-bearing. The most serious issue in the manuscript is Eq. (12), which as printed sums log-probabilities over all distinctive words at every token position and appears inconsistent with next-token generation; however, this is a correctness and reproducibility concern about whether the published equation describes the implemented loss, not a circularity in which the output is equivalent to the input by definition. Because no specific reduction of a claimed result to its own inputs can be exhibited with a quotation, the appropriate circularity score is 0.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central contribution is empirical; the main free parameters are hyperparameters tuned by ablation on the evaluation benchmark, and the key ad hoc assumption is the validity of the distinctive-word loss. No new physical or conceptual entities are postulated.

free parameters (7)
  • γ (Stage-I loss balance) = 0.5
    Tuned by ablation on MAD-Eval (Tab 4b); balances global video-AD and fine-grained frame-AD losses.
  • α (weight for compact reconstruction bH) = 3
    Tuned by ablation on MAD-Eval (Fig 4a); controls the contribution of EMA-reconstructed features in Hsum.
  • β (weight for cross-attended features eH) = 1
    Tuned by ablation on MAD-Eval (Fig 4b); controls the contribution of the cross-attention branch.
  • K (number of EMA bases) = 32
    Tuned by ablation on MAD-Eval (Fig 4c); number of learned bases in the Contextual EMA module.
  • N (number of consecutive context clips) = 16
    Tuned by ablation on MAD-Eval (Fig 4d); context window size during training and inference.
  • τ (softmax temperatures in Eq. 2 and Eq. 6) = not reported
    Temperatures for frame-word attention and EMA responsibility assignment are not specified, though they affect how sharply attention is concentrated.
  • R (EMA iteration count) = not reported
    The number of responsibility-estimation and likelihood-maximization iterations in Contextual EMA is not specified.
assumptions (4)
  • domain assumption CLIP text embeddings contain enough information to reconstruct ADs with a small projector and GPT-2, so the text encoder can be frozen during adaptation.
    Assumed in Section 3.1 and validated empirically in Appendix A (Tab A.1), but the reconstruction is from text-only inputs and does not prove the vision side is the only source of misalignment.
  • domain assumption Consecutive movie clips share redundant visual content, so removing common bases via EMA improves distinctiveness.
    Hypothesized in Section 3.2 and supported by Appendix B similarity matrices; the benefit depends on this redundancy being present in the test distribution.
  • domain assumption A multiple-instance relationship holds between frames and AD words: each word may appear in only some frames, allowing frame-word attention to align them.
    Invoked in Section 3.1 to justify fine-grained frame-AD matching in Eq. (2)-(3).
  • ad hoc to paper The distinctive word set wd, produced by filtering duplicates from context ADs, is a valid training signal for the LLM.
    Introduced in Section 3.2; the loss in Eq. (12) is not derived and is ambiguous as written, so this assumption is not independently grounded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DistinctAD: Distinctive Audio Description Generation in Contexts." pith.science (2026). https://pith.science/paper/KNR7U5OM

@misc{pith2026241118180,
  author       = {Pith},
  title        = {Pith review of: DistinctAD: Distinctive Audio Description Generation in Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KNR7U5OM}},
  note         = {Machine review of arXiv:2411.18180}
}
read the original abstract

Audio Descriptions (ADs) aim to provide a narration of a movie in text form, describing non-dialogue-related narratives, such as characters, actions, or scene establishment. Automatic generation of ADs remains challenging due to: i) the domain gap between movie-AD data and existing data used to train vision-language models, and ii) the issue of contextual redundancy arising from highly similar neighboring visual clips in a long movie. In this work, we propose DistinctAD, a novel two-stage framework for generating ADs that emphasize distinctiveness to produce better narratives. To address the domain gap, we introduce a CLIP-AD adaptation strategy that does not require additional AD corpora, enabling more effective alignment between movie and AD modalities at both global and fine-grained levels. In Stage-II, DistinctAD incorporates two key innovations: (i) a Contextual Expectation-Maximization Attention (EMA) module that reduces redundancy by extracting common bases from consecutive video clips, and (ii) an explicit distinctive word prediction loss that filters out repeated words in the context, ensuring the prediction of unique terms specific to the current AD. Comprehensive evaluations on MAD-Eval, CMD-AD, and TV-AD benchmarks demonstrate the superiority of DistinctAD, with the model consistently outperforming baselines, particularly in Recall@k/N, highlighting its effectiveness in producing high-quality, distinctive ADs.

Figures

Figures reproduced from arXiv: 2411.18180 by the authors.

Figure 1
Figure 1. (a) Previous methods approach the AD task similar to video captioning, using only a single video clip as input, which leads to repetitive ADs due to highly similar neighboring clips. (b) Our DistinctAD method generates distinctive ADs across N con￾secutive clips, with three key innovations: VLM-AD adaptation, the Distinct Module, and explicit distinctive words prediction. sighted viewers in supporting eye-free activ… view at source ↗
Figure 2
Figure 2. Illustration of Stage-I: CLIP-AD Adaptation. This pro [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pipeline of Stage-II: Distinctive AD Narration. Stage-II processes [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Ablation studies for hyperparameter in Stage-II, with [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results. We present ground-truth (GT) ADs, publicly released AutoAD-Zero outputs, and our DistinctAD predictions [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Visualizations of Contextual EMA. (a) A set of ran [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 58 canonical work pages

  1. [1]

    https://github

    Vlog: Video as a long document. https://github. com/showlab/VLog, 2023. GitHub repository. 6

    2023

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  3. [3]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024. 6, 2

    2024

  4. [4]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 35: 23716–23736, 2022. 1, 3, 4

    2022

  5. [5]

    Spice: Semantic propositional image cap- tion evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In ECCV, pages 382–398. Springer, 2016. 5

    2016

  6. [6]

    Condensed movies: Story based retrieval with con- textual embeddings

    Max Bain, Arsha Nagrani, Andrew Brown, and Andrew Zis- serman. Condensed movies: Story based retrieval with con- textual embeddings. In ACCV, 2020. 5

    2020

  7. [7]

    Whisperx: Time-accurate speech transcription of long- form audio

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisser- man. Whisperx: Time-accurate speech transcription of long- form audio. arXiv preprint arXiv:2303.00747, 2023. 2

    arXiv 2023

  8. [8]

    Livedescribe: can amateur describers create high-quality audio description? Journal of Visual Impairment & Blindness, 106(3):154–165,

    Carmen J Branje and Deborah I Fels. Livedescribe: can amateur describers create high-quality audio description? Journal of Visual Impairment & Blindness, 106(3):154–165,

Show all 88 references
  1. [9]

    End-to-end speaker seg- mentation for overlap-aware resegmentation

    Herv ´e Bredin and Antoine Laurent. End-to-end speaker seg- mentation for overlap-aware resegmentation. In Interspeech,

  2. [10]

    Pyannote

    Herv ´e Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, and Marie-Philippe Gill. Pyannote. audio: neural building blocks for speaker diariza- tion. In ICASSP, pages 7124–7128. IEEE, 2020. 2

    2020

  3. [11]

    Groupcap: Group-based image captioning with structured relevance and diversity constraints

    Fuhai Chen, Rongrong Ji, Xiaoshuai Sun, Yongjian Wu, and Jinsong Su. Groupcap: Group-based image captioning with structured relevance and diversity constraints. In CVPR, pages 1345–1353, 2018. 3

    2018

  4. [12]

    Towards bridging event captioner and sentence localizer for weakly supervised dense event captioning

    Shaoxiang Chen and Yu-Gang Jiang. Towards bridging event captioner and sentence localizer for weakly supervised dense event captioning. In CVPR, pages 8425–8435, 2021. 2

    2021

  5. [13]

    Llm-ad: Large language model based audio description system

    Peng Chu, Jiang Wang, and Andre Abrantes. Llm-ad: Large language model based audio description system. arXiv preprint arXiv:2405.00983, 2024. 1, 3, 6

    2024 arXiv

  6. [14]

    The ides of march

    George Clooney. The ides of march. Columbia Pictures,

  7. [15]

    Contrastive learning for image cap- tioning

    Bo Dai and Dahua Lin. Contrastive learning for image cap- tioning. NeurIPS, 30, 2017. 3

    2017

  8. [16]

    Maximum likelihood from incomplete data via the em al- gorithm

    Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the em al- gorithm. Journal of the royal statistical society: series B (methodological), 39(1):1–22, 1977. 2, 4

    1977

  9. [17]

    Sketch, ground, and refine: Top-down dense video caption- ing

    Chaorui Deng, Shizhe Chen, Da Chen, Yuan He, and Qi Wu. Sketch, ground, and refine: Top-down dense video caption- ing. In CVPR, pages 234–243, 2021. 2

    2021

  10. [18]

    Describing differences in image sets with natural language

    Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In CVPR, pages 24199–24208, 2024. 3

    2024

  11. [19]

    An introduction to audio description: A prac- tical guide, 2016

    Louise Fryer. An introduction to audio description: A prac- tical guide, 2016. 1

    2016

  12. [20]

    Autoad: Movie description in context

    Tengda Han, Max Bain, Arsha Nagrani, G ¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad: Movie description in context. In CVPR, pages 18930–18940, 2023. 1, 2, 3, 4, 5, 6, 7

    2023

  13. [21]

    Autoad ii: The sequel-who, when, and what in movie audio description

    Tengda Han, Max Bain, Arsha Nagrani, Gul Varol, Weidi Xie, and Andrew Zisserman. Autoad ii: The sequel-who, when, and what in movie audio description. In ICCV, pages 13645–13655, 2023. 1, 3, 4, 5, 6, 7

    2023

  14. [22]

    Autoad iii: The prequel-back to the pixels

    Tengda Han, Max Bain, Arsha Nagrani, G ¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad iii: The prequel-back to the pixels. In CVPR, pages 18164–18174, 2024. 1, 2, 3, 5, 6, 7

    2024

  15. [23]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 6

    2021 arXiv

  16. [24]

    A better use of audio-visual cues: Dense video captioning with bi-modal transformer

    Vladimir Iashin and Esa Rahtu. A better use of audio-visual cues: Dense video captioning with bi-modal transformer. In BMVC, 2020. 2

    2020

  17. [25]

    Multi-modal dense video captioning

    Vladimir Iashin and Esa Rahtu. Multi-modal dense video captioning. In CVPRW, pages 958–959, 2020. 2

    2020

  18. [26]

    Expectation- maximization contrastive learning for compact video-and- language representations

    Peng Jin, Jinfa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David Clifton, and Jie Chen. Expectation- maximization contrastive learning for compact video-and- language representations. NeurIPS, 35:30291–30306, 2022. 2

    2022

  19. [27]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 6

    2014 arXiv

  20. [28]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In ICCV, pages 706–715, 2017. 2

    2017

  21. [29]

    Tvqa: Localized, compositional video question answering

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. In EMNLP, 2018. 5

    2018

  22. [30]

    Deep dive: How audio description benefits ev- eryone, 2021

    Elisa Lewis. Deep dive: How audio description benefits ev- eryone, 2021. Accessed on, pages 11–13, 2023. 1 9

    2021

  23. [31]

    Mimic- it: Multi-modal in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic- it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.05425, 2023. 3

    2023 arXiv

  24. [32]

    Otter: a multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: a multi-modal model with in-context instruction tuning. corr abs/2305.03726 (2023), 2023. 3

    2023 arXiv

  25. [33]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023. 1, 3

    2023 arXiv

  26. [34]

    Expectation-maximization attention net- works for semantic segmentation

    Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation-maximization attention net- works for semantic segmentation. In ICCV, pages 9167– 9176, 2019. 2, 4

    2019

  27. [35]

    Jointly localizing and describing events for dense video captioning

    Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. Jointly localizing and describing events for dense video captioning. In CVPR, pages 7492–7500, 2018. 2

    2018

  28. [36]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 5

    2004

  29. [37]

    Swinbert: End-to-end transformers with sparse attention for video cap- tioning

    Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video cap- tioning. In CVPR, pages 17949–17958, 2022. 2

    2022

  30. [38]

    Mm-vid: Advanc- ing video understanding with gpt-4v (ision)

    Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. Mm-vid: Advanc- ing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310.19773, 2023. 1, 2, 6

    arXiv

  31. [39]

    Learning video context as interleaved multimodal sequences

    Kevin Qinghong Lin, Pengchuan Zhang, Difei Gao, Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, and Mike Zheng Shou. Learning video context as interleaved multimodal sequences. arXiv preprint arXiv:2407.21757 ,

    arXiv

  32. [40]

    Swem: Towards real- time video object segmentation with sequential weighted expectation-maximization

    Zhihui Lin, Tianyu Yang, Maomao Li, Ziyu Wang, Chun Yuan, Wenhao Jiang, and Wei Liu. Swem: Towards real- time video object segmentation with sequential weighted expectation-maximization. In CVPR, pages 1362–1372,

  33. [41]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 1

    2024

  34. [42]

    Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data

    Xihui Liu, Hongsheng Li, Jing Shao, Dapeng Chen, and Xi- aogang Wang. Show, tell and discriminate: Image captioning by self-retrieval with partially labeled data. In ECCV, pages 338–354, 2018. 3

    2018

  35. [43]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 6

    2017 arXiv

  36. [44]

    Univl: A unified video and language pre-training model for multimodal understanding and generation

    Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353, 2020. 2

    2002 arXiv

  37. [45]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neu- rocomputing, 508:293–304, 2022. 3

    2022

  38. [46]

    Discriminability objective for training de- scriptive captions

    Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich. Discriminability objective for training de- scriptive captions. In CVPR, pages 6964–6974, 2018. 3

    2018

  39. [47]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pages 2630–2640, 2019. 6, 7

    2019

  40. [48]

    End-to-end learning of visual representations from uncurated instruc- tional videos

    Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instruc- tional videos. In CVPR, pages 9879–9889, 2020. 4

    2020

  41. [49]

    Clip- cap: Clip prefix for image captioning

    Ron Mokady, Amir Hertz, and Amit H Bermano. Clip- cap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734, 2021. 6

    2021 arXiv

  42. [50]

    Streamlined dense video captioning

    Jonghwan Mun, Linjie Yang, Zhou Ren, Ning Xu, and Bo- hyung Han. Streamlined dense video captioning. In CVPR, pages 6588–6597, 2019. 2

    2019

  43. [51]

    Text-only training for image captioning using noise-injected clip.arXiv preprint arXiv:2211.00575, 2022

    David Nukrai, Ron Mokady, and Amir Globerson. Text-only training for image captioning using noise-injected clip.arXiv preprint arXiv:2211.00575, 2022. 6

    2022 arXiv

  44. [52]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. 2023. 1, 2, 6

    2023

  45. [53]

    Rescribe: Authoring and automatically editing audio descriptions

    Amy Pavel, Gabriel Reyes, and Jeffrey P Bigham. Rescribe: Authoring and automatically editing audio descriptions. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pages 747–759, 2020. 1, 2

    2020

  46. [54]

    Gains and losses of watching audio described films for sighted viewers

    Elisa Perego. Gains and losses of watching audio described films for sighted viewers. Target, 28(3):424–444, 2016. 1

    2016

  47. [55]

    Micap: A unified model for identity- aware movie descriptions

    Haran Raajesh, Naveen Reddy Desanur, Zeeshan Khan, and Makarand Tapaswi. Micap: A unified model for identity- aware movie descriptions. In CVPR, pages 14011–14021,

  48. [56]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 2, 1

    2019

  49. [57]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. In ICML, pages 8748–8763. PMLR, 2021. 2, 1

    2021

  50. [58]

    Watch, listen and tell: Multi-modal weakly supervised dense event captioning

    Tanzila Rahman, Bicheng Xu, and Leonid Sigal. Watch, listen and tell: Multi-modal weakly supervised dense event captioning. In ICCV, pages 8908–8917, 2019. 2

    2019

  51. [59]

    A dataset for movie description

    Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele. A dataset for movie description. In CVPR, pages 3202–3212, 2015. 2

    2015

  52. [60]

    Movie description

    Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. Movie description. IJCV, 123:94–120,

  53. [61]

    End-to-end generative pretraining for mul- timodal video captioning

    Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. End-to-end generative pretraining for mul- timodal video captioning. In CVPR, pages 17959–17968,

  54. [62]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In ACL, pages 2556–2565, 2018. 2

    2018

  55. [63]

    Weakly super- vised dense video captioning

    Zhiqiang Shen, Jianguo Li, Zhou Su, Minjun Li, Yurong Chen, Yu-Gang Jiang, and Xiangyang Xue. Weakly super- vised dense video captioning. In CVPR, pages 1916–1924,

    1916

  56. [64]

    Dense procedure captioning in narrated instructional videos

    Botian Shi, Lei Ji, Yaobo Liang, Nan Duan, Peng Chen, Zhendong Niu, and Ming Zhou. Dense procedure captioning in narrated instructional videos. In ACL, pages 6382–6391,

  57. [65]

    What does clip know about a red circle? visual prompt engineering for vlms

    Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vlms. In ICCV, pages 11987–11997,

  58. [66]

    Audio description: The visual made verbal

    Joel Snyder. Audio description: The visual made verbal. In International congress series , pages 935–939. Elsevier,

  59. [67]

    Mad: A scalable dataset for language grounding in videos from movie audio descriptions

    Mattia Soldan, Alejandro Pardo, Juan Le ´on Alc´azar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. In CVPR, pages 5026–5035,

  60. [68]

    Using descriptive video services to create a large data source for video annotation research

    Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville. Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070, 2015. 2

    2015 arXiv

  61. [69]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 6

    2023 arXiv

  62. [70]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008. 8

    2008

  63. [71]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In CVPR, pages 4566–4575, 2015. 5

    2015

  64. [72]

    Joint optimization for cooperative image captioning

    Gilad Vered, Gal Oren, Yuval Atzmon, and Gal Chechik. Joint optimization for cooperative image captioning. In ICCV, pages 8898–8907, 2019. 3

    2019

  65. [73]

    Contextual ad narration with interleaved mul- timodal sequence

    Hanlin Wang, Zhan Tong, Kecheng Zheng, Yujun Shen, and Limin Wang. Contextual ad narration with interleaved mul- timodal sequence. arXiv preprint arXiv:2403.12922, 2024. 1, 2, 3, 4, 5

    2024 arXiv

  66. [74]

    Bidirectional attentive fusion with context gating for dense video captioning

    Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context gating for dense video captioning. In CVPR, pages 7190–7198, 2018. 2

    2018

  67. [75]

    Compare and reweight: Distinctive image caption- ing using similar images sets

    Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. Compare and reweight: Distinctive image caption- ing using similar images sets. In ECCV, pages 370–386. Springer, 2020. 3

    2020

  68. [76]

    Group-based distinctive image captioning with mem- ory attention

    Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. Group-based distinctive image captioning with mem- ory attention. In ACMMM, pages 5020–5028, 2021

    2021

  69. [77]

    On distinctive image captioning via comparing and reweighting

    Jiuniu Wang, Wenjia Xu, Qingzhong Wang, and Antoni B Chan. On distinctive image captioning via comparing and reweighting. TPAMI, 45(2):2088–2103, 2022. 3

    2022

  70. [78]

    Event-centric hierarchical representation for dense video captioning

    Teng Wang, Huicheng Zheng, Mingjing Yu, Qian Tian, and Haifeng Hu. Event-centric hierarchical representation for dense video captioning. TCSVT, 31(5):1890–1900, 2020. 2

    1900

  71. [79]

    End-to-end dense video captioning with parallel decoding

    Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In ICCV, pages 6847–6857, 2021. 2

    2021

  72. [80]

    Non-local neural networks

    Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaim- ing He. Non-local neural networks. In CVPR, pages 7794– 7803, 2018. 4

    2018

  73. [81]

    Autoad-zero: A training-free framework for zero-shot audio description

    Junyu Xie, Tengda Han, Max Bain, Arsha Nagrani, G ¨ul Varol, Weidi Xie, and Andrew Zisserman. Autoad-zero: A training-free framework for zero-shot audio description. arXiv preprint arXiv:2407.15850, 2024. 1, 2, 3, 4, 5, 6, 7, 8

    2024 arXiv

  74. [82]

    Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning

    Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, An- toine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a vi- sual language model for dense video captioning. In CVPR, pages 10714–10726, 2023. 2

    2023

  75. [83]

    Image difference cap- tioning with pre-training and contrastive learning

    Linli Yao, Weiying Wang, and Qin Jin. Image difference cap- tioning with pre-training and contrastive learning. In AAAI, pages 3108–3116, 2022. 3

    2022

  76. [84]

    Videoblip, 2023

    Keunwoo Peter Yu. Videoblip, 2023. 6

    2023

  77. [85]

    Mm-narrator: Narrating long-form videos with multimodal in-context learning

    Chaoyi Zhang, Kevin Lin, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. Mm-narrator: Narrating long-form videos with multimodal in-context learning. In CVPR, pages 13647–13657, 2024. 1, 2, 4, 6

    2024

  78. [86]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 6

    2023 arXiv

  79. [87]

    Bertscore: Evaluating text genera- tion with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text genera- tion with bert. arXiv preprint arXiv:1904.09675, 2019. 5

    1904 arXiv

  80. [88]

    nuns” mistakenly appears in (d). AutoAD-II tends to gen- erates similar AD words, e.g. “furrowed brow

    Luowei Zhou, Yingbo Zhou, Jason J Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In CVPR, pages 8739–8748, 2018. 2 11 DistinctAD: Distinctive Audio Description Generation in Contexts Supplementary Material A. Analysis of AD...

    2002

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.