Pith. sign in

REVIEW 4 major objections 5 minor 42 references

Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a zero-shot video captioner can be trained entirely on synthetic video latents generated from captions, with a polishing step making those latents close enough to real-video latents for direct inference.

desk verdict A promising synthetic-visual training pipeline for zero-shot video captioning with strong reported gains, but the headline comparison is weakened by training on target-dataset captions and an unmeasured synthetic-to-real latent transfer. read the letter →

arxiv 2608.11013 v1 pith:NZGBQNV5 submitted 2026-08-11 cs.CV

Liangyu Fu , Junbo Wang , Yuke Li , Ya Jing , Xuecheng Wu , Zhiyong Wang This is my paper · ORCID
classification cs.CV
keywords zero-shotvideocaptioningtext-onlytrainingmodalitygaptext-to-videogenerationsyntheticlatentsvisuallatentpolishingsoftpromptconditioningGPT-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to make zero-shot video captioning work without any real video during training by replacing the missing video distribution with synthetic video. The proposed WSV framework feeds text captions into a text-to-video model, intercepts the generated video's compressed internal representation (the latent), and trains a polisher to push that latent closer to real-video latents. A prompter then turns the polished latent into soft prompts that steer GPT-2 to generate captions. The paper reports that this scheme beats previous text-only and zero-shot methods on MSR-VTT, MSVD, and VATEX, with the largest gains on MSR-VTT and MSVD.

What carries the argument

The load-bearing mechanism is latent-space visual synthesis with a distribution polisher. CogVideoX synthesizes a latent video $V_g$ from text; the polisher $A_\phi$, a lightweight 3D residual CNN, maps it to $V_p$ while preserving the $C \times F \times H \times W$ spatio-temporal dimensions, using a frozen CLIP4Clip encoder as supervision to keep the polished latent semantically aligned with the caption. The prompter $P_\theta$ then compresses $V_p$ into a short sequence of learnable prompt embeddings via 3D CNN features and alternating cross-attention and self-attention, and GPT-2 generates captions from those prompts. This design is what lets training happen entirely on synthetic latents while inference runs on real VAE latents.

What would settle it

Measure a quantitative distributional distance, such as Frechet Video Distance or maximum mean discrepancy, between polished synthetic latents $V_p$ and real encoder latents $E_{\mathrm{VAE}}(v_{\mathrm{real}})$ on a held-out split. If that distance is not substantially smaller than the distance between unpolished synthetic latents and real latents, or if captioning performance does not track this distance across different text-to-video backbones, the central claim that the polisher bridges the modality gap is falsified.

Watch

Extended reading notes

Core claim

The core claim is that the cross-modal gap in text-only video captioning can be closed by making the visual distribution visible during training, rather than by trying to project text into visual space. Given a caption, the text-to-video model CogVideoX produces a latent video representation $V_g$; a 3D convolutional polisher $A_\phi$, trained with a symmetric cross-entropy contrastive loss under CLIP4Clip supervision, refines this into $V_p$ so that it better matches the latents produced by the VAE encoder on real videos. A prompter $P_\theta$ maps $V_p$ to a fixed-length sequence of prompt embeddings prepended to GPT-2, and the captioner is trained with the standard next-token objective. At inference the T2V model and polisher are discarded: a real video is encoded by the same 3D causal VAE and fed directly to the prompter. The paper reports 33.9 B@4, 27.7 METEOR, 55.7 ROUGE-L, and 45.5 CIDEr on MSR-VTT, and 52.0 B@4 and 95.7 CIDEr on MSVD, with ablations showing the polisher, prompter, and GPT-2 fine-tuning each contribute.

Load-bearing premise

The claim assumes that polished synthetic video latents are close enough to the raw latents of real videos from the same VAE encoder that a captioner trained on polished latents will work on unpolished real latents, even though the polisher is removed at inference.

Editorial extensions

If this is right

  • If WSV is right, text-only training for video captioning no longer needs to approximate visual space from text; it can simply generate the visual space.
  • The reported gains imply that synthetic visual supervision can substitute for real video in zero-shot settings: on MSR-VTT, CIDEr rises from 33.1 to 45.5 over the previous best zero-shot result.
  • The ablation results imply the polisher is essential: removing it drops MSR-VTT CIDEr from 45.5 to 40.4, so raw text-to-video latents alone are not enough.
  • The framework is not tied to one generator: swapping CogVideoX for Wan2.2-T2V keeps performance nearly unchanged, so any semantically coherent text-to-video model can supply the synthetic latents.
  • Fine-tuning GPT-2 with the prompter is part of the mechanism: freezing it lowers MSR-VTT CIDEr from 45.5 to 41.8, indicating the language model must adapt to the prompt distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if synthetic latents are a faithful stand-in for real video latents, the same recipe could be repurposed for other video-language tasks that currently require paired data, such as video question answering, by swapping the prompter and decoder for a task-specific head.
  • Editorial inference: the paper's t-SNE evidence is qualitative; a quantitative latent-distance benchmark would test the stronger claim that polishing, rather than prompter robustness, is what closes the modality gap.
  • Editorial inference: an implied prediction is that improvements in text-to-video generators should transfer directly into zero-shot captioning gains without any new video data, since the synthetic latents are the only visual supervision.
  • Editorial inference: a testable extension is to check whether the residual gap matters more for long, temporally complex videos than for short clips, since the paper's argument emphasizes temporal semantic information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes WSV, a zero-shot video captioning framework trained with text only. It first uses a pretrained text-to-video model (CogVideoX) to synthesize video latents from captions, then trains a 3D CNN 'polisher' with a CLIP4Clip contrastive loss so that the decoded synthetic videos align with the captions in CLIP embedding space. A 'prompter' is then trained, together with GPT-2, to map polished latents to soft prompts and autoregressively generate captions. At inference, a real video is encoded by the CogVideoX VAE encoder and fed directly to the prompter, bypassing the polisher. Experiments on MSVD, MSR-VTT, and VATEX report large gains over prior zero-shot methods, with B@4/CIDEr of 52.0/95.7 on MSVD and 33.9/45.5 on MSR-VTT. The central claim is that synthetic visual supervision can eliminate the modality gap in zero-shot video captioning.

Significance. If the central claim holds, the paper makes a useful conceptual contribution: instead of projecting text into visual space with linear maps, it synthesizes actual video latents from text and trains the captioner on those latents. The reported gains over prior zero-shot methods are substantial, and the ablations consistently show that the synthetic visual stage, the polisher, and the prompter each contribute. The paper is also unusually concrete in specifying the training stages and the frozen/generated components. However, the significance is conditional on two load-bearing issues: whether the protocol is truly zero-shot given that the training text corpus is drawn from the evaluation datasets, and whether the polisher actually closes the gap between synthetic and real VAE latents in the space that the prompter consumes. Without addressing these, the reported state-of-the-art numbers do not yet substantiate the method's central claim.

major comments (4)
  1. [§IV-A and §III-A] The training corpus is formed by mixing text from MSVD, MSR-VTT, and VATEX—the very datasets used for evaluation. If captions from the test splits are included, the model is directly trained on reference captions for the test videos, which is label leakage and invalidates the zero-shot comparison. Even if only training-split captions are used, the text corpus is from the target datasets, so the setting is not zero-shot in the standard sense used by prior methods like DeCap, which train on an external corpus. The comparison to external-corpus methods in Table I is therefore not apples-to-apples. The paper must either retrain with a text corpus disjoint from the evaluation datasets, or explicitly justify why training on the target datasets' captions is a legitimate zero-shot protocol.
  2. [§III-A Eq. (5) and §III-C Eq. (12)] The polisher is trained without ever seeing a real video latent. Its only supervision is the CLIP4Clip contrastive loss on decoded pixels, and it is removed at inference, where raw E_VAE(v_real) latents are fed to the prompter. The evidence for the polisher closing the real-vs-synthetic gap is t-SNE of CLIP video representations (Figs. 4 and 5), not a quantitative distributional comparison in the VAE latent space that the prompter actually consumes. t-SNE can hide large metric distances, and Fig. 4 itself shows a visible pre-polish gap. The paper should report a latent-space distributional statistic (e.g., MMD or FID computed on VAE latents) and, ideally, validate that a prompter trained on polished synthetic latents performs comparably when the polisher is applied to real latents versus when it is removed.
  3. [§III-A Eq. (1), §III-B Eq. (10)] Because V_g is generated from the caption t_i and the captioner is trained to predict t_i from V_p ≈ G_T2V(t_i), the captioner may partly learn to invert the text-to-video model rather than to ground captions in visual content. The synthetic latents are text-conditioned by construction, whereas real video latents at inference are not text-conditioned. The paper does not test whether the prompter generalizes to real latents beyond the reported end-task numbers. A concrete test would be to generate synthetic latents from captions that are disjoint from the training captions and evaluate the captioner on those, or to probe the prompter with real latents whose captions are held out.
  4. [Table I] The 'Zero-shot' section of Table I includes models that are not zero-shot under the paper's definition: AuroraCap-7B is trained on large-scale video-text pairs, and Video-LLaMA/LLaVa are instruction-tuned multimodal models. Including supervised models in the same block obscures the comparison. The paper should clearly separate methods by training protocol and avoid claiming 'state-of-the-art among zero-shot methods' if the strongest competitor is supervised. Additionally, all results in Tables I–VIII are reported without error bars or significance tests; given the small margins in several ablations (e.g., Table VII: 0.3 B@4 and 0.7 CIDEr), single-run numbers are not sufficient to establish the superiority of the chosen configurations.
minor comments (5)
  1. [§III-A Eq. (2)] The notation is inconsistent: Eq. (2) writes the polished latent as V, while the text and other equations use V_p. Please define V_p explicitly in Eq. (2).
  2. [§III-C] There is a missing period and a sentence fragment: 'The real latent V_{g-real} is then processed by the prompter The entire visual-to-prompt encoding can be expressed as Eq. (12).' This should be corrected.
  3. [Table III] In the row for two CNN blocks, the value appears as '233.9' instead of '33.9'; please fix the spacing/formatting.
  4. [Fig. 6] The visual comparison of 'before' and 'after' polisher is a single subjective example; consider adding quantitative video-quality metrics or multiple examples with a user study.
  5. [§IV-A] The datasets section says the standard splits include test sets, but it does not state which captions are used for the training text corpus. Please specify explicitly whether the corpus is the union of training-split captions only, or also includes validation/test captions.

Circularity Check

1 steps flagged · score 3.0 of 10

Training loop is self-referential (synthetic latents are generated from the same captions used as labels), but the ablations and real-video test evaluation give the central claim independent empirical content; the remaining transfer assumption is unverified rather than circular.

  1. self definitional [Section III-A Eq. (1) and Section III-B Eq. (6)-(10)]
    "we feed each caption t_i into CogVideoX, G_T2V... we intercept the generation process to extract the final latent representation V_g... This process results in a new pseudo-dataset D_L={(V_g^i, t_i)}... We use our pseudo-dataset D_L again. For each pair (V_g, t_i), the generative latent V_g is passed through the frozen polisher... the model is trained to predict the next token in the caption t_i."

    The pseudo-dataset's visual latent is produced by G_T2V from the same caption t_i that serves as the training label in Stage 2 (after polishing, V_p = A_phi(G_T2V(t_i))). The captioner is therefore optimized to invert a text-conditioned function on the training distribution; success on synthetic latents is by construction expected because the latent already encodes the caption's semantics. This does not by itself prove that the model can caption real videos; that transfer depends on the additional, unmeasured assumption that E_VAE(v_real) lies close to A_phi(G_T2V(t_i)) in the 4D VAE latent space. No term in L_con or L_CE compares polished synthetic latents with real VAE latents; Fig. 5 is a t-SNE of CLIP embeddings, not a distributional test in the latent space consumed by the prompter.

full rationale

The paper's central claim is an empirical transfer claim: a captioner trained on polished synthetic latents works on raw real latents at inference. The training-data construction is partly self-referential because CogVideoX generates V_g from the caption t_i that is later used as the label; this can inflate apparent train-time success. However, the reported numbers are test-time predictions on real videos, and the ablations (w/o Synthetic Visual, w/o Polisher, w/o Prompter) show that each component contributes gains on those test sets, providing independent empirical content beyond caption memorization. The main weakness is not a circular derivation but an unverified assumption: the polisher is removed at inference and its CLIP-based contrastive loss never sees a real VAE latent, so the residual gap between A_phi(G_T2V(t)) and E_VAE(v_real) is unmeasured. That is a correctness risk, not a circularity. No self-citation chain or imported uniqueness theorem is load-bearing. One self-referential training step warrants a score around 3 rather than 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 2 invented entities

The central result depends on two newly introduced modules (polisher and prompter), a pretrained T2V generator, a supervised CLIP4Clip signal, and several hyperparameters chosen by hand or by ablation. No new physical entities are proposed. The free parameters are standard for an empirical ML paper but are not reported with uncertainty, and the architecture searches in Tables III-IV select hyperparameters on the same benchmark used for the final claim.

free parameters (6)
  • Contrastive temperature tau = not reported
    Used in Eq. 5 for the CLIP4Clip contrastive loss; temperature is a scale parameter, typically tuned, and its value is not specified.
  • Polisher CNN block count = 2
    Selected as best in Table III ablation; 3 or 4 blocks performed worse.
  • Prompter attention block count = 4
    Selected from Table IV; 8 blocks nearly tied, 4 chosen for lower cost.
  • Learning rates = 1e-4 and 2e-5
    Chosen by hand for Stage 1 and Stage 2 (Section IV-C); no schedule or search is reported.
  • Number of prompt tokens N_p = not reported
    The prompter output sequence length is a hyperparameter that controls how much visual information reaches GPT-2; it is never specified.
  • Batch size B = not reported
    Both the contrastive loss and the captioner training depend on batch size, but the value is not given.
assumptions (4)
  • domain assumption Synthetic latents from CogVideoX diffusion and real latents from the CogVideoX VAE encoder share a semantically comparable latent space.
    Section III-A extracts V_g before the VAE decoder, and Section III-C encodes real videos with the same VAE encoder; the paper's transfer claim depends on this shared space but only offers t-SNE as evidence.
  • domain assumption CLIP4Clip similarity between decoded synthetic videos and captions is a valid training signal for closing the real-synthetic distribution gap.
    The polisher is trained exclusively with the contrastive loss in Eq. 5 on synthetic videos; real videos are never seen in Stage 1.
  • ad hoc to paper Using captions from the evaluation datasets as the text-only training corpus is a legitimate zero-shot protocol.
    Section IV-A mixes text from MSR-VTT, MSVD, and VATEX into training, so the caption distribution seen at test is already in the training text; this is a weaker setting than methods trained on external corpora.
  • domain assumption Fine-tuning GPT-2 on soft prompts derived from synthetic latents preserves its language modeling ability for real-video captions.
    Equations 10 and 11 update GPT-2 parameters on synthetic latent prompts; the paper assumes this does not degrade generation for the wider caption distribution.
invented entities (2)
  • Polisher A_phi
    purpose: 3D residual CNN that refines synthetic video latents to reduce mismatch with real video latents.
    The polisher is only evaluated indirectly through downstream captioning metrics; it is not applied to real videos at inference, so its distribution-alignment claim is not directly measured.
  • Prompter P_theta
    purpose: Maps 4D video latents to a fixed-length sequence of soft prompt embeddings for GPT-2.
    It is a new trainable module whose only evidence is the downstream benchmark gains; there is no external validation of the prompt embeddings.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning." pith.science (2026). https://pith.science/paper/NZGBQNV5

@misc{pith2026260811013,
  author       = {Pith},
  title        = {Pith review of: Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NZGBQNV5}},
  note         = {Machine review of arXiv:2608.11013}
}
read the original abstract

Text-only training is a popular paradigm in zero-shot video captioning, where the video distribution is not available to the model during training, leading to a cross-modal gap between the training (text-only) and the inference (video-only). Previous works attempt to bridge the gap through simple linear transformations. However, the inherent gap between text and video makes cross-modal representation space alignment insufficient, resulting in inaccurate sentences. To address this issue, we propose a novel zero-shot video captioning framework (WSV) consisting of two training stages, which first generates corresponding synthetic video latent representations via a pretrained text-to-video generation model. To strengthen the fidelity of the latent representations, we propose a polisher capable of bridging the gap between real and synthetic video distributions. Subsequently, we design a prompter that conditions GPT-2 on the polished latent representations to generate the captions in the second training stage. During inference, an input video is encoded by a pretrained 3D Causal VAE and then fed directly into the prompter, which in turn guides GPT-2 to produce the final caption. Experimental results conducted on MSVD, MSR-VTT, and VATEX datasets demonstrate that our proposed method achieves scores of 52 and 95.7 on the B@4 and CIDEr metrics, respectively.

Figures

Figures reproduced from arXiv: 2608.11013 by the authors.

Figure 1
Figure 1. The comparison between (a) Previous Zero-shot Captioning Method [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of WSV. Regular text denotes annotated textual descriptions from the corpus, while italic text denotes generated textual [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Three case studies from the MSR-VTT dataset. Each sample contains textual descriptions from two video captioning models and one labeled description. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The t-SNE of the real video representation and the synthetic video [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The t-SNE of the real video representation and the synthetic video [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Comparison of a video sample before and after the Polisher. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 25 canonical work pages

  1. [1]

    Learning to compose topic-aware mixture of experts for zero-shot video captioning,

    X. Wang, J. Wu, D. Zhang, Y . Su, and W. Y . Wang, “Learning to compose topic-aware mixture of experts for zero-shot video captioning,” inAAAI, vol. 33, no. 01, 2019, pp. 8965–8972

    work page 2019

  2. [2]

    Decap: Decoding clip la- tents for zero-shot captioning via text-only training,

    W. Li, L. Zhu, L. Wen, and Y . Yang, “Decap: Decoding clip la- tents for zero-shot captioning via text-only training,”arXiv preprint arXiv:2303.03032, 2023

    arXiv 2023

  3. [3]

    Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data,

    Y . Zhang, E. Sui, and S. Yeung-Levy, “Connect, collapse, corrupt: Learning cross-modal tasks with uni-modal data,”arXiv preprint arXiv:2401.08567, 2024

    arXiv 2024

  4. [4]

    IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioning

    S. Lee, S.-W. Kim, T. Kim, and D.-J. Kim, “Ifcap: Image-like retrieval and frequency-based entity filtering for zero-shot captioning,”arXiv preprint arXiv:2409.18046, 2024

    work page Pith review arXiv 2024

  5. [5]

    Improving cross-modal alignment with synthetic pairs for text-only image captioning,

    Z. Liu, J. Liu, and F. Ma, “Improving cross-modal alignment with synthetic pairs for text-only image captioning,” inAAAI, vol. 38, no. 4, 2024, pp. 3864–3872

    work page 2024

  6. [6]

    Retta: Retrieval-enhanced test-time adaptation for zero-shot video cap- tioning,

    Y . Ma, L. Qing, G. Li, Y . Qi, A. Beheshti, Q. Z. Sheng, and Q. Huang, “Retta: Retrieval-enhanced test-time adaptation for zero-shot video cap- tioning,”PR, p. 112170, 2025

    work page 2025

  7. [7]

    Text-only training for image captioning using noise-injected CLIP,

    D. Nukrai, R. Mokady, and A. Globerson, “Text-only training for image captioning using noise-injected CLIP,” inEMNLP. Association for Computational Linguistics, 2022, pp. 4055–4063

    work page 2022

  8. [8]

    Sequence to sequence-video to text,

    S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko, “Sequence to sequence-video to text,” inICCV, 2015, pp. 4534–4542

    work page 2015

Show all 42 references
  1. [9]

    Bidirectional long- short term memory for video description,

    Y . Bin, Y . Yang, F. Shen, X. Xu, and H. T. Shen, “Bidirectional long- short term memory for video description,” inACM MM, 2016, pp. 436– 440

    2016

  2. [10]

    Graph convolutional network meta- learning with multi-granularity pos guidance for video captioning,

    P. Li, P. Zhang, and X. Xu, “Graph convolutional network meta- learning with multi-granularity pos guidance for video captioning,” Neurocomputing, vol. 472, pp. 294–305, 2022

    2022

  3. [11]

    Describing videos by exploiting temporal structure,

    L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville, “Describing videos by exploiting temporal structure,” in ICCV, 2015, pp. 4507–4515

    2015

  4. [12]

    Icocap: Improving video captioning by compounding images,

    Y . Liang, L. Zhu, X. Wang, and Y . Yang, “Icocap: Improving video captioning by compounding images,”IEEE TMM, vol. 26, pp. 4389– 4400, 2024

    2024

  5. [13]

    Memory- based augmentation network for video captioning,

    S. Jing, H. Zhang, P. Zeng, L. Gao, J. Song, and H. T. Shen, “Memory- based augmentation network for video captioning,”IEEE Transactions on Multimedia, vol. 26, pp. 2367–2379, 2024

    2024

  6. [14]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”NIPS, vol. 30, 2017

    2017

  7. [15]

    Hierarchical modular network for video captioning,

    H. Ye, G. Li, Y . Qi, S. Wang, Q. Huang, and M.-H. Yang, “Hierarchical modular network for video captioning,” inCVPR, 2022, pp. 17 939– 17 948

    2022

  8. [16]

    Swinbert: End-to-end transformers with sparse attention for video captioning,

    K. Lin, L. Li, C.-C. Lin, F. Ahmed, Z. Gan, Z. Liu, Y . Lu, and L. Wang, “Swinbert: End-to-end transformers with sparse attention for video captioning,” inCVPR, 2022, pp. 17 949–17 958

    2022

  9. [17]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,”arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  10. [18]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding,

    B. He, H. Li, Y . K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S.-N. Lim, “Ma-lmm: Memory-augmented large multimodal model for long-term video understanding,” inCVPR, 2024, pp. 13 504–13 514

    2024

  11. [19]

    Text-only training for image captioning using noise-injected clip,

    D. Nukrai, R. Mokady, and A. Globerson, “Text-only training for image captioning using noise-injected clip,”arXiv preprint arXiv:2211.00575, 2022

    2022 arXiv

  12. [20]

    Language models can see: Plugging visual controls in text generation,

    Y . Su, T. Lan, Y . Liu, F. Liu, D. Yogatama, Y . Wang, L. Kong, and N. Collier, “Language models can see: Plugging visual controls in text generation,”arXiv preprint arXiv:2205.02655, 2022

    2022 arXiv

  13. [21]

    Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic,

    Y . Tewel, Y . Shalev, I. Schwartz, and L. Wolf, “Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic,” inCVPR, 2022, pp. 17 918–17 928

    2022

  14. [22]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inICML. PmLR, 2021, pp. 8748–8763

    2021

  15. [23]

    From association to genera- tion: Text-only captioning by unsupervised cross-modal mapping,

    J. Wang, M. Yan, Y . Zhang, and J. Sang, “From association to genera- tion: Text-only captioning by unsupervised cross-modal mapping,”arXiv preprint arXiv:2304.13273, 2023

    2023 arXiv

  16. [24]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning,

    H. Luo, L. Ji, M. Zhong, Y . Chen, W. Lei, N. Duan, and T. Li, “Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning,”Neurocomputing, vol. 508, pp. 293–304, 2022

    2022

  17. [25]

    Cogvideox: Text-to-video diffusion models with an expert transformer,

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y . Yang, W. Hong, X. Zhang, G. Fenget al., “Cogvideox: Text-to-video diffusion models with an expert transformer,”arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  18. [26]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  19. [27]

    Delving deeper into the decoder for video captioning,

    H. Chen, J. Li, and X. Hu, “Delving deeper into the decoder for video captioning,”arXiv preprint arXiv:2001.05614, 2020

    2001 arXiv

  20. [28]

    Improving video captioning with temporal composition of a visual-syntactic embedding,

    J. Perez-Martin, B. Bustos, and J. P ´erez, “Improving video captioning with temporal composition of a visual-syntactic embedding,” inProceed- ings of the IEEE/CVF winter conference on applications of computer vision, 2021, pp. 3039–3049

    2021

  21. [29]

    Zero- shot video captioning with evolving pseudo-tokens,

    Y . Tewel, Y . Shalev, R. Nadler, I. Schwartz, and L. Wolf, “Zero- shot video captioning with evolving pseudo-tokens,”arXiv preprint arXiv:2207.11100, 2022

    2022 arXiv

  22. [30]

    MultiCapCLIP: Auto-encoding prompts for zero-shot multilingual visual captioning,

    B. Yang, F. Liu, X. Wu, Y . Wang, X. Sun, and Y . Zou, “MultiCapCLIP: Auto-encoding prompts for zero-shot multilingual visual captioning,” in ACL, Jul. 2023, pp. 11 908–11 922

    2023

  23. [31]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”NIPS, vol. 36, pp. 34 892–34 916, 2023

    2023

  24. [32]

    Auroracap: Efficient, perfor- mant video detailed captioning and a new benchmark,

    W. Chai, E. Song, Y . Du, C. Meng, V . Madhavan, O. Bar-Tal, J.- N. Hwang, S. Xie, and C. D. Manning, “Auroracap: Efficient, perfor- mant video detailed captioning and a new benchmark,”arXiv preprint arXiv:2410.03051, 2024

    2024 arXiv

  25. [33]

    Msr-vtt: A large video description dataset for bridging video and language,

    J. Xu, T. Mei, T. Yao, and Y . Rui, “Msr-vtt: A large video description dataset for bridging video and language,” inCVPR, 2016, pp. 5288– 5296

    2016

  26. [34]

    Collecting highly parallel data for paraphrase evaluation,

    D. Chen and W. B. Dolan, “Collecting highly parallel data for paraphrase evaluation,” inACL, 2011, pp. 190–200

    2011

  27. [35]

    Vatex: A large-scale, high-quality multilingual dataset for video-and-language research,

    X. Wang, J. Wu, J. Chen, L. Li, Y .-F. Wang, and W. Y . Wang, “Vatex: A large-scale, high-quality multilingual dataset for video-and-language research,” inICCV, 2019, pp. 4581–4591

    2019

  28. [36]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inACL, 2002, pp. 311– 318

    2002

  29. [37]

    Meteor universal: Language specific translation evaluation for any target language,

    M. Denkowski and A. Lavie, “Meteor universal: Language specific translation evaluation for any target language,” inProceedings of the ninth workshop on statistical machine translation, 2014, pp. 376–380

    2014

  30. [38]

    Rouge: A package for automatic evaluation of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” inText summarization branches out, 2004, pp. 74–81

    2004

  31. [39]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” inCVPR, 2015, pp. 4566–4575

    2015

  32. [40]

    Decoupled weight decay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  33. [41]

    Expanding language-image pretrained models for general video recognition,

    B. Ni, H. Peng, M. Chen, S. Zhang, G. Meng, J. Fu, S. Xiang, and H. Ling, “Expanding language-image pretrained models for general video recognition,” inECCV. Springer, 2022, pp. 1–18

    2022

  34. [42]

    Wan: Open and advanced large-scale video generative models,

    T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, F. Yu, H. Zhao, J. Yanget al., “Wan: Open and advanced large-scale video generative models,”arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.