Pith. sign in

REVIEW 3 major objections 6 minor 63 references

DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that correcting three transfer gaps—vision, language, and alignment—lets a 0.56M-parameter CLIP adapter beat full fine-tuning on video-text retrieval.

desk verdict Solid and well-ablated parameter-efficient video-text retrieval method, but the SOTA claim needs error bars and a proper validation split before the 1.5% margin can be trusted. read the letter →

arxiv 2506.08887 v1 pith:TEAANA2D submitted 2025-06-10 cs.CV

Leqi Shen , Guoqiang Gong , Tianxiang Hao , Tao He , Yifeng Zhang , Pengzhang Liu , Sicheng Zhao , Jungong Han
show 1 more author
Guiguang Ding
This is my paper
classification cs.CV
keywords video-textretrievalparameter-efficienttransferlearningCLIPadaptationdiscrepancyreductionvision-languagealignmentpseudocaptiongenerationdistillationtemporalmodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiscoVLA is a parameter-efficient method that adapts the frozen image-text model CLIP to video-text retrieval. It argues that transferring CLIP from images to videos suffers from three separate discrepancies: vision (missing temporal information), language (image captions versus video captions), and alignment (video-level matching is more complex than image-level matching). Existing parameter-efficient methods mostly address the vision gap alone, so DiscoVLA tackles all three at once with an image-video feature fusion, pseudo image-level alignment from generated captions, and image-to-video alignment distillation. With only 0.56M trainable parameters, it reaches 50.5% text-to-video R@1 on MSRVTT and outperforms prior methods on four benchmark datasets.

What carries the argument

The load-bearing mechanism is IVFusion, a parameter-efficient attention variant that processes sampled frames with image-level attention and applies spatio-temporal attention only to the CLS tokens, merging the two branches with a lightweight adapter to keep complexity at $O(FN^2 + F^2N)$. PImgAlign supplies the fine-grained training signal: pseudo image captions produced from frames plus the video caption, matched to images through a max-over-row-and-column similarity, which tolerates one caption covering several frames. AlignDistill then distills the resulting image-level similarity distributions into the video-level similarity used for retrieval; both PImgAlign and AlignDistill are discarded at inference so the retrieval cost stays unchanged.

What would settle it

On a benchmark with ground-truth frame-level captions (or a human-annotated subset), compare the generated pseudo captions against the ground truth and check whether retrieval gains appear only when caption accuracy is high. Alternatively, corrupt the pseudo captions with random substitutions of nouns or verbs and measure the drop in R@1, which would reveal whether PImgAlign is learning from caption content or merely from the presence of extra training pairs.

Watch

Extended reading notes

Core claim

The central claim is that the transfer from image-level CLIP to video-level retrieval can be decomposed into vision, language, and alignment discrepancies, and that jointly reducing all three yields state-of-the-art parameter-efficient video-text retrieval. On MSRVTT with CLIP (ViT-B/16), DiscoVLA attains 50.5% text-to-video R@1 and 49.2% video-to-text R@1, surpassing previous parameter-efficient methods and the full fine-tuning baseline while updating only 0.56M parameters. The gains come from three coupled modules: IVFusion merges image- and video-level features in both encoders; PImgAlign learns fine-grained image-level alignment from pseudo image captions generated by a multimodal language model; AlignDistill transfers that image-level alignment knowledge to video level via KL divergence.

Load-bearing premise

DiscoVLA's alignment gains depend on the pseudo image captions generated by a multimodal language model from a few frames and the video caption being accurate enough to serve as a training signal; the paper does not measure caption quality and only ablates the video-caption guidance.

Editorial extensions

If this is right

  • If DiscoVLA's gains hold, parameter-efficient video-text retrieval can match or beat full fine-tuning while updating two orders of magnitude fewer parameters.
  • Methods that focus only on temporal modeling are leaving language and alignment discrepancies on the table; addressing all three yields consistent improvements across MSRVTT, LSMDC, ActivityNet, and DiDeMo.
  • Image-level alignment learned from pseudo captions, combined with distillation, transfers CLIP's image-text matching strength to video-level matching without any inference-time caption generation.
  • The IVFusion complexity reduction, which applies spatio-temporal attention only to CLS tokens, makes video-level feature fusion computationally affordable in large pre-trained encoders.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable corollary not in the paper: the value of PImgAlign should scale with the quality of the pseudo captions; replacing the multimodal language model with a weaker captioner (or adding caption noise) would isolate how much of the improvement is due to caption fidelity rather than the alignment loss itself.
  • The max-over-row-and-column similarity acts as a soft assignment between frames and captions; in datasets with one-to-many frame-caption relations, other assignment strategies such as optimal transport might perform better, which the paper does not explore.
  • Because PImgAlign is training-only, the method effectively uses the video caption as a prior to re-caption frames; this suggests a general recipe for upgrading image-text backbones to video tasks without changing the inference-time architecture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes DiscoVLA, a parameter-efficient adaptation of CLIP for video-text retrieval. It identifies three discrepancies between image-level and video-level understanding (vision, language, and alignment) and introduces three components: IVFusion, which fuses image-level and video-level features via a lightweight adapter in both the vision and text encoders; PImgAlign, which uses LLaVA-NeXT-generated pseudo image captions to learn fine-grained image-level alignment; and AlignDistill, which distills image-level alignment knowledge into video-level alignment via KL divergence. The method uses only 0.56M trainable parameters and reports state-of-the-art results on MSRVTT, LSMDC, ActivityNet, and DiDeMo.

Significance. If the empirical results are robust, this work makes a solid contribution to parameter-efficient video-text retrieval. The main strength is the IVFusion module, which provides a substantial gain in the ablation (8.7 R@sum over the LoRA baseline) while adding only about 0.05M parameters, and the overall framework is well-structured with extensive comparisons and ablations. The release of code is a strength. However, the central claim of state-of-the-art performance is weakened by the absence of any statistical significance testing and by the use of the MSRVTT test set for hyperparameter selection, which may bias the reported headline numbers.

major comments (3)
  1. [Section 4.2, Tables 1-4] The paper reports single-run results with no error bars, confidence intervals, or multiple seeds. The improvement over TempMe on MSRVTT is 0.9% R@1 (ViT-B/32) and 1.5% R@1 (ViT-B/16), and on other datasets the margins vary from 0.4 to 2.0 R@1. In video-text retrieval, seed-level variation is typically on the order of 1 R@1, so these margins may be within noise. The claim that DiscoVLA "significantly outperforms" prior methods is not supported by statistical evidence. Please report mean and standard deviation over at least 3 random seeds for the main comparisons and for the key ablations, or at minimum provide the number of seeds and a discussion of expected variance.
  2. [Appendix B, Figures 6-9; Section 4.1] The hyperparameters α, β, H_V, and H_L are tuned on MSRVTT using text-to-video results (Figures 6-9). According to Appendix A, the model is trained on the 9,000 train+val pairs and evaluated on the 1K-A test set, so the test set is used for model selection. This makes the MSRVTT results, including the headline 50.5% R@1, optimistically biased. Please clarify whether a separate validation split was used; if not, either use a held-out validation split for hyperparameter selection and then retrain on the combined training+validation set, or show that the chosen hyperparameters are stable by reporting results on a validation split for each dataset.
  3. [Section 3.4, Table 8] The PImgAlign contribution relies on the accuracy of LLaVA-NeXT pseudo image captions, but the paper does not evaluate caption quality directly. The only quantitative evidence is the indirect ablation in Table 8, which shows a 1.6 R@sum drop when video caption guidance is removed. This does not invalidate the overall method, because the IVFusion-only variant (B1 in Table 5) already outperforms prior methods, but it leaves the proposed mechanism of PImgAlign under-validated. Please include a caption-quality evaluation (e.g., comparison with ground-truth frame descriptions or human assessment) or a more direct test of how caption correctness affects retrieval performance.
minor comments (6)
  1. [Abstract] The GitHub URL "DsicoVLA" appears to be a typo; it should be "DiscoVLA".
  2. [Equations (2) and (3)] The contrastive loss expressions appear to be missing a negative sign and proper parentheses; the standard form is -log( exp(sim_ii/τ) / Σ_j exp(sim_ij/τ) ). Please correct the notation.
  3. [Section 4.3, last paragraph] The sentence "we evaluate the effects of hyperparameters α, β, H V and H V" should read "H V and H L".
  4. [Table 9 caption] In the table caption and text, "MSTVTT" should be "MSRVTT".
  5. [Figure 2] The text "Utilized ONL Y during the training phase" contains a typo; it should be "ONLY".
  6. [Section 3.4 / Experiments] The paper does not report the training-time computational cost of generating pseudo image captions with LLaVA-NeXT. Please add a note on the additional offline training overhead.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DiscoVLA's components are standard supervised objectives, PImgAlign is training-only, and the TempMe self-citation is a baseline comparison, not a load-bearing premise.

full rationale

DiscoVLA is an empirical supervised-learning paper, not a derivation chain, and I find no step in which an output is constructed from its own input. The three modules are trained with standard objectives: video-level contrastive loss L_A(Sim_vid) (Eq. 4), image-level contrastive loss L_A(Sim_img) via pseudo captions (Eqs. 4 and 10), and KL distillation L_KL (Eq. 13), combined in Eq. (14). PImgAlign's pseudo captions are generated by an external MLLM (LLaVA-NeXT [35]) and are used only in training; the paper explicitly states 'PImgAlign is omitted at inference' and 'The final similarity used for retrieval is based on video-level similarity in Eq. (1).' Thus the reported retrieval numbers are not equal by construction to the pseudo-caption signal or to any fitted constant. Hyperparameters alpha, beta, H_V, and H_L are selected by ablations on MSRVTT (Appendix B), which is a model-selection issue, not a circular prediction. The only self-citation is TempMe [44], a concurrent arXiv baseline by the same first author; it is cited solely as a comparison method in Table 1 and is not used to justify any mechanism, ansatz, or uniqueness claim. The quality of LLaVA-NeXT pseudo captions is not quantitatively measured (Section 3.4/Table 8), but that is an assumption about training-signal quality, not circularity. No equation reduces to another by construction, and no fitted input is renamed as a prediction.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper's central claim is an empirical performance improvement. It introduces no new physical or mathematical entities. The free parameters are standard hyperparameters but are tuned on the test/validation split of one benchmark (MSRVTT) without multi-seed variance, so they carry some selection risk. The main unstated premise is that MLLM-generated pseudo captions are a faithful image-level text source.

free parameters (6)
  • LoRA rank r = 8
    Set for all experiments; controls capacity of the low-rank adaptation in attention layers. Chosen value is standard for LoRA.
  • Adapter bottleneck dimension r = 8
    Bottleneck size of IVFusion adapter (Eq. 7), set to 8 for all experiments.
  • alpha (weight on image-level alignment loss) = 0.3
    Tuned on MSRVTT (Figure 6); controls relative weight of L_A(Sim_img).
  • beta (weight on distillation loss) = 1.0
    Tuned on MSRVTT (Figure 7); controls L_KL weight.
  • H_V (number of IVFusion layers in vision encoder) = 4
    Tuned on MSRVTT (Figure 8); higher values improve performance until 4.
  • H_L (number of IVFusion layers in text encoder) = 2
    Tuned on MSRVTT (Figure 9).
assumptions (5)
  • domain assumption CLIP provides transferable image-text alignment.
    The method pretrains from CLIP (ViT-B/16 or B/32) and assumes its frozen features are a sufficient base for video-text retrieval; this is the standard premise of the cited parameter-efficient line of work.
  • domain assumption LLaVA-NeXT pseudo captions are a reliable image-level supervision source.
    PImgAlign depends on MLLM-generated captions from frames. The paper ablates W/ and W/O video caption guidance (Table 8) but does not quantify caption quality or failure cases.
  • domain assumption Benchmarks are valid proxies for video-text retrieval performance.
    MSRVTT, LSMDC, ActivityNet, DiDeMo are used following standard protocol; the paper assumes R@K and MnR are appropriate metrics.
  • domain assumption Temporal average pooling is a fair video feature aggregation baseline.
    The method and all compared parameter-efficient baselines use mean pooling over frame features (Section 2, Table 1 caption); the paper assumes this aggregation does not handicap comparisons.
  • domain assumption Max-similarity over image-caption pairs correctly resolves ambiguity.
    Eqs. (8)-(10) assume that taking row/column maxima of the image-caption similarity matrix handles many-to-many correspondences; ablation in Table 7 supports this but the assumption is explicit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval." pith.science (2026). https://pith.science/paper/TEAANA2D

@misc{pith2026250608887,
  author       = {Pith},
  title        = {Pith review of: DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TEAANA2D}},
  note         = {Machine review of arXiv:2506.08887}
}
read the original abstract

The parameter-efficient adaptation of the image-text pretraining model CLIP for video-text retrieval is a prominent area of research. While CLIP is focused on image-level vision-language matching, video-text retrieval demands comprehensive understanding at the video level. Three key discrepancies emerge in the transfer from image-level to video-level: vision, language, and alignment. However, existing methods mainly focus on vision while neglecting language and alignment. In this paper, we propose Discrepancy Reduction in Vision, Language, and Alignment (DiscoVLA), which simultaneously mitigates all three discrepancies. Specifically, we introduce Image-Video Features Fusion to integrate image-level and video-level features, effectively tackling both vision and language discrepancies. Additionally, we generate pseudo image captions to learn fine-grained image-level alignment. To mitigate alignment discrepancies, we propose Image-to-Video Alignment Distillation, which leverages image-level alignment knowledge to enhance video-level alignment. Extensive experiments demonstrate the superiority of our DiscoVLA. In particular, on MSRVTT with CLIP (ViT-B/16), DiscoVLA outperforms previous methods by 1.5% in R@1, reaching a final score of 50.5% R@1. The code is available at https://github.com/LunarShen/DsicoVLA.

Figures

Figures reproduced from arXiv: 2506.08887 by the authors.

Figure 1
Figure 1. When parameter-efficient transferring image-level CLIP [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of DiscoVLA. Initially, we generate pseudo image captions for each sampled image. In both vision and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the encoder layers for vision and text encoders. (a) Image-Level Attn operates on each of the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Visualization of discrepancies between video-level and image-level data. Each example consists of a paired video and video [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Comparison of image caption generation with (W/) and [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Ablation study on α in Eq. (14) for text-to-video results on MSRVTT using CLIP (ViT-B/32). α represents the weight of the image-level alignment loss LA(Simimg). All other hyperpa￾rameters are kept constant [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Ablation study on β in Eq. (14) for text-to-video results on MSRVTT using CLIP (ViT-B/32). β represents the weight of the distillation loss LKL. All other hyperparameters are kept con￾stant. the video-level alignment loss LA(Simvid), the image-level alignment loss LA(S…
Figure 8
Figure 8. Figure 8: Ablation study on HV for text-to-video results on MSRVTT using CLIP (ViT-B/32). HV represents the number of IVFusion layers in the vision encoder. All other hyperparameters are kept constant [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Ablation study on HL for text-to-video results on MSRVTT using CLIP (ViT-B/32). HL represents the number of IVFusion layers in the text encoder. All other hyperparameters are kept constant. Post-processing Text-to-Video R@1 R@5 R@10 R@sum MSRVTT DiscoVLA 47.0 73.0 82.8…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 49 canonical work pages

  1. [1]

    Localizing mo- ments in video with natural language

    Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing mo- ments in video with natural language. InProceedings of the International Conference on Computer Vision, 2017. 5, 12

    work page 2017

  2. [2]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the In- ternational Conference on Computer Vision, 2015. 2

    work page 2015

  3. [3]

    Frozen in time: A joint video and image encoder for end-to-end retrieval

    Max Bain, Arsha Nagrani, G ¨ul Varol, and Andrew Zisser- man. Frozen in time: A joint video and image encoder for end-to-end retrieval. InProceedings of the International Conference on Computer Vision, 2021. 12

    work page 2021

  4. [4]

    Cross modal retrieval with querybank normalisation

    Simion-Vlad Bogolin, Ioana Croitoru, Hailin Jin, Yang Liu, and Samuel Albanie. Cross modal retrieval with querybank normalisation. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 8, 13

    work page 2022

  5. [5]

    RAP: Efficient text-video retrieval with sparse-and- correlated adapter

    Meng Cao, Haoran Tang, Jinfa Huang, Peng Jin, Can Zhang, Ruyang Liu, Long Chen, Xiaodan Liang, Li Yuan, and Ge Li. RAP: Efficient text-video retrieval with sparse-and- correlated adapter. InAssociation for Computational Lin- guistics, 2024. 2, 3, 5, 6, 7, 12

    work page 2024

  6. [6]

    Adaptformer: Adapt- ing vision transformers for scalable visual recognition.Ad- vances in Neural Information Processing Systems, 2022

    Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapt- ing vision transformers for scalable visual recognition.Ad- vances in Neural Information Processing Systems, 2022. 2

    work page 2022

  7. [7]

    Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.Advances in Neural Information Processing Sys- tems, 2023

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset.Advances in Neural Information Processing Sys- tems, 2023. 2

    work page 2023

  8. [8]

    Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss.arXiv preprint arXiv:2109.04290, 2021

    Xing Cheng, Hezheng Lin, Xiangyu Wu, Fan Yang, and Dong Shen. Improving video-text retrieval by multi-stream corpus alignment and dual softmax loss.arXiv preprint arXiv:2109.04290, 2021. 8, 13

    arXiv 2021

Show all 63 references
  1. [9]

    Prompt switch: Efficient clip adaptation for text-video re- trieval

    Chaorui Deng, Qi Chen, Pengda Qin, Da Chen, and Qi Wu. Prompt switch: Efficient clip adaptation for text-video re- trieval. InProceedings of the International Conference on Computer Vision, 2023. 1, 2

    2023

  2. [10]

    Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks

    Xiaohan Ding, Yuchen Guo, Guiguang Ding, and Jungong Han. Acnet: Strengthening the kernel skeletons for powerful cnn via asymmetric convolution blocks. InProceedings of the International Conference on Computer Vision, 2019. 2

    2019

  3. [11]

    Repvgg: Making vgg-style convnets great again

    Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021. 2

    2021

  4. [12]

    Improving clip training with language rewrites.Advances in Neural Information Processing Sys- tems, 2024

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites.Advances in Neural Information Processing Sys- tems, 2024. 2

    2024

  5. [13]

    Multi-modal transformer for video retrieval

    Valentin Gabeur, Chen Sun, Karteek Alahari, and Cordelia Schmid. Multi-modal transformer for video retrieval. In Proceedings of the European Conference on Computer Vi- sion, 2020. 12

    2020

  6. [14]

    X-pool: Cross-modal language-video attention for text- video retrieval

    Satya Krishna Gorti, No ¨el V ouitsis, Junwei Ma, Keyvan Golestan, Maksims V olkovs, Animesh Garg, and Guangwei Yu. X-pool: Cross-modal language-video attention for text- video retrieval. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 1, 2

    2022

  7. [15]

    Accurate, large mini- batch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017

    Priya Goyal, Piotr Doll ´ar, Ross Girshick, Pieter Noord- huis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large mini- batch sgd: Training imagenet in 1 hour.arXiv preprint arXiv:1706.02677, 2017. 12

    2017 arXiv

  8. [16]

    Framewise phoneme classification with bidirectional lstm and other neural net- work architectures.Neural Networks, 2005

    Alex Graves and J ¨urgen Schmidhuber. Framewise phoneme classification with bidirectional lstm and other neural net- work architectures.Neural Networks, 2005. 2

    2005

  9. [17]

    Towards a unified view of parameter-efficient transfer learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. InProceedings of the International Conference on Learning Representations,

  10. [18]

    Secret: Self-consistent pseudo label refinement for unsupervised domain adaptive person re-identification

    Tao He, Leqi Shen, Yuchen Guo, Guiguang Ding, and Zhen- hua Guo. Secret: Self-consistent pseudo label refinement for unsupervised domain adaptive person re-identification. InProceedings of the AAAI Conference on Artificial Intel- ligence, 2022. 2

    2022

  11. [19]

    Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415, 2016

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus).arXiv preprint arXiv:1606.08415, 2016. 4

    2016 arXiv

  12. [20]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InProceedings of the International Con- ference on Machine Learning, 2019. 2, 5, 6

    2019

  13. [21]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021. 2, 5, 6

    2021 arXiv

  14. [22]

    V op: Text-video co- operative prompt tuning for cross-modal retrieval

    Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, and Donglin Wang. V op: Text-video co- operative prompt tuning for cross-modal retrieval. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023. 1, 2, 3, 5, 6, 7, 12

    2023

  15. [23]

    Vi- sual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Vi- sual prompt tuning. InProceedings of the European Confer- ence on Computer Vision, 2022. 2

    2022

  16. [24]

    Video- text as game players: Hierarchical banzhaf interaction for cross-modal representation learning

    Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, and Jie Chen. Video- text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. InProceedings of the IEEE Conference on Computer Vision and Pattern Reco...

    2023

  17. [25]

    Mv-adapter: Multimodal video transfer learning for video text retrieval

    Xiaojie Jin, Bowen Zhang, Weibo Gong, Kai Xu, Xueqing Deng, Peng Wang, Zhao Zhang, Xiaohui Shen, and Jiashi Feng. Mv-adapter: Multimodal video transfer learning for video text retrieval. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024. 2, ...

    2024

  18. [26]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015. 2

    2015

  19. [27]

    Maple: Multi-modal prompt learning

    Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

  20. [28]

    Self-regulating prompts: Foundational model adaptation without forgetting

    Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shah- baz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. InProceedings of the Inter- national Conference on Computer Vision, 2023. 2

    2023

  21. [29]

    Dense-captioning events in videos

    Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the International Conference on Computer Vision, 2017. 5, 12

    2017

  22. [30]

    Courier Corporation, 1997

    Solomon Kullback.Information theory and statistics. Courier Corporation, 1997. 5

    1997

  23. [31]

    Less is more: Clipbert for video-and-language learning via sparse sampling

    Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. InPro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021. 12

    2021

  24. [32]

    Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in Neural Infor- mation Processing Systems, 2021

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learn- ing with momentum distillation.Advances in Neural Infor- mation Processing Systems, 2021. 2

    2021

  25. [33]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InProceed- ings of the International Conference on Machine Learning, 2022

    2022

  26. [34]

    Unmasked teacher: Towards training-efficient video foundation models

    Kunchang Li, Yali Wang, Yizhuo Li, Yi Wang, Yinan He, Limin Wang, and Yu Qiao. Unmasked teacher: Towards training-efficient video foundation models. InProceedings of the International Conference on Computer Vision, 2023. 2

    2023

  27. [35]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 4, 8

    2024

  28. [36]

    Sgdr: Stochastic gradient descent with warm restarts

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. InProceedings of the Interna- tional Conference on Learning Representations, 2016. 12

    2016

  29. [37]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neu- rocomputing, 2022

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning.Neu- rocomputing, 2022. 1, 2, 5, 6

    2022

  30. [38]

    Ea-vtr: Event-aware video-text retrieval

    Zongyang Ma, Ziqi Zhang, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing Li, Yingmin Luo, Xu Li, Xiaojuan Qi, Ying Shan, et al. Ea-vtr: Event-aware video-text retrieval. In Proceedings of the European Conference on Computer Vi- sion, 2024. 5

    2024

  31. [39]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. InProceedings of the International Conference on Computer Vision, 2019. 12

    2019

  32. [40]

    Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018. 3

    2018 arXiv

  33. [41]

    Language models are unsu- pervised multitask learners.OpenAI blog, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners.OpenAI blog, 2019. 2

    2019

  34. [42]

    Learn- ing transferable visual models from natural language super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn- ing transferable visual models from natural language super- vision. InProceedings of the International Conference o...

    2021

  35. [43]

    The long-short story of movie description

    Anna Rohrbach, Marcus Rohrbach, and Bernt Schiele. The long-short story of movie description. InPattern Recogni- tion, 2015. 5, 12

    2015

  36. [44]

    Tempme: Video temporal token merging for efficient text-video re- trieval.arXiv preprint arXiv:2409.01156, 2024

    Leqi Shen, Tianxiang Hao, Sicheng Zhao, Yifeng Zhang, Pengzhang Liu, Yongjun Bao, and Guiguang Ding. Tempme: Video temporal token merging for efficient text-video re- trieval.arXiv preprint arXiv:2409.01156, 2024. 6

    2024 arXiv

  37. [45]

    X-reid: Cross-instance transformer for identity-level person re- identification

    Leqi Shen, Tao He, Sicheng Zhao, Zhelun Shen, Yuchen Guo, Tianshi Xu, and Guiguang Ding. X-reid: Cross-instance transformer for identity-level person re- identification. InIEEE International Conference on Multi- media and Expo, 2024. 2

    2024

  38. [46]

    Zerocap: Zero-shot image-to-text generation for visual- semantic arithmetic

    Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text generation for visual- semantic arithmetic. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 2

    2022

  39. [47]

    Yolov10: Real-time end-to-end object de- tection.Advances in Neural Information Processing Systems,

    Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jun- gong Han, et al. Yolov10: Real-time end-to-end object de- tection.Advances in Neural Information Processing Systems,

  40. [48]

    Text is mass: Modeling as stochastic embedding for text-video retrieval

    Jiamian Wang, Guohao Sun, Pichao Wang, Dongfang Liu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Text is mass: Modeling as stochastic embedding for text-video retrieval. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024. 1, 2

    2024

  41. [49]

    Disentangled representation learning for text- video retrieval.arXiv preprint arXiv:2203.07111, 2022

    Qiang Wang, Yanhao Zhang, Yun Zheng, Pan Pan, and Xi- ansheng Hua. Disentangled representation learning for text- video retrieval.arXiv preprint arXiv:2203.07111, 2022. 1, 2

    2022 arXiv

  42. [50]

    Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning.arXiv preprint arXiv:2212.03191, 2022. 2

    2022 arXiv

  43. [51]

    Cap4video: What can auxiliary captions do for text-video retrieval? InProceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, 2023

    Wenhao Wu, Haipeng Luo, Bo Fang, Jingdong Wang, and Wanli Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? InProceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, 2023. 1, 2

    2023

  44. [52]

    Demystifying clip data

    Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. InProceedings of the International Conference on Learning Representations, 2024. 2

    2024

  45. [53]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. 5, 12

    2016

  46. [54]

    Show, attend and tell: Neural image caption gen- eration with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gen- eration with visual attention. InProceedings of the Interna- tional Conference on Machine Learning, 2015. 2

    2015

  47. [55]

    Clip-vip: Adapting pre-trained image-text model to video-language alignment

    Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, and Jiebo Luo. Clip-vip: Adapting pre-trained image-text model to video-language alignment. InProceedings of the International Conference on Learning Representations, 2022. 2

    2022

  48. [56]

    Llmi3d: Empowering llm with 3d perception from a single 2d image.arXiv preprint arXiv:2408.07422, 2024

    Fan Yang, Sicheng Zhao, Yanhao Zhang, Haoxiang Chen, Hui Chen, Wenbo Tang, Haonan Lu, Pengfei Xu, Zhenyu Yang, Jungong Han, et al. Llmi3d: Empowering llm with 3d perception from a single 2d image.arXiv preprint arXiv:2408.07422, 2024. 2

    2024 arXiv

  49. [57]

    Heie: Mllm-based hierarchical explainable aigc image implausibil- ity evaluator.arXiv preprint arXiv:2411.17261, 2024

    Fan Yang, Ru Zhen, Jianing Wang, Yanhao Zhang, Haoxiang Chen, Haonan Lu, Sicheng Zhao, and Guiguang Ding. Heie: Mllm-based hierarchical explainable aigc image implausibil- ity evaluator.arXiv preprint arXiv:2411.17261, 2024. 2

    2024 arXiv

  50. [58]

    Dgl: Dynamic global-local prompt tuning for text-video re- trieval.Proceedings of the AAAI Conference on Artificial Intelligence, 2024

    Xiangpeng Yang, Linchao Zhu, Xiaohan Wang, and Yi Yang. Dgl: Dynamic global-local prompt tuning for text-video re- trieval.Proceedings of the AAAI Conference on Artificial Intelligence, 2024. 1, 2, 3, 5, 6, 12

    2024

  51. [59]

    Cross-modal and hierarchical modeling of video and text

    Bowen Zhang, Hexiang Hu, and Fei Sha. Cross-modal and hierarchical modeling of video and text. InProceedings of the European Conference on Computer Vision, 2018. 12

    2018

  52. [60]

    Neural prompt search.arXiv preprint arXiv:2206.04673, 2022

    Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search.arXiv preprint arXiv:2206.04673, 2022. 2

    2022 arXiv

  53. [61]

    Conditional prompt learning for vision-language mod- els

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language mod- els. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 2

    2022

  54. [62]

    Learning to prompt for vision-language models.Inter- national Journal of Computer Vision, 2022

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.Inter- national Journal of Computer Vision, 2022. 2 DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval Supple...

    2022

  55. [63]

    14,αandβare set to0.3and1.0, respectively

    In Eq. 14,αandβare set to0.3and1.0, respectively. For the number of IVFusion layers, we setH V = 4for the vision encoder andH L = 2for the text encoder. B. Additional Experimental Results Ablation study onαandβin Eq. (14).Figures 6 and 7 present ablation studies on hyperparame...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.