Pith. sign in

REVIEW 3 major objections 4 minor 55 references

VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read VisCoP adapts video-language models to shifted domains by attaching learnable probes to a frozen vision encoder, reporting gains of +3.5 to +19.3 on target benchmarks while retaining source skills.

desk verdict VisCoP is a sensible parameter-efficient adaptation idea with good ablations, but the headline ego/depth gains are compromised by a likely train/eval overlap that the paper never rules out. read the letter →

arxiv 2510.13808 v2 pith:F54ABT46 submitted 2025-10-15 cs.CV

Dominick Reilly , Manish Kumar Govind , Le Xue , Srijan Das This is my paper · ORCID
classification cs.CV
keywords videodomainadaptationvision-languagemodelsvisualprobingparameter-efficientfinetuningcatastrophicforgettingegocentricdepthperceptionrobotcontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a vision-language model can be adapted to a new visual domain without retraining its vision encoder, by attaching a compact set of learnable visual probes that read from intermediate encoder layers. The authors claim these probes capture domain-specific cues — first-person viewpoint, depth imagery, robot scenes — that the frozen encoder's final output loses, so the model learns the target domain without catastrophic forgetting. The proposal, called VisCoP, is evaluated on three shifts: exocentric-to-egocentric video, RGB-to-depth video, and human-action-understanding-to-robot-control. In each setting VisCoP is reported to beat connector-only, vision-encoder, and full-model finetuning on target benchmarks while retaining more source-domain knowledge. If correct, it offers a parameter-efficient adaptation recipe that avoids the usual trade-off between specialization and forgetting.

What carries the argument

The central mechanism is the visual probe set (16 learnable tokens by default) together with per-layer interaction modules. Each interaction module is a multi-head cross-attention in which the probes query the frozen encoder's intermediate visual features, letting them aggregate cues across all spatio-temporal tokens at multiple abstraction levels. The final probe embeddings go through a dedicated connector and join the standard VLM embeddings as conditioning for the LLM. During adaptation, only the probes, interaction modules, connectors, and LLM low-rank adapters are trained; the vision encoder stays frozen. The probes act as a gradient bypass, steering learning toward domain-relevant stru

What would settle it

Compare the 45,888 ego and 28,176 depth training instruction pairs against the 3,991 Ego-in-Exo PerceptionMCQ evaluation samples; if any overlap exists, rerun adaptation on a non-overlapping split and check whether VisCoP's target gains survive.

Watch

Extended reading notes

Core claim

The paper's central claim is that a small set of learnable visual probes, inserted at every layer of a frozen vision encoder and updated through cross-attention, forms an alternative pathway for learning domain-specific visual representations. At each layer, probes attend to intermediate spatio-temporal features; after the final layer, probe embeddings are projected into the language model and added to the standard visual conditioning. Across cross-view, cross-modal, and cross-task adaptation, VisCoP reportedly yields the largest target-domain improvement among the compared strategies while maintaining or improving source-domain performance, where full finetuning of the vision encoder loses

Load-bearing premise

The ego and depth results assume that the Ego-in-Exo PerceptionMCQ evaluation videos and question-answer pairs are disjoint from the EgoExo4D-derived instruction pairs used for adaptation training, but no train/test split is reported.

Editorial extensions

If this is right

  • VisCoP is reported to improve target-domain performance over other adaptation strategies in all three settings: +3.53 on egocentric video, +19.27 on depth video, and +67.82 on simulated robot control.
  • It retains more source-domain knowledge than vision-encoder finetuning, with source deltas of +1.77 (ego), +1.84 (depth), and -4.58 (robot) versus -8.87 for full finetuning.
  • Layer-wise placement of the probes is load-bearing: using probes only at the final layer, or training only the last four encoder layers, gives much smaller or negative target gains.
  • A moderate probe count (16) is the sweet spot; 128 probes give no additional gain and can hurt, suggesting a compact representation is what matters.
  • In real-world robot manipulation, VisCoP improves success rates over vision-encoder finetuning in both zero-shot transfer from simulation and joint training with real-robot data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the large gap between VisCoP's depth gain (+19.27) and its ego gain (+3.53) suggests the benefit scales with the size of the visual shift; this predicts even larger gains on more extreme modality changes, which could be tested directly.
  • Editorial inference: the mechanism implies the frozen encoder preserves useful domain information in middle layers that the final token aggregation discards; the same probe design may transfer to other shifts such as extreme illumination, sensor artifacts, or synthetic-to-real data without modification.
  • Editorial inference: because the ego and depth evaluations draw on data derived from the same source as the adaptation training and no train/test split is reported, the magnitude of those two gains should be re-measured on disjoint splits before being used as a benchmark comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes VisCoP, a parameter-efficient domain-adaptation method for video VLMs. VisCoP augments a frozen vision encoder with a compact set of learnable visual probes and per-layer interaction modules that cross-attend to intermediate visual features. The resulting probe embeddings are fed to the LLM through a dedicated connector while the LLM is updated with LoRA (or fully, in the robot-control setting). The method is evaluated on three adaptation scenarios: exocentric-to-egocentric video understanding, RGB-to-depth video understanding (depth maps generated by DepthAnythingV2), and human-action-to-robot-control. Performance is reported as Δtarget and Δsource, the accuracy differences between the adapted expert and the base VLM on target and source benchmarks, respectively. The paper claims that VisCoP achieves superior target-domain performance while better retaining source-domain knowledge than alternative component-wise finetuning strategies.

Significance. The conceptual idea of extracting domain-specific features through layer-wise probing of a frozen vision encoder is interesting and potentially useful as a PEFT recipe, and the robot-control experiments on VIMA-Bench and a new real-robot dataset provide a partially external evaluation. However, the paper's headline cross-view and cross-modal claims rest almost entirely on Ego-in-Exo PerceptionMCQ, a benchmark created by the same authors from the same EgoExo4D data used for training, with no disclosed train/test split and with training instruction pairs recaptioned from the same source. This creates a serious risk that the reported target-domain gains are memorization rather than adaptation. The central claim is therefore not currently substantiated, although the issue is fixable by reporting a clean split or by re-evaluating on external benchmarks.

major comments (3)
  1. [§5.2, §5.2.1, Table 1] The ego target evaluation uses Ego-in-Exo PerceptionMCQ, derived from EgoExo4D and comprising QA pairs from [42]. Training uses 24,688 EgoExo4D keystep videos and 74,064 instruction pairs 'recaptioned from the instruction pairs provided in [42]'. No video-ID disjunction or QA-pair deduplication between training and evaluation is reported. Because the training objective is next-token prediction on the same QA format, overlap would allow memorization of answers. The reported Δtarget=+3.53 on this benchmark is therefore not interpretable as evidence of adaptation. Please report the exact disjointness of videos and QA pairs, and provide results on a held-out EgoExo4D split or an external ego-video benchmark.
  2. [§5.2.2, Table 2] The depth target is evaluated only on Ego-in-Exo PerceptionMCQ (Exo Depth), generated by running DepthAnythingV2 on the same EgoExo4D-derived videos used for training, with no split reported. The +19.27 Δtarget could thus reflect memorization of the 3,991-item benchmark rather than cross-modal adaptation. Furthermore, because both training and evaluation depth maps come from the same monocular depth estimator, the setting is a synthetic depth proxy rather than a genuine RGB-to-depth sensor shift. A held-out split or an independent depth benchmark is needed to support the cross-modal claim.
  3. [§5.2.3, Tables 3–4] The robot-control experiments on VIMA-Bench and xArm-Det are more credible and provide some external validation. However, they only cover the cross-task scenario; they do not substantiate the paper's central claims about cross-view and cross-modal adaptation, which rely on the contaminated ego/depth benchmarks. The authors should either provide clean ego/depth evaluations or substantially qualify the central claim.
minor comments (4)
  1. [§4.2, Eq. (3)] The probe update equation writes 'Pℓ = softmax(...)' on the left-hand side, but from the preceding definition Pℓ+1 = Φℓ(Pℓ,Xℓ), the updated probes should be Pℓ+1. Please fix the subscript.
  2. [§4.2] The sentence 'in some settings, such as robotic control, we restrict vision-probe cross-attention to spatial tokens only' is underspecified. Which settings and why? Please clarify for each experiment.
  3. [§5.3, Table 5 and Figures 3–4] Ablation results are reported without variance or multiple seeds. Differences such as QFormer-Style (+0.56) versus VisCoP (+3.53) could be within run-to-run noise. Please report the number of seeds and variance, or at least state the experimental protocol.
  4. [References] Reference [55] lists the authors in the wrong order and incorrectly capitalizes the title. Please check the citation against the original paper.

Circularity Check

2 steps flagged · score 6.0 of 10

Depth (and partly ego) target gains are measured on the authors' own [42]/EgoExo4D benchmark while training on [42]-recaptioned EgoExo4D pairs, with no train/test split reported.

  1. fitted input called prediction [§5.2 Training datasets; §5.2.1–5.2.2 Target benchmarks (Tables 1–2)]
    "We utilize a total of 24,688 videos from the keystep recognition subset to generate 74,064 video instruction pairs. These instructions are recaptioned from the instruction pairs provided in [42]. ... For evaluation on the target domain, we evaluate on the Ego-in-Exo PerceptionMCQ [42] ... Ego-in-Exo PerceptionMCQ is derived from EgoExo4D and comprises 3,991 video question-answer (video-QA) pairs"

    The same source feeds both sides of the reported 'adaptation': the model is fitted on 74,064 instruction pairs recaptioned from [42] (EgoExo4D keystep subset), and the target gains it 'predicts' are computed on Ego-in-Exo PerceptionMCQ, also from [42] and derived from EgoExo4D. No train/test split or video-ID disjunction is reported, so benchmark items may coincide with training videos or QA pairs. Since the training objective is exactly next-token prediction over these QA formats (P(A|E,Q,Z) = ∏ Pθ(a_j|E,Q,Z,A<j), §4.2), the headline Δtarget values of +19.27 (depth) and +3.53 (ego) could be produced by memorization rather than adaptation; the 'prediction' is not demonstrably independent of the fitted input.

  2. self citation load bearing [§5.2 Training datasets; §5.2.1–5.2.2 evaluation; reference [42]]
    "These instructions are recaptioned from the instruction pairs provided in [42]. ... [42] Dominick Reilly, Manish Kumar Govind, Le Xue, and Srijan Das. From my view to yours: Ego-augmented learning in large vision language models for understanding exocentric daily living activities, 2025."

    The sole target benchmark for the cross-modal depth claim (Δtarget = +19.27, Table 2) and the primary driver of the ego gain is the authors' own prior work [42], whose same four authors also wrote the present paper; the training instructions for both ego and depth are likewise 'recaptioned from the instruction pairs provided in [42]'. The paper presents Ego-in-Exo PerceptionMCQ as an independent testbed, but its provenance is a same-author, same-dataset chain, and no external depth benchmark is used. The load-bearing evidence for 'superior target-domain performance' in the cross-view/cross-modal scenarios therefore reduces to a self-citation loop rather than independent validation.

full rationale

The paper is not wholly circular: the architecture contribution is well-specified (frozen encoder + 16 learnable probes with per-layer cross-attention + LoRA), the ablations (Table 5, Figures 3–4) are internally controlled, and two target/benchmark branches rest on external evidence — EgoSchema (Mangalam et al., Ego4D) shows a +7.9 ego gain, and VIMA-Bench (Jiang et al.) supports the robot-control claim. The relative comparison among adaptation strategies is also run under the same evaluation, so VisCoP's margin over baselines is not trivially forced. However, the central 'superior target-domain performance' claim leans heavily on Ego-in-Exo PerceptionMCQ from [42], which is the authors' own prior work: the same [42] supplies the training instructions (recaptioned) and the evaluation QA pairs, both grounded in the same EgoExo4D keystep source, and no train/test split is reported anywhere (§5.2, §5.2.1, §5.2.2). The depth result in particular — the largest claimed gain (+19.27) and the paper's evidence for cross-modal adaptation — is measured entirely on this self-derived benchmark, so its validity reduces to a same-source, same-author loop unless a clean split is demonstrated. That is a partial but substantive circularity: one or more core 'predictions' are potentially forced by construction, while other claims retain independent content. Score 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 2 invented entities

The paper relies on standard pretrained components (SigLIP, Qwen, VideoLLaMA3) and adds new trainable components (probes, interaction modules). The main free parameters are the number of probes, LoRA rank, learning rates, and layer placement, none of which are derived from theory. The deepest assumption is that intermediate frozen features are a useful source of domain-specific signal.

free parameters (5)
  • M (number of visual probes) = 16
    Chosen based on the ablation in Figure 3 to optimize target/source trade-off.
  • LoRA rank r = 16
    Standard choice, no sensitivity analysis reported.
  • Learning rates = 1e-5 (LLM/connector), 2e-6 (vision encoder)
    Set by convention, no sweep reported.
  • Probe initialization std = 0.02
    Random normal initialization, no sensitivity analysis.
  • Interaction module placement = every layer
    Ablation in Figure 4 shows every-layer is best; still a design choice.
assumptions (4)
  • domain assumption Intermediate layers of a frozen vision transformer contain domain-specific cues that can be extracted by cross-attention probes.
    Central hypothesis of VisCoP; not formally proven, only empirically assessed.
  • standard math Autoregressive next-token prediction on target video-QA pairs is a valid objective for domain adaptation.
    Standard training objective used by all baselines; no evidence that it is suboptimal.
  • domain assumption DepthAnythingV2 depth maps are a faithful proxy for the true depth modality.
    Used to synthesize the depth target domain without verifying against real depth sensors.
  • domain assumption The selected benchmarks accurately measure source and target capabilities.
    No analysis of benchmark noise or reliability; contamination risk is present.
invented entities (2)
  • Visual probes (learnable tokens P) independent evidence
    purpose: Extract domain-specific visual features from intermediate layers of a frozen vision encoder.
    Concrete component tested in ablations and on external tasks like VIMA and EgoSchema; the paper provides a falsifiable handle (the method can be run on held-out benchmarks).
  • Interaction modules Φℓ independent evidence
    purpose: Cross-attention between probes and visual features at each layer, enabling multi-level aggregation.
    Compared against Q-Former-style and other variants in Table 5; a concrete architectural choice that can be independently re-implemented.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models." pith.science (2026). https://pith.science/paper/F54ABT46

@misc{pith2026251013808,
  author       = {Pith},
  title        = {Pith review of: VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F54ABT46}},
  note         = {Machine review of arXiv:2510.13808}
}
read the original abstract

Large Vision Language Models (VLMs) excel at general visual reasoning but experience significant performance degradation when deployed in novel domains that exhibit substantial distribution shifts from their pretraining data. Existing domain adaptation methods rely on finetuning standard VLM components; however, depending on which components are updated, these approaches either limit the model's ability to learn domain-specific representations or cause catastrophic forgetting of previously acquired capabilities. We introduce Vision Contextualized Probing (VisCoP), a parameter-efficient adaptation framework that augments the VLM vision encoder with a compact set of learnable visual probes. By learning domain-specific visual representations through these probes while requiring only minimal updates to pretrained model components, VisCoP effectively adapts to new domains without sacrificing existing knowledge. We evaluate VisCoP across three challenging adaptation settings: cross-view (exocentric to egocentric), cross-modal (RGB to depth), and cross-task (human understanding to robot control). Across all scenarios, VisCoP consistently outperforms existing domain adaptation strategies, achieving superior target-domain performance while preserving the pretrained VLM's capabilities on the source domain. These results demonstrate that lightweight visual probing provides an effective and robust solution for adapting VLMs under substantial distribution shifts. Code, models, and evaluation protocols are available at https://github.com/dominickrei/VisCoP.

Figures

Figures reproduced from arXiv: 2510.13808 by the authors.

Figure 1
Figure 1. Domain adaptation performance of different adaptation strategies. VISCOP achieves superior target domain performance while better retaining source domain knowl￾edge compared to other strategies. To this end, we introduce Vision Contextualized Probing, dubbed VISCOP, a mechanism that en￾ables adaptation of pretrained VLMs to a novel tar￾get domain, while retaining its general-purpose vi￾sual representations learned d… view at source ↗
Figure 2
Figure 2. Architecture of our proposed VISCOP. Learnable visual probes are conditioned on intermediate representations of a frozen vision encoder through vision-probe cross-attention, which extracts domain-specific features that may have otherwise been discarded by the frozen encoder. where N is the number of spatial patch tokens per frame and dv is the embedding dimension of the vision encoder. Concatenating these tokens ove… view at source ↗
Figure 4
Figure 4. Ablation on the positions of interaction modules in VISCOP [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Analysis of VISCOP. (a) Attentions between visual features and visual probes. (b) Atten￾tion of generated language tokens to visual embeddings. (c) t-SNE of visual and probe embeddings. Ellipses denote 95% confidence regions of a fitted 2D Gaussian, and cross markers i…
Figure 6
Figure 6. Figure 6: Examples from VIMA and VIMA-Bench. The first two rows show training examples, including the initial observations, final states, and task instructions. The bottom row illustrates the evaluation in VIMA-Bench, covering three levels of generalization. A.2 Details of Real-…
Figure 7
Figure 7. Figure 7: Real robot setup. Our setup uses an xArm7 robot arm and Intel RealSense D455 camera. We provide additional details of the experiments conducted in our novel robot environment, including the setup, data collection, and evaluation protocol. A.2.1 Real-Robot Setup Our set…
Figure 8
Figure 8. Figure 8: Visualization of the three real-world tasks. Each column shows the initial state (top) and the corresponding final state (bottom), along with the robot execution (from left to right): T1 (place the corn on the plate), T2 (rotate the cucumber by 90◦ ), and T3 (move all …
Figure 9
Figure 9. Figure 9: Qualitative results on Egocentric Video Understanding Experts. Q: Which object does the person fill with water ? A. Pitcher B. Measuring cup C. Bowl D. Bottle Base VLM : A VL-C+VE : B VISCOP (Ours): B Q: Which object does the person pour oil from into skillet ? A. Wash…
Figure 10
Figure 10. Figure 10: Qualitative results on Depth Video Understanding Experts. compared to the trained vision encoder (VL-C+VE). While VL-C+VE can adapt to the target domain, on the source domain it often introduces hallucinated details. In contrast, VISCOP preserves correctness, capturin…
Figure 11
Figure 11. Figure 11: Qualitative results on robot control Experts. Q: Describe the video in detail, keep it within 100 words, Include all the actions and object interactions of the person in the scene. Base VLM: The video shows a woman in a tie-dye shirt and black pants standing in front …
Figure 12
Figure 12. Figure 12: ADL-X descriptions using Ego Video Understanding Expert [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]
Figure 13
Figure 13. Figure 13: ADL-X captions from the Depth Video Understanding Expert [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 3 linked inside Pith

  1. [42]

    From my view to yours: Ego-augmented learning in large vision language models for understanding exocentric daily living activities, 2025

    Dominick Reilly, Manish Kumar Govind, Le Xue, and Srijan Das. From my view to yours: Ego-augmented learning in large vision language models for understanding exocentric daily living activities, 2025

    2025

  2. [1]

    Thinking with images, April 2025

    OpenAI. Thinking with images, April 2025. Accessed: October 16, 2025

    2025

  3. [2]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    arXiv 2025

  4. [3]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    arXiv 2025

  5. [4]

    xgen-mm (blip-3): A family of open large multimodal models, 2025

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Shaoyen Tseng, Gustavo A Lujan-Moreno, Matthew L Olson, Musashi Hinck, David Cobbley, Vasudev Lal, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke,...

    2025

  6. [5]

    Socratic models: Composing zero-shot multimodal rea- soning with language

    Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal rea- soning with language. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  7. [6]

    Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, 2024

    Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, 2024

    2024

  8. [7]

    Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692, 2023

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692, 2023

    arXiv 2023

Show all 55 references
  1. [8]

    Ryoo, and Tsung- Yu Lin

    Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S. Ryoo, and Tsung- Yu Lin. Learning to localize objects improves spatial reasoning in visual-llms.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12977–12987, 2024

    2024

  2. [9]

    Qwen2.5 technical report, 2025

    Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang...

    2025

  3. [10]

    The llama 3 herd of models, 2024

    Meta. The llama 3 herd of models, 2024

    2024

  4. [11]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  5. [12]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2023

    2023

  6. [13]

    Video instruction tuning with synthetic data, 2024

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024

    2024

  7. [14]

    Sharegpt4video: Improving video understanding and generation with better captions

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions. In Advances in Neura...

    2024

  8. [15]

    Videogpt+: Integrating image and video encoders for enhanced video understanding, 2024

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding, 2024

    2024

  9. [16]

    Cinepile: A long video question answering dataset and benchmark, 2024

    Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark, 2024

    2024

  10. [17]

    Is space-time attention all you need for video understanding? InProceedings of the International Conference on Machine Learning (ICML), July 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? InProceedings of the International Conference on Machine Learning (ICML), July 2021

    2021

  11. [18]

    Vivit: A video vision transformer

    Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lu ˇci´c, and Cordelia Schmid. Vivit: A video vision transformer. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6836–6846, October 2021

    2021

  12. [19]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. InAdvances in Neural Information Processing Systems, 2023

    2023

  13. [20]

    Aim: Adapting image models for efficient video understanding

    Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. Aim: Adapting image models for efficient video understanding. InInternational Conference on Learning Representations, 2023

    2023

  14. [21]

    Overcoming the pitfalls of vision- language model finetuning for ood generalization

    Yuhang Zang, Hanlin Goh, Josh Susskind, and Chen Huang. Overcoming the pitfalls of vision- language model finetuning for ood generalization. InInternational Conference on Learning Representations, 2024

    2024

  15. [22]

    Vision- language model fine-tuning via simple parameter-efficient modification

    Ming Li, Jike Zhong, Chenxin Li, Liuzhuozheng Li, Nie Lin, and Masashi Sugiyama. Vision- language model fine-tuning via simple parameter-efficient modification. InConference on Empirical Methods in Natural Language Processing, 2024

    2024

  16. [23]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Informa- tion Processing Systems, 2017

    2017

  17. [24]

    Video swin transformer.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3192–3201, 2021

    Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer.2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3192–3201, 2021

    2022

  18. [25]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning, 2023

    2023

  19. [26]

    Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bi´nkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Se...

    2022

  20. [27]

    Fusion of domain-adapted vision and language models for medical visual question answering

    Cuong Nhat Ha, Shima Asaadi, Sanjeev Kumar Karn, Oladimeji Farri, Tobias Heimann, and Thomas Runkler. Fusion of domain-adapted vision and language models for medical visual question answering. InProceedings of the Clinical Natural Language Processing Workshop at the 2024 Confe...

    2024

  21. [28]

    Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles

    Michael S. Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Jongwoo Park, Kanchana Ranasinghe, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen-mm-vid (blip-3-video): You only need 32 tokens to represent a video even in vlms, 2025. 11

    2025

  22. [29]

    Finetuned clip models are efficient video learners

    Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Finetuned clip models are efficient video learners. InThe IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  23. [30]

    Conditional prompt learning for vision-language models

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022

    2022

  24. [31]

    Prompt-aligned gradient for prompt tuning

    Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision, 2023

    2023

  25. [32]

    Visual-language prompt tuning with knowledge- guided context optimization

    Hantao Yao, Rui Zhang, and Changsheng Xu. Visual-language prompt tuning with knowledge- guided context optimization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023

    2023

  26. [33]

    Clip-adapter: Better vision-language models with feature adapters

    Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. In International Journal of Computer Vision, 2023

    2023

  27. [34]

    On domain-adaptive post-training for multimodal large language models

    Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Wayne Xin Zhao, Zhongzhi Luan, Bo Dai, and Zhenliang Zhang. On domain-adaptive post-training for multimodal large language models. InConference on Empirical Methods in Natural Language Processing Findings, 2025

    2025

  28. [35]

    Fnu Mohbat and Mohammed J. Zaki. Llava-chef: A multi-modal generative model for food recipes. InACM International Conference on Information and Knowledge Management, 2024

    2024

  29. [36]

    Llavidal: A large language-vision model for daily activities of living

    Dominick Reilly, Rajatsubhra Chakraborty, Arkaprava Sinha, Manish Kumar Govind, Pu Wang, Francois Bremond, Le Xue, and Srijan Das. Llavidal: A large language-vision model for daily activities of living. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2025

  30. [37]

    Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024

    Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Ruifei Zhang, Zhenyang Cai, Ke Ji, Guangjun Yu, Xiang Wan, and Benyou Wang. Huatuogpt-vision, towards injecting medical visual knowledge into multimodal llms at scale, 2024

    2024

  31. [38]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023

    2023

  32. [39]

    Apollo: An exploration of video understanding in large multimodal models

    Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, Serena Yeung-Levy, and Xide Xia. Apollo: An exploration of video understanding in large multimodal models. InProceedings of the IEEE/...

    2025

  33. [40]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  34. [41]

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Tri- antafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zach Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, Maria E...

    2025

  35. [43]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. InAdvances in Neural Information Processing Systems, 2024

    2024

  36. [44]

    Vima: General robot manipulation with multimodal prompts

    Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. InInternational Conference on Machine Learning, 2023

    2023

  37. [45]

    Xiang Li, Cristina Mata, Jongwoo Park, Kumara Kahatapitiya, Yoo Sung Jang, Jinghuan Shang, Kanchana Ranasinghe, Ryan Burgert, Mu Cai, Yong Jae Lee, and Michael S. Ryoo. Llara: Supercharging robot learning data for vision-language policy. InInternational Conference on Learning ...

    2025

  38. [46]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. InProceedings of the Thirty- seventh Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, 2023

    2023

  39. [47]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Ro- hit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Meng...

    2022

  40. [48]

    Next-qa: Next phase of question- answering to explaining temporal actions

    Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9777–9786, 2021

    2021

  41. [49]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal large language models in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Ke Li, Tong Xu, Xiawu Zheng, Enhong Chen, Caifeng Shan, Ran He, and Xing Sun. Video-mme: The first-ev...

    2025

  42. [50]

    Toyota smarthome: Real-world activities of daily living

    Srijan Das, Rui Dai, Michal Koperski, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome: Real-world activities of daily living. InPro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 833–842, 2019. 13

    2019

  43. [51]

    Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta

    Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Proceedings of the European Conference on Computer Vision, pages 510–526, 2016

    2016

  44. [52]

    Lemma: A multi- view dataset for learning multi-agent multi-task activities

    Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song-Chun Zhu. Lemma: A multi- view dataset for learning multi-agent multi-task activities. InProceedings of the European Conference on Computer Vision, pages 767–783, 2020

    2020

  45. [53]

    Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    Rui Dai, Srijan Das, Saurav Sharma, Luca Minciullo, Lorenzo Garattoni, Francois Bremond, and Gianpiero Francesca. Toyota smarthome untrimmed: Real-world untrimmed videos for activity detection.IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

    2022

  46. [54]

    Quantifying attention flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. InAnnual Meeting of the Association for Computational Linguistics, 2020

    2020

  47. [55]

    Place the carrot on the plate

    Neil Houlsby Matthias Minderer, Alexey Gritsenko. Scaling open-vocabulary object detection. NeurIPS, 2023. 14 A Appendix A.1 Details of Simulated Robot Control Experiments For our robot control simulation experiments, we use the VIMA-8K instruction set generated from the VIMA ...

    2023

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.