REVIEW 7 cited by
BT-Adapter: Video Conversation is Feasible Without Video Instruction Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The recent progress in Large Language Models (LLM) has spurred various advancements in image-language conversation agents, while how to build a proficient video-based dialogue system is still under exploration. Considering the extensive scale of LLM and visual backbone, minimal GPU memory is left for facilitating effective temporal modeling, which is crucial for comprehending and providing feedback on videos. To this end, we propose Branching Temporal Adapter (BT-Adapter), a novel method for extending image-language pretrained models into the video domain. Specifically, BT-Adapter serves as a plug-and-use temporal modeling branch alongside the pretrained visual encoder, which is tuned while keeping the backbone frozen. Just pretrained once, BT-Adapter can be seamlessly integrated into all image conversation models using this version of CLIP, enabling video conversations without the need for video instructions. Besides, we develop a unique asymmetric token masking strategy inside the branch with tailor-made training tasks for BT-Adapter, facilitating faster convergence and better results. Thanks to BT-Adapter, we are able to empower existing multimodal dialogue models with strong video understanding capabilities without incurring excessive GPU costs. Without bells and whistles, BT-Adapter achieves (1) state-of-the-art zero-shot results on various video tasks using thousands of fewer GPU hours. (2) better performance than current video chatbots without any video instruction tuning. (3) state-of-the-art results of video chatting using video instruction tuning, outperforming previous SOTAs by a large margin.
Forward citations
Cited by 7 Pith papers
-
Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedback
Oracle-RLAIF uses an oracle ranker and a GRPO variant with nDCG rank penalties to fine-tune video language models, reporting improved video QA accuracy over VLM-RLAIF.
-
Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
Period-LLM improves multimodal LLM performance on periodic tasks such as repetition counting and heart-rate estimation via easy-to-hard curriculum training and a channel-gradient weighting strategy.
-
Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness
The paper presents FDA, a manually annotated dataset, FaceTrack-MM, a face-tracking video MLLM, FEC-Bench, a benchmark, and TEM, a ChatGPT-based metric, all for dynamic facial expression captioning.
-
ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
ATP-LLaVA learns per-layer and per-instance visual token pruning thresholds, cutting average token counts by 75% with about 1.9% average benchmark degradation on LLaVA-1.5.
-
Generative Timelines for Instructed Visual Assembly
A fine-tuned multimodal large language model that represents visual collections and timelines as token sequences can execute natural language timeline editing instructions more accurately than GPT-4o on synthetic benchmarks.
-
DynFocus: Dynamic Cooperative Network Empowers LLMs with Video Understanding
DynFocus dynamically allocates a few tokens to selected frames and two tokens to the rest, reporting competitive video QA accuracy with lower token budgets.
-
Visual Large Language Models for Generalized and Specialized Applications
This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.
Discussion (0). Continue with ORCID to comment.