Pith. sign in

REVIEW 25 cited by

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.17043 v1 pith:UP3LW4KD submitted 2023-11-28 cs.CV cs.CL

Yanwei Li , Chengyao Wang , Jiaya Jia This is my paper
classification cs.CVcs.CL
keywords llama-vidtokencontextimagetokensvideosvisualcontent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID

Discussion (0). Sign in to comment.

Forward citations

Cited by 25 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  2. AffectVerse: Emotional World Models for Multimodal Affective Computing

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    AffectVerse improves multimodal emotion recognition by at least 2.57% on nine benchmarks through an Emotion World Module that performs short-horizon latent affective prediction via cross-modal temporal imagination and...

  3. Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    SceneBench shows VLMs lose accuracy on scene-level questions in long videos due to forgetting, and Scene-RAG retrieval improves performance by 2.5%.

  4. LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models

    cs.CV 2024-07 unverdicted novelty 7.0 of 10

    LLaVA-NeXT-Interleave unifies multi-image, video, and 3D capabilities in large multimodal models via a new 1.18M-sample interleaved dataset and benchmark, achieving leading results across those tasks while preserving ...

  5. LVBench: An Extreme Long Video Understanding Benchmark

    cs.CV 2024-06 accept novelty 7.0 of 10

    LVBench is a new benchmark for extreme long video understanding that evaluates multimodal large language models on hour-scale videos using tasks designed to probe extended memory and comprehension.

  6. MLVU: Benchmarking Multi-task Long Video Understanding

    cs.CV 2024-06 conditional novelty 7.0 of 10

    MLVU is a new benchmark for long video understanding that uses extended videos across diverse genres and multi-task evaluations, revealing that current MLLMs struggle significantly and degrade sharply with longer durations.

  7. QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding

    cs.CV 2026-07 accept novelty 6.0 of 10

    QSVideo reformulates questions into structured queries, ranks frames by object-action-location relevance plus diversity, and applies temporal strategies to boost VLM accuracy under tight frame budgets.

  8. Latent Visual Cache for Video Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A recurrent latent visual cache in the decoder, trained with contrastive key-frame alignment and a vision-grounded GRPO reward, improves video reasoning accuracy while shortening answers.

  9. TRACE: Evidence Grounding-Guided Multi-Video Event Understanding and Claim Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    TRACE improves multi-video event understanding by grounding evidence in structured timelines before visual reasoning, raising MiRAGE F1 from 0.705 to 0.811 on MAGMaR 2026.

  10. TRACE: Evidence Grounding-Guided Multi-Video Event Understanding and Claim Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    TRACE builds structured text timelines from videos via OCR and detection, then applies text-only LLM evidence localization before LVLM claim generation, raising MiRAGE F1 from 0.705 to 0.811 on MAGMaR.

  11. Video Reasoning without Training

    cs.CV 2025-10 conditional novelty 6.0 of 10

    An entropy-guided, inference-time value-cache controller improves video reasoning accuracy and cuts output tokens versus RL-trained baselines.

  12. SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A surgical-video multimodal LLM with instrument-centric masked pretraining and interleaved temporal embeddings beats general video LLMs on a new CholecT50-derived benchmark.

  13. ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs

    cs.CV 2025-07 unverdicted novelty 6.0 of 10

    ReGATE introduces a teacher-student adaptive token elision method that reduces training tokens to 38% while matching or exceeding baseline accuracy on multimodal benchmarks.

  14. Uni-NaVid: A Video-based Vision-Language-Action Model for Unifying Embodied Navigation Tasks

    cs.RO 2024-12 unverdicted novelty 6.0 of 10

    Uni-NaVid unifies diverse embodied navigation tasks into one video-based vision-language-action model trained on 3.6 million samples from four sub-tasks, achieving state-of-the-art performance on benchmarks and real-w...

  15. LLaVA-Video: Video Instruction Tuning With Synthetic Data

    cs.CV 2024-10 unverdicted novelty 6.0 of 10

    LLaVA-Video-178K is a new synthetic video instruction dataset that, when combined with existing data to train LLaVA-Video, produces strong results on video understanding benchmarks.

  16. LongVILA: Scaling Long-Context Visual Language Models for Long Videos

    cs.CV 2024-08 unverdicted novelty 6.0 of 10

    LongVILA scales visual-language models from 8 to 2048 video frames with 99.8% needle-in-a-haystack accuracy using long-context extension, supervised fine-tuning, and multi-modal sequence parallelism on up to 256 GPUs.

  17. What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction

    cs.CV 2024-07 unverdicted novelty 6.0 of 10

    Introduces the QEVD benchmark for asynchronous situated interaction in fitness coaching and proposes a streaming baseline to address limitations of existing vision-language models.

  18. TempCompass: Do Video LLMs Really Understand Videos?

    cs.CV 2024-03 unverdicted novelty 6.0 of 10

    TempCompass benchmark reveals that state-of-the-art Video LLMs have poor ability to perceive temporal aspects such as speed, direction, and ordering in videos.

  19. NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

    cs.CV 2024-02 unverdicted novelty 6.0 of 10

    NaVid, a video-based VLM trained on 510k navigation and 763k web samples, achieves SOTA VLN performance using only monocular RGB video for next-step action planning in sim and real environments.

  20. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

  21. Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

    cs.CV 2026-03 unverdicted novelty 5.0 of 10

    SceneBench shows VLMs sharply lose accuracy on scene-level long-video questions, and Scene-RAG only partially mitigates that forgetting (+2.50%).

  22. InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

    cs.CV 2024-07 conditional novelty 5.0 of 10

    InternLM-XComposer-2.5 is a 7B vision-language model supporting up to 96K context that reaches GPT-4V-level performance on image, video, and multi-turn tasks and adds LoRA-driven text-image composition capabilities.

  23. PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioning

    cs.CV 2024-04 conditional novelty 5.0 of 10

    A temporal pooling layer added to LLaVA smooths video feature distributions and lifts performance on dense video captioning and QA to new SOTA levels without extra parameters.

  24. Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

    cs.CV 2024-03 unverdicted novelty 5.0 of 10

    Mini-Gemini enhances VLMs via high-resolution visual refinement, curated reasoning data, and self-guided generation to reach leading zero-shot benchmark results across 2B-34B LLMs.

  25. SV3.3B: A Sports Video Understanding Model for Action Recognition

    cs.CV 2025-07 reject novelty 4.0 of 10

    A fine-tuned 3.3B video description model using DWT-VGG16-LDA keyframe sampling reports 29.2% higher validation scores than GPT-4o on a 1,315-clip NBA play-by-play subset.

Pith tools