Pith. sign in

REVIEW 11 cited by

LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2311.17043 v1 pith:UP3LW4KD submitted 2023-11-28 cs.CV cs.CL

Yanwei Li , Chengyao Wang , Jiaya Jia This is my paper
classification cs.CVcs.CL
keywords llama-vidtokencontextimagetokensvideosvisualcontent
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this work, we present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. Current VLMs, while proficient in tasks like image captioning and visual question answering, face computational burdens when processing long videos due to the excessive visual tokens. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. The context token encodes the overall image context based on user input, whereas the content token encapsulates visual cues in each frame. This dual-token strategy significantly reduces the overload of long videos while preserving critical information. Generally, LLaMA-VID empowers existing frameworks to support hour-long videos and pushes their upper limit with an extra context token. It is proved to surpass previous methods on most of video- or image-based benchmarks. Code is available https://github.com/dvlab-research/LLaMA-VID}{https://github.com/dvlab-research/LLaMA-VID

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MentalThink: Shaping Thoughts in Mental SVG World

    cs.AI 2026-07 conditional novelty 7.0 of 10

    MLLMs that generate and render SVG sketches as multi-turn intermediate reasoning steps reach 55.1% on VSIBench and 76.0% on MindCube, far above the Qwen2.5-VL-7B backbone.

  2. QSVideo: Query-Conditioned Semantic Temporal Retrieval for Video Understanding

    cs.CV 2026-07 accept novelty 6.0 of 10

    QSVideo reformulates questions into structured queries, ranks frames by object-action-location relevance plus diversity, and applies temporal strategies to boost VLM accuracy under tight frame budgets.

  3. Latent Visual Cache for Video Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A recurrent latent visual cache in the decoder, trained with contrastive key-frame alignment and a vision-grounded GRPO reward, improves video reasoning accuracy while shortening answers.

  4. Video Reasoning without Training

    cs.CV 2025-10 conditional novelty 6.0 of 10

    An entropy-guided, inference-time value-cache controller improves video reasoning accuracy and cuts output tokens versus RL-trained baselines.

  5. SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding

    cs.CV 2025-08 conditional novelty 6.0 of 10

    A surgical-video multimodal LLM with instrument-centric masked pretraining and interleaved temporal embeddings beats general video LLMs on a new CholecT50-derived benchmark.

  6. AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.

  7. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

  8. Task-Aware KV Compression For Cost-Effective Long Video Understanding

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-X2L uses bi-level KV compression with task-aware selective reloading to improve long-video QA accuracy and reduce decode-time memory versus uniform KV compression.

  9. Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Video-XL-2 cuts long-video inference cost with chunked pre-filling and query-gated dense-or-sparse KV reloading, reporting half the FLOPs and a third less decoding memory at roughly equal benchmark scores.

  10. Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

    cs.AI 2025-06 conditional novelty 5.0 of 10

    Stream-Omni uses CTC-based layer-dimension mapping to align speech with text, achieving vision, speech, and text interaction in one 8B model trained on 23,000 hours of speech.

  11. SV3.3B: A Sports Video Understanding Model for Action Recognition

    cs.CV 2025-07 reject novelty 4.0 of 10

    A fine-tuned 3.3B video description model using DWT-VGG16-LDA keyframe sampling reports 29.2% higher validation scores than GPT-4o on a 1,315-clip NBA play-by-play subset.

Pith tools