Pith. sign in

REVIEW 2 major objections 6 minor 51 references

One learnable token, scored against value vectors, lets frozen vision-language models pick the few frames that answer the question.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 01:40 UTC pith:IXX6HMK5

load-bearing objection A small, honest systems win: one learned token scored in value space beats attention retrieval and transfers image→long-video, with the main limit already quantified by the authors. the 2 major comments →

arxiv 2607.28627 v1 pith:IXX6HMK5 submitted 2026-07-30 cs.CV cs.AIcs.LG

ReToken: One Token to Improve Vision-Language Models for Visual Retrieval

Yao Xiao , Reuben Tan , Zhen Zhu , Yuqun Wu , Jianfeng Gao , Derek Hoiem This is my paper
classification cs.CV cs.AIcs.LG
keywords vision-language modelsvisual retrievalvalue-space scoringKV cachelong video understandinglearnable retrieval tokenmulti-image QA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language models struggle when the visual input is long and most of it is irrelevant: accuracy falls as distractors pile up, and stuffing every frame into the model at once can exhaust GPU memory. This paper argues that the usual fix—reading the model’s own attention scores—is a weak retrieval signal in VLMs, while the value projections, which carry the content that actually flows through attention, are much more informative. It introduces ReToken, a single learnable embedding appended to the question and trained as an explicit retrieval target against each frame’s mean final-layer value vector. With the backbone frozen and training limited to a small multi-image QA set, the method raises accuracy on image haystacks by more than twenty percent relative and transfers zero-shot to hour-scale video, all while training and inference fit on one H100.

Core claim

In pretrained vision-language models, retrieval scores computed in the value space are substantially more reliable than conventional query–key attention for selecting query-relevant visual frames. A single learnable token, supervised only with a class-balanced retrieval loss against final-layer mean value vectors, is enough to turn that signal into a practical sparse retriever over a pre-filled visual KV cache, and the same token trained only on multi-image QA transfers zero-shot to long video.

What carries the argument

ReToken: one learnable embedding plus a single projection matrix. At the final layer it scores each frame by cosine similarity to that frame’s mean value vector; top-K frames are then broadcast so answer generation attends only to their cached KV states.

Load-bearing premise

Relevance can be recovered by matching a query to per-frame average content, so evidence that only appears across several frames or is spread over the whole video is not the main case the method must solve.

What would settle it

On a long-video benchmark where answers require dispersed or cross-frame evidence (for example summarization or ‘what happened before X’), measure whether ReToken’s top-K frames still raise accuracy over uniform sampling; a consistent drop or no gain would falsify the claim that value-space frame means are a sufficient retrieval representation.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Sparse retrieval over a once-encoded visual KV cache becomes a practical default for multi-image and long-video QA on a single GPU.
  • Value-space scoring, not attention weights, becomes the preferred internal retrieval signal for frozen VLMs.
  • A token trained only on multi-image relevance labels can improve hour-scale video understanding without video fine-tuning.
  • When many questions share one video, encoding cost is paid once and each question only reloads a small retrieved subset.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If value features are the better retrieval space, other internal-memory methods (block selection, streaming caches) may gain more from value matching than from query–key attention.
  • Tasks that need temporal displacement or set-level evidence will likely need the paper’s own suggested extensions—token-level scores or consecutive-frame sets—rather than independent frame means.
  • Cleaning the stored KV cache via light early-layer tuning helps image haystacks but can hurt video transfer, suggesting a real trade-off between distractor isolation and cross-frame binding.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes ReToken, a single learnable embedding X_r plus a final-layer projection W_r that is appended to the question and trained as an explicit retrieval target. Retrieval scores are cosine similarities between the projected ReToken and per-frame mean value vectors at the last transformer layer (Eq. 3), supervised by class-balanced BCE against ground-truth frame relevance (Eq. 4). The VLM is frozen by default; training uses a filtered multi-image MIRAGE subset. At inference a two-pass retrieve-then-answer pipeline selects top-K frames from a pre-filled visual KV cache. Empirically, with K=1 on Visual Haystacks the method improves frozen Qwen3VL-8B by up to +13.4 points (C=50) and InternVL3.5-8B by +12.4, beats SigLIP2/ReKV/CoT/MIRAGE baselines (Tables 2, 4), and transfers zero-shot from image-only training to long video (+8.0 on LVBench; Tables 5–6). Controlled ablations support value-over-key scoring (Tables 1, 7; Fig. 6), two-pass inference (Table 9), and quantify when the method helps or hurts by question type (Table 11).

Significance. If the results hold, this is a practical and lightweight contribution to long-context VLM retrieval: one token and one projection, single-H100 training and inference, public code, and large gains under tight retrieval budgets plus genuine zero-shot image-to-video transfer. The diagnostic that final-layer value means are a stronger text-aligned retrieval space than query–key attention in pretrained VLMs is useful beyond the specific method and is backed by controlled comparisons. The work cleanly targets the retriever bottleneck in retrieval-based long-video pipelines rather than proposing another external dual encoder. Scope is appropriately limited by the authors’ own error analysis (localized/nameable evidence vs. dispersed summarization) and Limitations section. Strengths include reproducible code, multi-backbone evidence, runtime/memory breakdown (Table 10), and explicit self-reporting of failure modes (Table 11).

major comments (2)
  1. [§3.2 Inference; Fig. 5; Tables 2, 5] §3.2 Inference and the paragraph noting the K1 vs K asymmetry: ReToken’s retrieval pass may attend to K1=256 frames per early layer before final-layer ranking, while answer generation and several baselines (e.g., single-pass ReKV, Uniformly) operate under the much smaller answer budget K. Part of the reported gap vs ReKV (Fig. 5, Tables 2, 5) may therefore come from a larger ranking context rather than solely from the learned value-space token. Please add a controlled comparison that equalizes the early-layer ranking budget (or reports ReKV with an analogous multi-frame ranking pass) so the contribution of X_r / value scoring is isolated from the K1 budget.
  2. [Supp. B.1; Table 12; §4.1 Training Datasets] Supp. B.1 and Table 12: the training set is filtered using generation loss and question–target attention scores from frozen Qwen3VL-8B. That filter is reasonable for removing memorized pairs, but it selects examples on which Qwen’s own attention already prefers the target, which can inflate Qwen-centric gains and make the recipe less backbone-agnostic. Clarify whether InternVL3.5 was trained on the identical Qwen-filtered set or re-filtered with InternVL signals, and report at least one unfiltered or cross-filtered ablation on both backbones (beyond the single Qwen row in Table 12) so the central multi-backbone claim does not rest on a Qwen-specific curriculum.
minor comments (6)
  1. [§4.2–4.3; Tables 4–6] Report variance or multiple seeds for the main VHs and LVBench numbers; all headline deltas appear to be single-run.
  2. [Figure 1] Fig. 1a caption and axis labels are hard to read in grayscale; consider a clearer distinction between GT window and attention curve.
  3. [§3.1] Eq. (1)–(2) use bar-q as mean over question tokens; later ReToken replaces this. A short forward pointer in §3.1 to Eq. (3) would help readers track the design change.
  4. [Table 4; §4.1] Table 4 marks InternVL C=100 gain as only +4.4 and attributes it to 1-epoch training; state the epoch budget in the table caption or a footnote so the asymmetry with Qwen (3 epochs) is visible without turning to §4.1.
  5. [Abstract; §3.1] Typos / notation: “RETOKENyields” missing space in abstract; inconsistent RETOKEN vs ReToken capitalization; “rIvrSplq Kss” formatting in §3.1 is difficult to parse.
  6. [§2; §5 Future Work] Related Work could briefly contrast token-level vs frame-level retrieval given Future Work’s token-level suggestion; optional.

Circularity Check

0 steps flagged

No significant circularity: empirical retrieval method supervised on external relevance labels and measured on held-out QA benchmarks.

full rationale

ReToken is an empirical systems paper, not a first-principles derivation. The load-bearing chain is: (i) diagnose weak query–key attention for visual retrieval (Fig. 1, Tab. 1); (ii) train one learnable token plus a projection with class-balanced BCE against ground-truth frame/image relevance labels from MIRAGE-style annotations (Eqs. 3–4); (iii) report QA accuracy and recall on held-out benchmarks (Visual Haystacks, QAEgo4DTest-MC, LVBench, Video-MME). None of these evaluation metrics is defined to equal the training objective, and gains are not forced by construction from fitted inputs. Value-space scoring is motivated partly by overlapping-author TextRegion and by segmentation literature, but is independently ablated in-paper (Tab. 1, Tab. 7, Fig. 6). Training-data filters that use the frozen backbone’s loss/attention (Supp. B.1) are ordinary curriculum choices on the train split only; they do not make reported test accuracy equal the filter by definition. Zero-shot image→video transfer further separates train objective from claimed gains. No self-definitional loop, fitted-input-as-prediction, or load-bearing uniqueness/self-citation chain is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 1 invented entities

Load-bearing content is mostly standard transformer mechanics plus empirical design choices. The method invents one trained embedding and one projection; free parameters are the usual optimization and inference knobs. No new physical entities. Domain assumptions include frozen pretrained VLMs, frame-level relevance labels, and mean value pooling as the retrieval key.

free parameters (4)
  • ReToken embedding X_r and projection W_r (plus logit scale τ) = d-dimensional vectors/matrices; τ learnable (values not reported)
    Only trained parameters in the default frozen-VLM setting; fit on filtered MIRAGE multi-image QA via class-balanced BCE on retrieval scores.
  • Retrieval budget K and early-layer budget K1 = K1=256 default; K task-dependent (e.g. 1 on VHs, 100 on LVBench)
    Inference hyperparameters that control precision/recall and memory; K1 default 256, K varied (1–100+) in experiments.
  • Training hyperparameters (lr, epochs, batch, λ for partial-tune) = lr 3e-4 frozen / 2e-5 partial; batch 64; 3 epochs Qwen / 1 InternVL
    Chosen for single-H100 training; partial-tune uses early stopping after 1 epoch.
  • Video encode chunk size and sliding-window length l_m = chunk 128 frames; l_m=30,000 tokens
    Engineering knobs for long-video KV cache construction following ReKV-style chunking.
axioms (5)
  • domain assumption Decoder-only VLM attention computes Q,K,V such that values carry content propagated to attending tokens while QK only routes aggregation.
    Stated in §3.1 'Why Values Carry Retrieval Signal'; standard transformer mechanics used to motivate value-space scoring.
  • domain assumption Ground-truth per-image/frame relevance labels in multi-image QA are a valid supervision signal for retrieval that should transfer to video frames.
    Training uses y_f ∈ {0,1} from MIRAGE-style annotations (§3.2, §4.1); zero-shot video transfer assumes this label structure generalizes.
  • ad hoc to paper Mean-pooling visual tokens within a frame into one value vector preserves enough signal to rank frames for QA.
    Eq. 3 defines s_f via cos(W_r X_r, mean_i v_i); Limitations explicitly flag token-level scoring and multi-frame sets as open.
  • ad hoc to paper Broadcasting final-layer top-K to all layers (two-pass) is a valid sparse attention pattern for answer generation.
    §3.2 Inference; Table 9 shows two-pass is required for ReToken though not for ReKV.
  • standard math Standard next-token CE and cosine-similarity retrieval scoring are appropriate losses/metrics.
    Eqs. 4–5; routine ML objectives.
invented entities (1)
  • ReToken (single learnable retrieval embedding X_r + final-layer projection W_r) independent evidence
    purpose: Serve as an explicit, learned retrieval query in value space to rank frames from a pre-filled visual KV cache.
    Core proposed module; not a physical entity but a new model component trained only for retrieval, distinct from Q-Former multi-query compressors.

pith-pipeline@v1.2.0-daily-grok45 · 23326 in / 3729 out tokens · 66795 ms · 2026-07-31T01:40:48.935870+00:00 · methodology

0 comments
read the original abstract

Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, ReToken yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.0-point gain with Qwen3VL-8B. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken

Figures

Figures reproduced from arXiv: 2607.28627 by Derek Hoiem, Jianfeng Gao, Reuben Tan, Yao Xiao, Yuqun Wu, Zhen Zhu.

Figure 1
Figure 1. Figure 1: Attention-based retrieval mechanisms are not effective for retrieving relevant visual information in VLMs. (a) shows attention scores from question tokens (as queries) to frame tokens (as keys), across frames in one video; (b) shows whether the top-1 retrieved frame falls within the ground-truth time window (240 candidate frames per video on average, sampled at 0.5 FPS). 1 Introduction Long visual contexts… view at source ↗
Figure 2
Figure 2. Figure 2: Retrieval Example. Question/Sentence: "For the image with a cow, is there a truck?". Target phrase: "a cow". where S plq K denotes the K frames most relevant to the question at layer l. Given the per-layer retrieval sets tS plq K u N l“1 , the model re-runs generation through the frozen VLM with layer-wise sparse attention: at each layer l, the question and answer tokens attend only to the visual tokens be… view at source ↗
Figure 3
Figure 3. Figure 3: Training Pipeline. The visual tokens, question tokens, and a learnable embedding (RETOKEN) are fed into the LLM decoder. The RETOKEN output from the final layer serves as a retrieval query, scoring each frame based on its cosine similarity to the frame’s pooled value features. The resulting scores are supervised using ground-truth frame-relevance labels and a class-balanced binary cross-entropy loss. By de… view at source ↗
Figure 4
Figure 4. Figure 4: Inference Pipeline. Our method first caches features for all input frames. Given a text query, ReToken retrieves the relevant frames, and the answer is generated from their cached features. Cache video once: At video ingestion time, the VLM processes the visual tokens and stores the resulting per-layer KV cache in a persistent cache. For short videos, this is a single full-context forward pass. For long vi… view at source ↗
Figure 5
Figure 5. Figure 5: Accuracy vs. Retrieve K Images. Frozen Qwen3VL-8B [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training Comparison. The RETOKEN ˆValue variant (a) converges much faster in retrieval loss, and (b) learns a substantially larger retrieval score gap between relevant and irrelevant images. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 15 linked inside Pith

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    System card addendum: Claude Opus 4.1

    Anthropic. System card addendum: Claude Opus 4.1. Technical report, Anthropic, August 2025

    2025

  3. [3]

    Goldfish: Vision-language understanding of arbitrarily long videos

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Mingchen Zhuge, Jian Ding, Deyao Zhu, Jürgen Schmidhuber, and Mohamed Elhoseiny. Goldfish: Vision-language understanding of arbitrarily long videos. InEuropean Conference on Computer Vision, pages 251–267. Springer, 2024

    2024

  4. [4]

    Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025

    Pith/arXiv arXiv 2025

  5. [5]

    Perception encoder: The best visual embeddings are not at the output of the network.arXiv preprint arXiv:2504.13181, 2025

    Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network.arXiv preprint arXiv:2504.13181, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Where did i leave my keys? — episodic-memory-based question an- swering on egocentric videos

    Leonard Bärmann and Alex Waibel. Where did i leave my keys? — episodic-memory-based question an- swering on egocentric videos. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1559–1567, 2022

    2022

  7. [7]

    Webqa: Multihop and multimodal qa

    Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16495–16504, 2022

    2022

  8. [8]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.arXiv preprint arXiv:2412.05271, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185–24198, 2024

    2024

  10. [10]

    Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  11. [11]

    Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning.Advances in neural information processing systems, 36:49250–49267, 2023

    2023

  12. [12]

    Grounded question-answering in long egocentric videos

    Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12934–12943, 2024

    2024

  13. [13]

    Streaming video question-answering with in-context video kv-cache retrieval

    Shangzhe Di, Zhelun Yu, Guanghao Zhang, Haoyuan Li, Tao Zhong, Hao Cheng, Bolin Li, Wanggui He, Fangxun Shu, and Hao Jiang. Streaming video question-answering with in-context video kv-cache retrieval. arXiv preprint arXiv:2503.00540, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Human-inspired episodic memory for infinite context llms

    Zafeirios Fountas, Martin A Benfeghoul, Adnan Oomerjee, Fenia Christopoulou, Gerasimos Lampouras, Haitham Bou-Ammar, and Jun Wang. Human-inspired episodic memory for infinite context llms. In13th International Conference on Learning Representations Iclr 2025. ICLR, 2025

    2025

  15. [15]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24108–24118, 2025. 12

    2025

  16. [16]

    Lm-infinite: Zero-shot extreme length generalization for large language models

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero-shot extreme length generalization for large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3991–4008, 2024

    2024

  17. [17]

    Ma-lmm: Memory-augmented large multimodal model for long-term video understanding

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-lmm: Memory-augmented large multimodal model for long-term video understanding. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13504– 13514, 2024

    2024

  18. [18]

    E5-v: Universal embeddings with multimodal large language models.arXiv preprint arXiv:2407.12580, 2024

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. E5-v: Universal embeddings with multimodal large language models.arXiv preprint arXiv:2407.12580, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700–13710, 2024

    2024

  20. [20]

    Ren: Fast and efficient region encodings from patch-based image encoders.arXiv preprint arXiv:2505.18153, 2025

    Savya Khosla, Sethuraman TV , Barnett Lee, Alexander Schwing, and Derek Hoiem. Ren: Fast and efficient region encodings from patch-based image encoders.arXiv preprint arXiv:2505.18153, 2025

    arXiv 2025

  21. [21]

    Proxyclip: Proxy attention improves clip for open-vocabulary segmentation

    Mengcheng Lan, Chaofeng Chen, Yiping Ke, Xinjiang Wang, Litong Feng, and Wayne Zhang. Proxyclip: Proxy attention improves clip for open-vocabulary segmentation. InEuropean Conference on Computer Vision, pages 70–88. Springer, 2024

    2024

  22. [22]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  23. [23]

    Llama-vid: An image is worth 2 tokens in large language models

    Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. InEuropean Conference on Computer Vision, pages 323–340. Springer, 2024

    2024

  24. [24]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean conference on computer vision, pages 740–755. Springer, 2014

    2014

  25. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024

    2024

  26. [26]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  27. [27]

    Lamra: Large multimodal model as your advanced retrieval assistant

    Yikun Liu, Yajie Zhang, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiangchao Yao, Yanfeng Wang, and Weidi Xie. Lamra: Large multimodal model as your advanced retrieval assistant. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 4015–4025, 2025

    2025

  28. [28]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    Yongdong Luo, Xiawu Zheng, Guilin Li, Shukang Yin, Haojia Lin, Chaoyou Fu, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, et al. Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    arXiv 2024

  29. [29]

    Drvideo: Document retrieval based long video understanding

    Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. Drvideo: Document retrieval based long video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 18936–18946, 2025

    2025

  30. [30]

    An- swer mining from a pool of images: towards retrieval-based visual question answering.arXiv preprint arXiv:2306.16713, 2023

    Abhirama Subramanyam Penamakuri, Manish Gupta, Mithun Das Gupta, and Anand Mishra. An- swer mining from a pool of images: towards retrieval-based visual question answering.arXiv preprint arXiv:2306.16713, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  32. [32]

    Longvu: Spatiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding.arXiv preprint arXiv:2410.17434, 2024. 13

    Pith/arXiv arXiv 2024

  33. [33]

    Video-xl: Extra-long vision language model for hour-scale video understanding

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 26160–26169, 2025

    2025

  34. [34]

    Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaugh- lin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221–18232, 2024

    2024

  36. [36]

    Slidevqa: A dataset for document visual question answering on multiple images

    Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito, and Kuniko Saito. Slidevqa: A dataset for document visual question answering on multiple images. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13636–13645, 2023

    2023

  37. [37]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology.arXiv preprint arXiv:2403.08295, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786, 2025

    Pith/arXiv arXiv 2025

  40. [40]

    Sclip: Rethinking self-attention for dense vision-language inference

    Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. InEuropean conference on computer vision, pages 315–332. Springer, 2024

    2024

  41. [41]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, et al. Lvbench: An extreme long video understanding benchmark. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 22958–22967, 2025

    2025

  42. [42]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  43. [43]

    Videoagent: Long-form video under- standing with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video under- standing with large language model as agent. InEuropean Conference on Computer Vision, pages 58–76. Springer, 2024

    2024

  44. [44]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 3272–3283, 2025

    2025

  45. [45]

    Visual haystacks: A vision-centric needle-in-a-haystack benchmark.arXiv preprint arXiv:2407.13766, 2024

    Tsung-Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Visual haystacks: A vision-centric needle-in-a-haystack benchmark.arXiv preprint arXiv:2407.13766, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Infllm: Training-free long-context extrapolation for llms with an efficient context memory

    Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Infllm: Training-free long-context extrapolation for llms with an efficient context memory. Advances in neural information processing systems, 37:119638–119661, 2024

    2024

  47. [47]

    Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks.arXiv preprint arXiv:2309.17453, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    Textregion: Text-aligned region tokens from frozen image-text models.arXiv preprint arXiv:2505.23769, 2025

    Yao Xiao, Qiqian Fu, Heyi Tao, Yuqun Wu, Zhen Zhu, and Derek Hoiem. Textregion: Text-aligned region tokens from frozen image-text models.arXiv preprint arXiv:2505.23769, 2025

    arXiv 2025

  49. [49]

    Self-chained image-language model for video localization and question answering.Advances in Neural Information Processing Systems, 36:76749–76771, 2023

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering.Advances in Neural Information Processing Systems, 36:76749–76771, 2023

    2023

  50. [50]

    Extract free dense labels from clip

    Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. InEuropean conference on computer vision, pages 696–712. Springer, 2022. 14

    2022

  51. [51]

    w/o filtered

    Yutao Zhu, Zhaoheng Huang, Zhicheng Dou, and Ji-Rong Wen. One token can help! learning scalable and pluggable virtual tokens for retrieval-augmented large language models. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26166–26174, 2025. 15 A Prompt Templates In this section, we detail the prompt templates used during tr...

    2025