Pith. sign in

REVIEW 4 major objections 6 minor 5 cited by

Towards General Continuous Memory for Vision-Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A vision-language model can serve as its own continuous memory encoder, compressing retrieved multimodal and multilingual knowledge into eight embeddings that improve reasoning without lengthening the context.

desk verdict A genuinely new memory recipe for VLMs that mostly works; send it to review, but ask for the input-injection ablation and error bars. read the letter →

arxiv 2505.17670 v2 pith:YPMNBMVR submitted 2025-05-23 cs.LG cs.AI

Wenyi Wu , Zixuan Song , Kun Zhou , Yifei Shao , Zhiting Hu , Biwei Huang This is my paper · ORCID
classification cs.LGcs.AI
keywords continuousmemoryvision-languagemodelsretrieval-augmentedgenerationcontextcompressionmultimodalreasoningmultilingualQ-Formerparameter-efficientfine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a vision-language model can serve as its own continuous memory encoder: the VLM produces dense embeddings for retrieved multimodal and multilingual knowledge, and those embeddings, rather than long token sequences, are what the model reads at inference. The proposed method, CoMEM, compresses each retrieved image-text pair into just eight continuous vectors with a lightweight Q-Former while fine-tuning only LoRA layers (1.2% of parameters) on 15.6K self-synthesized samples. Because the inference-time VLM stays frozen, the memory module is plug-and-play. Across eight knowledge-intensive benchmarks the authors report average gains of 8.0 points on English reasoning and 5.1 points on multilingual reasoning over the base VLM, and show the compressed memory also transfers to a language-only model. If these results hold, external knowledge can be supplied to VLMs in a compact, detachable form that avoids the context-length degradation seen with retrieval-augmented generation.

What carries the argument

The mechanism is a two-part compressor riding on the frozen VLM. First, the VLM encodes a retrieved image-text pair and its late-layer hidden states are collected. Second, a Q-Former—a lightweight transformer with eight learnable query embeddings whose parameters are shared across layers—attends to those hidden states and outputs eight continuous vectors per knowledge item. Training updates only the Q-Former and LoRA matrices (1.2% of parameters) on 15.6K self-synthesized samples, and at inference the resulting vectors are simply prepended to the VLM's input embedding sequence. This gives a compression ratio above 80× relative to the average 643.7 tokens per knowledge item, and it is what makes the memory plug-and-play: the inference model is never modified.

What would settle it

Concretely, measure InfoSeek or OK-VQA accuracy with the eight memory vectors replaced by eight vectors sampled from the same embedding distribution but carrying no information about the retrieved item: if accuracy stays high, the frozen VLM is ignoring the memory content, and if accuracy drops to the no-memory baseline, the vectors are demonstrably content-bearing.

Watch

Extended reading notes

Core claim

The paper's central claim is that a VLM can serve as its own continuous memory encoder: hidden states taken from the later layers of a frozen VLM, after being pooled by a trainable Q-Former, are semantically aligned enough that the same VLM can consume them as ordinary input embeddings and reason better with them than with the original text-and-image tokens. The authors establish this in two steps: first, a training-free study shows that simply concatenating the VLM's own layer-wise hidden states at selected layers beats both vanilla RAG and token pruning; second, the full CoMEM method adds a shared-parameter Q-Former with eight query embeddings and LoRA fine-tuning, trained only on self-synthesized samples that a vanilla RAG run answered correctly. The resulting eight-vector memories improve accuracy on six English benchmarks and two multilingual benchmarks, and the same VLM-encoded memory can be prepended to a language-only LLM that never sees images.

Load-bearing premise

The method assumes the eight compressed vectors produced by the Q-Former already occupy the same internal space as the frozen VLM's ordinary inputs, so inserting them needs no extra alignment step and the model will read them as meaningful context.

Editorial extensions

If this is right

  • On Qwen2-VL-Instruct, CoMEM raises average accuracy on six English benchmarks by 8.0 points and on two multilingual benchmarks by 5.1 points; the corresponding gains on Qwen2.5-VL-Instruct are 7.7 and 4.3 points.
  • Because the memory is compact, retrieval scale stops hurting: accuracy remains stable from top-3 to top-50 retrieved pairs, whereas vanilla RAG degrades beyond about 30 pairs.
  • The memory transfers across model families: a language-only LLM reading VLM-encoded memory reaches 17.8% average accuracy on InfoSeek and OVEN, versus 7.0% with text RAG and 3.1% with no augmentation.
  • The recipe is inexpensive enough to be practical: 15.6K self-synthesized samples, 1.2% trainable parameters, one epoch, and about 20 hours on a single H100; scaling data or parameters by 4× gives only marginal gains.
  • CoMEM is detachable by design: since the inference-time VLM is frozen, the same model can run with or without memory, making the module an optional augmentation rather than a retrained system.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the embedding-space compatibility generalizes, the same memory encoding could serve as a shared knowledge bus: one VLM encodes once, and any model with a sufficiently aligned embedding space consumes the vectors without retraining; the paper demonstrates this only for Qwen2.5-Instruct reading Qwen2.5-VL memory, but the architecture places no obvious barrier to other aligned models.
  • A natural stress test the paper leaves implicit is to replace the eight memory vectors with eight random vectors drawn from the same distribution: if accuracy does not drop, the frozen VLM is not actually reading the memory, and if it drops sharply, the vectors are content-bearing and the compression claim is doing real work.
  • Because training data is filtered by whether a vanilla RAG run answered correctly, the encoder is taught to compress knowledge that RAG can already surface; extending CoMEM to settings where retrieval is noisy or the answer is absent would test whether the memory adds information or merely repackages retrieved context.
  • The paper's training-free experiment suggests that even attention-based selection of hidden states helps, so a lighter compressor without a Q-Former might capture part of the gain; comparing these variants would isolate what LoRA-plus-Q-Former training actually contributes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes CoMEM, a continuous memory mechanism for vision-language models (VLMs). The core idea is to use the VLM itself as a memory encoder: retrieved image-text knowledge items are encoded by the (frozen, plus LoRA-tuned) VLM, then compressed into eight continuous embeddings by a Q-Former, and these embeddings are prepended to the input embedding sequence of a frozen inference VLM. The authors report a training-free empirical study suggesting that VLM hidden states can serve as memory, and then introduce a training recipe that fine-tunes only 1.2% of parameters on 15.6k self-synthesized samples. Experiments on eight benchmarks show average gains over base VLMs and RAG baselines, with additional studies on long-context robustness and transferability to a text-only LLM.

Significance. If the central claim holds, CoMEM would provide a highly data- and parameter-efficient way to augment VLMs with external multimodal and multilingual knowledge, using only a compact set of continuous vectors instead of long token sequences. The paper includes several strengths: a training-free empirical analysis of VLM-as-memory, a long-context stability study, a transferability experiment to a pure LLM, public code and data, and scaling ablations on data size and parameter count. The reported average improvements of +8.0% (English) and +5.1% (multilingual) over base VLMs are notable and would be of practical interest for knowledge-intensive visual question answering.

major comments (4)
  1. [Section 3.2, Eq. (1)] The final CoMEM mechanism prepends Q-Former output vectors V_t directly to the input embedding sequence E_I of the frozen inference VLM, but the training-free evidence in Section 2 (Table 2) uses a different interface: hidden states concatenated at layers 17–19. The paper never shows that the frozen VLM can consume input-level continuous vectors without a learned projection; the training loss can in principle align them, yet no distributional analysis, no projection-layer ablation, and no training-free input-level test are provided. This leaves the central 'VLM as its own memory encoder' claim without direct support for the deployed interface.
  2. [Table 3] On MRAG-Bench, CoMEM+Qwen2-VL (35.1) and CoMEM+Qwen2.5-VL (38.1) are below the base models (39.3 and 42.0) and below their RAG counterparts (40.4 and 42.1). The paper claims 'significant gains across multimodal reasoning benchmarks' without acknowledging this regression or providing an explanation. The claim of consistent improvement needs to be qualified or mechanistically explained.
  3. [Section 2, Table 2] For Qwen2-VL-Instruct, the VLM-as-Memory method scores 38.9 on A-OKVQA versus a base score of 41.8, a clear degradation. This contradicts the statement that VLM-as-Memory 'outperforms both approaches across most tasks' and reduces the strength of the empirical motivation for the proposed method.
  4. [Section 3.3] The training corpus is synthesized by Qwen2.5-VL-Instruct and filtered by whether that same model produces correct answers, and the main evaluation uses Qwen2-VL and Qwen2.5-VL. The cross-model results with Qwen2-VL are encouraging, but no experiment trains the memory encoder on one architecture and evaluates on another. The claim of a 'general continuous memory' would be substantially strengthened by such a cross-architecture control.
minor comments (6)
  1. [Section 2.1] In the sentence 'our method outperforms the baseline even at a high compression rate of 5This suggests ...', a period is missing after '5'.
  2. [Table 3] The mPLUG-Owl3 row contains the value '1.945.031.9', which appears to be missing separators between the MRAG and OKVQA columns and should be reformatted.
  3. [Section 3.2] The Q-Former formulation writes 'H(0) = q', but q is a set of k query embeddings; it should be specified as a k×d matrix rather than a single vector.
  4. [Figure 2] The x-axis label '5' appears cut off at the high-compression end; the label should read '5%' or similar.
  5. [Table 1] Several entries use inconsistent spacing, e.g., 'V oCo-LLaMA' and 'ReflectiV A'; these should be typeset consistently.
  6. [Appendix D, Table 6] The row for '4x' reads '4x34.828.431.3' with missing separators; the values need proper spacing for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the VLM-as-Memory insight is supported by an independent training-free experiment, and the trained CoMEM is validated on external benchmarks.

full rationale

The paper's derivation chain does not reduce to its own inputs by construction or by self-citation. The central claim that a VLM can serve as its own memory encoder is supported by the training-free empirical study in Section 2, where hidden states of the VLM are extracted and reinjected at layers 17-19 (Table 2). This measurement is independent of the later training objective and does not presuppose the result. The trained CoMEM method then adds a Q-Former and LoRA to compress VLM representations, and its evaluation includes held-out benchmarks (OVEN, MRAG-Bench, A-OKVQA, ViQuAE, CVQA) that are not in the 15.6k self-synthesized training corpus. The self-synthesized data filtering (retaining only correct RAG answers) is a standard self-distillation/compression setup, not a circular definition of the measured accuracy. The one notable weakness is a non-circular correctness concern: the Section 2 experiment injects hidden states at middle layers, whereas the final plug-and-play mechanism prepends Q-Former outputs to the input embedding sequence (Section 3.2), so the preliminary evidence does not fully cover the final deployment mechanism. That gap is an unsupported inference, not a circular step. The minor citation to co-authored work [17] for the general intuition that continuous embeddings are expressive is not load-bearing and does not import any ansatz or uniqueness theorem. Therefore, no circular step meets the evidentiary bar, and the correct finding is no significant circularity.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several unstated or lightly stated premises: the CLIP retriever returns relevant knowledge for every query; hidden states of the VLM preserve answer-relevant information after compression to eight vectors; prepended Q-Former vectors are readable by the frozen VLM without a projection; self-synthesized, correctness-filtered training data is representative; and exact-match accuracy on static benchmarks captures the value of the memory. Hyperparameters (k=8, LoRA rank 16, top-10 retrieval, 15.6k samples) are chosen by hand and are not derived from first principles.

free parameters (7)
  • number of memory embeddings k = 8
    Chosen by hand as the compression rate; not derived from the data or task.
  • LoRA rank = 16
    Chosen by hand; controls the 1.2% trainable parameter count.
  • Q-Former layers = not reported
    Shared across layers, but the exact number is not stated in the main text.
  • top-k retrieved knowledge pairs = 10
    Used consistently across all experiments; affects the content of the memory and the comparison baselines.
  • training corpus size = 15.6k
    Curated by filtering teacher-correct outputs and by translating 200 samples with GPT-4o-mini; chosen for efficiency, not derived.
  • attention selection threshold for VLM-as-Memory+Attn = 25% (top-25%)
    Used only in the preliminary study in Section 2.1 to select continuous embeddings.
  • layers for memory injection in preliminary study = 17-19
    Chosen for efficiency in the training-free VLM-as-Memory experiment; not used in the final method.
assumptions (6)
  • domain assumption Transformer self-attention hidden states aggregate answer-relevant semantic information.
    Relied on in Section 2 to argue that VLM embeddings can serve as memory; supported by cited work but not proven here.
  • domain assumption CLIP-based top-10 retrieval from WIT returns knowledge items relevant to each query.
    Assumed in Section 2.1 and Section 4.1; if retrieval fails, memory cannot help.
  • domain assumption Compression to 8 embeddings preserves the answer-relevant content of each knowledge item.
    Central to the method; the paper provides empirical evidence but no formal guarantee.
  • ad hoc to paper Q-Former output vectors can be prepended to the frozen VLM input embedding sequence without a learned projection.
    Stated in Section 3.2 as a simple concatenation; no projection layer or compatibility analysis is provided.
  • ad hoc to paper Self-synthesized and correctness-filtered training samples are representative enough to generalize to unseen benchmarks and languages.
    Assumed in Section 3.3; the filtering step could induce selection bias.
  • domain assumption Exact-match and multiple-choice accuracy on static benchmarks measure the value of the memory module.
    Used throughout Section 4 and Appendix A; standard practice but static benchmarks may not reflect dynamic or noisy inputs, as the authors acknowledge in Appendix C.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards General Continuous Memory for Vision-Language Models." pith.science (2026). https://pith.science/paper/YPMNBMVR

@misc{pith2026250517670,
  author       = {Pith},
  title        = {Pith review of: Towards General Continuous Memory for Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YPMNBMVR}},
  note         = {Machine review of arXiv:2505.17670}
}
read the original abstract

Language models (LMs) and their extension, vision-language models (VLMs), have achieved remarkable performance across various tasks. However, they still struggle with complex reasoning tasks that require multimodal or multilingual real-world knowledge. To support such capabilities, an external memory system that can efficiently provide relevant multimodal information is essential. Existing approaches generally concatenate image and text tokens into a long sequence as memory, which, however, may drastically increase context length and even degrade performance. In contrast, we propose using continuous memory, a compact set of dense embeddings to more effectively and efficiently represent multimodal and multilingual knowledge. Our key insight is that a VLM can serve as its own continuous memory encoder. We empirically show that this design improves performance on complex multimodal reasoning tasks. Building on this, we introduce a data-efficient and parameter-efficient method to fine-tune the VLM into a memory encoder, requiring only 1.2% of the model's parameters and a small corpus of 15.6K self-synthesized samples. Our approach CoMEM utilizes VLM's original capabilities to encode arbitrary multimodal and multilingual knowledge into just 8 continuous embeddings. Since the inference-time VLM remains frozen, our memory module is plug-and-play and can be flexibly integrated as needed. Extensive experiments across eight multimodal reasoning benchmarks demonstrate the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2505.17670 by the authors.

Figure 1
Figure 1. CoMEM architecture in comparison to the traditional RAG method. Through large-scale training, language mod￾els (LMs) [1, 2] have demonstrated remarkable performance across diverse real-world tasks. LMs even surpass human capabilities in language reasoning tasks [3] such as mathematical problem solving [4], commonsense reasoning[5], and code synthesis [6]. However, when confronted with complex reasoning tasks that de… view at source ↗
Figure 2
Figure 2. Qwen2.5 accuracy with vary￾ing attention-based compression rates. Effectiveness Study of VLM-as-Memory Methods. As shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Overview of the CoMEM architecture. Given a vision-language query, the system retrieves [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of Long Context Ability of RAG [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Case studies comparing CoMEM with baseline model and model with RAG. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MMAgent-R$^2$: Learning to Rerank and Reject for Agentic mRAG

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An agentic mRAG framework uses GRPO-trained visual reranking and active rejection to verify retrieved candidate entities, achieving state-of-the-art on three KB-VQA benchmarks.

  2. Dual Latent Memory for Visual Multi-agent System

    cs.AI 2026-01 conditional novelty 6.0 of 10

    L2-VMAS replaces text-based inter-agent communication in visual multi-agent systems with decoupled latent perception and thinking memories, improving accuracy by 2.7–5.4% and cutting token use by 21.3–44.8%.

  3. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  4. The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlook

    cs.AI 2026-04 accept novelty 5.0 of 10

    A large survey organizes latent-space work in language-based models by foundation, evolution, four mechanisms, seven abilities, and open challenges.

  5. Recurrence Meets Transformers for Universal Multimodal Retrieval

    cs.CV 2025-09 conditional novelty 5.0 of 10

    A single multimodal retriever with a gated recurrent Transformer cell, one-token output, and three-layer sampling achieves strong results on M2KR and M-BEIR and improves RAG-VQA accuracy.

Reference graph

Works this paper leans on

59 extracted references · 16 canonical work pages · cited by 5 Pith papers

  1. [1]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  2. [2]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [3]

    Language models show human-like content effects on reasoning tasks.arXiv preprint arXiv:2207.07051, 2022

    Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effects on reasoning tasks.arXiv preprint arXiv:2207.07051, 2022

    arXiv 2022

  4. [4]

    Large language models for mathematical reasoning: Progresses and challenges.arXiv preprint arXiv:2402.00157, 2024

    Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges.arXiv preprint arXiv:2402.00157, 2024

    arXiv 2024

  5. [5]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  6. [6]

    A survey on large language models for code generation.arXiv preprint arXiv:2406.00515, 2024

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation.arXiv preprint arXiv:2406.00515, 2024. 10

    arXiv 2024

  7. [7]

    Token pruning in multimodal large language models: Are we solving the right problem?arXiv preprint arXiv:2502.11501, 2025

    Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. Token pruning in multimodal large language models: Are we solving the right problem?arXiv preprint arXiv:2502.11501, 2025

    arXiv 2025

  8. [8]

    Retrieval-augmented generation for knowledge-intensive nlp tasks

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Angela Fan, Vishrav Chaudhary, Matthias Gallé, Veselin Stoyanov, and Wen-tau Yih. Retrieval-augmented generation for knowledge-intensive nlp tasks. InAdvances in Neural Information Processing Systems (NeurIPS), 2020

    work page 2020

Show all 59 references
  1. [9]

    Augmenting language models with long-term memory.arXiv preprint arXiv:2306.07174, 2023

    Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory.arXiv preprint arXiv:2306.07174, 2023

    2023 arXiv

  2. [10]

    Realm: Retrieval-augmented language model pre-training

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. InInternational Conference on Machine Learning (ICML), 2020

    2020

  3. [11]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  4. [12]

    Visrag: Vision-based retrieval-augmented generation on multi-modality documents.arXiv preprint arXiv:2410.10594, 2024

    Shi Yu, Chaoyue Tang, Bokai Xu, Junbo Cui, Junhao Ran, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, et al. Visrag: Vision-based retrieval-augmented generation on multi-modality documents.arXiv preprint arXiv:2410.10594, 2024

    arXiv

  5. [13]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. InEuropean Conference on Computer Vision, pages 19–35. Springer, 2024

    2024

  6. [14]

    Sparsevlm: Visual token sparsification for efficient vision-language model inference.arXiv preprint arXiv:2410.04417, 2024

    Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevlm: Visual token sparsification for efficient vision-language model inference.arXiv preprint arXiv:2410.04417, 2024

    arXiv

  7. [15]

    A mathematical framework for transformer circuits.Transformer Circuits Thread, 2021.https://transformer-circuits.pub/2021/framework/index.html

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

    2021

  8. [16]

    In-context learning and induction heads.arXiv preprint arXiv:2209.11895, 2022

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022 arXiv

  9. [17]

    Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769, 2024

    Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space.arXiv preprint arXiv:2412.06769, 2024

    2024 arXiv

  10. [18]

    Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver io: A general ar...

    2021 arXiv

  11. [19]

    Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien

    Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. InProceedings of the 34th Internat...

    2017

  12. [20]

    The trade-offs of domain adaptation for neural language models

    David Grangier and Dan Iter. The trade-offs of domain adaptation for neural language models. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), pages 3802–3813, Dublin, Ireland, May 2022. Association for Computat...

    2022

  13. [21]

    Attention is all you need.arXiv preprint arXiv:1706.03762, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need.arXiv preprint arXiv:1706.03762, 2017

    2017 arXiv

  14. [22]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does bert look at? an analysis of bert’s attention.arXiv preprint arXiv:1906.04341, 2019

    1906 arXiv

  15. [23]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phil Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  16. [24]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  17. [25]

    Echosight: Advancing visual-language models with wiki knowledge

    Yibin Yan and Weidi Xie. Echosight: Advancing visual-language models with wiki knowledge. arXiv preprint arXiv:2407.12735, 2024. Accepted at EMNLP 2024 Findings

    2024 arXiv

  18. [26]

    Augmenting multimodal llms with self-reflective tokens for knowledge-based visual question answering.arXiv preprint arXiv:2411.16863, 2024

    Federico Cocchi, Nicholas Moratelli, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Augmenting multimodal llms with self-reflective tokens for knowledge-based visual question answering.arXiv preprint arXiv:2411.16863, 2024. Accepted at CVPR 2025

    2024 arXiv

  19. [27]

    RORA-VLM: Robust retrieval augmentation for vision language models.arXiv preprint arXiv:2410.08876, 2024

    Jingyuan Qi, Zhiyang Xu, Rulin Shao, Yang Chen, Jin Di, Yu Cheng, Qifan Wang, and Lifu Huang. RORA-VLM: Robust retrieval augmentation for vision language models.arXiv preprint arXiv:2410.08876, 2024

    2024 arXiv

  20. [28]

    xrag: Extreme context compression for retrieval-augmented generation with one token.arXiv preprint arXiv:2405.13792, 2024

    Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. xrag: Extreme context compression for retrieval-augmented generation with one token.arXiv preprint arXiv:2405.13792, 2024

    2024 arXiv

  21. [29]

    Kv-distill: Nearly lossless learnable context compression for llms.ArXiv, abs/2503.10337, 2025

    Vivek Chari, Guanghui Qin, and Benjamin Van Durme. Kv-distill: Nearly lossless learnable context compression for llms.ArXiv, abs/2503.10337, 2025

    2025 arXiv

  22. [30]

    V oco-llama: Towards vision compression with large language models.arXiv preprint arXiv:2406.12275, 2024

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, and Yansong Tang. V oco-llama: Towards vision compression with large language models.arXiv preprint arXiv:2406.12275, 2024

    2024 arXiv

  23. [31]

    MA-LMM: Memory-augmented large multimodal model for long-term video understanding.arXiv preprint arXiv:2404.05726, 2024

    Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. MA-LMM: Memory-augmented large multimodal model for long-term video understanding.arXiv preprint arXiv:2404.05726, 2024

    2024 arXiv

  24. [32]

    M+: Extending memoryllm with scalable long-term memory.arXiv preprint arXiv:2502.00592, 2025

    Yu Wang, Dmitry Krotov, Yuanzhe Hu, Yifan Gao, Wangchunshu Zhou, Julian McAuley, Dan Gutfreund, Rogerio Feris, and Zexue He. M+: Extending memoryllm with scalable long-term memory.arXiv preprint arXiv:2502.00592, 2025

    2025 arXiv

  25. [33]

    Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560, 2024

    Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G Patil, Ion Stoica, and Joseph E Gonzalez. Memgpt: Towards llms as operating systems.arXiv preprint arXiv:2310.08560, 2024

    2024 arXiv

  26. [34]

    Can pre-trained vision and language models answer visual information-seeking questions?arXiv preprint arXiv:2302.11713, 2023

    Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- Wei Chang. Can pre-trained vision and language models answer visual information-seeking questions?arXiv preprint arXiv:2302.11713, 2023

    2023 arXiv

  27. [35]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019

    2019

  28. [36]

    A-okvqa: A benchmark for visual question answering using world knowledge

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European conference on computer vision, pages 146–162. Springer, 2022. 12

    2022

  29. [37]

    Wiki-LLaV A: Hierarchical retrieval-augmented generation for multimodal llms.arXiv preprint arXiv:2404.15406, 2024

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Wiki-LLaV A: Hierarchical retrieval-augmented generation for multimodal llms.arXiv preprint arXiv:2404.15406, 2024

    2024 arXiv

  30. [38]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  31. [39]

    Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning

    Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. InPro- ceedings of the 44th international ACM SIGIR conference on research and development in information r...

    2021

  32. [40]

    Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories

    Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories. InProceedings of the IEEE/CVF International Conferenc...

    2023

  33. [41]

    Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities

    Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. InProceedings of the IEEE/CVF International Conference on Computer V...

    2023

  34. [42]

    Mrag-bench: Vision-centric evaluation for retrieval-augmented multimodal models.arXiv preprint arXiv:2410.08182, 2024

    Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng. Mrag-bench: Vision-centric evaluation for retrieval-augmented multimodal models.arXiv preprint arXiv:2410.08182, 2024

    2024 arXiv

  35. [43]

    Viquae, a dataset for knowledge-based visual question answering about named entities

    Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé Le Borgne, Romaric Besançon, José G Moreno, and Jesús Lovón Melgarejo. Viquae, a dataset for knowledge-based visual question answering about named entities. InProceedings of the 45th international ACM SIGIR conference on r...

    2022

  36. [44]

    CVQA: Culturally-diverse multilingual visual question answering benchmark

    David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. CVQA: Culturally-diverse multilingual visual question answering benchmark. arXiv preprint arXiv:240...

    2024 arXiv

  37. [45]

    Gpt-4o technical report, 2024

    OpenAI. Gpt-4o technical report, 2024. Accessed: 2025-05-10

    2024

  38. [46]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  39. [47]

    Llava-next: Open large multimodal models

    LLaV A-VL Team. Llava-next: Open large multimodal models. https://github.com/ LLaVA-VL/LLaVA-NeXT, 2024. Accessed: 2025-05-10

    2024

  40. [48]

    Internlm- xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output.arXiv preprint arXiv:2407.03320, 2024

    Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, J...

    2024 arXiv

  41. [49]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models.arXiv preprint arXiv:2408.04840, 2024

    2024 arXiv

  42. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  43. [51]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024. 13

    2024 arXiv

  44. [52]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InInternational conference on machine learning, pages 1597–1607. PmLR, 2020

    2020

  45. [53]

    Mm1: methods, analysis and insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Anton Belyi, et al. Mm1: methods, analysis and insights from multimodal llm pre-training. InEuropean Conference on Computer Vision, pages 304–323....

    2024

  46. [54]

    Ulip-2: Towards scalable multimodal pre-training for 3d understanding

    Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. InProceedings of the IEEE/CVF Conference on Computer Vision ...

    2024

  47. [55]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  48. [56]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36:19327–19352, 2023

    2023

  49. [57]

    In- context former: Lightning-fast compressing context for large language model.arXiv preprint arXiv:2406.13618, 2024

    Xiangfeng Wang, Zaiyi Chen, Zheyong Xie, Tong Xu, Yongyi He, and Enhong Chen. In- context former: Lightning-fast compressing context for large language model.arXiv preprint arXiv:2406.13618, 2024

    2024 arXiv

  50. [58]

    Adapting llms for efficient context processing through soft prompt compression

    Cangqing Wang, Yutian Yang, Ruisi Li, Dan Sun, Ruicong Cai, Yuzhu Zhang, and Chengqian Fu. Adapting llms for efficient context processing through soft prompt compression. InProceedings of the International Conference on Modeling, Natural Language Processing and Machine Learnin...

    2024

  51. [59]

    The probabilistic relevance framework: Bm25 and beyond.F oundations and Trends in Information Retrieval, 3(4):333–389, 2009

    Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond.F oundations and Trends in Information Retrieval, 3(4):333–389, 2009. 14 A Benchmark Details InfoSeekInfoSeek is a visual question answering (VQA) dataset tailored for information-seeki...

    2009

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.