Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Zero-Shot Vision Encoder Grafting via LLM Surrogates

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Vision encoders trained on small LLM surrogates can be grafted directly into the full-size model, matching full decoder training on several benchmarks and cutting Llama-70B VLM training cost by roughly 45%.

desk verdict Solid method with a genuine grafting mechanism; the ~45% cost savings headline needs data-subsampling and repeated-run details before I'd quote it. read the letter →

arxiv 2505.22664 v2 pith:4Y6FBZIX submitted 2025-05-28 cs.CV

Kaiyu Yue , Vasu Singla , Menglin Jia , John Kirchenbauer , Rifaa Qadri , Zikui Cai , Abhinav Bhatele , Furong Huang
show 1 more author
Tom Goldstein
This is my paper
classification cs.CV
keywords vision-languagemodelsvisionencodersurrogatemodelzero-shotgraftinglayertransitionanalysistrainingcostreductionLlama-70B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Training a vision-language model normally means fine-tuning a large language decoder, and with decoders the size of Llama-70B that step dominates the compute bill. This paper argues the decoder does not need to sit in the loop for most of encoder training: you can first train the vision encoder against a small 'surrogate' language model built from the target LLM's own shallow layers, then plug the finished encoder straight into the full-size model. The authors call this zero-shot grafting and report that the grafted pair surpasses the encoder-with-surrogate pair, matches full decoder training on several benchmarks, and cuts overall VLM training cost by about 45% when the target is Llama-70B. If true, the result separates vision-encoder alignment from expensive decoder-scale training and makes VLM development substantially cheaper.

What carries the argument

The load-bearing object is the surrogate model: a small transformer that keeps the target LLM's early-phase layers unchanged, keeps its last layer, and inserts a single 'translator' transformer layer at the transition point where the late phase begins. The translator is initialized from the shallowest replaced layer and fine-tuned on text-only instructions, so the surrogate speaks the same embedding language as the target. The transition point itself is located by computing, for each layer, the Kullback-Leibler divergence between the layer's softmax next-token distribution and the final distribution, and finding where 300 random text samples abruptly converge; the paper verifies with layer-replacement ablations and a control that perturbs early layers, showing that preserving the inherited early parameters is what makes zero-shot grafting work.

What would settle it

Re-run the decoder-training comparison with the 10% and 20% subsets drawn as random stratified samples of all 665K vision-language instructions, repeated over several seeds, and report the full benchmark table; if the surrogate-trained encoder no longer matches the 100%-data baseline at 10% data, the convergence and ~45% cost-reduction claims are artifacts of how the subsets were chosen.

Watch

Extended reading notes

Core claim

The central discovery is that a vision encoder can be aligned to a much larger LLM by training it against a small stand-in that shares that LLM's early layers. Tracing the Kullback-Leibler divergence between each layer's next-token distribution and the model's final output reveals a sharp transition point: layers before it form an early phase that fixes the model's representation space and carries the ability to transfer an encoder, while layers after it mostly settle on a concrete prediction. Replacing those late layers with a single translator layer yields a surrogate that keeps the target's embedding space intact. An encoder trained on this surrogate, when grafted into the target LLM with no further training, produces image features the target can read directly; on several benchmarks the grafted model exceeds the encoder-surrogate pair and approaches or matches full decoder fine-tuning. The same encoder also serves as a warm start, letting the full decoder reach baseline-level performance with roughly ten to twenty percent of the training data and cutting total training hours from about 35 to 19 for a Llama-70B VLM.

Load-bearing premise

The cost and convergence claims assume that the 10% and 20% slices of the instruction set used for decoder fine-tuning are representative of the full 665K-example set; if those slices are an easier or unbalanced portion, the apparent speed-up could come from data ordering rather than from the surrogate-trained encoder.

Editorial extensions

If this is right

  • A newly released LLM can be turned into a vision-language model by building a surrogate from its early layers and training only a vision encoder, avoiding full decoder fine-tuning.
  • The surrogate-trained encoder can be grafted into the target LLM without any further training, giving an immediate VLM that on several benchmarks matches a model trained end-to-end with the full decoder.
  • If further decoder fine-tuning is desired, the grafted encoder acts as a warm start and reaches baseline-level performance after training on only about 10-20% of the instruction set.
  • The approach appears to transfer across model families, since the layer-trajectory transition point is observed in Llama, Gemma, and Qwen models of different sizes.
  • Full decoder fine-tuning on top of the surrogate-trained encoder preserves the LLM's text-benchmark performance, whereas baseline VLM training degrades it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not pursue is applying the same surrogate construction to align encoders for other modalities, such as audio or depth, whenever a large autoregressive decoder is the training bottleneck.
  • The saturation seen at 20% data suggests the alignment signal is concentrated and that better data selection or curricula could shrink the required fraction below the paper's 10-20% range.
  • If the transition point can be predicted from model size or architecture, surrogate construction could be automated without recomputing the layer-trajectory analysis for each new LLM.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a method for training vision encoders for VLMs without repeatedly training against a large target LLM. It first identifies a 'transition point' in the layer-wise prediction trajectory of an LLM, then constructs a small 'surrogate' decoder by preserving the early layers before that point and replacing later layers with a single translator layer initialized from the preserved block. A vision encoder is fine-tuned on this surrogate decoder, and the resulting encoder is then either plugged directly into the full-size target LLM ('zero-shot grafting') or used as a warm start for full decoder training. Experiments on Llama-3B, Llama-8B, Llama-70B, and Qwen3-4B show that grafting often improves on the surrogate's own VLM performance, sometimes matches or exceeds full decoder training on individual benchmarks, and that the warm-start approach reduces total Llama-70B training time by approximately 45%.

Significance. If the claims hold, the paper offers a practical and conceptually interesting recipe for decoupling vision-encoder training from the cost of a giant LLM decoder. The strongest support comes from controlled ablations: the T(16,26)* experiment in Section 1.2 shows that preserving the early-phase parameters of the target is necessary for zero-shot grafting, and the Llama-8B ablations T(6,30), T(12,30), and T(17,30) in Tables 3-4 demonstrate a monotonic relationship between early-layer preservation and grafting quality. The evaluation spans multiple model families and scales, includes external VLM benchmarks, and the code is released. The main caveats are experimental: all numbers come from single runs, the construction of the X% training subsets in Table 8 is not stated, and the MME/POPE prompt protocol for zero-shot grafting is not fully specified. These issues directly affect the quantitative headline claim of ~45% cost reduction.

major comments (3)
  1. [Section 2.2, Tables 8-9] The headline ~45% cost reduction rests on the decoder-training time at 20% data in Table 9 (5.56 h) and on the claim that 10% of surrogate-trained data already matches 100% baseline performance. However, the paper does not state how the 10%, 20%, 30%, and 60% subsets of LLaVA-1.5-665K were sampled, and all numbers are single-run with no error bars. The baseline curve is non-monotonic (average score 0.5127 at 10%, 0.5153 at 20%, 0.6277 at 30%, 0.6538 at 100%), so an unrepresentative 20% slice (for example, a prefix or an unbalanced split) could explain the '20% of data suffices' result. Please specify the subsetting procedure, report repeated runs or variance, and, if possible, use a random stratified sampling scheme.
  2. [Section 1.2 and Section 2.1.1, Tables 2, 4, and 7] The evaluation protocol for MME and POPE appears to differ between zero-shot grafting and full decoder training. The text states that for grafting, because the target LLM has not been instruction-tuned on VLM data, 'we prompt the model with binary prompts' (Section 1.2), while the full decoder-training baseline is presumably evaluated under the standard LLaVA 'single word or phrase' protocol. The exact prompt template used for each row is not given, so it is unclear whether the comparisons labeled 'MMEbinary' and 'POPEbinary' are apples-to-apples across all rows. Since a binary yes/no prompt can change scores substantially, the claim that zero-shot grafting 'surpasses the full-size Llama-70B decoder training on some benchmarks' requires the same prompt protocol for every method compared. Please specify the prompt used for each row, and ideally report both protocols for both methods.
  3. [Tables 2, 4, 6, and 7] All reported VLM scores are from single runs without error bars or significance measures. Some differences that support the central narrative, such as the grafted encoder outperforming the surrogate decoder on LLaVA-Wild or MMBench, may be within run-to-run variability of VLM evaluation. It would strengthen the paper to report at least two or three seeds for the main Llama-70B comparisons in Tables 7-8, or to provide evaluation with multiple prompt variants, before claiming performance parity or superiority.
minor comments (4)
  1. [Figure 5 caption] The word 'cabability' is a typo and should be 'capability'.
  2. [Section A.11] The dynamic loss weight normalization in Section A.11 appears notational: the update w_i ← w_i / ((Σ_j w_j)/(Σ_j L_j)) is dimensionally inconsistent. Please clarify whether the intent is to rescale the average weight to 1, and rewrite the equation accordingly.
  3. [Section 1.1] The transition point is identified visually from KL-trajectory plots (Figure 3). A quantitative criterion, such as a threshold on the KL distance or a change-point detector, would make the method easier to reproduce for a new target LLM.
  4. [Table 7 and Table 8] The column headers in Tables 7 and 8 are extremely compact (e.g., 'MMEbinary MME POPE binary POPE'), and the reader cannot tell at a glance which columns correspond to which benchmark subset and protocol. Expanding the headers and adding a short caption explaining the binary vs. original protocol would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the grafting mechanism is empirically tested against external benchmarks and controlled ablations, with only a minor built-in design transfer.

full rationale

The paper's central derivation—constructing a surrogate by inheriting the target LLM's shallow layers and then training a vision encoder on that surrogate—is not circular. The 'zero-shot grafting' success is an empirical finding evaluated on external VLM benchmarks (MME, POPE, SEED-Bench, MMVet, LLaVA-Wild, MMBen, etc.) and against baselines that include full decoder training. The shared early layers provide an alignment mechanism by construction, but the paper does not stop there: it ablates the design by removing early layers (T(1,11)) and by unfreezing early layers (T(16,26)*), showing that grafting fails in both cases, and it varies the number of preserved early layers (T(6,30), T(12,30), T(17,30)) to validate the transition point. No equation in the paper reduces a predicted quantity to a fitted parameter; the KL trajectory in Eq. (4) is a descriptive analysis, not a fitted prediction. The ~45% cost reduction is a reported measurement of training hours for a specific 20%-data run, not a prediction derived from a fitted quantity; concerns about how the X% subsets were sampled are statistical validity issues rather than circularity. The only self-referential element is the use of GenQA [5], which shares authors, as text instruction data for stage-1 translator training; this is a data choice, not a load-bearing theoretical claim, and the method's core grafting result does not depend on the uniqueness of that dataset. Therefore no circular step rises to the level of the enumerated patterns.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central method rests on a handful of hand-chosen layer indices (transition points), one loss-weighting exponent, and empirical assumptions about layer-phase behavior and translator fidelity; the data-subset assumption behind the cost claim is the least-controlled part.

free parameters (3)
  • Transition point (translator insertion layer) per model = 16 (Llama-3B), 17 (Llama-8B), 40 (Llama-70B), 22 (Qwen3-4B)
    Selected by visual inspection of KL-divergence curves in Figure 3 and Appendix A.4-A.5. This layer index determines how many early layers are preserved and is central to surrogate construction.
  • Dynamic loss weight exponent ord = 0.5
    Set by hand in Appendix A.11 to balance loss contributions from responses of different lengths when fine-tuning small-model decoders; not used in the main 70B cost experiment.
  • Number of encoder layers fine-tuned in stage 2 = 8 (last eight layers)
    Main 70B experiments fine-tune only the last eight CLIP encoder layers; analysis experiments fine-tune the entire encoder. This hand-chosen setting affects the encoder's grafting capability.
assumptions (5)
  • domain assumption The KL-divergence trajectory (Eq. 4) computed under teacher-forced feeding reveals a universal early/late phase transition in LLMs.
    Section 1.1, Figure 3; the transition point is read visually. The paper ablates teacher-forced feeding in A.7, but the phase dichotomy remains an empirical hypothesis.
  • domain assumption A single trained translator layer inserted at the transition point faithfully emulates the target LLM's late-phase computation for both text and image-conditioned inputs.
    Section 1.2 and Figure 4; the encoder is trained against the translator, so grafting success depends on the translator's fidelity to the target's late layers.
  • domain assumption The X% training-data subsets in Table 8 are representative and consistently sampled across methods.
    Section 2.2.1 and Table 8; the paper does not describe how subsets are formed (random, stratified, or prefix) nor report multiple seeds, so the 10% to 20% convergence claim relies on this assumption.
  • domain assumption Benchmark evaluations with modified binary prompts for MME/POPE in zero-shot grafting are valid measures of visual understanding.
    Section 1.2, footnote on binary prompts; the target LLM is not instruction-tuned for VQA, so the protocol is altered, which may affect comparability with standard benchmark scores.
  • standard math Standard transformer and softmax machinery (Eq. 1-4) and pretrained model weights are taken as given.
    Section 1.1; the analysis relies on model logits and layer-wise hidden states.
invented entities (1)
  • Early phase / late phase transition in LLM inference independent evidence
    purpose: Conceptual split of transformer layers used to decide which layers to preserve in the surrogate and where to insert the translator.
    The KL-trajectory measurement (Eq. 4, Figure 3) is a falsifiable, reproducible probe; the paper demonstrates it on Llama, Gemma, and Qwen families, so the phase transition is not an untestable postulate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Zero-Shot Vision Encoder Grafting via LLM Surrogates." pith.science (2026). https://pith.science/paper/4Y6FBZIX

@misc{pith2026250522664,
  author       = {Pith},
  title        = {Pith review of: Zero-Shot Vision Encoder Grafting via LLM Surrogates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4Y6FBZIX}},
  note         = {Machine review of arXiv:2505.22664}
}
read the original abstract

Vision language models (VLMs) typically pair a modestly sized vision encoder with a large language model (LLM), e.g., Llama-70B, making the decoder the primary computational burden during training. To reduce costs, a potential promising strategy is to first train the vision encoder using a small language model before transferring it to the large one. We construct small "surrogate models" that share the same embedding space and representation language as the large target LLM by directly inheriting its shallow layers. Vision encoders trained on the surrogate can then be directly transferred to the larger model, a process we call zero-shot grafting -- when plugged directly into the full-size target LLM, the grafted pair surpasses the encoder-surrogate pair and, on some benchmarks, even performs on par with full decoder training with the target LLM. Furthermore, our surrogate training approach reduces overall VLM training costs by ~45% when using Llama-70B as the decoder. The code is at https://github.com/facebookresearch/zero.

Figures

Figures reproduced from arXiv: 2505.22664 by the authors.

Figure 1
Figure 1. Zero-shot vision encoder grafting via a small language surrogate (srgt) model to trigger the target LLM to perform visual understanding task without any additional training. Abstract Vision language models (VLMs) typically pair a modestly sized vision encoder with a large language model (LLM), e.g., Llama-70B, making the decoder the primary computa￾tional burden during training. To reduce costs, a potential promisin… view at source ↗
Figure 3
Figure 3. The trajectory of prediction across different layers of Llama-3B, 8B, and 70B, and Gemma-2B from a different model family. The arrow marks the transition point where the trajectories of 300 random samples converge. process information from individual token embeddings and combine simple representations together to form higher or￾der concepts, then layers in the late phase converge towards a specific next-token predic… view at source ↗
Figure 4
Figure 4. Replacing layers with a translator. Despite the rel￾ative size in the illustration, our translator is simply an identical transformer layer inherited from the target LLM. The translator bypasses many network layers, and is initialized from the shallow￾est original layer that it replaced. 1.2. Studying the Transition Phases To test our hypothesis on the transition point, we exper￾iment with Llama-3B5 by replacing con… view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Qualitative results on zero-shot grafting capability of encoders trained with small models for Llama-3B. For comparison, we also include responses from the encoder trained with Llama-3B and the fine-tuned Llama-3B. The encoder trained on T (16, 26) achieves strong zero…
Figure 6
Figure 6. Figure 6: Qualitative results on zero-shot grafting capability of encoders trained with surrogate models for Llama-8B. For comparison, we also include responses from the encoder trained with Llama-8B and the fine-tuned Llama-8B. More early-phase layers preserved lead to stronger…
Figure 7
Figure 7. Figure 7: Qualitative results on strong zero-shot grafting ability of surrogate-trained encoder for Llama-70B, which produces fine￾grained image features to trigger Llama-70B to perform complex visual understanding tasks. Response is sampled with greedy decoding. periments demon…
Figure 8
Figure 8. Figure 8: Qualitative OCR results on strong zero-shot grafting ability of surrogate-trained encoder for Llama-70B. The input im￾age size is 3362 . small model mmlu hellaswag arceasy arcchallenge winogrande piqa boolq openbookqa Llama-70B 82.6 86.9 83.4 71.2 85.4 83.7 89.1 47.6 s…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 39 canonical work pages

  1. [1]

    Understanding Inter- mediate Layers Using Linear Classifier Probes

    Guillaume Alain and Yoshua Bengio. Understanding Inter- mediate Layers Using Linear Classifier Probes. In ICLR,

    work page

  2. [2]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a Visual Language Model for Few-Shot Learning. InNeurIPS,

    work page

  3. [3]

    Eliciting Latent Predictions From Transform- ers With the Tuned Lens

    Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting Latent Predictions From Transform- ers With the Tuned Lens. arXiv preprint arXiv:2303.08112,

    arXiv

  4. [4]

    PIQA: Reasoning about Physical Commonsense in Natural Language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about Physical Commonsense in Natural Language. In AAAI, 2020. 11

    work page 2020

  5. [5]

    GenQA: Generating Millions of Instructions from a Handful of Prompts

    Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Goldstein. GenQA: Generating Millions of Instructions from a Handful of Prompts. arXiv preprint arXiv:2406.10323, 2024. 2, 3, 12

    arXiv 2024

  6. [6]

    InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. In CVPR, 2024. 7, 8, 13

    work page 2024

  7. [7]

    BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In ACL Anthology, 2019. 11

    work page 2019

  8. [8]

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv preprint arXiv:1803.05457,

    arXiv

Show all 53 references
  1. [9]

    The Llama 3 Herd of Models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783,

    arXiv

  2. [10]

    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394, 2023. 3, 11

    2023 arXiv

  3. [11]

    A Framework for Few-Shot Language Model Evaluation, 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Gold- ing, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle Mc- Donell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Linta...

    2024

  4. [12]

    The Unreasonable Ineffectiveness of the Deeper Layers

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The Unreasonable Ineffectiveness of the Deeper Layers. In ICLR, 2025. 8

    2025

  5. [13]

    VizWiz Grand Challenge: Answering Visual Questions from Blind People

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. VizWiz Grand Challenge: Answering Visual Questions from Blind People. In CVPR, 2018. 11

    2018

  6. [14]

    Word Embed- dings Are Steers for Language Models

    Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. Word Embed- dings Are Steers for Language Models. In ACL Anthology,

  7. [15]

    Measur- ing Massive Multitask Language Understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measur- ing Massive Multitask Language Understanding. In ICLR,

  8. [16]

    LoRA: Low-Rank Adaptation of Large Language Models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2022. 8, 9

    2022

  9. [17]

    GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering

    Drew A Hudson and Christopher D Manning. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In CVPR, 2019. 11

    2019

  10. [18]

    InternVL2: Better than the Best—Expanding Performance Boundaries of Open-Source Multimodal Mod- els with the Progressive Scaling Strategy

    InternVL2. InternVL2: Better than the Best—Expanding Performance Boundaries of Open-Source Multimodal Mod- els with the Progressive Scaling Strategy. https : / / internvl.github.io/blog/2024- 07- 02- InternVL- 2.0/, 2024. 8

    2024

  11. [19]

    A Diagram Is Worth a Dozen Images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A Diagram Is Worth a Dozen Images. In ECCV, 2016. 11

    2016

  12. [20]

    Propulsion: Steering LLM with Tiny Fine-Tuning

    Md Kowsher, Nusrat Jahan Prottasha, and Prakash Bhat. Propulsion: Steering LLM with Tiny Fine-Tuning. In ACL Anthology, 2024. 8

    2024

  13. [21]

    SEED-Bench: Benchmarking Multi- modal LLMs with Generative Comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. SEED-Bench: Benchmarking Multi- modal LLMs with Generative Comprehension. In CVPR,

  14. [22]

    LLaV A-Next: Stronger Llms Supercharge Multimodal Capabilities in the Wild

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. LLaV A-Next: Stronger Llms Supercharge Multimodal Capabilities in the Wild. https://llava- vl.github. io/blog/2024-05-10-llava-next-stronger-llms/ ,

    2024

  15. [23]

    LMMs-Eval: Accelerating the Development of Large Multimodal Models

    Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. LMMs-Eval: Accelerating the Development of Large Multimodal Models. https: //github.com/EvolvingLMMs-Lab/lmms- eval , 2024. 11

    2024

  16. [24]

    LLaV A-OneVision: Easy Visual Task Trans- fer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Zi- wei Liu, et al. LLaV A-OneVision: Easy Visual Task Trans- fer. arXiv preprint arXiv:2408.03326, 2024. 7, 8

    2024 arXiv

  17. [25]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In ICML, 2023. 8

    2023

  18. [26]

    Evaluating Object Hallucination in Large Vision-Language Models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating Object Hallucination in Large Vision-Language Models. In EMNLP, 2023. 3, 11

    2023

  19. [27]

    Improved Baselines with Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved Baselines with Visual Instruction Tuning. In CVPR, 2024. 3, 7, 11, 13

    2024

  20. [28]

    LLaV A-Next: Im- proved Reasoning, Ocr, and World Knowledge

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaV A-Next: Im- proved Reasoning, Ocr, and World Knowledge. https: //llava- vl.github.io/blog/2024- 01- 30- llava- next/, 2024. 8

    2024

  21. [29]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurIPS, 2024. 11, 13

    2024

  22. [30]

    MMBench: Is Your Multi-modal Model an All-around Player? In ECCV, 2025

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. MMBench: Is Your Multi-modal Model an All-around Player? In ECCV, 2025. 11

    2025

  23. [31]

    Chartqa: A Benchmark for Question Answering About Charts With Visual and Logical Reason- ing

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A Benchmark for Question Answering About Charts With Visual and Logical Reason- ing. In ACL Findings, 2022. 11

    2022

  24. [32]

    Docvqa: A Dataset for VQA on Document Images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A Dataset for VQA on Document Images. In WACV, 2021. 11

    2021

  25. [33]

    Infograph- icVQA

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infograph- icVQA. In WACV, 2022. 11

    2022

  26. [34]

    ShortGPT: Layers in Large Language Models are More Re- dundant Than You Expect.arXiv preprint arXiv:2403.03853,

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT: Layers in Large Language Models are More Re- dundant Than You Expect.arXiv preprint arXiv:2403.03853,

    arXiv

  27. [35]

    Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sab- harwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In ACL An- thology, 2018. 11

    2018

  28. [36]

    interpreting GPT: the logit lens

    nostalgebraist. interpreting GPT: the logit lens. https: //www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens , 2020. 8

    2020

  29. [37]

    Learning Transferable Visual Models From Natural Language Super- vision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models From Natural Language Super- vision. In ICML, 2021. 1

    2021

  30. [38]

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. In ACM Communications, 2021. 11

    2021

  31. [39]

    Open Problems in Mechanistic Interpretability

    Lee Sharkey, Bilal Chughtai, Joshua Batson, Jack Lindsey, Jeff Wu, Lucius Bushnaq, Nicholas Goldowsky-Dill, Stefan Heimersheim, Alejandro Ortega, Joseph Bloom, et al. Open Problems in Mechanistic Interpretability. arXiv preprint arXiv:2501.16496, 2025. 8

    2025 arXiv

  32. [40]

    Towards VQA Models That Can Read

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards VQA Models That Can Read. In CVPR, 2019. 11

    2019

  33. [41]

    Transformer Layers as Painters

    Qi Sun, Marc Pickett, Aakash Kumar Nain, and Llion Jones. Transformer Layers as Painters. arXiv preprint arXiv:2407.09298, 2024. 8

    2024 arXiv

  34. [42]

    Gemma Team. Gemma-2. Kaggle, 2024. 12

    2024

  35. [43]

    Gemma Team. Gemma-3. Kaggle, 2025. 9

    2025

  36. [44]

    Qwen2.5 Technical Report

    Qwen Team. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115, 2025. 10

    2025 arXiv

  37. [45]

    Qwen3 Technical Report

    Qwen Team. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388, 2025. 10

    2025 arXiv

  38. [46]

    Cambrian- 1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. In NeurIPS, 2025. 8, 11

    2025

  39. [47]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localiza- tion, and Dense Features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muham- mad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localiza- tion, ...

    arXiv

  40. [48]

    Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution. arXiv preprint arXiv:2409.12191, 2024. 8

    2024 arXiv

  41. [49]

    CogVLM: Visual Expert for Pretrained Lan- guage Models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. CogVLM: Visual Expert for Pretrained Lan- guage Models. In NeurIPS, 2024. 8

    2024

  42. [50]

    MM-Vet: Evaluating Large Multimodal Models for Inte- grated Capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. MM-Vet: Evaluating Large Multimodal Models for Inte- grated Capabilities. In ICML, 2024. 11

    2024

  43. [51]

    HellaSwag: Can a Machine Really Finish Your Sentence? In ACL Anthology, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In ACL Anthology, 2019. 11

    2019

  44. [52]

    Sigmoid Loss for Language Image Pre- Training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Training. In ICCV, 2023. 1

    2023

  45. [53]

    PyTorch FSDP: Experi- ences on Scaling Fully Sharded Data Parallel

    Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien- Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. PyTorch FSDP: Experi- ences on Scaling Fully Sharded Data Parallel. arXiv preprint arXiv:2304.11277, 2023. 11

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.