Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Diffusion Instruction Tuning

T0 review · 3 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a vision-language model to copy Stable Diffusion's per-word attention maps during supervised fine-tuning boosts its accuracy on 19 of 20 benchmarks, with up to 30% relative gains and a 68% improvement on out-of-distribution…

desk verdict Novel attention-alignment SFT method with unusually broad gains, but the diffusion-specific mechanism is underdetermined for want of a teacher-control ablation. read the letter →

arxiv 2502.06814 v2 pith:BCX2TBY2 submitted 2025-02-04 cs.LG cs.AIcs.GR

Chen Jin , Ryutaro Tanno , Amrutha Saseendran , Tom Diethe , Philip Teare This is my paper · ORCID
classification cs.LGcs.AIcs.GR
keywords Diffusioninstructiontuningattentionalignmentvision-languagemodelssupervisedfine-tuningStablevisualgroundingout-of-distributiongeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Lavender, a supervised fine-tuning method that adds an attention-alignment loss to the standard next-token prediction loss while adapting a vision-language model (VLM): the VLM's text-vision attention maps are pushed toward the per-word attention maps of a frozen Stable Diffusion model. The premise is that diffusion models, because they must reconstruct images at the pixel level, learn more concentrated word-to-region attention than VLMs trained purely to predict text tokens, and that this concentration is closer to an ideal attention distribution for visual question answering. On Llama-3.2-11B the method improves performance on 19 of 20 benchmarks relative to autoregressive fine-tuning, with up to 30% relative gains, and a 68% relative gain on the out-of-distribution WorldMedQA-V medical benchmark; it also works, with smaller gains, on the self-attention-only MiniCPM-V2.5. All of this is achieved with 0.13 million training examples, about 2.5% of typical large-scale SFT datasets, on 8 GPUs in a day. If the claim holds, it means the visual expertise of image generators can be transferred into language models without new annotations, offering a data-efficient route to grounded, generalizing vision-language systems.

What carries the argument

The load-bearing object is the attention-alignment loss $L_{\text{att}}(\theta) = \sum_i \|\text{Aligner}(p_{\text{VLM}}(a|x^{(i)},y^{(i)};\theta)) - p_{\text{DM}}(a|x^{(i)},y^{(i)};\theta_D)\|^2$, added with weight $\lambda$ to the standard next-token loss. Three components make it work: (1) per-word attention targets extracted offline from frozen Stable Diffusion v1.4 via a short DDIM inversion (5 inversion steps, 10 diffusion steps, 20 seconds per image), with attention averaged across layers and denoising steps following the well-known cross-attention-control recipe; (2) the Aligner network, a small 3-layer ConvNet inspired by Squeeze-and-Excitation blocks, which converts the VLM's multi-head, multi-layer attention into a single-channel saliency map at a fixed 32x32 resolution so it can be compared to the diffusion target; and (3) attention aggregation in the VLM, where the paper compares mean/max pooling, attention flow, and a learned 'parallel attention' scheme that adds new $Q_d, K_d$ projections in one fifth of the layers, finding learned aggregation most scalable. The alignment loss is derived as a Gaussian likelihood term in a Bayesian posterior over VLM parameters, treating the diffusion attention maps as observations.

What would settle it

Run Lavender with three attention targets on the same VLM and data: (a) genuine Stable Diffusion maps, (b) random maps reshaped to the same entropy, and (c) the VLM's own maps sharpened by temperature. If (b) or (c) reproduce (a)'s gains, the specific semantic content of diffusion attention is not what drives the improvement. Alternatively, an out-of-distribution task where Stable Diffusion attention is known to mislocalize, such as dense text or unusual objects, should show Lavender underperforming plain autoregressive fine-tuning if the core claim is wrong.

Watch

Extended reading notes

Core claim

Lavender's central claim is that the cross-attention maps of a text-to-image diffusion model are closer than a VLM's own attention maps to the ideal posterior attention distribution $p^*(a|x,y)$ for vision-centric reasoning, and that pulling the VLM toward them during fine-tuning improves task performance. Formally, the paper defines the difference $\delta^{(i)}(\theta) = p_{\text{VLM}}(a|x^{(i)},y^{(i)};\theta) - p_{\text{DM}}(a|x^{(i)},y^{(i)};\theta_D)$, models it as Gaussian noise, and derives the total objective $L_{\text{total}}(\theta) = L_{\text{VLM}}(\theta) + \lambda L_{\text{att}}(\theta)$ with $L_{\text{att}} = \sum_i \|\delta^{(i)}(\theta)\|^2$, where a lightweight Aligner network (a 3-layer ConvNet) projects the VLM's aggregated attention into a single-channel map comparable to the diffusion model's per-word attention. The empirical payload is that this alignment regularizer beats plain autoregressive SFT on 19 of 20 benchmarks for Llama-3.2-11B (LoRA variant), up to 30% relative improvement, and lifts accuracy on the severely out-of-distribution WorldMedQA-V medical QA benchmark by 68%, narrowing the gap to closed-source models from 43% to 10%. The paper further reports that learned attention aggregation outperforms manual pooling or attention flow, that aligning all eight cross-attention layers works best, and that pretraining the Aligner before joint fine-tuning prevents catastrophic forgetting on small datasets.

Load-bearing premise

The paper assumes that Stable Diffusion's attention maps are closer to the ideal attention for a VLM answering questions than the VLM's own attention maps, and that forcing the VLM to copy them improves answers; the support is lower attention entropy and qualitative visualizations, not a causal test that diffusion attention carries the task-relevant semantics.

Editorial extensions

If this is right

  • If Lavender's claim is right, vision-language models can be improved by an attention-level transfer of 'visual expertise' from frozen image generators, with no new annotations beyond the image-text pairs already used for SFT.
  • The dependency on large SFT datasets weakens: 0.13M examples (2.5% of typical scales) suffice for up to 30% relative gains, making the method accessible to groups with modest compute (8 GPUs, one day of training).
  • Attention alignment improves out-of-distribution robustness: the 68% gain on WorldMedQA-V medical QA suggests that grounding-based fine-tuning generalizes better than purely text-loss fine-tuning when data shifts.
  • The method is architecture-agnostic: it works on cross-attention VLMs (OpenFlamingo, Llama-3.2-11B) and, with smaller gains, on self-attention-only models (MiniCPM-V2.5), and the same loss is proposed as a drop-in 'diffusion feedback' for RL post-training.
  • Lavender scales better with data and mitigates overfitting relative to autoregressive SFT on small datasets, which would make it a safer default for fine-tuning in low-data regimes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own evidence for the core assumption is correlational (lower attention entropy, qualitative maps). A sharper test, which the authors do not run, would be to align the VLM to random or deliberately mislocalized peaked attention maps: if gains persist, the benefit is attention sharpening rather than the semantic content of diffusion attention.
  • The method's stated compatibility with RL post-training, replacing human visual feedback with 'diffusion feedback', is proposed but not demonstrated; a natural extension would be to use the alignment loss as a reward model or KL-type regularizer in preference optimization.
  • If the attention-transfer principle generalizes, it could cross modalities: text-to-audio or text-to-structure generative models (e.g., protein diffusion) could supply attention targets for audio-language or structure-language models, an application the paper mentions but does not test.
  • The OCRVQA mixing results in the paper's scaling study show that adding one dataset can reduce overall performance, suggesting the choice of attention targets matters per domain; testing Lavender on text-heavy images where Stable Diffusion attention is known to be unreliable would map its failure boundary.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Lavender, a supervised fine-tuning method that augments the standard next-token prediction loss with an auxiliary loss aligning the VLM's text-visual attention maps to those of a frozen Stable Diffusion model. The method is evaluated on three VLMs (OpenFlamingo, MiniCPM-V-2.5, Llama-3.2-11B) across 20 benchmarks, with ablations of attention aggregation, Aligner design, layer choice, and scaling behavior. The headline results are consistent gains on Llama-3.2-11B (claimed 19/20 benchmarks, up to 30%) and a 68% relative improvement on the out-of-distribution WorldMedQA-V medical benchmark.

Significance. If the central claim—that diffusion attention is a superior alignment target because it is closer to an ideal task-relevant attention distribution—is substantiated, the method would be a valuable data-efficient transfer technique for VLM fine-tuning. The paper's strengths include an unusually broad evaluation suite (20 benchmarks, 3 models, 23 baselines), extensive ablations of aggregation and layer-selection choices, a dedicated OOD medical QA test, and a commitment to release code, data, and models. However, the current evidence for the diffusion-specific mechanism is correlational (entropy, visualizations) and the paper lacks the key control experiment that would separate diffusion-specific attention quality from generic regularization or added capacity.

major comments (3)
  1. [4.3, 7.2, 8] Every Lavender variant uses Stable Diffusion attention as the target; there is no condition that replaces it with an alternative teacher (e.g., the VLM's own attention, a sharpened version of it, CLIP/DINO attention, or a randomly initialized attention map). Such a control is needed to attribute the observed gains to the diffusion model's 'visual expertise' rather than to the auxiliary loss acting as a regularizer, the added parallel-attention/Aligner capacity, or any entropy-reducing target. Without this control, the central mechanistic claim of the paper is not supported.
  2. [3.2, 7.1, Appendix B] The argument that lower attention entropy implies closer proximity to an ideal distribution p* is incomplete. Appendix B explicitly assumes that the cross-entropy terms between p_DM and p* and p_VLM and p* are approximately equal (footnote 1), but this assumption is untested and cannot be inferred from lower entropy; indeed, a more concentrated distribution can have arbitrarily large cross-entropy with p* if it concentrates on the wrong regions. The paper should provide a direct test of the KL-divergence claim or at least a sensitivity analysis of the cross-entropy assumption.
  3. [7.2, Table 1] The text states that Lavender 'outperforms autoregressive fine-tuning by up to 30% on 19 out of 20 benchmarks with LoRA' (Section 7.2). However, Table 1 shows that on CCBench, Lavender LoRA scores 39.2 versus 39.8 for Autoregressive LoRA—a decrease. The claim appears to be an 18/20 comparison against the autoregressive baseline; the '19/20' figure seems to refer to improvement over the base Llama-3.2 model. Please correct this inconsistency, which affects a headline claim.
minor comments (5)
  1. [Figure 3] The caption contains two typos: 'ridding' should be 'riding' in both 'flamingo ridding bicycle' and 'bear ridding bicycle'.
  2. [Table 2 caption] The caption cites WorldMedQA-V as (Duan et al., 2024), but the reference list also contains Matos et al. (2024) for the same dataset; please disambiguate the correct citation.
  3. [Section 5] DM attention extraction uses a shortened inversion process with 5 inversion steps and 10 diffusion steps, but the paper does not report a sensitivity analysis of these step counts on the quality of the attention target, even though the limitation section acknowledges potential inaccuracy for infrequent words.
  4. [Section 3.3 and Algorithm 1] The hyperparameter λ is introduced in the total loss but no sensitivity analysis is reported; given that it is a free parameter, a sweep or at least a discussion of its effect would strengthen the practical guidance.
  5. [Table 1] The main results table does not report standard deviations or significance tests, despite error bars appearing in Figure 1; please indicate how many seeds and whether the reported gains are statistically stable.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: Lavender's improvements are measured on external benchmarks, and the auxiliary attention-alignment loss is not forced by construction.

full rationale

Lavender's central claim is an empirical one: adding an MSE attention-alignment loss to standard supervised fine-tuning improves measured performance on 20 external benchmarks, including the out-of-distribution WorldMedQA-V set. The reported gains are not forced by construction: the diffusion attention target A_DM is precomputed once from a frozen Stable Diffusion model and held fixed, while the VLM parameters and Aligner are optimized; nothing in the loss definition constrains the benchmark scores, and the same supervised data with only L_VLM is the baseline against which Lavender is measured. The Bayesian derivation in Section 3.3 is a standard MAP/regularization interpretation of the auxiliary loss, not a derivation of downstream accuracy; it only re-expresses L_att as a negative log-likelihood under a Gaussian assumption, and Appendix C explicitly treats lambda as a tunable hyperparameter. The assumption that DM attention is closer to an ideal p* is stated as a hypothesis and supported by entropy comparisons and visualizations; even if that assumption were false, the benchmark gains would remain an empirical finding rather than a tautology. The one self-citation (Jin et al., 2023) is used for the shortened inversion technique that extracts DM attention maps and is itself acknowledged in the limitations as potentially limiting accuracy for infrequent words; it is not invoked as a uniqueness theorem and does not carry the argument. The skeptical observation that no teacher-ablation was run (e.g., replacing SD targets with CLIP/DINO, self-distillation, or temperature-sharpened VLM attention) is a legitimate causal-attribution concern about whether the specific diffusion teacher explains the gains, but it is an experimental design limitation, not a circular derivation. No step in the paper equates an input with an output by definition, renames a fitted quantity as a prediction, or imports a load-bearing result from the authors' prior work.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. Its core relies on the empirical assumption that diffusion-model attention is a better target for VLM attention, plus standard probabilistic and architectural assumptions. The Aligner network is a learned component, not a newly postulated entity. The main free parameter is the regularization weight lambda, which is not specified in the paper.

free parameters (2)
  • lambda (attention alignment weight) = not disclosed in the main text
    The hyperparameter balancing the task loss and the attention alignment loss (Equation 10). Its value is chosen by hand/validation, and the paper does not report the specific value used in the main experiments.
  • selection of layers to align = all 8 cross-attention layers in Llama-3.2-11B
    The paper chooses to align all cross-attention layers based on an ablation study (Section 8), but this is a design choice that affects the method's behavior.
assumptions (3)
  • domain assumption Stable Diffusion's attention distribution is closer to an ideal attention distribution for VLM tasks than the VLM's own attention distribution.
    This is the central premise of Lavender (Section 3.2). It is supported only by entropy comparisons (Figure 7) and qualitative visualizations, not by a rigorous proof or causal test.
  • standard math The difference between VLM and DM attention maps follows a Gaussian distribution with equal variance, justifying the MSE loss.
    Used in the Bayesian derivation (Section 3.3, Appendix C). This is a standard simplifying assumption, not independently validated for attention maps.
  • domain assumption A small Aligner network can project the VLM's attention maps into a space comparable to the DM's attention maps without losing critical information.
    The whole alignment loss depends on this comparability (Section 4.3). The paper does not formally analyze the expressive power or information loss of the Aligner.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diffusion Instruction Tuning." pith.science (2026). https://pith.science/paper/BCX2TBY2

@misc{pith2026250206814,
  author       = {Pith},
  title        = {Pith review of: Diffusion Instruction Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BCX2TBY2}},
  note         = {Machine review of arXiv:2502.06814}
}
read the original abstract

We introduce Lavender, a simple supervised fine-tuning (SFT) method that boosts the performance of advanced vision-language models (VLMs) by leveraging state-of-the-art image generation models such as Stable Diffusion. Specifically, Lavender aligns the text-vision attention in the VLM transformer with the equivalent used by Stable Diffusion during SFT, instead of adapting separate encoders. This alignment enriches the model's visual understanding and significantly boosts performance across in- and out-of-distribution tasks. Lavender requires just 0.13 million training examples, 2.5% of typical large-scale SFT datasets, and fine-tunes on standard hardware (8 GPUs) in a single day. It consistently improves state-of-the-art open-source multimodal LLMs (e.g., Llama-3.2-11B, MiniCPM-Llama3-v2.5), achieving up to 30% gains and a 68% boost on challenging out-of-distribution medical QA tasks. By efficiently transferring the visual expertise of image generators with minimal supervision, Lavender offers a scalable solution for more accurate vision-language systems. All code, training data, and models will be shared at https://astrazeneca.github.io/vlm/.

Figures

Figures reproduced from arXiv: 2502.06814 by the authors.

Figure 1
Figure 1. Average Performance on 20 Vision-Language Reasoning Benchmarks (Grouped into 4 Categories). Abstract We introduce Lavender, a simple supervised fine￾tuning (SFT) method that boosts the performance of advanced vision-language models (VLMs) by leveraging state-of-the-art image generation mod￾els such as Stable Diffusion. Specifically, Laven￾der aligns the text-vision attention in the VLM transformer with the equivalen… view at source ↗
Figure 2
Figure 2. Lavender: Diffusion Instruction Tuning. Lavender uses the text-vision attention maps of a Stable Diffusion Model, AttentionSDM, as a guiding objective for the attention of the target vision-language model (VLM), AttentionV LM. The Atten￾tion Alignment module employs a 3-Layer ConvNet to transform AttentionV LM to match AttentionSDM via an MSE loss, act￾ing as a regularisation term during supervised fine-tuning. 1. I… view at source ↗
Figure 3
Figure 3. Image generation models (Stable Diffusion on the left) exhibit stronger word-to-region attention alignment than VLMs (Open-Flamingo on the right). Per-word average attention maps suggest that diffusion models may be closer to an ideal distribution correlating image regions with textual tokens. Ablation studies reveal that the method of attention aggre￾gation and the choice of layers for fine-tuning are critical to p… view at source ↗
Figures from the paper (23 more)
Figure 4
Figure 4. Figure 4: Sketch of Diffusion Instruction Tuning (left) and a short pseudo code (right), whose full version is available in Appendix A. Algorithm 1 Diffusion Instruction Tuning Require: D = {(x (i) , y(i) )}, θD, θ, scale λ Ensure: Fine-tuned VLM parameters θ Stage 1: Precompute…
Figure 5
Figure 5. Figure 5: Illustration of attention aggregation in VLMs. Atten￾tion weights between text tokens and image patches are aggregated to form per-word saliency maps. 4.2.1. SIMPLE AGGREGATION FUNCTIONS A straightforward approach is to pool attention weights A (i.e., w hl (t,p) ) acro…
Figure 6
Figure 6. Figure 6: Learning to aggregate with parallel attention. The demonstration is based on a self-attention VLM. The parallel at￾tention constitutes about 1/5 th of the total VLM layers (L ′ ∈ L). (WQ, WK). This enables us to capture richer semantic correlations without overwriting …
Figure 7
Figure 7. Figure 7: DM attention is more concentrated than VLM at￾tention, aligning closer to the optimal posterior distribution p ∗ (a|x, y). Measured across OpenFlamingo, MiniCPM-v2.5, and Llama 3.2-11B. Full results in Appendix F.1. Lavender + 'root match' + MLP + LR 1e-5 Lavender + 'i…
Figure 8
Figure 8. Figure 8: Aligning VLM attention with DM for “guitar.” Each row adds a training technique. More examples in Appendix F.1. shown in [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 11
Figure 11. Figure 11: Lavender demonstrates comparable performance to certain High-Resource State-of-the-Art Models. 7.3. Data Overlapping Analysis Qualitative Analysis. A key question arising from the above benchmark results is how to interpret the observed ab￾solute numbers. In this sect…
Figure 12
Figure 12. Figure 12: Lavender improves MiniCPM-V-2.5 and Llama-3.2-11B, surpassing Small Budget-Constrained SOTA by up to 50%. This plot highlights the key zero-shot accuracy results across 16 VLM benchmarks from [PITH_FULL_IMAGE:figures/full_fig_p010_12.png]
Figure 13
Figure 13. Figure 13: Lavender exhibits minimal data overlap during fine￾tuning, supporting its generalisability. Overlap scores are as￾signed as follows: 3 for explicitly shared benchmark datasets, 1 for datasets with shared sources (e.g., COCO) or web-crawled im￾ages, and 0 for no overla…
Figure 14
Figure 14. Figure 14: Lavender scales better and mitigates overfitting com￾pared to autoregressive fine-tuning, with larger datasets reduc￾ing variability. This plot shows the mean normalised performance across eight benchmarks for two dataset configurations after fine￾tuning Llama 3.2-11B…
Figure 15
Figure 15. Figure 15: Relative Improvement Over Baseline on 20 Benchmarks. (a) Results for MiniCPM-Llama3-V-2.5 fine-tuned on its original RV83k dataset. Lavender improves performance on 16/20 benchmarks with gains up to 4%, while limiting performance drops to -1%, primarily on an out-of-d…
Figure 16
Figure 16. Figure 16: Lavender boosts Llama-3.2-11B’s performance on the OOD WorldMedQA benchmark by 68%. Results are based on fine-tuning Llama-3.2-11B-Vision-Instruct on RV83k, Flk30k, and OV30k using autoregressive or Lavender methods with LoRA. Accuracy reflects average performance acr…
Figure 17
Figure 17. Figure 17: The per-word VLM attention maps are aligned to the Stable Diffusion (SD) after tuning with Lavender. Results are from Lavender-Llama 3.2-11B implementation. More results are available in Appendix [PITH_FULL_IMAGE:figures/full_fig_p013_17.png]
Figure 18
Figure 18. Figure 18: Examples demonstrating Lavender’s enhanced fine-granularity vision alignment leading to improved accuracy on various VQA benchmarks. In the DocVQA example on the top-left, with a question about the title of a table, Llama-3.2 mistakenly extracts the section/page title…
Figure 20
Figure 20. Figure 20: Mean performance across eight benchmarks over training epochs with various aggregation functions. The results are derived from Lavender-Llama 3.2-11B, fully fine-tuned on the Flickr-1k subset [PITH_FULL_IMAGE:figures/full_fig_p015_20.png]
Figure 22
Figure 22. Figure 22: Aligning all eight cross-attention layers in Llama-3.2 with Lavender is most effective. This ablation study examines the impact of aligning subsets of the eight cross-attention layers {8, 13, 18, 23, 28, 33, 38} in Llama-3.2, denoted as ‘First 3/8,’ ‘Mid 4/8,’ ‘Last 4…
Figure 23
Figure 23. Figure 23: Attention reconstruction under the tiling and tokenization procedure in Llama-3.2, Example highlighting the importance of proper reconstruction. Improper rearrangement disrupts spatial continuity, while correct reconstruction preserves semantic alignment. [Back to Con…
Figure 24
Figure 24. Figure 24: Appropriate rearrangement and reconstruction are crucial. Results are based on attention maps extracted from Llama-3.2 (without Lavender fine-tuning) on OCRVQA samples. Poor rearrangement disrupts semantic alignment, while proper reconstruction corrects the spatial ar…
Figure 25
Figure 25. Figure 25: Attention map entropy histograms from three models (OpenFlamingo, MiniCPM-v2.5, and Llama 3.2-11B) are generated by processing a small subset of Flickr30k, RLAIF-V83k, and OCRVQA30k, totalling approximately 10k samples. [Back to Contents] F. Extended Supplementary Res…
Figure 26
Figure 26. Figure 26: More visual verification of learning VLM attention aggregation compared to SD attention for the matched word ’man’ and ’arms’, based on the OpenFlamingo implementation of our method. The first row shows the plain version of our method, and in each subsequent row, we a…
Figure 27
Figure 27. Figure 27: Lavender scales better and reduces overfitting compared to autoregressive fine-tuning, with larger datasets lowering variations. The plot shows the mean normalised performance of four dataset configurations across eight benchmarks after LoRA and Full fine-tuning of Ll…
Figure 28
Figure 28. Figure 28: Scaling Behaviour Across Eight Benchmarks. Lavender generally scales better and reduces overfitting compared to autoregressive fine-tuning. Larger mixed datasets further reduce overfitting and variation. Results are based on LoRA fine-tuning of Llama 3.2-11B with both…
Figure 29
Figure 29. Figure 29: Impact of aggregation functions on tuning iterations (Flickr-1k subset) across eight benchmarks. The results are derived from Lavender-Llama 3.2-11B, fully fine-tuned on the Flickr-1k subset and evaluated using ‘exact match’ without an LLM judge. The averaged result i…
Figure 30
Figure 30. Figure 30: Visually aligned examples from Lavender-Llama 3.2-11B. Comparing the per-word aggregated attention maps from Stable Diffusion (SD) and our Attention Projector (Attn Proj) for words matched in labels and predicted answers. [Back to Contents] 38 [PITH_FULL_IMAGE:figure…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 7 canonical work pages

  1. [1]

    and Zuidema, W

    Abnar, S. and Zuidema, W. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928,

    arXiv 2005

  2. [3]

    Tasks with structured data (e.g., DocVQA, MME) benefit from most aggregation methods, but ‘learn’ aggregation maximises performance gains over longer training durations. F.4. Extra Visual Results[Back to Contents] In this section, we present additional results with Lavender-Llama-3.2-11B across various benchmarks: Table 2 provides example results from the...

    work page 2024

  3. [4]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024a

    Chen, L., Li, J., Dong, X., Zhang, P., Zang, Y ., Chen, Z., Duan, H., Wang, J., Qiao, Y ., Lin, D., et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024a. Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., and Lin, D. Sharegpt4v: Improving large multi-modal models with better cap...

    arXiv

  4. [6]

    Locality align- ment improves vision-language models

    Covert, I., Sun, T., Zou, J., and Hashimoto, T. Locality align- ment improves vision-language models. arXiv preprint arXiv:2410.11087,

    arXiv

  5. [7]

    URL https://arxiv.org/abs/ 2305.06500. DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., Dai, D., Guo, D., Yang, D., Chen, D., Ji, D., Li, E., Lin, F., Dai, F., Luo, F., Hao, G., Chen, G., Li, G., Zhang, H., Bao, H., Xu, H., Wang, H., Zhang, H., Ding, H., Xin, H., Gao, H., Li, H., Qu, H., Cai, J. L...

    arXiv

  6. [8]

    Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y ., Park, J

    URL https://arxiv.org/abs/2412.19437. Deitke, M., Clark, C., Lee, S., Tripathi, R., Yang, Y ., Park, J. S., Salehi, M., Muennighoff, N., Lo, K., Soldaini, L., et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146,

    arXiv

  7. [9]

    An image is worth 16x16 words: Trans- formers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    arXiv 2010

  8. [10]

    The llama 3 herd of models

    Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

Show all 39 references
  1. [11]

    MME: A com- prehensive evaluation benchmark for multimodal large language models

    Fu, C., Chen, P., Shen, Y ., Qin, Y ., Zhang, M., Lin, X., Qiu, Z., Lin, W., Yang, J., Zheng, X., et al. MME: A com- prehensive evaluation benchmark for multimodal large language models. arXiv:2306.13394,

    arXiv

  2. [12]

    Llama-adapter v2: Parameter-efficient visual instruction model

    Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., et al. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010,

    arXiv

  3. [15]

    An image is worth multiple words: Learning object level concepts using multi-concept prompt learning

    Jin, C., Tanno, R., Saseendran, A., Diethe, T., and Teare, P. An image is worth multiple words: Learning object level concepts using multi-concept prompt learning. arXiv preprint arXiv:2310.12274,

    arXiv

  4. [16]

    Yes I’m not able to provide a name for the person in this picture

    [Back to Contents] 36 Diffusion Instruction Tuning index category question answer prediction extracted score 1356 landmark Is this a photo of Serbian Ortho- dox Cathedral in Sarajevo? Please answer yes or no. Yes I’m not able to provide a name for the person in this picture. U...

  5. [17]

    A diagram is worth a dozen images

    Kembhavi, A., Salvato, M., Kolve, E., Seo, M., Hajishirzi, H., and Farhadi, A. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pp. 235–251. Springer,

    2016

  6. [18]

    Seed-Bench: Benchmarking multimodal llms with generative comprehension

    Li, B., Wang, R., Wang, G., Ge, Y ., Ge, Y ., and Shan, Y . Seed-Bench: Benchmarking multimodal llms with generative comprehension. arXiv:2307.16125, 2023a. Li, B., Zhang, Y ., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Zhang, P., Li, Y ., Liu, Z., et al. Llava- onevisi...

    arXiv

  7. [20]

    HallusionBench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models

    Liu, F., Guan, T., Li, Z., Chen, L., Yacoob, Y ., Manocha, D., and Zhou, T. HallusionBench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv:2310.14566, 202...

    arXiv

  8. [22]

    Matos, J., Chen, S., Placino, S., Li, Y ., Pardo, J. C. C., Idan, D., Tohyama, T., Restrepo, D., Nakayama, L. F., Pascual-Leone, J. M., et al. Worldmedqa-v: a mul- tilingual, multimodal medical examination dataset for multimodal language models evaluation. arXiv preprint arXiv...

    arXiv

  9. [23]

    K., and Chakraborty, A

    Mishra, A., Shekhar, S., Singh, A. K., and Chakraborty, A. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947–952. IEEE,

    2019

  10. [24]

    org/abs/2211.09794

    URLhttps://arxiv. org/abs/2211.09794. OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  11. [25]

    Eagle: Exploring the design space for multi- modal llms with mixture of encoders

    Shi, M., Liu, F., Wang, S., Liao, S., Radhakrishnan, S., Huang, D.-A., Yin, H., Sapra, K., Yacoob, Y ., Shi, H., et al. Eagle: Exploring the design space for multi- modal llms with mixture of encoders. arXiv preprint arXiv:2408.15998,

    arXiv

  12. [26]

    Singh, A., Natarajan, V ., Shah, M., Jiang, Y ., Chen, X., Batra, D., Parikh, D., and Rohrbach, M

    URL https://arxiv.org/abs/2412.15188. Singh, A., Natarajan, V ., Shah, M., Jiang, Y ., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards VQA models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326,

    arXiv

  13. [27]

    Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context

    Team, G. Gemini 1.5: Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530,

    arXiv

  14. [28]

    C., Yang, J., Yang, S., Iyer, A., Pan, X., Wang, A., Fergus, R., LeCun, Y ., and Xie, S

    Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., Wang, A., Fergus, R., LeCun, Y ., and Xie, S. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. arXiv preprint arXiv:2406.16860, 2024a. Tong, S., Fa...

    arXiv

  15. [30]

    Minicpm-v: A gpt- 4v level mllm on your phone

    21 Diffusion Instruction Tuning Yao, Y ., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., Chen, Q., Zhou, H., Zou, Z., Zhang, H., Hu, S., Zheng, Z., Zhou, J., Cai, J., Han, X., Zeng, G., Li, D., Liu, Z., and Sun, M. Minicpm-v: A gpt- 4v level ...

    arXiv

  16. [31]

    Ferret: Refer and ground anything anywhere at any granularity

    You, H., Zhang, H., Gan, Z., Du, X., Zhang, B., Wang, Z., Cao, L., Chang, S.-F., and Yang, Y . Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704,

    arXiv

  17. [33]

    Transfusion: Predict the next token and dif- fuse images with one multi-modal model

    Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and dif- fuse images with one multi-modal model. arXiv preprint arXiv:2408.11039,

    arXiv

  18. [34]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592,

    arXiv

  19. [35]

    Mova: Adapting mixture of vision experts to multimodal context

    Zong, Z., Ma, B., Shen, D., Song, G., Shao, H., Jiang, D., Li, H., and Liu, Y . Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046,

    arXiv

  20. [39]

    Lavender-Llama3.2-11B occasionally refuses to answer questions for privacy reasons, resulting in a FALSE score and reduced performance on MME as shown in Figure

    Failure case analysis on the MME benchmark (Fu et al., 2023). Lavender-Llama3.2-11B occasionally refuses to answer questions for privacy reasons, resulting in a FALSE score and reduced performance on MME as shown in Figure

    2023

  21. [2014]

    Rlaif-v: Aligning mllms through open-source ai feed- back for super gpt-4v trustworthiness

    Yu, T., Zhang, H., Yao, Y ., Dang, Y ., Chen, D., Lu, X., Cui, G., He, T., Liu, Z., Chua, T.-S., and Sun, M. Rlaif-v: Aligning mllms through open-source ai feed- back for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220,

  22. [2015]

    Janus-pro: Unified multimodal understand- ing and generation with data and model scaling, 2025b

    Chen, X., Wu, Z., Liu, X., Pan, Z., Liu, W., Xie, Z., Yu, X., and Ruan, C. Janus-pro: Unified multimodal understand- ing and generation with data and model scaling, 2025b. URL https://arxiv.org/abs/2501.17811. Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., ...

    arXiv

  23. [2017]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024a. Wang, W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zen...

    arXiv

  24. [2018]

    Generative vi- sual instruction tuning

    Hernandez, J., Villegas, R., and Ordonez, V . Generative vi- sual instruction tuning. arXiv preprint arXiv:2406.11262,

    arXiv

  25. [2019]

    X., Tan, J

    Masry, A., Long, D. X., Tan, J. Q., Joty, S., and Hoque, E. ChartQA: A benchmark for question an- swering about charts with visual and logical reasoning. arXiv:2203.10244,

    arXiv

  26. [2020]

    to aggregate attention maps across layers in VLMs. Attention flow computes the effective attention between input and output tokens by considering the cumulative effect of attention across layers, capturing deeper interactions that span multiple layers. This method has been uti...

    2024

  27. [2021]

    From clip to dino: Visual encoders shout in multi-modal large language models

    Jiang, D., Liu, Y ., Liu, S., Zhao, J., Zhang, H., Gao, Z., Zhang, X., Li, J., and Xiong, H. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825,

    arXiv

  28. [2022]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. PMLR, 2023b. Li, Y ., Du, Y ., Zhou, K., Wang, J., Zhao, W. X., a...

    arXiv

  29. [2023]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J

    URL https://arxiv.org/ abs/2309.16609. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901 arXiv

  30. [2024]

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y ., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al

    URL https://arxiv.org/abs/2410.07073. Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y ., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35...

    arXiv

  31. [2025]

    Prismatic vlms: Investigating the design space of visually-conditioned language models

    Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865,

    arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.