Differential Multimodal Transformers

Jerry Li; Joseph Hoang; Timothy Oh; Vardhit Veeramachaneni

REVIEW 4 major objections 6 minor 20 references

Differential Multimodal Transformers

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that adapting Differential Attention to the multimodal model PaliGemma via LoRA fine-tuning improves noisy information retrieval, reporting higher needle-in-a-haystack accuracy than plain fine-tuning.

desk verdict The paper's differential attention reduces to a scalar multiple of standard softmax attention, so the central claim collapses and the experiments are confounded. read the letter →

arxiv 2507.15875 v1 pith:W72BCTO3 submitted 2025-07-17 cs.AI cs.MM

Jerry Li , Timothy Oh , Joseph Hoang , Vardhit Veeramachaneni This is my paper

classification cs.AIcs.MM

keywords DifferentialAttentionmultimodalvision-languagemodelsPaliGemmaLoRAfine-tuningnoisevisualquestionansweringneedle-in-a-haystackhallucinationmitigation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that Differential Attention, a mechanism originally developed for text-only Transformers, can be adapted to the multimodal vision-language model PaliGemma through LoRA-based fine-tuning, and that the adaptation improves retrieval when the input contains distraction. The authors' version of Differential Attention does not train a new pair of query and key projections; it duplicates the existing single attention map and subtracts a learned scalar multiple of it from itself, reasoning that the model can learn an optimal scalar during fine-tuning. They report that on the Multimodal Needle in a Haystack benchmark, the differential fine-tuned model reaches 34.72% index accuracy versus 30.42% for plain fine-tuning and 28.75% for the unfine-tuned baseline, while on VQAv2 the differential model scores slightly lower than the plain fine-tuned model. The paper's stated goal is to demonstrate that this attention modification can be integrated into existing models to reduce noise and improve retrieval and question-answering.

What carries the argument

The central object is the modified Differential Attention head defined in Eq. 5, which computes $\text{softmax}(QK^T/\sqrt{d_{\text{head}}}) - \lambda \, \text{softmax}(QK^T/\sqrt{d_{\text{head}}})$ and multiplies by $V$, with $\lambda$ a per-layer learned scalar initialized by Eq. 4. In the original Differential Transformer, noise reduction comes from subtracting two separate attention maps built from distinct query and key projections; the paper's version duplicates a single predefined attention map and relies on $\lambda$ to down-weight noisy attention. This is paired with LoRA low-rank updates on the frozen PaliGemma attention projections and head outputs normalized by RMSNorm scaled by $(1-\lambda_{\text{init}})$, following the gradient-stability recipe of the original Differential Transformer.

What would settle it

A controlled fine-tuning experiment that replaces the differential head with a standard attention head whose logits are scaled by a learned per-head constant would falsify the causal claim if it matches or exceeds the 34.72% MMNeedle accuracy, because it would show the subtraction structure itself is not needed.

Watch

Extended reading notes

Core claim

The central claim is that the Differential Attention mechanism can be moved from text-only Transformers to a multimodal model by fine-tuning, and that a modified version using only one set of pretrained query and key projections is enough to reduce attention noise. The authors reason that since the pretrained attention weights are already expressive, subtracting a learned scalar multiple of the same attention map lets the model down-weight irrelevant parts of the context during fine-tuning. They support this with the Multimodal Needle in a Haystack evaluation, where the differential fine-tuned PaliGemma reaches 34.72% index accuracy, above the 30.42% of the plain fine-tuned model and the 28.75% of the baseline, while also appearing more robust across needle positions.

Load-bearing premise

The load-bearing premise is that subtracting a scaled copy of the same attention map from itself still reduces attention noise, even though algebraically the operation only multiplies the original softmax attention by $(1-\lambda)$.

Editorial extensions

If this is right

The recipe can be applied to other pretrained multimodal models without retraining from scratch, since it only adds LoRA matrices and a small set of scalar $\lambda$ parameters.
On noisy retrieval tasks, a differential fine-tuned model can outperform both the locked pretrained model and the conventionally fine-tuned model.
The attention-modification approach can improve robustness across spatial positions in an image grid, not only average accuracy.
Standard question-answering accuracy may not rise in tandem; the authors' VQAv2 score with Differential Attention is 51.91 versus 53.21 for plain fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Eq. 5 factors to $(1-\lambda)$ times the original softmax attention, the reported gains could come from a learned reweighting of attention rather than from the cancellation mechanism of the original Differential Transformer; the paper does not disentangle these.
A natural control is to fine-tune with standard attention whose softmax temperature or logit scale is a learned per-head scalar; matching 34.72% would show the two-map subtraction is unnecessary.
The MMNeedle gain is computed on 200 samples; a larger evaluation would be needed to confirm the 4.3-point difference is not noise.
Testing the same LoRA-plus-scalar-subtraction recipe on a model with a wider context window would show whether the approach scales beyond PaliGemma's limited token capacity.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The thing to know about arXiv:2507.15875 is that its headline mechanism, Eq 5, is a scalar multiple of standard softmax attention. The paper defines DiffAttn as softmax(QK^T/sqrt(d)) − λ·softmax(QK^T/sqrt(d)) using the same Q and K in both terms, so it equals (1−λ) times ordinary attention. The original Differential Transformer gets its denoising from two distinct learned query and key projections; subtracting identical maps cannot do that. The abstract's claim that this adapts Differential Attention to multimodal retrieval is not supported by the math as written.

What the paper does well: it is transparent. The authors state their computational constraints, describe the VQAv2 and MMNeedle setups, and report enough hyperparameter detail that a reader can see the comparisons are confounded. Section 6 explicitly acknowledges that time and hardware limited the evaluation. That honesty is real, but it does not rescue the central result.

The soft spots are substantial. First, the algebra kills the novelty: no new operator is introduced, and the 'differential' model is equivalent to scaling attention weights by a per-layer scalar, which is a routine temperature-like adjustment. Second, the experiments do not isolate the claimed mechanism. Table 1 varies the feedforward block, LoRA rank/alpha, learning rate, and weight decay across models, and the two models selected for MMNeedle (Model 1 vs Model 2) differ in both attention and feedforward. Table 2's difference (30.42 vs 34.72) is therefore not attributable to the differential attention component. Third, the evaluation is small: 200 samples, one run, and the code link is a placeholder. On the positive side, the conclusion is appropriately cautious, but the caveat is about compute, not about the validity of the mechanism.

My verdict matches the reader's: reject. The central claim is false as written, and the empirical evidence is confounded. The only reader who gets value is a student or teacher using it as a cautionary example of an attention variant collapsing into a scalar. This is not a paper that needs a serious referee; it is a project that missed a basic algebraic identity. A desk reject with feedback pointing to Eq 5 and the need for two distinct projections would be appropriate. That said, the writing is clear and the limitations are honest, so with a corrected mechanism and a clean comparison the authors could produce something more credible.

Recommendation: do not send to peer review; a desk reject with detailed guidance is the right call.

Referee Report

4 major / 6 minor

Summary. This paper proposes to adapt the Differential Attention mechanism of Ye et al. (2024) to the multimodal model PaliGemma by fine-tuning with LoRA. The authors define a 'single-set' differential attention in Eq. (5) that subtracts a scalar-weighted copy of the same softmax attention map from itself, and they evaluate the resulting 'Differential PaliGemma' on VQAv2 and the Multimodal Needle-in-a-Haystack (MMNeedle) benchmark. The central claim is that this adaptation preserves the noise-reduction properties of the original differential attention, improving noisy information retrieval and question answering in a multimodal setting.

Significance. If the proposed mechanism were sound, this work could offer a lightweight way to retrofit differential attention into existing multimodal models, which would be of practical interest. The paper uses public benchmarks and describes its fine-tuning procedure, providing some reproducibility even if the code link is not expanded. However, the core mathematical operation in Eq. (5) is algebraically degenerate, reducing to standard softmax attention multiplied by a per-layer scalar. Consequently, the claimed noise-reduction mechanism is not implemented, and the main empirical comparison is confounded by simultaneous changes to the feedforward network. These issues invalidate the central claim, so the significance of the contribution is not established.

major comments (4)

[§3.3, Eq. (5)] Because the two softmax terms in Eq. (5) use the same query and key matrices, DiffAttn(X) equals (1 − λ) · softmax(QK^T/√d_head)V. This is exactly standard softmax attention with a per-layer scalar multiplier; it cannot re-weight which tokens are attended to and therefore does not implement the differential denoising mechanism of [17]. Any effect of tuning λ is equivalent to adjusting a softmax temperature, so the claimed noise-reduction property is absent from the proposal.
[§3.3] The load-bearing assertion that 'the concept of subtracting two sets of attention weights to reduce noise can be directly applied to a single predefined set' is contradicted by Eq. (5): subtracting a scaled copy of the same attention map is a scalar multiplication, not a subtraction of two distinct attention distributions. The claim that the model 'learns optimal λ values' to 'diminish attention on noisy or irrelevant information' is not supported by the mathematics, since a scalar multiplication of the entire attention map cannot selectively suppress individual noisy tokens.
[Table 1] The VQAv2 results contradict the paper's claim of improved question-answering capability: the differential-attention model (row 2) scores 51.91, below the non-differential model (row 1) at 53.21 under the same learning rate, LoRA rank/alpha, and weight decay. Section 4.1.1 then selects these two models for the MMNeedle benchmark as the 'best potential' configurations, which is not justified by the VQAv2 scores.
[Table 2 and §5.2] The MMNeedle comparison between 'Finetuned PaliGemma' (30.42%) and 'Finetuned with Diff. Attn. PaliGemma' (34.72%) is confounded because, according to Table 1, these two configurations differ in the feedforward network (MLP vs SwiGLU) as well as in the attention mechanism. The reported 4.3-point improvement cannot be attributed to the proposed differential attention without a controlled ablation that changes only the attention module.

minor comments (6)

[Throughout] The manuscript contains numerous typographical errors, including 'parametres', 'confirguation', 'district', 'ontop', and 'than' used in place of 'then'; a thorough proofread is needed.
[§3.2, Eq. (3)] The text says that λ is created from learnable parameters 'for each query and key', but Eq. (3) defines a single scalar per layer; the relationship between the per-vector parameters and the per-layer scalar should be stated precisely.
[Figure 1] Figure 1 is referenced in the text but no image appears in the manuscript; either include the architecture diagram or remove the reference.
[§5.1] The prompt template appears to concatenate the caption with the literal string 'Where is the caption? Top or Bottom?'; the exact format, including where the caption is placed relative to the question, should be spelled out because the benchmark's validity depends on it.
[§5.2] The claim that the models achieve 'high accuracy' for bottom-left needles is not quantified; reporting per-cell accuracies would make the heatmap in Figure 2 interpretable.
[Table 2] The sample size of 200 and the absence of error bars or significance tests make the reported differences (30.42 vs 34.72) difficult to evaluate; the authors should report confidence intervals or repeated runs.

Circularity Check

1 steps flagged · score 8.0 of 10

Eq. 5 is algebraically degenerate: subtracting λ times the identical softmax term equals (1−λ) times standard attention, so the claimed 'differential' denoising mechanism reduces to a fitted scalar.

self definitional [Section 3.3, Eq. (5)]
"Rather than creating two separate sets of queries and keys like the original Differential Attention, we take the original query and keys and simply duplicate itself as shown below. ... DiffAttn(X) = (softmax(QK^T/sqrt(d_head)) − λ softmax(QK^T/sqrt(d_head))) V. ... Instead, the concept of subtracting two sets of attention weights to reduce noise can be directly applied to a single predefined set."

The two softmax terms in Eq. 5 are identical by construction (same Q, same K, same d_head), so DiffAttn(X) = (1−λ)·softmax(QK^T/sqrt(d_head))V. This is standard scaled softmax attention with a scalar multiplier; it cannot reweight which tokens are attended to or implement any 'subtraction of two attention maps'. The claimed noise-reduction mechanism is therefore not an independent component being tested; any effect attributed to 'Differential Attention' is, by definition, just the effect of training a per-layer scalar λ on top of the original attention. The empirical MMNeedle comparison cannot separate a differential mechanism from ordinary attention scaling.

full rationale

The paper's central technical claim, that the Differential Attention concept can be applied to a single predefined set of queries and keys, reduces to the identity S − λS = (1−λ)S. Both copies of the attention matrix in Eq. 5 are literally the same softmax(QK^T/sqrt(d_head)) because W_Q and W_K are not duplicated and no separate projections are introduced. Consequently the 'differential' operator is a scalar multiplier, and the claimed ability to 'diminish attention on noisy or irrelevant information while enhancing focus on more critical areas' is not supported by any mechanism in the equation; a shared scalar cannot change the relative attention distribution. The paper's own Section 6 limitation ('Due to time and hardware constraints we were unable to finetune on further epochs and hyperparameter settings') is acknowledged, but it does not address this algebraic degeneracy. Table 2's comparison is also confounded: the 'Finetuned with Diff. Attn.' run uses SwiGLU while the 'Finetuned PaliGemma' run uses MLP, so the 34.72 vs 30.42 difference cannot be isolated to Eq. 5. The identified reduction is by construction, making the central mechanism self-definitional rather than an empirical discovery. No self-citation chain is involved, and the cited Differential Transformer paper [17] is external to the present authors, but the local Eq. 5 identity is sufficient to establish the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The paper introduces a single learned scalar lambda per layer as the only new component; all other architecture pieces are borrowed from PaliGemma and the Differential Transformer paper. The key presupposition is that subtracting a duplicate softmax map denoises attention, which is algebraically false. The simplified MMNeedle protocol is an assumption because it is not the original benchmark and is not validated.

free parameters (2)

lambda (per-layer scalar attention weight) = learned during fine-tuning, not reported
Equation 5 shows the entire 'differential' component is a single scalar lambda multiplying softmax; any performance difference is attributable to this fitted value rather than a new mechanism.
LoRA rank and alpha configuration = rank 32/alpha 64 or rank 16/alpha 32 depending on run
Chosen by hand from a small grid; the selected best configuration differs across compared models, confounding the MMNeedle comparison.

assumptions (4)

ad hoc to paper Subtracting two identical softmax attention maps can reduce attention noise.
Eq 5 duplicates Q and K so the subtraction collapses to scalar scaling; no mechanism for noise reduction is derived. Section 3.3.
domain assumption Attention in both the Siglip encoder and the Gemma decoder can be replaced by Eq 5 and then recovered through LoRA fine-tuning.
The paper fine-tunes only LoRA adapters plus new lambda parameters and assumes the base weights remain stable and useful; no stability analysis is provided. Sections 3.3 and 4.1.1.
domain assumption A model selected as best on VQAv2 is the correct model to compare on MMNeedle.
The two selected models differ by attention, feedforward block, and hyperparameters, so selection on VQAv2 does not isolate the attention mechanism. Section 4.1.1.
domain assumption Sequential top/bottom then left/right binary queries give a valid measure of needle-in-a-haystack retrieval.
This protocol differs from the original MMNeedle benchmark and has no validation in the paper; it is evaluated on only 200 samples. Section 5.1.

invented entities (1)

Single-set Differential Attention operator (Eq 5)
purpose: Claimed to reduce noise by subtracting duplicate attention maps while fine-tuning PaliGemma.
It is algebraically identical to (1 - lambda) times standard attention; no separate mechanism or falsifiable prediction outside the fitted lambda is provided. Section 3.3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Differential Multimodal Transformers." pith.science (2026). https://pith.science/paper/W72BCTO3

@misc{pith2026250715875,
  author       = {Pith},
  title        = {Pith review of: Differential Multimodal Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W72BCTO3}},
  note         = {Machine review of arXiv:2507.15875}
}

read the original abstract

Small language models have gained significant popularity due to their efficiency and growing capabilities. However, incorporating additional modalities, such as vision, can exacerbate the challenge of limited context windows by introducing noise. Recent studies have highlighted that Transformer attention mechanisms often disproportionately focus on irrelevant contexts. In this work, we extend the Differential Attention mechanism, originally designed for text-only models, to the text-vision model PaliGemma. Our aim is to evaluate its ability to mitigate noisy information retrieval and reduce hallucinations. To this end, we fine-tuned the PaliGemma 3B model using LoRA, incorporating Differential Attention, and experimented with various parameter settings and configurations. We demonstrate that Differential Attention can be adapted and integrated into the fine-tuning of existing models to enhance noisy information retrieval and question-answering capabilities.

Figures

Figures reproduced from arXiv: 2507.15875 by the authors.

**Figure 2.** MMNeedle evaluation performance comparison. The x-axis plots each model’s performance, [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 4 canonical work pages

[17]

Differential transformer, 2024

Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer, 2024. URL https://arxiv.org/abs/2410.05258

arXiv 2024
[1]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza 7 Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Shar- ifzadeh, Mikolaj Bi...

arXiv 2022
[2]

Paligemma: A versatile 3b vlm for transfer, 2024

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bau...

arXiv 2024
[3]

Pali: A jointly-scaled multilingual language-image model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carl...

arXiv 2023
[4]

Making the v in vqa matter: Elevating the role of image understanding in visual question answering

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. arXiv preprint arXiv:1612.00837, 2017. URL https://arxiv.org/pdf/1612.00837. Version 3

arXiv 2017
[5]

Textbooks are all you need

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. arXiv preprint arXiv:2...

arXiv 2023
[6]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. URL https://arxiv.org/abs/2106.09685. Version 2

arXiv 2021
[7]

Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 66–

work page 2018

Show all 20 references

[8]

Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts, 2023. URL https://arxiv.org/abs/2307.03172

2023 arXiv
[9]

Minesh Mathew, Dimosthenis Karatzas, and C. V . Jawahar. Docvqa: A dataset for vqa on document images, 2021. URL https://arxiv.org/abs/2007.00398

2021 arXiv
[10]

Gpt-4o mini: Advancing cost-efficient intelligence, July 2024

OpenAI. Gpt-4o mini: Advancing cost-efficient intelligence, July 2024. URL https:// openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ . Ac- cessed: 2024-12-07

2024
[11]

Learning transferable visual models from natural language supervision

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings o...
[13]

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. URL https://arxiv.org/abs/2403. 05530. Version 4. 8

2024 arXiv
[14]

Clark, et al

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose ...

2024 arXiv
[15]

Multimodal needle in a haystack: Benchmarking long-context capability of multimodal large language models, 2024

Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay Nambi, Tanuja Ganu, Hao Wang, Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai...

2024 arXiv
[16]

A comprehensive review of multimodal large language models: Performance and challenges across different tasks, 2024

Jiaqi Wang, Hanqi Jiang, Yiheng Liu, Chong Ma, Xu Zhang, Yi Pan, Mengyuan Liu, Peiran Gu, Sichen Xia, Wenjun Li, Yutong Zhang, Zihao Wu, Zhengliang Liu, Tianyang Zhong, Bao Ge, Tuo Zhang, Ning Qiang, Xintao Hu, Xi Jiang, Xin Zhang, Wei Zhang, Dinggang Shen, Tianming Liu, and S...

2024 arXiv
[18]

Sigmoid loss for language image pre-training

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986, 2023. URL https://doi.org/10.48550/ arXiv.2303.15343. 9

2023 doi arXiv
[71]

doi: 10.18653/v1/D18-2012

Association for Computational Linguistics, 2018. doi: 10.18653/v1/D18-2012. URL https://aclanthology.org/D18-2012

2018 doi
[2021]

URL https://proceedings.mlr.press/v139/radford21a.html
[2024]

Version 1

URL https://arxiv.org/abs/2410.18613. Version 1

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [17]

Differential transformer, 2024

Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, and Furu Wei. Differential transformer, 2024. URL https://arxiv.org/abs/2410.05258

arXiv 2024

[2] [1]

Flamingo: a visual language model for few-shot learning

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Has- son, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza 7 Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Mon- teiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Shar- ifzadeh, Mikolaj Bi...

arXiv 2022

[3] [2]

Paligemma: A versatile 3b vlm for transfer, 2024

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bau...

arXiv 2024

[4] [3]

Pali: A jointly-scaled multilingual language-image model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carl...

arXiv 2023

[5] [4]

Making the v in vqa matter: Elevating the role of image understanding in visual question answering

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. arXiv preprint arXiv:1612.00837, 2017. URL https://arxiv.org/pdf/1612.00837. Version 3

arXiv 2017

[6] [5]

Textbooks are all you need

Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tau- man Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need. arXiv preprint arXiv:2...

arXiv 2023

[7] [6]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. URL https://arxiv.org/abs/2106.09685. Version 2

arXiv 2021

[8] [7]

Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 66–

work page 2018