Pith. Sign in

REVIEW

Sigmoid Loss for Language Image Pre-Training

PaperPDF from arXiv

data files

ReviewThe objections and verdict, as JSON Paper breakdownThe authors' claims, conditions and measurements, as JSON Signed receiptProof this review has not been altered
X Bluesky LinkedIn Reddit Hacker News
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas Beyer

Computer Vision and Pattern Recognition · Submitted 27 Mar 2023 · 60 references

Pith's verdict: SigLIP's pairwise sigmoid loss removes the global normalization step from contrastive pretraining, which lets batch size vary freely and supports the reported 84.5% zero-shot result on minimal hardware, but the numbers need full experimental controls to confirm they generalize.

T0 · AI review

How this review was made: An AI referee (grok-4.3) read the paper on May 16, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

  1. T0 AI review An AI read the paper and answered a fixed list of questions.
  2. T1 Audit Automatic checks that the references and files the paper relies on exist and match.
  3. T2 Compute The paper's code or calculations re-run with the same results.
  4. T3 Formal statement The claims written out in a formal math language a computer can read.
  5. T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 2 major objections · 1 minor · Ask about this paper · Read the full review →
Challenge this review Comment (0) Read on arXiv ↗
Cite ▾
arXiv ID2303.15343 (version 4)
Pith IDX5N46CMJ, Pith's permanent record of this paper and its review
CitationAlexander Kolesnikov, Basil Mustafa, Lucas Beyer, Xiaohua Zhai. "Sigmoid Loss for Language Image Pre-Training." arXiv:2303.15343 (2023). https://pith.science/paper/X5N46CMJ
Cite this reviewBibTeX for Pith's review of this paper
Claim paper

Why Pith says this

Highlight any text to comment on it.

Referee report

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

A pairwise sigmoid loss for image-text pre-training achieves 84.5% zero-shot ImageNet accuracy using only four TPU chips in two days.

The reading

The paper introduces a sigmoid loss that computes a loss for each image-text pair independently, without needing to normalize across the entire batch like in standard contrastive methods. This design allows training with much larger batch sizes and also works well with smaller ones, removing the coupling between batch size and loss function. When paired with locked-image tuning, the resulting SigLiT model reaches 84.5 percent accuracy on ImageNet zero-shot classification after training for two days on four TPUv4 chips. The separation of batch size also enables experiments varying the number of examples, pairs, and negative-to-positive ratios. Tests up to batch size one million show that gains level off, with 32 thousand being adequate.

What carries the argument

The pairwise sigmoid loss, which applies a sigmoid activation to the dot product of image and text embeddings for each pair independently.

Extended reading notes

Core claim

The sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization, simultaneously allowing further scaling up of the batch size while also performing better at smaller batch sizes.

Load-bearing premise

The sigmoid loss will keep producing high-quality representations at new scales or on new data without needing hyper-parameter adjustments.

Editorial extensions

If this is right

  • Training becomes possible with extremely large batch sizes up to one million without issues from global normalization.
  • A moderate batch size of 32k provides most of the benefits, making training more practical.
  • The loss allows independent control over the number of examples and the negative-to-positive ratio.
  • High zero-shot performance is achievable with minimal hardware resources when combined with locked-image tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Pre-training could become accessible on smaller compute budgets or even single machines with further optimizations.
  • The method might apply to other contrastive learning setups beyond vision-language models.
  • Future work could explore even larger scales or different modalities using the same loss structure.
Original abstract

We propose a simple pairwise Sigmoid loss for Language-Image Pre-training (SigLIP). Unlike standard contrastive learning with softmax normalization, the sigmoid loss operates solely on image-text pairs and does not require a global view of the pairwise similarities for normalization. The sigmoid loss simultaneously allows further scaling up the batch size, while also performing better at smaller batch sizes. Combined with Locked-image Tuning, with only four TPUv4 chips, we train a SigLiT model that achieves 84.5% ImageNet zero-shot accuracy in two days. The disentanglement of the batch size from the loss further allows us to study the impact of examples vs pairs and negative to positive ratio. Finally, we push the batch size to the extreme, up to one million, and find that the benefits of growing batch size quickly diminish, with a more reasonable batch size of 32k being sufficient. We release our models at https://github.com/google-research/big_vision and hope our research motivates further explorations in improving the quality and efficiency of language-image pre-training.

References

How the literature uses it

213 arXiv papers cite this in 315 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 119 sentences without a confident label

Builds on it 1 paper
  • We extend this question to VFMs—large Transformers trained with different datasets (, LAION [ref] , WebLI [ref] , LVD-142M [ref] ), objectives (contrastive vs. reconstruction), and modality mixes (vision-language vs. pure vision)—including CLIP [ref] , DINOv2 [ref] , DINOv3 [ref] and SigLIP 2 [ref] .

    Revisiting Model Stitching In the Foundation Model Era · Introduction · 0.82

Uses its method or tool 98 papers
Compares with it 6 papers
  • obtains 2.8% better average performance on 38 datasets and significantly better retrieval performance when compared to ViT-B/32-256 model trained 2.6$×$ longer on DataComp [ref] . is 1.5$×$ smaller and 1.4$×$ faster than ViT-B/32-256 model. obtains 2.9% better average performance on 38 datasets and better retrieval performance while being 26.3% smaller than SigLIP-B/16 [ref] model, which is trained approximately 3$×$ longer on WebLI dataset.

    MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training · Comparison with State-of-the-art · 1.00

  • We compared NLLB-CLIP with state-of-the-art models – mSigLIP [ref] and PaLI [ref] .

    NLLB-CLIP -- train performant multilingual image retrieval model on a budget · Crossmodal-3600 · 1.00

  • Baselines We compare CAIP against a representative set of vision encoders spanning self-supervised, language-supervised, video, and robotics-pretrained representations: R3M [ref] , MVP [ref] , DINOv2 [ref] , SigLIP [ref] , SigLIP 2 [ref] , VideoMAE [ref] , VC-1 [ref] , and the native Qwen3.5-0.8B vision encoder [ref] .

    Contrastive Action-Image Pre-training for Visuomotor Control · Real-world Evaluation · 0.99

Cites it as support 1 paper
Cites it as background 67 papers

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.