Pith. sign in

REVIEW 5 major objections 5 minor 16 references

The paper claims that a fully Mamba-based CLIP—no transformers in either tower—matches transformer-based CLIP on retrieval and beats it on out-of-distribution robustness while using less memory at high resolution.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 11:15 UTC pith:BDQM27C2

load-bearing objection First fully Mamba-based CLIP with credible retrieval/OOD gains, but the headline classification edge rests on a single unexplained dataset outlier that likely flips when removed. the 5 major comments →

arxiv 2601.06891 v2 pith:BDQM27C2 submitted 2026-01-11 cs.CV

CLIMP: Contrastive Language-Image Mamba Pretraining

Nimrod Shabtay , Itamar Zimerman , Eli Schwartz , Raja Giryes This is my paper
classification cs.CV
keywords contrastive vision-language pretrainingstate space modelsMambaVMambaCLIPzero-shot robustnessimage-text retrievalresolution scalability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CLIMP asks whether a contrastive vision-language model can be built entirely from state-space models instead of transformers. The paper's answer is yes: replacing both the image encoder (with VMamba) and the text encoder (with Mamba-1 or Mamba-2) produces an embedding space with tighter cross-modal alignment and lower hubness, which shows up as higher retrieval recall and better out-of-distribution accuracy. A reader should care because the same swap removes two long-standing CLIP constraints—fixed 224×224 inputs and a 77-token text window—while cutting memory and compute at high resolution. If the results hold, the path to robust, resolution-flexible vision-language models does not require attention at all.

Core claim

On its own terms, the paper discovers that a fully state-space architecture can learn a shared image-text embedding space that is better aligned and less dominated by hub embeddings than transformer-based CLIP, yielding higher retrieval recall and stronger out-of-distribution accuracy. The headline evidence is that a CLIMP model trained on 12M image-text pairs surpasses a ViT-based CLIP trained on roughly 167× more data on the ImageNet-O distribution-shift benchmark, and that both CLIMP variants top the average across five ImageNet robustness variants. At 896×896 input resolution, CLIMP keeps retrieval accuracy while transformer baselines degrade sharply, with 5× lower resolution-specific me

What carries the argument

The central mechanism is Mamba, a selective state-space model that updates a recurrent hidden state with input-dependent parameters, giving linear-time sequence processing instead of attention's quadratic pairwise comparisons. VMamba adapts Mamba to images with a 2D cross-scan that traverses patches along four directions, so spatial layout is carried by the scan order rather than by learned position embeddings; this is what lets CLIMP accept any resolution without interpolation or retraining. On the text side, Mamba LLMs are autoregressive, and CLIMP pools the hidden state of the last non-padding token, a choice that naturally extends beyond the standard 77-token context and enables dense-ca

Load-bearing premise

The load-bearing premise is that the reported classification superiority is genuine, especially the 71.3% score on the diabetic_retinopathy dataset; if that number is a label-alignment or evaluation artifact, the paper's top classification result collapses.

What would settle it

Re-run the zero-shot evaluation on diabetic_retinopathy alone and inspect per-class predictions; if the 71.3% top-1 is not reproducible, or is explained by majority-class guessing or label-order leakage, then the claimed +2.3% classification advantage disappears. Recomputing the 31-dataset average with diabetic_retinopathy excluded would settle whether CLIMP's top classification rank is robust.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • CLIP-style zero-shot models can be built with sub-quadratic complexity in both towers, making high-resolution and long-context deployment practical.
  • Mamba's spatial inductive bias appears to improve sample efficiency and OOD robustness, so training at larger data scales should widen the reported margins.
  • Native variable resolution means images can be indexed at their original size, including very high-resolution domains, without resizing or positional-encoding tricks.
  • Long, paragraph-length captions become usable as retrieval queries, going beyond CLIP's fixed token window.
  • Matched SSM encoders produce better cross-modal alignment than mixing a Mamba vision tower with a transformer text tower, informing architecture choices for future vision-language models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the ImageNet-O result suggests OOD robustness may be driven more by architecture than by training-data scale; if replicated, it would push the field toward inductive-bias-oriented design rather than data scaling alone.
  • The authors note that SSMs and transformers have different failure modes, which implies an ensemble of a Mamba-based and a ViT-based CLIP could be more robust than either; this is a cheap, direct next experiment.
  • The spatial-inductive-bias experiment with shuffled patches suggests a testable extension: CLIMP should be unusually sensitive to patch-order corruption at inference, and robust to it only if scan order is semantically meaningful.
  • Because no positional embeddings are needed, the same architecture could extend to tiled gigapixel images or video frames, which the paper does not evaluate.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. CLIMP replaces both encoders of a CLIP-style contrastive model with state-space models: VMamba-B for vision and Mamba-1/Mamba-2 (1.4B/1.3B) for text, trained on CC12M at 224×224. The paper claims that CLIMP outperforms Transformer-based counterparts (ViT/FlexViT/NaFlex/RoPE-ViT, paired with LLaMA-3.2-1B) on CLIP-Benchmark retrieval, on five ImageNet OOD variants, at high resolution (up to 896×896), and for dense-caption retrieval, while being more memory/FLOP efficient. Additional analyses address spatial inductive bias, embedding geometry, scaling, and architectural synergy. The headline results are Table 1's classification/retrieval averages, Table 2's OOD averages, and Table 4's high-resolution retrieval.

Significance. If the empirical claims hold, the paper would be a meaningful step: it shows a fully SSM dual-encoder can be trained with CLIP-style contrastive learning, handles variable resolutions without positional-encoding surgery, and is compute-efficient. The open code/models, controlled inductive-bias experiment (Table 14), and alignment/hubness analysis (Table 6) are useful. However, the main classification advantage is not robust: it rests on one unexplained outlier, and there are internal inconsistencies between tables. The contribution is therefore potentially significant but needs verification and more careful reporting before the claims can be accepted.

major comments (5)
  1. [Table 1 / Appendix Table 9 / §4.1] The best-Acc@1 claim in Table 1 (CLIMP-Mamba-1 29.6 vs RoPE-ViT 27.3, +2.3) depends on one anomalous dataset. In Appendix Table 9, diabetic_retinopathy gives CLIMP-Mamba-1 71.3% top-1, while CLIMP-Mamba-2 scores 10.4% and every transformer baseline scores 2.3–30.7%; all models have 100% top-5. Removing this single row from the 25-row table changes Mamba-1's average to ~27.9 and RoPE-ViT's to ~28.3, reversing the reported ranking. The paper offers no explanation, no per-class/balanced accuracy, and no majority-class baseline for a benchmark that is known to be highly imbalanced. This outlier is load-bearing for the 'match or exceed transformers' conclusion and must be analyzed (or the claim retracted). Please also reconcile the '31 datasets' in Table 1 with the 25 classification + 3 retrieval rows in Tables 9–10.
  2. [Table 2 vs Appendix Table 9 / §4.2] The ImageNet-O top-1 numbers are swapped between the two tables. Table 2 reports CLIMP-Mamba-2 49.8 and CLIMP-Mamba-1 48.1, while Appendix Table 9 lists VMamba+Mamba2 48.1 and VMamba+Mamba1 49.8. Section 4.2 uses these values to claim gains of +9.7/+8.0 over RoPE-ViT and a +7.5 advantage over CLIP-ViT-B/16. Both tables cannot be correct; please provide the underlying logs, correct the tables, and recompute the OOD averages.
  3. [Abstract / §4.2 footnote / References] The ImageNet-O comparison is attributed to 'OpenAI's CLIP-ViT-B/16 trained on LAION-2B'. OpenAI's CLIP-ViT-B/16 was trained on WIT-400M, not LAION-2B; the 42.3 number is from OpenCLIP results, as the footnote indicates. This misattribution affects the headline 'surpassing OpenAI's CLIP-ViT-B by 7.5%' and the '167× smaller data' narrative. Please rephrase to name the actual OpenCLIP checkpoint.
  4. [§4.4 / Table 5 / Table 13] The claim that 'CLIMP variants consistently outperform all transformer baselines' in dense-caption retrieval is contradicted by the DOCCI text-retrieval numbers: NaFlex+LLaMA has T→I Recall@5 53.4, vs 51.7 for CLIMP-Mamba-2 and 50.4 for CLIMP-Mamba-1. The image-retrieval results are strong, but the blanket statement and the 'up to 11.8%' summary should be restricted to the image→text direction or to the specific baselines where CLIMP actually leads.
  5. [§4.3 / Table 4 / Abstract] The statement that both CLIMP variants 'consistently outperform all baselines at every resolution' is not supported by Table 4: at 512×512, RoPE-ViT has TR@5 70.0 vs 67.2 (Mamba-2) and 66.8 (Mamba-1). The 896×896 advantage is substantial and real, but the overclaim should be corrected. In addition, the abstract's 'up to 6.6% higher retrieval accuracy at 16x training resolution' should name the comparator (apparently NaFlex) and metric, since it is not the strongest transformer baseline.
minor comments (5)
  1. [Abstract] Typo: 'To address these limitations, We present' should be lowercase 'we'.
  2. [§3 / §3.1] The λ^k decay argument for inductive bias in selective SSMs is heuristic, since B, C, and Δ are input-dependent. Please state the assumptions more carefully and cite the prior analysis it relies on.
  3. [Table 10] Column order is inconsistent with Table 1: Table 10 lists T-R@5 before I-R@5 while Table 1 reports IR@5/TR@5. Make the order and labels uniform.
  4. [Figure 4] The caption says 'trained on CC dataset'; specify that this is CC12M and describe how the 1M–12M subsets were sampled.
  5. [Figure 6] Typo in caption: 'Orignal' should be 'Original'.

Circularity Check

0 steps flagged

No circularity: the paper's claims are empirical comparisons against external baselines; the diabetic-retinopathy outlier is a data-integrity concern, not a circular derivation.

full rationale

The paper's central claims are empirical outcomes from training CLIMP and transformer baselines under a shared protocol and evaluating on external benchmarks (CLIP-Benchmark, ImageNet variants, NoCaps, Crossmodal-3600, Flickr8k-R, DOCCI). No fitted parameter is renamed as a prediction, and no result is derived from an equation that assumes the conclusion. The analytical remark in Section 3 that Mamba-2's recurrent update with Ā = λI gives λ^k decay is a direct property of the stated dynamics, used only as motivation, not as a substitute for experimental evidence. The inductive-bias experiment in Section 4.5.1 / Table 14 is a controlled comparison on CIFAR-10 with shuffled patch orders, testing the architectural prior rather than assuming it. The only self-citation (Zimerman & Wolf, 2024) is background and the paper explicitly notes that prior analyses 'do not directly extend to selective SSMs,' so it is not load-bearing. The anomalous diabetic_retinopathy cell (Table 9) and the swapped ImageNet-O values between Tables 2 and 9 are serious verification concerns about the reported numbers, but they do not constitute circularity: the reported averages are not forced by construction. The Limitations section also appropriately flags scale uncertainty. Overall, the derivation chain is self-contained with respect to the claim of architectural advantage; the risks are empirical validity, not circular reasoning.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted constants or new entities are introduced. The paper relies on standard architecture components and the assumptions above; the diabetic_retinopathy outlier is a data-integrity concern, not an axiom.

axioms (4)
  • domain assumption Mamba-2's per-step transition with A=lambda I yields exponentially decaying past-state influence approximately lambda^k under stability conditions.
    Used in Section 3 to argue a locality/smoothness inductive bias; selective SSMs have input-dependent effective dynamics, so this is an approximation, not a theorem for the actual model.
  • domain assumption VMamba's SS2D scanning order provides sufficient implicit positional information to generalize across resolutions without positional embeddings.
    Relied on throughout Section 4.3; plausible and supported by experiments, but not proven.
  • ad hoc to paper Last-token pooling of a causal Mamba LLM yields a representation suitable for contrastive alignment.
    Introduced in Eq. (1); the paper does not compare this to other pooling strategies (e.g., mean pooling) or to a bidirectional text encoder.
  • domain assumption The external CLIP-ViT-B/16 ImageNet-O result (42.3%) is obtained under the same zero-shot prompt protocol as CLIMP.
    Used in Section 4.2 for the '167x larger data' comparison; the paper does not specify the prompt templates or verify protocol matching.

pith-pipeline@v1.3.0-alltime-deepseek · 16417 in / 20322 out tokens · 189039 ms · 2026-08-03T11:15:11.515144+00:00 · methodology

0 comments
read the original abstract

Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI's CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP's fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.The code and models are publicly available at https://github.com/NimrodShabtay/CLIMP}

Figures

Figures reproduced from arXiv: 2601.06891 by Eli Schwartz, Itamar Zimerman, Nimrod Shabtay, Raja Giryes.

Figure 1
Figure 1. Figure 1: CLIP vs. CLIMP. By replacing Transformer encoders with Mamba-based models, CLIMP achieves sub-quadratic O(L) complexity instead of quadratic O(L 2 ), while removing the fixed resolution and 77-token text limitations inherent to standard CLIP. We empirically confirm that VMamba’s spatial inductive bias transfers to the vision-language setting, enabling improved sample efficiency and robustness (Section 4.5)… view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of Image-Text similarity for the caption: "A large porch with a wooden fence and no roof." [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Efficiency analysis. CLIMP achieves superior memory and computational efficiency across all resolutions. (Left) Memory overhead is 4–57× lower. (Right) FLOPs scale linearly, yielding up to 1.8× reduction—a gap that widens with resolution. Params CLIMP FlexViT RoPE-ViT (Mamba2) (LLaMA) (LLaMA) 22-30M 38.8 32.5 33.2 50M 42.8 – – 87M 43.5 38.3 40.7 [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Scaling laws of CLIMP trained on CC dataset [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Visualization of image-text similarity. CLIMP exhibits more interpretable alignment and better similarity [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of the original and distorted [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 13 linked inside Pith

  1. [2]

    Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt

    Understanding robustness of visual state space models for image classification.arXiv preprint arXiv:2403.10935. Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt

    Pith/arXiv arXiv

  2. [4]

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    CLIP-Mamba: CLIP pretrained mamba models with OOD and hessian evaluation.arXiv preprint arXiv:2404.19394. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig

    Pith/arXiv arXiv

  3. [7]

    Simba: Simplified mamba-based architecture for vision and multivariate time series.arXiv preprint arXiv:2403.15360. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever

    Pith/arXiv arXiv

  4. [8]

    arXiv preprint arXiv:2402.02491

    VM-UNet: Vision mamba UNet for medical image segmentation. arXiv preprint arXiv:2402.02491. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, and 1 others

    Pith/arXiv arXiv

  5. [12]

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, and 1 others

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971. Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, and 1 others

    Pith/arXiv arXiv

  6. [13]

    Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing

    SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.arXiv preprint arXiv:2502.14786. Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing

    Pith/arXiv arXiv

  7. [14]

    Yuqing Zhou and Ziwei Zhu

    Point mamba: A novel point cloud backbone based on state space model with octree-based ordering strategy.arXiv preprint arXiv:2403.06467. Yuqing Zhou and Ziwei Zhu

    Pith/arXiv arXiv

  8. [15]

    Fighting spurious correlations in text classification via a causal learning perspective. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4264–4274. Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xi...

    2025

  9. [2009]

    Technical report, University of Toronto

    Learn- ing multiple layers of features from tiny images. Technical report, University of Toronto. 10 Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. 2024a. VideoMamba: State space model for efficient video understanding. InEuropean Conference on Computer Vision, pages 237–255. Springer. Qizhou Li, Weiran Huang, Jian Lin, Yan...

    Pith/arXiv arXiv

  10. [2019]

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692. Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, Jianbin Jiao, and Yunfan Liu

    Pith/arXiv arXiv 1907

  11. [2020]

    2021 ieee

    The many faces of robustness: A critical analysis of out-of-distribution generalization. 2021 ieee. InCVF International Conference on Computer Vision (ICCV), volume

    2021

  12. [2021]

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.arXiv preprint arXiv:2111.02114. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu

    Pith/arXiv arXiv

  13. [2022]

    at different resolutions. A.4 Dense Captioning Retrieval Image Retrieval Text Retrieval Vision Tower Text TowerR@1 R@5 R@10 R@1 R@5 R@10 FlexViT LLaMA 46.4 75.0 84.5 55.7 79.0 87.8NaFlex LLaMA 50.9 80.2 88.9 66.1 86.8 93.8ViT LLaMA 53.9 82.0 90.1 64.8 85.8 92.0RoPE-ViT LLaMA 60.7 85.8 93.3 77.6 93.6 97.0 VMamba Mamba-2 63.3 88.2 94.6 75.7 92.6 96.2VMamba ...

    2024

  14. [2023]

    Camilo Tamayo-Rousseau, Yunjia Zhao, Yiqun Zhang, and Randall Balestriero

    EV A-CLIP: Improved training techniques for CLIP at scale.arXiv preprint arXiv:2303.15389. Camilo Tamayo-Rousseau, Yunjia Zhao, Yiqun Zhang, and Randall Balestriero

    Pith/arXiv arXiv

  15. [2024]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei

    Naflex: Training-free flexible image classification.arXiv preprint arXiv:2406.04662. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei

    Pith/arXiv arXiv

  16. [2025]

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt

    Your attention matters: to improve model robustness to noise and spurious correlations.arXiv preprint arXiv:2507.20453. Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt

    Pith/arXiv arXiv