Pith. sign in

REVIEW 3 major objections 6 minor 58 references

(Almost) Free Modality Stitching of Foundation Models

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single hypernetwork, jointly trained across all model pairings, matches grid-search ranking and connector quality while cutting the search cost by an order of magnitude.

desk verdict Useful and honest approach for cheap VLM model-pair search, but the abstract overclaims by saying it matches grid-search connector performance when Table 3 shows a systematic 2–6 point gap. read the letter →

arxiv 2507.10015 v3 pith:7BJ2K2YL submitted 2025-07-14 cs.CV cs.AIcs.LG

Jaisidh Singh , Diganta Misra , Boris Knyazev , Antonio Orvieto This is my paper
classification cs.CVcs.AIcs.LG
keywords hypernetworksmodelstitchingvision-languagemodelsmultimodalalignmentselectionconnectortrainingcontrastivelearningefficientsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes on the pairing problem in multimodal model design: given a set of pretrained image encoders and a set of text encoders, which pair should be stitched together, and how should the stitching connector be trained? It proposes Hypernetwork Model Alignment (HYMA), a hypernetwork that predicts connector weights for every image-text pair at once. The central claim is that this joint training both identifies the best-performing pair and produces connectors nearly as good as individually trained ones, at roughly one-tenth of the FLOPs of a full grid search. The claim is tested on up to 27 image-text pairs across classification, retrieval, and question answering benchmarks. A sympathetic reader would care because model selection cost is the practical bottleneck when foundation models are assembled by stitching rather than trained from scratch.

What carries the argument

The hypernetwork is an MLP $F_\varrho$ that predicts connector parameters layer by layer. Each layer prediction is conditioned on the sum of a learnable pair embedding $c_k = W^\sigma[k]$ and a learnable layer embedding $e_j = E^\omega[j]$; the output is sliced to the layer's shape and concatenated into the full connector parameter vector. Mini-batching over model combinations makes joint training tractable: at each step a batch of $B_m$ pairs is sampled and the same data batch is passed through the generated connectors, so the hypernetwork sees the whole dataset across steps while each individual pair sees only a subset. This dual mini-batching is what converts the $N \times M$ training problem into something cheaper than grid search.

What would settle it

A concrete check: on the $N \times M = 27$ VLM zoo, compute the Spearman rank correlation between HYMA's predicted pair ranking and the grid-search oracle ranking after a full training run of both; the paper reports values around 0.87-0.99 across tasks. If on a new zoo of comparable size the correlation drops toward chance level, or if the best HYMA connector's gap to the best independently trained connector exceeds the 1.85-5.91 point range reported here, the 'almost free' claim would need to be restricted to the tested encoder families.

Watch

Extended reading notes

Core claim

HYMA treats model-pair search as parameter prediction. A hypernetwork $H_\phi$ takes a learnable embedding $c_k$ for each of the $N \times M$ encoder combinations and outputs the weights $\theta_k$ of an MLP connector $f_{\theta_k}$; the connector is trained with the contrastive InfoNCE loss while the encoders stay frozen. Because all connectors share one generator, a single training run yields both a ranking of all pairs and usable connectors for each pair. On the vision-language experiments, the paper reports that HYMA's ranking closely matches the grid-search oracle (high NDCG@k and Spearman's $\rho$), and the best HYMA connector lands within a few points of the best individually trained connector, while the FLOP cost is about $10\times$ lower. The same recipe does not transfer to causal language modeling: in the appendix, connectors generated by HYMA for multimodal LLMs underperform independently stitched connectors by a clear margin, and the predicted ranking drifts toward uni-modal performance.

Load-bearing premise

One shared hypernetwork can generate connectors for all model pairs without losing much quality compared with training each connector separately; the paper's own appendix shows this assumption fails in the causal language modeling setting, where HYMA connectors underperform independent stitching.

Editorial extensions

If this is right

  • On the tested vision-language zoos, model-pair selection no longer requires training every pair: a single HYMA run ranks all 27 pairs like the grid-search oracle, at about one-tenth of the FLOPs.
  • The connectors HYMA generates land within a few accuracy points of connectors trained individually for the best pair, while beating random pairing, best-unimodal selection, and an LLM-based selector across classification, retrieval, and VQA benchmarks.
  • A budget-matched iterative search baseline (AutoPair) finishes with noticeably weaker connectors than HYMA at the same FLOP cost.
  • Giving grid search the same per-pair data budget as HYMA makes grid search worse than HYMA, suggesting the benefit is not simply less data per pair.
  • Because the connector objective is not tied to image and text, the same hypernetwork formulation extends to stitching other modalities such as audio and text, at least in principle.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 10x figure is measured on MLP connectors for contrastive VLMs; if connectors grow to transformer-scale projectors, the hypernetwork's own cost and training instability could erode or reverse the saving.
  • HYMA's per-pair data exposure shrinks with $N \times M$, so part of its advantage may be a form of implicit data pruning; varying $B_m$ at fixed total compute would separate that effect from the benefit of parameter sharing.
  • A natural two-stage recipe the paper does not test is to use HYMA for ranking and then fine-tune only the top few generated connectors; this could recover the small gap to grid search at modest extra cost.
  • The MLLM results suggest the shared generator struggles when the stitching objective is causal language modeling rather than contrastive alignment; extending the claim to generative multimodal LLMs likely needs per-pair adapters or a differently conditioned hypernetwork.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces HYMA, a hypernetwork-based method that jointly generates connector parameters for all N x M pairs of pretrained uni-modal encoders, targeting what the authors call Multi-modal Optimal Pairing and Stitching (M-OPS). The hypernetwork is trained end-to-end with a contrastive InfoNCE loss on LLaVA-CC558K while mini-batching over model pairs; after training, the generated connectors are used both to rank the N x M pairs and to evaluate the best pair on downstream benchmarks. The experiments compare HYMA against random pairing, uni-modal top-1 selection, an LLM-prompted selector, a search-and-prune baseline (AutoPair), and full grid search (Oracle) across three connector architectures and four task families. The authors claim a 10x FLOP reduction and high ranking correlation with grid search on the VLM benchmarks, and they include an appendix applying HYMA to multimodal large language models (MLLMs).

Significance. The paper addresses a well-defined and practically important problem: selecting and stitching uni-modal encoders without exhaustively training all pairs. The evaluation is broad, covering multiple baselines, three connector capacities, and six downstream datasets, and the efficiency comparison is quantified in FLOPs. The ranking evidence for the VLM setting is strong: NDCG@k and Spearman rho values in Table 3 are mostly high, and because HYMA is trained on LLaVA-CC558K while the ranking benchmarks are unseen, this comparison is not circular. The limitations section is candid about training instability, including the exclusion of MaxViT models. The main weakness is that the abstract's claim of matching grid-search connector performance is not supported by Table 3, and the MLLM appendix shows a clear failure to generalize. With a recalibrated claim, the paper would be a solid empirical contribution; as written, it overstates the evidence.

major comments (3)
  1. [Abstract; Section 5.2, Table 3] The abstract claims that HYMA "matches" the ranking and trained connector performance obtained via grid search, but every entry in the Oracle (GS) Delta column of Table 3 is negative. Averaged over Linear, MLP1, and MLP2, the best HYMA connector underperforms grid search by 4.84 points on ImageNet-1K, 2.88 points on CIFAR-100, 3.03 recall@5 points on MSCOCO, and 2.52 recall@5 points on Flickr-8K. The Section 5.2 text itself acknowledges these gaps, and the Introduction's "on par" phrasing makes the same overclaim. The ranking evidence is strong, but the performance claim should be revised to something like "approaches" or "slightly underperforms" the oracle, or the paper should explicitly frame the results as a cost-accuracy trade-off.
  2. [Appendix A.2, Table 6] In the MLLM (causal language modeling) setting, HYMA does not match grid search either in ranking or in perplexity: Spearman's rho is -0.6 for Linear connectors on MSCOCO, and the text states that independently stitched connectors outperform HYMA "by a significant margin." This is a direct counterexample to the unqualified abstract claim that HYMA matches ranking and trained connector performance across a suite of multi-modal benchmarks. The claims should be restricted to the VLM/contrastive stitching setting, and the MLLM results should be presented as a boundary of the method rather than as supporting evidence.
  3. [Limitations; Appendix D] The advertised all-in-one property is qualified by the statement in the Limitations section that including MaxViT models caused training instability and led to their exclusion from the final zoo. The main efficiency results also depend on the model-mini-batch size Bm, with Bm=1 for N x M = 3 and Bm=9 for N x M = 27. The paper does not provide a criterion for which model pools are admissible, nor an ablation over Bm. Without this, the 10x-reduction claim is tied to a specific, curated zoo and may not generalize to the open-ended model collection that motivates the paper.
minor comments (6)
  1. [Table 6 caption] The caption says that HYMA exhibits strong correlation with grid-search rankings, but the table contains Spearman values as low as -0.6 and 0.12; the caption should be corrected or qualified to match the reported numbers.
  2. [Section 4.3] The Linear connector is described as mapping from the text-encoder embedding space to the image-encoder space, while Definition 2 and the methodology in Section 3.1 define the connector as mapping modality-A features to modality-B; please make the direction consistent throughout.
  3. [Throughout] There are numerous typos and inconsistencies, including "with an text model," "EV A-2" in Table 1, "LaV A" in Section 4.4, "N x = 12" in Appendix D, and "performative" in Section 5.1; a careful proofread is needed.
  4. [Section 5.4] The comparison with the constrained grid search (C-GS) is interesting, but it is not fully clear whether the one-third data budget refers to the same number of optimizer steps or the same number of samples per pair for both methods; please clarify the exact matching of computational budgets.
  5. [Appendix G.1] The Ask-LLM baseline uses a specific proprietary model (Claude 4 Sonnet) without a version date or reproducibility details; please document the exact model version and access date.
  6. [References] Some references are incomplete, for example the Minderer et al. entry lacks a year and the Anthropic reference is only a URL; these should be completed.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: HYMA's rankings and connector performance are measured from generated connectors, not derived from the method's own inputs.

full rationale

HYMA's claimed derivation chain is an empirical method, not a derivation: a hypernetwork H_phi is trained jointly over all N x M pairs with loss L_HYMA = (1/NM) sum_k L_task(f_{H_phi(c_k)}), and the resulting connectors are then evaluated on held-out downstream benchmarks (ImageNet-1K, CIFAR-100, MSCOCO, Flickr-8K, etc.). The ranking reported in Table 3 is obtained by actually scoring the generated connectors, not by fitting those benchmarks. The hypernetwork is not defined in terms of the downstream ranking; its conditioning inputs are a learnable lookup table over pair indices, and its training target is the contrastive InfoNCE loss on LLaVA-CC558K. Consequently, the 'prediction' of the best pair is an approximate result of joint training, not a restatement of its inputs. No parameter is fitted to the evaluated benchmarks and then relabeled a prediction. The only notable self-citation is model mini-batching from Knyazev et al. (2023), a co-authored prior work, but this is a training-scale implementation detail and is not the load-bearing claim that HYMA matches grid-search ranking and connector performance; removing or replacing it would not make the central result true by construction. The paper's own limitations (MaxViT instability, Appendix A.2 MLLM underperformance) weaken the strength of the empirical claim, and Table 3's uniformly negative Oracle deltas show the abstract overstates 'matching trained connector performance,' but these are correctness and robustness concerns, not circularity. Verdict: no significant circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

HYMA is an empirical ML method, so the ledger contains design choices and background assumptions rather than derived constants or invented physical entities. The key free choices are the model mini-batch size (which drives the efficiency claim), the tuned beta_2, training hyperparameters, and the exclusion of MaxViT models. The axioms are the frozen-encoder assumption, the transferability of the contrastive training objective, the shared-hypernetwork expressiveness assumption, and the FLOPs accounting assumption.

free parameters (4)
  • B_m (model mini-batch size) = 1 (N=3), 9 (N=27), 4 (N=12 AutoPair)
    Controls how many model pairs share each data batch. The claimed efficiency gain (e.g., 10x) scales with N*M/B_m, so the central cost claim depends on this hand-chosen value.
  • beta_2 of Adam (hypernetwork) = tuned, exact value not reported
    The Limitations section states beta_2 was tuned to stabilize training but the exact value is not given, making it a hidden free choice that affects reproducibility and performance.
  • Training epochs and learning rate = 10 epochs, lr 1e-2 (VLM), 1e-3 (MLLM)
    Selected per setting in Appendix D for best validation performance and stability; these choices determine both final performance and the FLOPs efficiency numbers.
  • Exclusion of MaxViT models from zoo = excluded
    The authors state that including MaxViT caused training instability, so these models were removed from the model zoo. This is a post-hoc selection that narrows the claimed generality of the method.
assumptions (4)
  • domain assumption Pretrained unimodal encoders are frozen and the connector alone is sufficient to align their representation spaces.
    Standard in the stitching literature (Section 2, Definition 2); if a connector cannot express the needed alignment, HYMA's generated connectors would be limited regardless of training.
  • domain assumption The InfoNCE contrastive loss on LLaVA-CC558K is a valid proxy for downstream zero-shot performance on ImageNet, CIFAR, retrieval, and VQA benchmarks.
    The ranking of pairs produced by HYMA is only useful if training signal transfers to the evaluation tasks; this is an empirical assumption not proven in the paper.
  • ad hoc to paper A single hypernetwork can generate parameters for all N x M connectors with acceptable quality, i.e., no pair needs individually trained parameters to perform well.
    This is the central inductive-bias assumption of HYMA. The paper's own Limitations (MaxViT instability) and Appendix A.2 (MLLM failure) show this assumption is fragile.
  • domain assumption FLOPs are counted consistently across baselines (Appendix E), including the hypernetwork's own forward and backward cost.
    The 10x efficiency claim depends entirely on this accounting, and the paper's accounting is complex and not independently verifiable from the text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of (Almost) Free Modality Stitching of Foundation Models." pith.science (2026). https://pith.science/paper/7BJ2K2YL

@misc{pith2026250710015,
  author       = {Pith},
  title        = {Pith review of: (Almost) Free Modality Stitching of Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7BJ2K2YL}},
  note         = {Machine review of arXiv:2507.10015}
}
abstract

Foundation multi-modal models are often designed by stitching of multiple existing pretrained uni-modal models: for example, an image classifier with an text model. This stitching process is performed by training a connector module that aims to align the representation spaces of these uni-modal models towards a multi-modal objective. However, given the complexity of training such connectors on large scale web-based datasets coupled with the ever-increasing number of available pretrained uni-modal models, the task of uni-modal models selection and subsequent connector module training becomes computationally demanding. To address this under-studied critical problem, we propose Hypernetwork Model Alignment (Hyma), a novel all-in-one solution for optimal uni-modal model selection and connector training by leveraging hypernetworks. Specifically, our framework utilizes the parameter prediction capability of a hypernetwork to obtain jointly trained connector modules for $N \times M$ combinations of uni-modal models. In our experiments, Hyma reduces the cost of searching for the best performing uni-modal model pair by $10\times$, while matching the ranking and trained connector performance obtained via grid search across a suite of diverse multi-modal benchmarks.

Figures

Figures reproduced from arXiv: 2507.10015 by the authors.

Figure 1
Figure 1. We train connectors between pretrained uni [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Given multiple options for uni-modal models, pair-wise grid search can be an expensive way to determine [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. A visual walkthrough of our hypernetwork architecture is provided above. We take the example of [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: MLP1 | N × M = 3: We show the trade-off between computational resources (measured in FLOPs) and performance of the best stitched model pairs across all comparative baselines. We find that HYMA is able to predict a highly performance pairing at a significantly reduced F…
Figure 5
Figure 5. Figure 5: Evaluation of HYMA for MLLMs, on MSCOCO and Flickr-8K (N = 1, M = 3, Bm = 1). We report the model combination exhibiting the best final performance for each evaluation benchmark and search method. Dataset Connector NDCG @ k (↑) ρ (↑) ∆Perplexity (↓) k=5 k=7 k=9 N×M=9 R…
Figure 6
Figure 6. Figure 6: PyTorch (Paszke et al., 2019) pseudocode for HYMA training procedure on N × M models. up the learning rate from 0 for 50 steps. After that, the learning rate is decayed to 0 following a cosine curve. Training HYMA for VLMs is quite sensitive to hyperparameters, as is t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 28 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Victor Akinwande, Mohammad Sadegh Norouzzadeh, Devin Willmott, Anna Bair, Madan Ravi Ganesh, and J Zico Kolter. 2024. Hyperclip: Adapting vision-language models with hypernetworks. arXiv preprint arXiv:2412.16777

    work page Pith review arXiv 2024

  4. [4]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, and 8 others. 2022. Flamingo: a visual language model for few-s...

    work page 2022

  5. [5]

    Claude 4 sonnet

    Anthropic. Claude 4 sonnet. https://www.anthropic.com/claude

    work page

  6. [6]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    arXiv 2023

  7. [7]

    Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025. https://arxiv.org/abs/2502.12119 Prism: Self-pruning intrinsic selection method for training-free multimodal data selection . Preprint, arXiv:2502.12119

    arXiv 2025

  8. [8]

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397--2430. PMLR

    2023

Show all 58 references
  1. [9]

    Cattaneo and Boris Shigida

    Matias D. Cattaneo and Boris Shigida. 2025. T uning adam(w): D efault β2 may be too large. https://mdcattaneo.github.io/papers/Cattaneo-Shigida_2025_TuningAdam.pdf

    2025

  2. [10]

    Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu. 2023. Stablevideo: Text-driven consistency-aware diffusion video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 23040--23050

    2023

  3. [11]

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558--3568

    2021

  4. [12]

    Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. 2024. A brief review of hypernetworks in deep learning. Artificial Intelligence Review, 57(9):250

    2024

  5. [13]

    Keyan Chen, Xiaolong Jiang, Yao Hu, Xu Tang, Yan Gao, Jianqi Chen, and Weidi Xie. 2023. Ovarnet: Towards open-vocabulary object attribute recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23518--23527

    2023

  6. [14]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248--255. IEEE

    2009

  7. [15]

    Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. 2021. Redcaps: Web-curated image-text data created by the people, for the people. arXiv preprint arXiv:2111.11431

    2021 arXiv

  8. [16]

    Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. 2023. https://arxiv.org/abs/2309.17425 Data filtering networks . Preprint, arXiv:2309.17425

    2023 arXiv

  9. [17]

    Andrey Guzhov, Federico Raue, J \"o rn Hees, and Andreas Dengel. 2022. Audioclip: Extending clip to image, text and audio. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 976--980. IEEE

    2022

  10. [18]

    David Ha, Andrew Dai, and Quoc V Le. 2016. Hypernetworks. arXiv preprint arXiv:1609.09106

    2016 arXiv

  11. [19]

    Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853--899

    2013

  12. [20]

    Junlong Jia, Ying Hu, Xi Weng, Yiming Shi, Miao Li, Xingjian Zhang, Baichuan Zhou, Ziyu Liu, Jie Luo, Lei Huang, and 1 others. 2024. Tinyllava factory: A modularized codebase for small-scale large multimodal models. arXiv preprint arXiv:2405.11788

    2024 arXiv

  13. [21]

    Boris Knyazev, Doha Hwang, and Simon Lacoste-Julien. 2023. Can we scale transformers to predict parameters of diverse imagenet models? In International Conference on Machine Learning, pages 17243--17259. PMLR

    2023

  14. [22]

    Alex Krizhevsky, Geoffrey Hinton, and 1 others. 2009. Learning multiple layers of features from tiny images

    2009

  15. [23]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  16. [24]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

    2022

  17. [25]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740--755. Springer

    2014

  18. [26]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.04489

    2021 arXiv

  20. [28]

    Anas Mahmoud, Mostafa Elhoushi, Amro Abbas, Yu Yang, Newsha Ardalani, Hugh Leather, and Ari Morcos. 2024. https://arxiv.org/abs/2310.02110 Sieve: Multimodal dataset pruning using image captioning models . Preprint, arXiv:2310.02110

    2024 arXiv

  21. [29]

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195--3204

    2019

  22. [30]

    Jack Merullo, Louis Castricato, Carsten Eickhoff, and Ellie Pavlick. 2022. Linearly mapping from image to text space. arXiv preprint arXiv:2209.15162

    2022 arXiv

  23. [31]

    Simple open-vocabulary object detection with vision transformers

    M Minderer, A Gritsenko, A Stone, M Neumann, D Weissenborn, A Dosovitskiy, A Mahendran, A Arnab, M Dehghani, Z Shen, and 1 others. Simple open-vocabulary object detection with vision transformers. arxiv 2022. arXiv preprint arXiv:2205.06230

    2022 arXiv

  24. [32]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    2018 arXiv

  25. [33]

    Jose Javier Gonzalez Ortiz, John Guttag, and Adrian Dalca. 2023. Magnitude invariant parametrizations improve hypernetwork learning. arXiv preprint arXiv:2304.07645

    2023 arXiv

  26. [34]

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing...

    2019

  27. [35]

    Jason Phang, Yi Mao, Pengcheng He, and Weizhu Chen. 2023. Hypertuning: Toward adapting large language models without back-propagation. In International Conference on Machine Learning, pages 27854--27875. PMLR

    2023

  28. [36]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine...

    2021

  29. [37]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, and 1 others. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  30. [38]

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text-conditional image generation with clip latents, 2022. arXiv preprint arXiv:2204.06125

    2022 arXiv

  31. [39]

    Nils Reimers and Iryna Gurevych. 2019. http://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  32. [40]

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj \"o rn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684--10695

    2022

  33. [41]

    Elan Rosenfeld, Preetum Nakkiran, Hadi Pouransari, Oncel Tuzel, and Fartash Faghri. 2022. Ape: Aligning pretrained encoders to quickly learn aligned multimodal representations. arXiv preprint arXiv:2210.03927

    2022 arXiv

  34. [42]

    J \"u rgen Schmidhuber. 1992. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131--139

    1992

  35. [43]

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, and 1 others. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in N...

    2022

  36. [44]

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114

    2021 arXiv

  37. [45]

    Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. 2021. How much can clip benefit vision-and-language tasks? arXiv preprint arXiv:2107.06383

    2021 arXiv

  38. [46]

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. 2022. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15...

    2022

  39. [47]

    Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317--8326

    2019

  40. [48]

    no" to say

    Jaisidh Singh, Ishaan Shrivastava, Mayank Vatsa, Richa Singh, and Aparna Bharati. 2024. Learn" no" to say" yes" better: Improving vision-language models via negations. arXiv preprint arXiv:2403.20312

    2024 arXiv

  41. [49]

    Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. 2022. https://arxiv.org/abs/2203.07190 Clip models are few-shot learners: Empirical studies on vqa and visual entailment . Preprint, arXiv:2203.07190

    2022 arXiv

  42. [50]

    Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. 2016. Yfcc100m: The new data in multimedia research. Communications of the ACM, (2):64--73

    2016

  43. [51]

    Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. 2022. Maxvit: Multi-axis vision transformer. In European conference on computer vision, pages 459--479. Springer

    2022

  44. [52]

    Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. 2022. Omnivl: One foundation model for image-language and video-language tasks. In Advances in Neural Information Processing Systems, volume 35, pages 569...

    2022

  45. [53]

    Ross Wightman. 2019. https://doi.org/10.5281/zenodo.4414861 Pytorch image models . https://github.com/rwightman/pytorch-image-models

    2019 doi

  46. [54]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and ...

    2020 arXiv

  47. [55]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975--11986

    2023

  48. [56]

    Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, and 1 others. 2024. Hyperllava: Dynamic visual and language expert tuning for multimodal large language models. arXiv preprint arXiv:2403.13447

    2024 arXiv

  49. [57]

    Zhengkun Zhang, Wenya Guo, Xiaojun Meng, Yasheng Wang, Yadao Wang, Xin Jiang, Qun Liu, and Zhenglu Yang. 2022. Hyperpelt: Unified parameter-efficient language model tuning for both language and vision-and-language tasks. arXiv preprint arXiv:2203.03878

    2022 arXiv

  50. [58]

    Xun Zhu, Zheng Zhang, Xi Chen, Yiming Shi, Miao Li, and Ji Wu. 2025. Connector-s: A survey of connectors in multi-modal large language models. arXiv preprint arXiv:2502.11453

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.