Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval

Gyu-Hwung Cho; Ilhyeon Song; Jonghyun Song; Saehun Kim; Yohan Jo; Youngjune Lee

REVIEW 3 major objections 6 minor 38 references

Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that jointly training sparse and dense retrievers, with a weighted integrated score as a shared teacher, lets the sparse retriever match or beat fully fine-tuned dense models on MSCOCO and Flickr30k while also improving th

desk verdict Useful sparse-retrieval recipe with an honestly good ablation, but the conclusion overstates the dense and sparse-vs-dense results; fix Table 2 interpretation and report hyperparameters. read the letter →

arxiv 2508.16707 v1 pith:2ZQVKHLA submitted 2025-08-22 cs.CL cs.IRcs.LG

Jonghyun Song , Youngjune Lee , Gyu-Hwung Cho , Ilhyeon Song , Saehun Kim , Yohan Jo This is my paper

classification cs.CLcs.IRcs.LG

keywords learnedsparseretrievalcross-modaltext-imageself-knowledgedistillationvision-languagepre-trainingdense-sparsejointoptimizationinvertedindex

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that sparse and dense retrieval representations in vision-language models are not rivals to be trained separately: if you optimize them together, each gets better. It introduces a shared integrated score, a weighted sum of dense and sparse similarities, that acts as the teacher for both via self-knowledge distillation. On MSCOCO and Flickr30k, the resulting sparse retriever beats prior sparse baselines and often matches or exceeds the dense backbone, while the dense retriever also improves. This matters because sparse retrieval is cheaper to serve and more interpretable; if sparse can match dense quality, large-scale multimodal search can run on inverted indexes.

What carries the argument

The central mechanism is the integrated similarity score s_inter = w1·s_dense + w2·s_sparse, a weighted sum that serves as a single soft teacher in a self-knowledge distillation loss for both s_dense and s_sparse. The same integrated score also participates in a triple contrastive loss. The sparse path is a shared MLP projection head f with ReLU and log transformations, initialized with the transposed word-embedding matrix, turning [CLS] embeddings into vocabulary term weights; only the final encoder layer and the head are trainable. This setup makes the teacher a moving, mutually informed target rather than a frozen dense model.

What would settle it

Re-run the exact recipe with the shared projection head initialized randomly instead of with the transposed text word-embedding matrix, keeping everything else fixed; if R@1 does not drop materially, the assumed text-vocabulary basis is not doing the work. Alternatively, replace the integrated teacher with a fixed ensemble of separately trained dense and sparse models; if the reported gains vanish, the claimed bidirectional self-distillation is not the source of improvement.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that bidirectional self-distillation works. Instead of freezing a dense encoder and training a sparse projection head to imitate it, the paper has three similarity scores, dense, sparse, and their weighted integration, trained jointly under a contrastive loss, with the integrated score supervising the two individual scores. The sparse projection head, initialized from the transposed word-embedding matrix of the frozen text encoder, maps both text and image [CLS] embeddings to vocabulary weights; only the final encoder layer and this head are fine-tuned. The result is that on MSCOCO and Flickr30k, the BLIP-backed sparse retriever reaches R@1 of 57.6

Load-bearing premise

The load-bearing premise is that projecting image embeddings through the text-word-embedding-initialized projection head yields meaningful token weights; if image and text [CLS] embeddings do not share a semantic basis in that vocabulary space, the sparse scores are noise and the integrated teacher would poison the dense retriever instead of improving it.

Editorial extensions

If this is right

Sparse retrieval can be made competitive with dense retrieval on standard image-text datasets without expensive end-to-end contrastive pretraining; only the final encoder layer and a projection head are fine-tuned.
Dense retrievers benefit from co-training with sparse signals, so the two representation types can mutually improve rather than dense always being the teacher.
Because sparse scores support inverted-index lookup, a competitive sparse retriever can serve cross-modal search with lower query-time compute than dense retrieval.
The framework is architecture-agnostic: any VLP model can be adapted by loading dataset-fine-tuned checkpoints and fine-tuning two modules.
The weight analysis indicates the sparse signal should not be treated as a weak auxiliary: larger sparse weight in the integrated teacher generally improves R@1.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the paper leaves implicit is that the text encoder's word-embedding space may serve as a common semantic basis for both modalities, since image embeddings projected through it produce useful sparse retrieval scores; this could be tested by measuring how well the sparse scores generalize to vocabulary far from the pretraining distribution.
The method's reliance on the transposed text word-embedding initialization is a testable vulnerability: on specialized domains such as medical or e-commerce imagery, a randomly initialized or modality-specific projection basis might behave very differently.
Because only the final encoder layer is updated, the observed gains behave like a low-rank adaptation of the VLP encoder; unfreezing deeper layers could amplify the mutual enhancement or destabilize the pretrained features, and the paper does not explore that trade-off.
A natural extension is to apply the same joint objective to text-only retrieval or to reranking pipelines where dense and sparse components already coexist, using the integrated score as a teacher to unify their training.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the core idea—bidirectional self-distillation between dense and sparse scores with an integrated weighted teacher—is new enough, and the sparse-retrieval gains over D2S are consistent and statistically tested. The paper deserves a serious referee. But the conclusion oversells: Table 2 shows the sparse retriever beats the dense backbone in only one of four R@1 comparisons (BLIP/MSCOCO), and the dense model degrades on BLIP/Flickr. The 'often surpassing' language doesn't survive contact with their own table.

What's actually good. The method is simple: fine-tune only the final layer of the VLP encoder plus a sparse projection head initialized with the transposed word-embedding matrix. The integrated teacher is a weighted sum of dense and sparse similarities, and both students are trained against it. That composes existing ingredients (self-KD, distillation, sparse projection) but the combination is new relative to D2S's unidirectional dense-to-sparse setup. The ablation shows both components help (Table 3). They reimplemented D2S under the same conditions, which is the right comparison, and the daggers for statistical significance are a plus. Training cost is honestly reported: 4 hours on one RTX 3090.

Soft spots. The main one is the mismatch between Section 4 and Table 2. The sparse retriever is below the dense backbone on three of four R@1 numbers. The dense retriever improves in three of four, but not on BLIP/Flickr. So 'often surpassing' is wrong for sparse, and 'improves beyond original backbones' needs a footnote. That's a load-bearing overstatement because the 'Learn Better Together' framing leans on it. Fixing it is easy: qualify the claims, or report average recall across datasets. Second, hyperparameters (λ1,2,3, w1,2, η, τ, learning rates) are described as 'tuned' but never listed. For a 5-page paper that's less excusable than for a longer one. Third, no seed variance; given the small absolute gains (0.2–1.7 R@1), a single seed is thin. The integrated teacher's self-referential nature is a legitimate conceptual worry, but it doesn't undermine the contribution because the benchmark recall is externally grounded—the circularity affects interpretation, not the empirical claim.

Who it's for: people working on learned sparse retrieval or efficient VLP fine-tuning. It's not a field-reshaping paper, but it advances a practical recipe. As a reviewer I'd ask for the conclusion rewrite, hyperparameters, and seeds; the core method and ablation are solid.

Referee Report

3 major / 6 minor

Summary. The paper proposes a lightweight framework for jointly optimizing dense and sparse retrievers in text-image retrieval. Starting from a VLP backbone (BLIP or ALBEF), the method fine-tunes only the final encoder layer and a sparse projection head, while using a weighted sum of dense and sparse similarity scores as an integrated teacher for self-knowledge distillation. The training objective combines contrastive losses on dense, sparse, and integrated scores, a distillation loss from the integrated score to both dense and sparse scores, and L1 sparsity regularization. Experiments on MSCOCO and Flickr30k report sparse retrieval improvements over VisualSparta, LexLIP, and D2S, as well as dense retrieval improvements over the backbone in most settings. The abstract and conclusion claim that the sparse retriever 'often surpasses' dense models and that the same training 'improves the performance of the dense models beyond their original backbones.'

Significance. If the claims are supported, the paper contributes a simple, efficient recipe for adapting existing VLP models to learned sparse retrieval, with a bidirectional self-distillation mechanism. The code is publicly available, D2S is re-implemented under matched conditions, and the ablation study shows that both the distillation objective and final-layer fine-tuning contribute to sparse retrieval performance. The sparse-vs-sparse improvements in Table 1 are consistently positive and statistically significant, which is a solid empirical contribution. However, the headline claims about surpassing dense models and improving dense models are only partially supported by the reported numbers, as detailed below. The method itself is interesting and potentially practical, but the evidence needs to be presented more carefully.

major comments (3)

[Section 4, Table 2] The claim that 'Our sparse retriever...often surpassing fully fine-tuned dense models' is contradicted by Table 2. For R@1, Ours (Sparse) exceeds the dense backbone in only one of four configurations (BLIP/MSCOCO: 57.6 vs 57.0); it is lower in ALBEF/MSCOCO (53.2 vs 53.3), ALBEF/Flickr (78.6 vs 79.1), and BLIP/Flickr (82.0 vs 83.2). Likewise, 'also improves the performance of the dense models beyond their original backbones' is not supported by the BLIP/Flickr row of Ours (Dense), where R@1 drops from 83.2 to 82.8 and M@10 from 89.3 to 88.7. This is load-bearing for the 'Learn Better Together' framing. The conclusion and abstract should be qualified to report the mixed results, and the paper should explicitly discuss the configurations where sparse or dense degrade.
[Section 3.1, Eq. (3)] Hyperparameters are not reported. The objective in Eq. (3) contains λ1, λ2, λ3, w1, w2, η_t, η_i, and the dense score uses a learnable temperature τ. The text states that models were 'tuned using the same set of hyperparameters' but gives no values, learning rate, optimizer, or the quadratic schedule for η_t/η_i. Since the method is a fine-tuning recipe, these settings are essential for reproducibility and for interpreting the sensitivity of the approach. Please provide the complete configuration in the final version or an appendix.
[Table 2] The dense comparison in Table 2 is reported without significance tests or variance estimates, even though differences are small (often 0.1–1.5 R@1). For example, Ours (Dense) improves ALBEF/MSCOCO by 1.2 R@1 and BLIP/MSCOCO by 1.7 R@1, but drops 0.4–0.6 R@1 on BLIP/Flickr. The sparse comparisons in Table 1 are daggered with a paired t-test, but the dense claims rely on raw point estimates. Reporting significance, confidence intervals, or results over multiple seeds would make the dense-improvement claim supportable.

minor comments (6)

[Section 2.3, Eq. (2)] The notation L^{s*}_{a→b} is slightly confusing because the superscript * is used as a placeholder. Consider writing L^{s_dense}, L^{s_sparse}, L^{s_inter} explicitly.
[Table 3] The row order (✓ ✗, ✗ ✓, ✓ ✓) with the caption '✓ ✓ is our best configuration' is hard to parse. Label the rows (e.g., 'Distillation only', 'Fine-tuning only', 'Full model') or add a column for each component.
[Section 3.1, Tables 1–2] The text says the evaluation metric is MRR@10, but the tables and later text use 'M@10'. Standardize the notation.
[Figure 2 (Right)] The heatmap has no axis labels, colorbar, or specified ranges for w1 and w2. The claim that 'higher weight to the sparse score generally leads to better R@1' is difficult to verify without the figure. Add concrete values and a legend.
[Section 2.3] The integrated score s_inter = w1·s_dense + w2·s_sparse is used as the teacher for both dense and sparse students. Since s_inter includes the student's own score, this is a form of self-distillation; the paper should discuss the potential for degenerate solutions (e.g., if w1≫w2, the dense student is trained to match itself) and how the weight values are chosen to avoid this.
[Section 3.3, Figure 2 (Left)] The description refers to 'Variant 1 and 2 represent earlier and later checkpoints from the same training run' but the figure does not show which checkpoint each variant corresponds to. Clarify the training stage or checkpoint selection.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the self-distillation teacher is self-referential by design, but the paper's central claims are empirically evaluated on external benchmarks and do not reduce to their inputs.

full rationale

No circular derivation chain is present. The integrated score s_inter = w1·s_dense + w2·s_sparse is used as a teacher in the distillation loss Ldistill(s_inter, s_*) for s_* in {s_dense, s_sparse}, so each student is trained toward a target that includes itself. This is genuinely self-referential as a training mechanism, but it is a standard self-knowledge distillation objective and not a logical derivation of the paper's claims. The paper's headline results are empirical: Recall@{1,5} and MRR@10 on MSCOCO and Flickr30k, obtained by training and then evaluating with sparse or dense scores. None of these numbers is forced by the objective's definition alone; the ablations and comparisons against external baselines provide independent evidence. There are no load-bearing self-citations: the cited prior methods (D2S, VisualSparta, LexLIP, BLIP, ALBEF) are external, and no uniqueness theorem or ansatz is imported from the authors' own prior work. The conclusion's language ('often surpassing fully fine-tuned dense models', 'improves the performance of the dense models beyond their original backbones') is not fully supported by Table 2 — e.g., Ours (Sparse) BLIP/Flickr R@1 is 82.0 vs BLIP's 83.2, and Ours (Dense) BLIP/Flickr R@1 is 82.8 vs 83.2 — but that is an overstatement or correctness concern, not circularity. The derivation chain from method to reported benchmark numbers is therefore self-contained, and no circular step can be exhibited.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The contribution rests on an engineered training objective with at least five tuned hyperparameters (w1, w2, lambda1-3, eta_t, eta_i, tau) whose values are not reported, plus two domain assumptions about projection-head initialization and final-layer-only fine-tuning. No new physical entities are postulated.

free parameters (6)

w1 = not reported
Weight of dense similarity in integrated score s_inter; tuned per dataset; Figure 2 explores it and concludes moderate values are best.
w2 = not reported
Weight of sparse similarity in integrated score s_inter; tuned; heatmap analysis says higher w2 generally helps, so choice is data-dependent.
lambda1, lambda2, lambda3 = not reported
Weights of contrastive losses for dense, sparse, and integrated scores in L = lambda1 L_dense + lambda2 L_sparse + lambda3 L_inter.
eta_t, eta_i = not reported (quadratically increased over 200 epochs)
L1 sparsity regularization weights in Eq. 3; schedule and final values not given.
tau = not reported
Learnable temperature in dense similarity; no value or initialization given.
learning rate and optimizer settings = not reported
All models are said to use the same hyperparameters, but no values are provided.

assumptions (6)

domain assumption The transposed word embedding matrix of the frozen text encoder is a meaningful initialization for the shared sparse projection head over images as well as text.
Section 2.2: 'we initialize the projection head with the transposed word embedding matrix from the frozen text encoder, guiding it to produce semantically meaningful token-level scores.' This assumes image [CLS] embeddings live in a space aligned with this text-vocabulary projection.
domain assumption Fine-tuning only the final encoder layer plus the projection head is sufficient to adapt the frozen VLP backbone without degrading its generalization.
Section 2.2: 'we preserve the strong generalization abilities of the pre-trained encoder, while efficiently adapting to sparse retrieval tasks.' No layer-sensitivity analysis is given.
ad hoc to paper The weighted sum s_inter = w1 s_dense + w2 s_sparse is a valid teacher for both dense and sparse students.
Section 2.3: 'This integrated score acts as a soft teacher signal...' The weights are tuned by hand; no theoretical grounding is given.
standard math InfoNCE contrastive objective with in-batch negatives is a valid training signal for both sparse and dense similarities.
Section 2.3, Eq. 2; standard contrastive loss used in prior retrieval work.
standard math Cross-entropy between softmax distributions of teacher and student scores is an appropriate distillation objective.
Section 2.3, Ldistill; standard knowledge-distillation formulation.
domain assumption L1 regularization on term weights induces useful sparsity without harming retrieval quality.
Section 2.3, Eq. 3; commonly used in SPLADE-style models, assumed to transfer to the multimodal setting.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval." pith.science (2026). https://pith.science/paper/2ZQVKHLA

@misc{pith2026250816707,
  author       = {Pith},
  title        = {Pith review of: Sparse and Dense Retrievers Learn Better Together: Joint Sparse-Dense Optimization for Text-Image Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ZQVKHLA}},
  note         = {Machine review of arXiv:2508.16707}
}

read the original abstract

Vision-Language Pretrained (VLP) models have achieved impressive performance on multimodal tasks, including text-image retrieval, based on dense representations. Meanwhile, Learned Sparse Retrieval (LSR) has gained traction in text-only settings due to its interpretability and efficiency with fast term-based lookup via inverted indexes. Inspired by these advantages, recent work has extended LSR to the multimodal domain. However, these methods often rely on computationally expensive contrastive pre-training, or distillation from a frozen dense model, which limits the potential for mutual enhancement. To address these limitations, we propose a simple yet effective framework that enables bi-directional learning between dense and sparse representations through Self-Knowledge Distillation. This bi-directional learning is achieved using an integrated similarity score-a weighted sum of dense and sparse similarities-which serves as a shared teacher signal for both representations. To ensure efficiency, we fine-tune the final layer of the dense encoder and the sparse projection head, enabling easy adaptation of any existing VLP model. Experiments on MSCOCO and Flickr30k demonstrate that our sparse retriever not only outperforms existing sparse baselines, but also achieves performance comparable to-or even surpassing-its dense counterparts, while retaining the benefits of sparse models.

Figures

Figures reproduced from arXiv: 2508.16707 by the authors.

**Figure 2.** (Left) Effectiveness vs. efficiency trade-off. Model variant 3 applies Probabilistic Expansion Control (PEC), whereas variants 1 and 2 do not. (Right) R@1 performance heatmap under varying dense (𝑤1) and sparse (𝑤2) score weights. for a similarity computation between a text and an image.2 . Variants 1 and 2 represent earlier and later checkpoints from the same training run, both trained without PEC. They achieve be… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 22 canonical work pages

[1]

Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022. The unreasonable effectiveness of CLIP features for image captioning: an experimental analysis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 4662–4670

work page 2022
[2]

Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini
[3]

Peter Bühlmann. 2011. Bagging, boosting and ensemble methods. In Handbook of computational statistics: Concepts and methods . Springer, 985–1022

work page 2011
[4]

Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Jose, Alexander Toshev, Yantao Zheng, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. 2023. STAIR: Learning Sparse Text and Image Representation in Grounded Tokens. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and...

work page 2023
[5]

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216 (2024)

arXiv 2024
[6]

Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv:1910.10687 (2019)

arXiv 2019
[7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186

2019
[8]

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 (2017)

arXiv 2017

Show all 38 references

[9]

Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant
[10]

Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2288–2292

2021
[11]

Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. arXiv preprint arXiv:2104.07186 (2021)

2021 arXiv
[12]

Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 17980–17989

2022
[13]

PJ Jeshmol and Binsu C Kovoor. 2025. A CLIP-based Video Question Answering framework with Explainable AI. In 2025 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS) . IEEE, 1–6

2025
[14]

Ding Jiang and Mang Ye. 2023. Cross-modal implicit relation reasoning and align- ing for text-to-image person retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2787–2797

2023
[15]

Andrej Karpathy and Li Fei-Fei. 2015. Deep Visual-Semantic Alignments for Generating Image Descriptions. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

2015
[16]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning . PMLR, 12888–12900

2022
[17]

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language repre- sentation learning with momentum distillation. Advances in neural information processing systems 34 (2021), 9694–9705

2021
[18]

Yifan Li, Yikai Wang, Yanwei Fu, Dongyu Ru, Zheng Zhang, and Tong He. 2024. Unified lexical representation for interpretable visual-language alignment. Ad- vances in Neural Information Processing Systems 37 (2024), 1141–1161

2024
[19]

Jimmy Lin and Xueguang Ma. 2021. A few brief notes on deepimpact, coil, and a conceptual framework for information retrieval techniques. arXiv preprint arXiv:2106.14807 (2021)

2021 arXiv
[20]

Lawrence Zitnick

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision – ECCV 2014 , David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.)....

2014
[21]

Xiaopeng Lu, Tiancheng Zhao, and Kyusong Lee. 2021. VisualSparta: An em- barrassingly simple approach to large-scale text-to-image search with weighted bag-of-words. arXiv preprint arXiv:2101.00265 (2021)

2021 arXiv
[22]

Ziyang Luo, Pu Zhao, Can Xu, Xiubo Geng, Tao Shen, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. LexLIP: Lexicon-Bottlenecked Language- Image Pre-Training for Large-Scale Image-Text Sparse Retrieval. InProceedings of the IEEE/CVF International Conference on Compu...

2023
[23]

Haoyu Ma, Handong Zhao, Zhe Lin, Ajinkya Kale, Zhangyang Wang, Tong Yu, Jiuxiang Gu, Sunav Choudhary, and Xiaohui Xie. 2022. Ei-clip: Entity-aware interventional contrastive learning for e-commerce cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vi...

2022
[24]

Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learn- ing passage impacts for inverted indexes. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1723–1727

2021
[25]

Ron Mokady, Amir Hertz, and Amit H Bermano. 2021. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.09734 (2021)

2021 arXiv
[26]

Thong Nguyen, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. 2024. Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control. In Proceedings of the 46th European Conference on Information Retrieval (ECIR) (Glas- gow, United Kingdom). Springer-Verlag, Ber...

2024 doi
[27]

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

2018 arXiv
[28]

Manish Patil, Sharma V Thankachan, Rahul Shah, Wing-Kai Hon, Jeffrey Scott Vitter, and Sabrina Chandrasekaran. 2011. Inverted indexes for phrases and strings. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval . 555–564

2011
[29]

Plummer, Liwei Wang, Chris M

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k Entities: Collecting Region- to-Phrase Correspondences for Richer Image-to-Sentence Models. In Proceedings of the IEEE International Conference on Compu...

2015
[30]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

2021
[31]

Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, and Furu Wei. 2022. Clip models are few-shot learners: Empirical studies on vqa and visual entailment. arXiv preprint arXiv:2203.07190 (2022)

2022 arXiv
[32]

Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. 2025. Cross-modal retrieval: a systematic review of methods and future directions. Proc. IEEE (2025)

2025
[33]

Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, and Changsheng Xu. 2024. Towards Visual Grounding: A Survey. arXiv preprint arXiv:2412.20206 (2024)

2024 arXiv
[34]

Linhui Xiao, Xiaoshan Yang, Fang Peng, Ming Yan, Yaowei Wang, and Chang- sheng Xu. 2023. Clip-vg: Self-paced curriculum adapting of clip for visual ground- ing. IEEE Transactions on Multimedia 26 (2023), 4334–4347

2023
[35]

Yan Zhang, Zhong Ji, Di Wang, Yanwei Pang, and Xuelong Li. 2024. USER: Unified semantic enhancement with momentum contrast for image-text retrieval. IEEE Transactions on Image Processing 33 (2024), 595–609

2024
[36]

Xiaoyang Zheng, Zilong Wang, Sen Li, Ke Xu, Tao Zhuang, Qingwen Liu, and Xiaoyi Zeng. 2023. Make: Vision-language pre-training based product retrieval in taobao search. In Companion Proceedings of the ACM Web Conference 2023 . 356–360

2023
[2021]

arXiv preprint arXiv:2109.10086 (2021)

SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086 (2021)

2021 arXiv
[2024]

In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

Efficient inverted indexes for approximate retrieval over learned sparse representations. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval . 152–162

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. 2022. The unreasonable effectiveness of CLIP features for image captioning: an experimental analysis. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 4662–4670

work page 2022

[2] [2]

Sebastian Bruch, Franco Maria Nardini, Cosimo Rulli, and Rossano Venturini

[3] [3]

Peter Bühlmann. 2011. Bagging, boosting and ensemble methods. In Handbook of computational statistics: Concepts and methods . Springer, 985–1022

work page 2011

[4] [4]

Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Jose, Alexander Toshev, Yantao Zheng, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. 2023. STAIR: Learning Sparse Text and Image Representation in Grounded Tokens. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and...

work page 2023

[5] [5]

Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216 (2024)

arXiv 2024

[6] [6]

Zhuyun Dai and Jamie Callan. 2019. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv preprint arXiv:1910.10687 (2019)

arXiv 2019

[7] [7]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers). 4171–4186

2019

[8] [8]

Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2017. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint arXiv:1707.05612 (2017)

arXiv 2017