Pith. sign in

REVIEW 4 major objections 4 minor 97 references

Needle: A Generative AI-Powered Multi-modal Database for Answering Complex Natural Language Queries

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces NEEDLE, a system that answers complex natural language queries over image collections by first generating synthetic 'guide' images from the query and then retrieving nearest neighbors to those guides in an ensemble…

desk verdict Genuinely new guide-image retrieval idea with a working system, but Theorem 1's proof is invalid and the reported 'significant' edge lacks variance and contradicts Table 2 on Winoground MRR. read the letter →

arxiv 2412.00639 v2 pith:3NCZPRJ7 submitted 2024-12-01 cs.IR cs.DB

Mahdi Erfanian , Mohsen Dehghankar , Abolfazl Asudeh This is my paper · ORCID
classification cs.IRcs.DB
keywords generativeAImulti-modalretrievalnaturallanguagequeryMonteCarlomethodtext-to-imagefoundationmodelsembeddingensemblenearestneighborsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that complex natural-language queries over image collections can be answered by generating synthetic guide images from the query and using them as stand-ins for the query in embedding space. The proposed system, NEEDLE, generates several guide images with foundation models, embeds both the guides and the dataset with an ensemble of pretrained embedders, and aggregates nearest-neighbor rankings into a final answer. Experimental comparisons on object-detection and compositional-query benchmarks are reported as showing that this approach beats contrastive-learning baselines such as CLIP and ALIGN, as well as caption-then-embed pipelines, especially on hard queries. If correct, the contribution is a deployment-ready, open-source retrieval database whose accuracy improves automatically as foundation models and embedders improve.

What carries the argument

The load-bearing object is the guide tuple: a synthetic image generated by a foundation model from the query text, treated as an i.i.d. sample from a distribution centered at an optimal representation of the query. The argument runs through the Monte Carlo estimator that averages cosine distances between each guide tuple and each dataset image across multiple embedders, with a Chernoff/union-bound theorem giving a high-probability guarantee that the estimated distance is close to the true distance. The system also adds a dynamic embedder weighting scheme, an outlier filter, a query-complexity classifier, and caching.

What would settle it

Generate the same query many times with the same foundation model, embed the outputs with one embedder, and check whether the mean of those guide embeddings equals the embedding of the query text; if it does not, the theorem's premise fails. A simpler check is to use the paper's own observation on compositional benchmarks: if guide generation often produces the wrong object ordering, the mean guide embedding cannot be centered on the true query.

Watch

Extended reading notes

Core claim

The central claim is that a query text and an image collection can be compared in a single embedding space without joint text-image training tailored to the query: instead, the query is converted into images. The paper argues that generated images sampled around an ideal query image, combined with an ensemble of embedders, give an unbiased estimate of semantic distance, and hence nearest-neighbor search over images with the generated guides approximates the true answer to the query. The proposed method uses a Monte Carlo aggregation over guide images and embedders, and the paper reports that this outperforms state-of-the-art baselines on both simple object-retrieval and compositional benchmarks.

Load-bearing premise

The whole argument assumes that the images generated from a query are, on average, a faithful stand-in for the query itself, so that averaging their distances to database images gives the true semantic distance, and that this averaging behaves well with cosine similarity, which is not automatically true.

Editorial extensions

If this is right

  • Queries that contrastive models fail on, such as 'an unripe banana' or compositional scenes, become answerable without retraining, just by generating guides.
  • Because embedders and generators are swappable, the system's accuracy should rise automatically as better foundation models and embedders appear.
  • With the query-complexity classifier, simple queries skip generation entirely, so the added cost of the method is concentrated on the queries that actually need it.
  • Experimental ablations show performance improves with more guide images and more embedders, implying the ensemble is the main driver of gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's reported gains may owe more to ensemble diversity across embedders than to the unbiasedness theorem; a natural test is to keep the guide images fixed and vary only the embedder count, or to replace generated guides with a single captioned image.
  • If the query-complexity classifier generalizes, the effective cost of the system in practice is dominated by a small fraction of hard queries, which changes the deployment calculus for personal photo repositories.
  • The same Monte Carlo scheme should transfer to audio or video retrieval once generative models for those modalities exist, since the method is modality-agnostic apart from the generator and embedders.
  • A deeper implication is that text-to-image retrieval need not rely on a shared embedding space trained for the task; any embedder that ranks images by semantic similarity can be used as long as the query can be rendered as an image.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces NEEDLE, a database system for answering complex natural-language queries over image collections. The core idea is to use generative foundation models to synthesize 'guide images' from the input query, embed those guide images with an ensemble of pretrained embedders, run k-NN searches per (guide image, embedder) pair, and aggregate the rankings. The authors describe a Monte Carlo aggregation method, prove (they claim) a concentration bound for the estimated distances, and implement practical optimizations such as dynamic embedder trust weighting, outlier detection, caching, and a query-complexity classifier. The system is evaluated on object-detection benchmarks (Caltech256, COCO, LVIS, BDD100k) and complex-NLQ benchmarks (COLA, Winoground, SentiCap, NoCaps), with additional ablations and a human study.

Significance. If the empirical advantage is real, the contribution is useful: the system is open-source, deployment-oriented, and the main retrieval pipeline does not train on the target test labels; embedder weights are taken from an external leaderboard and the guide images are produced by external foundation models. The paper also reports concrete monetary costs and includes a human preference study. These are genuine strengths. However, the theoretical guarantee is not established as written, and the headline empirical claim is weakened by an internal inconsistency and by the complete absence of error bars or significance testing. The contribution is therefore conditional on a corrected analysis and a more rigorous empirical report.

major comments (4)
  1. [Appendix B and §3.1, Theorem 1] The proof of Theorem 1 is invalid because it assumes E[δ(E^ℓ(¯g_j), v^ℓ_i)] = δ_{φ,i} follows from E[E_o(¯g_φ)] = E_o(g_φ). This is false for cosine distance: cosine is not linear, so E[cos(X, v)] ≠ cos(E[X], v) in general. The same issue affects the earlier statement in §3.1 that E[E_o(¯g_φ)] = E_o(g_φ) implies the generated images are aligned with the query in expectation. Moreover, the unbiasedness assumption itself is asserted rather than derived, and §5.2.2 later admits that foundation models 'frequently fail to produce images with the correct compositional ordering,' which directly contradicts the assumption that guide images are unbiased samples of the query's ideal representation. As written, Theorem 1 provides no valid concentration guarantee and should be repaired, replaced with a condition under which the bound holds, or removed from the paper's claims.
  2. [§5.2.2 and Table 2] The text states that 'NEEDLE outperforms the baselines in both PAcc and MRR,' but Table 2 shows the opposite on Winoground MRR: ALIGN achieves 0.501 while NEEDLE achieves 0.490. This is a direct internal contradiction in the central experimental section. The claim needs to be corrected, and the discrepancy needs discussion (for example, whether the Winoground MRR difference is within noise).
  3. [§5.1 and Tables 1–2, overall evaluation] No error bars, confidence intervals, or significance tests are reported anywhere in the experimental section. Guide-image generation is stochastic, with a small number of guide images (e.g., m=9 in the object-detection setup), so the reported point estimates are single draws from a random procedure. The abstract's claim that NEEDLE 'significantly outperforms' state-of-the-art methods is therefore not supported by the evidence as presented. The authors should provide repeated-run statistics or a formal statistical comparison, at least for the main benchmarks in Tables 1 and 2.
  4. [§5.1, E.1, and Table 1] The 'hard' subset in Table 1 is defined by CLIP Average Precision below 0.5, i.e., by the performance of the weakest named baseline. The authors should make clear that 'hard' is relative to CLIP, not an intrinsic property of the query set, and should report the number of queries in each hard subset. This is not circular for NEEDLE because no training on those labels occurs, but the framing as 'hard-set improvements' overstates the generality of the conclusion.
minor comments (4)
  1. [Appendix E.3] The text says 'the pairing accuracy for a random baseline is 0.25%'; since random chance among four pairings is 0.25, this should read '0.25' or '25%'.
  2. [Appendix D.2] The sentence 'some illustrative examples are provided in Listing??' contains a missing cross-reference; the listing should be included or the sentence removed.
  3. [§3.3 and Appendix C.1] The default embedder weights are derived from the timm leaderboard, which is an image-classification benchmark, whereas the embedders are used here for retrieval. This mismatch is worth a sentence of justification or a caveat.
  4. [Appendix A] The limitations paragraph on reliance on foundation models is welcome and appropriately candid; it would be even more useful if it noted that the current experiments already show one failure mode (compositional ordering) that the theory does not account for.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: retrieval results are benchmarked against external labels and the theoretical bound is conditional on stated, not fitted, assumptions.

full rationale

No circularity found. NEEDLE's retrieval pipeline never fits parameters to the evaluation labels: default embedder weights come from the external timm leaderboard (Section 4), guide images are generated from the query prompt alone, and the reported experiments use fixed weights rather than the on-the-fly feedback mechanism. The theoretical guarantee in Theorem 1 is conditional on explicitly stated unbiasedness assumptions (E[delta^ell_{i,j}] = delta^o_{i,j} for all tuples and E[E_o(gbar_phi)] = E_o(g_phi)); those assumptions are unverified and the proof contains an expectation-versus-cosine gap, but relying on an assumption is not equivalent to defining the prediction in terms of the target. The Table 2 inconsistency (Winoground MRR: NEEDLE 0.490 vs ALIGN 0.501 despite Section 5.2.2 claiming NEEDLE outperforms in both PAcc and MRR) and the absence of error bars are evidence-quality and statistical concerns, not circularity. Self-citations ([16], [59]) appear only in related work and are not load-bearing for the central claim. The core experimental result is therefore self-contained with respect to the benchmark ground truth, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the unverified assumption that generated guide images and embedder distances are unbiased samples of the query's semantic location. The hyperparameters m, l, and the trust weights are not fitted to the test labels in the main experiments, but they do shape the results. No new entities beyond the synthetic 'guide tuple' concept are introduced.

free parameters (6)
  • m (number of guide images per foundation model) = 3 (default; varied 1-3 in ablation)
    Controls the sample size of the Monte Carlo estimate; larger m improves MAP in Figure 7.
  • l (number of embedders) = 6 (default; varied 1-6)
    Ensemble size; larger l improves MAP in Figure 7.
  • Embedder trust weights w_i^t = timm leaderboard scores (Table 3)
    Chosen externally, not fitted to the target datasets; still determines how much each embedder contributes to the final ranking.
  • Outlier threshold tau = not specified
    Hyperparameter in the anomaly detection module (Appendix C.2); trades precision against recall.
  • Query complexity routing threshold gamma_AP = not specified
    Used by the query complexity classifier to decide whether to short-circuit generation (Appendix C.4).
  • Image quality level = MEDIUM in main experiments
    Ablation (Figure 10) shows performance varies with generated image resolution.
assumptions (5)
  • domain assumption Generated guide images are iid samples from a distribution whose mean is g_phi, and E[E_o(gbar_phi)] = E_o(g_phi)
    Section 3.1 states this assumption; it is load-bearing for the Monte Carlo averaging and Theorem 1, but no evidence is given.
  • domain assumption Each embedder is an unbiased estimator of the optimal semantic distance: E[delta^ell_{i,j}] = delta^o_{i,j}
    Section 3.2 assumes each embedder is a random sample from the space of embedders; this is not a defined distribution.
  • domain assumption Cosine distance between embeddings is monotonically related to semantic distance
    Equation 2; a standard idealization for embedding-based retrieval.
  • standard math Chernoff bound applies to the sum of 1-cosine distances
    Used in Appendix B, but the required independence and boundedness of the terms are not established.
  • domain assumption Foundation models can generate images that reflect compositional and relational query content
    Section 5.2.2 admits the generators often fail at compositional ordering, yet the retrieval advantage depends on partial success.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Needle: A Generative AI-Powered Multi-modal Database for Answering Complex Natural Language Queries." pith.science (2026). https://pith.science/paper/3NCZPRJ7

@misc{pith2026241200639,
  author       = {Pith},
  title        = {Pith review of: Needle: A Generative AI-Powered Multi-modal Database for Answering Complex Natural Language Queries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3NCZPRJ7}},
  note         = {Machine review of arXiv:2412.00639}
}
read the original abstract

Multi-modal datasets, like those involving images, often miss the detailed descriptions that properly capture the rich information encoded in each item. This makes answering complex natural language queries a major challenge in this domain. In particular, unlike the traditional nearest neighbor search, where the tuples and the query are represented as points in a single metric space, these settings involve queries and tuples embedded in fundamentally different spaces, making the traditional query answering methods inapplicable. Existing literature addresses this challenge for image datasets through vector representations jointly trained on natural language and images. This technique, however, underperforms for complex queries due to various reasons. This paper takes a step towards addressing this challenge by introducing a Generative-based Monte Carlo method that utilizes foundation models to generate synthetic samples that capture the complexity of the natural language query and represent it in the same metric space as the multi-modal data. Following this method, we propose Needle, a database for image data retrieval. Instead of relying on contrastive learning or metadata-searching approaches, our system is based on synthetic data generation to capture the complexities of natural language queries. Our system is open-source and ready for deployment, designed to be easily adopted by researchers and developers. The comprehensive experiments on various benchmark datasets verify that this system significantly outperforms state-of-the-art text-to-image retrieval methods in the literature. Any foundation model and embedder can be easily integrated into Needle to improve the performance, piggybacking on the advancements in these technologies.

Figures

Figures reproduced from arXiv: 2412.00639 by the authors.

Figure 1
Figure 1. The query results for (a) [a banana] (row [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. NEEDLE Architecture 5 Experimental Evaluation In this section, we evaluate several key aspects of NEEDLE to demonstrate its capabilities and to evaluate its performance. We use multiple benchmark datasets and several baselines for this purpose. In the following, we first detail our experiments setup (§ 5.1), followed by a proof-of-concept analysis (§ 5.2) that demonstrates NEEDLE’s efficacy in text-to-image retrieva… view at source ↗
Figure 3
Figure 3. Illustration of Complex Natural Language Queries extracted from NoCaps [ [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Correlation between CLIP cosine similarity and Mean Average Precision (MAP) across [PITH_FULL_IMAGE:figures/full_fig_p021_4.png]
Figure 5
Figure 5. Figure 5: Screenshots of key interfaces and features of N [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Illustration of COLA compositional benchmark queries, using each image description, [PITH_FULL_IMAGE:figures/full_fig_p027_6.png]
Figure 7
Figure 7. Figure 7: Illustration of how varying number of guide images and number of embedders involved [PITH_FULL_IMAGE:figures/full_fig_p029_7.png]
Figure 8
Figure 8. Figure 8: Comparison of Performance of Different Image Generators on Object Detection and [PITH_FULL_IMAGE:figures/full_fig_p030_8.png]
Figure 9
Figure 9. Figure 9: The effect of number of Foundation Models on the Performance of [PITH_FULL_IMAGE:figures/full_fig_p031_9.png]
Figure 10
Figure 10. Figure 10: Illustration of variation of NEEDLE MAP based on size of guide image for different foundation models It is evident that foundation models are typically fine-tuned for a specific image resolution, which in turn yields the best quality and prompt alignment. In our exper…
Figure 11
Figure 11. Figure 11: Results from the human evaluation case study on the [PITH_FULL_IMAGE:figures/full_fig_p032_11.png]
Figure 12
Figure 12. Figure 12: Illustration of how NEEDLE Query Complexity Classifier improves NEEDLE inference speed The model was trained on a combined dataset derived from the training splits of the Caltech256 [21] and LVIS [22] benchmarks. The ground-truth target for training was the actual AP …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

97 extracted references · 48 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    nocaps: novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. InProceedings of the IEEE International Conference on Computer Vision, pages 8948–8957, 2019

    2019

  3. [3]

    Natural language interfaces to databases–an introduction.Natural language engineering, 1(1):29–81, 1995

    Ion Androutsopoulos, Graeme D Ritchie, and Peter Thanisch. Natural language interfaces to databases–an introduction.Natural language engineering, 1(1):29–81, 1995

    1995

  4. [4]

    Modeling score distributions in information retrieval

    Avi Arampatzis and Stephen Robertson. Modeling score distributions in information retrieval. Information Retrieval, 14:26–46, 2011

    2011

  5. [5]

    The multiplicative weights update method: a meta-algorithm and applications.Theory of computing, 8(1):121–164, 2012

    Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a meta-algorithm and applications.Theory of computing, 8(1):121–164, 2012

    2012

  6. [6]

    Deep neural architecture for multi-modal retrieval based on joint embedding space for text and images

    Saeid Balaneshin-kordan and Alexander Kotov. Deep neural architecture for multi-modal retrieval based on joint embedding space for text and images. InProceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pages 28–36, 2018

    2018

  7. [7]

    Beit: Bert pre-training of image trans- formers.arXiv preprint arXiv:2106.08254, 2021

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image trans- formers.arXiv preprint arXiv:2106.08254, 2021

    arXiv 2021

  8. [8]

    Lof: identifying density-based local outliers

    Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. Lof: identifying density-based local outliers. InProceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93–104, 2000

    2000

Show all 97 references
  1. [9]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    Tom B Brown. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  2. [10]

    Large-scale content-based audio retrieval from text queries

    Gal Chechik, Eugene Ie, Martin Rehn, Samy Bengio, and Dick Lyon. Large-scale content-based audio retrieval from text queries. InProceedings of the 1st ACM international conference on Multimedia information retrieval, pages 105–112, 2008

    2008

  3. [11]

    Image search with text feedback by visiolin- guistic attention learning

    Yanbei Chen, Shaogang Gong, and Loris Bazzani. Image search with text feedback by visiolin- guistic attention learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3001–3011, 2020

    2020

  4. [12]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1– 113, 2023

    2023

  5. [13]

    On the inference of average precision from score distributions

    Ronan Cummins. On the inference of average precision from score distributions. InProceedings of the 21st ACM international conference on Information and knowledge management, pages 2435–2438, 2012

    2012

  6. [14]

    Advanced generative ai methods for academic text summarization

    Zaema Dar, Muhammad Raheel, Usman Bokhari, Akhtar Jamil, Esraa Mohammed Alazawi, and Alaa Ali Hameed. Advanced generative ai methods for academic text summarization. In 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI), pages 1–7. IEEE, 2024

    2024

  7. [15]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  8. [16]

    Chameleon: Foundation models for fairness-aware multi-modal data augmentation to enhance coverage of minorities.Proceedings of the VLDB Endowment, 17(11):3470–3483, 2024

    Mahdi Erfanian, HV Jagadish, and Abolfazl Asudeh. Chameleon: Foundation models for fairness-aware multi-modal data augmentation to enhance coverage of minorities.Proceedings of the VLDB Endowment, 17(11):3470–3483, 2024

    2024

  9. [17]

    Optimal aggregation algorithms for middleware

    Ronald Fagin, Amnon Lotem, and Moni Naor. Optimal aggregation algorithms for middleware. InProceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 102–113, 2001. 11

    2001

  10. [18]

    Eva: Exploring the limits of masked visual representation learning at scale

    Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, page...

    2023

  11. [19]

    Learning to rank for content-based image retrieval

    Fabio F Faria, Adriano Veloso, Humberto M Almeida, Eduardo Valle, Ricardo da S Torres, Marcos A Gonçalves, and Wagner Meira Jr. Learning to rank for content-based image retrieval. InProceedings of the international conference on Multimedia information retrieval, pages 285–294, 2010

    2010

  12. [20]

    Tagging personal photos with transfer deep learning

    Jianlong Fu, Tao Mei, Kuiyuan Yang, Hanqing Lu, and Yong Rui. Tagging personal photos with transfer deep learning. InProceedings of the 24th International Conference on World Wide Web, pages 344–354, 2015

    2015

  13. [21]

    Caltech 256, Apr 2022

    Gregory Griffin, Alex Holub, and Pietro Perona. Caltech 256, Apr 2022

    2022

  14. [22]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356–5364, 2019

    2019

  15. [23]

    Query-aware locality- sensitive hashing for approximate nearest neighbor search.Proceedings of the VLDB Endow- ment, 9(1):1–12, 2015

    Qiang Huang, Jianlin Feng, Yikai Zhang, Qiong Fang, and Wilfred Ng. Query-aware locality- sensitive hashing for approximate nearest neighbor search.Proceedings of the VLDB Endow- ment, 9(1):1–12, 2015

    2015

  16. [24]

    A survey on locality sensitive hashing algorithms and their applications.arXiv preprint arXiv:2102.08942, 2021

    Omid Jafari, Preeti Maurya, Parth Nagarkar, Khandker Mushfiqul Islam, and Chidambaram Crushev. A survey on locality sensitive hashing algorithms and their applications.arXiv preprint arXiv:2102.08942, 2021

    2021 arXiv

  17. [25]

    Scaling up visual and vision-language representation learning with noisy text supervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. InInternational conference on machine learning, pages 4904–4916. ...

    2021

  18. [26]

    Semantically- enhanced kernel canonical correlation analysis: a multi-label cross-modal retrieval.Multimedia Tools and Applications, 78:13169–13188, 2019

    Yuhua Jia, Liang Bai, Shuang Liu, Peng Wang, Jinlin Guo, and Yuxiang Xie. Semantically- enhanced kernel canonical correlation analysis: a multi-label cross-modal retrieval.Multimedia Tools and Applications, 78:13169–13188, 2019

    2019

  19. [27]

    Learning image embeddings using convolutional neural networks for improved multi-modal semantics

    Douwe Kiela and Léon Bottou. Learning image embeddings using convolutional neural networks for improved multi-modal semantics. InProceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP), pages 36–45, 2014

    2014

  20. [28]

    Audio retrieval with natural language queries: A benchmark study.IEEE Transactions on Multimedia, 25:2675–2685, 2022

    A Sophia Koepke, Andreea-Maria Oncescu, João F Henriques, Zeynep Akata, and Samuel Albanie. Audio retrieval with natural language queries: A benchmark study.IEEE Transactions on Multimedia, 25:2675–2685, 2022

    2022

  21. [29]

    V oronoi-based k nearest neighbor search for spatial network databases

    Mohammad Kolahdouzan and Cyrus Shahabi. V oronoi-based k nearest neighbor search for spatial network databases. InProceedings of the Thirtieth international conference on Very large data bases-Volume 30, pages 840–851, 2004

    2004

  22. [30]

    Using synthetic data to train neural networks is model-based reasoning

    Tuan Anh Le, Atilim Giine¸ s Baydin, Robert Zinkov, and Frank Wood. Using synthetic data to train neural networks is model-based reasoning. In2017 international joint conference on neural networks (IJCNN), pages 3514–3521. IEEE, 2017

    2017

  23. [31]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in Neural Information Processing Systems,...

    2020

  24. [32]

    Nalir: an interactive natural language interface for querying relational databases

    Fei Li and Hosagrahar V Jagadish. Nalir: an interactive natural language interface for querying relational databases. InProceedings of the 2014 ACM SIGMOD international conference on Management of data, pages 709–712, 2014. 12

    2014

  25. [33]

    Cross- modal retrieval: a systematic review of methods and future directions.arXiv preprint arXiv:2308.14263, 2023

    Fengling Li, Lei Zhu, Tianshi Wang, Jingjing Li, Zheng Zhang, and Heng Tao Shen. Cross- modal retrieval: a systematic review of methods and future directions.arXiv preprint arXiv:2308.14263, 2023

    2023 arXiv

  26. [34]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.Advances in Neural Information Processing Systems, 36, 2024

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.Advances in Neural Information Processing Systems, 36, 2024

    2024

  27. [35]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. InInternational conference on machine learning, pages 12888–12900. PMLR, 2022

    2022

  28. [36]

    W2vv++ fully deep learning for ad-hoc video search

    Xirong Li, Chaoxi Xu, Gang Yang, Zhineng Chen, and Jianfeng Dong. W2vv++ fully deep learning for ad-hoc video search. InProceedings of the 27th ACM international conference on multimedia, pages 1786–1794, 2019

    2019

  29. [37]

    Using llm to select the right sql query from candidates.arXiv preprint arXiv:2401.02115, 2024

    Zhenwen Li and Tao Xie. Using llm to select the right sql query from candidates.arXiv preprint arXiv:2401.02115, 2024

    2024 arXiv

  30. [38]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings,...

    2014

  31. [39]

    Learning a recurrent residual fusion network for multimodal matching

    Yu Liu, Yanming Guo, Erwin M Bakker, and Michael S Lew. Learning a recurrent residual fusion network for multimodal matching. InProceedings of the IEEE international conference on computer vision, pages 4107–4116, 2017

    2017

  32. [40]

    Clip4clip: An empirical study of clip for end to end video clip retrieval.arXiv preprint arXiv:2104.08860, 2021

    Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval.arXiv preprint arXiv:2104.08860, 2021

    2021 arXiv

  33. [41]

    Mention extraction and linking for sql query generation.arXiv preprint arXiv:2012.10074, 2020

    Jianqiang Ma, Zeyu Yan, Shuai Pang, Yang Zhang, and Jianping Shen. Mention extraction and linking for sql query generation.arXiv preprint arXiv:2012.10074, 2020

    2012 arXiv

  34. [42]

    Yu A Malkov and Dmitry A Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.IEEE transactions on pattern analysis and machine intelligence, 42(4):824–836, 2018

    2018

  35. [43]

    Query and keyframe representations for ad-hoc video search

    Foteini Markatopoulou, Damianos Galanopoulos, Vasileios Mezaris, and Ioannis Patras. Query and keyframe representations for ad-hoc video search. InProceedings of the 2017 ACM on international conference on multimedia retrieval, pages 407–411, 2017

    2017

  36. [44]

    Senticap: Generating image descriptions with sentiments

    Alexander Mathews, Lexing Xie, and Xuming He. Senticap: Generating image descriptions with sentiments. InProceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  37. [45]

    Umap: Uniform manifold approximation and projection for dimension reduction.arXiv preprint arXiv:1802.03426, 2018

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction.arXiv preprint arXiv:1802.03426, 2018

    2018 arXiv

  38. [46]

    Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. InProceedings of the IEEE/CVF international conference on computer vision, pages 2...

    2019

  39. [47]

    Learning joint embedding with multimodal cues for cross-modal video-text retrieval

    Niluthpol Chowdhury Mithun, Juncheng Li, Florian Metze, and Amit K Roy-Chowdhury. Learning joint embedding with multimodal cues for cross-modal video-text retrieval. In Proceedings of the 2018 ACM on international conference on multimedia retrieval, pages 19–27, 2018

    2018

  40. [48]

    Seesaw: interactive ad-hoc search over image databases.Proceedings of the ACM on Management of Data, 1(4):1–26, 2023

    Oscar Moll, Manuel Favela, Samuel Madden, Vijay Gadepally, and Michael Cafarella. Seesaw: interactive ad-hoc search over image databases.Proceedings of the ACM on Management of Data, 1(4):1–26, 2023. 13

    2023

  41. [49]

    Randomized algorithms.ACM Computing Surveys (CSUR), 28(1):33–37, 1996

    Rajeev Motwani and Prabhakar Raghavan. Randomized algorithms.ACM Computing Surveys (CSUR), 28(1):33–37, 1996

    1996

  42. [50]

    Learning and transferring mid- level image representations using convolutional neural networks

    Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid- level image representations using convolutional neural networks. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 1717–1724, 2014

    2014

  43. [51]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  44. [52]

    Composing object relations and attributes for image-text matching

    Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava. Composing object relations and attributes for image-text matching. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14354–14363, 2024

    2024

  45. [53]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  46. [54]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  47. [55]

    A new approach to cross-modal multimedia retrieval

    Nikhil Rasiwasia, Jose Costa Pereira, Emanuele Coviello, Gabriel Doyle, Gert RG Lanckriet, Roger Levy, and Nuno Vasconcelos. A new approach to cross-modal multimedia retrieval. In Proceedings of the 18th ACM international conference on Multimedia, pages 251–260, 2010

    2010

  48. [56]

    Cola: A benchmark for compositional text-to-image retrieval.Advances in Neural Information Processing Systems, 36:46433–46445, 2023

    Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan Plummer, Ranjay Krishna, and Kate Saenko. Cola: A benchmark for compositional text-to-image retrieval.Advances in Neural Information Processing Systems, 36:46433–46445, 2023

    2023

  49. [57]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  50. [58]

    Multi-modal joint embedding for fashion product retrieval

    Antonio Rubio, LongLong Yu, Edgar Simo-Serra, and Francesc Moreno-Noguer. Multi-modal joint embedding for fashion product retrieval. In2017 IEEE International Conference on Image Processing (ICIP), pages 400–404. IEEE, 2017

    2017

  51. [59]

    Coverage-based data-centric approaches for responsible and trustworthy ai.IEEE Data Eng

    Nima Shahbazi, Mahdi Erfanian, and Abolfazl Asudeh. Coverage-based data-centric approaches for responsible and trustworthy ai.IEEE Data Eng. Bull., 47(1):3–17, 2024

    2024

  52. [60]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15638–15650, 2022

    2022

  53. [61]

    Data augmentation using gans

    Fabio Henrique Kiyoiti Dos Santos Tanaka and Claus Aranha. Data augmentation using gans. arXiv preprint arXiv:1904.09135, 2019

    1904 arXiv

  54. [62]

    Interacting- enhancing feature transformer for cross-modal remote-sensing image and text retrieval.IEEE Transactions on Geoscience and Remote Sensing, 61:1–15, 2023

    Xu Tang, Yijing Wang, Jingjing Ma, Xiangrong Zhang, Fang Liu, and Licheng Jiao. Interacting- enhancing feature transformer for cross-modal remote-sensing image and text retrieval.IEEE Transactions on Geoscience and Remote Sensing, 61:1–15, 2023

    2023

  55. [63]

    Winoground: Probing vision and language models for visio-linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2022

  56. [64]

    Learning language-visual embedding for movie understanding with natural-language.arXiv preprint arXiv:1609.08124, 2016

    Atousa Torabi, Niket Tandon, and Leonid Sigal. Learning language-visual embedding for movie understanding with natural-language.arXiv preprint arXiv:1609.08124, 2016. 14

    2016 arXiv

  57. [65]

    Training deep networks with synthetic data: Bridging the reality gap by domain randomization

    Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. InProceedings of the IEEE conferen...

    2018

  58. [66]

    Codexdb: Synthesizing code for query processing from natural language instructions using gpt-3 codex.Proceedings of the VLDB Endowment, 15(11):2921–2928, 2022

    Immanuel Trummer. Codexdb: Synthesizing code for query processing from natural language instructions using gpt-3 codex.Proceedings of the VLDB Endowment, 15(11):2921–2928, 2022

    2022

  59. [67]

    Demonstrating gpt-db: Generating query-specific and customizable code for sql processing with gpt-4.Proceedings of the VLDB Endowment, 16(12):4098–4101, 2023

    Immanuel Trummer. Demonstrating gpt-db: Generating query-specific and customizable code for sql processing with gpt-4.Proceedings of the VLDB Endowment, 16(12):4098–4101, 2023

    2023

  60. [68]

    Dif- fusion models for tabular data imputation and synthetic data generation.arXiv preprint arXiv:2407.02549, 2024

    Mario Villaizán-Vallelado, Matteo Salvatori, Carlos Segura, and Ioannis Arapakis. Dif- fusion models for tabular data imputation and synthetic data generation.arXiv preprint arXiv:2407.02549, 2024

    2024 arXiv

  61. [69]

    Image-text cross- modal retrieval via modality-specific feature learning

    Jian Wang, Yonghao He, Cuicui Kang, Shiming Xiang, and Chunhong Pan. Image-text cross- modal retrieval via modality-specific feature learning. InProceedings of the 5th ACM on International Conference on Multimedia Retrieval, pages 347–354, 2015

    2015

  62. [70]

    Cluster-sensitive structured correlation analysis for web cross-modal retrieval.Neurocomputing, 168:747–760, 2015

    Shuhui Wang, Fuzhen Zhuang, Shuqiang Jiang, Qingming Huang, and Qi Tian. Cluster-sensitive structured correlation analysis for web cross-modal retrieval.Neurocomputing, 168:747–760, 2015

    2015

  63. [71]

    Cross- modal retrieval: a systematic review of methods and future directions.Proceedings of the IEEE, 2025

    Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. Cross- modal retrieval: a systematic review of methods and future directions.Proceedings of the IEEE, 2025

    2025

  64. [72]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  65. [73]

    Convnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16133–16142, 2023

    2023

  66. [74]

    Improving text-audio retrieval by text-aware attention pooling and prior matrix revised loss

    Yifei Xin, Dongchao Yang, and Yuexian Zou. Improving text-audio retrieval by text-aware attention pooling and prior matrix revised loss. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  67. [75]

    Regnet: self-regulated network for image classification.IEEE Transactions on Neural Networks and Learning Systems, 34(11):9562–9567, 2022

    Jing Xu, Yu Pan, Xinglin Pan, Steven Hoi, Zhang Yi, and Zenglin Xu. Regnet: self-regulated network for image classification.IEEE Transactions on Neural Networks and Learning Systems, 34(11):9562–9567, 2022

    2022

  68. [76]

    Re- thinking label-wise cross-modal retrieval from a semantic sharing perspective

    Yang Yang, Chubing Zhang, Yi-Chu Xu, Dianhai Yu, De-Chuan Zhan, and Jian Yang. Re- thinking label-wise cross-modal retrieval from a semantic sharing perspective. InIJCAI, pages 3300–3306, 2021

    2021

  69. [77]

    Saliencycut: Augmenting plausible anomalies for anomaly detection.Pattern Recognition, 153:110508, 2024

    Jianan Ye, Yijie Hu, Xi Yang, Qiu-Feng Wang, Chao Huang, and Kaizhu Huang. Saliencycut: Augmenting plausible anomalies for anomaly detection.Pattern Recognition, 153:110508, 2024

    2024

  70. [78]

    Data structures and algorithms for nearest neighbor search in general metric spaces

    Peter N Yianilos. Data structures and algorithms for nearest neighbor search in general metric spaces. InSoda, volume 93, pages 311–21, 1993

    1993

  71. [79]

    Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pa...

    2020

  72. [80]

    Text-to-image diffusion models in generative ai: A survey.arXiv preprint arXiv:2303.07909, 2023

    Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion models in generative ai: A survey.arXiv preprint arXiv:2303.07909, 2023. 15

    2023 arXiv

  73. [81]

    Interactive retrieval based on faceted feedback

    Lanbo Zhang and Yi Zhang. Interactive retrieval based on faceted feedback. InProceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 363–370, 2010

    2010

  74. [82]

    Editing-based sql query generation for cross- domain context-dependent questions.arXiv preprint arXiv:1909.00786, 2019

    Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. Editing-based sql query generation for cross- domain context-dependent questions.arXiv preprint arXiv:1909.00786, 2019

    1909 arXiv

  75. [83]

    Recognize anything: A strong image tagging model

    Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1724–...

    2024

  76. [84]

    Relevance feedback in image retrieval: A comprehen- sive review.Multimedia systems, 8:536–544, 2003

    Xiang Sean Zhou and Thomas S Huang. Relevance feedback in image retrieval: A comprehen- sive review.Multimedia systems, 8:536–544, 2003

    2003

  77. [85]

    complex

    Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey.Proceedings of the IEEE, 111(3):257–276, 2023. 16 APPENDIX A Discussions and Limitations Reliance on Existing Models:NEEDLErelies on Foundation Models for guide tuple gene...

    2023

  78. [86]

    hard categories

    CLIP [53]:Developed by OpenAI 26, this model uses a ViT-B/32 image encoder and a Transformer- based text encoder. It learns a shared embedding space via contrastive learning on large-scale image– text pairs, and its widespread use and benchmarking make it a standard reference....

  79. [87]

    However, while CLIP (in our clip-vit-base-patch32 variant) uses a Vision Transformer for image encoding, ALIGN typically uses a CNN image encoder along with a Transformer for text

    ALIGN [ 25]:Originally developed by Google, ALIGN (A Large-scale ImaGe and Noisy-text Embedding) employs a dual-encoder architecture similar to CLIP, using separate encoders for images and text trained with a contrastive loss. However, while CLIP (in our clip-vit-base-patch32 ...

  80. [88]

    FLA V A [60]:28 a unified multimodal model developed by Facebook that jointly learns representa- tions for images and text. Unlike CLIP, which primarily relies on a contrastive approach applied to image-text pairs, and ALIGN, which emphasizes scaling up representation learning...

  81. [89]

    The generated captions are then transformed into embeddings with MiniLM [72]30, a robust text encoder widely adopted in industrial applications

    BLIP + MiniLM:This pipeline approach first converts images into descriptive captions us- ing BLIP [35]29—a state-of-the-art image captioning model from Salesforce known for its high- quality, informative captions. The generated captions are then transformed into embeddings wit...

  82. [90]

    Its diverse object classes make it a useful benchmark for assessing the robustness of detection and retrieval models

    Caltech256 [21]: Contains 30,607 images spanning 256 object categories. Its diverse object classes make it a useful benchmark for assessing the robustness of detection and retrieval models

  83. [91]

    Its comprehensive annotations and varied scene compositions provide a challenging testbed

    MS COCO [38]: With over 118K images and 80 object categories, MS COCO is widely used for object detection and segmentation. Its comprehensive annotations and varied scene compositions provide a challenging testbed

  84. [92]

    Its focus on rare and fine-grained objects is critical for evaluating retrieval performance on less frequent classes

    LVIS [22]: offers instance segmentation with a long-tail distribution of over 1,200 categories. Its focus on rare and fine-grained objects is critical for evaluating retrieval performance on less frequent classes

  85. [93]

    BDD100k [79]: This dataset is tailored to urban driving scenarios with detailed annotations, making it valuable for testing retrieval in real-world, dynamic contexts. Natural Language Query Datasets: For evaluating the performance of various baselines on complex natural langua...

  86. [94]

    COLA [56]: As a benchmark for compositional text-to-image retrieval, this dataset features nuanced captions that require the system to distinguish between multiple similar images by capturing subtle semantic and spatial details

  87. [95]

    This dataset challenges models to accurately map nuanced language to the corresponding image, serving as a stringent test of fine-grained retrieval performance

    Winoground [63]: Designed to evaluate visio-linguistic alignment, Winoground presents pairs of images and captions that differ only in their compositional structure. This dataset challenges models to accurately map nuanced language to the corresponding image, serving as a stri...

  88. [96]

    It is crucial for evaluating a model’s ability to handle zero-shot retrieval on images containing objects not commonly found in standard captioning datasets

    NoCaps [2]: A large-scale benchmark for novel object captioning, featuring images from Open Images with human-generated captions. It is crucial for evaluating a model’s ability to handle zero-shot retrieval on images containing objects not commonly found in standard captioning...

  89. [97]

    Left is better,

    SentiCap [44]: Derived from datasets like MS COCO, SentiCap provides images with captions specifically annotated for positive or negative sentiment. It allows for evaluating retrieval systems’ ability to understand and retrieve images based on subjective or emotional language ...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.