Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

Changyou Chen; Jingchen Sun; Ming Liu; Naresh Kumar Devulapally; Ruiyi Zhang; Shaobo Han; Vishnu Suresh Lokhande; Yitao Long

REVIEW 4 major objections 6 minor 92 references

Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bayesian Data Reweighting, which infers per-negative importance weights via closed-form posterior updates, consistently improves retrieval and downstream answer generation for knowledge-based visual question answering.

desk verdict A useful reweighting trick for contrastive retrieval, dressed in Bayesian language; the empirical gains are consistent and worth engaging with, despite the pseudo-likelihood caveat. read the letter →

arxiv 2608.02907 v1 pith:7TV6CLXO submitted 2026-08-03 cs.LG

Jingchen Sun , Shaobo Han , Ruiyi Zhang , Naresh Kumar Devulapally , Ming Liu , Yitao Long , Vishnu Suresh Lokhande , Changyou Chen This is my paper · ORCID

classification cs.LG

keywords multimodalretrievalknowledge-basedvisualquestionansweringcontrastivelearningfalsenegativesBayesiandatareweightinglatentimportanceweightsstochasticexpectation-maximizationrecall

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that standard contrastive training for knowledge-based visual question answering is hurt by false negatives — unpaired documents that are actually relevant or partially useful — and that this can be fixed by reweighting each negative according to a Bayesian posterior over its importance. The proposed method, Bayesian Data Reweighting, replaces the uniform negative term in the InfoNCE loss with a weighted sum, treats each weight as a latent variable with a Gamma or Bernoulli prior, and infers it with closed-form updates inside a stochastic EM loop. Across three retriever backbones and seven KB-VQA benchmarks, the paper reports consistent gains in Recall@K and Pseudo Recall@K over InfoNCE and six existing reweighting baselines, and the trained retriever improves answer generation when paired with frozen generators on InfoSeek and EVQA. The practical significance would be a cheap, drop-in training objective that makes multimodal retrieval more robust to incomplete annotations in knowledge-intensive settings.

What carries the argument

The load-bearing object is the weighted contrastive objective $\mathcal{L}_{\text{BDR}}(\mathcal{B};\theta)=-\frac{1}{B}\sum_i \log \frac{s_{i+}}{s_{i+}+\sum_k w_{ik}s_{ik-}}$, with latent importance weights $w_{ik}\ge 0$. The enabling identity is the Laplace transform $\lambda^{-1}=\int_0^\infty e^{-\lambda u}\,du$, applied to the normalization term $\lambda_i=s_{i+}+\sum_k w_{ik}s_{ik-}$; this turns the coupled denominator into a factorized product $s_{i+}e^{-u_i s_{i+}}\prod_k e^{-u_i w_{ik}s_{ik-}}$, restoring conditional conjugacy. With $u_i\sim\text{Gamma}(a_u,b_u)$ and $w_{ik}\sim\text{Gamma}(a_w,b_w)$ (or Bernoulli), the conditional posteriors are closed-form, and stochastic approximation EM alternates sampling them with gradient updates to the encoder. The machinery's work is to convert what could be a heuristic similarity-threshold schedule into a posterior inference procedure: expected weights drop monotonically as negative similarity grows, so noisy supervision is suppressed instance-by-instance without needing false-negative labels.

What would settle it

Construct or annotate a set of true false negatives in a KB-VQA corpus, train the same retriever with BDR and with a deterministic decreasing function of cosine similarity used as weights, and compare both Recall@K and how well the inferred weights predict the human false-negative labels. If the deterministic schedule matches BDR's retrieval gains and predicts the labels equally well, the posterior inference is not the active ingredient.

Watch

Extended reading notes

Core claim

The central discovery is that the false-negative problem in KB-VQA retrieval can be reduced to a tractable Bayesian reweighting problem. The paper formulates a sample-wise pseudo-likelihood from the weighted contrastive objective, augments it with an auxiliary variable via the Laplace identity so that the denominator factorizes, and then derives closed-form conditional posteriors for the negative weights under Gamma and Bernoulli priors. The Gamma posterior expectation $a_w/(b_w+u_i s_{ik}^-)$ decreases with negative similarity, so the method automatically downweights the most suspicious false negatives; the Bernoulli posterior performs hard keep-or-drop gating. Optimized with stochastic approximation EM, this reweighting is what the paper argues produces better-separated query-document embeddings, a larger positive-negative similarity gap, higher Recall@K at every tested K, and lower inferred weights for manually verified false negatives, translating into improved downstream VQA answers with negligible training overhead.

Load-bearing premise

The method assumes that the weights it infers from similarity scores genuinely measure whether an unpaired document is actually relevant, rather than just re-encoding those scores in another form; if that fails, the Bayesian machinery adds nothing beyond a tuned similarity threshold.

Editorial extensions

If this is right

A retriever trained with BDR should rank relevant documents higher at small K, where retrieval budgets for downstream retrieval-augmented generation are tightest.
The Gamma-prior variant is the recommended configuration: it gives the best average Recall@K across the seven M2KR benchmarks, and continuous reweighting outperforms Bernoulli hard gating.
BDR works across model scales from 2B to 7B and attaches to the training objective rather than the architecture, so existing fine-tuned multimodal retrievers can adopt it directly.
When the BDR-trained retriever is paired with a frozen generative VQA model, downstream answer accuracy on InfoSeek and EVQA improves, narrowing the gap to oracle retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond KB-VQA, the same latent-weight construction could transfer to other contrastive learners with noisy negatives, such as image-text pretraining or open-domain passage retrieval; the paper only tests KB-VQA, so this is an extension rather than a result.
Because the Gamma posterior mean is a decreasing function of negative similarity, BDR's practical behavior resembles a soft similarity gate; the Bayesian framing matters mainly if the posterior weights are better calibrated than raw scores, which is directly testable.
A controlled comparison against a well-tuned deterministic similarity-threshold reweighting scheme would isolate how much of the gain comes from the probabilistic inference itself rather than from the simple act of downweighting high-similarity negatives.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's the short version: the paper proposes a simple, effective trick for contrastive retrieval — downweight negatives that look semantically similar to the query — and shows it consistently helps KB-VQA across several benchmarks and backbones. The Bayesian apparatus is more packaging than substance, but the underlying idea is sound and the experiments are substantial enough to take seriously.

The genuinely new piece is the Laplace augmentation: introducing an auxiliary variable per sample turns a non-factorable pseudo-likelihood into a conditionally conjugate problem, giving closed-form Gibbs updates for the latent weights. That's nontrivial and could carry over to other contrastive settings. The weight updates under a Gamma prior are effectively a decreasing function of similarity, so you could approximate this with a hand-designed schedule, but the closed-form derivation provides a coherent path. The paper also includes a useful calibration analysis showing inferred weights track manually labeled false negatives.

Soft spots. First, the 'Bayesian' posterior is explicitly a pseudo-likelihood, and Algorithm 1 is not actually SAEM as Delyon et al. define it (no stochastic smoothing step). The convergence guarantees from SAEM therefore do not apply, and the paper offers no alternative analysis. That weakens the 'principled' claim, but not necessarily the empirical results. Second, the main comparisons in Tables 1 and 2 use test corpora filtered to ground-truth-containing documents; gains there could be inflated relative to a full open corpus. The stress-test note says every number is filtered, which is not right: Table 3 reports full test-set results, and BDR still improves, though with smaller margins. So the concern is real but not fatal. Third, no code or data, and hyperparameter defaults for the main runs are not stated; the sensitivity analysis only varies them on one dataset.

Bottom line: a competent empirical paper with a modest but real contribution. The Bayesian framing overpromises, but the method works and is cheap. A referee should ask for open-corpus results across all datasets, a corrected optimization description, and code release. I'd send it to peer review.

Referee Report

4 major / 6 minor

Summary. The paper proposes Bayesian Data Reweighting (BDR), a training objective for contrastively learned multimodal retrievers in knowledge-based VQA. BDR introduces latent importance weights w_ik for each negative query-document pair, reformulates the InfoNCE objective as a sample-wise pseudo-likelihood, and uses Laplace augmentation to obtain conditionally conjugate closed-form updates under Gamma or Bernoulli priors. Optimization is described as stochastic approximation EM. Experiments compare BDR against InfoNCE and several reweighting baselines on filtered subsets of OKVQA, EVQA, and the M2KR benchmarks, reporting consistent Recall@K gains across three retriever backbones, improved downstream answer generation with frozen generators, a weight-calibration analysis, and negligible training overhead.

Significance. If the empirical claims hold on the open retrieval corpus, BDR would be a practical, low-overhead training objective for KB-VQA retrieval: it is simple to implement, adds negligible cost, and is evaluated across a broad set of benchmarks and backbones. The paper's strengths include the breadth of the experiments, the explicit calibration analysis linking learned weights to false-negative prevalence, and the efficiency measurements. However, the significance is currently bounded by three issues: all retrieval numbers are obtained on a test corpus filtered to ground-truth-contained documents; the reweighting behavior is largely dictated by the prior rather than discovered from data; and the proposed optimization is not actually the stochastic approximation EM that the paper claims. These concerns affect the central claim of consistent improvement for the open-domain task that motivates the work.

major comments (4)

[Section 3.2] The claim that BDR 'adaptively infers posterior weights' and thereby discovers false negatives is an encoding of the prior, not an inference outcome. With the Gamma prior, E[w_ik | -] = a_w / (b_w + u_i s_ik-), which is monotonically decreasing in similarity by construction; the Bernoulli posterior has the same monotone behavior. The observed downweighting of annotated false negatives in Figure 4 is therefore a direct consequence of the chosen prior family. To substantiate the Bayesian contribution, the authors should compare BDR against a deterministic schedule with the same functional form, e.g., w_ik = 1/(b + u s_ik-) with matched hyperparameters, or demonstrate that the posterior updates change the ranking of weights beyond this monotone schedule. Without such a comparison, the calibration analysis does not validate the Bayesian mechanism.
[Algorithm 1 and Section 3.3] The algorithm is not stochastic approximation EM as claimed. SAEM requires a stochastic approximation update of the surrogate objective, typically Q_{t+1} = Q_t + rho_t (Q_hat - Q_t) with a step size rho_t satisfying the Robbins-Monro conditions. Algorithm 1 instead sets Q_{t+1}(theta) <- L_BDR(B_t;theta), which is a single-sample Monte Carlo EM update, and no convergence analysis is provided for the pseudo-likelihood objective. Since the paper explicitly claims 'stochastic approximation EM' and cites Delyon et al. (1999), the optimization section must either be corrected to describe the actual algorithm, or the claims about SAEM and convergence should be weakened to 'EM-style optimization with a finite-sample surrogate.'
[Section 4.1] All retrieval results in Tables 1-3 are produced on a test corpus filtered to retain only ground-truth-contained documents, not on the open corpus that the introduction motivates. In a filtered corpus, the negative set is enriched with passages that are gold for other queries, which is precisely the regime where downweighting high-similarity negatives can have the largest effect. The paper's Limitations section explicitly defers full open-corpus validation, yet the abstract states 'consistently improves retrieval accuracy across seven knowledge-based VQA benchmarks' without this qualification. The central empirical claim therefore needs either an open-corpus evaluation or a concrete argument, with numbers, that the filtered protocol is representative. As it stands, the delivered claim is established only for a restricted benchmark, not for the open-domain KB-VQA retrieval task the paper motivates.
[Section 3] The construction p(w,theta|X) proportional to product over i of pi(w_i,theta) treats a product of sample-wise pseudo-likelihoods as a joint posterior, although pi(w_i,theta) is not a normalized likelihood over any data-generating process and the shared parameter theta couples all samples. The Limitations section acknowledges that this is not a fully normalized posterior, but the 'principled Bayesian manner' framing and the closed-form posterior derivations in Section 3.2 rest on this step. The paper should state explicitly what statistical object is being optimized (e.g., a penalized surrogate objective) and show that the conditional updates used are consistent with that object. Otherwise, the probabilistic interpretation is largely cosmetic, and the method reduces to a similarity-threshold reweighting schedule with tunable hyperparameters.

minor comments (6)

[Table 3] The 'Average' column averages R@5 for OVEN and KVQA with R@1 for LLaVA, so it is not a single retrieval metric; please report per-metric averages or clarify the aggregation.
[Table 4] The row label 'Qwen2-VL-7B (Ours)' is ambiguous; it presumably denotes Qwen2-VL-7B trained with BDR, and the retriever name should appear consistently across tables.
[Throughout] The dataset name 'LLaV A' should be 'LLaVA' throughout, and Section 4.4 contains the typo 'Tables 5' for 'Table 5'.
[Algorithm 1] The loop 'for n = 1,...,N' samples w_ik, but N is not defined in the algorithm's input line or in the surrounding text; please define it explicitly.
[Page 1 figure material] The text after the introduction containing 'This passage provides related background about Iris setosa...' and similar lines appears to be corpus annotation or a corrupted figure caption; please remove or properly integrate this material.
[Reproducibility] No code or checkpoints are released, which hampers reproduction of the filtered evaluation; please release the code and training/evaluation scripts.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor self-confirming calibration analysis; the core Bayesian derivation and external benchmark comparisons are not circular.

self definitional [Section 4.3, Weight Calibration Analysis, with Section 3.2, Gamma Prior]
"E[wik|−] = aw/(bw + uisik−). ... when a negative sample has high similarity to the query, sik− increases the posterior rate and decreases the expected weight. ... We first use an external LLM embedding model to measure passage-level semantic similarity, followed by manual verification to identify false negatives (FNs). ... the FN proportion decreases as the weight decile increases, indicating that lower-weight bins contain more FNs."

Under the Gamma prior, the BDR expected weight is a deterministic decreasing function of the retriever's similarity sik−, with only fixed hyperparameters and the auxiliary ui; no FN label enters the weight computation. The calibration study identifies FNs by first screening for high semantic similarity with an external LLM and then manually verifying those candidates. Because FNs are thereby concentrated among high-similarity negatives, and because the BDR weight is constructed to be low for high-similarity negatives, the reported correlation (lower weights for FNs) is largely entailed by the two definitions.

full rationale

The main derivation in Section 3 is self-contained: the closed-form conditional posteriors follow from the stated Laplace augmentation and conjugate Gamma/Bernoulli priors, and no external theorem or prior paper is invoked as a load-bearing uniqueness argument. The central retrieval claims are evaluated against InfoNCE and several reweighting baselines on held-out test sets, so those improvements are not fitted inputs or renamed predictions. The only circular-adjacent element is the weight-calibration analysis: FNs are identified using semantic similarity while the Gamma-prior expected weight is a decreasing function of similarity, making part of the reported calibration structurally expected. The paper also honestly notes in the Limitations that the posterior is a pseudo-likelihood rather than a fully normalized posterior, which weakens the 'principled Bayesian' framing but is not itself circularity. The filtered test corpus is a validity concern for open-corpus generalization, not a circularity issue. Overall, the derivation chain is not circular in its core claims.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on five unproved inputs: the pseudo-likelihood treatment, the Laplace identity (standard), conjugate-prior calculations (standard), the inductive bias that high-similarity negatives should be downweighted, and the filtered-test-corpus assumption. The first and last are the most consequential: the pseudo-likelihood is acknowledged as non-normalized, and the filtered evaluation limits the external validity of the empirical claim.

free parameters (5)

b_w (Gamma prior rate for negative weights) = not stated for main tables; sensitivity range 0.1-5.0
Controls the scale of downweighting for high-similarity negatives; tuned by hand in Section 4.4.
b_u (Gamma prior rate for auxiliary variable u) = not stated for main tables; sensitivity range 0.1-20.0
Controls prior strength on the normalization auxiliary variable; too large values degrade performance per Table 5.
p (Bernoulli retaining probability) = not stated for main tables; sensitivity range 0.5-0.95
Prior probability of keeping a negative in the Bernoulli variant; tuned per dataset in Section 4.4.
a_w = a_u = 1 (Gamma shape parameters) = 1 (fixed)
Shape parameters fixed to 1 without sensitivity analysis; they affect the weight schedule.
temperature tau = 0.02
InfoNCE temperature inherited from the base model; it scales similarities and therefore influences the posterior weights.

assumptions (5)

ad hoc to paper pseudo-likelihood as joint posterior: p(w,theta|X) proportional to product over i of s_i+/(s_i+ + sum_k w_ik s_ik-) times priors
Section 3 defines the posterior from the contrastive term; the Limitations section confirms it is not a fully normalized posterior.
standard math Laplace augmentation identity lambda^{-1} = integral_0^infty exp(-lambda u) du
Used in Section 3.1 to factorize the denominator; standard calculus.
standard math Conjugate Gamma/Bernoulli priors yield closed-form conditional posteriors for Gibbs sampling
Section 3.2 applies standard conjugate Bayesian updates.
domain assumption High-similarity unmatched documents are likely false negatives and should be downweighted
The expected weight E[w_ik]=a_w/(b_w+u_i s_ik-) encodes this assumption; it is motivated in Section 1 and Figure 1 but not tested independently.
domain assumption Filtered test corpora preserve the relative ranking of retrieval methods
Section 4.1 filters the test corpus to ground-truth-containing documents; reported improvements assume this ranking transfers to open-corpus retrieval.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering." pith.science (2026). https://pith.science/paper/7TV6CLXO

@misc{pith2026260802907,
  author       = {Pith},
  title        = {Pith review of: Bayesian Data Reweighting Improves Multimodal Retrieval for Knowledge-Based Visual Question Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7TV6CLXO}},
  note         = {Machine review of arXiv:2608.02907}
}

read the original abstract

Multimodal retrievers are essential for knowledge-based visual question answering, where they retrieve external evidence for image-question pairs. However, existing contrastive training methods typically treat all unmatched query-document pairs as equally informative negatives, which is problematic because many unmatched documents may still be semantically relevant or partially useful. We propose Bayesian Data Reweighting, a probabilistic framework that models query-document importance as latent variables and adaptively infers posterior weights to downweight likely false negatives. With closed-form posterior updates under conjugate priors and stochastic EM optimization, our method consistently improves retrieval accuracy across three retrievers and seven knowledge-based VQA benchmarks.

Figures

Figures reproduced from arXiv: 2608.02907 by the authors.

**Figure 1.** Examples of false negative reweighting. We select representative FN samples from EVQA to illustrate the false-negative issue in KB-VQA, where unpaired documents may still describe related entities or partially relevant evidence. Unlike standard InfoNCE, which uniformly treats these samples as negatives and pushes them away from the query, our method BDR adaptively infers weights for FN passages, suppressing noisy co… view at source ↗

**Figure 2.** Distribution of query–passage cosine similarities on EVQA. The dashed lines denote the mean of similarities, and the annotated gap measures the positive– negative separation. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Recall@K comparison on EVQA. We evaluate retrieval performance by varying K and reporting whether the ground-truth document appears in the topK results. 4.3 How does Bayesian Reweighting work? Decision Boundary Visualization. We analyze the cosine similarity distributions of positive and negative query–passage pairs to examine the effect of Bayesian Reweighting on the decision boundary. As shown in [PITH_FULL_IMA… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Weight calibration analysis. Left: FN samples receive lower Bayesian weights than TN samples on average. Right: the FN fraction generally decreases across increasing weight deciles, suggesting that lowerweight samples are more likely to be false negatives. Weight Cal…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

92 extracted references · 44 canonical work pages

[1]

Annals of statistics , pages=

Convergence of a stochastic approximation version of the EM algorithm , author=. Annals of statistics , pages=. 1999 , publisher=

1999
[2]

Aho and Jeffrey D

Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

1972
[3]

Publications Manual , year = "1983", publisher =

1983
[4]

Chandra and Dexter C

Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

arXiv 1981
[5]

Scalable training of

Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of
[6]

Dan Gusfield , title =. 1997

1997
[7]

Tetreault , title =

Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

2015
[8]

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

Show all 92 references

[9]

International Conference on Machine Learning , pages=

Position: Bayesian Deep Learning is Needed in the Age of Large-Scale AI , author=. International Conference on Machine Learning , pages=. 2024 , organization=

2024
[10]

International Conference on Machine Learning , pages=

Not all samples are created equal: Deep learning with importance sampling , author=. International Conference on Machine Learning , pages=. 2018 , organization=

2018
[11]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

End-to-end learning of visual representations from uncurated instructional videos , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[12]

International Conference on Machine Learning , pages=

Learning to reweight examples for robust deep learning , author=. International Conference on Machine Learning , pages=. 2018 , organization=

2018
[13]

International Conference on Machine Learning , pages=

Robust probabilistic modeling with bayesian data reweighting , author=. International Conference on Machine Learning , pages=. 2017 , organization=

2017
[14]

Proceedings of the AAAI Conference on Artificial Intelligence , year=

ConceptNet 5.5: An Open Multilingual Graph of General Knowledge , author=. Proceedings of the AAAI Conference on Artificial Intelligence , year=
[15]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=

Don't just assume; look and answer: Overcoming priors for visual question answering , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=
[16]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

Can pre-trained vision and language models answer visual information-seeking questions? , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

2024
[17]

Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

Encyclopedic vqa: Visual questions about detailed properties of fine-grained categories , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
[18]

Communications of the ACM , volume=

Wikidata: a free collaborative knowledgebase , author=. Communications of the ACM , volume=. 2014 , publisher=

2014
[19]

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

Retrieval Augmented Visual Question Answering with Outside Knowledge , author=. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

2022
[20]

Advances in Neural Information Processing Systems , volume=

Fine-grained late-interaction multi-modal retrieval for retrieval augmented visual question answering , author=. Advances in Neural Information Processing Systems , volume=
[21]

Lin, Weizhe and Mei, Jingbiao and Chen, Jinghong and Byrne, Bill , booktitle=
[22]

European Conference on Computer Vision , pages=

Uniir: Training and benchmarking universal multimodal information retrievers , author=. European Conference on Computer Vision , pages=. 2024 , organization=

2024
[23]

Proceedings of the Twelfth International Conference on Learning Representations (ICLR) , year=

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks , author=. Proceedings of the Twelfth International Conference on Learning Representations (ICLR) , year=
[24]

Liu, Yikun and Chen, Pingan and Cai, Jiayin and Jiang, Xiaolong and Hu, Yao and Yao, Jiangchao and Wang, Yanfeng and Xie, Weidi , booktitle=
[25]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=

Recurrence-Enhanced Vision-and-Language Transformers for Robust Multimodal Document Retrieval , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , year=
[26]

arXiv preprint arXiv:1807.03748 , year=

Representation learning with contrastive predictive coding , author=. arXiv preprint arXiv:1807.03748 , year=

arXiv
[27]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Understanding the behaviour of contrastive loss , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[28]

Advances in Neural Information Processing Systems , volume=

Debiased contrastive learning , author=. Advances in Neural Information Processing Systems , volume=
[29]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Self-supervised learning of pretext-invariant representations , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[30]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Momentum contrast for unsupervised visual representation learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[31]

Advances in Neural Information Processing Systems , volume=

Big self-supervised models are strong semi-supervised learners , author=. Advances in Neural Information Processing Systems , volume=
[32]

IEEE Transactions on Knowledge and Data Engineering , volume=

Self-supervised learning: Generative or contrastive , author=. IEEE Transactions on Knowledge and Data Engineering , volume=. 2021 , publisher=

2021
[33]

Bert: Pre-training of deep bidirectional transformers for language understanding , author=. Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , pages=

2019
[34]

International Conference on Learning Representations , year=

An image is worth 16x16 words: Transformers for image recognition at scale , author=. International Conference on Learning Representations , year=
[35]

IEEE Transactions on Big Data , volume=

Billion-scale similarity search with GPUs , author=. IEEE Transactions on Big Data , volume=. 2019 , publisher=

2019
[36]

Advances in Neural Information Processing Systems , volume=

Why do we need large batchsizes in contrastive learning? a gradient-bias perspective , author=. Advances in Neural Information Processing Systems , volume=
[37]

International Conference on Machine Learning , year=

Bayesian self-supervised contrastive learning , author=. International Conference on Machine Learning , year=
[38]

Escaping local maxima and handling intractable sampling , author=

A new class of stochastic EM algorithms. Escaping local maxima and handling intractable sampling , author=. Computational Statistics & Data Analysis , volume=. 2021 , publisher=

2021
[39]

Advances in Neural Information Processing Systems , volume=

Stochastic expectation maximization with variance reduction , author=. Advances in Neural Information Processing Systems , volume=
[40]

Annual Asian Computing Science Conference , pages=

Online stochastic and robust optimization , author=. Annual Asian Computing Science Conference , pages=. 2004 , organization=

2004
[41]

arXiv preprint arXiv:2112.08614 , year=

Kat: A knowledge augmented transformer for vision-and-language , author=. arXiv preprint arXiv:2112.08614 , year=

arXiv
[42]

arXiv preprint arXiv:2305.06407 , year=

Combo of thinking and observing for outside-knowledge vqa , author=. arXiv preprint arXiv:2305.06407 , year=

arXiv
[43]

Advances in Neural Information Processing Systems , volume=

Revive: Regional visual representation matters in knowledge-based visual question answering , author=. Advances in Neural Information Processing Systems , volume=
[44]

arXiv preprint arXiv:2404.14219 , year=

Phi-3 technical report: A highly capable language model locally on your phone , author=. arXiv preprint arXiv:2404.14219 , year=

arXiv
[45]

Qwen2.5-VL , url =

Qwen Team , month =. Qwen2.5-VL , url =
[46]

arXiv preprint arXiv:2409.12191 , year=

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution , author=. arXiv preprint arXiv:2409.12191 , year=

arXiv
[47]

2023 , eprint=

Improved Baselines with Visual Instruction Tuning , author=. 2023 , eprint=

2023
[48]

Advances in Neural Information Processing Systems , volume=

Flamingo: a visual language model for few-shot learning , author=. Advances in Neural Information Processing Systems , volume=
[49]

Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

Promptcap: Prompt-guided image captioning for vqa with gpt-3 , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
[50]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Prompting large language models with answer heuristics for knowledge-based visual question answering , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[51]

Proceedings of the AAAI conference on artificial intelligence , volume=

An empirical study of gpt-3 for few-shot knowledge-based vqa , author=. Proceedings of the AAAI conference on artificial intelligence , volume=
[52]

International Conference on Machine Learning , year=

Palm-e: An embodied multimodal language model , author=. International Conference on Machine Learning , year=
[53]

arXiv preprint arXiv:2303.08774 , year=

Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

arXiv
[54]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Facenet: A unified embedding for face recognition and clustering , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[55]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Class-balanced loss based on effective number of samples , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[56]

International Conference on Machine Learning , pages=

Learning transferable visual models from natural language supervision , author=. International Conference on Machine Learning , pages=. 2021 , organization=

2021
[57]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Hardness-aware deep metric learning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[58]

arXiv preprint arXiv:2010.04592 , year=

Contrastive learning with hard negative samples , author=. arXiv preprint arXiv:2010.04592 , year=

2010 arXiv
[59]

Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages=

Boosting contrastive self-supervised learning with false negative cancellation , author=. Proceedings of the IEEE/CVF winter conference on applications of computer vision , pages=
[60]

International Conference on Learning Representations , year=

Mm-embed: Universal multimodal retrieval with multimodal llms , author=. International Conference on Learning Representations , year=
[61]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=

OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=
[62]

European Conference on Computer Vision , pages=

A-okvqa: A benchmark for visual question answering using world knowledge , author=. European Conference on Computer Vision , pages=. 2022 , organization=

2022
[63]

Proceedings of the AAAI Conference on Artificial Intelligence , volume=

Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=
[64]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Wiki-llava: Hierarchical retrieval-augmented generation for multimodal llms , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[65]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

EchoSight: Advancing Visual-Language Models with Wiki Knowledge , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

2024
[66]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

Large Language Models Know What is Key Visual Entity: An LLM-assisted Multimodal Retrieval for VQA , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

2024
[67]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

From recognition to cognition: Visual commonsense reasoning , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[68]

The Thirteenth International Conference on Learning Representations , year=

Colpali: Efficient document retrieval with vision language models , author=. The Thirteenth International Conference on Learning Representations , year=
[69]

Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

Self-Bootstrapped Visual-Language Model for Knowledge Selection and Question Answering , author=. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

2024
[70]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[71]

International Conference on Learning Representations , year=

Relative representations enable zero-shot latent space communication , author=. International Conference on Learning Representations , year=
[72]

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

Weakly-supervised visual-retriever-reader for knowledge-based question answering , author=. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP) , year=

2021
[73]

arXiv preprint arXiv:2112.01488 , year=

Colbertv2: Effective and efficient retrieval via lightweight late interaction , author=. arXiv preprint arXiv:2112.01488 , year=

arXiv
[74]

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

Dense passage retrieval for open-domain question answering , author=. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages=

2020
[75]

Foundations and Trends

The probabilistic relevance framework: BM25 and beyond , author=. Foundations and Trends. 2009 , publisher=

2009
[76]

International Conference on Machine Learning , pages=

IGLUE: A benchmark for transfer learning across modalities, tasks, and languages , author=. International Conference on Machine Learning , pages=. 2022 , organization=

2022
[77]

Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages=

Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning , author=. Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval , pages=
[78]

Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=

Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities , author=. Proceedings of the IEEE/CVF International Conference on Computer Vision , pages=
[79]

Advances in Neural Information Processing Systems , volume=

Visual instruction tuning , author=. Advances in Neural Information Processing Systems , volume=
[80]

Advances in neural information processing systems , volume=

Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS) , author=. Advances in neural information processing systems , volume=
[81]

Journal of the American statistical Association , volume=

The calculation of posterior distributions by data augmentation , author=. Journal of the American statistical Association , volume=. 1987 , publisher=

1987
[82]

International Conference on Learning Representations , year=

Conditional negative sampling for contrastive learning of visual representations , author=. International Conference on Learning Representations , year=
[83]

Advances in neural information processing systems , volume=

Hard negative mixing for contrastive learning , author=. Advances in neural information processing systems , volume=
[84]

International Conference on Learning Representations , year=

Approximate nearest neighbor negative contrastive learning for dense text retrieval , author=. International Conference on Learning Representations , year=
[85]

arXiv preprint arXiv:2106.03719 , year=

Incremental false negative detection for contrastive learning , author=. arXiv preprint arXiv:2106.03719 , year=

arXiv
[86]

Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=

Mafa: Managing false negatives for vision-language pre-training , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages=
[87]

International conference on machine learning , pages=

Robust probabilistic modeling with bayesian data reweighting , author=. International conference on machine learning , pages=. 2017 , organization=

2017
[88]

Statistics and Computing , volume=

Exact Bayesian inference via data augmentation , author=. Statistics and Computing , volume=. 2015 , publisher=

2015
[89]

Proceedings of the IEEE international conference on computer vision , pages=

Focal loss for dense object detection , author=. Proceedings of the IEEE international conference on computer vision , pages=
[90]

arXiv preprint arXiv:1707.05612 , year=

Vse++: Improving visual-semantic embeddings with hard negatives , author=. arXiv preprint arXiv:1707.05612 , year=

arXiv
[91]

European conference on computer vision , pages=

Decoupled contrastive learning , author=. European conference on computer vision , pages=. 2022 , organization=

2022
[92]

Proceedings of the IEEE international conference on computer vision , pages=

Sampling matters in deep embedding learning , author=. Proceedings of the IEEE international conference on computer vision , pages=

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Annals of statistics , pages=

Convergence of a stochastic approximation version of the EM algorithm , author=. Annals of statistics , pages=. 1999 , publisher=

1999

[2] [2]

Aho and Jeffrey D

Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

1972

[3] [3]

Publications Manual , year = "1983", publisher =

1983

[4] [4]

Chandra and Dexter C

Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

arXiv 1981

[5] [5]

Scalable training of

Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

[6] [6]

Dan Gusfield , title =. 1997

1997

[7] [7]

Tetreault , title =

Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

2015

[8] [8]

A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =