{"id":"5c362fb8-46fa-4fcd-a6ef-5c3bc5dde259","arxiv_id":"2608.02907","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Bayesian Data Reweighting adaptively lowers the loss weight of likely false-negative documents, improving multimodal retrieval accuracy for knowledge-based visual question answering across several benchmarks.","lead":"This paper proposes Bayesian Data Reweighting, a training method that learns to downweight negative documents that look similar to the query during contrastive retrieval training, reducing the harm of false negatives. The method reports consistent retrieval gains for knowledge-based visual question answering, a task where external evidence must be retrieved from large text corpora.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Filtered test corpus (ground-truth-only passages) may inflate BDR's consistent gains; open-corpus evaluation is needed before accepting the central claim.","rationale":"The reader's named weakest assumption is the pseudo-likelihood foundation, but their rationale also identifies the filtered test corpus as a selection bias. I agree with the reader's CONDITIONAL verdict, and my concern is the filtered evaluation protocol because it directly threatens the empirical generalization claim that is the paper's central contribution. The pseudo-likelihood issue, while real, is explicitly acknowledged in the Limitations section and does not by itself invalidate the method's empirical value: even a heuristic reweighting rule can be useful, and the authors describe the inferred weights as uncertainty-aware importance weights rather than a strictly normalized posterior. The filtered-corpus issue is different: it affects every retrieval number in the paper and is not acknowledged in the main text, only deferred as future work. Since the reader already conditioned acceptance on open-corpus evaluation and code release, my attack reinforces that condition rather than changing the verdict. I mark agreement as partial because the reader's formal weakest_assumption field points at pseudo-likelihood, whereas I judge the test-corpus filter to be the more load-bearing concern for the central empirical assertion.","tokens_in":13967,"tokens_out":11275,"duration_ms":109530,"concrete_test":"Reproduce the Table 2 comparison (BDR Gamma vs. Uniform Negative Sampling, Debiased Negative Reweighting, and Hardness Negative Reweighting) on EVQA and InfoSeek with Qwen2-VL-2B, but retrieve from the full unfiltered M2KR corpus (millions of passages) instead of the ground-truth-only subset, using the same training runs. If BDR's Recall@1 and Recall@5 gains over baselines persist on the open corpus, the central claim survives; if the gains shrink or reverse, the filtered evaluation is the load-bearing artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that BDR consistently improves Recall@K over InfoNCE and existing reweighting baselines. Every reported retrieval number, however, is produced on a test corpus filtered to retain only ground-truth-contained documents (§4.1), not the open corpus that the method is motivated for. In such a filtered corpus, the negative set is enriched with passages that are gold for other queries; this is precisely the regime where downweighting high-similarity negatives can have the largest effect, because semantically related false negatives are overrepresented relative to the full corpus. The consistent gains on this filtered setup may therefore not transfer to the full open-corpus retrieval task described in the abstract and introduction. The paper's own Limitations section explicitly defers full open-corpus validation to future work, and no code or checkpoints are released to reproduce the filtered evaluation. Because the load-bearing assertion is the consistent relative improvement over baselines, and the evaluation protocol is the one place where that assertion could be an artifact of the benchmark construction rather than a property of the method, the claim is not yet established for the task the paper motivates.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Bayesian Data Reweighting (BDR), a training objective for contrastively learned multimodal retrievers in knowledge-based VQA. BDR introduces latent importance weights w_ik for each negative query-document pair, reformulates the InfoNCE objective as a sample-wise pseudo-likelihood, and uses Laplace augmentation to obtain conditionally conjugate closed-form updates under Gamma or Bernoulli priors. Optimization is described as stochastic approximation EM. Experiments compare BDR against InfoNCE and several reweighting baselines on filtered subsets of OKVQA, EVQA, and the M2KR benchmarks, reporting consistent Recall@K gains across three retriever backbones, improved downstream answer generation with frozen generators, a weight-calibration analysis, and negligible training overhead.","tokens_in":14183,"tokens_out":4244,"duration_ms":40312,"significance":"If the empirical claims hold on the open retrieval corpus, BDR would be a practical, low-overhead training objective for KB-VQA retrieval: it is simple to implement, adds negligible cost, and is evaluated across a broad set of benchmarks and backbones. The paper's strengths include the breadth of the experiments, the explicit calibration analysis linking learned weights to false-negative prevalence, and the efficiency measurements. However, the significance is currently bounded by three issues: all retrieval numbers are obtained on a test corpus filtered to ground-truth-contained documents; the reweighting behavior is largely dictated by the prior rather than discovered from data; and the proposed optimization is not actually the stochastic approximation EM that the paper claims. These concerns affect the central claim of consistent improvement for the open-domain task that motivates the work.","major_comments":[{"comment":"The claim that BDR 'adaptively infers posterior weights' and thereby discovers false negatives is an encoding of the prior, not an inference outcome. With the Gamma prior, E[w_ik | -] = a_w / (b_w + u_i s_ik-), which is monotonically decreasing in similarity by construction; the Bernoulli posterior has the same monotone behavior. The observed downweighting of annotated false negatives in Figure 4 is therefore a direct consequence of the chosen prior family. To substantiate the Bayesian contribution, the authors should compare BDR against a deterministic schedule with the same functional form, e.g., w_ik = 1/(b + u s_ik-) with matched hyperparameters, or demonstrate that the posterior updates change the ranking of weights beyond this monotone schedule. Without such a comparison, the calibration analysis does not validate the Bayesian mechanism.","section":"Section 3.2"},{"comment":"The algorithm is not stochastic approximation EM as claimed. SAEM requires a stochastic approximation update of the surrogate objective, typically Q_{t+1} = Q_t + rho_t (Q_hat - Q_t) with a step size rho_t satisfying the Robbins-Monro conditions. Algorithm 1 instead sets Q_{t+1}(theta) <- L_BDR(B_t;theta), which is a single-sample Monte Carlo EM update, and no convergence analysis is provided for the pseudo-likelihood objective. Since the paper explicitly claims 'stochastic approximation EM' and cites Delyon et al. (1999), the optimization section must either be corrected to describe the actual algorithm, or the claims about SAEM and convergence should be weakened to 'EM-style optimization with a finite-sample surrogate.'","section":"Algorithm 1 and Section 3.3"},{"comment":"All retrieval results in Tables 1-3 are produced on a test corpus filtered to retain only ground-truth-contained documents, not on the open corpus that the introduction motivates. In a filtered corpus, the negative set is enriched with passages that are gold for other queries, which is precisely the regime where downweighting high-similarity negatives can have the largest effect. The paper's Limitations section explicitly defers full open-corpus validation, yet the abstract states 'consistently improves retrieval accuracy across seven knowledge-based VQA benchmarks' without this qualification. The central empirical claim therefore needs either an open-corpus evaluation or a concrete argument, with numbers, that the filtered protocol is representative. As it stands, the delivered claim is established only for a restricted benchmark, not for the open-domain KB-VQA retrieval task the paper motivates.","section":"Section 4.1"},{"comment":"The construction p(w,theta|X) proportional to product over i of pi(w_i,theta) treats a product of sample-wise pseudo-likelihoods as a joint posterior, although pi(w_i,theta) is not a normalized likelihood over any data-generating process and the shared parameter theta couples all samples. The Limitations section acknowledges that this is not a fully normalized posterior, but the 'principled Bayesian manner' framing and the closed-form posterior derivations in Section 3.2 rest on this step. The paper should state explicitly what statistical object is being optimized (e.g., a penalized surrogate objective) and show that the conditional updates used are consistent with that object. Otherwise, the probabilistic interpretation is largely cosmetic, and the method reduces to a similarity-threshold reweighting schedule with tunable hyperparameters.","section":"Section 3"}],"minor_comments":[{"comment":"The 'Average' column averages R@5 for OVEN and KVQA with R@1 for LLaVA, so it is not a single retrieval metric; please report per-metric averages or clarify the aggregation.","section":"Table 3"},{"comment":"The row label 'Qwen2-VL-7B (Ours)' is ambiguous; it presumably denotes Qwen2-VL-7B trained with BDR, and the retriever name should appear consistently across tables.","section":"Table 4"},{"comment":"The dataset name 'LLaV A' should be 'LLaVA' throughout, and Section 4.4 contains the typo 'Tables 5' for 'Table 5'.","section":"Throughout"},{"comment":"The loop 'for n = 1,...,N' samples w_ik, but N is not defined in the algorithm's input line or in the surrounding text; please define it explicitly.","section":"Algorithm 1"},{"comment":"The text after the introduction containing 'This passage provides related background about Iris setosa...' and similar lines appears to be corpus annotation or a corrupted figure caption; please remove or properly integrate this material.","section":"Page 1 figure material"},{"comment":"No code or checkpoints are released, which hampers reproduction of the filtered evaluation; please release the code and training/evaluation scripts.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful and potentially publishable experimental contribution, but the evaluation protocol is the main load-bearing issue: the consistent gains may be an artifact of the filtered corpus. The optimization and pseudo-likelihood concerns are also substantive but addressable by reframing. I recommend major revision rather than rejection, because the empirical comparisons are internally coherent and the method is simple enough to re-evaluate on an open corpus within a revision cycle."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper proposes a simple, effective trick for contrastive retrieval — downweight negatives that look semantically similar to the query — and shows it consistently helps KB-VQA across several benchmarks and backbones. The Bayesian apparatus is more packaging than substance, but the underlying idea is sound and the experiments are substantial enough to take seriously.\n\nThe genuinely new piece is the Laplace augmentation: introducing an auxiliary variable per sample turns a non-factorable pseudo-likelihood into a conditionally conjugate problem, giving closed-form Gibbs updates for the latent weights. That's nontrivial and could carry over to other contrastive settings. The weight updates under a Gamma prior are effectively a decreasing function of similarity, so you could approximate this with a hand-designed schedule, but the closed-form derivation provides a coherent path. The paper also includes a useful calibration analysis showing inferred weights track manually labeled false negatives.\n\nSoft spots. First, the 'Bayesian' posterior is explicitly a pseudo-likelihood, and Algorithm 1 is not actually SAEM as Delyon et al. define it (no stochastic smoothing step). The convergence guarantees from SAEM therefore do not apply, and the paper offers no alternative analysis. That weakens the 'principled' claim, but not necessarily the empirical results. Second, the main comparisons in Tables 1 and 2 use test corpora filtered to ground-truth-containing documents; gains there could be inflated relative to a full open corpus. The stress-test note says every number is filtered, which is not right: Table 3 reports full test-set results, and BDR still improves, though with smaller margins. So the concern is real but not fatal. Third, no code or data, and hyperparameter defaults for the main runs are not stated; the sensitivity analysis only varies them on one dataset.\n\nBottom line: a competent empirical paper with a modest but real contribution. The Bayesian framing overpromises, but the method works and is cheap. A referee should ask for open-corpus results across all datasets, a corrected optimization description, and code release. I'd send it to peer review.","headline":"A useful reweighting trick for contrastive retrieval, dressed in Bayesian language; the empirical gains are consistent and worth engaging with, despite the pseudo-likelihood caveat.","tokens_in":14747,"tokens_out":4239,"would_cite":false,"duration_ms":34819,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Bayesian Data Reweighting, which infers per-negative importance weights via closed-form posterior updates, consistently improves retrieval and downstream answer generation for knowledge-based visual question answering.","keywords":["multimodal retrieval","knowledge-based visual question answering","contrastive learning","false negatives","Bayesian data reweighting","latent importance weights","stochastic expectation-maximization","retrieval recall"],"falsifier":"Construct or annotate a set of true false negatives in a KB-VQA corpus, train the same retriever with BDR and with a deterministic decreasing function of cosine similarity used as weights, and compare both Recall@K and how well the inferred weights predict the human false-negative labels. If the deterministic schedule matches BDR's retrieval gains and predicts the labels equally well, the posterior inference is not the active ingredient.","tokens_in":13721,"feed_emoji":"🔎","tokens_out":6692,"duration_ms":56479,"temperature":0.7,"pith_summary":"This paper claims that standard contrastive training for knowledge-based visual question answering is hurt by false negatives — unpaired documents that are actually relevant or partially useful — and that this can be fixed by reweighting each negative according to a Bayesian posterior over its importance. The proposed method, Bayesian Data Reweighting, replaces the uniform negative term in the InfoNCE loss with a weighted sum, treats each weight as a latent variable with a Gamma or Bernoulli prior, and infers it with closed-form updates inside a stochastic EM loop. Across three retriever backbones and seven KB-VQA benchmarks, the paper reports consistent gains in Recall@K and Pseudo Recall@K over InfoNCE and six existing reweighting baselines, and the trained retriever improves answer generation when paired with frozen generators on InfoSeek and EVQA. The practical significance would be a cheap, drop-in training objective that makes multimodal retrieval more robust to incomplete annotations in knowledge-intensive settings.","feed_headline":"Bayesian reweighting lifts VQA retrieval on 7 benchmarks","feed_subtitle":"It beats InfoNCE and existing reweighting baselines across datasets while adding almost no training overhead.","key_machinery":"The load-bearing object is the weighted contrastive objective $\\mathcal{L}_{\\text{BDR}}(\\mathcal{B};\\theta)=-\\frac{1}{B}\\sum_i \\log \\frac{s_{i+}}{s_{i+}+\\sum_k w_{ik}s_{ik-}}$, with latent importance weights $w_{ik}\\ge 0$. The enabling identity is the Laplace transform $\\lambda^{-1}=\\int_0^\\infty e^{-\\lambda u}\\,du$, applied to the normalization term $\\lambda_i=s_{i+}+\\sum_k w_{ik}s_{ik-}$; this turns the coupled denominator into a factorized product $s_{i+}e^{-u_i s_{i+}}\\prod_k e^{-u_i w_{ik}s_{ik-}}$, restoring conditional conjugacy. With $u_i\\sim\\text{Gamma}(a_u,b_u)$ and $w_{ik}\\sim\\text{Gamma}(a_w,b_w)$ (or Bernoulli), the conditional posteriors are closed-form, and stochastic approximation EM alternates sampling them with gradient updates to the encoder. The machinery's work is to convert what could be a heuristic similarity-threshold schedule into a posterior inference procedure: expected weights drop monotonically as negative similarity grows, so noisy supervision is suppressed instance-by-instance without needing false-negative labels.","core_discovery":"The central discovery is that the false-negative problem in KB-VQA retrieval can be reduced to a tractable Bayesian reweighting problem. The paper formulates a sample-wise pseudo-likelihood from the weighted contrastive objective, augments it with an auxiliary variable via the Laplace identity so that the denominator factorizes, and then derives closed-form conditional posteriors for the negative weights under Gamma and Bernoulli priors. The Gamma posterior expectation $a_w/(b_w+u_i s_{ik}^-)$ decreases with negative similarity, so the method automatically downweights the most suspicious false negatives; the Bernoulli posterior performs hard keep-or-drop gating. Optimized with stochastic approximation EM, this reweighting is what the paper argues produces better-separated query-document embeddings, a larger positive-negative similarity gap, higher Recall@K at every tested K, and lower inferred weights for manually verified false negatives, translating into improved downstream VQA answers with negligible training overhead.","pith_inferences":["Beyond KB-VQA, the same latent-weight construction could transfer to other contrastive learners with noisy negatives, such as image-text pretraining or open-domain passage retrieval; the paper only tests KB-VQA, so this is an extension rather than a result.","Because the Gamma posterior mean is a decreasing function of negative similarity, BDR's practical behavior resembles a soft similarity gate; the Bayesian framing matters mainly if the posterior weights are better calibrated than raw scores, which is directly testable.","A controlled comparison against a well-tuned deterministic similarity-threshold reweighting scheme would isolate how much of the gain comes from the probabilistic inference itself rather than from the simple act of downweighting high-similarity negatives."],"forward_implications":["A retriever trained with BDR should rank relevant documents higher at small K, where retrieval budgets for downstream retrieval-augmented generation are tightest.","The Gamma-prior variant is the recommended configuration: it gives the best average Recall@K across the seven M2KR benchmarks, and continuous reweighting outperforms Bernoulli hard gating.","BDR works across model scales from 2B to 7B and attaches to the training objective rather than the architecture, so existing fine-tuned multimodal retrievers can adopt it directly.","When the BDR-trained retriever is paired with a frozen generative VQA model, downstream answer accuracy on InfoSeek and EVQA improves, narrowing the gap to oracle retrieval."],"supporting_citations":[{"why":"Supplies the InfoNCE objective that BDR modifies by replacing uniform negative weights with latent importance weights.","marker":"Oord et al., 2018"},{"why":"Provides the Bayesian data reweighting formulation that motivates modeling sample importance as latent variables.","marker":"Wang et al., 2017"},{"why":"Supplies the data-augmentation technique used to factorize the non-conjugate normalization term.","marker":"Tanner and Wong, 1987"},{"why":"Gives the stochastic approximation EM algorithm used to alternate posterior sampling and retriever updates.","marker":"Delyon et al., 1999"},{"why":"Debiased contrastive learning baseline that BDR is compared against as a false-negative mitigation method.","marker":"Chuang et al., 2020"},{"why":"Hardness-aware negative reweighting baseline that BDR must beat in the retrieval comparisons.","marker":"Robinson et al., 2020"},{"why":"Provides the M2KR benchmark suite and the PreFLMR retriever baseline used for evaluation.","marker":"Lin et al., 2024"},{"why":"Supplies the VLM2Vec embedding setup, including last-token embedding and LoRA fine-tuning, adopted for the retrievers.","marker":"Jiang et al., 2025"}],"fun_headline_variants":["Bayesian reweighting downs false negatives in VQA retrieval","A Bayesian cure for false negatives in knowledge-based VQA","Reweight negatives Bayesian-style: VQA retrieval improves","Bayesian latent weights fix VQA's false negative problem","Adaptive Bayesian reweighting lifts VQA retrieval accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the weights it infers from similarity scores genuinely measure whether an unpaired document is actually relevant, rather than just re-encoding those scores in another form; if that fails, the Bayesian machinery adds nothing beyond a tuned similarity threshold.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian reweighting downs false negatives in VQA retrieval","A Bayesian cure for false negatives in knowledge-based VQA","Reweight negatives Bayesian-style: VQA retrieval improves","Bayesian latent weights fix VQA's false negative problem","Adaptive Bayesian reweighting lifts VQA retrieval accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1445,"prompt_tokens":838,"completion_tokens":607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":526}},"tokens_in":454,"tokens_out":607,"duration_ms":5718,"temperature":1.0,"reasoning_tokens":526,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:55:54.760456+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct or annotate a set of true false negatives in a KB-VQA corpus, train the same retriever with BDR and with a deterministic decreasing function of cosine similarity used as weights, and compare both Recall@K and how well the inferred weights predict the human false-negative labels. If the deterministic schedule matches BDR's retrieval gains and predicts the labels equally well, the posterior inference is not the active ingredient.","supporting_citations":[],"review_version":2}