{"id":"97a1b364-e3af-4f5f-b0ce-95dd2f1f4faa","arxiv_id":"2412.18800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"BRMGR independently reranks retrieved and LLM-generated passages with zero-shot likelihood scores and combines them by greedy matching, giving small exact match gains on open-domain QA.","lead":"The paper proposes BRMGR, an unsupervised method that reranks retrieved passages and LLM-generated passages separately, then pairs them by rank to combine both knowledge sources for question answering. It reports 1 to 2 point exact match gains over the COMBO baseline on two of three QA datasets and a comparable result on the third.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theorem's justification for greedy matching is ambiguous: under standard log-loss the factorization makes every matching optimal, and without a pairing-only ablation the claimed optimality is untested.","rationale":"Stress-test reading: the paper's headline empirical result is a +1.7/+1.6 exact-match gain over COMBO on NQ/WebQ with FiD, with the theoretical contribution being the claim that independently reranking both sources and pairing by rank is equivalent to optimal bipartite matching. That equivalence is what makes the method sound rather than ad hoc. After checking the proof, the equivalence holds for the sum-of-products objective, but the paper invokes the standard bipartite matching loss, which is additive in logs and hence permutation-invariant under factorization. This is not a direct contradiction, but it means the theorem's force depends on which objective is intended, and the paper does not clarify. More importantly, the factorization premise is not tested. A retrieved passage and a generated passage are not conditionally independent given the query: they often share answer strings or topic phrasing, and a compatibility-oriented method like COMBO exists precisely to model such interactions. If those interactions matter, greedy rank pairing could pair a misleading retrieved passage with a generated passage that contains the answer, diluting the signal for FiD. The absence of a pairing-only ablation means the empirical contribution of the greedy matching step is unidentified. This is a genuine gap, but it does not overturn the empirical result; it makes the result conditional on an assumption. Hence the reader's CONDITIONAL verdict stands. My read agrees with the reader's weakest_assumption in spirit, and the concrete test proposed would settle whether the assumption is harmful.","tokens_in":7655,"tokens_out":11877,"duration_ms":109953,"concrete_test":"On the NQ test set, keep BRMGR's independently reranked lists (same 10 generated and 10 retrieved passages, same order from Eqs. 4 and 5), but compare three pairings: (a) greedy rank pairing; (b) optimal bipartite matching using a non-factorized pair score, e.g., a cross-encoder scoring (q, lpi, rpj) jointly or COMBO's trained compatibility discriminator; (c) random pairing averaged over several seeds. Report FiD exact match for each. If (b) significantly exceeds (a), the factorization assumption is load-bearing and Theorem 1's premise is violated; if (a) and (b) are close, the heuristic is benign and the theoretical simplification is acceptable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is Theorem 1, which states that greedy matching is equivalent to optimal bipartite matching when Eq. (1) holds. The paper's proof maximizes the sum of a_ij = b_i c_j, for which greedy rank pairing is indeed optimal by the rearrangement inequality. However, the cited 'bipartite matching loss' in set prediction is normally the sum of negative log-likelihoods. Under that loss, if p(lpi,rpj|q) = b_i c_j, the total cost is -sum_i log b_i - sum_j log c_j, which is invariant to the permutation; every one-to-one matching is optimal, and the Hungarian algorithm does not select the rank pairing. Thus the equivalence either is vacuous (under log-loss) or rests on an unvalidated factorization (under a product objective). The factorization itself is strong: retrieved and generated passages are often correlated (e.g., both contain the same answer span or are about the same subtopic), so p(lpi,rpj|q) may deviate substantially from p(lpi|q)p(rpj|q). The paper never checks this empirically. In particular, there is no ablation that keeps the two independently reranked lists and varies only the pairing mechanism. Without such a check, a reader cannot tell whether the reported gains come from reranking each source individually or from the greedy matching step that the paper emphasizes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BRMGR, an unsupervised method for combining retrieved passages and LLM-generated passages in open-domain QA. Each source is reranked independently with zero-shot likelihood scoring (UPR-style query likelihood for retrieved passages, passage likelihood conditioned on the query for generated passages), and the two ranked lists are merged by greedy rank pairing. The paper claims that, under a conditional-independence factorization of the combination relevance score (Eq. 1), greedy matching is equivalent to optimal bipartite matching (Theorem 1). Experiments on TriviaQA, NQ, and WebQ report exact-match improvements of +1.7 and +1.6 over the COMBO baseline on NQ and WebQ, with comparable performance on TriviaQA, using FiD as the reader and T0-3B as the reranker.","tokens_in":7861,"tokens_out":3546,"duration_ms":34232,"significance":"If the method holds up, it is a useful contribution: it avoids silver-label mining, uses only off-the-shelf pretrained models, and is simple to reproduce. The paper is honest about its unsupervised nature and does not fit parameters to the evaluation data, which is a genuine strength. However, the theoretical justification is the load-bearing part of the contribution, and it is currently ambiguous. The empirical gains are small and reported without variance or significance testing, so the central claim that greedy rank pairing is both optimal and responsible for the gains is not yet established. With a clarified theorem and a direct pairing ablation, the method would be a solid incremental contribution to retrieval/generation fusion for QA.","major_comments":[{"comment":"The theorem's statement is ambiguous about the 'bipartite matching loss' cited from [23,24]. If that loss is the standard set-prediction negative log-likelihood, then under Eq. (1) the total loss for any bijection pi is -sum_i log p(lp_i|q) - sum_j log p(rp_pi(j)|q), which is independent of the permutation; every matching is optimal, and greedy rank pairing is only one of many optimal matchings, so the claimed equivalence to the Hungarian solution is vacuous. If instead the intended objective is maximizing sum_i b_i c_pi(i), the proof by induction is incomplete: the step from 'the top-1 combination is optimal' to 'the remaining pairs are solved greedily' requires an explicit exchange argument or an appeal to the rearrangement inequality, and the equal-cardinality assumption should be stated in Eq. (1) rather than only in the theorem.","section":"Section II, Theorem 1"},{"comment":"The factorization p(lp_i,rp_j|q) = p(lp_i|q)p(rp_j|q) is a strong conditional-independence assumption that is never tested. Retrieved and generated passages can be correlated (e.g., both contain the same answer span or concern the same subtopic), and when the assumption fails the greedy matching is not guaranteed to be optimal. The paper contains no ablation that varies only the pairing mechanism while keeping the independently reranked lists fixed; without a comparison against random pairing, Hungarian on the full score matrix, or an alternative pairing rule, the reported gains cannot be attributed to the greedy matching step that Theorem 1 is meant to justify.","section":"Section II, Eq. (1); Section III-C"},{"comment":"The main QA results are reported from a single run with no standard deviations, confidence intervals, or significance tests, and the gains over COMBO are 1-2 exact-match points (NQ +1.7, WebQ +1.6, TriviaQA -0.2). The claim that BRMGR 'achieves the strongest overall performance' is therefore not supported at a conventional statistical standard. The authors should report variance across multiple seeds or a paired significance test, and ideally show results with the other reranker variants (Flan-T5 base/large/xlarge) rather than only T0-3B.","section":"Section III-B, Table II"}],"minor_comments":[{"comment":"The derivation switches between p(rp_j|q) and p(q|rp_j) without clearly stating the uniform-prior assumption each time; Eq. (6) should state the proportionality constant and the fact that p(rp_j) is assumed uniform.","section":"Section II, Eqs. (2)-(6)"},{"comment":"The figure labels contain typographical artifacts ('R e rieved Knowledge', 'Genera ed Knowledge', 'Exac  Ma ch Score'); these should be corrected.","section":"Figure 1"},{"comment":"The text refers to 'Flan-T5 models' but the table lists 'T5-base', 'T5-large', and 'T5-xlarge'; please clarify whether these are Flan-T5 variants and keep the naming consistent.","section":"Section III-C, Table III"},{"comment":"Figure 3 shows reranking improvements without error bars or significance information, so it is hard to judge whether the differences between p(lp|q) and p(q|lp) are meaningful beyond the one development set.","section":"Figure 3"},{"comment":"The conclusion contains a sentence fragment ('Rather than relying on mined silver labels for computing compatibility scores between the two types of passages.'); it should be merged with the following sentence.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The central theoretical claim needs reworking: under the standard log-loss interpretation, Theorem 1 is vacuous, and under the product-score interpretation the proof is incomplete and the factorization is unvalidated. The single most useful addition would be a pairing-only ablation. The self-citation [24] is used as background, but the reliance on it for the theorem's framing makes the ambiguity more consequential. Scope is appropriate for a QA/IR journal if the experimental claims are tightened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful bit in this paper is the empirical observation that p(lp|q) reranks LLM-generated passages better than the usual query-likelihood p(q|lp), and that merging two independently reranked lists by rank is a cheap zero-shot replacement for COMBO. The paper also shows a sensible extension of UPR to generated passages. Those are real, though modest, contributions.\n\nThe numbers support the method only weakly. BRMGR beats COMBO by 1.7 EM on NQ and 1.6 on WebQ, and ties on TriviaQA. The gains are small and the paper reports no variance or significance tests, so the results are suggestive, not conclusive.\n\nThe stress-test note lands. The theorem is the weakest part. The paper never defines the \"bipartite matching loss\" it claims equivalence with. If that loss is the standard sum of negative log-likelihoods, then under the factorization in Eq. (1) every matching has the same loss, and the greedy equivalence is vacuous. If the objective is instead maximizing the sum of products b_i c_j, then greedy rank pairing is optimal by the rearrangement inequality, but the paper doesn't state or prove that cleanly. Either way the theorem doesn't carry the weight the paper places on it. The factorization itself is also strong: retrieved and generated passages can be correlated, and nobody checks. There is no ablation that keeps the two reranked lists fixed and varies only the pairing, so the reader cannot tell whether the gains come from the individual rerankers or from the matching step. That is the one experiment I would want before believing the central claim.\n\nThe citation pattern looks normal; the one self-citation is to a set-prediction paper and is not padding. No code or data is shipped, and the theorem is not machine-checked, so the reproducible surface is only the tables.\n\nThis is a competent, useful short paper, not a breakthrough. A reader working on zero-shot fusion or retrieval-augmented QA will get something from it. I would send it to serious review, but with the expectation that the authors either tighten the theorem or drop it, and add the pairing ablation plus error bars. It belongs at a workshop or a short-paper track, not at a main conference in its current form.","headline":"Useful empirical recipe for zero-shot fusion of retrieved and generated passages, but the theorem is vacuous under the usual loss and the gains are thin.","tokens_in":8429,"tokens_out":3036,"would_cite":false,"duration_ms":29914,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Independent reranking plus greedy rank pairing merges retrieved and generated knowledge optimally under a factorized score, lifting exact match on NQ and WebQ.","keywords":["open-domain question answering","passage reranking","retrieved knowledge","generated knowledge","greedy matching","bipartite matching","zero-shot generation","unsupervised reranking"],"falsifier":"Test the theorem directly: on real or constructed questions where a generated passage duplicates a retrieved passage, compare rank-pair greedy merging against exhaustive optimal bipartite matching over all pairings; if the two methods ever select different top-K pairs or different exact-match scores, the factorized-score assumption has failed.","tokens_in":7434,"feed_emoji":"📚","tokens_out":9744,"duration_ms":79994,"temperature":0.7,"pith_summary":"Open-domain QA usually has two imperfect knowledge sources: passages retrieved from a corpus and passages generated by an LLM. This paper argues that they can be merged without any labels saying which generated passage goes with which retrieved passage. Its method reranks each source with a pretrained language model, then pairs the two ranked lists in order; a theorem shows that under a factorized relevance score this greedy pairing equals optimal bipartite matching. On Natural Questions and WebQuestions the merged set outperforms the trained COMBO baseline by +1.7 and +1.6 exact match, and matches it on TriviaQA. The practical point is that knowledge fusion can improve with a zero-shot, label-free procedure.","feed_headline":"Reranking both sources lifts QA by +1.7 on NQ and +1.6 on WebQ","feed_subtitle":"A zero-shot reranker pairs each retrieved passage with a generated one, no labels needed.","key_machinery":"The engine of the method is the factorized combination score $p(lp_i,rp_j\\mid q)=p(lp_i\\mid q)\\,p(rp_j\\mid q)$, with each factor computed as an average token log-likelihood by a frozen pretrained language model. For retrieved passages the relevant factor is $p(q\\mid rp_j)$, the UPR zero-shot query-likelihood score, and for generated passages it is $p(lp_i\\mid q)$, the likelihood of generating the passage from the question. This outer-product structure turns the $M\\times N$ pairing problem into two independent rankings, so the optimal merge is achieved by sorting each list and matching by rank. Theorem 1 formalizes the equivalence to bipartite matching when the lists have equal length, which is why no Hungarian assignment is actually needed.","core_discovery":"The central claim is that the best merged set of retrieved and generated passages is obtained by independently scoring each source and gluing them rank-to-rank. Retrieved passages are scored by the log-likelihood of generating the question from the passage; generated passages are scored by the log-likelihood of generating the passage from the question. Because the joint score is assumed to factorize, the $10\\times10$ compatibility matrix is an outer product, and the paper proves that greedy matching over the sorted lists recovers the same pairing as the Hungarian-algorithm bipartite match. In FiD-based evaluation, this unsupervised scheme gives the strongest overall exact match of the compared methods, improving over COMBO by +1.7 on NQ and +1.6 on WebQ and staying comparable on TriviaQA.","pith_inferences":["The independence assumption is most vulnerable when generated passages paraphrase the retrieved ones or both are drawn from the same underlying document; there, a learned compatibility term could add signal the factorized score misses.","The equality proof assumes equal numbers of generated and retrieved passages, so practical deployments with unequal list sizes will need a padding or selection rule before greedy pairing is provably optimal.","Document-likelihood reranking of LLM output could serve as a general zero-shot filter for generated evidence in other knowledge-intensive tasks such as fact verification, beyond question answering."],"forward_implications":["With factorization, merging is just reranking each source and pairing by rank; no assignment solver or trained compatibility scorer is required.","Reranking generated passages by $p(lp\\mid q)$ helps, while reranking them by $p(q\\mid lp)$ hurts, indicating that document-likelihood scoring captures what makes generated passages useful.","The label-free recipe transfers to new domains by replacing the pretrained scorer, since no silver labels or discriminators are involved.","On two of three benchmarks, the merged reranked set beats the strongest compared compatibility-based baseline by more than a point of exact match."],"supporting_citations":[{"why":"Supplies the zero-shot query-likelihood reranking (UPR) used to score retrieved passages.","marker":"[18]"},{"why":"Supplies the LLM-generated passages that form the generated knowledge source.","marker":"[4]"},{"why":"Defines the COMBO compatibility-based merge used as the main baseline and the dataset setup.","marker":"[13]"},{"why":"Provides the Fusion-in-Decoder reader that converts the merged passage sets into final answers.","marker":"[3]"},{"why":"Provides the Dense Passage Retrieval model that produces the top-10 retrieved passages.","marker":"[2]"},{"why":"Provides the Hungarian algorithm that the equivalence argument uses to define optimal bipartite matching.","marker":"[25]"},{"why":"Provides the Flan-T5 models used as reranking language models.","marker":"[31]"},{"why":"Provides the T0-3B model used for reranking in the main retrieval experiments.","marker":"[32]"}],"fun_headline_variants":["Zero-shot reranking pairs passages, boosts NQ by +1.7","No labels needed: reranker lifts QA on NQ and WebQ","Rerank both sources: +1.7 NQ, +1.6 WebQ without training","Unsupervised pairing of passages yields gains on NQ, WebQ","Greedy matching of reranked passages: +1.7 on NQ, +1.6 on WebQ"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the assumption that, once the question is fixed, a retrieved passage's usefulness and a generated passage's usefulness do not affect each other, so their combined score is simply one score times the other.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot reranking pairs passages, boosts NQ by +1.7","No labels needed: reranker lifts QA on NQ and WebQ","Rerank both sources: +1.7 NQ, +1.6 WebQ without training","Unsupervised pairing of passages yields gains on NQ, WebQ","Greedy matching of reranked passages: +1.7 on NQ, +1.6 on WebQ"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000119,"raw_usage":{"total_tokens":1048,"prompt_tokens":872,"completion_tokens":176,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":488,"completion_tokens_details":{"reasoning_tokens":62}},"tokens_in":488,"tokens_out":176,"duration_ms":2064,"temperature":1.0,"reasoning_tokens":62,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:26:48.667117+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Test the theorem directly: on real or constructed questions where a generated passage duplicates a retrieved passage, compare rank-pair greedy merging against exhaustive optimal bipartite matching over all pairings; if the two methods ever select different top-K pairs or different exact-match scores, the factorized-score assumption has failed.","supporting_citations":[{"cited_title":"Improving passage retrieval with zero-shot question generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the zero-shot query-likelihood reranking (UPR) used to score retrieved passages."},{"cited_title":"Generate rather than retrieve: Large language models are strong context generators,","cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-generated passages that form the generated knowledge source."},{"cited_title":"Merging generated and retrieved knowledge for open-domain qa,","cited_arxiv_id":null,"evidence_quote":"Defines the COMBO compatibility-based merge used as the main baseline and the dataset setup."},{"cited_title":"Leveraging passage retrieval with generative models for open domain question answering,","cited_arxiv_id":null,"evidence_quote":"Provides the Fusion-in-Decoder reader that converts the merged passage sets into final answers."},{"cited_title":"Dense passage retrieval for open-domain question answering,","cited_arxiv_id":null,"evidence_quote":"Provides the Dense Passage Retrieval model that produces the top-10 retrieved passages."},{"cited_title":"Multitask prompted training enables zero-shot task generalization,","cited_arxiv_id":null,"evidence_quote":"Provides the T0-3B model used for reranking in the main retrieval experiments."}],"review_version":1}