{"id":"3d9a460a-cd59-430a-a909-73ed7c8bb20d","arxiv_id":"2509.02558","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"BRIGHT's BM25 results come from query-side BM25 weighting rather than standard bag-of-words, and the paper shows this variant helps on the benchmark's long queries while shipping reproducible toolkits and a corpus audit.","lead":"BRIGHT's published BM25 scores actually come from query-side BM25, a variant that scores query terms themselves, and this paper shows that variant beats standard BM25 on the benchmark's long reasoning queries. The paper ships reproducible baselines in Anserini, Pyserini, and RankLLM, and audits the BRIGHT corpus for duplicates and missing gold documents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"BM25Q advantage over bag-of-words is not established beyond noise: 0.011 nDCG@10 aggregate, 7-4-1 per-task, no significance test, post hoc length buckets; abstract's five-benchmark generalization claim is absent from the body (Section 5 only BRIGHT).","rationale":"Reader's verdict is CONDITIONAL; my independent read reaches the same conclusion. The reproducibility contribution is real: implementations are integrated into Anserini/Pyserini/RankLLM, runs and indexes are released, and the Table 1 match to the original BRIGHT BM25 column is a useful confirmation. The weak point is not the implementation but the inference drawn from it. The BM25Q-vs-BoW comparison has no uncertainty quantification, mixed per-task outcomes, and a post hoc length stratification, so the headline recommendation is not yet supported. The abstract/body mismatch on five benchmarks is a separate but compounding concern: the manuscript asserts a study that Section 5 explicitly says was not done. Neither issue requires rejecting the paper, but both require revision before the general claims are accepted. I therefore leave the CONDITIONAL verdict unchanged, with the concrete significance and threshold test as the condition for upgrading.","tokens_in":14420,"tokens_out":7498,"duration_ms":67409,"concrete_test":"Using the released Anserini/Pyserini runs, compute per-query nDCG@10 for BoW and BM25Q with the paper's k1=0.9, b=0.4. Run a paired bootstrap stratified by task (10,000 resamples) and a randomization test on the per-query differences; report 95% CIs for the overall and 16-256-token subsets. Also re-fit the Figure 1 bucket analysis with pre-registered equal-width log buckets and a held-out split (e.g., train threshold on 6 tasks, test on 6). If the CI includes zero or p>0.05, the claim should be downgraded to an observed trend rather than a recommendation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that query-side BM25 (BM25Q) is the strongest lexical baseline for BRIGHT's long queries. The evidence is Table 1 (0.148 vs 0.137 nDCG@10) and Figure 1's length-stratified deltas. Three things undercut this. (1) No significance test is reported at query or task level; across 12 tasks the score is 7 wins, 4 losses, 1 tie, which is not significant by a sign test (one-sided p≈0.27, two-sided p≈0.55), so the aggregate advantage may be sampling noise. (2) The length buckets in Figure 1 and the 16-256 token recommendation are chosen post hoc on the same data, and Section 4.1 explicitly says BRIGHT has no dedicated dev/validation split, so there is no check that the threshold generalizes. (3) The abstract claims a five-benchmark generalization study and a fusion conclusion, but Section 5 states query-side BM25 was evaluated only on BRIGHT; the promised experiments are not in the manuscript. The released runs and code make these points testable, but as written the 'consistently outperforms' language exceeds the support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents reproducible first-stage retrieval and reranking baselines for the BRIGHT benchmark, integrated into Anserini, Pyserini, and RankLLM. It identifies that the BRIGHT repository's BM25 implementation weights query terms with BM25 ('query-side BM25') rather than using the usual bag-of-words query vectors, replicates the BRIGHT BM25 scores using the 'accurate' variant, evaluates dense (BGE-large-en-v1.5), sparse (SPLADE-v3), and RRF/fusion combinations, adds LLM listwise reranking results with Qwen3-8b and gpt-oss-20b, and audits the corpus for duplicate, short, and empty documents plus missing gold IDs. The abstract additionally claims a five-benchmark generalization study of BM25Q, but the body's Section 5 states the method was evaluated only on BRIGHT.","tokens_in":14619,"tokens_out":3924,"duration_ms":32637,"significance":"The toolkit integrations and released runs/indexes are valuable and reproducible; the discovery that BRIGHT's BM25 scores come from query-side weighting is a concrete, useful observation for the community, and the exact replication (Table 1 columns d vs e) is a strength. However, the central generalization claims—'consistently outperforms' and the 16–256 token recommendation—are not supported by significance testing or a validation split, and the five-benchmark study promised in the abstract is absent from the body. The paper is a useful resource but needs revision before its headline claims can be accepted.","major_comments":[{"comment":"The abstract states that the paper 'study[ies] the generalizability of BM25Q across five additional benchmarks,' but Section 5 (Limitations and Future Work) explicitly says 'we evaluated query-side BM25 only on BRIGHT.' No section of the manuscript reports experiments on additional benchmarks. This is a direct factual inconsistency: either the experiments must be added or the abstract must be revised to state the scope.","section":"Abstract / §5"},{"comment":"The claim that BM25Q 'consistently outperforms' bag-of-words is not statistically established. The aggregate nDCG@10 difference is 0.011 (Table 1, 0.148 vs 0.137), but the per-task comparison is 7 wins, 4 losses, 1 tie, which is not significant by a sign test (one-sided p≈0.27). No paired significance test at the query level is reported, and Figure 1's length-stratified deltas are inspected post hoc on the same data for which no development split exists (stated in §4.1). As written, the recommendation to prefer BM25Q for 16–256 token queries is an observed trend, not a validated rule.","section":"§4.1, Table 1, Figure 1"},{"comment":"The 16–256 token recommendation is derived from the same BRIGHT data used to produce the comparison, with no held-out validation or significance test for the bucket boundaries. The paper should either provide a proper validation methodology (e.g., cross-task splits, bootstrap confidence intervals) or soften the recommendation to a hypothesis for future work.","section":"§4.1, 'recommend applying BM25 weighting to queries of roughly 16–256 tokens'"},{"comment":"The text claims that in the RRF section 'all scores in the RRF section exceed their counterparts in the individual section,' but this is not true for every cell: for example, Biology BM25 alone is 0.197 while BM25+BGE fusion (column e) is 0.175. The claim of consistent fusion gains should be qualified to overall averages or to a stated subset of tasks, and the actual per-task behavior should be described accurately.","section":"§4.2, Table 3"}],"minor_comments":[{"comment":"There is a typo in the last paragraph: 'Anserini baslines' should be 'Anserini baselines.'","section":"§3.1"},{"comment":"The figure label 'BM25 - BoW' is confusing because the paper elsewhere distinguishes 'BM25Q' (query-side BM25) from 'BM25' (which in Table 1 refers to the generic method); the axis label should read 'BM25Q - BoW' or the caption should explicitly define the notation.","section":"Figure 1"},{"comment":"The header 'BRIGHT' for column (e) is ambiguous; it is the BRIGHT-reported score, so the caption or table header should say 'BRIGHT reported' to distinguish it from the authors' own 'BM25Q Acc.' column.","section":"Table 1"},{"comment":"The sentence 'Combining BM25 with either S-v3 or BGE proves more effective than fusing S-v3 and BGE' is based on overall averages; readers should be reminded that per-task exceptions exist (e.g., Psychology in Table 3 where S-v3+BGE exceeds BM25+BGE).","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is primarily a reproducibility/engineering contribution, and the authors' maintenance of the involved toolkits makes self-citation expected and appropriate. The editorial decision should hinge on whether the abstract's overclaim about five-benchmark generalization and the unsubstantiated 'consistently outperforms' language can be corrected: either by adding the missing experiments or by reframing the paper as a single-benchmark reproducibility study with clearly hedged claims. The released code and runs are a genuine asset, but the current text overstates the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a useful reproducibility paper with a real finding, but its abstract promises more than the body delivers. The core contribution—that BRIGHT's BM25 baseline uses query-side BM25 rather than bag-of-words—is convincingly established, and the authors have shipped runnable integrations in Anserini, Pyserini, and RankLLM, plus prebuilt indexes and adjusted qrels. That is concrete, reproducible work, and the replication evidence (Table 1, columns d vs. e) is strong. The corpus audit also earns its keep: duplicate and near-empty chunks are quantified, and the adjusted-qrels effect on nDCG is small but honestly reported.\n\nThe soft spots are real and line up with the stress-test. Most seriously, the abstract claims a five-benchmark generalizability study and a fusion conclusion, but Section 5 explicitly says query-side BM25 was evaluated only on BRIGHT. That is a direct internal contradiction, not a minor wording issue. Second, the \"consistently outperforms\" language is not supported by the numbers: the aggregate advantage is 0.011 nDCG@10, the per-task split is 7 wins / 4 losses / 1 tie, and no significance test is reported. The stress-test's sign-test calculation is correct—seven of twelve is not significant. The length-bucket recommendation (16–256 tokens) is also post hoc, and the authors themselves note BRIGHT has no dedicated dev split, so the threshold is an observed trend, not a validated rule.\n\nThat said, I would not sink the paper over these issues. The descriptive finding is solid: BRIGHT's BM25 is query-side, and on these long queries it often helps. The overreach is in the abstract and the prescriptive recommendation, not in the experiments. The body's own limitations section is candid; the authors seem aware that generalization work is still ahead. Fix the abstract, add a significance caveat (or at least stop saying \"consistently\"), and this is a worthwhile baseline paper for anyone working with BRIGHT or with long-query lexical retrieval. It deserves a serious referee; I'd send it out with a request for revision rather than desk-reject it.","headline":"Useful reproducibility paper that correctly identifies BRIGHT's BM25 as query-side BM25, but the abstract overclaims a five-benchmark study the body explicitly says was not done.","tokens_in":15211,"tokens_out":2022,"would_cite":true,"duration_ms":19134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that the BRIGHT benchmark's reported BM25 baseline was actually produced by query-side BM25—applying BM25 weighting to query terms—and that on long reasoning queries this detail beats standard bag-of-words BM25, with…","keywords":["query-side BM25","BM25","BRIGHT benchmark","long queries","retrieval-augmented generation","reciprocal rank fusion","LLM reranking","reproducible baselines"],"falsifier":"A paired significance test over per-query scores on the twelve BRIGHT tasks comparing query-side BM25 with bag-of-words, or an independent long-query benchmark with queries of 16-256 tokens, would settle it; if the advantage fails to exceed noise or to replicate, the central recommendation collapses.","tokens_in":14161,"feed_emoji":"🔍","tokens_out":11020,"duration_ms":86124,"temperature":0.7,"pith_summary":"The paper argues that in the era of long, reasoning-intensive queries, the standard way BM25 builds query vectors—counting query terms as a bag of words—should be reconsidered. Reproducing the lexical baseline of the BRIGHT benchmark, the authors find that the originally reported BM25 scores come from an implementation that weights each query term with BM25 itself (query-side BM25), and that on BRIGHT's long multi-sentence queries this variant outperforms bag-of-words: 0.148 vs 0.137 nDCG@10 overall. They turn the finding into reproducible baselines in the standard open-source retrieval and reranking toolkits they extend, and show that fusing BM25 with learned sparse and dense retrievers, then applying LLM listwise reranking, gives further large gains (overall nDCG@10 up to 0.274). A sympathetic reader would care because search toolkits default to bag-of-words query vectors, and this paper shows a small implementation choice that can matter as RAG prompts and LLM-expanded queries grow longer. The paper also audits the BRIGHT corpus, finding duplicate and degenerate documents and missing gold IDs that slightly distort evaluation.","feed_headline":"Query-side BM25 beats bag-of-words on long retrieval prompts","feed_subtitle":"The under-documented query-weighting trick behind BRIGHT's BM25 baseline lifts nDCG@10 to 0.148.","key_machinery":"The load-bearing mechanism is the query-side BM25 query vector, in which each query term is weighted by the BM25 formula applied to the query itself—so terms that are frequent in the query and the corpus receive saturating, rarity-adjusted weights instead of raw counts. This replaces the bag-of-words query vector inside the same inner-product similarity scoring, isolating the query-representation choice from everything else. The mechanism is what allows long, multi-sentence queries with repeated token distributions to be reweighted, and the paper's experiments compare it directly against bag-of-words while controlling for document-length normalization quantization.","core_discovery":"The central discovery is that the BRIGHT benchmark's reported BM25 baseline was produced by applying the BM25 scoring function to each query token to build the query vector, rather than by the bag-of-words weighting that standard toolkit defaults use; both approaches score documents by the inner product of the query and document vectors. On BRIGHT's long, reasoning-intensive queries, query-side BM25 is more effective overall (nDCG@10 0.148 vs 0.137), winning 7 of 12 tasks, with per-query analysis showing gains grow in frequency and size as query length increases until BM25's length-normalization term dominates. The authors make query-side BM25 available as a standard option in the toolkits they extend, recommend it as the strongest lexical baseline for reasoning-oriented retrieval, and find that its advantage does not generalize to five other benchmarks, while fusing BM25 with learned retrievers (or with standard BM25) is the most consistently helpful choice.","pith_inferences":["The 16-256 token recommendation is post hoc and untested for significance; a natural extension is a paired per-query significance test across the twelve tasks, which the authors do not report.","Because BRIGHT has no dedicated development/validation split, the length-bucket pattern in Figure 1 is an observed trend; holding out a random query subset and checking whether BM25Q's advantage persists would separate the length effect from task-specific quirks.","If query-side weighting helps long queries, it may compound with LLM query expansion, whose outputs are long and token-repetitive; the paper does not test BM25Q on the LLM-expanded queries BRIGHT provides, which would be a direct next experiment.","The corpus audit implies a cheap benchmark repair—deduplicating documents and propagating gold labels across duplicate IDs—that the adjusted qrels approximate; the paper predicts stronger retrievers will be even more penalized by missing gold IDs, which is testable with the released qrels."],"forward_implications":["On BRIGHT, query-side BM25 becomes the lexical baseline to beat: 0.148 vs 0.137 nDCG@10 over bag-of-words, with wins on 7 of 12 tasks.","For queries of roughly 16-256 tokens, the paper recommends applying BM25 weighting to the query; below 16 tokens the two query representations are nearly identical.","Fusing BM25 with learned sparse or dense retrievers via reciprocal rank fusion beats every individual retriever on BRIGHT (overall 0.172-0.173 vs 0.137-0.156 nDCG@10), and the BM25-plus-dense combination is best.","LLM listwise reranking of the top 100 results raises overall nDCG@10 from about 0.16-0.17 at first stage to 0.241-0.274, with the larger open-weight reranker consistently better.","Duplicate documents, near-empty chunks, and missing gold IDs in BRIGHT distort evaluation by up to 0.010 nDCG@10 after reranking; the authors release adjusted relevance labels."],"supporting_citations":[{"why":"Defines the BRIGHT benchmark, its queries, qrels, and the original BM25 implementation whose query-side weighting this paper identifies.","marker":"[32]"},{"why":"Provides the standard BM25 formulation and the probabilistic relevance framework that the paper contrasts with query-side weighting.","marker":"[28]"},{"why":"Records the original Okapi BM25 treatment of query term frequency, the practice the paper argues should be revisited for long queries.","marker":"[27]"},{"why":"Documents later Okapi experiments that dropped query-term saturation, which the paper cites as the origin of bag-of-words query vectors.","marker":"[26]"},{"why":"Supplies the reproducibility study of BM25 scoring variants that underlies the paper's conclusion that length-normalization quantization has negligible impact.","marker":"[9]"},{"why":"Describes the Lucene-based toolkit whose default bag-of-words query generation is the baseline being compared.","marker":"[37]"},{"why":"Supplies the Python toolkit and analyzer used for tokenization, indexing, and retrieval in the paper's experiments.","marker":"[17]"},{"why":"Provides the reranking package into which BRIGHT is integrated for the LLM reranking experiments.","marker":"[31]"},{"why":"Supplies the learned sparse retriever used as a first-stage baseline in fusion and comparison.","marker":"[11]"},{"why":"Supplies the dense retriever used as a first-stage baseline and in the best fused configuration.","marker":"[36]"}],"fun_headline_variants":["Query-side BM25 beats standard on long reasoning queries","BM25Q: strongest lexical baseline for reasoning-oriented retrieval","BRIGHT's BM25 baseline uses query-side weighting, beats bag-of-words","Query-side BM25 lifts nDCG on BRIGHT, but gains don't generalize","Under-documented query weighting makes BM25 stronger on BRIGHT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the observed advantage of query-side BM25 on mid-length queries is a real effect rather than noise: it rests on a small 0.011-point average gain, mixed per-task results, and a length-bucket trend from a benchmark that has no development/validation split to confirm it.","fun_headline_variants_meta":{"raw":{"variants":["Query-side BM25 beats standard on long reasoning queries","BM25Q: strongest lexical baseline for reasoning-oriented retrieval","BRIGHT's BM25 baseline uses query-side weighting, beats bag-of-words","Query-side BM25 lifts nDCG on BRIGHT, but gains don't generalize","Under-documented query weighting makes BM25 stronger on BRIGHT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000765,"raw_usage":{"total_tokens":3397,"prompt_tokens":951,"completion_tokens":2446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":2354}},"tokens_in":567,"tokens_out":2446,"duration_ms":14166,"temperature":1.0,"reasoning_tokens":2354,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:36:41.895313+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A paired significance test over per-query scores on the twelve BRIGHT tasks comparing query-side BM25 with bag-of-words, or an independent long-query benchmark with queries of 16-256 tokens, would settle it; if the advantage fails to exceed noise or to replicate, the central recommendation collapses.","supporting_citations":[{"cited_title":"The Probabilistic Relevance Framework: BM25 and Beyond","cited_arxiv_id":null,"evidence_quote":"Provides the standard BM25 formulation and the probabilistic relevance framework that the paper contrasts with query-side weighting."},{"cited_title":"Walker, S","cited_arxiv_id":null,"evidence_quote":"Records the original Okapi BM25 treatment of query term frequency, the practice the paper argues should be revisited for long queries."},{"cited_title":"Walker, M","cited_arxiv_id":null,"evidence_quote":"Documents later Okapi experiments that dropped query-term saturation, which the paper cites as the origin of bag-of-words query vectors."},{"cited_title":"de Vries, Leonid Boytsov, and Jimmy Lin","cited_arxiv_id":null,"evidence_quote":"Supplies the reproducibility study of BM25 scoring variants that underlies the paper's conclusion that length-normalization quantization has negligible impact."},{"cited_title":"Anserini: Reproducible Ranking Baselines Using Lucene","cited_arxiv_id":null,"evidence_quote":"Describes the Lucene-based toolkit whose default bag-of-words query generation is the baseline being compared."},{"cited_title":"Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations","cited_arxiv_id":null,"evidence_quote":"Supplies the Python toolkit and analyzer used for tokenization, indexing, and retrieval in the paper's experiments."},{"cited_title":"RankLLM: A Python Package for Reranking with LLMs","cited_arxiv_id":null,"evidence_quote":"Provides the reranking package into which BRIGHT is integrated for the LLM reranking experiments."}],"review_version":2}