{"id":"673425a8-5ead-412f-aacf-a68cfe83fb24","arxiv_id":"2506.14086","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"InsertRank shows that injecting BM25 scores into listwise LLM reranking prompts improves retrieval effectiveness on BRIGHT and R2MED across multiple LLM families.","lead":"The paper presents InsertRank, a reranking method that appends each document's BM25 score to the prompt of a listwise large language model reranker. On two reasoning-focused retrieval benchmarks, the added scores improve average ranking quality across GPT, Gemini, and Deepseek models at negligible extra cost.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Most paired gains are 0.003–0.012 NDCG@10, with no variance, CI, or significance test; single-run API evaluation cannot support the central improvement claim as stated.","rationale":"Reading in good faith: the method is simple and the experiments are plausibly executed; no fitted parameters, no circularity, no obvious leakage. The central assertion is not that BM25 scores always help every domain, but that they improve average retrieval effectiveness across families of LLMs on two benchmarks. That assertion requires the measured differences to be distinguishable from run-to-run noise. The paper gives no evidence of this, and the smallest differences (0.003 NDCG@10) are the most common. The reader identified exactly this as the weakest assumption; I agree. I would not reject the paper: the 8/8 positive sign pattern and the large Gemini 2.5 gain on R2MED (0.012) are suggestive, and the method is cheap and easy to test. But the reported evidence is insufficient at the reported granularity, so CONDITIONAL is the right verdict. The appendix's mismatched gold answer (Section 7.3.4) is a proofreading artifact and does not bear on the central results. No code is released, which compounds the reproducibility problem, but that is secondary to the missing variance information.","tokens_in":10551,"tokens_out":6815,"duration_ms":67521,"concrete_test":"Rerun each of the 8 main conditions (vanilla vs InsertRank for Gemini 2.0 flash, Gemini 2.5 flash, GPT-4o, Deepseek-r1 on BRIGHT and R2MED) five times with identical prompts and a fixed decoding configuration, then compute a query-level paired bootstrap 95% confidence interval for the mean NDCG@10 difference. If any of the four 0.003-level differences has a CI that includes zero, the 'consistent improvement' claim is not supported; if all 8 CIs exclude zero, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in Section 4 ('consistent improvements... with no extra cost'), is an empirical one about average NDCG@10. The evidence is Tables 1 and 2, which give one number per condition. Eight paired differences are reported; four are exactly 0.003 (GPT-4o and Deepseek-r1 on both BRIGHT and R2MED), one is 0.004 (Gemini 2.0 flash on R2MED), and the largest is 0.012 (Gemini 2.5 flash on R2MED). No standard deviation, confidence interval, significance test, or repeated-sampling information appears anywhere in the paper, including the ablation tables (Tables 3–8), and no decoding configuration (temperature, top-p, number of runs) is given. For BRIGHT's ~1300 queries, a plausible per-query NDCG@10 standard deviation of ~0.3 implies a standard error of ~0.008 on the mean; the 0.003 differences are below that. For R2MED's 876 queries the SE is similar. The sign test across 8 aggregate averages is suggestive (8/8 positive), but it ignores within-condition stochasticity and is contradicted at domain level by Table 1, where several domains decrease for Gemini 2.0 flash and Deepseek-r1. Thus the load-bearing assumption—single-run API evaluations are stable enough to resolve 0.003-level differences—is not supported. This is an epistemic gap in the central empirical claim, not an internal contradiction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces InsertRank, a zero-shot listwise reranking method that injects BM25 scores from a first-stage retrieval into the prompt of an LLM reranker. The method is evaluated on two reasoning-centric retrieval benchmarks, BRIGHT and R2MED, using Gemini 2.0 Flash, Gemini 2.5 Flash, GPT-4o, and Deepseek-R1. The main results (Tables 1 and 2) report average NDCG@10 improvements of 0.003 to 0.012 over vanilla listwise reranking, with the best setting (Deepseek-R1 + InsertRank) reaching 37.5 on BRIGHT and 51.1 on R2MED. The paper also presents ablations on BM25 score normalization and on document-order shuffling. The authors claim this is the first demonstration that injecting retrieval scores into a zero-shot listwise LLM reranker improves reasoning-centric retrieval.","tokens_in":10842,"tokens_out":12521,"duration_ms":110721,"significance":"If the average gains are real, InsertRank is a simple and practically valuable finding: it requires no fine-tuning, adds negligible token overhead, and appears to help across four LLM families and two benchmarks. The method has no fitted parameters, so overfitting to the benchmarks is not a concern. The ablation on document-order shuffling is a useful control, and the normalization experiments give some evidence that the method is not brittle to score scaling. However, the current statistical evidence is too weak to support the strength of the claims, and several numeric inconsistencies in the ablation discussion reduce confidence in the reported details.","major_comments":[{"comment":"The central claim of consistent improvement rests on single runs per condition. The paper reports no variance, confidence intervals, significance tests, or decoding configuration (temperature, top-p, number of runs). The reported differences are between 0.003 and 0.012 NDCG@10; given the benchmark sizes (about 1,300 queries for BRIGHT and 876 for R2MED), a plausible per-query NDCG@10 standard deviation of about 0.3 yields a standard error near 0.008, so the smallest gains are below the noise floor. Please provide repeated-run statistics (for example, paired bootstrap or permutation tests over queries) and state the sampling parameters. Additionally, many individual domains regress (for example, Gemini 2.0 Flash on Stack, Sust, and Pony in BRIGHT; Deepseek-R1 on Psych, Sust, and Pony in BRIGHT; and several R2MED tasks for most models), so the word 'consistent' should be explicitly limited to the average level unless per-domain significance is demonstrated.","section":"Section 4, Tables 1-2"},{"comment":"The ablation text contains numerical claims that do not match the tables. In Section 5.2, the text says that for BRIGHT, the shuffled setting with BM25 injection gives a 9.4% increase relative to the vanilla setting and a 1.1-point absolute decrease compared with the ordered setting, but Table 4 reports Shuffled w/ BM25 = .322, Vanilla = .334, and ordered w/ BM25 = .345. These values imply a -3.6% relative change and a .023 absolute decrease. In Section 5.1, the reported '0.58% decrease' for the BRIGHT 0-1 normalization is also inconsistent with Table 3, where the 0-1 value (.340) is about 1.45% below the raw-BM25 value (.345). The authors should correct these numbers and state explicitly whether percentages are relative to the vanilla condition, the raw-BM25 condition, or the 0-1 condition.","section":"Sections 5.1-5.2, Tables 3-4"},{"comment":"The claim that InsertRank 'surpasses previous methods' compares against published numbers that are not generated under the same protocol. The Deepseek-R1 + InsertRank result uses the GPT-4o CoT query formulation for BRIGHT and the HyDE formulation for R2MED, while the cited baselines (ReasonIR, JudgeRank, Rank-K, Rank1-32B, Rank-R1) use their own retrievers and query processing. The differences involved (for example, .375 vs .369 for ReasonIR on BRIGHT) are small and within the noise range identified above. Please either run the baselines under the same first-stage and query-formulation conditions or soften the 'surpassing' claim to a leaderboard-style comparison with the appropriate caveats.","section":"Section 4, Tables 1-2"}],"minor_comments":[{"comment":"The abstract says BRIGHT spans '12 diverse domains' while Section 2.1 says '11 domains'; Table 1 lists 12 domain columns. Please reconcile the count.","section":"Abstract and Section 2.1"},{"comment":"Table 4's caption 'Effect of shuffling on R2MED' is misleading because the table reports both BRIGHT and R2MED averages; Table 7's caption should also indicate that it reports BRIGHT results.","section":"Table 4 and Table 7"},{"comment":"The notation for BM25 scores is inconsistent: equation (4) uses b_i, the prompt template in Section 3 uses s_1 and then b_n, and the appendix prompt uses 'BM25 score: {score_1}'. Please use one notation throughout.","section":"Section 3 and Appendix 7.4"},{"comment":"The prose says 'as evidenced in 3' and 'as evidenced in 4' rather than 'Table 3' and 'Table 4'.","section":"Sections 5.1-5.2"},{"comment":"The 'Gold Answer' in Appendix 7.3.4 is about drug overdose, which does not match the bedtools/Ubuntu query used in the reasoning traces in Sections 7.3.1-7.3.3; please replace it with the correct gold answer.","section":"Appendix 7.3.4"},{"comment":"The paper does not state whether the reported NDCG@10 values are computed with the official benchmark scripts; please state the metric and the evaluation setup explicitly.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal and the proposed method is simple enough to be a useful empirical contribution. The main obstacle is statistical: the authors should run repeated trials and report paired significance tests, and they should correct the numerical inconsistencies in the ablation section. I would not recommend rejection because the direction of the effect is plausible and the method is easily testable, but the claims need to be scaled to the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does one simple thing—appends BM25 scores to each document in a zero-shot listwise LLM reranking prompt—and shows positive aggregate NDCG@10 gains on BRIGHT and R2MED across four LLM families. The core idea is borrowed from cross-encoder work [2], and the authors say so. What's new is the listwise zero-shot setting, and the ablations on normalization and document order are genuinely informative.\n\nWhere it earns credit: the method is cheap, clearly described, and the comparisons to prior work look fair. The paper doesn't oversell novelty. The shuffle ablation—BM25 injection helps when documents are ordered by score but hurts when that order is destroyed—is a nice sanity check that the model is using the signal, not just copying the input order.\n\nThe main soft spot is statistical. Most reported gains are 0.003–0.012 NDCG@10, with one exception (Gemini 2.5 flash on BRIGHT, +0.048). There are no error bars, repeated runs, significance tests, or decoding configuration. With ~1300 queries in BRIGHT, the standard error on the mean is plausibly around 0.008, so differences below that are indistinguishable from noise. The 8/8 positive sign across model-benchmark pairs is suggestive, but several domains regress in Table 1 and the paper doesn't address that. This is the load-bearing weakness for the central 'consistent improvements' claim. It's fixable by running each condition multiple times and reporting paired significance.\n\nThere are also minor production issues: the appendix gold answer is mismatched (a toxicology text for a bedtools query), no code is released, and the writing has a few typos. None of that affects the core results, but it does suggest hasty preparation.\n\nWho this is for: anyone doing zero-shot listwise reranking who wants a quick, cheap trick to try. I would not build a strong conclusion on the current numbers, but the idea is worth knowing. Send it to peer review—the method is legitimate and the ablations are useful—but push for variance estimates before publication.","headline":"A simple, honestly framed BM25-prompting trick for listwise LLM reranking, with plausible but statistically under-supported gains.","tokens_in":11397,"tokens_out":3015,"would_cite":false,"duration_ms":30098,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Inserting BM25 scores into a listwise prompt makes zero-shot LLM rerankers more effective on reasoning queries.","keywords":["listwise reranking","BM25 score injection","zero-shot LLM reranking","reasoning-centric retrieval","retrieval effectiveness","prompt engineering","BRIGHT benchmark","R2MED benchmark"],"falsifier":"Re-run the vanilla and InsertRank conditions on a sample of BRIGHT and R2MED queries several times per model, varying sampling temperature, and compute confidence intervals for the NDCG@10 difference; if zero frequently falls inside the interval, the claimed improvement is not distinguishable from noise. A second decisive check is replacing the true BM25 scores with permuted or random scores: if the gain persists, the model is not actually using the lexical signal.","tokens_in":10357,"feed_emoji":"🔍","tokens_out":6850,"duration_ms":59626,"temperature":0.7,"pith_summary":"InsertRank is a test-time change to listwise reranking: instead of feeding an LLM only the query and candidate documents, it also includes the BM25 score each document received from the first-stage retriever. The paper's central claim is that this lexical signal improves retrieval effectiveness on reasoning-centric queries, where simple keyword or semantic matching is not enough. The authors report consistent NDCG@10 gains across GPT, Gemini, and Deepseek model families on two benchmarks, BRIGHT and R2MED, without any fine-tuning and with negligible extra token cost. The best setting, Deepseek-R1 with InsertRank, reaches 37.5 on BRIGHT and 51.1 on R2MED, surpassing previously published results. If right, the result is a low-cost plug-in improvement for any pipeline that already uses BM25 to retrieve candidates and an LLM to rerank them.","feed_headline":"BM25 scores in the prompt lift LLM reranking on reasoning queries","feed_subtitle":"Zero-shot InsertRank beats vanilla listwise reranking across GPT, Gemini, and Deepseek on BRIGHT and R2MED.","key_machinery":"The central object is the score-augmented prompt: each candidate document is listed with its BM25 score, and the documents are ordered from highest to lowest score before the LLM ranks them. This tuple format is what lets the reranker combine lexical evidence with its own reasoning; a single line of instruction tells the model the scores come from a lexical retriever. The ablations show the mechanism is not just \"add a number\": shuffling the document order removes most of the benefit on R2MED, and rescaling to 0-1 slightly hurts while rescaling to 0-100 preserves the gain, suggesting the LLM reads both relative order and approximate magnitude.","core_discovery":"The discovery is that a reasoning-capable LLM reranker can use the retriever's own BM25 scores as a grounding signal when documents are presented in descending score order. On the authors' terms, injecting the scores anchors the model's step-by-step reasoning to the lexical relevance of the candidate set, countering overthinking and concept drift that can derail long reasoning chains. The paper demonstrates this by adding the BM25 score next to each document in a zero-shot listwise prompt and observing consistent average NDCG@10 improvements over the vanilla listwise prompt across the four model families tested on both BRIGHT and R2MED. Gains in the main tables range from roughly 0.003 to 0.012 in absolute average NDCG@10, and the strongest configuration surpasses the best published baselines on both benchmarks.","pith_inferences":["A natural extension is to test whether other cheap first-stage signals—dense similarity scores, hybrid fusion scores, or even rank positions—produce the same grounding effect; if they do, the mechanism is general score injection rather than BM25 specifically.","The ablation pattern hints that the LLM uses relative order more than absolute magnitude; a testable prediction is that replacing BM25 values with any strictly decreasing sequence of numbers preserves most of the gain.","The R2MED shuffle result suggests the score signal and the order signal interact; an experiment that keeps order but drops scores, or keeps scores but randomizes order, would isolate which component carries the benefit.","Because the paper only evaluates reasoning-heavy benchmarks, the boundary of the effect is unknown; testing on standard ad-hoc retrieval collections could show whether the gain is specific to reasoning queries or general."],"forward_implications":["Any existing BM25-then-LLM reranking pipeline can adopt InsertRank by editing only the prompt, adding no training data and almost no token cost.","The consistent gains across GPT, Gemini, and Deepseek suggest the effect transfers across model families, not just one architecture.","Because the document order itself matters, a pipeline should keep the first-stage BM25 ordering rather than re-order randomly before reranking, even if no scores are shown.","Reasoning-centric retrieval, where overthinking and concept drift are common failure modes, is the setting where score injection helps; applying it to simple keyword queries may not yield the same benefit."],"supporting_citations":[{"why":"Shows that writing the BM25 score as text improves BERT-based cross-encoders, the direct precedent InsertRank extends to zero-shot listwise LLM reranking.","marker":"[2]"},{"why":"Supplies the BRIGHT benchmark and the GPT-4o chain-of-thought query reformulation used for the BRIGHT first-stage retrieval.","marker":"[21]"},{"why":"Supplies the R2MED benchmark used for the medical reasoning retrieval evaluation.","marker":"[10]"},{"why":"Supplies the HyDE query reformulation that gives the best BM25 first-stage retrieval results on R2MED.","marker":"[7]"},{"why":"Provides the Rank1 baseline and the query-augmentation inspiration for the BRIGHT experimental setting.","marker":"[23]"},{"why":"Provides the Rank-R1 GRPO finetuned listwise baseline that InsertRank compares against.","marker":"[25]"},{"why":"Provides the ReasonIR finetuned pointwise reranking baseline reported in the comparison tables.","marker":"[20]"},{"why":"Provides the Rank-K sliding-window listwise reranking baseline reported in the comparison tables.","marker":"[24]"},{"why":"Motivates the document-order shuffle ablation, which supports the claim that descending BM25 order is part of the mechanism.","marker":"[11]"}],"fun_headline_variants":["InsertRank: BM25 scores anchor LLM reasoning for reranking","BM25 scores in prompt help LLMs rerank reasoning queries","InsertRank: Lexical scores as reasoning anchors for rerankers","Use BM25 scores to ground LLM listwise reranking","Grounding LLM reranking with BM25 scores improves reasoning queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported NDCG@10 differences—about 0.003 to 0.012 on average—are larger than the run-to-run noise of the stochastic LLM APIs, since each condition is evaluated once without variance or significance testing.","fun_headline_variants_meta":{"raw":{"variants":["InsertRank: BM25 scores anchor LLM reasoning for reranking","BM25 scores in prompt help LLMs rerank reasoning queries","InsertRank: Lexical scores as reasoning anchors for rerankers","Use BM25 scores to ground LLM listwise reranking","Grounding LLM reranking with BM25 scores improves reasoning queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3282,"prompt_tokens":971,"completion_tokens":2311,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2222}},"tokens_in":587,"tokens_out":2311,"duration_ms":15641,"temperature":1.0,"reasoning_tokens":2222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:53:59.078390+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the vanilla and InsertRank conditions on a sample of BRIGHT and R2MED queries several times per model, varying sampling temperature, and compute confidence intervals for the NDCG@10 difference; if zero frequently falls inside the interval, the claimed improvement is not distinguishable from noise. A second decisive check is replacing the true BM25 scores with permuted or random scores: if the gain persists, the model is not actually using the lexical signal.","supporting_citations":[],"review_version":1}