{"id":"6e4e5b3e-8ff3-4c1d-b30e-3d32a3e59a8a","arxiv_id":"2608.00183","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"On a windowed financial-filing corpus, untuned reciprocal rank fusion is the best tested retriever, and lightweight per-query weighting routers fail to beat a fixed equal-weight blend.","lead":"This paper tests hybrid search over financial filings and finds that simple rank fusion of BM25 and a dense model beats either alone, while per-query weighting routers do not reliably improve on the fixed blend. It also exposes an evaluation confound: if retrieval units are longer than the dense encoder's window, the dense model never sees much of the labeled evidence, so fair comparisons require window-sized units.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Automatic reference-to-window mapping is the load-bearing risk: only 30.3% literal containment, no verified precision, and 621 unresolved queries could systematically bias both the RRF advantage and the router nulls.","rationale":"The reader's conditionality is well placed. The paper is methodologically careful: company-grouped cross-validation, cluster-robust inference, a prespecified decision rule, and a literal-containment robustness check are all genuine strengths. The main soft spot is the automatic label mapping, which is the foundation for every reported number. The paper itself discloses the key limitations: only 30.3% literal containment, no verified precision, 621 unresolved queries, and failures concentrated in table-heavy financial statements. The literal-containment subset is a reasonable mitigation and shows that the fusion-vs-component gains persist on the cleanest labels, but it is too small to give the RRF-vs-blend comparison statistical power (p=0.11) and says nothing about the 621 excluded queries. Consequently, the strongest claim is conditional: if the label mapping is systematically biased, both the positive RRF result and the negative router results could shift. The proposed sensitivity analysis would directly test whether the unresolved-queries exclusion and label noise can move the conclusions. Because the concern is already identified by the reader and the paper's own robustness check partially mitigates it, the conditional verdict should remain unchanged.","tokens_in":10067,"tokens_out":7148,"duration_ms":84953,"concrete_test":"Perform a bounding sensitivity analysis for the 621 unresolved queries: assign them, under two extreme scenarios, (i) zero hits for all methods and (ii) the literal-containment-subset mean Hit@10 for all methods, then recompute Table I and the RRF-vs-blend comparison. If the RRF advantage (+0.0090) remains positive and all router comparisons remain non-significant in both scenarios, the exclusion is not load-bearing; if the advantage flips sign or the ridge router crosses the 0.0167 threshold in either scenario, the conclusions are not robust to the unresolved 10.9% of queries. To cover mis-mapped resolved references, repeat the same computation on the 1,518-query literal-containment subset already reported: if the RRF advantage is not significant and router differences stay null there, the central negative result survives the cleanest labels, and only the RRF-best claim remains explorat","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—fusion is a strong baseline, RRF is best, and per-query routers do not beat the equal-weight blend—rests on automatically mapped reference-to-window relevance labels (Section III-C). Only 30.3% of references pass literal containment and no precision figure is verified; 621/5,703 queries are excluded entirely. The paper's own qualitative inspection says failures concentrate in financial-statement tables (Section III-D), the category where numeric evidence matters most and where BM25's exact-number matching is expected to help. If missing or mis-mapped references are systematically distributed, the measured +0.0090 RRF advantage and the null router differences could be artifacts of label noise rather than retrieval quality. The literal-containment subset check (n=1,518) is reassuring for the fusion-vs-components comparison, but it is underpowered for the RRF-vs-blend comparison (p=0.11) and cannot speak to the 621 excluded queries. Thus the practical guidance 'untuned RRF is the defensible default' is conditionally supported at best.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies sparse-dense hybrid retrieval over FinDER, a financial 10-K retrieval benchmark, and makes two main claims. First, it identifies an 'evidence-unit fairness' problem: the parsed retrieval units are much larger than the 512-token input limit of the dense encoder, so 46.2% of labeled evidence begins at or beyond word 512, biasing comparisons against dense retrieval. The authors segment the corpus into 160-word overlapping windows and automatically map references to windows. On this corrected corpus, BM25 and a compact dense encoder each achieve reference-level Hit@10 near 0.14, while fusion roughly doubles either component; reciprocal rank fusion (RRF) scores highest at 0.1892, ahead of the equal-weight blend at 0.1802. Second, the paper asks whether per-query fusion weights help: a grid oracle shows 21.8% headroom, but three lightweight routers (a score-confidence heuristic, a random forest, and a ridge regressor over embeddings) fail to beat the equal-weight blend reliably under company-grouped cross-validation with cluster-robust inference. The paper concludes that untuned RRF is a sensible default and that effort may be better spent on evidence-unit construction than on lightweight per-query weighting.","tokens_in":10363,"tokens_out":5057,"duration_ms":62076,"significance":"If the results hold, the paper makes a useful methodological contribution: it quantifies a real retrieval-unit confound and offers a clean protocol for removing it. The evaluation design is a strength: company-grouped cross-validation with clustered bootstrap inference, a fixed decision rule for the router comparisons, an explicit oracle upper bound, and a literal-containment sensitivity subset. The negative result on lightweight query-adaptive fusion is also valuable, especially because the equal-weight blend is simple and the routers are described precisely enough to reproduce. The paper is not circular: the fixed baselines are training-free or independently tuned, and the routers' failure is the claim rather than a fitted success. However, the central comparison rests on automatically constructed relevance labels with no verified precision and with 10.9% of queries excluded; this makes the current evidence conditional rather than definitive. A manual label audit and analysis of excluded queries would substantially increase confidence.","major_comments":[{"comment":"The automatic reference-to-window mapping is the load-bearing component of the evaluation. It resolves only 5,082 of 5,703 queries (89.1%) and achieves literal containment for only 30.3% of references; no verified precision figure is reported. The paper's own qualitative inspection says failures concentrate in financial-statement tables, which is exactly the category where BM25's exact-number matching is expected to help. If unresolved or silently mis-mapped references are systematically distributed, the measured +0.0090 RRF advantage and the null router differences could be artifacts of label construction rather than retrieval quality. The literal-containment subset (n=1,518) is a reasonable robustness check for the fusion-vs-components comparison, but it cannot speak to the 621 excluded queries or to references that are mapped incorrectly despite high overlap. I ask for a manual audit","section":"Section III-C/III-D; Tables I-II"},{"comment":"The only comparison that is statistically reliable on the full set — RRF over the equal-weight blend — is not reliable on the cleaner literal-containment subset. The paper reports a similar point estimate (+0.0092) but p=0.11 on that subset. Given that the practical guidance in Section VI recommends RRF as the default, the evidence for that recommendation is weaker than the full-table result implies. The revision should either temper the practical claim or provide additional evidence, such as a larger manually verified label set or an analysis showing that the full-set advantage is not driven by label noise.","section":"Section V-D; Table II"},{"comment":"The router comparison is statistically careful, but the study appears underpowered to detect small positive gains. The ridge router's one-sided p is 0.0555, close to the unadjusted 0.05 level, and the random-forest estimate is effectively zero with a wide interval. The conclusion that no router 'establishes' an improvement is formally correct, but the practical advice 'do not assume routing will help' goes further. I request a minimum-detectable-effect analysis under the company-clustered bootstrap, or a pre-specified equivalence/non-inferiority margin, so readers can distinguish 'no evidence of benefit' from 'evidence of no benefit.'","section":"Section IV-B; Table II"},{"comment":"The paper states that mapping failures concentrate in financial-statement tables and that the heading-only mapping may make some numeric queries' answer-bearing rows absent from any retrievable window. This is an important caveat, but it is not quantified for the 621 excluded queries. If the excluded queries are disproportionately numeric, the evaluation set is biased toward narrative queries, which would directly affect both the fusion gain and the router nulls. Please report the category distribution and other query features for the resolved versus unresolved sets, and if possible bound the impact of the exclusions on the main comparisons.","section":"Section III-D; Section V-D"}],"minor_comments":[{"comment":"The phrase 'these audited pairs' is unclear; define what 'audited' means here and how the 46.2%/55.2% figures were computed.","section":"Section III-B"},{"comment":"The training target description contains a likely typo: 'we take the best rank ... and computer d/(rb + rd)' should probably be 'compute' with the formula written consistently. Also clarify whether the exclusion of queries where neither retriever finds a relevant window in the top 100 applies only to the training set or also to evaluation.","section":"Section IV-B"},{"comment":"The ridge router row reports 'n/a' for window-level Recall@10 without explanation. Since the other rows report this secondary metric, please either compute it or state why it is not applicable.","section":"Table I"},{"comment":"The explanation of the α=1 endpoint difference from the BM25-only row is useful, but it would be clearer in the main text rather than only in the caption.","section":"Figure 1 caption"},{"comment":"The sentence describing the Bonferroni adjustment is convoluted: 'each fixed before that router's own final evaluation and tested in sequence rather than planned jointly at the outset.' Please clarify whether the three router tests were planned sequentially or jointly, since the stated correction assumes joint testing of three methods.","section":"Section IV-C"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the protocol is strong, but the automatic label quality is the central risk. I would not reject, because the issue is fixable: a manual precision audit of a stratified sample, plus an unresolved-query analysis, would largely resolve my concern. Please ask the authors to treat the 621 excluded queries and the 30.3% literal-containment rate as first-class evidence, not as a limitation paragraph. The RRF recommendation should be softened unless the label-noise analysis supports it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you care about hybrid retrieval or benchmark evaluation. It makes a point that should be obvious but isn't: when you compare a sparse retriever against a dense encoder that truncates the input, you're measuring truncation as much as semantics. The paper quantifies that on FinDER (46% of evidence begins past word 512), then fixes it by windowing the corpus to 160 words and mapping references to windows. That fairness correction is the real contribution, and it's done carefully.\n\nThe evaluation protocol is genuinely strong. Company-grouped cross-validation, cluster-robust bootstrap, a prespecified decision rule, and a literal-containment subset check. They report their label diagnostics honestly instead of hiding them. Fusion beats either component by ~28% at reference-level Hit@10, which is plausible and holds on the literal subset. The negative result—none of three lightweight per-query routers beats the fixed blend under company clustering—is credible and useful. The grid oracle showing 21.8% headroom makes that null result meaningful, not a failure of the setup.\n\nThe soft spots are real but manageable. The automatic reference-to-window mapping resolves only 89.1% of queries, with literal containment for only 30.3% of references and no verified precision. That's a genuine threat to the RRF-vs-blend comparison. The paper's own subset check is the right move: fusion-vs-components and the router nulls survive there, but RRF's edge over the blend drops to p=0.11, so the 'RRF as default' guidance is based on an exploratory difference. The stress-test note worries about systematic bias in the 621 excluded queries; that's possible, but the evidence doesn't show it, and the authors flag it as a limitation.\n\nOne more thing: no code or derived data. For a paper whose main value is a corrected corpus, that's a real reproducibility gap. It doesn't make the results wrong, but exact replication is impossible.\n\nWho's this for? Researchers building retrieval pipelines over long, table-dense documents, and anyone working on benchmark construction. It deserves a serious referee. The subfield would be better off with more null results this carefully done.","headline":"A solid, honest empirical study: the evidence-unit fairness point is real, the router null result is credible, and the label mapping is the main caveat—worth serious referee time.","tokens_in":10797,"tokens_out":2415,"would_cite":true,"duration_ms":28426,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Untuned reciprocal rank fusion outperforms three adaptive per-query weighting schemes on 10-K evidence retrieval, once retrieval units are cut to the encoder's window.","keywords":["financial document retrieval","sparse-dense hybrid retrieval","reciprocal rank fusion","10-K filings","evidence-unit fairness","query-adaptive fusion","dense retrieval","retrieval evaluation"],"falsifier":"Manually audit a random sample of the 5,082 resolved query-reference-window mappings, stratifying by query category and by the literal-containment diagnostic; if mis-mapping concentrates in table-heavy categories where BM25 and fusion appear to excel, recompute Hit@10 on verified mappings and check whether the 28 percent fusion gain and the router null results survive. A second check: rerun the router comparisons only on the 1,518 literal-containment queries with the full protocol; the paper already reports the router effects shrink there, so a confirmed null on a fully verified subset would s","tokens_in":9962,"feed_emoji":"🔍","tokens_out":4501,"duration_ms":47784,"temperature":0.7,"pith_summary":"Retrieving answers from corporate 10-K filings is hard because queries are short and the evidence sits deep inside long, table-heavy documents. This paper establishes two things. First, comparisons between lexical and dense retrieval are unfair unless the retrieval unit is cut to the dense encoder's reading window; on the FinDER benchmark, nearly half the labeled evidence begins beyond that window. Second, once that mismatch is fixed, fusing BM25 with a compact dense encoder improves reference-level Hit@10 by roughly 28 percent over either component alone, and an untuned reciprocal rank fusion scores highest of all tested methods. Three lightweight per-query weighting schemes fail to reliably beat the fixed equal-weight blend, despite an oracle showing 21.8 percent headroom, so the paper argues simple fusion is the defensible default and the next gains lie in better evidence-unit construction.","feed_headline":"Untuned rank fusion beats per-query routers on 10-K filings","feed_subtitle":"BM25 plus a small dense encoder, merged without tuning, beats three adaptive weighting schemes on 10-K evidence retrieval.","key_machinery":"The central mechanism is the evidence-unit fairness correction: segmenting every parsed filing unit into overlapping 160-word windows that fit the dense encoder's 512-token input, and mapping each labeled reference to its windows with provenance, so reference-level Hit@10 can be measured fairly. This correction turns a confounded comparison into one where the two retrievers are equally exposed to the evidence. The second working part is fusion: reciprocal rank fusion (combining ranked lists by summing reciprocal ranks with a fixed offset, here k=60) and weighted score fusion with an interpolation weight. The third is the oracle/router apparatus: a per-query grid oracle defines an upper bound","core_discovery":"The paper's central claim is that the apparent performance gap between sparse and dense retrieval on financial filings is partly a measurement artifact: when the retrieval unit is a large table or section, a dense encoder with a 512-token budget never sees most of the labeled evidence, while a lexical index sees all of it. After segmenting the corpus into overlapping windows that fit the encoder budget and mapping each expert reference to its windows, BM25 and a compact dense encoder score nearly the same alone (reference-level Hit@10 around 0.139 and 0.142), and fusing them raises the score to 0.180–0.189, a relative gain of roughly 28 percent. Reciprocal rank fusion with no training and no","pith_inferences":["Editorial inference: if the automatic label mapping is accurate, these results suggest that the main yield left on the table in financial-filing retrieval is not router sophistication but evidence-unit construction—keeping table headings attached to their numeric rows—and the same lesson should transfer to any retrieval pipeline over XBRL-sourced documents.","Editorial inference: the router failure is bounded by reachability: 77.6 percent of queries are unanswered under any weight, so even a perfect per-query weighting can only affect the reachable minority; a router that first scores reachability, not just weight, would be a more direct test of the headroom.","Editorial inference: the fairness correction is dataset-agnostic; any dense retriever used over long documents should be evaluated on encoder-sized windows, and this paper gives a template for reporting reference-level provenance that other benchmarks could adopt."],"forward_implications":["On encoder-windowed 10-K corpora, fusing BM25 with a compact dense encoder improves reference-level Hit@10 by roughly 28 percent over either component; the equal-weight blend is within a flat optimum, so it is not sharply tuned.","Untuned reciprocal rank fusion is the highest-scoring deployable method in this study, exceeding the equal-weight blend by 0.009 in reference-level Hit@10 with a clustered 95% confidence interval excluding zero, and requires no training or weight selection.","Per-query weighting is not worth the complexity here: none of the three lightweight routers reliably beats the fixed blend, even though the grid oracle shows 21.8 percent headroom.","A category-based hard switch between pure retrievers is also dominated by the fixed blend (0.158 versus 0.180), so the most direct use of category information does not beat simple fusion.","Tuning the fixed weight on a small validation split can slightly underperform simply setting it to 0.5, reported as suggestive, reinforcing the untuned-default recommendation."],"fun_headline_variants":["Simple fusion beats adaptive routers on 10-K evidence","28% fusion boost on 10-K retrieval","Untuned fusion wins; per-query weighting doesn't","Why adaptive fusion routing fails on 10-K filings","Per-query weighting can't beat fixed blend on 10-K"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The automatic reference-to-window mapping is accurate enough for the comparison; only 89.1 percent of queries resolved, with literal containment for just 30.3 percent of references and no verified precision figure, so systematically mis-mapped labels—especially in table-heavy accounting questions—could distort the measured fusion gains and the routers' failure.","fun_headline_variants_meta":{"raw":{"variants":["Simple fusion beats adaptive routers on 10-K evidence","28% fusion boost on 10-K retrieval","Untuned fusion wins; per-query weighting doesn't","Why adaptive fusion routing fails on 10-K filings","Per-query weighting can't beat fixed blend on 10-K"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00167,"raw_usage":{"total_tokens":6485,"prompt_tokens":787,"completion_tokens":5698,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":5619}},"tokens_in":531,"tokens_out":5698,"duration_ms":40514,"temperature":1.0,"reasoning_tokens":5619,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:03:43.309568+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually audit a random sample of the 5,082 resolved query-reference-window mappings, stratifying by query category and by the literal-containment diagnostic; if mis-mapping concentrates in table-heavy categories where BM25 and fusion appear to excel, recompute Hit@10 on verified mappings and check whether the 28 percent fusion gain and the router null results survive. A second check: rerun the router comparisons only on the 1,518 literal-containment queries with the full protocol; the paper already reports the router effects shrink there, so a confirmed null on a fully verified subset would s","supporting_citations":[],"review_version":1}