{"id":"edec423a-83ed-40b4-9891-5a3c60c4584e","arxiv_id":"2412.06009","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LeSeR, a hybrid dense-plus-BM25 reranking pipeline, achieves competitive recall and mAP on the RegNLP regulatory QA retrieval task.","lead":"1-800-SHARED-TASKS presents LeSeR, a retrieval pipeline that first uses a fine-tuned dense embedding model to gather candidates and then reranks them with BM25 lexical scores for regulatory question answering. On the RegNLP RIRAG benchmark, the best LeSeR configuration reaches recall@10 of 0.8201 and map@10 of 0.6655, surpassing both dense-only and BM25-only baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LeSeR's claimed gains rest on an unspecified weighted score fusion; without weights or an ablation, the semantic component's contribution is unverified.","rationale":"The paper's central empirical claim is Table 2's BGE_LeSeR row. I checked whether that row can be supported from the described method. The method section states that the dense top-20 are reranked by 'combining their dense semantic scores with sparse relevance scores generated by BM25 using a weighted aggregation approach,' but gives no formula, no weights, no normalization, and no mention of how the weights were chosen. This is exactly the reader's weakest assumption. My analysis strengthens it: BM25's baseline mAP@10 is 0.6237, close to LeSeR's 0.6655, while the dense model's mAP@10 is only 0.1077. Therefore even a fusion strongly dominated by BM25 would produce a score close to the reported value, and the semantic contribution might be negligible. The paper's Limitations section acknowledges the dense model's poor ranking precision, making this gap material rather than cosmetic. The proposed concrete test — an ablation that discards the dense score and reranks the dense top-20 by BM25 alone — would settle whether the hybrid claim is real or whether LeSeR is effectively BM25 applied to a high-recall candidate set. The reader's CONDITIONAL verdict remains appropriate; no change is needed, but the condition (releasing fusion details and ablation) is essential for the claim to be verified. Other potential issues, such as the selection of BGE_MNSR as the base model, are secondary because even with that choice the fusion step is the unverified link.","tokens_in":5284,"tokens_out":4606,"duration_ms":43424,"concrete_test":"Ask the authors for the exact fusion formula, weights, and normalization, or for the reranking code. Then run an ablation on the dev set: replace the dense cosine score in the weighted sum with a constant (i.e., rerank the dense top-20 candidates by BM25 score alone), keeping the candidate generation and all other settings identical. If this BM25-only reranking already yields mAP@10 ≈ 0.6655 and recall@10 ≈ 0.8201, then the semantic score contributes nothing to LeSeR's final ranking, and the central hybrid claim is unsupported. Additionally, report the chosen weight and confirm it was selected on dev, not on the test split.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LeSeR's central claim is that a two-phase dense-then-lexical pipeline outperforms both BM25 and BGE_MNSR on the RIRAG retrieval task (Table 2). The entire precision gain — mAP@10 from 0.1077 for BGE_MNSR to 0.6655 for BGE_LeSeR — is produced by the 'weighted aggregation' of dense cosine and BM25 scores described in Section 3. The paper never reports the fusion weights, the normalization of the two score distributions, or the procedure used to choose them. This is not a cosmetic omission. BM25 alone already reaches mAP@10 = 0.6237, so any fusion that gives even modest weight to the BM25 score will reproduce most of the reported mAP. Because the dense stage has very low mAP (0.1077), a weighted sum with even 80% BM25 weight would look similar to pure BM25 reranking of the dense top-20 candidates, and the claimed 'strict modularity' of LeSeR would be vacuous. The paper's Limitations section concedes that dense models 'underperformed in ranking precision,' which raises the burden of demonstrating that the dense scores contribute anything to the final ordering. As written, the central claim cannot be checked: the load-bearing fusion step is a black box, and the recall gain (0.8068 to 0.8201) may come entirely from restricting BM25 to the high-recall dense candidate set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a system description for the COLING 2025 RegNLP RIRAG shared task. For passage retrieval, the authors fine-tune several open-source dense embedding models (Stella, BGE, CDE, MedEmbed, MPNet) with MNSR loss on ObliQA, retrieve top-20 candidates with FAISS, and then rerank by a 'weighted aggregation' of dense cosine and BM25 scores (LeSeR). On the test split, the best system BGE_LeSeR reports recall@10 = 0.8201 and mAP@10 = 0.6655, compared with the BM25 baseline (0.7611/0.6237) and dense-only BGE_MNSR (0.8068/0.1077). For answer generation, the paper evaluates Qwen2.5 7B, Mistral 7B, Mistral Nemo 12B, and Gemma-2 9B using the RePASs metric, with Qwen2.5 7B performing best.","tokens_in":5597,"tokens_out":4287,"duration_ms":40215,"significance":"If the reported numbers are taken at face value, the paper offers a useful practical recipe for regulatory passage retrieval: dense candidates provide high recall and lexical reranking restores precision. The use of a public benchmark (ObliQA), standard metrics, and a systematic comparison across five embedding models are strengths, and a shared-task system description is a legitimate contribution to the workshop literature. However, the central methodological novelty is not currently verifiable: the fusion step is described only as a 'weighted aggregation' with no weights, normalization, or tuning protocol, and the reported gain in mAP is exactly the quantity that this unspecified component controls. The significance of the paper therefore depends on the authors disclosing the fusion details and providing an ablation; as written, the LeSeR contribution cannot be assessed independently of the unspecified fusion hyperparameters.","major_comments":[{"comment":"The 'weighted aggregation' that defines LeSeR is never specified. The paper does not state how dense cosine similarities and BM25 scores are normalized before combination, what weights are used, or how the weights were chosen. This matters because BM25 alone already achieves mAP@10 = 0.6237, while dense-only BGE_MNSR achieves 0.1077; a fusion that is even mostly lexical would trivially reproduce most of the reported 0.6655. As written, the central claim that BGE_LeSeR 'outperforms all other models' cannot be checked.","section":"§3, Methodology; Table 2"},{"comment":"The manuscript does not disclose whether the fusion weights, the candidate pool size of 20, or the fine-tuning checkpoints were selected on the development split or on the test split. Since Table 2 reports test-set numbers and the core claim is that LeSeR generalizes, the selection protocol must be stated. If test-set information was used to choose any hyperparameter, the reported numbers would be circular; if the dev set was used, that should be said explicitly.","section":"§3, dev/test protocol; Table 2"},{"comment":"The paper's own Limitations section concedes that dense models 'underperformed in ranking precision,' and the recall improvement from BGE_MNSR to BGE_LeSeR is only 0.0133 (0.8068 to 0.8201). To support the mechanistic story that semantic retrieval supplies high-recall candidates and lexical reranking supplies precision, the authors should report an ablation: dense-only, BM25-only, dense-top-20 with BM25 reranking, and dense-plus-BM25 at several fusion weights and candidate-pool sizes. Without this, the contribution of the dense stage to the final ranking is unverified.","section":"§4, Limitations; Table 2"}],"minor_comments":[{"comment":"LeSeR is expanded inconsistently: the title and abstract say 'Lexical Reranking of Semantic Retrieval,' while §3 introduces 'LeSeR (Lexical-Semantic Retrieval).' Please unify the expansion.","section":"Title, §1, §3"},{"comment":"There are several typos and formatting inconsistencies: 'reprsent' in the Table 2 caption, 'baselines model' in §4, and 'map@10' in the abstract should be 'mAP@10'.","section":"Table 2, §4, Abstract"},{"comment":"Figure 1 is cited in §3 as 'shown in Figure 1' but does not appear in the manuscript text; the reference should be either implemented or removed.","section":"§3, Figure 1"},{"comment":"No code, configuration files, or data availability statement are included. For a system description, even a short statement on release of the retrieval/reranking code and fine-tuning configuration would substantially aid reproducibility.","section":"Overall, Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of a shared-task system-description venue, and the reported results are plausible but not yet verifiable because the fusion step is a black box. I recommend major revision rather than rejection because the missing details (weights, normalization, tuning protocol, ablation) are straightforward to supply. If the revised paper shows that LeSeR is essentially BM25 reranking of dense candidates, that is still a useful finding as long as it is stated honestly. Please also check whether the workshop requires code release; the absence of any reproducibility statement is a gap for a system-description paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a solid shared-task system description wrapped in slightly inflated language. The actual method is a standard dense-retrieve-then-BM25-rerank hybrid, and the paper's central number—mAP jumping from 0.1077 for the dense model to 0.6655 for LeSeR—is real only if the undisclosed fusion weights are what make it work.\n\nThe paper does several things well. It evaluates five embeddings, fine-tunes with MNSR loss, reports both recall and mAP, and includes a limitations section that honestly concedes dense models underperform on ranking precision. The recall gain from 0.8068 to 0.8201 is modest but plausible, and the answer-generation comparisons are straightforward.\n\nThe soft spots are real and load-bearing. The weighted aggregation of BM25 and dense scores is never specified: no weights, no normalization, no tuning procedure. Because BM25 alone reaches mAP@10 of 0.6237, any fusion with even modest BM25 weight will reproduce most of that number. The stress-test concern is on target: as written, the dense component's contribution to the final ranking is unverified. The paper's own limitation statement says dense models 'underperformed in ranking precision.' That is not a flaw in itself, but it raises the bar for showing that the dense scores add anything beyond candidate recall. Also missing are error bars, significance tests, and code or data release; for a shared-task paper these are less critical but still worth noting.\n\nThe 'novel approach' claim is overstated. Decoupling dense retrieval and lexical reranking is a routine extension of hybrid retrieval. The paper is best read as a description of what worked on the RIRAG benchmark, not a conceptual advance.\n\nRecommendation: this deserves a serious referee, but the referee should condition acceptance on the authors disclosing the fusion weights and ideally an ablation with BM25-only reranking on the same candidate set. For a shared-task system description, that is a reasonable bar. I would bring it to a reading group only if someone is working on regulatory QA or hybrid retrieval recipes.","headline":"A competent shared-task system description whose headline gain rests on an unspecified fusion weight; worth a conditional accept, not a pass.","tokens_in":6124,"tokens_out":2578,"would_cite":false,"duration_ms":24492,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage dense-then-BM25 retrieval beats both pure approaches on regulatory QA.","keywords":["regulatory question answering","hybrid retrieval","lexical reranking","BM25","dense retrieval","MNSR fine-tuning","ObliQA","retrieval-augmented generation"],"falsifier":"On a held-out set of regulatory queries, fix the fusion weights using only the development split, then sweep the weight from 0 to 1 and record mAP@10; if no interior weight is better than the pure dense and pure BM25 endpoints, the LeSeR advantage disappears, and if the optimum is at an endpoint, the hybrid claim reduces to whichever single retriever won.","tokens_in":5138,"feed_emoji":"⚖️","tokens_out":9060,"duration_ms":79189,"temperature":0.7,"pith_summary":"This paper is a system description for the RIRAG regulatory question-answering challenge, and its central claim is that retrieval improves when the job is split into two phases: a dense semantic search that gathers candidate passages and a lexical BM25 rerank that fixes their order. On the ObliQA regulatory benchmark, this LeSeR pipeline reaches a recall@10 of 0.8201 and a mean average precision@10 of 0.6655, beating both the pure lexical baseline BM25 (0.7611 and 0.6237) and the best purely dense model (0.8068 and 0.1077). The authors argue this is worth caring about because regulatory texts depend on exact terminology, so a model that can retrieve broadly but cannot rank precisely is not enough for compliance work. They also note, in their own Limitations section, that dense models alone underperform on ranking precision and that answer generation might improve if only the most relevant retrieved contexts were fed to the generator.","feed_headline":"Hybrid reranking beats BM25 and dense search on regulatory QA","feed_subtitle":"Adding BM25 scores to a fine-tuned dense retriever lifts recall@10 to 0.8201 and mAP@10 to 0.6655.","key_machinery":"LeSeR (Lexical reranking of Semantic Retrieval) is a two-stage pipeline that separates recall from precision: a fine-tuned embedding model returns the top 20 passages by cosine similarity, and BM25 sparse scores are then combined with those dense scores in a weighted aggregation, with the top 10 kept as the final ranking. The decoupling is the load-bearing design: dense matching supplies broad recall, and lexical matching supplies exact-term precision. The quality of the final ranking therefore depends on how the two score types are mixed.","core_discovery":"On its own terms, the paper's finding is that lexical reranking repairs the ranking weakness of dense retrieval in a legal domain. The authors fine-tune several open embedding models with a contrastive loss that treats every other example in the batch as a negative, retrieve the top 20 passages by cosine similarity of the embeddings, and then reorder those candidates by combining the dense scores with BM25 sparse scores in a weighted aggregation, keeping only the top 10. The fine-tuned BGE-small model, called BGE_MNSR, is the strongest candidate generator of the models tested, and after this reranking step, BGE_LeSeR outperforms every other system on both recall@10 and mAP@10, including the BM25 baseline. The same retrieval pipeline, when paired with the Qwen2.5 7B language model, also gives the best answer-generation score on the RePASs metric among the four LLMs compared.","pith_inferences":["The paper's numerical gain in mAP@10 appears to come almost entirely from the fusion step rather than from the dense retriever; a natural extension is to sweep the fusion weight between pure dense and pure BM25 on a fixed split and check whether any interior mixture beats both extremes.","Since the paper does not report the fusion weights, normalisation, or tuning procedure, a reproducibility test is to re-derive the weights on the development split and see whether the 0.6655 mAP@10 holds on unseen queries.","The authors' own Limitations section concedes that only the most relevant contexts should be appended to the generator prompt; a natural follow-up is to replace the top-10 context with the reranked top 3-5 passages and measure RePASs again.","The same two-stage recipe should transfer to other terminology-heavy domains such as medical or financial compliance, where exact phrases carry regulatory weight, but that is a testable guess rather than a claim of the paper."],"forward_implications":["A hybrid that reranks dense candidates lexically can beat both BM25 and dense-only retrieval on the same benchmark: BGE_LeSeR gives recall@10 0.8201 and mAP@10 0.6655, against 0.7611/0.6237 for BM25 and 0.8068/0.1077 for the best dense-only model.","Fine-tuning a small general embedding model with symmetric contrastive loss makes it the strongest candidate generator among the tested dense models, even before reranking.","For answer generation, the same retrieved context works best with Qwen2.5 7B, which reaches the top RePASs score (0.4340) of the four LLMs tested.","Regulatory question-answering systems should not rely on a dense retriever alone when rankings matter, because the dense models here show high recall but very low mAP."],"supporting_citations":[{"why":"Provides BM25, the sparse retrieval algorithm whose scores are fused with dense scores in the LeSeR reranking step.","marker":"(Robertson et al., 1994)"},{"why":"Introduces the ObliQA dataset, the RIRAG task, and the RePASs metric against which all retrieval and answer-generation results are measured.","marker":"(Gokhan et al., 2024)"},{"why":"Implements the dense vector search library used to retrieve the top-20 candidate passages.","marker":"(Douze et al., 2024)"},{"why":"Releases BGE-small-en-v1.5, the base embedding model that is fine-tuned into BGE_MNSR, the best candidate generator.","marker":"(Xiao et al., 2023)"}],"fun_headline_variants":["LeSeR reranking lifts recall@10 to 0.82 in regulatory QA","Lexical reranking on fine-tuned dense retriever wins RegNLP","Hybrid dense+BM25 reranking tops regulatory retrieval","BGE_MNSR plus LeSeR achieves top retrieval scores","Adding BM25 scores to dense retriever improves regulatory QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported superiority of LeSeR rests on the way the dense cosine scores and BM25 scores are merged, but the paper does not give the weights, normalisation, or tuning procedure, so if that merger was chosen arbitrarily or fitted to the test set, the advantage would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["LeSeR reranking lifts recall@10 to 0.82 in regulatory QA","Lexical reranking on fine-tuned dense retriever wins RegNLP","Hybrid dense+BM25 reranking tops regulatory retrieval","BGE_MNSR plus LeSeR achieves top retrieval scores","Adding BM25 scores to dense retriever improves regulatory QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1230,"prompt_tokens":904,"completion_tokens":326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":231}},"tokens_in":520,"tokens_out":326,"duration_ms":3646,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:05:09.739321+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of regulatory queries, fix the fusion weights using only the development split, then sweep the weight from 0 to 1 and record mAP@10; if no interior weight is better than the pure dense and pure BM25 endpoints, the LeSeR advantage disappears, and if the optimum is at an endpoint, the hybrid claim reduces to whichever single retriever won.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides BM25, the sparse retrieval algorithm whose scores are fused with dense scores in the LeSeR reranking step."}],"review_version":1}