{"id":"b02d4644-08a6-428e-b774-40975e37c30d","arxiv_id":"2412.19265","paper_version":1,"verdict":"REJECT","confidence":"LOW","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A language-model-only, two-phase retrieval pipeline with hard-negative training and a grid-searched ensemble is reported to outperform sparse, dense, and generative baselines on a Japanese legal retrieval test set and a small MS MARCO subset.","lead":"This paper proposes a two-phase, language-model-only retrieval pipeline for Japanese legal documents and adds an ensemble of three embedding models. The authors report large gains over BM25, dense, and generative baselines, but the ensemble weights are tuned on the test set and no code or data is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported SOTA gaps are within the noise of a 130-query test set, and the 0.72 ensemble score is produced by grid-searching weights on that same test set; neither claim is statistically supported.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the evaluation is too small and the ensemble weights are tuned on the test set. This is the single most load-bearing issue because if it lands, the central state-of-the-art claim collapses. The paper's contribution is an engineering recipe, and its reported improvements over the strongest baselines are within one standard error of a proportion on 130 examples. The ensemble score is especially vulnerable because selecting the best of 231 weight combinations on the test set is a form of test-set overfitting; the reported 0.72 is the maximum of a noisy search, not an unbiased estimate. A simple bootstrap or a held-out split would settle the matter. The paper has no machine-checked proofs or released code, so the empirical claim is the only support for the central conclusion. The internal inconsistencies in method naming (e.g., 'BM25Plus (Round 2)' vs. 'LMS (Round 2)') and the unexplained failure of DSI-QG are secondary; they reduce clarity but are not the decisive flaw. Because the identified concern directly undermines the stated claim and the reader already judged the paper as REJECT, no verdict change is needed.","tokens_in":9697,"tokens_out":3589,"duration_ms":31868,"concrete_test":"Run a paired bootstrap over the 130 test queries: resample queries with replacement 10,000 times, recompute Recall@5, Recall@10 and Recall@200 for LMS (Finetuned MLM) Round 2 and for the best baseline (DSI or CoCondenser), and report the 95% confidence interval for each difference. Then evaluate the ensemble fairly by splitting the 130 queries (e.g., 65/65): select alpha, beta, theta on the first half via the same grid search and report MyRecall@3 on the second half, repeated over multiple random splits. If the bootstrap CI for Recall@10 includes zero, or the held-out ensemble score falls below the non-ensembled R@3 of 68.72, the headline SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on Table II and the ensemble result in Section IV. The Japanese legal test set contains only 130 examples (Section V-A). At Recall@10 = 79.53 (about 103 of 130 queries), the standard error of a binomial proportion is sqrt(0.795 * 0.205 / 130) ≈ 3.5 percentage points; the claimed lead over DSI at Recall@10 is 4.32 points (79.53 vs 75.21), barely over one standard error, and at Recall@5 the lead is 3.64 points with a similar standard error. Paired comparisons do not rescue this because the methods are correlated. In addition, the ensemble weights alpha, beta, theta are selected by grid search over 231 combinations (Section IV) to maximize MyRecall@3 on the same 130-example test set used for the headline table. Maximizing over 231 noisy configurations on 130 queries yields an optimistic estimate; the reported 0.72 may reflect selection bias rather than a genuine ensemble gain. Without error bars, a significance test, or a held-out weight-selection split, the state-of-the-art claim is not distinguishable from chance. The absence of released code and data for the self-cited companion dataset compounds the problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-phase, language-model-only retrieval pipeline for Japanese legal text, with variants LMS (Round 1/2) and LMS (Finetuned MLM) (Round 1/2), and an ensemble that linearly combines three models with weights alpha, beta, and theta. The central claims are that LMS (Finetuned MLM) Round 2 achieves state-of-the-art results on a Japanese legal test set (e.g., Recall@200 of 97.46, Recall@10 of 79.53, Recall@5 of 72.50) and that a grid-searched ensemble reaches a final My Recall@3 score of 0.72, outperforming sparse, dense, and generative baselines. The paper also reports results on a small nonstandard split of MS MARCO.","tokens_in":9963,"tokens_out":5200,"duration_ms":49546,"significance":"If the reported results were properly supported, the two-phase hard-negative training recipe and the ensemble formulation would be a useful practical contribution to domain-specific legal retrieval. The manuscript gives a fairly detailed algorithmic description, including Algorithm 1 for the contrastive loss, and compares many baselines across multiple recall and ranking metrics. However, the central claims rest on an evaluation protocol whose load-bearing parts are not statistically sound: the ensemble weights are selected on the test set, the test set has only 130 queries, no error bars or significance tests are provided, and the MS MARCO comparison uses a nonstandard subset. The paper does not release code or data and relies on a self-cited companion dataset, which further limits verifiability.","major_comments":[{"comment":"The ensemble weights are selected by maximizing the same metric that is then reported as the ensemble's final performance. The text states that each weight combination was evaluated using My Recall@3 and that the selected weights produce a final My Recall@3 score of 0.72. Because this value is the maximum over a grid of 231 weight configurations evaluated on the same 130-query test set used in Table II, it is a selected maximum, not an unbiased estimate. The claimed ensemble improvement over individual models is therefore not demonstrated. A proper protocol would tune the weights on a separate validation split (or with cross-validation) and report the test score only for the fixed weights.","section":"Section IV, Eq. (4), Table I"},{"comment":"The Japanese legal test set contains only 130 examples. At Recall@10 = 79.53, the binomial standard error is about 3.5 percentage points, so the reported lead over DSI at Recall@10 (79.53 vs. 75.21, a 4.32-point gap) is barely above one standard error; the Recall@5 gap (72.50 vs. 68.86, 3.64 points) is below one standard error. The manuscript reports no error bars, no confidence intervals, and no significance tests, and paired comparisons cannot be assumed to rescue the result because the methods are correlated. The state-of-the-art claim in the Abstract and Section V-A is not statistically supported by the evidence presented.","section":"Section V-A, Table II"},{"comment":"The MS MARCO evaluation uses a nonstandard subset (15,270 training rows, 862 test rows, and roughly 134,000 corpus documents) rather than the full MS MARCO passage collection. The resulting numbers are not comparable to published MS MARCO results, and the text even refers to evaluating on the 'training split' while reporting test-like metrics. The Abstract's claim of strong performance on 'widely recognized benchmarks like MS-MARCO' is therefore not supported.","section":"Section V-B, Table III"},{"comment":"There are serious naming inconsistencies that make the central comparison unauditable. For example, the text in Section V-A says 'our method BM25Plus (Round 2) reaches 65.38' at Recall@3, but Table II lists BM25Plus (Round 2) with Recall@3 = 65.38 under Dense Retrieval while LMS (Finetuned MLM) Round 2 has Recall@3 = 68.72; Section V-B discusses BM25Plus (Round 1) and BM25Plus (Round 2) as the proposed methods, whereas Section III defines the proposed pipelines as LMS variants. The ensemble in Section IV uses LMS (Finetuned MLM) Round 2, but later text sometimes refers to the same or similar model under different names. Without consistent model names, the reader cannot determine which system produced which reported result.","section":"Section III, Section V-A, Tables II and III"}],"minor_comments":[{"comment":"The equation states 'where α + β + θ' but omits '= 1'; the constraint is introduced only in the prose below the equation.","section":"Equation (4)"},{"comment":"The paragraph says 'our method BM25Plus (Round 2)' when discussing a result that Table II attributes under Dense Retrieval; this conflates the method name with the sparse baseline BM25+ and should be corrected.","section":"Section V-A, paragraph on sparse retrieval"},{"comment":"The phrase 'on the training split of the MS MARCO passage dataset' is confusing because the section reports evaluation metrics; if the evaluation is on the test split, the wording should be changed, and if it is truly on a training split, the evaluation is invalid as a benchmark comparison.","section":"Section V-B"},{"comment":"Figure 5 is described only as a visualization of the grid search; the axes, color scale, and the location of the optimal weights are not explained in the caption or text, making the figure difficult to interpret.","section":"Figure 5"}],"recommendation":"reject","confidential_remarks":"The paper depends heavily on the self-cited companion dataset [Trung et al., 2024] and does not release code or data, so the experimental results cannot be independently reproduced. The weight-selection-on-test-set issue is a fundamental protocol flaw, not a presentation issue; even with reanalysis, the reported SOTA numbers would require a new evaluation. I would encourage the authors to resubmit after re-running the experiments with a proper held-out weight-selection split and with statistically grounded comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the pipeline is a coherent engineering recipe, but the state-of-the-art claim is not supported by the evaluation as reported. The headline numbers come from a 130-query test set, with no error bars, and the ensemble score is literally the maximum of a grid search over weights on that same test set.\n\nWhat is actually new: the paper takes the CoCondenser idea (corpus-aware MLM pretraining plus contrastive fine-tuning with hard negatives) and removes the BM25 component, relying entirely on a language model encoder. That is a reasonable simplification, and the three-stage refinement (regular negatives, then hard negatives) is described clearly. The ensemble of two off-the-shelf multilingual sentence encoders with the trained model is a practical trick that appears to help. The authors also include a broad set of baselines, including generative retrieval, and they are honest that the data comes from their own companion paper.\n\nWhere it falls short: first, the evaluation cannot carry the weight of the claims. With 130 test examples, the standard error on Recall@10 is around 3.5 percentage points, so the reported 4.3-point lead over DSI is within noise. Second, the ensemble weights (alpha, beta, theta) are selected by grid search over 231 combinations to maximize MyRecall@3 on the same 130 examples. The reported 0.72 is thus the result of fitting to the test set, not a predictive performance. There is no held-out split, no cross-validation, no significance test. Third, the naming is inconsistent: the text refers to 'BM25Plus (Round 2)' where the tables list 'LMS (Finetuned MLM) Round 2' as the best model, and the tables and text sometimes disagree about which method is being discussed. Fourth, no code or data is released, so the self-cited dataset and the MS MARCO split cannot be checked. The MS MARCO split (17,132 rows) is nonstandard, which means those numbers are not comparable to standard leaderboards.\n\nWho this is for: someone working on domain-specific dense retrieval might pick up the pipeline idea, but they should not quote the SOTA numbers. The paper deserves a serious referee because the recipe is coherent and the application is relevant, but the referee would need to require a much more careful evaluation before the claims can stand.","headline":"A coherent pipeline with an unsupported SOTA claim: 130 test queries, no error bars, and ensemble weights tuned on the test set make the headline numbers uninterpretable.","tokens_in":10472,"tokens_out":3389,"would_cite":false,"duration_ms":30419,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that language-model-only two-phase training with hard negatives beats sparse, dense, and generative baselines on Japanese legal text, and that a grid-searched ensemble reaches My Recall@3 of 0.72.","keywords":["two-phase retrieval","Japanese legal text","masked language model","contrastive learning","hard negatives","ensemble retrieval","dense passage retrieval","MS MARCO"],"falsifier":"Run the same pipelines on a larger held-out Japanese legal query set, or use repeated random 130-example splits, and check whether LMS (Finetuned MLM) Round 2 still beats CoCondenser and DSI by the reported margins; also choose ensemble weights on a training split and see whether My Recall@3 remains near 0.72.","tokens_in":9505,"feed_emoji":"⚖️","tokens_out":9944,"duration_ms":82064,"temperature":0.7,"pith_summary":"The paper seeks to establish that a retrieval pipeline built entirely from language models, with no BM25-style sparse stage, can achieve top results on Japanese legal text retrieval. Its two-phase recipe first continues pretraining an encoder with a masked-language-model objective on the legal corpus, then fine-tunes it in stages: retrieve candidate documents to build positive and negative pairs, train with a contrastive loss, and retrain on hard negatives. On a 130-example Japanese legal test set, the best variant, LMS (Finetuned MLM) Round 2, reports Recall@200 of 97.46, Recall@10 of 79.53, and Recall@5 of 72.50, beating sparse, dense, and generative baselines. The paper also reports that a weighted ensemble of three multilingual encoders, with weights found by grid search on the same data, reaches a My Recall@3 of 0.72, where My Recall@3 is the top-three recall metric used in the evaluation. A transfer experiment on a 17,132-row MS MARCO split supports the claim that the recipe generalizes beyond Japanese law.","feed_headline":"Hard negatives push legal retrieval past 97% recall","feed_subtitle":"Two-phase LM training with hard negatives tops sparse, dense, and generative baselines on Japanese legal data.","key_machinery":"The central machinery is a two-phase, three-stage encoder-training loop. Phase 1 runs masked-language-model pretraining on the legal corpus. Phase 2 Stage 1 uses the resulting encoder to retrieve candidate documents and builds positive and negative pairs; Stage 2 fine-tunes a dual encoder with a contrastive loss based on cosine similarity and a margin; Stage 3 generates hard negatives from the Stage 2 model and retrains. The ensemble step is a weighted linear combination $S = \\alpha s_1 + \\beta s_2 + \\theta s_3$ of scores from three encoders, with $\\alpha + \\beta + \\theta = 1$ and weights chosen by grid search. The load-bearing pieces are the hard-negative stage and the replacement of BM25+ by a language-model retriever, since the paper attributes most of the gain to Stage 3 and to eliminating sparse retrieval.","core_discovery":"The central claim is that multi-stage language-model training can replace sparse retrieval entirely in a domain-specific legal search system. Phase 1 masked-language-model pretraining adapts a multilingual encoder to the legal corpus; Phase 2 uses the adapted encoder to mine positive and negative document pairs, trains with a contrastive loss, and then repeats training with hard negatives. The paper reports that the hard-negative round (Stage 3) is the decisive step: LMS (Finetuned MLM) Round 2 improves Recall@10 over Round 1 by 4.62 points (79.53 vs. 74.91) on the Japanese legal set. The same method reports the best Recall@200 (97.46), beating the strongest dense baseline, CoCondenser, by 8.64 points at Recall@10 (79.53 vs. 70.89) and by 4.54 points at Recall@200 (97.46 vs. 92.92).","pith_inferences":["One test the paper does not run: measure each ensemble member's individual My Recall@3 and ablate Stage 3; if the domain-tuned member alone already reaches 0.72, the ensemble contribution is smaller than claimed.","With only 130 test examples, the reported percentage-point margins are uncertain; bootstrap confidence intervals or repeated random splits would tell whether the ordering over baselines is stable.","Because the ensemble weights were optimized on the same test set where performance is reported, a held-out split or cross-validated weight search would be a stronger check of the 0.72 number.","The same hard-negative recipe could plausibly transfer to other specialized legal domains (for example German or Korean statutes) with minimal change, provided a multilingual encoder covers the language."],"forward_implications":["If these results are right, a fully neural, BM25-free pipeline is sufficient for competitive domain-specific retrieval on Japanese legal text.","The reported 4.62-point Recall@10 gain from Stage 3 implies hard-negative retraining is the highest-leverage component of the pipeline.","The MS MARCO split results imply the same two-phase recipe transfers to a general English passage-retrieval benchmark, at least at the 17k-row scale tested.","Because grid-searching ensemble weights on the test metric raised My Recall@3 to 0.72, the paper implies that combining a domain-tuned model with generic multilingual encoders can improve ranking stability."],"supporting_citations":[{"why":"Supplies the Japanese legal dataset and the baseline retrieval task the paper optimizes.","marker":"[Trung et al., 2024]"},{"why":"CoCondenser is the method the paper adapts by removing BM25+; its scores serve as the strongest dense baseline.","marker":"[Gao and Callan, 2021]"},{"why":"MS MARCO provides the benchmark whose subset is used for the transfer experiment.","marker":"[Bajaj et al., 2016]"},{"why":"DSI is the generative baseline that the paper's methods outperform on both datasets.","marker":"[Tay et al., 2022]"},{"why":"DSI-QG is the query-generation variant whose low Japanese scores motivate the domain-specific pipeline.","marker":"[Zhuang et al., 2022]"},{"why":"DPR (GC-DPR) supplies a standard dense-retrieval baseline for the Japanese legal comparison.","marker":"[Karpukhin et al., 2020]"},{"why":"Sentence-BERT Siamese is a dense baseline used in the Japanese legal comparison.","marker":"[Reimers, 2019]"},{"why":"The multi-stage re-ranking model 3SRM serves as a domain-specific baseline.","marker":"[Sasazawa et al., 2023]"}],"fun_headline_variants":["Hard negatives lift legal retrieval recall by 4.6 points","Two-phase LM beats dense baselines on Japanese legal","Stage 3 hard negatives: the key to 97% recall","Legal retrieval: why hard negatives matter","From 74.9 to 79.5: hard negatives boost Recall@10"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim depends on treating a 130-example Japanese legal test set as large enough that gaps of a few percentage points are meaningful, and on choosing the ensemble weights by searching over that same test data.","fun_headline_variants_meta":{"raw":{"variants":["Hard negatives lift legal retrieval recall by 4.6 points","Two-phase LM beats dense baselines on Japanese legal","Stage 3 hard negatives: the key to 97% recall","Legal retrieval: why hard negatives matter","From 74.9 to 79.5: hard negatives boost Recall@10"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000325,"raw_usage":{"total_tokens":1784,"prompt_tokens":867,"completion_tokens":917,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":832}},"tokens_in":483,"tokens_out":917,"duration_ms":7350,"temperature":1.0,"reasoning_tokens":832,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:46:01.852095+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipelines on a larger held-out Japanese legal query set, or use repeated random 130-example splits, and check whether LMS (Finetuned MLM) Round 2 still beats CoCondenser and DSI by the reported margins; also choose ensemble weights on a training split and see whether My Recall@3 remains near 0.72.","supporting_citations":[{"cited_title":"Adaptive Two-Phase Finetuning LLMs for Japanese Legal Text Retrieval","cited_arxiv_id":"2412.13205","evidence_quote":"Supplies the Japanese legal dataset and the baseline retrieval task the paper optimizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DSI is the generative baseline that the paper's methods outperform on both datasets."},{"cited_title":"Text Retrieval with Multi-Stage Re-Ranking Models","cited_arxiv_id":"2311.07994","evidence_quote":"The multi-stage re-ranking model 3SRM serves as a domain-specific baseline."}],"review_version":1}