{"id":"c9dbfa61-347f-40a1-bc3b-dd28125ef418","arxiv_id":"2412.13205","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-phase fine-tuning pipeline (global contrastive pretraining, then domain-specific hard-negative training) improves LLaMA-based dense retrieval on a Gemini-generated Japanese legal dataset and on a subset of MS MARCO.","lead":"This paper adds a second hard-negative fine-tuning stage to a LLaMA-based dense retriever and evaluates it on a new Japanese legal dataset and on a split of MS MARCO. If the reported gains hold under independent evaluation, the recipe is a modest but useful extension for low-resource legal retrieval.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set relevance labels are Gemini-generated and manually verified without expert audit, so the reported Japanese-dataset gains may measure agreement with Gemini rather than legal relevance.","rationale":"The reader's weakest assumption correctly identifies the Achilles heel: the evaluation ground truth is partly model-authored and lacks expert validation. This is the most load-bearing concern because the paper's headline result on the Japanese legal dataset depends entirely on the accuracy of those labels. The concern is strengthened by the fact that training data were also generated by Gemini from the same laws, creating a shared-provenance risk: the model can learn to mimic Gemini's relevance judgments, and the test labels will reward exactly that mimicry. This is not an accusation of intentional bias but a structural property of using the same LLM for both training-data generation and test-label creation, with only a non-expert manual screen. The paper does provide real independent support: it releases code and model checkpoints, and the pipeline is a concrete extension of RepLLaMA, so replication is feasible. However, the reported margins over RepLLaMA on the 130-query test set are also statistically fragile, which compounds the label concern. A focused expert audit of the test labels is the decisive check: it directly tests whether the dataset measures legal relevance as a human legal professional would understand it. If experts confirm the labels, the concern is mitigated; if they do not, the Japanese-dataset claims are not yet established. The MS MARCO split result, while also non-standard and not significance-tested, is secondary because the paper's primary contribution is the Japanese legal dataset and pipeline. On balance, the reader's CONDITIONAL verdict remains appropriate: the paper is a plausible engineering contribution but should be accepted only with the expert audit and significance reporting requested.","tokens_in":23268,"tokens_out":3337,"duration_ms":38600,"concrete_test":"Have two or more Japanese legal experts independently label a blinded sample of all 130 test queries' top-10 documents from Ours and RepLLaMA (or a random 50-query subset if full annotation is infeasible). Compute expert-expert agreement (e.g., Cohen's kappa), agreement between experts and the Gemini-derived labels, and recompute R@10/MRR@10 system rankings using expert labels. Also report bootstrap 95% confidence intervals for the R@10 difference on the original 130-query test set. If expert labels disagree substantially with Gemini labels, or if the system ordering flips under expert labels, the paper's central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 describes the Japanese legal dataset: contract chunks were matched to law articles by Gemini 1.5 Pro, then manually verified only by removing clearly unrelated chunks; no legal experts, annotator counts, or inter-annotator agreement are reported. Stage (v) further uses Gemini to generate 3,172 training sentences from the same laws. Thus the training data and the test labels share the same model-authored provenance. A LLaMA retriever fine-tuned on Gemini-generated positives will learn Gemini's notion of relevance, and the test labels, also produced by Gemini, may reward that alignment rather than true legal relevance. The central claim that the two-phase pipeline outperforms RepLLaMA on this dataset (R@10 76.87 vs 73.73, MRR@10 82.54 vs 81.58) could therefore be an artifact of label bias. Even if the labels were perfect, the 130-query test set yields a standard error of roughly 5.4 percentage points for per-query binary R@10, so the 3.1-point margin is within noise; no confidence intervals or significance tests are reported anywhere in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-phase fine-tuning pipeline for LLM-based dense retrieval, tailored to Japanese legal text, and introduces a new corpus: 743 Japanese statutes, 302 human-identified contract-chunk-law pairs (with 3,172 Gemini-generated training sentences), split into 99/73/130 train/validation/test examples. Phase 1 trains a LLaMA-2-7B retriever with BM25+ negatives and in-batch negatives; Phase 2 fine-tunes on hard negatives mined from Phase 1. An ensemble variant combines checkpoints and/or BM25+ scores. Experiments report improved Recall@k, MRR@10, MAP@10, and nDCG@10 over sparse, dense, and generative baselines on the Japanese legal dataset, and improved results on a small custom split of MS MARCO. The authors release code and checkpoints.","tokens_in":23530,"tokens_out":3678,"duration_ms":37108,"significance":"If the reported results are reliable, the two-phase training strategy is a plausible and useful contribution for non-English legal retrieval, and the Japanese legal dataset (code, checkpoints released) could be a resource for the community. The paper's inclusion of well-known baselines and its attempt to show English generalization are positive. However, the Japanese label construction relies on a single LLM with light manual verification, the test set is small (130 queries), and no statistical uncertainty is reported; the ensemble weights are tuned on the test set. These issues currently prevent the paper from establishing its central empirical claims. The MS MARCO split is non-standard, further weakening the English-generalization claim. The release of code and model checkpoints is a strength that enables independent verification.","major_comments":[{"comment":"The test-set relevance labels are produced by Gemini 1.5 Pro and then only manually verified by removing 'clearly unrelated' chunks; no legal experts, annotator counts, or inter-annotator agreement are reported. The training positives in stage (v) are also Gemini-generated. The reported gains over RepLLaMA on the Japanese dataset (R@10 76.87 vs 73.73, MRR@10 82.54 vs 81.58) may therefore reflect agreement with Gemini's notion of relevance rather than legal relevance. The paper should either provide an expert audit of the test labels (with agreement statistics) or explicitly condition the conclusions on the validity of the Gemini-generated labels.","section":"§3.1 (iv)–(v), §5.1"},{"comment":"The ensemble weights w1=0.6 and w2=0.4 for 'Ours(ckpt*)' are selected based on test-set performance ('based on our experiments ... yields the best performance'). This is test-set tuning, so the reported ensemble numbers (e.g., R@10 83.42, MRR@10 83.00 in the text) are optimistically biased. Model selection should be performed on the validation set or via nested cross-validation, and the paper should report the variance of the chosen weights.","section":"§4, Table 3"},{"comment":"With 130 test queries, a per-query binary R@10 has a standard error of approximately 5.4 percentage points. The reported difference over RepLLaMA (76.87 vs 73.73, a 3.14-point margin) is well within this noise. No confidence intervals, significance tests, or per-query variance are reported anywhere in the paper. The in-domain superiority claim is therefore not statistically supported in its current form.","section":"§5.1, Table 4"},{"comment":"The MS MARCO experiment uses a non-standard 'split of 17,132 rows' (15,270 train / 1,000 validation / 862 test) and a corpus of ~134,000 documents, rather than the full MS MARCO passage collection or the standard development set. It is not clear whether the baselines in Table 5 were retrained and evaluated on this same split with matched hyperparameters. Without a common, well-defined evaluation protocol, the claim that the pipeline 'proves effective in English contexts' and 'surpasses comparable baselines on the MS MARCO dataset' is not established.","section":"§5.2, Table 5"},{"comment":"The DSI-QG baseline on the Japanese dataset uses the English docT5query generator (as the paper acknowledges), which is known to be a poor fit for Japanese. Listing this as a standard baseline and then using its very low scores (e.g., R@10 28.77) to demonstrate superiority is unfair. Either run DSI-QG with a Japanese-appropriate query generator, or remove the row and clearly state that DSI-QG was evaluated only under a degenerate configuration.","section":"§5.1, Table 4 (DSI-QG row)"}],"minor_comments":[{"comment":"The 'Single' row appears to contain merged digits (e.g., '98.2183'), and the text states MRR@10 = 83.00 while the table's MRR@10 column parses inconsistently. Please correct the table formatting and verify the numbers against the text.","section":"Table 3"},{"comment":"The notation 'LoRa' is used inconsistently (it should be 'LoRA', Low-Rank Adaptation), and the terms 'globalDataset', 'domainSpecif icDataset', and 'humanLabeledDos' in Algorithms 1–2 are informal; please use standard camelCase or mathematical notation.","section":"§4 and algorithms"},{"comment":"The hyperparameters a1 and a2 from Algorithm 1 are not explicitly defined in the text; Section 5.1 describes 'sampled 50 relevant documents from BM25+ results' and 'top 50 relevant documents from the Phase 1 model,' but the mapping to a1 and a2 should be stated in the algorithm or implementation details.","section":"§5.1"},{"comment":"The embedding notation 'VT = Decoder(\"t1 t2 . . . tk <EOS>\")[−1]' is ambiguous; clarify that [−1] denotes the hidden state at the <EOS> position, and define the vector for queries and documents separately.","section":"§3.2, Eq. (1)"},{"comment":"There is a stray period in 'Japanese e-Gov website. 1,' and the sentence 'Laws were retrieved from the Japanese e-Gov website. 1, which provides access...' should be reworded.","section":"§1 and §3.1"},{"comment":"The two experiments use different max sequence lengths (512 vs 128), batch sizes, and numbers of negatives; the paper should discuss whether these differences are justified by dataset characteristics and whether the MS MARCO experiment is intended as a matched-domain test or a domain-shift test.","section":"§5.1 vs §5.2"}],"recommendation":"major_revision","confidential_remarks":"The core idea—a two-phase fine-tuning strategy with hard-negative deepening—is reasonable and the released code/checkpoints are valuable. However, the label provenance and the lack of statistical testing are serious obstacles to the main empirical claims. If the authors can supply an expert audit or a human-labeled test subset, report confidence intervals, and move ensemble selection to a validation set, the paper could become acceptable. The MS MARCO comparison also needs to be placed on a standard footing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper gives you a genuinely new Japanese legal retrieval dataset, a clear two-phase fine-tuning recipe on top of RepLLaMA, and code. The second phase is essentially hard-negative mining, familiar from ANCE/RocketQA, but the combination with RepLLaMA training and batch-size-1 fine-tuning is not something I've seen reported for Japanese legal IR. The single-model results are consistently a few points above RepLLaMA on both their Japanese test set and their MS MARCO split. That is a real outcome, not a rehash.\n\nThe soft spots are mostly statistical and one is about label provenance. The Japanese relevance labels were produced by Gemini 1.5 Pro and then manually filtered only to remove clearly unrelated chunks. No legal experts, no annotator counts, no agreement. The training sentences in stage (v) are also Gemini-generated. So the model may be learning Gemini's notion of relevance and the test labels may reward that alignment. I don't think this is fatal, but it means the dataset needs an expert audit before anyone treats the numbers as measuring legal relevance. Second, the test set is 130 queries; the R@10 margin over RepLLaMA is about 3 points, which is within one standard error. There are no confidence intervals or significance tests anywhere. Third, the ensemble weights in Section 4 are chosen on the test set, so those specific numbers are cherry-picked. The single-model comparison is the clean one, and even that is underpowered. Fourth, the MS MARCO split is non-standard and small, so the English generalization claim is suggestive, not strong.\n\nWhat the paper does well: it is transparent about the dataset construction, the baselines are re-run on the same split, and the code/checkpoints are promised. That is the right way to build a resource paper. The DSI-QG comparison on Japanese is unfair because the query generator is English, and the paper admits it, so I don't hold that against them.\n\nWho is this for: people working on low-resource legal IR, especially Japanese, who want a starting dataset and a simple recipe. It deserves peer review, but the reviewers should send it back for a revised evaluation with error bars, held-out hyperparameter selection, and an expert label audit.","headline":"A useful but statistically fragile extension of RepLLaMA: the two-phase recipe and Gemini-generated Japanese legal dataset are worth a look, but the reported gains need error bars and a clean label audit.","tokens_in":24055,"tokens_out":3336,"would_cite":false,"duration_ms":36426,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a two-phase fine-tuning pipeline, in which a LLaMA-based bi-encoder is first trained on broad positives and in-batch negatives and then on query-specific hard negatives, outperforms existing sparse, dense, and…","keywords":["Japanese legal text retrieval","two-phase fine-tuning","LLM bi-encoder","hard negative mining","dense retrieval","legal dataset","MS MARCO","ensemble retrieval"],"falsifier":"Have several licensed Japanese labor-law attorneys independently re-annotate the 130 test queries against the 743-article corpus, blind to the model outputs; if inter-annotator agreement is low or the corrected labels shift, recompute Table 4 and check whether the margin over RepLLaMA survives. A cheaper supplementary test is to evaluate only on the 302 human-authored contract chunks, excluding the 3,172 Gemini-generated sentences, to see whether the phase-2 gains depend on generated training data.","tokens_in":23084,"feed_emoji":"⚖️","tokens_out":6765,"duration_ms":64563,"temperature":0.7,"pith_summary":"The paper is trying to establish that large language models can be made to retrieve Japanese legal texts well by fine-tuning them in two deliberate stages rather than one. In the first stage the model learns broad context from positive documents, BM25+ negatives, and in-batch easy negatives; in the second it is pushed to distinguish the query's own top-ranked non-relevant documents, called hard negatives, from true positives. The authors support the claim with a new Japanese legal corpus, 743 statute articles and roughly 3.5 thousand contract chunks labeled with relevant articles, and report that the pipeline beats sparse, dense, and generative baselines, including RepLLaMA, on recall and ranking metrics. They further report that the same pipeline transfers to a split of MS MARCO, suggesting the recipe is not Japanese-specific. A sympathetic reader would care because it offers a parameter-efficient path to building retrievers for low-resource legal domains without expert annotation at scale.","feed_headline":"Two-phase finetuning beats RepLLaMA on Japanese legal retrieval","feed_subtitle":"New 3.5k-row Japanese legal dataset plus phase-2 hard negatives lift Recall@10 to 76.87 versus 73.73.","key_machinery":"The load-bearing mechanism is contrastive training of a LLaMA-2 bi-encoder whose query and document vectors are the hidden states of an appended <EOS> token, optimized by InfoNCE loss. Phase 1 assembles a globalDataset of true positives, top-BM25+ negatives, and in-batch negatives from other queries, teaching broad generalization; Phase 2 builds a domainSpecificDataset containing only positives and hard negatives mined by the Phase 1 model itself, with batch size 1 so each query competes only against its own near-miss documents. LoRA and 16-bit quantization make the 7-billion-parameter model trainable on two 24GB GPUs, and an ensemble variant adds BM25+ lexical scores and multiple checkpoints with learned weights.","core_discovery":"The central claim, stated on the paper's own terms, is that a two-phase fine-tuning schedule built on RepLLaMA's LLaMA-based bi-encoder yields better retrieval than any of the compared single-phase, sparse, dense, or generative methods. On the Japanese legal test set of 130 queries, the single two-phase model reaches Recall@10 of 76.87 and MRR@10 of 82.54, against 73.73 and 81.58 for RepLLaMA; adding BM25+ and a second checkpoint to the ensemble raises Recall@10 to 84.12. On a split of MS MARCO, the same two-phase approach reaches Recall@10 of 80.71 versus 77.55 for RepLLaMA. The paper also claims its dataset, built in Japanese with Gemini 1.5 Pro and manual verification rather than translated, is itself a contribution for future research in non-English legal retrieval.","pith_inferences":["Extension beyond the paper: because the test set has only 130 queries and all labels trace back to one LLM plus unnamed manual verification, the reported margins are likely to shrink under independent re-labeling by several legal experts; that check is the most direct way to test the headline gains.","Extension beyond the paper: the same pipelined generation (LLM writes contract-like sentences, LLM evaluates their relevance) may create a label source that correlates with the retriever's own inductive biases, so a retriever trained and evaluated on it could look better than it would on fully human-authored contracts.","Extension beyond the paper: the phase-1/phase-2 schedule with self-mined hard negatives should be testable as a drop-in addition to other LLM bi-encoders, such as those based on LLaMA-3, without changing the loss or the batch-size-1 deepening step."],"forward_implications":["If the two-phase claim holds, the same pipeline should transfer to other non-English legal systems, since the only Japanese-specific components are the corpus and the BM25+ tokenizer settings.","Phase 2's batch-size-1 hard-negative deepening is credited with the gains over single-phase RepLLaMA; removing it should lower precision at small k on the Japanese test set.","Combining BM25+ with fine-tuned LLM embeddings in an ensemble gives the best published numbers on this corpus (Recall@10 84.12), implying lexical and semantic signals are complementary rather than redundant.","The dataset itself, 743 statute articles and 3,259 labeled training rows, gives the community a reusable non-English legal retrieval benchmark."],"supporting_citations":[{"why":"Supplies the RepLLaMA architecture this pipeline builds on: LLaMA-based bi-encoder with <EOS> token embeddings and multi-stage training.","marker":"Ma et al., 2024"},{"why":"Provides the DPR contrastive-training formulation and in-batch negative mechanism adapted in Phase 1.","marker":"Karpukhin et al., 2020"},{"why":"CoCondenser is cited as one of the inspirations for the two-phase design and as a dense-retrieval baseline.","marker":"Gao and Callan, 2022"},{"why":"BM25+ is the sparse retriever used to mine Phase 1 negatives and as an ensemble component.","marker":"Robertson and Walker, 1994"},{"why":"LLaMA-2-7B is the base model fine-tuned with LoRA and quantization in both phases.","marker":"Touvron et al., 2023a,b"},{"why":"MS MARCO is the English benchmark whose split is used to show cross-lingual generality.","marker":"Nguyen et al., 2016"}],"fun_headline_variants":["Two-phase finetuning wins Japanese legal retrieval","New dataset plus two-phase finetuning for Japanese legal","Adaptive two-phase finetuning boosts Japanese legal search","Two-phase finetuning beats RepLLaMA on Japanese legal","Japanese and English legal retrieval improved by two-phase finetuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's relevance labels are accurate enough to support the reported margins: the labels were produced by Gemini 1.5 Pro and then manually verified, but the paper reports no legal experts, no annotator counts, and no agreement statistics, and the test set has only 130 queries.","fun_headline_variants_meta":{"raw":{"variants":["Two-phase finetuning wins Japanese legal retrieval","New dataset plus two-phase finetuning for Japanese legal","Adaptive two-phase finetuning boosts Japanese legal search","Two-phase finetuning beats RepLLaMA on Japanese legal","Japanese and English legal retrieval improved by two-phase finetuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1435,"prompt_tokens":907,"completion_tokens":528,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":523,"tokens_out":528,"duration_ms":5173,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:32:58.879348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several licensed Japanese labor-law attorneys independently re-annotate the 130 test queries against the 743-article corpus, blind to the model outputs; if inter-annotator agreement is low or the corrected labels shift, recompute Table 4 and check whether the margin over RepLLaMA survives. A cheaper supplementary test is to evaluate only on the 302 human-authored contract chunks, excluding the 3,172 Gemini-generated sentences, to see whether the phase-2 gains depend on generated training data.","supporting_citations":[{"cited_title":", Walker , S","cited_arxiv_id":null,"evidence_quote":"BM25+ is the sparse retriever used to mine Phase 1 negatives and as an ensemble component."}],"review_version":1}