{"id":"5c35fe76-b1b4-404e-aeeb-9d883e2c2ad8","arxiv_id":"2507.14619","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A lightweight Bi-Encoder plus Cross-Encoder pipeline with random top-candidate negative sampling achieves 79.1% MRR@10 on Vietnamese legal retrieval.","lead":"A small two-stage retrieval and re-ranking system for Vietnamese legal documents reaches competitive accuracy in a national hackathon, using carefully chosen wrong answers to train the re-ranker. The paper compares negative sampling strategies and introduces a recall-style metric for evaluating the first retrieval stage.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The local 79.11% MRR@10 may be inflated by qid leakage: Section 5.4 splits after Section 5.2 expands multi-answer questions, with no qid grouping described; only the private-test score is independent, and it lacks evaluation details.","rationale":"The reader's conditional verdict is appropriate. My stress-test pass concentrates on the single most load-bearing assumption: the validity of the local 90/10 split. The paper's own pipeline description contains all the ingredients for leakage: multi-answer questions are expanded in Section 5.2, and Section 5.4 says only that processed train.csv is split 90/10. No group-by-qid statement appears, and no code is released to verify the split. If qids straddle the split, the contrastive and BCE losses can memorize exact train rows, inflating every number measured on eval_df, including the claimed 23% relative improvement in the abstract (which, as the reader notes, does not match Table 2 in any case). The private test score is the strongest independent support and should be credited; however, a single reported number without evaluation details cannot establish the strategy-level comparisons (semi-hard vs hard vs easy), which are all local. The semi-hard labeling issue is secondary but real: Table 3's own statistics contradict the standard meaning of semi-hard, so the mechanism story is weakly supported even if the local numbers survive. My proposed check is cheap and decisive: reproduce the split, measure qid overlap, and if leakage is present, retrain with grouped splits. This is a standard correctness check and does not question the authors' integrity. Verdict remains CONDITIONAL: the paper is a useful competition-system description, but the headline local result should be re-validated under a grouped split, and the semi-hard terminology should be corrected or supported with a similarity-based selection criterion.","tokens_in":9745,"tokens_out":4915,"duration_ms":56104,"concrete_test":"Check the actual split for qid leakage and, if needed, re-run with a grouped split: after Section 5.4, compute the number and percentage of qids appearing in both train_df and eval_df. If non-zero (expected, given Section 5.2), retrain the best Bi-Encoder (11 epochs) and Cross-Encoder (semi-hard n=10, seed 28) using GroupShuffleSplit on qid with the same 90/10 proportion and report MRR@10 and Exist@90 on the held-out groups. If the grouped validation MRR@10 drops materially from 79.11, the local headline is inflated and only the private-test number remains; if it stays within about 1 MRR point, leakage is not the cause.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical evidence for the headline score is the local eval_df split, and the paper does not establish that this split measures generalization. Section 5.2 expands each multi-answer question into multiple rows with the same qid; Section 5.4 then performs a 90/10 split on the processed file without any stated qid-level grouping. Since roughly 10% of questions have multiple cids, random splitting can place rows with the same qid in both train and validation. The Bi-Encoder is trained with MultipleNegativesRankingLoss and the Cross-Encoder is trained on question-negative pairs, so a validation row whose qid appeared in training can be memorized; this would inflate the 79.11% MRR@10 and the Exist@90 values in Tables 1–3. The 77.54% private-test score is genuine independent evidence, but it is reported as a single number without submission protocol, number of runs, or confidence interval, so it cannot independently validate the strategy-level comparisons. A secondary concern is that the “semi-hard” negatives in Section 7.2 are uniform random draws from the Bi-Encoder top-90 after removing correct answers; Table 3 puts their mean cosine similarity at 0.2072 with 79% below 0.5, close to easy negatives (mean 0.0008) and far from the classic “similar-but-incorrect” definition. This weakens the mechanistic claim that semi-hard negatives, rather than simply negatives from the candidate distribution, drive the improvement, even though the empirical ordering in Table 2 is consistent across seeds.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a two-stage pipeline for Vietnamese legal document retrieval: a fine-tuned Vietnamese Bi-Encoder retrieves top-90 candidates, and a PhoRanker Cross-Encoder re-ranks them. The authors propose a recall-style metric Exist@m for the retrieval stage and compare three negative-mining strategies for the re-ranker, reporting that semi-hard negatives (random candidates from the Bi-Encoder top-90) outperform hard and easy negatives, with 79.11% MRR@10 on a local eval split and 77.54% on the SoICT 2024 private test.","tokens_in":10079,"tokens_out":7709,"duration_ms":81608,"significance":"If the empirical comparisons are valid, the main contribution is a practical demonstration that, for a legal QA retrieval task, training a cross-encoder with negatives sampled from the retriever's candidate distribution gives large gains over corpus-wide easy negatives and over the hardest in-candidate negatives, with consistent results across three random seeds. The paper is clearly written and does not overstate the architecture's novelty relative to ensemble competitors. However, the local evaluation is compromised by the data-splitting pipeline (Section 5.2 followed by Section 5.4) and by an ambiguous statement that the Bi-Encoder was fine-tuned using both train_df and eval_df (Section 7.1); until those issues are fixed, the headline local numbers cannot be regarded as evidence for generalization, and the private-test score lacks the protocol details needed to carry the comparison alone.","major_comments":[{"comment":"The sentence 'The Bi-Encoder is fine-tuned using the preprocessed train_df and eval_df datasets' is, if taken literally, direct training-data evaluation: the model would be evaluated on eval_df after training on it, invalidating all Exist@90 and MRR@10 values in Table 1 and the choice of the 11-epoch retriever. Please clarify whether this is a typo (i.e., trained on train_df and evaluated on eval_df) or re-run the experiments with a strict separation; this also affects the negative-mining step in Section 7.2 because the Bi-Encoder used to mine negatives would have seen eval_df.","section":"§7.1, Table 1"},{"comment":"The local validation split is made after Section 5.2 expands multi-answer questions into separate rows, and Section 5.4 does not state that the 90/10 split groups by qid. Since roughly 10% of questions are linked to multiple cids, rows with the same qid can fall on both sides of the split, so eval_df is not a held-out set for those questions. Both the Bi-Encoder (MultipleNegativesRankingLoss) and the Cross-Encoder (BCEWithLogitsLoss) are trained on question–document pairs, so a validation row whose qid appeared in training can be memorized; this inflates the local 79.11% MRR@10 and the Exist@90 values in Tables 1–3, and it undermines the strategy-level comparison in Table 2, which is the paper's central empirical claim. Please re-split by qid (or report both row-level and qid-level splits) and re-run all local tables; alternatively, provide per-strategy results on the private test.","section":"§5.2, §5.4, Table 2"},{"comment":"The operational definition of semi-hard negatives as uniform random draws from the Bi-Encoder top-90 after removing correct answers produces negatives with a mean cosine similarity of 0.2072 (79.44% below 0.5), which is much closer to the easy-negative distribution (mean 0.0008) than to the 'similar to the correct answers but still incorrect' definition stated in Section 3.3. Table 3 therefore does not demonstrate that the improvement in Table 2 is due to semi-hardness; it demonstrates only that negatives from the candidate set help. The paper should rename the strategy or provide an analysis that isolates hardness (for example, sampling from specified cosine-similarity bands), and the mechanistic claims in Sections 6.3 and 8 should be moderated accordingly.","section":"§7.2, Table 3"},{"comment":"The 77.54% MRR@10 private-test score is the only independent evidence for the headline result, but it is reported without any evaluation protocol: number of submitted runs, selection procedure (e.g., best-of-k), and whether the same negative-mining strategy was used. Because the local eval_df comparison is compromised by the qid split, this single number cannot independently validate the Table 2 strategy comparisons. Please add a subsection describing the private-test submission and, if possible, report private-test scores for at least the main negative-mining variants.","section":"Abstract / Conclusion / private test"}],"minor_comments":[{"comment":"Exist@m is recall@m for multi-document questions; the novelty claim in Section 1 should be calibrated by relating it to standard recall-oriented IR metrics.","section":"§6.2, Eq. (1)"},{"comment":"The text says 'Using PhoRanker, we measured the initial cosine similarity', but PhoRanker is a Cross-Encoder and does not naturally produce query–document cosine similarities; specify the model and how the embeddings or scores were obtained.","section":"§7.2, Table 3"},{"comment":"The BM25Plus configuration with b=0 is described as 'tuned k1 and b' in the text, but the table lists many configurations; clarify whether these are a grid search and how the best configuration was selected to avoid selection-on-test concerns.","section":"§7.1, Table 1"},{"comment":"Figure 3 places the 0.9/0.1 split after the 'separate if multiple answers' step; annotate the figure to make the qid-grouping choice explicit, or redraw the pipeline to split before expansion.","section":"Figure 3"},{"comment":"The conclusion states the model 'achieves same performance compared to competitors', but no table with other teams' private-test scores is provided; please add the competition leaderboard excerpt or precise ranking context.","section":"§8, Conclusion"},{"comment":"Section 4.4 says models with 1024-token capacity may be beneficial, but no such experiment is reported; either remove the speculation or add a sentence clarifying it is outside the scope.","section":"§4.4"},{"comment":"There is no statement about code or data availability; for reproducibility of an empirical benchmark paper, please add one.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The qid-leakage issue and the train-on-eval_df statement are severe enough that I would not rely on any local number until clarified. The private-test claim may be true, but the paper needs restructuring around a clean split. Given that a fix (qid-grouped re-split and re-run) is feasible, major_revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real contribution here is the systematic comparison of negative mining strategies for reranker training on Vietnamese legal text. The ordering semi-hard > easy > hard across three seeds in Table 2 is visible and consistent, and the discussion of why hard negatives hurt using the similarity distribution in Table 3 is a genuine attempt at explanation. The paper is clearly written, the pipeline is lightweight and competitive, and the private test score (77.54% MRR@10, top three) is independent evidence that the full system works. I'd give credit for that.\n\nBut the soft spots are real. The qid leakage concern holds up: Section 5.2 expands multi-answer questions into separate rows, and Section 5.4 does a 90/10 split on the expanded file without any described qid-level grouping. Since roughly 10% of questions have multiple cids, random splitting can place the same question in both train and validation. That likely inflates the local 79.11% MRR@10 and the Exist@90 numbers. The private test score is the only clean evidence, and it is reported as a single number without submission protocol or confidence interval, so it cannot validate the strategy-level comparison.\n\nThe 'semi-hard' negatives are also misnamed. They are random draws from the Bi-Encoder's top-90 list, and Table 3 puts their mean cosine similarity at 0.2072—close to easy negatives, far from the classic 'similar but incorrect' definition. The empirical ordering may still hold, but the mechanism claimed is weakly supported.\n\nTwo smaller issues: the abstract's '23% relative improvement' does not match Table 2 (the best semi-hard result is about 42% relative improvement over the PhoRanker baseline), and Exist@m is recall-at-m under a new name. No code or data are released, which hurts reproducibility.\n\nOverall, this is a competition system description rather than a methodological breakthrough, but it is a useful one for people working on Vietnamese legal IR or on reranker training with limited data. The split issue is fixable by re-splitting with qid grouping, and the numbers need alignment. I would send this to peer review, but the authors should expect major revision and would be wise to release code and data. A serious referee can verify whether the effect survives a clean split.","headline":"A useful Vietnamese legal retrieval system paper with a solid-looking negative-mining comparison, but the local validation split likely leaks question IDs and the 'semi-hard' negatives are closer to easy ones than the name suggests.","tokens_in":10611,"tokens_out":2562,"would_cite":false,"duration_ms":31515,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight two-stage Vietnamese legal retriever reaches 79.11% MRR@10 locally and 77.54% on the private test, matching far larger ensembles.","keywords":["legal document retrieval","Vietnamese information retrieval","two-stage retrieval pipeline","semi-hard negative mining","bi-encoder candidate retrieval","cross-encoder re-ranking","Exist@m metric","MRR@10"],"falsifier":"Group the processed training questions by their question ID before the 90/10 split and re-run fine-tuning: if MRR@10 falls from 79.11% toward 77.54%, the local split was leaking. Independently, train the re-ranker on easy negatives matched to the same mean cosine similarity as the claimed semi-hard negatives: if MRR@10 stays near 79%, the paper's mechanism is not the semi-hard difficulty band.","tokens_in":9545,"feed_emoji":"⚖️","tokens_out":6065,"duration_ms":67437,"temperature":0.7,"pith_summary":"The paper tries to show that legal document retrieval in Vietnamese does not require huge ensemble systems: a fine-tuned Bi-Encoder that retrieves 90 candidates, followed by a Cross-Encoder re-ranker trained on semi-hard negatives, can reach competitive accuracy on its own. The authors report 97% Exist@90 for candidate retrieval and a 23% relative improvement in re-ranking from semi-hard negative mining, culminating in 79.11% MRR@10 on the local evaluation split and 77.54% MRR@10 on the competition's hidden test set. Their message is that careful data preprocessing, a contrastive loss for the encoder, and balanced negative sampling matter as much as model scale.","feed_headline":"Semi-hard negatives lift Vietnamese legal retrieval to 79% MRR@10","feed_subtitle":"A light two-stage retriever and reranker beats large ensembles on a Vietnamese legal QA benchmark.","key_machinery":"The load-bearing mechanism is the two-stage pipeline: a Bi-Encoder quickly embeds the query and each legal document independently, allowing cosine search over the full 261,446-document corpus to return 90 candidates, and a Cross-Encoder then scores each query-candidate pair jointly and returns the top 10. Candidate negatives for the re-ranker are mined from the Bi-Encoder's own top-90 lists: after removing correct answers, hard negatives take the highest-scoring wrong documents, semi-hard negatives randomly sample from the remaining candidates, and easy negatives sample from the whole corpus. Exist@m is the metric that makes the retriever's job explicit, checking only whether the correct document is present in the candidate set rather than its rank, since the re-ranker will do the final ordering.","core_discovery":"The paper's central discovery is an engineering result: a two-stage retrieval and re-ranking pipeline, built from a fine-tuned Vietnamese Bi-Encoder and a pretrained Vietnamese Cross-Encoder, can place in the top three of a legal document retrieval competition while using far fewer parameters than ensemble approaches. The retriever is trained with MultipleNegativesRankingLoss and evaluated with a new metric, Exist@m, which measures whether at least one correct document appears among the top m candidates; the fine-tuned retriever reaches roughly 97% Exist@90, beating BM25. The re-ranker is trained with binary cross-entropy on positive pairs plus negatives mined from the retriever's top-90 candidates. Among hard, semi-hard, and easy negative mining, only semi-hard negatives give large and reliable gains, and increasing the number of such negatives from 2 to 10 steadily improves MRR@10, while hard negatives are unstable until the sample size becomes large.","pith_inferences":["Editorial extension: the paper's local 79.11% score would be more convincing if the 90/10 split grouped by question ID; the authors do not describe such grouping, so a reader should treat the private-test 77.54% as the stronger evidence.","Editorial extension: Table 3 shows the random top-90 negatives have mean cosine similarity 0.2072, close to the easy-negative distribution, so the paper's label 'semi-hard' may overstate how close these negatives are to true positives; the benefit could come more from sampling candidates on the retrieval frontier than from the difficulty band itself.","Editorial extension: a natural transfer test is to apply the same retrieval-plus-rerank recipe with retriever-mined semi-hard negatives to other low-resource domains or languages, where ensembles are less affordable.","Editorial extension: the authors attribute hard-negative instability to large gradients from near-positive samples; this could be tested directly by monitoring gradient norms or positive-negative score margins during training under each mining strategy."],"forward_implications":["If the result holds, a single fine-tuned Bi-Encoder plus Cross-Encoder is enough for competitive specialized legal retrieval, so expensive ensembles are not a prerequisite for top performance.","Retrieval stages in a two-stage pipeline can be tuned for candidate coverage with Exist@m instead of MRR@10, which should make retriever development faster and more directly aligned with the re-ranker's needs.","Semi-hard negatives mined from the retriever's own candidates appear to be a reliable and cheap training signal for re-rankers, with more negatives per question consistently improving performance.","The 23% relative MRR@10 gain suggests that data preparation steps like replacing truncated answers with full legal documents, splitting multi-answer questions, and Vietnamese word segmentation are not incidental but carry substantial value.","A lightweight pipeline of this kind is more practical for real-time or large-scale legal information retrieval than ensemble systems, because only 90 pairs per query reach the expensive Cross-Encoder."],"supporting_citations":[{"why":"Supplies the pretrained Vietnamese Bi-Encoder that is fine-tuned for candidate retrieval.","marker":"[15]"},{"why":"Supplies the pretrained Vietnamese Cross-Encoder used as the re-ranker.","marker":"[16]"},{"why":"Provides the MultipleNegativesRankingLoss that trains the Bi-Encoder without explicit negative labels.","marker":"[10]"},{"why":"Establishes the semi-hard negative sampling idea that the re-ranking stage builds on.","marker":"[9]"},{"why":"Defines BM25, the lexical baseline the fine-tuned Bi-Encoder must outperform.","marker":"[3]"},{"why":"Provides the Sentence-BERT-style Bi-Encoder and Cross-Encoder architecture underlying the pipeline.","marker":"[4]"}],"fun_headline_variants":["Semi-hard negatives boost Vietnamese legal retrieval reranking","Light retriever+reranker cracks top-3 in Vietnamese legal task","Semi-hard negative mining lifts legal rerank with fewer params","Two-stage pipeline beats big ensembles on Vietnamese law QA","Semi-hard negatives: key to efficient legal document ranking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The local 79.11% result depends on the 90/10 split of processed training questions not letting the same question appear in both training and validation; if that fails, only the 77.54% private-test score is independent evidence.","fun_headline_variants_meta":{"raw":{"variants":["Semi-hard negatives boost Vietnamese legal retrieval reranking","Light retriever+reranker cracks top-3 in Vietnamese legal task","Semi-hard negative mining lifts legal rerank with fewer params","Two-stage pipeline beats big ensembles on Vietnamese law QA","Semi-hard negatives: key to efficient legal document ranking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1652,"prompt_tokens":926,"completion_tokens":726,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":642}},"tokens_in":542,"tokens_out":726,"duration_ms":7856,"temperature":1.0,"reasoning_tokens":642,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:51:53.515173+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Group the processed training questions by their question ID before the 90/10 split and re-run fine-tuning: if MRR@10 falls from 79.11% toward 77.54%, the local split was leaking. Independently, train the re-ranker on easy negatives matched to the same mean cosine similarity as the claimed semi-hard negatives: if MRR@10 stays near 79%, the paper's mechanism is not the semi-hard difficulty band.","supporting_citations":[{"cited_title":"PhoRanker: A Cross-encoder Model for Vietnamese Text Rank- ing","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained Vietnamese Cross-Encoder used as the re-ranker."},{"cited_title":"Learning Dense Representa- tions for Entity Retrieval","cited_arxiv_id":null,"evidence_quote":"Establishes the semi-hard negative sampling idea that the re-ranking stage builds on."},{"cited_title":"Some Simple Effective Approxima- tions to the 2-Poisson Model for Probabilistic Weighted Retrieval","cited_arxiv_id":null,"evidence_quote":"Defines BM25, the lexical baseline the fine-tuned Bi-Encoder must outperform."},{"cited_title":"Sentence-BERT: Sentence Embeddings us- ing Siamese BERT-Networks","cited_arxiv_id":null,"evidence_quote":"Provides the Sentence-BERT-style Bi-Encoder and Cross-Encoder architecture underlying the pipeline."}],"review_version":1}