Pith. sign in

REVIEW 4 major objections 7 minor 16 references

Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A lightweight two-stage Vietnamese legal retriever reaches 79.11% MRR@10 locally and 77.54% on the private test, matching far larger ensembles.

desk verdict A useful Vietnamese legal retrieval system paper with a solid-looking negative-mining comparison, but the local validation split likely leaks question IDs and the 'semi-hard' negatives are closer to easy ones than the name suggests. read the letter →

arxiv 2507.14619 v1 pith:JXIUSSWI submitted 2025-07-19 cs.IR cs.CL

classification cs.IRcs.CL
keywords legaldocumentretrievalVietnameseinformationtwo-stagepipelinesemi-hardnegativeminingbi-encodercandidatecross-encoderre-rankingExist@mmetricMRR@10
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that legal document retrieval in Vietnamese does not require huge ensemble systems: a fine-tuned Bi-Encoder that retrieves 90 candidates, followed by a Cross-Encoder re-ranker trained on semi-hard negatives, can reach competitive accuracy on its own. The authors report 97% Exist@90 for candidate retrieval and a 23% relative improvement in re-ranking from semi-hard negative mining, culminating in 79.11% MRR@10 on the local evaluation split and 77.54% MRR@10 on the competition's hidden test set. Their message is that careful data preprocessing, a contrastive loss for the encoder, and balanced negative sampling matter as much as model scale.

What carries the argument

The load-bearing mechanism is the two-stage pipeline: a Bi-Encoder quickly embeds the query and each legal document independently, allowing cosine search over the full 261,446-document corpus to return 90 candidates, and a Cross-Encoder then scores each query-candidate pair jointly and returns the top 10. Candidate negatives for the re-ranker are mined from the Bi-Encoder's own top-90 lists: after removing correct answers, hard negatives take the highest-scoring wrong documents, semi-hard negatives randomly sample from the remaining candidates, and easy negatives sample from the whole corpus. Exist@m is the metric that makes the retriever's job explicit, checking only whether the correct document is present in the candidate set rather than its rank, since the re-ranker will do the final ordering.

What would settle it

Group the processed training questions by their question ID before the 90/10 split and re-run fine-tuning: if MRR@10 falls from 79.11% toward 77.54%, the local split was leaking. Independently, train the re-ranker on easy negatives matched to the same mean cosine similarity as the claimed semi-hard negatives: if MRR@10 stays near 79%, the paper's mechanism is not the semi-hard difficulty band.

Watch

Extended reading notes

Core claim

The paper's central discovery is an engineering result: a two-stage retrieval and re-ranking pipeline, built from a fine-tuned Vietnamese Bi-Encoder and a pretrained Vietnamese Cross-Encoder, can place in the top three of a legal document retrieval competition while using far fewer parameters than ensemble approaches. The retriever is trained with MultipleNegativesRankingLoss and evaluated with a new metric, Exist@m, which measures whether at least one correct document appears among the top m candidates; the fine-tuned retriever reaches roughly 97% Exist@90, beating BM25. The re-ranker is trained with binary cross-entropy on positive pairs plus negatives mined from the retriever's top-90 candidates. Among hard, semi-hard, and easy negative mining, only semi-hard negatives give large and reliable gains, and increasing the number of such negatives from 2 to 10 steadily improves MRR@10, while hard negatives are unstable until the sample size becomes large.

Load-bearing premise

The local 79.11% result depends on the 90/10 split of processed training questions not letting the same question appear in both training and validation; if that fails, only the 77.54% private-test score is independent evidence.

Editorial extensions

If this is right

  • If the result holds, a single fine-tuned Bi-Encoder plus Cross-Encoder is enough for competitive specialized legal retrieval, so expensive ensembles are not a prerequisite for top performance.
  • Retrieval stages in a two-stage pipeline can be tuned for candidate coverage with Exist@m instead of MRR@10, which should make retriever development faster and more directly aligned with the re-ranker's needs.
  • Semi-hard negatives mined from the retriever's own candidates appear to be a reliable and cheap training signal for re-rankers, with more negatives per question consistently improving performance.
  • The 23% relative MRR@10 gain suggests that data preparation steps like replacing truncated answers with full legal documents, splitting multi-answer questions, and Vietnamese word segmentation are not incidental but carry substantial value.
  • A lightweight pipeline of this kind is more practical for real-time or large-scale legal information retrieval than ensemble systems, because only 90 pairs per query reach the expensive Cross-Encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's local 79.11% score would be more convincing if the 90/10 split grouped by question ID; the authors do not describe such grouping, so a reader should treat the private-test 77.54% as the stronger evidence.
  • Editorial extension: Table 3 shows the random top-90 negatives have mean cosine similarity 0.2072, close to the easy-negative distribution, so the paper's label 'semi-hard' may overstate how close these negatives are to true positives; the benefit could come more from sampling candidates on the retrieval frontier than from the difficulty band itself.
  • Editorial extension: a natural transfer test is to apply the same retrieval-plus-rerank recipe with retriever-mined semi-hard negatives to other low-resource domains or languages, where ensembles are less affordable.
  • Editorial extension: the authors attribute hard-negative instability to large gradients from near-positive samples; this could be tested directly by monitoring gradient norms or positive-negative score margins during training under each mining strategy.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper describes a two-stage pipeline for Vietnamese legal document retrieval: a fine-tuned Vietnamese Bi-Encoder retrieves top-90 candidates, and a PhoRanker Cross-Encoder re-ranks them. The authors propose a recall-style metric Exist@m for the retrieval stage and compare three negative-mining strategies for the re-ranker, reporting that semi-hard negatives (random candidates from the Bi-Encoder top-90) outperform hard and easy negatives, with 79.11% MRR@10 on a local eval split and 77.54% on the SoICT 2024 private test.

Significance. If the empirical comparisons are valid, the main contribution is a practical demonstration that, for a legal QA retrieval task, training a cross-encoder with negatives sampled from the retriever's candidate distribution gives large gains over corpus-wide easy negatives and over the hardest in-candidate negatives, with consistent results across three random seeds. The paper is clearly written and does not overstate the architecture's novelty relative to ensemble competitors. However, the local evaluation is compromised by the data-splitting pipeline (Section 5.2 followed by Section 5.4) and by an ambiguous statement that the Bi-Encoder was fine-tuned using both train_df and eval_df (Section 7.1); until those issues are fixed, the headline local numbers cannot be regarded as evidence for generalization, and the private-test score lacks the protocol details needed to carry the comparison alone.

major comments (4)
  1. [§7.1, Table 1] The sentence 'The Bi-Encoder is fine-tuned using the preprocessed train_df and eval_df datasets' is, if taken literally, direct training-data evaluation: the model would be evaluated on eval_df after training on it, invalidating all Exist@90 and MRR@10 values in Table 1 and the choice of the 11-epoch retriever. Please clarify whether this is a typo (i.e., trained on train_df and evaluated on eval_df) or re-run the experiments with a strict separation; this also affects the negative-mining step in Section 7.2 because the Bi-Encoder used to mine negatives would have seen eval_df.
  2. [§5.2, §5.4, Table 2] The local validation split is made after Section 5.2 expands multi-answer questions into separate rows, and Section 5.4 does not state that the 90/10 split groups by qid. Since roughly 10% of questions are linked to multiple cids, rows with the same qid can fall on both sides of the split, so eval_df is not a held-out set for those questions. Both the Bi-Encoder (MultipleNegativesRankingLoss) and the Cross-Encoder (BCEWithLogitsLoss) are trained on question–document pairs, so a validation row whose qid appeared in training can be memorized; this inflates the local 79.11% MRR@10 and the Exist@90 values in Tables 1–3, and it undermines the strategy-level comparison in Table 2, which is the paper's central empirical claim. Please re-split by qid (or report both row-level and qid-level splits) and re-run all local tables; alternatively, provide per-strategy results on the private test.
  3. [§7.2, Table 3] The operational definition of semi-hard negatives as uniform random draws from the Bi-Encoder top-90 after removing correct answers produces negatives with a mean cosine similarity of 0.2072 (79.44% below 0.5), which is much closer to the easy-negative distribution (mean 0.0008) than to the 'similar to the correct answers but still incorrect' definition stated in Section 3.3. Table 3 therefore does not demonstrate that the improvement in Table 2 is due to semi-hardness; it demonstrates only that negatives from the candidate set help. The paper should rename the strategy or provide an analysis that isolates hardness (for example, sampling from specified cosine-similarity bands), and the mechanistic claims in Sections 6.3 and 8 should be moderated accordingly.
  4. [Abstract / Conclusion / private test] The 77.54% MRR@10 private-test score is the only independent evidence for the headline result, but it is reported without any evaluation protocol: number of submitted runs, selection procedure (e.g., best-of-k), and whether the same negative-mining strategy was used. Because the local eval_df comparison is compromised by the qid split, this single number cannot independently validate the Table 2 strategy comparisons. Please add a subsection describing the private-test submission and, if possible, report private-test scores for at least the main negative-mining variants.
minor comments (7)
  1. [§6.2, Eq. (1)] Exist@m is recall@m for multi-document questions; the novelty claim in Section 1 should be calibrated by relating it to standard recall-oriented IR metrics.
  2. [§7.2, Table 3] The text says 'Using PhoRanker, we measured the initial cosine similarity', but PhoRanker is a Cross-Encoder and does not naturally produce query–document cosine similarities; specify the model and how the embeddings or scores were obtained.
  3. [§7.1, Table 1] The BM25Plus configuration with b=0 is described as 'tuned k1 and b' in the text, but the table lists many configurations; clarify whether these are a grid search and how the best configuration was selected to avoid selection-on-test concerns.
  4. [Figure 3] Figure 3 places the 0.9/0.1 split after the 'separate if multiple answers' step; annotate the figure to make the qid-grouping choice explicit, or redraw the pipeline to split before expansion.
  5. [§8, Conclusion] The conclusion states the model 'achieves same performance compared to competitors', but no table with other teams' private-test scores is provided; please add the competition leaderboard excerpt or precise ranking context.
  6. [§4.4] Section 4.4 says models with 1024-token capacity may be beneficial, but no such experiment is reported; either remove the speculation or add a sentence clarifying it is outside the scope.
  7. [General] There is no statement about code or data availability; for reproducibility of an empirical benchmark paper, please add one.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are measured on held-out and private data, and the self-referential metric and split concerns are validity issues rather than circular derivations.

full rationale

This is an empirical systems paper rather than a mathematical derivation, so the circularity burden is low. The Bi-Encoder and Cross-Encoder are trained on processed training rows and evaluated on a 90/10 split plus an external private test; the headline MRR@10 and the 23% relative improvement from semi-hard negatives are computed from trained model outputs, not forced by the definitions of the losses or metrics. Exist@m is a metric the authors define and then use for retriever selection, but reporting Exist@90 measured after training is not a prediction that reduces to its own input. The operationalization of semi-hard negatives as random draws from the Bi-Encoder top-90 is arguably a naming/mechanism concern, especially because Table 3 shows those negatives have mean cosine similarity 0.2072, close to easy negatives, but the Table 2 comparisons are empirical outcomes rather than identities. The most serious issue is the local evaluation split: Section 5.2 expands multi-answer questions into multiple rows before the Section 5.4 90/10 split, with no qid-level grouping described, so rows with the same qid could appear in both training and validation and inflate the 79.11% MRR@10. That is a data-split validity risk, not a circularity-by-construction, and the 77.54% private-test score remains independent external evidence. No load-bearing self-citation is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on dataset construction choices, such as the split, tokenization, and negative sampling, and on pre-trained checkpoints from prior work. There are no derived constants or physical entities; the main free parameters are the candidate count, negative count, and baseline tuning values.

free parameters (5)
  • top_m candidate count = 90
    Number of candidates retrieved by Bi-Encoder before reranking, chosen by design and directly defines Exist@90; affects the reranker's ceiling.
  • negative sample count n = 2, 5, 10
    Grid-searched in Section 7.2; the best MRR@10 occurs at n=10 for semi-hard and hard mining.
  • BM25 k1 and b = k1 in {0.8, 1.2, 2.0}, b in {0, 0.75, 1}
    Tuned for the BM25Plus baseline in Table 1; not central to the main claim but shows baseline sensitivity.
  • fine-tuning epochs = 7, 9, 11 for Bi-Encoder, 2 for Cross-Encoder
    Selected by validation performance; learning rate and batch size are also fixed by tuning.
  • random seeds = 28, 42, 2025
    Used for semi-hard and easy negative sampling; results are reported per seed and appear stable.
assumptions (4)
  • domain assumption Pre-trained Vietnamese Bi-Encoder and PhoRanker checkpoints are suitable starting points for legal retrieval fine-tuning.
    Section 6.1 relies on transfer learning from references [15] and [16] without verifying that their pretraining distribution covers legal terminology.
  • domain assumption Random negatives sampled from the Bi-Encoder top-90 candidates are semi-hard negatives.
    Section 6.3 and Table 3 show these negatives have mean cosine 0.2072, closer to easy (0.0008) than hard (0.6806), so the label is a modeling choice, not a demonstrated property.
  • domain assumption The 90/10 train/validation split is a fair estimate of generalization.
    Section 5.4 splits the processed train.csv after multi-answer expansion in Section 5.2, and no question-level grouping is described, so the same qid can appear in both splits.
  • domain assumption Pyvi tokenization preserves semantic content needed for retrieval.
    Section 5.3 applies Vietnamese word segmentation to all text before training and inference, and the paper does not ablate this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining." pith.science (2026). https://pith.science/paper/JXIUSSWI

@misc{pith2026250714619,
  author       = {Pith},
  title        = {Pith review of: Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Mining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JXIUSSWI}},
  note         = {Machine review of arXiv:2507.14619}
}
read the original abstract

Large Language Models (LLMs) face significant challenges in specialized domains like law, where precision and domain-specific knowledge are critical. This paper presents a streamlined two-stage framework consisting of Retrieval and Re-ranking to enhance legal document retrieval efficiency and accuracy. Our approach employs a fine-tuned Bi-Encoder for rapid candidate retrieval, followed by a Cross-Encoder for precise re-ranking, both optimized through strategic negative example mining. Key innovations include the introduction of the Exist@m metric to evaluate retrieval effectiveness and the use of semi-hard negatives to mitigate training bias, which significantly improved re-ranking performance. Evaluated on the SoICT Hackathon 2024 for Legal Document Retrieval, our team, 4Huiter, achieved a top-three position. While top-performing teams employed ensemble models and iterative self-training on large bge-m3 architectures, our lightweight, single-pass approach offered a competitive alternative with far fewer parameters. The framework demonstrates that optimized data processing, tailored loss functions, and balanced negative sampling are pivotal for building robust retrieval-augmented systems in legal contexts.

Figures

Figures reproduced from arXiv: 2507.14619 by the authors.

Figure 1
Figure 1. Dataset statistics: token-length profile (left) and document-ID usage per question [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Distribution of legal documents by token length [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Overview of the data processing pipeline [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Retrieve and re-rank pipeline structure Our implementation follows this general structure, but with careful consideration of model selection, training bias, loss function choice, and evaluation metrics tailored to the legal document retrieval task [PITH_FULL_IMAGE:fig…
Figure 5
Figure 5. Figure 5: Negative example mining using a fine-tuned Bi-Encoder [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: End-to-end fine-tuning and evaluation process [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Inference process for a legal question 7 Experimental Setup and Evaluation Strategy This section presents the experiments conducted to evaluate the effectiveness of our retrieval framework. We focus on both components of the pipeline: the initial retrieval stage using …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages

  1. [1]

    Retrieval-Augmented Gen- eration for Knowledge-Intensive NLP Tasks

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K ¨uttler, Mike Lewis, Wen-tau Yih, Tim Rockt¨aschel, Sebastian Riedel, and Douwe Kiela. “Retrieval-Augmented Gen- eration for Knowledge-Intensive NLP Tasks”. In: Advances in Neural Informa- tion Processing Systems 33: Annual Conference on Neural ...

  2. [2]

    A statistical interpretation of term specificity and its appli- cation in retrieval

    Karen Sp ¨arck Jones. “A statistical interpretation of term specificity and its appli- cation in retrieval”. In: J. Documentation 60.5 (2004), pp. 493–502

  3. [3]

    Some Simple Effective Approxima- tions to the 2-Poisson Model for Probabilistic Weighted Retrieval

    Stephen E. Robertson and Steve Walker. “Some Simple Effective Approxima- tions to the 2-Poisson Model for Probabilistic Weighted Retrieval”. In: Proceed- ings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval. Dublin, Ireland, 3-6 July 1994 (Special Issue of the SIGIR Forum) . Ed. by W . Bruce Croft...

  4. [4]

    Sentence-BERT: Sentence Embeddings us- ing Siamese BERT-Networks

    Nils Reimers and Iryna Gurevych. “Sentence-BERT: Sentence Embeddings us- ing Siamese BERT-Networks”. In: Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Ed. by Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan. Hong Kong, ...

  5. [5]

    Universal Sentence Encoder for English

    Daniel Cer, Yinfei Y ang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. “Universal Sentence Encoder for English”. In: Pro- ceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Ed. by Eduardo Blan...

  6. [6]

    Legal Document Retrieval using Document Vector Embeddings and Deep Learning

    Keet Sugathadasa, Buddhi Ayesha, Nisansa de Silva, Amal Shehan Perera, Vin- dula Jayawardana, Dimuthu Lakmal, and Madhavi Perera. “Legal Document Retrieval using Document Vector Embeddings and Deep Learning”. In: CoRR abs/1805.10685 (2018). arXiv: 1805.10685

  7. [7]

    Attentive deep neural networks for legal doc- ument retrieval

    Ha-Thanh Nguyen, Manh-Kien Phi, Xuan-Bach Ngo, Vu D. Tran, Le-Minh Nguyen, and Minh-Phuong Tu. “Attentive deep neural networks for legal doc- ument retrieval”. In: Artif. Intell. Law 32.1 (2024), pp. 57–86

  8. [8]

    Enhancing Legal Document Retrieval: A Multi-Phase Approach with Large Language Models

    Hai-Long Nguyen, Duc-Minh Nguyen, Tan-Minh Nguyen, Ha-Thanh Nguyen, Thi-Hai-Y en Vuong, and Ken Satoh. “Enhancing Legal Document Retrieval: A Multi-Phase Approach with Large Language Models”. In: CoRR abs/2403.18093 (2024). arXiv: 2403.18093

Show all 16 references
  1. [9]

    Learning Dense Representa- tions for Entity Retrieval

    Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. “Learning Dense Representa- tions for Entity Retrieval”. In: Proceedings of the 23rd Conference on Compu- tational Natural Language Learning (CoNLL) . Ed. by ...

  2. [10]

    Ef- ficient Natural Language Response Suggestion for Smart Reply

    Matthew L. Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, Lás- zló Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. “Ef- ficient Natural Language Response Suggestion for Smart Reply”. In: CoRR abs/1705.00652 (2017). arXiv: 1705.00652

  3. [11]

    Multi-Stage Document Ranking with BERT

    Rodrigo Nogueira, Wei Y ang, Kyunghyun Cho, and Jimmy Lin. “Multi-Stage Document Ranking with BERT”. In: CoRR abs/1910.14424 (2019). arXiv: 1910.14424

  4. [12]

    In De- fense of Cross-Encoders for Zero-Shot Retrieval

    Guilherme Rosa, Luiz Henrique Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Marzieh Fadaee, Roberto A. Lotufo, and Rodrigo Nogueira. “In De- fense of Cross-Encoders for Zero-Shot Retrieval”. In: CoRR abs/2212.06121 (2022). arXiv: 2212.06121

  5. [13]

    T wente-BMS-NLP at PerspectiveArg 2024: Combining Bi-Encoder and Cross-Encoder for Argument Retrieval

    Leixin Zhang and Daniel Braun. “T wente-BMS-NLP at PerspectiveArg 2024: Combining Bi-Encoder and Cross-Encoder for Argument Retrieval”. In: Pro- ceedings of the 11th Workshop on Argument Mining (ArgMining 2024) . Ed. by Y amen Ajjour, Roy Bar-Haim, Roxanne El Baff, Zhexiong Li...

  6. [14]

    ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting

    Steven H. Wang, Maksim Zubkov, Kexin Fan, Sarah Harrell, Yuyang Sun, Wei Chen, Andreas Plesner, and Roger Wattenhofer. “ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting”. In:CoRR abs/2501.06582 (2025). arXiv: 2501.06582

  7. [15]

    Towards Comprehensive Viet- namese Retrieval-Augmented Generation and Large Language Models

    Nguyen Quang Duc, Le Hai Son, Nguyen Duc Nhan, Nguyen Dich Nhat Minh, Le Thanh Huong, and Dinh Viet Sang. “Towards Comprehensive Viet- namese Retrieval-Augmented Generation and Large Language Models”. In: CoRR abs/2403.01616 (2024). arXiv: 2403.01616

  8. [16]

    PhoRanker: A Cross-encoder Model for Vietnamese Text Rank- ing

    Dai Nguyen Ba. PhoRanker: A Cross-encoder Model for Vietnamese Text Rank- ing. https://huggingface.co/itdainb/PhoRanker. 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.