{"id":"fefe30cb-5f2a-4f52-ad5e-695f1d291788","arxiv_id":"2412.00657","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Llama 3 generated 507,152 synthetic Vietnamese legal queries; using them for CoT-MAE pre-training and contrastive fine-tuning improved bi-encoder and ColBERT retrieval scores on TVPL and Legal Zalo 21.","lead":"The authors used Llama 3 to write more than 500,000 practice questions for Vietnamese legal texts, then trained search models on that synthetic data. The models beat existing Vietnamese retrieval baselines on their new TVPL benchmark, but the experiment does not cleanly isolate how much of the gain comes from the synthetic questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The TVPL benchmark is built from the same thuvienphapluat.vn source as the synthetic query-generation passages; without an overlap analysis, the headline gains in Table 3 may reflect training/evaluation leakage rather than a generalizable synthetic-data benefit.","rationale":"Read in good faith, the paper's contributions are the synthetic dataset, the TVPL benchmark, and the training recipe. The strongest claim is attribution of the Table 3 gains to synthetic data. The most load-bearing condition for that claim is that TVPL is a genuinely held-out evaluation relative to the synthetic-data pipeline. The paper itself states that both the synthetic generation passages and the TVPL benchmark come from thuvienphapluat.vn, which places the burden on the authors to demonstrate disjointness. The concern is not that the authors are dishonest; it is that the reported numbers may not mean what the abstract says. The proposed overlap audit is cheap and decisive: if passage or query overlap is high, the TVPL column of Table 3 is invalid for the central claim; if overlap is low, the remaining ablation gap (no model trained without synthetic queries) still prevents clean attribution, so the verdict should remain CONDITIONAL rather than ACCEPT. The out-of-domain Zalo QA 19 results provide some independent evidence of generalization, but the in-domain benchmark is where the headline numbers live. The reader's weakest_assumption is the same train/evaluation source overlap, so agreement is 'agree'. The paper lacks formal verification, but that is not unusual for applied IR and is not itself a defect. The verdict stays UNCHANGED because the reader already marked the paper CONDITIONAL with medium correctness risk, and the proposed tests would either confirm or remove that conditionality.","tokens_in":10671,"tokens_out":5575,"duration_ms":53605,"concrete_test":"Run a documented overlap audit: for each of the 143,261 synthetic-generation passages, record the source document and article identifiers, then compute exact and near-duplicate overlap (e.g., MinHash or normalized edit distance) with the 224,006 TVPL corpus passages and with the passages retrieved by the 10,000 TVPL test queries. Additionally, for each TVPL test query, find its nearest neighbor among the 507k synthetic training queries by BGE-M3 cosine similarity and report the fraction above a chosen threshold. If either overlap is non-negligible, the TVPL results in Table 3 are contaminated and the central claim is unsubstantiated for that benchmark; if overlap is negligible, re-run the same fine-tuning recipe with and without the synthetic component to isolate its contribution.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 507,152 Llama-3-generated queries produce 'strong improvement' depends on TVPL being a genuinely held-out benchmark. Section 3.2 generates synthetic queries from 143,261 passages scraped from thuvienphapluat.vn, and Section 4.1 constructs the TVPL corpus (224,006 passages) and its 10,000 test queries from legal QA articles on the same site. The paper never quantifies passage-level overlap between the synthetic-generation set and the TVPL corpus/test set, nor query-level similarity between synthetic queries and TVPL test queries. If a non-trivial fraction of TVPL test passages (or near-duplicates) were among the passages fed to Llama-3, the Query-as-Context CoT-MAE pretraining and the synthetic fine-tuning pairs would let the retriever memorize exact passage content, inflating MRR@10 74.61 and related numbers. Moreover, Table 2 lists TVPL as a fine-tuning dataset, and no model is trained on the same mixture with the synthetic component removed, so even without exact overlap the design cannot separate 'synthetic data helps' from 'in-domain supervised fine-tuning on TVPL-style data helps.' The out-of-domain Zalo QA 19 results are less exposed to this specific leakage, but they do not rescue the paper's headline attribution for TVPL.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a pipeline for improving Vietnamese legal passage retrieval: (1) collect legal documents from thuvienphapluat.vn and split them into 143,261 passages; (2) use Llama 3 70B with an aspect-guided prompt to generate over 620,000 synthetic queries, filtered by BGE-M3 top-40 retrieval and by excluding queries with deictic references, yielding 507,152 query-passage pairs; (3) pre-train PhoBERT-base-v2 with the Query-as-Context CoT-MAE objective; (4) fine-tune bi-encoder and ColBERT models with contrastive InfoNCE loss and BGE-M3-mined hard negatives on a mixture of MS-MARCO, SQuAD 2.0, Legal Zalo 21 (80%), the newly introduced TVPL dataset, and the synthetic queries. Evaluation on TVPL, Legal Zalo 21 (20%), and Vietnamese Wiki QA from Zalo AI Challenge 2019 shows that the proposed models outperform the listed sparse and dense baselines, with the best CoT-MAE ColBERT achieving MRR@10 of 74.61 on TVPL and 84.08 on Legal Zalo 21.","tokens_in":10929,"tokens_out":2547,"duration_ms":21258,"significance":"If the reported gains are attributable to the synthetic-data pipeline rather than to confounds, the work is a useful contribution to low-resource and Vietnamese legal retrieval: it provides a large synthetic query dataset, a new benchmark (TVPL), and a reproducible recipe combining LLM-generated queries with CoT-MAE pre-training and hard-negative contrastive fine-tuning. The authors also release the synthetic data and TVPL queries on Hugging Face, which is a concrete asset for the community. However, the central claim of 'strong improvement' depends on TVPL being a genuinely held-out benchmark and on a controlled comparison that isolates the synthetic-data component; the current manuscript does not provide either, so the significance is conditional on the results of the requested analyses.","major_comments":[{"comment":"The TVPL benchmark may not be held out from the synthetic-data generation pipeline. Section 3.2 states that synthetic queries are generated from 143,261 passages scraped from thuvienphapluat.vn, and Section 4.1 states that the TVPL corpus (224,006 passages) and its 10,000 test queries are also sourced from that site's legal QA articles. The paper does not report any passage-level overlap analysis between the synthetic-generation set and the TVPL corpus/test set, nor any query-similarity analysis between synthetic queries and TVPL test queries. If a substantial fraction of TVPL test passages or near-duplicates were among the passages fed to Llama 3, the headline gains in Table 3 (e.g., CoT-MAE ColBERT MRR@10 74.61 on TVPL) could reflect memorization of passage content rather than a generalizable synthetic-data benefit. Please add a quantitative overlap analysis, ideally with exact-match and near-duplicate (e.g., n-gram or embedding-similarity) thresholds, and report the overlap rate for both the TVPL corpus and the TVPL test set.","section":"§3.2, §4.1, Table 3"},{"comment":"The baseline comparisons in Table 3 are not controlled for fine-tuning data and protocol. The dense baselines (vietnamese-sbert, vietnamese-bi-encoder, mE5base, BGE-M3) are used as pre-trained off-the-shelf retrievers, whereas the proposed models are fine-tuned on a large in-domain mixture that includes TVPL training queries and 80% of Legal Zalo 21. Part of the gap is therefore expected from in-domain supervised fine-tuning, not from synthetic data. Moreover, the paper does not ablate the synthetic component from the fine-tuning mixture: no model is trained on the same mixture with the 507,152 synthetic pairs removed, so the contribution of synthetic data to fine-tuning is not isolated. The cleanest within-paper comparison (with vs. without CoT-MAE pre-training, both fine-tuned identically) shows small gains (Table 3: MRR@10 +0.32 for bi-encoder, +0.71 for ColBERT on TVPL), which do not by themselves establish the abstract's 'strong improvement.' Please add an ablation that trains the same bi-encoder and ColBERT models on exactly the same fine-tuning mixture without the synthetic queries, and report the delta.","section":"§4.3, Table 3, §3.5"},{"comment":"The out-of-domain results do not support the claim of generalizable improvement. In Table 4, CoT-MAE Bi-encoder (68.22 MRR@10) is numerically worse than Bi-encoder without pre-training (69.57), and the CoT-MAE ColBERT gain over ColBERT is 2.04 MRR@10 but still below BGE-M3 (72.38 vs. 76.69). The paper's statement that 'our models demonstrated improved performance on the out-of-domain dataset' is only partially true and is not quantified against the non-pre-trained models. Please either temper the claim or provide a statistical significance test and a direct discussion of the mixed out-of-domain results; as written, the out-of-domain evaluation undermines rather than supports the central attribution.","section":"§4.5, Table 4"},{"comment":"The same retriever, BGE-M3, is used in two roles that shape the training distribution and is also a baseline in Table 3: it filters synthetic queries (Section 3.3) and mines hard negatives for fine-tuning (Section 3.5). While this is not circular in the strict sense—final evaluation uses separate test queries—it means the training distribution is biased toward passages that BGE-M3 can retrieve, and the baseline comparison against BGE-M3 in Table 3 is not apples-to-apples with respect to negative mining. Please discuss this potential bias explicitly and, if feasible, report an ablation that uses hard negatives mined with the trained model itself or with BM25 to check sensitivity.","section":"§3.3, §3.5, Table 3"}],"minor_comments":[{"comment":"The mE5base row reports '72,76' with a comma as the decimal separator, while all other numbers in the table use periods; please standardize.","section":"Table 4"},{"comment":"The reference is titled 'Vienamese sbert' and the Hugging Face URL is informal; please correct the typo ('Vietnamese') and, if possible, include a version or accession date for the model.","section":"Reference [10]"},{"comment":"The description of fine-tuning says the bi-encoder was trained for 170,000 steps (5 epochs) and ColBERT for 290,000 steps (9 epochs), but the total number of training queries in the mixture is not reported; please state the per-epoch query count or the number of updates per dataset so a reader can assess whether the longer ColBERT schedule is a confound.","section":"§4.2"},{"comment":"The sentence 'Applying this method, we generated over 620,000 legal queries from 140,292 passages' uses a different passage count from the 143,261 quoted in §3.2; please reconcile the two numbers.","section":"§3.3"},{"comment":"The passage hit rates in Table 5 (8.26% for basic prompting at k=10 vs. 82.06% for aspect-guided prompting) are striking; please clarify whether the basic-prompt queries were also filtered with BGE-M3, because the filtering step in §3.3 could differentially remove low-quality queries from the two prompt variants.","section":"§5.1, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a retrieval or NLP venue, but the submission currently reads like a technical report rather than a fully controlled study. The two load-bearing issues are (i) possible train/eval overlap between TVPL and the synthetic-data generation source, and (ii) absence of an ablation that separates the synthetic-data contribution from in-domain fine-tuning. Both are fixable with additional analysis and experiments, so I do not recommend rejection. The authors should also be asked to temper the abstract's 'strong improvement' claim until the overlap and ablation results are in. The released datasets are a positive aspect worth acknowledging in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful part of this paper is the resources: 507k synthetic Vietnamese legal queries and the TVPL 10k-query test set, both released on HuggingFace. That is real, citable value for a low-resource language, and the pipeline is described clearly enough to reproduce.\n\nThe method itself is not new — LLM-generated queries, CoT-MAE/Query-as-Context pre-training, hard-negative mining, ColBERT fine-tuning are all published components. The paper's own contribution is the domain application and the measurements. The aspect-guided prompt analysis (Table 5) is a genuine, useful result: passage hit rate jumps from 8.26% to 82.06% at k=10. The compression comparison in Table 6 is also worth having.\n\nNow the soft spots, in order of importance. The central claim that synthetic data produces 'strong improvement' on TVPL is not isolated by any ablation. No model is fine-tuned on the same mixture without the synthetic component, so the gain could just be more in-domain supervised data. Second, the TVPL benchmark shares its source with the synthetic generation: sections 3.2 and 4.1 both use thuvienphapluat.vn, and the paper never measures passage overlap between the two, nor similarity between synthetic queries and TVPL test queries. That leaves a real leakage path. The baselines in Table 3 (except BM25) are not fine-tuned on the same in-domain data, so part of the gap is expected and not evidence for the synthetic data per se.\n\nThe other results are more mixed. The CoT-MAE pre-training adds only +0.32 MRR@10 for the bi-encoder and +0.71 for ColBERT on TVPL; on Zalo QA 19 the CoT-MAE bi-encoder actually loses a point. On the out-of-domain Zalo QA 19, their best ColBERT (72.38) trails BGE-M3 (76.69) and mE5base (72.76), which tempers the generalization claim.\n\nNone of this sinks the paper. The datasets are still useful, and the overlap and ablation concerns are addressable in revision. But the headline attribution should be reworded until the ablation is done.\n\nI'd send this to peer review: the resources alone justify referee time, and the methodological issues are fixable. For my own work I'd cite the datasets, not the method.","headline":"Useful new Vietnamese legal retrieval resources, but the headline gain from synthetic data is not isolated from in-domain supervision and a shared source with the test set.","tokens_in":11520,"tokens_out":1939,"would_cite":true,"duration_ms":17479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"500k synthetic queries lift Vietnamese legal retrieval to 74.61 MRR","keywords":["Information Retrieval","Vietnamese legal text","synthetic query generation","LLM-generated data","dense retrieval","ColBERT","Query-as-Context pre-training","contrastive fine-tuning"],"falsifier":"Compute exact and near-duplicate overlap between the 143,261 passages used to generate synthetic queries and the 224,006 TVPL corpus passages, and measure retrieval-based similarity between the 507,152 synthetic queries and the 10,000 TVPL test queries. If even a modest fraction of test passages or test queries are covered by the synthetic generation passages, the reported TVPL gains are explained by leakage rather than by synthetic data improving generalization.","tokens_in":10426,"feed_emoji":"⚖️","tokens_out":8263,"duration_ms":71333,"temperature":0.7,"pith_summary":"This paper argues that a large set of synthetic Vietnamese legal queries, generated by Llama 3 70B from real legal passages and filtered for quality, can stand in for the human-annotated training data that is missing for legal retrieval. The authors show that pre-training a PhoBERT (Vietnamese BERT) based retriever on these queries with a Query-as-Context masked auto-encoder objective, then fine-tuning with contrastive loss and hard negatives, produces large gains for both single-vector bi-encoders and multi-vector ColBERT models. On the new TVPL benchmark and on Legal Zalo 21, the best ColBERT model reaches MRR@10 (how high the correct passage ranks in the top ten results) of 74.61 and 84.08, outperforming the listed dense baselines. If correct, this gives a cheap, scalable path to building legal search and question-answering systems in Vietnamese and other low-resource legal domains.","feed_headline":"500k synthetic queries lift Vietnamese legal retrieval to 74.61 MRR","feed_subtitle":"Fine-tuning a 124M-parameter ColBERT on LLM-generated legal questions beats much larger multilingual models.","key_machinery":"The load-bearing mechanism is an aspect-guided synthetic query generator coupled with a Query-as-Context pre-training objective. The generator prompts Llama 3 to identify 1-5 distinct aspects covered by a legal passage and write one question per aspect; this step alone raises the BGE-M3 passage hit rate at k=10 from 8.26% to 82.06% compared with direct prompting. Pre-training applies the CoT-MAE loss, in which an encoder reconstructs the masked passage while a decoder reconstructs the generated query from its unmasked tokens conditioned on the passage, so the encoder learns to align query and passage representations before any supervised fine-tuning. Fine-tuning then uses the InfoNCE contrastive loss with BGE-M3-mined hard negatives and in-batch negatives, and the final deployed model applies ColBERT's 2-bit residual compression.","core_discovery":"The central discovery is that synthetic queries are not just a data-augmentation trick; they carry the training signal that would otherwise require tens of thousands of human-annotated legal QA pairs. Starting from 143,261 legal passages, the authors prompt Llama 3 to first list 1-5 aspects of a passage and then write one question for each aspect, producing 507,152 queries after BGE-M3-based filtering and removal of queries that explicitly reference the source passage. These pairs are used twice: to pre-train the encoder under the CoT-MAE Query-as-Context objective, and again as fine-tuning data alongside translated MS-MARCO and SQuAD 2.0 plus existing legal benchmarks. The result is consistent across settings: Query-as-Context pre-training improves over fine-tuning alone, ColBERT outperforms the bi-encoder, and the best ColBERT reaches MRR@10 of 74.61 on TVPL and 84.08 on Legal Zalo 21, ahead of the reported baselines. On the out-of-domain Zalo QA 19 benchmark the same model reaches MRR@10 of 72.38, close to much larger multilingual retrievers.","pith_inferences":["If the TVPL evaluation passages overlap the passages used for synthetic query generation, the in-domain gains would be partly a training/evaluation leakage artifact; the paper does not report any overlap check, so an external held-out legal corpus would be the clean test.","The aspect-first prompting insight (generate aspects, then a question per aspect) likely generalizes to other specialized domains such as medical or financial regulation, and could be tested by measuring hit-rate improvements on those corpora.","The top-40 BGE-M3 filter may bias the synthetic set toward queries that dense retrieval can already answer, potentially under-representing hard, long-tail legal questions; an unfiltered sample comparison would reveal how much this matters.","A pure-synthetic training run (without the translated MS-MARCO and SQuAD data) would isolate how much of the gain comes from synthetic legal data versus multilingual data; the paper's ablations currently bundle both."],"forward_implications":["The 507,152 synthetic query-passage pairs are released under CC BY 4.0, giving other Vietnamese NLP teams a ready-made retrieval training set.","Query-as-Context pre-training yields measurable gains over fine-tuning alone on in-domain benchmarks, suggesting the pre-training step will transfer to other low-resource legal languages.","Because 1-bit compressed ColBERT (647 MB) beats the 672 MB bi-encoder on MRR@10 and MAP@10 on TVPL, practitioners can get better accuracy without paying a storage penalty.","The improved out-of-domain Zalo QA 19 scores suggest the recipe transfers beyond legal text, despite being trained solely on legal data.","The aspect-guided prompting recipe and the BGE-M3 filtering step together form a reusable pipeline for generating synthetic retrieval data for domains where annotated data is scarce."],"supporting_citations":[{"why":"Supplies the Llama 3 70B model used to generate the 507,152 synthetic legal queries.","marker":"[2]"},{"why":"Provides BGE-M3, used both to filter low-quality synthetic queries and to mine hard negatives for fine-tuning.","marker":"[4]"},{"why":"Defines the ColBERT multi-vector late-interaction architecture that achieves the best results.","marker":"[11]"},{"why":"Contributes the MS MARCO passage ranking dataset, translated to Vietnamese, used in fine-tuning.","marker":"[17]"},{"why":"Establishes the vietnamese-bi-encoder baseline and the translation approach for using English datasets in Vietnamese.","marker":"[20]"},{"why":"Contributes SQuAD 2.0, translated to Vietnamese, used in fine-tuning.","marker":"[21]"},{"why":"Supplies ColBERT's residual compression scheme, evaluated for storage and accuracy in the footprint analysis.","marker":"[23]"},{"why":"Introduces Query-as-Context pre-training, the objective adapted with CoT-MAE for this work.","marker":"[26]"},{"why":"Motivates LLM-generated query-passage pairs and the distillation of large-model knowledge into smaller encoders.","marker":"[27]"},{"why":"Defines the CoT-MAE masked auto-encoder loss used during pre-training.","marker":"[29]"}],"fun_headline_variants":["LLM-generated queries boost Vietnamese legal retrieval to 74.61 MRR","Synthetic data powers Vietnamese legal search to top MRR","From 143k passages, LLM writes 500k legal queries for better retrieval","Vietnamese legal retrieval hits 74.61 MRR thanks to synthetic queries","Pre-training on synthetic queries beats larger models in legal retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result hinges on the TVPL evaluation being genuinely unseen, because the synthetic training queries and the TVPL test queries come from the same legal website and the paper never measures passage overlap or query similarity between them.","fun_headline_variants_meta":{"raw":{"variants":["LLM-generated queries boost Vietnamese legal retrieval to 74.61 MRR","Synthetic data powers Vietnamese legal search to top MRR","From 143k passages, LLM writes 500k legal queries for better retrieval","Vietnamese legal retrieval hits 74.61 MRR thanks to synthetic queries","Pre-training on synthetic queries beats larger models in legal retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000609,"raw_usage":{"total_tokens":2814,"prompt_tokens":903,"completion_tokens":1911,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":1817}},"tokens_in":519,"tokens_out":1911,"duration_ms":27600,"temperature":1.0,"reasoning_tokens":1817,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:08:56.223732+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute exact and near-duplicate overlap between the 143,261 passages used to generate synthetic queries and the 224,006 TVPL corpus passages, and measure retrieval-based similarity between the 507,152 synthetic queries and the 10,000 TVPL test queries. If even a modest fraction of test passages or test queries are covered by the synthetic generation passages, the reported TVPL gains are explained by leakage rather than by synthetic data improving generalization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Llama 3 70B model used to generate the 507,152 synthetic legal queries."},{"cited_title":"In: Ku, L.W., Martins, A., Srikumar, V","cited_arxiv_id":null,"evidence_quote":"Provides BGE-M3, used both to filter low-quality synthetic queries and to mine hard negatives for fine-tuning."},{"cited_title":"In: Proceedings of the 43rd Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval","cited_arxiv_id":null,"evidence_quote":"Defines the ColBERT multi-vector late-interaction architecture that achieves the best results."},{"cited_title":"In: Besold, T.R., Bordes, A., d’Avila Garcez, A.S., Wayne, G","cited_arxiv_id":null,"evidence_quote":"Contributes the MS MARCO passage ranking dataset, translated to Vietnamese, used in fine-tuning."},{"cited_title":"arXiv e-prints (Mar 2024)","cited_arxiv_id":null,"evidence_quote":"Establishes the vietnamese-bi-encoder baseline and the translation approach for using English datasets in Vietnamese."},{"cited_title":"In: Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers)","cited_arxiv_id":null,"evidence_quote":"Contributes SQuAD 2.0, translated to Vietnamese, used in fine-tuning."},{"cited_title":"In: Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022","cited_arxiv_id":null,"evidence_quote":"Supplies ColBERT's residual compression scheme, evaluated for storage and accuracy in the footprint analysis."},{"cited_title":"In: Bouamor, H., Pino, J., Bali, K","cited_arxiv_id":null,"evidence_quote":"Introduces Query-as-Context pre-training, the objective adapted with CoT-MAE for this work."},{"cited_title":"In: Ku, L.W., Martins, A., Srikumar, V","cited_arxiv_id":null,"evidence_quote":"Motivates LLM-generated query-passage pairs and the distillation of large-model knowledge into smaller encoders."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CoT-MAE masked auto-encoder loss used during pre-training."}],"review_version":1}