{"id":"0d4d907a-0538-4425-959d-9f69945a4e43","arxiv_id":"2412.07462","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors extend the French BSARD legal retrieval dataset to Dutch (bBSARD) and benchmark retrieval models, showing small fine-tuned language-specific models can outperform zero-shot proprietary embeddings.","lead":"This paper introduces bBSARD, a bilingual French-Dutch dataset for Belgian statutory article retrieval, built by aligning 22,417 legal articles and translating 1,108 legal questions into Dutch. It then benchmarks lexical, zero-shot dense, and fine-tuned retrieval models, finding that BM25 stays competitive and small fine-tuned language-specific models can match or beat proprietary zero-shot APIs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Dutch benchmark's validity depends on 1,108 GPT-4o-translated questions, yet only 100 were human-checked (2% legally inaccurate); systemic errors in the remaining 91% could corrupt transferred relevance labels and skew all Dutch retrieval results.","rationale":"The central claim is that bBSARD is a valid bilingual resource and a reliable Dutch retrieval benchmark. The article corpus (parallel French/Dutch, 99% alignment) is well constructed and not the weak point. The weak point is the Dutch question set: 1,108 machine translations with only 100 human-checked, and the check itself found 2% legally inaccurate. Because the gold labels are transferred from the French questions without independent Dutch annotation, any mistranslation that changes legal meaning invalidates the labels. This is load-bearing because the entire Dutch evaluation, including the headline result that fine-tuned small models match proprietary zero-shot models, is measured against these labels. The concern is not fatal—it is fixable by full human review—but it is not currently addressed. The reader identified the same assumption, and we agree. Thus the verdict remains CONDITIONAL: accept once translation quality is verified and, ideally, chunk-score aggregation is documented. We found no other issue as central: the fine-tuning differences between models are large enough to support the qualitative claims, and the dataset's limitations (coverage, version) are honestly stated.","tokens_in":14324,"tokens_out":5959,"duration_ms":58934,"concrete_test":"Recruit two independent Dutch legal experts to review all 1,108 translated questions (or a pre-registered sample of at least 300) against the original French questions, judging whether the legal meaning is preserved and whether the BSARD relevance labels remain correct. Then re-run the Dutch retrieval experiments (Tables 1 and 4) using only the verified high-quality questions. If the relative model rankings and headline conclusions (e.g., fine-tuned RobBERT beating voyage-3) are unchanged, the translation concern is resolved; if rankings shift materially, the benchmark requires corrected translations or a reduced test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 describes translating all 1,108 BSARD questions with GPT-4o and checking a random sample of 100, finding 2% legally inaccurate and 6% minor semantic/grammatical/lexical issues. The Dutch relevance labels are simply inherited from the French BSARD labels on the assumption that translation preserves legal meaning. With only 9% of translations verified, the error rate and types of errors in the unchecked 91% are unknown. If GPT-4o made systematic legal-terminology errors (e.g., confusing 'usufruit' with 'gebruiksrecht', or procedural terms), the Dutch test set would contain mislabeled queries. This directly undermines the paper's central claim that bBSARD 'offers a much-needed retrieval benchmark for the Dutch language': both fine-tuning (training on noisy queries) and evaluation (ranking against wrong gold articles) would be distorted. The paper's own cross-language comparison (Section 5.3) concedes that 'potential translation issues ... particularly affect lexical models,' so the observed French-Dutch gaps may partly reflect translation artifacts rather than genuine model/language differences. Because the bilingual resource's usefulness as a benchmark hinges on this untested assumption, the Dutch-side results are not yet reliable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces bBSARD, a Dutch extension of the French BSARD statutory article retrieval dataset. The authors scrape parallel French and Dutch Belgian legislation from the Justel portal (22,417 aligned articles), translate the 1,108 BSARD questions into Dutch with GPT-4o, and human-check 100 of the translations. They then benchmark lexical models (TF-IDF, BM25), zero-shot dense models (open and proprietary), and fine-tuned small language models (RobBERT, Tik-to-Tok, CamemBERT, FlauBERT, XLM-R) on both language subsets. The reported findings are that BM25 remains a strong baseline, proprietary embedding models lead the zero-shot setting, and fine-tuned language-specific small models can match or surpass those proprietary models; French results are on average higher than Dutch results. The dataset and evaluation code are publicly released.","tokens_in":14543,"tokens_out":8111,"duration_ms":79054,"significance":"If the resource is valid, bBSARD fills a genuine gap: there is no public Dutch statutory article retrieval benchmark with expert relevance labels, and the parallel French--Dutch corpus enables cross-lingual retrieval studies. The evaluation is broad, covering 19 models across lexical, zero-shot dense, and fine-tuned settings, and the public release of data and code is a concrete community asset. The paper is also transparent about corpus coverage and about the fact that the Dutch question set is machine-translated. However, the validity of the Dutch benchmark hinges on the assumption that the translated questions preserve the French relevance labels; the evidence for that assumption is currently thin, and a few reproducibility details are missing. These issues are fixable and do not appear to require a change in the overall approach, but they need to be addressed before the resource can be used with confidence.","major_comments":[{"comment":"The Dutch side of bBSARD inherits BSARD relevance labels solely through translation of the 1,108 questions, but only 100 translations (9%) were human-checked. The check already found 2% legally inaccurate and 6% minor semantic/grammatical/lexical issues, and the error rate and error types in the remaining 91% are unknown. If the unverified translations contain systematic legal-terminology errors, both fine-tuning (noisy training queries) and evaluation (wrong gold articles) on the Dutch subset would be corrupted. The paper itself notes in Section 5.3 that translation issues may particularly affect lexical models, and the Conclusions state that future work will 'improve bBSARD's quality by manually checking/correcting all translated questions,' which confirms that the current release is not fully verified. Please either verify and correct all question translations before release, or provide additional systematic evidence (e.g., full back-translation with legal review, error analysis by legal subdomain, inter-annotator agreement) that the inherited labels remain valid.","section":"Section 3.2"},{"comment":"The paper does not specify how chunk-level scores are aggregated into article-level scores for models with a 512-token input limit. Section 4.1.2 says that text is divided into overlapping chunks of 200 tokens with 20-token overlap, but it never states whether the article score is the maximum, mean, or some other aggregate of the chunk scores. Since all dense-model results in Tables 1 and 2 depend on this choice, the benchmark is not reproducible without this information. Please state the exact aggregation rule and, ideally, release the scoring script.","section":"Section 4.1.2"},{"comment":"The claim that 'zero-shot dense models only fully outperforming [BM25] starting from 300 million parameters' is contradicted by Table 1: mE5base (278M parameters) exceeds BM25 on all seven reported metrics (e.g., R@100 50.14 vs. 40.19, MAP@100 16.47 vs. 16.07, MRR@100 25.64 vs. 22.63). The threshold should be corrected or the claim qualified (for example, 'clearly' or 'consistently'), and the analogous statement for the French subset should be rechecked against Table 2.","section":"Section 5.1 / Table 1"},{"comment":"The Conclusions report an experimental observation that XLM-Roberta performs better in Dutch when fine-tuned for 50+50 epochs on French and Dutch data than when fine-tuned for 100 epochs on Dutch data. No details of this experiment appear in Section 4 or in the results tables: the data-mixing procedure, hyperparameters, and metric values are all unspecified. Since this is a non-trivial cross-lingual training claim, it should either be presented with full experimental details and results in a table or removed.","section":"Section 6"}],"minor_comments":[{"comment":"Comparative statements such as 'FlauBERTbase beating voyage-3 in 4 out of 7 metrics' are based on a single evaluation run on a 222-question test set. Please add confidence intervals, bootstrap estimates, or multiple-seed runs so that small differences are not over-interpreted.","section":"Section 5.2 / Table 2"},{"comment":"The translation check was performed by a single 'native speaker'; it is not stated whether this annotator has legal expertise. For legal terminology, legal-expert annotation or at least a second annotator would strengthen the quality claim.","section":"Section 3.2"},{"comment":"The alignment pipeline is described as using 'ChatGPT queries' without specifying the model version or prompt. Please provide the exact model and protocol so that the alignment process is reproducible.","section":"Appendix A"},{"comment":"The bBSARD French article counts differ slightly from the original BSARD counts for several codes (e.g., Electoral Code 218 vs. 217, Code of Various Rights and Taxes 191 vs. 189). A sentence explaining whether these differences affect comparability with the original BSARD results would be helpful.","section":"Table 3"},{"comment":"There is a typo: 'obtaining any comprehensive legal information or advise' should read 'advice.'","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"This is a useful dataset paper, and the evaluation splits are clean: fine-tuned models are trained on the official training split and tested on held-out questions, so there is no circularity. The main risk is the Dutch question translation quality: since the benchmark's central value is the Dutch resource, the 9% verification rate is too thin. The missing chunk-aggregation rule and the incorrect BM25 threshold in Section 5.1 are easily fixable. I would not reject the paper; it should be given the opportunity to strengthen the translation-validation evidence and to add the missing reproducibility details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: bBSARD is a legitimate new resource. It extends BSARD to Dutch with parallel articles aligned at 99% and translated questions, plus a broad benchmark of lexical, zero-shot dense, and fine-tuned models on both languages. The dataset and code are public, which is real value for legal NLP. The French-Dutch parallel structure is the genuinely new thing, and the authors use it for a cross-language comparison rather than just dumping a dataset.\n\nWhat the paper does well: the article alignment is careful—they control for enforcement dates, handle the missing 216 articles, and document the per-code breakdown. The benchmarking is broad (22 models), and the fine-tuned small models beating proprietary zero-shot ones is a concrete, reproducible finding. The paper is honest about inherited limitations.\n\nSoft spots, in order of weight. First, the Dutch question translations: only 100 of 1,108 were human-checked, and that sample already showed 2% legally inaccurate and 6% minor issues. The relevance labels are inherited from French, so systematic legal-terminology errors in the unchecked 91% would directly corrupt both fine-tuning and evaluation on the Dutch side. The authors acknowledge potential translation issues in the cross-language discussion but don't quantify the risk. This is the load-bearing weakness of the Dutch benchmark claim. Second, there are no confidence intervals or repeated runs; several reported differences between models are small, and we can't tell if they're noise. Third, the chunk-score aggregation for long articles is not described—how chunk scores become article scores matters for the dense models and is easy to document. These are fixable, not fatal.\n\nI read the cross-language gap (French better on average) as cautiously reported; the authors note the data-availability hypothesis and ask for more analysis. That's fair.\n\nWho this is for: anyone working on multilingual legal retrieval or Dutch retrieval generally. It deserves a serious referee; the dataset contribution outweighs the benchmark novelty. I'd want the translation validation expanded and the experimental details filled in before accepting, but the resource is worth engaging now.","headline":"A useful bilingual legal retrieval benchmark with a real translation-validation gap; worth engaging, but the Dutch-side numbers need more verification.","tokens_in":15085,"tokens_out":1675,"would_cite":false,"duration_ms":16057,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents bBSARD, a parallel French–Dutch statutory article retrieval benchmark for Belgian law, and shows that fine-tuning small language-specific models matches or beats proprietary zero-shot embedding models.","keywords":["legal information retrieval","statutory article retrieval","multilingual retrieval","Dutch language","French language","parallel corpus","dense retrieval","BM25"],"falsifier":"Have two qualified legal translators independently translate all 1,108 questions from French to Dutch and compare against GPT-4o's translations, then re-run the benchmark on the corrected translations; if the corrected translations change which articles are relevant, the Dutch-side results and model rankings in the paper are not reliable.","tokens_in":14117,"feed_emoji":"⚖️","tokens_out":12267,"duration_ms":99861,"temperature":0.7,"pith_summary":"The paper introduces bBSARD, a bilingual extension of the French BSARD legal retrieval dataset, adding Dutch translations of Belgian statutory articles and of the dataset's 1,108 legal questions. It argues that bBSARD fills a gap by providing a public retrieval benchmark for Dutch statutory law, enabling reliable evaluation of Dutch retrieval models in the legal domain. On this benchmark, the paper reports that BM25 remains a competitive baseline against many zero-shot dense models, that proprietary embedding APIs lead the zero-shot setting, and that fine-tuning small language-specific models such as RobBERT-2023 and FlauBERT can match or surpass those proprietary results. The paper also documents a consistent performance advantage for French over Dutch, which it attributes tentatively to differences in pre-training data availability. If the benchmark is sound, it gives the community a reproducible testbed for improving Dutch legal search and for studying cross-lingual transfer between French and Dutch.","feed_headline":"Small fine-tuned models outrank big APIs on Dutch legal retrieval","feed_subtitle":"New parallel French–Dutch benchmark for Belgian law shows fine-tuning rivals commercial embeddings.","key_machinery":"The central object is the bBSARD dataset itself: a parallel corpus of 22,417 Belgian statutory articles aligned in French and Dutch by law code and article number, paired with 1,108 legal questions (886 train, 222 test) whose Dutch translations were produced by GPT-4o with the original articles as context, and whose relevance labels are inherited from BSARD's expert annotations. The mechanism that produces the paper's headline result is the Siamese bi-encoder fine-tuning protocol—queries and articles encoded by the same transformer, trained with contrastive loss, temperature 0.05, and in-batch negatives—which lets small language-specific models such as RobBERT-2023 and FlauBERT surpass much larger zero-shot models on ranking metrics.","core_discovery":"The central claim is that bBSARD is a valid benchmark for statutory article retrieval in Dutch and French, constructed by scraping and aligning 22,417 parallel Belgian law articles (99% of the BSARD corpus) from the Justel database and translating BSARD's 1,108 expert-labeled legal questions from French to Dutch with GPT-4o at temperature 0, followed by human inspection of a 100-question sample. The paper's benchmarking over lexical models (TF-IDF, BM25), context-independent embeddings, zero-shot dense models (open and proprietary), and fine-tuned small foundation models leads to three findings: (1) BM25 is competitive with zero-shot dense models below 300M parameters; (2) proprietary models (voyage-3, text-embedding-3-large) are the best zero-shot performers; and (3) fine-tuned small language-specific models—RobBERT-2023 for Dutch and FlauBERT for French—match or exceed those proprietary models on ranking metrics. The paper also finds that French retrieval consistently outperforms Dutch across most model types, and observes preliminary evidence that training on combined French and Dutch data improves a multilingual model's Dutch performance.","pith_inferences":["If the unverified 91% of Dutch translations contain systematic legal errors at a rate similar to the 100-question sample (2% legally inaccurate, 6% minor), then the Dutch subset's relevance labels are noisy, and any conclusions about the exact ordering of models on Dutch should be read with caution until the translations are fully corrected.","The parallel nature of bBSARD could also support adversarial robustness studies—for example, testing whether a model trained only on French articles can retrieve Dutch articles via cross-lingual transfer, which the paper only begins to explore.","A natural next experiment would be to add generative or LLM-based re-rankers to the bBSARD leaderboard; the paper's benchmark currently only covers bi-encoders and lexical models, leaving out a class of methods now common in retrieval pipelines.","The French-over-Dutch gap, if driven by pre-training data size, implies that investing in Dutch pre-training corpora is likely to narrow the gap; this is a testable prediction that could be checked by comparing models pre-trained on different Dutch data volumes."],"forward_implications":["Dutch legal information retrieval now has a public, reproducible benchmark with expert relevance labels, so future Dutch retrieval models can be evaluated on a common test set.","Fine-tuning small language-specific models offers a cost-effective alternative to proprietary embedding APIs for legal retrieval, achieving higher MAP, MRR, and nDCG on both Dutch and French.","BM25 should be kept as a standard baseline in future legal-retrieval evaluations, since it outperforms many dense models in the zero-shot setting.","The parallel French–Dutch structure enables cross-lingual transfer experiments; the paper's observation that XLM-RoBERTa improves in Dutch when trained on mixed-language data suggests bilingual training can benefit the lower-resource language.","The consistent French-over-Dutch performance gap points to pre-training data volume as a likely factor, motivating investment in Dutch corpora and Dutch-specific models."],"supporting_citations":[{"why":"Provides the French BSARD dataset, the 1,108 expert-labeled legal questions, and the fine-tuning codebase that bBSARD extends with Dutch articles and questions.","marker":"Louis and Spanakis (2022)"},{"why":"Defines the Siamese bi-encoder architecture used to fine-tune small language-specific models for retrieval.","marker":"Reimers and Gurevych (2019)"},{"why":"Supports the temperature-0 setting for GPT-4o translation, which the paper relies on to produce the Dutch questions.","marker":"Peng et al. (2023)"},{"why":"Supplies the in-batch negative training objective used in the contrastive fine-tuning of the small models.","marker":"Karpukhin et al. (2020)"},{"why":"Introduces RobBERT-2023, the Dutch language model whose fine-tuned version is the top performer on the Dutch subset.","marker":"Delobelle and Remy (2024)"},{"why":"Provides voyage-3, the proprietary embedding model that is the strongest zero-shot baseline on the French subset and a close baseline on Dutch.","marker":"VoyageAI (2024)"}],"fun_headline_variants":["Small fine-tuned models beat big APIs on Dutch legal retrieval","bBSARD benchmark shows BM25 still strong for legal search","Open fine-tuned models match proprietary on Dutch law retrieval","Dutch legal retrieval: fine-tuning small models rivals commercial APIs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Dutch evaluation rests on the assumption that GPT-4o's translations of the 1,108 legal questions preserve the legal meaning closely enough that BSARD's expert relevance labels still apply to the Dutch corpus; only 100 translations were human-checked, with 2% legally inaccurate and 6% minor issues, leaving 91% unverified.","fun_headline_variants_meta":{"raw":{"variants":["Small fine-tuned models beat big APIs on Dutch legal retrieval","bBSARD benchmark shows BM25 still strong for legal search","Open fine-tuned models match proprietary on Dutch law retrieval","Dutch legal retrieval: fine-tuning small models rivals commercial APIs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000116,"raw_usage":{"total_tokens":1075,"prompt_tokens":948,"completion_tokens":127,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":58}},"tokens_in":564,"tokens_out":127,"duration_ms":2054,"temperature":1.0,"reasoning_tokens":58,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:48:53.334945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two qualified legal translators independently translate all 1,108 questions from French to Dutch and compare against GPT-4o's translations, then re-run the benchmark on the corrected translations; if the corrected translations change which articles are relevant, the Dutch-side results and model rankings in the paper are not reliable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the French BSARD dataset, the 1,108 expert-labeled legal questions, and the fine-tuning codebase that bBSARD extends with Dutch articles and questions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces RobBERT-2023, the Dutch language model whose fine-tuned version is the top performer on the Dutch subset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides voyage-3, the proprietary embedding model that is the strongest zero-shot baseline on the French subset and a close baseline on Dutch."}],"review_version":1}