REVIEW 3 major objections 6 minor 17 references
A comparison of data filtering techniques for English-Polish LLM-based machine translation in the biomedical domain
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Data filtering by LASER sentence-embedding similarity lets an English-Polish biomedical translation model match full-corpus fine-tuning while training on only 60% of the data, and the paper recommends LASER as the most effective filter…
desk verdict A useful, honest empirical data point that overstates its LASER recommendation relative to single-seed BLEU gaps within noise. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is cosine similarity between language-agnostic sentence embeddings, used as a quality score for each English-Polish sentence pair. Three encoder families produce the embeddings: LASER learns shared cross-lingual sentence representations; MUSE aligns monolingual word embeddings into a shared space; LaBSE produces BERT-based bilingual sentence embeddings. The paper ranks all pairs in the medical corpus by this score and keeps the top 20% or 60%, then fine-tunes mBART50 on the surviving pairs. Random subsets of the same sizes and the full corpus serve as controls, so any BLEU difference can be attributed to the filter's selection rather than to dataset size.
What would settle it
Run the same filtering and fine-tuning protocol but evaluate on a second biomedical English-Polish test set, or apply bootstrap resampling to the Khresmoi scores, and check whether LASER-60% still matches or beats the full-corpus model; if the advantage disappears or reverses, the recommendation fails to generalize. Alternatively, take a human-annotated sample of medical sentence pairs from the source corpus and test whether the pairs ranked highest by LASER cosine similarity are actually judged the best training examples by bilingual medical translators; disagreement would falsify the premise that the filter scores quality.
Extended reading notes
Core claim
The authors claim that data filtering by cross-lingual sentence-embedding similarity is a viable substitute for full-corpus fine-tuning in domain-adapted machine translation. Concretely, keeping the 60% of English-Polish medical sentence pairs with the highest LASER cosine-similarity scores and fine-tuning mBART50 on them yields 17.411 BLEU on Khresmoi, above the full-corpus baseline of 17.402 and above the random-60% average of 17.234. At 20% retention, LASER (17.114) and MUSE (17.071) outperform random selection (16.801 average) but fall short of the full-corpus model. Bilingual-speaker assessment puts LASER-60% and the full-corpus model at the top for fluency and naturalness, while MUSE-20% loses medical terminology; LaBSE is not beneficial at either size despite correlating strongly with LASER scores ($r = 0.81$). The paper concludes that LASER is the most effective data filtering method for English-Polish biomedical MT.
Load-bearing premise
The whole comparison rests on the premise that a high cosine similarity between the English and Polish embeddings of a sentence pair marks it as good training data for medical translation; the paper does not independently verify that premise, and the final BLEU differences among methods are small and untested for statistical significance.
Editorial extensions
If this is right
- A LASER-filtered 60% subset can replace the full corpus for English-Polish biomedical fine-tuning with no measurable BLEU loss and noticeably lower training time.
- The gain from filtering is domain- and language-pair-specific: LaBSE's negative result shows a filter that works elsewhere can hurt here, so each new pair needs its own comparison.
- At 20% retention, embedding filters still beat random pruning and the unfiltered pretrained model, but cannot match full-corpus quality, indicating a practical floor on how much data can be cut.
- Automatic BLEU alone is not enough to pick a filter, since MUSE-20%'s BLEU looks acceptable while human inspection finds it garbles medical terms; the qualitative check changes the ranking.
Reading between the lines
- Editorial extension: the honest practical takeaway from the small BLEU gaps is compute savings, not quality improvement; a team wanting faster iterations could adopt LASER-60% and expect roughly unchanged quality at reduced cost.
- Because LaBSE and LASER scores correlate at $r = 0.81$ yet the resulting models differ, inspecting the low-agreement sentence pairs may reveal the specific selection signal that makes LASER effective.
- The same filtering pipeline is likely to transfer to other medium-resource language pairs, but the LaBSE failure is a warning that the transfer must be tested rather than assumed for each pair and domain.
- Combining cosine-similarity filtering with a domain-specificity measure, such as medical-term density, could remove off-topic sentences that similarity alone lets through.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares three multilingual-embedding-based data filtering methods (LASER, MUSE, LaBSE) against random downsampling and full-corpus training for fine-tuning mBART50 on English-to-Polish biomedical machine translation. The authors filter the UFAL Medical Corpus to 20% and 60% subsets, fine-tune mBART50 on each subset, and evaluate on the Khresmoi test set with SacreBLEU, supplemented by a small human quality assessment. The main reported result is that the LASER-filtered 60% model achieves BLEU 17.411, versus 17.402 for the full-corpus model and 17.234 for a random 60% subset, and the paper recommends LASER as the most effective filtering method for this language pair and domain.
Significance. If the reported hierarchy were statistically robust, this would be a practically useful contribution: it would provide concrete evidence that embedding-based filtering can reduce training data size without sacrificing, and possibly improving, translation quality in a moderately low-resource domain, and the authors make their code publicly available for reproducibility. The study also addresses a gap in the literature by targeting English-Polish, a pair for which previous filtering results may not transfer. However, the empirical basis for the central recommendation is narrow: the headline advantage over the full corpus is +0.009 BLEU, the filtered models are trained once with no variance estimate, no significance tests are reported, and the human evaluation is unblinded and performed by the authors. The paper is transparent about several of these limitations, which is commendable, but the strength of the conclusions still substantially exceeds what the evidence supports.
major comments (3)
- [Table 1, §5] The central result that LASER filtering outperforms the full corpus rests on a BLEU difference of +0.009 (Filtered-60% LASER: 17.411 vs. Base-all: 17.402). Meanwhile, the random 60% baseline, trained with three seeds, has a standard deviation of 0.05, and the random 20% baseline has a standard deviation of 0.13. The filtered models were trained only once, so no variance estimate is available for LASER, MUSE, or LaBSE. With seed noise of this magnitude, the observed ordering of methods could plausibly change under different fine-tuning seeds, and the hierarchy 'LASER > MUSE > Baseline > LaBSE' stated in §6 is not statistically established. A paired bootstrap or a small multi-seed comparison for at least the leading filtered condition is needed to support the recommendation.
- [§4, §5 (human evaluation)] The qualitative assessment was conducted by two of the authors, who are native Polish speakers, but the report does not state that they were blind to which model produced each translation, nor does it provide a quantitative protocol, a sample size, or inter-annotator agreement. The Results section nonetheless states that LASER-60% and Base-all produce 'the most accurate and well-sounding translations' (see example in Table 2), and the Discussion uses this to reinforce the LASER recommendation. The Limitations section correctly acknowledges the absence of a quantitative human evaluation by an expert, but the earlier unblinded assessment should not be presented as supporting evidence without a clear caveat.
- [§6 Discussion and Limitations] The recommendation 'we recommend LASER as the most effective data filtering method for LLM-based machine translation from English to Polish in the biomedical domain' is stated categorically, despite the acknowledged absence of significance testing, single-seed runs for filtered models, and a single 1,500-sentence test set. The Discussion should be rephrased to describe LASER as showing a consistent but statistically unverified advantage in this specific setup, and the possibility that the methods are indistinguishable from random selection (as found in prior work cited by the authors, e.g., Bane and Zaretskaya 2021) should be explicitly discussed.
minor comments (6)
- [§3] The refined dataset size is written as '700,00' and should be '700,000'.
- [§2] The sentence 'The latter indicates that filtering results might not be universally applicable across different pairs or topics of languages, which motivated our study design to specifically investigate the Polish-English translation' uses the reverse direction of the language pair investigated in the experiments; it should refer to English-Polish translation.
- [Table 3] The filtered subset sizes are reported as 426k and 158k for the 60% and 20% conditions, while §4 states approximately 420k and 150k pairs; please clarify whether the discrepancy is due to rounding, deduplication, or a different counting convention.
- [§6] The sentence 'LASER and MUSE achieved relatively lower validation BLEU scores (-0.288 and -0.331 respectively) compared to the baseline model (Base-all)' is confusing because the numbers are BLEU differences, not scores; rephrase to say that LASER-20% and MUSE-20% decrease BLEU by 0.288 and 0.331 relative to Base-all.
- [Table 2] The example showing MUSE-20% rendering 'Meningococcal Disease' as 'Choroba gruczołu krokowego' is a striking error, but a single example does not quantify translation quality; consider reporting how frequently such terminology errors occurred across the evaluated sentences.
- [§4.2] The abstract says translations were 'assessed by bilingual speakers,' but the experimental setup says two authors performed the assessment; please clarify the evaluation procedure and the number of sentences reviewed.
Circularity Check
No circularity: the filtering methods, fine-tuning procedure, and Khresmoi evaluation are independent, so the LASER recommendation is an empirical finding rather than a constructional or self-referential result.
full rationale
The paper's central claim is an empirical comparison of data filtering methods for English-Polish biomedical MT. The training data are filtered by cosine similarity scores from LASER, MUSE, and LaBSE embeddings; the resulting subsets are used to fine-tune mBART50; and the models are then evaluated on the Khresmoi test set, which the paper explicitly describes as an independent dataset: 'Evaluation was conducted on an independent dataset.' No parameter is fitted to Khresmoi, no filtering score is derived from the test set, and the paper does not define any filtering method in terms of the BLEU outcome it later predicts. The recommendation of LASER rests on observed BLEU scores and qualitative judgments, not on an identity between inputs and outputs. The limitations section candidly notes that significance testing and multiple train/dev splits were not performed, and the observed BLEU gaps are small, but these are statistical robustness concerns, not circularity: the evaluation is not forced by construction or by a self-citation chain. The authors cite prior external work for the filtering methods and models, and none of those citations substitutes for the empirical result reported here. Therefore no circular step is present, and the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Cosine similarity in LASER, MUSE, and LaBSE embedding spaces is a valid proxy for parallel sentence-pair quality.
- domain assumption Khresmoi is a high-quality independent test set and SacreBLEU on it reflects translation quality.
- domain assumption Training all models for exactly three epochs is a fair comparison setup.
Cite this review
Pith. "Pith review of A comparison of data filtering techniques for English-Polish LLM-based machine translation in the biomedical domain." pith.science (2026). https://pith.science/paper/RES2PA6K
@misc{pith2026250116533,
author = {Pith},
title = {Pith review of: A comparison of data filtering techniques for English-Polish LLM-based machine translation in the biomedical domain},
year = {2026},
howpublished = {\url{https://pith.science/paper/RES2PA6K}},
note = {Machine review of arXiv:2501.16533}
}
read the original abstract
Large Language Models (LLMs) have become state-of-the-art in Machine Translation (MT), often trained on massive bilingual parallel corpora scraped from the web, that contain low-quality entries and redundant information, leading to significant computational challenges. Various data filtering methods exist to reduce dataset sizes, but their effectiveness largely varies based on specific language pairs and domains. This paper evaluates the impact of commonly used data filtering techniques, such as LASER, MUSE, and LaBSE, on English-Polish translation within the biomedical domain. By filtering the UFAL Medical Corpus, we created varying dataset sizes to fine-tune the mBART50 model, which was then evaluated using the SacreBLEU metric on the Khresmoi dataset, having the quality of translations assessed by bilingual speakers. Our results show that both LASER and MUSE can significantly reduce dataset sizes while maintaining or even enhancing performance. We recommend the use of LASER, as it consistently outperforms the other methods and provides the most fluent and natural-sounding translations.
Reference graph
Works this paper leans on
-
[1]
Haluk A c ar c i c ek, Talha C olako g lu, P nar Ece Aktan Hatipo g lu, Chong Hsuan Huang, and Wei Peng. 2020. Filtering noisy parallel corpus using transformers with proxy task learning. In Proceedings of the Fifth Conference on Machine Translation, pages 940--946
work page 2020
-
[2]
Fred Bane, Celia Soler Uguet, Wiktor Stribi \.z ew, and Anna Zaretskaya. 2022. A comparison of data filtering methods for neural machine translation. In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 2: Users and Providers Track and Government Track), pages 313--325
work page 2022
-
[3]
Fred Bane and Anna Zaretskaya. 2021. Selecting the best data filtering method for NMT training. In Proceedings of Machine Translation Summit XVIII: Users and Providers Track, pages 89--97
work page 2021
-
[4]
Vishrav Chaudhary, Yuqing Tang, Francisco Guzm \'a n, Holger Schwenk, and Philipp Koehn. 2019. Low-resource corpus filtering using multilingual sentence embeddings. arXiv preprint arXiv:1906.08885
work page Pith review arXiv 2019
-
[5]
Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Herv \'e J \'e gou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087
arXiv 2017
-
[6]
Lei Cui, Dongdong Zhang, Shujie Liu, Mu Li, and Ming Zhou. 2013. Bilingual data cleaning for SMT using graph-based random walk. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 340--345
work page 2013
-
[7]
Ond r ej Du s ek, Jan Haji c , Jaroslava Hlav \'a c ov \'a , Jind r ich Libovick \'y , Pavel Pecina, Ale s Tamchyna, and Zde n ka Ure s ov \'a . 2017. http://hdl.handle.net/11234/1-2122 Khresmoi summary translation test data 2.0 . LINDAT / CLARIAH - CZ digital library at the Institute of Formal and Applied Linguistics ( \'U FAL ), Faculty of Mathematics a...
work page 2017
-
[8]
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. https://arxiv.org/abs/2007.01852 Language-agnostic BERT sentence embedding . Preprint, arXiv:2007.01852
arXiv 2022
Show all 17 references
-
[9]
Philipp Koehn, Huda Khayrallah, Kenneth Heafield, and Mikel L. Forcada. 2018. https://doi.org/10.18653/v1/W18-6453 Findings of the WMT 2018 shared task on parallel corpus filtering . In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 726--...
2018 doi
-
[10]
Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771
2018 arXiv
-
[11]
Holger Schwenk and Matthijs Douze. 2017. https://doi.org/10.18653/v1/W17-2619 Learning joint multilingual sentence representations with neural machine translation . In Proceedings of the 2nd Workshop on Representation Learning for NLP , pages 157--167, Vancouver, Canada. Assoc...
2017 doi
-
[12]
Kaveh Taghipour, Shahram Khadivi, and Jia Xu. 2011. Parallel corpus refinement as an outlier detection algorithm. In Proceedings of Machine Translation Summit XIII: Papers
2011
-
[13]
Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. https://arxiv.org/abs/2008.00401 Multilingual translation with extensible multilingual pretraining and finetuning
2020 arXiv
-
[14]
Asahi Ushio, Yi Zhou, and Jose Camacho-Collados. 2023. An efficient multilingual language model compression through vocabulary trimming. arXiv preprint arXiv:2305.15020
2023 arXiv
-
[15]
Hainan Xu and Philipp Koehn. 2017. Zipporah: a fast and scalable data cleaning system for noisy web-crawled parallel corpora. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2945--2950
2017
-
[16]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[17]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.