{"id":"bd57caa2-a42b-40d3-b2ca-f9aad9e072ad","arxiv_id":"2501.16533","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Filtering an English-Polish biomedical corpus with LASER embeddings lets mBART50 match full-corpus BLEU while using 60 percent of the training data.","lead":"This paper tested three automated filters (LASER, MUSE, LaBSE) for cleaning an English-Polish medical translation corpus before fine-tuning a neural machine translation model. It reports that LASER filtering cut training data by 40 percent without losing translation quality, and recommends LASER for this language pair and domain.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LASER-over-baseline recommendation rests on single-seed BLEU gaps as small as 0.009; without seed variance or significance testing, the claimed hierarchy is not established.","rationale":"The paper's central claim is a comparative recommendation, so the load-bearing requirement is that the observed BLEU differences are reproducible rather than artifacts of a single training run. The strongest numerical evidence is small: LASER-60 surpasses Base-all by 0.009 BLEU, and the random baselines themselves vary by 0.05-0.13 BLEU across seeds. Since the filtered models were each trained once, no statement can currently be made about whether LASER would beat MUSE, LaBSE, or even random selection under reruns. The paper's own Limitations explicitly call for significance testing and multiple train/dev splits, which supports this concern rather than refuting it. I do not see an internal inconsistency or a fatal flaw; the experimental setup is clear, reproducible code is provided, and the broad 'filtering can cut data with little loss' conclusion is plausible. The issue is calibration: the firm LASER recommendation exceeds what the evidence establishes. The reader's CONDITIONAL verdict already captures this, so I would keep the verdict unchanged rather than moving to reject. My focus on seed variance and significance testing is slightly narrower than the reader's emphasis on the cosine-similarity ranking assumption, hence partial agreement.","tokens_in":7007,"tokens_out":2844,"duration_ms":31381,"concrete_test":"Retrain Filtered-60% and Filtered-20% LASER, MUSE, and LaBSE with three random seeds each, mirroring the Base-60% and Base-20% protocol; evaluate with SacreBLEU on Khresmoi and report mean, standard deviation, and paired bootstrap 95% confidence intervals for LASER-60 vs Base-all and LASER-60 vs Base-60. If the LASER-60 advantage over Base-all is within one seed standard deviation or its confidence interval includes zero, the recommendation should be downgraded to 'LASER filtering can match full-data BLEU while using 60% of the data,' not 'enhances performance.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Table 1's headline comparison is Filtered-60% LASER at 17.411 vs Base-all at 17.402, a +0.009 BLEU difference, and vs Base-60% at +0.177. Base-60% itself has a standard deviation of 0.05 over three seeds, and Base-20% has 0.13, so the observed gains are comparable to or smaller than seed noise. Each filtered model was trained only once, so there is no variance estimate for the LASER, MUSE, or LaBSE conditions. With a single 1,500-sentence test set and no bootstrap or paired significance test, the hierarchy LASER > MUSE > Baseline > LaBSE could plausibly be reordered under different fine-tuning seeds. The authors candidly list the missing significance testing and multiple-split evaluation in the Limitations, yet still issue a firm recommendation. The unvalidated cosine-similarity ranking assumption also matters, but the immediate correctness risk is that even a perfect filter would not be detectable from these deltas; the central recommendation therefore overstates what the evidence supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three multilingual-embedding-based data filtering methods (LASER, MUSE, LaBSE) against random downsampling and full-corpus training for fine-tuning mBART50 on English-to-Polish biomedical machine translation. The authors filter the UFAL Medical Corpus to 20% and 60% subsets, fine-tune mBART50 on each subset, and evaluate on the Khresmoi test set with SacreBLEU, supplemented by a small human quality assessment. The main reported result is that the LASER-filtered 60% model achieves BLEU 17.411, versus 17.402 for the full-corpus model and 17.234 for a random 60% subset, and the paper recommends LASER as the most effective filtering method for this language pair and domain.","tokens_in":7173,"tokens_out":3860,"duration_ms":39790,"significance":"If the reported hierarchy were statistically robust, this would be a practically useful contribution: it would provide concrete evidence that embedding-based filtering can reduce training data size without sacrificing, and possibly improving, translation quality in a moderately low-resource domain, and the authors make their code publicly available for reproducibility. The study also addresses a gap in the literature by targeting English-Polish, a pair for which previous filtering results may not transfer. However, the empirical basis for the central recommendation is narrow: the headline advantage over the full corpus is +0.009 BLEU, the filtered models are trained once with no variance estimate, no significance tests are reported, and the human evaluation is unblinded and performed by the authors. The paper is transparent about several of these limitations, which is commendable, but the strength of the conclusions still substantially exceeds what the evidence supports.","major_comments":[{"comment":"The central result that LASER filtering outperforms the full corpus rests on a BLEU difference of +0.009 (Filtered-60% LASER: 17.411 vs. Base-all: 17.402). Meanwhile, the random 60% baseline, trained with three seeds, has a standard deviation of 0.05, and the random 20% baseline has a standard deviation of 0.13. The filtered models were trained only once, so no variance estimate is available for LASER, MUSE, or LaBSE. With seed noise of this magnitude, the observed ordering of methods could plausibly change under different fine-tuning seeds, and the hierarchy 'LASER > MUSE > Baseline > LaBSE' stated in §6 is not statistically established. A paired bootstrap or a small multi-seed comparison for at least the leading filtered condition is needed to support the recommendation.","section":"Table 1, §5"},{"comment":"The qualitative assessment was conducted by two of the authors, who are native Polish speakers, but the report does not state that they were blind to which model produced each translation, nor does it provide a quantitative protocol, a sample size, or inter-annotator agreement. The Results section nonetheless states that LASER-60% and Base-all produce 'the most accurate and well-sounding translations' (see example in Table 2), and the Discussion uses this to reinforce the LASER recommendation. The Limitations section correctly acknowledges the absence of a quantitative human evaluation by an expert, but the earlier unblinded assessment should not be presented as supporting evidence without a clear caveat.","section":"§4, §5 (human evaluation)"},{"comment":"The recommendation 'we recommend LASER as the most effective data filtering method for LLM-based machine translation from English to Polish in the biomedical domain' is stated categorically, despite the acknowledged absence of significance testing, single-seed runs for filtered models, and a single 1,500-sentence test set. The Discussion should be rephrased to describe LASER as showing a consistent but statistically unverified advantage in this specific setup, and the possibility that the methods are indistinguishable from random selection (as found in prior work cited by the authors, e.g., Bane and Zaretskaya 2021) should be explicitly discussed.","section":"§6 Discussion and Limitations"}],"minor_comments":[{"comment":"The refined dataset size is written as '700,00' and should be '700,000'.","section":"§3"},{"comment":"The sentence 'The latter indicates that filtering results might not be universally applicable across different pairs or topics of languages, which motivated our study design to specifically investigate the Polish-English translation' uses the reverse direction of the language pair investigated in the experiments; it should refer to English-Polish translation.","section":"§2"},{"comment":"The filtered subset sizes are reported as 426k and 158k for the 60% and 20% conditions, while §4 states approximately 420k and 150k pairs; please clarify whether the discrepancy is due to rounding, deduplication, or a different counting convention.","section":"Table 3"},{"comment":"The sentence 'LASER and MUSE achieved relatively lower validation BLEU scores (-0.288 and -0.331 respectively) compared to the baseline model (Base-all)' is confusing because the numbers are BLEU differences, not scores; rephrase to say that LASER-20% and MUSE-20% decrease BLEU by 0.288 and 0.331 relative to Base-all.","section":"§6"},{"comment":"The example showing MUSE-20% rendering 'Meningococcal Disease' as 'Choroba gruczołu krokowego' is a striking error, but a single example does not quantify translation quality; consider reporting how frequently such terminology errors occurred across the evaluated sentences.","section":"Table 2"},{"comment":"The abstract says translations were 'assessed by bilingual speakers,' but the experimental setup says two authors performed the assessment; please clarify the evaluation procedure and the number of sentences reviewed.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its limitations, and the code release is a positive feature. However, the main practical recommendation rests on BLEU differences that are within or near the seed noise of the baseline conditions, and the human evaluation is not blinded. The manuscript could be made acceptable by substantially softening the conclusions and adding at least a bootstrap significance test on the test set or a small multi-seed comparison for the leading filtered condition; otherwise the central claim is not supported by the reported evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper is that its headline result—LASER filtering beats training on the full corpus—is a single-seed BLEU difference of +0.009. That number is pure noise. The more honest reading of the table is that LASER at 60% data slightly outperforms a random 60% subset (+0.177), but even that gap comes from one run of the filtered model against three-seed baselines, so you can't rule out seed luck.\n\nWhat the paper does well: it's the first systematic filtering comparison for English-Polish biomedical MT, it uses an independent test set (Khresmoi) that wasn't touched by filtering or training, and it compares against both random subsets and the full corpus. Code is released. The limitations section is unusually candid—it names the missing significance testing, the fixed three-epoch setup, and the lack of blinded expert evaluation. Those are real strengths and they make the paper useful as a data point.\n\nThe soft spots are exactly where the stress-test lands. Filtered models get one training run each, so there's no variance estimate for the LASER/MUSE/LaBSE conditions. The random baselines show seed noise of 0.05 and 0.13 BLEU, which brackets the reported gaps. The human quality judgments were made by the authors, unblinded, and the qualitative example in Table 2 doesn't show a clear winner either. On top of that, the cosine-similarity scoring assumption—that embedding distance ranks translation-pair quality correctly for biomedical Polish—is never validated independently. The paper's own limitations mention bootstrapping as future work, but the abstract and conclusion still issue a firm recommendation for LASER. That overstates the evidence.\n\nThe central practical message survives: embedding filtering can cut training data substantially while staying within a few tenths of a BLEU point of the full-corpus model, and the compute savings are real. But the specific hierarchy LASER > MUSE > LaBSE is not established at the reported granularity.\n\nThis paper deserves a serious referee. It's a legitimate extension of existing filtering work to an untested pair and domain, with reproducible code. A referee should ask for multiple fine-tuning seeds for the filtered conditions, bootstrap significance intervals, and either a blinded human evaluation or a toned-down qualitative claim.\n\nMy bottom line: send it out, but expect revision.","headline":"A useful, honest empirical data point that overstates its LASER recommendation relative to single-seed BLEU gaps within noise.","tokens_in":7726,"tokens_out":3006,"would_cite":false,"duration_ms":28396,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Data filtering by LASER sentence-embedding similarity lets an English-Polish biomedical translation model match full-corpus fine-tuning while training on only 60% of the data, and the paper recommends LASER as the most effective filter…","keywords":["English-Polish machine translation","biomedical domain","data filtering","multilingual sentence embeddings","cosine similarity","LASER","MUSE","LaBSE"],"falsifier":"Run the same filtering and fine-tuning protocol but evaluate on a second biomedical English-Polish test set, or apply bootstrap resampling to the Khresmoi scores, and check whether LASER-60% still matches or beats the full-corpus model; if the advantage disappears or reverses, the recommendation fails to generalize. Alternatively, take a human-annotated sample of medical sentence pairs from the source corpus and test whether the pairs ranked highest by LASER cosine similarity are actually judged the best training examples by bilingual medical translators; disagreement would falsify the premise that the filter scores quality.","tokens_in":6829,"feed_emoji":"🧬","tokens_out":8969,"duration_ms":77009,"temperature":0.7,"pith_summary":"The paper sets out to test whether pruning a large biomedical parallel corpus to its top-quality sentence pairs can shrink the cost of fine-tuning an LLM-based machine translator without sacrificing quality, specifically for English-to-Polish. It compares three embedding-based filters — LASER, MUSE, and LaBSE — at two retention levels (about 20% and 60% of a cleaned 700k-pair medical corpus), fine-tunes mBART50 on each subset, and evaluates on the independent Khresmoi test set using SacreBLEU plus native-speaker inspection. The central result is that the LASER-filtered 60% model scores 17.411 BLEU, marginally above the full-corpus model's 17.402, while using roughly 60% of the training data; LASER and MUSE both beat random subsets of equal size, while LaBSE does not. This is the first systematic comparison of these filters for English-Polish biomedical translation, and the paper's recommendation is to use LASER for this setup.","feed_headline":"LASER filter matches full-corpus Polish medical MT on 60% of data","feed_subtitle":"Fine-tuning on LASER-scored pairs scores 17.411 BLEU vs 17.402 for the full 700k-pair corpus, at 60% of the data.","key_machinery":"The central mechanism is cosine similarity between language-agnostic sentence embeddings, used as a quality score for each English-Polish sentence pair. Three encoder families produce the embeddings: LASER learns shared cross-lingual sentence representations; MUSE aligns monolingual word embeddings into a shared space; LaBSE produces BERT-based bilingual sentence embeddings. The paper ranks all pairs in the medical corpus by this score and keeps the top 20% or 60%, then fine-tunes mBART50 on the surviving pairs. Random subsets of the same sizes and the full corpus serve as controls, so any BLEU difference can be attributed to the filter's selection rather than to dataset size.","core_discovery":"The authors claim that data filtering by cross-lingual sentence-embedding similarity is a viable substitute for full-corpus fine-tuning in domain-adapted machine translation. Concretely, keeping the 60% of English-Polish medical sentence pairs with the highest LASER cosine-similarity scores and fine-tuning mBART50 on them yields 17.411 BLEU on Khresmoi, above the full-corpus baseline of 17.402 and above the random-60% average of 17.234. At 20% retention, LASER (17.114) and MUSE (17.071) outperform random selection (16.801 average) but fall short of the full-corpus model. Bilingual-speaker assessment puts LASER-60% and the full-corpus model at the top for fluency and naturalness, while MUSE-20% loses medical terminology; LaBSE is not beneficial at either size despite correlating strongly with LASER scores ($r = 0.81$). The paper concludes that LASER is the most effective data filtering method for English-Polish biomedical MT.","pith_inferences":["Editorial extension: the honest practical takeaway from the small BLEU gaps is compute savings, not quality improvement; a team wanting faster iterations could adopt LASER-60% and expect roughly unchanged quality at reduced cost.","Because LaBSE and LASER scores correlate at $r = 0.81$ yet the resulting models differ, inspecting the low-agreement sentence pairs may reveal the specific selection signal that makes LASER effective.","The same filtering pipeline is likely to transfer to other medium-resource language pairs, but the LaBSE failure is a warning that the transfer must be tested rather than assumed for each pair and domain.","Combining cosine-similarity filtering with a domain-specificity measure, such as medical-term density, could remove off-topic sentences that similarity alone lets through."],"forward_implications":["A LASER-filtered 60% subset can replace the full corpus for English-Polish biomedical fine-tuning with no measurable BLEU loss and noticeably lower training time.","The gain from filtering is domain- and language-pair-specific: LaBSE's negative result shows a filter that works elsewhere can hurt here, so each new pair needs its own comparison.","At 20% retention, embedding filters still beat random pruning and the unfiltered pretrained model, but cannot match full-corpus quality, indicating a practical floor on how much data can be cut.","Automatic BLEU alone is not enough to pick a filter, since MUSE-20%'s BLEU looks acceptable while human inspection finds it garbles medical terms; the qualitative check changes the ranking."],"supporting_citations":[{"why":"Supplies the LASER sentence-embedding architecture that the paper uses as its top-performing filter.","marker":"Schwenk and Douze (2017)"},{"why":"Provides the premise that multilingual sentence-embedding filtering is competitive with classifier-based corpus cleaning.","marker":"Chaudhary et al. (2019)"},{"why":"Prior comparison of filtering methods showing results vary by language pair, which motivates testing English-Polish specifically.","marker":"Bane and Zaretskaya (2021)"},{"why":"Characterizes the strengths and weaknesses of embedding-based filters, used to interpret LASER, MUSE, and LaBSE behavior.","marker":"Bane et al. (2022)"},{"why":"Supplies LaBSE, one of the three filter models evaluated.","marker":"Feng et al. (2022)"},{"why":"Supplies MUSE, one of the three filter models evaluated.","marker":"Conneau et al. (2017)"},{"why":"Supplies the Khresmoi test set on which all models are evaluated.","marker":"Dušek et al. (2017)"},{"why":"Supplies SacreBLEU, the evaluation metric used for all BLEU comparisons.","marker":"Post (2018)"},{"why":"Supplies mBART50, the pretrained multilingual model fine-tuned in all experiments.","marker":"Tang et al. (2020)"},{"why":"Establishes the rationale that low-quality parallel data contributes little to model performance, motivating the filtering approach.","marker":"Koehn et al. (2018)"}],"fun_headline_variants":["LASER filter beats full corpus in Polish medical MT at 60% data","Less data, better BLEU: LASER filtering lifts Polish medical MT","LASER tops full-corpus Polish medical translation with 60% of data","Filtered data outperforms full corpus for Polish medical MT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison rests on the premise that a high cosine similarity between the English and Polish embeddings of a sentence pair marks it as good training data for medical translation; the paper does not independently verify that premise, and the final BLEU differences among methods are small and untested for statistical significance.","fun_headline_variants_meta":{"raw":{"variants":["LASER filter beats full corpus in Polish medical MT at 60% data","Less data, better BLEU: LASER filtering lifts Polish medical MT","LASER tops full-corpus Polish medical translation with 60% of data","Filtered data outperforms full corpus for Polish medical MT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001468,"raw_usage":{"total_tokens":5919,"prompt_tokens":979,"completion_tokens":4940,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":4861}},"tokens_in":595,"tokens_out":4940,"duration_ms":35887,"temperature":1.0,"reasoning_tokens":4861,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:31:34.007840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same filtering and fine-tuning protocol but evaluate on a second biomedical English-Polish test set, or apply bootstrap resampling to the Khresmoi scores, and check whether LASER-60% still matches or beats the full-corpus model; if the advantage disappears or reverses, the recommendation fails to generalize. Alternatively, take a human-annotated sample of medical sentence pairs from the source corpus and test whether the pairs ranked highest by LASER cosine similarity are actually judged the best training examples by bilingual medical translators; disagreement would falsify the premise that the filter scores quality.","supporting_citations":[{"cited_title":"Low-Resource Corpus Filtering using Multilingual Sentence Embeddings","cited_arxiv_id":"1906.08885","evidence_quote":"Provides the premise that multilingual sentence-embedding filtering is competitive with classifier-based corpus cleaning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior comparison of filtering methods showing results vary by language pair, which motivates testing English-Polish specifically."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Characterizes the strengths and weaknesses of embedding-based filters, used to interpret LASER, MUSE, and LaBSE behavior."}],"review_version":1}