{"id":"e8db44da-4de8-4c74-9196-a1561ae5826b","arxiv_id":"2506.12966","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A quality filter trained only on English labels can select better French, German, and Chinese pretraining data, improving bilingual model performance and cutting the monolingual-bilingual gap to about 1%.","lead":"Training a language model on two languages at once usually makes it worse at each language than a single-language model. This study finds that unequal data quality, not just amount of data, drives that drop, and shows a filtering recipe that uses only English quality labels to improve French, German, and Chinese bilingual models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The English-only quality filter is only validated on translated-from-English benchmarks for German and Chinese, so the claimed cross-lingual transfer and 1% gap reduction may reflect selection of English-like documents rather than higher target-language quality.","rationale":"The reader identifies the English-only filter's cross-lingual transfer as the weakest assumption, and I agree that transfer is central. However, the end-to-end improvements reported for German and Chinese in Tables 8-9 are themselves a form of transfer evidence, so the absence of an embedding histogram for those languages is not the sharpest problem. The sharper issue is the evaluation design: every German and Chinese result is measured on benchmarks translated from English, and the paper's own Limitations section warns that such evaluations can favor models trained on English-derived or translated data. This makes the headline 'reduces bilingual gaps to 1%' vulnerable to a distribution-matching artifact: the filter may be selecting target-language documents that are similar to the English/translated evaluation distribution rather than documents that are higher quality for native-language use. The Appendix D cluster histogram does not resolve this because it only checks English versus translated French. I still credit the controlled quality/language manipulations in Section 3 and the French native evaluations in Section 5.5, which give the paper partial independent support. The concern is a genuine evidential gap, not an internal contradiction, and it does not overturn the paper's direction; it strengthens the case for a conditional verdict pending native-language evaluation of the German and Chinese filtered models.","tokens_in":32559,"tokens_out":10197,"duration_ms":112802,"concrete_test":"Hold out native-language benchmarks for German and Chinese (for example, native German tasks, Chinese CMMLU/CLUE-style multiple-choice sets, and French FrenchBench as a control) and evaluate the same FW2 base versus FW2 90%-filtered monolingual and bilingual models from Tables 8-9 with the same 200K-step protocol. If the filtering advantage persists at comparable magnitude on native evaluation, the cross-lingual quality transfer claim is supported; if the advantage shrinks or reverses, the reported 1% gap reduction is an artifact of English-translated evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central recipe in Section 4 trains a logistic classifier on multilingual SBERT embeddings using English OpenHermes/ELI5 labels, then applies it to filter French, German, and Chinese pretraining data. The headline claim that this reduces bilingual gaps to about 1% (Section 6) requires that the classifier's notion of quality transfers across languages and that the selected documents are genuinely better for native-language use. For German and Chinese, the only evidence is performance on Core tasks translated from English (Tables 8-9) and, in Section 3.5, CMMLU items evaluated in English and Chinese translations; no native German or Chinese benchmark is used. The Limitations section (7) concedes that translated evaluations can reward models trained on English-derived or translated data. Under this setup, the observed 2-4% filter gains and the narrow monolingual-versus-bilingual gaps are equally consistent with a filter that selects documents resembling the English/translated evaluation distribution, not with a language-agnostic quality signal. The Appendix D cluster histogram compares English only to translated French, so it does not close this gap for German and Chinese. The practical claim that English-only labels identify better non-English pretraining data is therefore underdetermined by the reported evaluations.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies whether data quality, rather than data quantity alone, accounts for monolingual-vs-bilingual performance gaps in language models. Section 3 presents controlled experiments on 1.3B models trained on a 2x2 grid of data quality (mC4 low vs FineWebEDU high) and language (native English vs translated French/German/Chinese), finding that high-quality bilingual models match monolingual models within about 1% at 200K steps, while gaps of 2-2.5% appear at 30K steps. Section 4 proposes a language-agnostic filter: a logistic regression trained on multilingual Sentence-BERT embeddings of English DCLM-quality labels (OpenHermes/ELI5 vs RefinedWeb), applied to RedPajama2 and FineWeb2. Section 5 reports French experiments showing filtering improves monolingual and bilingual performance and is competitive with public models; Section J extends the filter to German and Chinese with improvements on Core benchmarks translated from English. The abstract claims 2-4% monolingual gains and reduction of bilingual gaps to 1% for French, German, and Chinese.","tokens_in":32811,"tokens_out":8287,"duration_ms":77921,"significance":"If the stated results hold, the paper makes a useful practical contribution: data quality can be curated with English-only labels, and filtering reduces the bilingual penalty. The controlled experiments in Section 3.3 are well designed, particularly the quality-by-language grid and the 30K-step analysis, and the paper is honest in its Limitations section. The comparison with public models in Table 2 and the ablations with FineWeb2 HQ in Appendix I are valuable reference points. However, the cross-lingual claim for German and Chinese rests on translated benchmarks only, and the headline '1% gap' is obtained after post hoc selection of the filter percentile. The core French evidence is credible, but the broader claim is not yet supported.","major_comments":[{"comment":"The German and Chinese filtering results are evaluated only on Core benchmarks translated from English; no native German or Chinese benchmark is used to evaluate the Section 4 filter. The paper's own Limitations section states that translated evaluations can reward training on English-derived or translated data. The gains in Tables 8-9 are therefore consistent with the filter selecting English-like documents rather than higher target-language quality, and the abstract's 'improves monolingual performance by 2-4% and reduces bilingual model performance gaps to 1%' is not established for German and Chinese. Reporting native benchmark results (e.g., German NLI or regional tasks, Chinese CMMLU for the filtered models) or explicitly limiting the cross-lingual claim to French would address this.","section":"Section J, Tables 8-9; Section 7"},{"comment":"The 90% filter percentile is selected after examining evaluation results for the 30th, 60th, 90th, and 95th percentiles, with the 95% run discarded due to 'over-filtering.' Because the same evaluation set is used for both selection and reporting, the 1% gap figure is a post hoc optimum and is likely optimistic. Please report results for all percentiles without selecting one, or pre-specify the threshold on a validation split.","section":"Section 5.4, Figure 6"},{"comment":"The only evidence for the embedding-space transfer that underlies the language-agnostic filter is a cluster histogram comparing English and translated French data (Figure 8). No alignment check is shown for German or Chinese original corpora before the filter is applied in Section J. Since the filter's applicability to these languages is load-bearing for the paper's main claim, please add a direct alignment evaluation (e.g., cluster overlap, retrieval accuracy, or similarity distributions) for German and Chinese data, or state the transfer assumption explicitly as a limitation.","section":"Appendix D, Section 4"},{"comment":"The claim that bilingual performance gaps are reduced to 1% is conditional on 200K training steps: at 30K steps, the same experiments show gaps of 2-2.5% (Figures 2 and 10). The abstract states the 1% figure without this qualifier, which overstates the recipe's effect for shorter training runs that are common in practice. Please qualify the claim with the training regime.","section":"Abstract; Sections 3.4 and 5.4"}],"minor_comments":[{"comment":"Typo: 'trainng' should be 'training.'","section":"Section 3, first paragraph"},{"comment":"Typo: 'Tabl 5' should be 'Table 5.'","section":"Appendix B.2"},{"comment":"The dataset name is spelled inconsistently as 'RedPajama2' and 'Redpajamav2' (e.g., Section 5.1 vs. Appendix B.1); please standardize.","section":"Throughout"},{"comment":"Abbreviations such as 'SBert OH FR,' 'FT FR,' 'FT-M FR,' and 'fwe_fr' are not defined in the captions or the surrounding text; please define them at first use.","section":"Figure 5 and Figure 13 captions"},{"comment":"The token counts for 'RPJ2 (90%)' (260B French, 248B German) are inconsistent with a 10% retention filter applied to the base pools (310B and 297B). Please clarify whether these counts include repeated epochs or whether the percentile notation refers to something else.","section":"Table 4"},{"comment":"The phrase 'totaling 100K steps each' is ambiguous: it is unclear whether the monolingual models train for 100K or 200K steps and whether the bilingual model trains for 200K total steps with 100K per language; please restate.","section":"Section 3.2, methodology"},{"comment":"The statement 'up to 15× fewer' than other models lacks a unit; please specify that it refers to training tokens.","section":"Section 5.5, first paragraph"},{"comment":"The claim that English and French distributions are 'similar' is based on visual inspection of histograms; please add a quantitative divergence or overlap measure.","section":"Appendix D, Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The paper's French results are convincing and the controlled experiments are well executed. My main concern is the mismatch between the abstract's broad claim and the evidence: the German/Chinese filter gains are measured only on translated benchmarks, and the 90% percentile is selected post hoc. Both are fixable by adding native evaluations (or narrowing the claim) and by reporting all percentiles. The paper would be a solid contribution after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nThe short version: this paper has one genuinely solid empirical result and one useful but partially overclaimed recipe. The controlled experiments in Section 3 showing that data quality, not just quantity, drives the bilingual performance gap are well designed and convincing. The filtering recipe (English-only labels + multilingual SBERT + logistic regression) is practical and shows real gains in French, with suggestive gains in German and Chinese. But the headline \"gaps to 1%\" depends on translated-from-English benchmarks for all three languages, and the filter itself is trained on English quality labels, so there's a real circularity risk for German and Chinese. Treat the cross-lingual generality as promising, not established.\n\nWhat's new: the controlled quality-by-language grid at 30K and 200K steps, using mC4 vs FineWebEDU in native and translated form, cleanly shows that mixed-quality/mixed-language training is what costs performance. The CMMLU experiment (Section 3.5) is a nice control: high-quality translated English data is not enough for a native Chinese knowledge benchmark. That's a genuine insight. The comparison with concurrent work (Messmer et al.) is fair, and the limitations section is unusually candid about translated-eval bias.\n\nSoft spots, in rough order: (1) The filter's transfer to German and Chinese is validated only on translated evaluations. The appendix's cluster histogram covers English vs French, not the other two. The authors do flag translated-eval bias, but the abstract's \"reduces gaps to 1%\" goes beyond what the evidence supports for non-French languages. (2) The 90% threshold is chosen after comparing several percentiles on the eval set; it is an ablation, not an a priori setting. (3) Main tables lack error bars, though some appendix tables have them. (4) No code or data release, which will slow adoption.\n\nNone of this kills the paper. The core data-quality claim holds up; the filtering recipe is worth building on. It deserves serious refereeing. My recommendation: accept the quality-over-quantity finding, and push the authors to either reframe the German/Chinese results as preliminary or add a native benchmark in the filtering setup (e.g., German Regional, Chinese CMMLU). I'd bring it to reading group to discuss the eval-transfer confound.\n\nVerdict: worth a real review, conditional on de-emphasizing or further supporting the cross-lingual transfer claim.","headline":"Solid controlled experiments on data quality vs. quantity in bilingual models; the English-only filtering recipe is useful but its cross-lingual generality is validated via translated benchmarks, so treat the 1% gap claim as French-confirmed and promising elsewhere.","tokens_in":33342,"tokens_out":4095,"would_cite":true,"duration_ms":44407,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unequal data quality, not data quantity, drives bilingual language model performance gaps, and an English-only quality filter can close most of the gap in French, German, and Chinese.","keywords":["data quality","bilingual language models","multilingual pretraining","cross-lingual data filtering","Sentence-BERT","quality classifier","data curation","performance gap"],"falsifier":"Compare the Sentence-BERT embedding clusters of English high- and low-quality documents with those of German and Chinese web data: if the target-language documents do not separate along the same axis, the filter would select a different distribution and the reported gains would not reproduce. A second concrete check is to run the French, German, and Chinese filtered models on native target-language benchmarks such as CMMLU; if the filtered model fails to beat the unfiltered baseline there, the quality-transfer claim collapses.","tokens_in":32312,"feed_emoji":"🌐","tokens_out":5206,"duration_ms":52552,"temperature":0.7,"pith_summary":"The paper argues that the familiar 'curse of multilinguality' performance drops are driven less by adding a second language than by feeding that language lower-quality data. Its controlled experiments hold data constant by translating a single corpus, and the bilingual gap disappears. The paper then shows that a quality classifier trained only on English labels can select better French, German, and Chinese pretraining data from large raw corpora, improving monolingual performance by 2-4% and shrinking bilingual-monolingual gaps to about 1%. If correct, the practical consequence is that balanced bilingual models can be built through data curation rather than longer training or bigger models.","feed_headline":"Data quality, not quantity, drives bilingual model gaps","feed_subtitle":"An English-only quality filter lifts French, German, and Chinese performance 2-4% and cuts bilingual gaps to about 1%.","key_machinery":"The load-bearing mechanism is the multilingual Sentence-BERT embedding space used as a shared geometry: comparable documents in different languages are mapped close together, so a logistic-regression quality classifier trained on English 'high quality' (OpenHermes/ELI5-style) versus 'low quality' (RefinedWeb) labels can rank documents in French, German, and Chinese by their proximity to the English high-quality region. Selecting the 90th percentile of this score yields the training set. The paper also uses translated corpora as a control to hold document content constant while varying language and quality.","core_discovery":"On the paper's own terms, the discovery is that data quality, not language identity or dataset size, is a primary cause of bilingual model degradation. Matching quality between languages—by translating one corpus and training monolingual and bilingual models on the same content—removes the performance gap that appears when models are trained on unequal native corpora. Building on this, a lightweight logistic-regression classifier over multilingual Sentence-BERT embeddings, trained only on English high- and low-quality document labels, selects the top decile of raw French, German, and Chinese web data; the filtered data improves target-language performance and brings bilingual models within about 1% of monolingual models across six common-sense-reasoning tasks and MMLU-style knowledge tasks.","pith_inferences":["Extension: If the embedding-space transfer holds beyond the three tested languages, one English quality seed could curate pretraining data for the 50+ languages covered by the embedding model, centralizing data curation.","Extension: The paper's evaluations for German and Chinese rely on translated English benchmarks; native-language benchmarks would test whether the quality gains reflect genuine target-language understanding rather than translation artifacts.","Extension: A direct alignment check before applying the filter in a new language—comparing the embedding clusters of English high-quality data with the target language's raw data—would cheaply predict whether the filter will transfer.","Extension: Because the filter selects by document-level embedding similarity, it may systematically prefer text that resembles English-style writing; culturally distinct but high-quality target-language documents could be filtered out."],"forward_implications":["Bilingual performance gaps can be reduced to roughly 1% by curating the non-English data, without extra training steps or larger models.","Filtering raw multilingual corpora with an English-trained quality score improves monolingual French, German, and Chinese performance by 2-4% over the unfiltered corpus.","A 1.3B model trained on filtered data matches or beats public bilingual models such as CroissantLLM on French benchmarks, despite being trained on far fewer tokens.","Filtering acts like training acceleration: the same benchmark score is reached about 5x faster for monolingual and 2x faster for bilingual models.","High-quality English data alone, even translated, is not enough for strong non-English performance; native high-quality curation still matters."],"supporting_citations":[{"why":"Supplies the multilingual Sentence-BERT embeddings that define the shared space for cross-lingual filtering.","marker":"Reimers and Gurevych, 2019"},{"why":"Provides the English high/low quality seed data (DCLM classifier training data) and the 10% filtering regime.","marker":"Li et al., 2024"},{"why":"Supplies the classifier-based data selection method the paper adapts to multilingual filtering.","marker":"Grangier et al., 2024a"},{"why":"Provides FineWeb2, the heuristically curated multilingual corpus the filter is applied to.","marker":"Penedo et al., 2024b"},{"why":"Provides RedPajama2, the raw large-scale French and German corpora used as unfiltered baselines.","marker":"Weber et al., 2024"},{"why":"Supplies FineWebEDU, the high-quality English corpus used as the English training data and as the basis for translated reference data.","marker":"Penedo et al., 2024a"},{"why":"Provides TransWebEDU, machine-translated high-quality data used as an upper-bound comparison.","marker":"Wang et al., 2025"},{"why":"Provides CroissantLLM, the public French-English bilingual model the filtered models are compared against.","marker":"Faysse et al., 2024"},{"why":"Provides the bilingual training setup and translation system used in the controlled quality experiments.","marker":"Seto et al., 2024"}],"fun_headline_variants":["Quality beats quantity for bilingual LLMs","English-only filter lifts bilingual models","Data filtering narrows bilingual gap to 1%","Better data, not more, for bilingual AI","Data quality fix boosts bilingual performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the multilingual sentence-embedding space aligns English quality with quality in the target language, so a classifier trained only on English labels ranks French, German, and Chinese documents correctly; the direct alignment evidence is shown for French but not for German or Chinese.","fun_headline_variants_meta":{"raw":{"variants":["Quality beats quantity for bilingual LLMs","English-only filter lifts bilingual models","Data filtering narrows bilingual gap to 1%","Better data, not more, for bilingual AI","Data quality fix boosts bilingual performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1134,"prompt_tokens":845,"completion_tokens":289,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":225}},"tokens_in":461,"tokens_out":289,"duration_ms":3875,"temperature":1.0,"reasoning_tokens":225,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:14.905111+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the Sentence-BERT embedding clusters of English high- and low-quality documents with those of German and Chinese web data: if the target-language documents do not separate along the same axis, the filter would select a different distribution and the reported gains would not reproduce. A second concrete check is to run the French, German, and Chinese filtered models on native target-language benchmarks such as CMMLU; if the filtered model fails to beat the unfiltered baseline there, the quality-transfer claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides RedPajama2, the raw large-scale French and German corpora used as unfiltered baselines."}],"review_version":1}