Pith. sign in

REVIEW 4 major objections 6 minor 40 references

Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM quality judgments, distilled into lightweight regressors on frozen multilingual embeddings, filter pretraining data better than heuristic methods across 35 languages and unseen scripts.

desk verdict Useful artifacts and a solid multilingual pipeline, but the headline downstream-quality claim is weakened by benchmark alignment and unmatched token counts. read the letter →

arxiv 2505.22232 v2 pith:3B77TNQM submitted 2025-05-28 cs.CL cs.AIcs.LG

Mehdi Ali , Manuel Brack , Max Lübbering , Elias Wendt , Abbas Goher Khan , Richard Rutmann , Alex Jude , Maurice Kraus
show 10 more authors
Alexander Arno Weber David Kaczér Florian Mai Lucie Flek Rafet Sifa Nicolas Flores-Herr Joachim Köhler Patrick Schramowski Michael Fromm Kristian Kersting
This is my paper
classification cs.CLcs.AIcs.LG
keywords multilingualpretrainingdatafilteringLLM-as-a-judgeknowledgedistillationcross-lingualtransfereducationalvaluescoringFineWeb2lightweightannotators
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that high-quality multilingual pretraining data can be curated without expensive LLM inference or language-specific rules. JQL does this in four steps: humans score 511 English documents on educational value; those scores are carried over to machine translations in 35 languages; strong LLMs are ranked by how well their judgments correlate with the human ground truth; and the best LLMs' scores are distilled into small regression heads on a frozen multilingual embedding model. The resulting annotators, applied with percentile thresholds and ensemble agreement, filter raw FineWeb2 data into a corpus that trains better 2B-parameter models than the heuristic Fineweb2 filter. The paper reports a +6.70% average and +7.2% final downstream benchmark gain at the 0.7 percentile threshold with 15.8% fewer retained tokens, and shows the same annotators work zero-shot on Arabic, Thai, and Chinese.

What carries the argument

The load-bearing mechanism is distillation of LLM judgment into a lightweight regressor: a single-layer MLP (under 1% of total parameters) with ReLU activation, trained on three-score-aggregated labels from Gemma-3-27B-it, Mistral-3.1-24B-it, and LLaMA-3.3-70B-it, over the frozen Snowflake Arctic Embed v2 multilingual embeddings. Because that embedding space aligns semantically equivalent text across languages, the same head scores documents in unseen languages and scripts without retraining. The second mechanism is dynamic percentile filtering: each head's scores are thresholded at the same quantile (0.6 or 0.7), and a document is kept only if all three heads place it above their threshold, which removes the need for an absolute quality scale shared across models.

What would settle it

Have native speakers manually score a few hundred documents in, say, German, Arabic, and Thai, then re-run judge selection and lightweight-annotator evaluation against those native scores and check whether the Spearman correlations and the +6-7% downstream gains survive.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that semantic quality filtering of multilingual web text reduces to a distillation problem: a small regression head trained on the judgments of three strong multilingual LLMs, sitting on top of a frozen cross-lingual embedding backbone, reproduces the ranking behavior of those LLMs and transfers to languages never seen in training. The evidence for this is the downstream comparison: at the 0.7 percentile threshold, JQL-filtered training data outperforms the heuristic Fineweb2 baseline by +6.70% on average benchmark performance and +7.2% at the final checkpoint while retaining 15.8% fewer tokens; at the 0.6 threshold JQL retains more tokens than the baseline in most languages and still trains better models. The paper further claims that rank correlation with human ground truth, not classification accuracy, is the right selection criterion for judges, and that percentile-based ensemble filtering makes absolute score scales unnecessary.

Load-bearing premise

The load-bearing premise is that machine-translated versions of 511 English documents, scored with the original English human judgments, form a valid multilingual ground truth for selecting judges and training annotators; if translation distorts educational quality, every downstream comparison inherits that distortion.

Editorial extensions

If this is right

  • Filtering raw multilingual web data with JQL yields higher downstream benchmark scores than the heuristic Fineweb2 filter at the same or even larger data budgets.
  • At the 0.7 percentile threshold, JQL-filtered data improves average and final downstream performance by +6.70% and +7.2% relative to the FW2 baseline while retaining 15.8% fewer tokens.
  • The trained annotators rank documents in Arabic, Thai, and Chinese as reliably as in the European languages, so a new language can be added without retraining the filter.
  • Because the regression head is small and the embedding backbone is shared, additional filtering criteria can be implemented as separate heads at little extra cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If translated ground truth preserves enough of the educational-quality ordering, the method implies that a small human annotation effort in one language can bootstrap data curation for many languages; the paper itself only tests translated English ground truth, so this generalization is an inference.
  • The paper's emphasis on Spearman correlation and percentile thresholds suggests that downstream curation systems should select judges by ranking consistency rather than classification accuracy; that principle is argued for here but not yet established as a general design rule.
  • A natural next test, left implicit in the paper, is applying the same pipeline to other filtering criteria such as code quality, mathematical correctness, or adult-content removal, where the same aligned embeddings might transfer the learned judgment across languages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces JQL, a four-stage pipeline for multilingual pretraining-data filtering: human ground-truth annotations on 511 English documents are machine-translated into 35 languages; LLM judges are selected by Spearman correlation against this translated ground truth; lightweight regression heads are distilled from the selected LLMs' annotations on top of frozen multilingual embeddings; and the resulting annotators are used to filter FineWeb2 with percentile-based thresholds. The main empirical claims are that JQL-filtered data improves 2B-parameter model performance on multilingual MMLU/HellaSwag/ARC by +6.70% (average) and +7.2% (final) relative to the FW2 heuristic baseline at the 0.7 percentile threshold while retaining 15.8% fewer tokens, and that the annotators generalize zero-shot to Arabic, Thai, and Chinese. The paper releases the human ground-truth set, LLM annotations, distilled heads, and pipeline code.

Significance. If the central claims hold, JQL is a valuable, efficient, and open alternative to heuristic multilingual filtering, with particular strengths in cross-lingual zero-shot transfer and low-cost annotation at scale. The paper ships extensive artifacts, detailed ablations on backbone choice, context length, training-data amount, and label balancing, and a falsifiable zero-shot evaluation on three typologically unseen languages. These are concrete strengths that make the contribution reproducible and useful to the community. However, the downstream evidence is confined to educational/knowledge benchmarks, token budgets are not matched across conditions, and no random-retention control is provided; as a result, the abstract's broad claim that JQL 'enhances downstream model training quality' is stronger than the current experimental support.

major comments (4)
  1. [§5.1, §5.3, Table 2] The central claim that JQL 'enhances downstream model training quality' is evaluated only on MMLU, HellaSwag, and ARC, all educational/knowledge benchmarks whose content aligns with the FineWeb-Edu 'educational value' objective used by the filter (Sec. 2.1). This is a benchmark-alignment concern: a filter trained to keep educationally valuable documents is expected to improve such benchmarks even if it does not improve code, math, translation, or general language modeling. Please add non-educational downstream tasks (e.g., multilingual code, math, translation, and held-out perplexity) or explicitly restrict the claim to educational-value-filtered data. The zero-shot results in Fig. 6 use the same family of educational QA benchmarks and do not resolve this issue.
  2. [§5.1, Table 2, App. D.1] The comparison is not token-matched and the training protocol is under-specified. At the 0.7 threshold JQL retains 15.8% fewer tokens than FW2; the paper does not state epochs, repetitions, or total training-token budgets per condition. If runs are trained to the same total token count, the smaller JQL corpus is repeated more often, which can inflate benchmark scores; if runs are single-epoch, the 'Final' checkpoints occur at different total token counts, making the reported relative gains ill-defined. No control that randomly removes a matched fraction of FW2 tokens is reported. Please report the exact step/epoch/token budgets, match total tokens across conditions, and include a random-retention control so the improvement can be attributed to the quality signal.
  3. [§2.2, §9] The multilingual ground truth is machine-translated (DeepL/GPT-4o) from 511 English documents with English human scores carried over, and the same translated set is used to select LLM judges and validate every lightweight-annotator choice. The paper's Limitations section acknowledges the translation issue but provides no quantification of translation quality or any native-speaker validation for a subset of languages. Because translation artifacts would propagate through judge selection, head training, and downstream filtering, please add a small per-language human validation (or at least systematic back-translation/quality scores) and report per-language Spearman agreement; without this, the robustness of the multilingual claims is not established.
  4. [§5.1, Fig. 5, Table 2] All downstream conclusions rest on single 2B-scale runs per language/threshold, with no seeds or standard errors reported. Given the headline numbers are aggregated relative changes in a token-normalized probability on heavily under-trained models, the absence of variance estimates weakens the 'consistently outperforms' and 'notably enhances' statements. Please provide multiple seeds for the headline conditions, or at minimum a variance estimate from the 13-language spread, and state whether per-language differences are within expected run-to-run noise.
minor comments (6)
  1. [§2.2, App. G] The main text says the ground truth is translated into 35 languages, but Table 5 shows several test sets have fewer than 511 samples after translation filtering (e.g., 502 for Croatian, 505 for Irish); please state this in Sec. 2.2.
  2. [§5.3, Figs. 5-6] The abbreviation 'Gold Label Prop.' used in Figs. 5-6 is not defined in the main text; please define it as the token-normalized probability of the correct answer and explain why the y-axis ranges differ across languages.
  3. [App. E.1] The header 'Evaluation of Lightweight PQL-Annotator' contains a typo; it should read 'JQL-Annotator'.
  4. [Related Work, §5.3] Given the related work discusses neural filters such as Qurating and DataComp-LM, the empirical comparison would be more convincing with a neural filtering baseline in Table 2; as written the 'outperforms current heuristic filtering' claim is established only versus FW2.
  5. [§5.1] The phrase 'Common Crawl W ARC files' appears to be a typo for 'WARC files'.
  6. [App. H] Table 8 lists Chinese-SQuAD as having no license information; please clarify whether the authors can redistribute it under the released dataset.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the educational-value objective is inherited from FineWeb-Edu, but the downstream benchmark comparison is external and empirically contingent.

full rationale

JQL's ground-truth labels are human annotations collected under the FineWeb-Edu educational prompt (Sec. 2.1), so the notion of 'quality' is operationally inherited from an external LLM rubric rather than derived from JQL's own outputs. That is an explicit modeling choice, not a hidden reduction: the lightweight annotators are trained on separate LLM-generated labels (Sec. 4) and evaluated against human annotations (Fig. 3), and the headline claim is tested by training 2B models on filtered versus heuristic corpora and scoring them on MMLU/HellaSwag/ARC and multilingual QA (Secs. 5-6). These benchmarks are external to the annotation loop, so the reported improvement is not a restatement of the filter scores. No fitted parameter is renamed as a prediction, and no load-bearing step depends on a self-citation: the only self-citations (e.g., Brack et al. 2024 in Related Work) support background claims. Acknowledged limitations (machine-translated ground truth, single-scale ablations, annotator overlap in App. A.1) affect validity and generalizability but do not make the derivation circular. Token-count differences across conditions are an experimental confound, not a circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The pipeline introduces no new physical or theoretical entities. The main free parameters are the percentile thresholds and the annotator training-set size. The axioms listed are the load-bearing domain assumptions that connect the annotation procedure to downstream pretraining quality; all are acknowledged to some degree in the paper, but none is independently proven.

free parameters (2)
  • percentile filtering thresholds = 0.6 and 0.7 quantiles
    Two relative thresholds are chosen to filter per annotator head; results are reported for both, and 0.7 is applied in unseen-language experiments. The central comparison depends on these chosen values.
  • annotator training set size = 500,000 documents across 35 languages
    Selected after a data-amount ablation (Fig. 17); not fitted to the downstream benchmark, but a hand-chosen design point.
assumptions (4)
  • domain assumption FineWeb-Edu's educational-value scoring (0-5 prompt) is a valid proxy for pretraining data quality
    The entire pipeline optimizes for 'educational value' as defined by the FineWeb-Edu prompt (Sec. 2.1); downstream benchmark gains are assumed to follow from this proxy.
  • domain assumption Small 2B-parameter, 14-27B-token training runs predict dataset quality ordering at larger scales
    Invoked in Sec. 5.1 and Limitations via DataDecide (Magnusson et al., 2025); the main downstream claim is established only at this scale.
  • domain assumption Machine translation preserves the relative educational quality of documents
    All non-English ground truth is obtained by translating the 511 English documents and keeping English human scores (Sec. 2.2); no human validation of translated documents is performed.
  • domain assumption Spearman rank correlation with human scores is the right selection criterion for filtering
    Sec. 2.4 argues ranking matters more than classification; this justifies using Spearman to select LLM judges and annotators, and is central to the pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models." pith.science (2026). https://pith.science/paper/3B77TNQM

@misc{pith2026250522232,
  author       = {Pith},
  title        = {Pith review of: Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3B77TNQM}},
  note         = {Machine review of arXiv:2505.22232}
}
read the original abstract

High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets remains limited. Existing state-of-the-art datasets mostly rely on heuristic filtering methods, restricting both their cross-lingual transferability and scalability. Here, we introduce JQL, a systematic approach that efficiently curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. JQL distills LLMs' annotation capabilities into lightweight annotators based on pretrained multilingual embeddings. These models exhibit robust multilingual and cross-lingual performance, even for languages and scripts unseen during training. Evaluated empirically across 35 languages, the resulting annotation pipeline substantially outperforms current heuristic filtering methods like Fineweb2. JQL notably enhances downstream model training quality and increases data retention rates. Our research provides practical insights and valuable resources for multilingual data curation, raising the standards of multilingual dataset development.

Figures

Figures reproduced from arXiv: 2505.22232 by the authors.

Figure 1
Figure 1. The multilingual data filtering approach JQL: In the first stage (Sec. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. LLMs show varying ranking performance for educational quality. Some models exhibit strong multilingual [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Lightweight JQL annotators show strong multilingual and cross-lingual performance. Training on [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (28 more)
Figure 4
Figure 4. Figure 4: Lightweight annotators trained on different [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Our JQL annotators improve pre-training data [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Our JQL lightweight annotators generalize to [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Histogram on the distribution of the document scores judged by the human annotators. [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Cumulative distribution of spread within annotations. Aligned with the majority agreement of 78.5% and [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Invalid scores predictions (in percent) B LLM Based Annotator Evaluation In this Section we provide further details and ablations on our LLM based annotators discussed in Section 3. B.1 Invalid Predictions Similar to the human annotators, we prompted the LLM-based anno…
Figure 10
Figure 10. Figure 10: Percentages of invalid scores (aggregated) for each model across all languages. An aggregated score [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Multilingual LLM classification performance (macro F1-score) on human-annotated ground truth. Scores [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Ranking performance in terms of Spearman correlation for each model across all languages. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Classification performance in terms of macro F1 score for each model across all languages. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]
Figure 14
Figure 14. Figure 14: Confusion matrices of the three ablated LLMs on the 511 human annotated ground truth documents in [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Right cumulative distribution of the scores predicted by the three ablated models. Alternatively, the [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Validation performance (Spearman correlation) as a function of the number of processed training samples, [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Ten training runs (one per row), utilizing between 10k and 10M training samples (text documents). The [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Full cross-lingual transfer; One plot per Annotation model (balanced); training/evaluation setup is [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]
Figure 19
Figure 19. Figure 19: Distribution of different lightweight annotation heads on CC release 2024-14 over 13 languages. Training [PITH_FULL_IMAGE:figures/full_fig_p030_19.png]
Figure 20
Figure 20. Figure 20: Distribution of edu score annotations by language. Dotted lines represent the respective mean. [PITH_FULL_IMAGE:figures/full_fig_p031_20.png]
Figure 22
Figure 22. Figure 22: Dataset training performance for German. [PITH_FULL_IMAGE:figures/full_fig_p031_22.png]
Figure 24
Figure 24. Figure 24: Dataset training performance for Spanish. [PITH_FULL_IMAGE:figures/full_fig_p031_24.png]
Figure 26
Figure 26. Figure 26: Dataset training performance for French. [PITH_FULL_IMAGE:figures/full_fig_p031_26.png]
Figure 28
Figure 28. Figure 28: Dataset training performance for Italian. [PITH_FULL_IMAGE:figures/full_fig_p032_28.png]
Figure 30
Figure 30. Figure 30: Dataset training performance for Norwegian [PITH_FULL_IMAGE:figures/full_fig_p032_30.png]
Figure 31
Figure 31. Figure 31: Dataset training performance for Polish. [PITH_FULL_IMAGE:figures/full_fig_p032_31.png]
Figure 33
Figure 33. Figure 33: Dataset training performance for Ukrainian. [PITH_FULL_IMAGE:figures/full_fig_p033_33.png]
Figure 35
Figure 35. Figure 35: Dataset training performance for Thai. 5 10 15 20 25 Training Tokens in Billion 2.0 2.5 3.0 3.5 4.0 4.5 MMLU – Gold Label Prop. (%) Chinese 20 30 40 50 QA Tasks – Gold Label Prop. (%) Benchmark MMLU CMRC 2018 Chinese SQuAD Quality Filter FW2 JQL-Edu-0.7 (Ours) −11.39%…
Figure 37
Figure 37. Figure 37: Strong cross-lingual performance of our lightweight JQL annotators on unseen languages (Arabic, Thai, [PITH_FULL_IMAGE:figures/full_fig_p033_37.png]
Figure 38
Figure 38. Figure 38: Direct comparison of Gemma and Llama as annotators. [PITH_FULL_IMAGE:figures/full_fig_p034_38.png]
Figure 39
Figure 39. Figure 39: Increased context length of lightweight JQL-annotators improved performance. [PITH_FULL_IMAGE:figures/full_fig_p035_39.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

40 extracted references · 21 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Julien Abadji, Pedro Javier Ortiz Su \' a rez, Laurent Romary, and Beno \^ t Sagot. 2022. https://aclanthology.org/2022.lrec-1.463 Towards a cleaner document-oriented multilingual crawled corpus . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, LREC . European Language Resources Association

    work page 2022

  4. [4]

    Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werr...

    arXiv 2025

  5. [5]

    Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2019. https://arxiv.org/abs/1910.11856 On the cross-lingual transferability of monolingual representations . arXiv preprint arXiv1910.11856:

    arXiv 2019

  6. [6]

    Manuel Brack, Malte Ostendorff, Pedro Ortiz Suarez, Jos \'e Javier Saiz, I \ n aki Lacunza Castilla, Jorge Palomar-Giner, Alexander Shvets, Patrick Schramowski, Georg Rehm, Marta Villegas, and Kristian Kersting. 2024. https://aclanthology.org/2024.mrl-1.19/ Community oscar: A community effort for multilingual web data . In Proceedings of the Fourth Worksh...

    work page 2024

  7. [7]

    Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Pinzhen Chen, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Jindřich Helcl, Erik Henriksson, Mateusz Klimaszewski, Ville Komulainen, Andrey Kutuzov, Joona Kytöniemi, Veronika Laippala, Petter Mæhlum, Bhavitvya Malik, Farrokh Mehryary, Vladislav Mikhailov, Nikita Moghe, ...

    arXiv 2025

  8. [8]

    Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki

    Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. https://aclanthology.org/2020.tacl-1.30/ Tydi qa: A benchmark for information-seeking question answering in typologically diverse languages . Transactions of the Association for Computational Linguistics

    work page 2020

Show all 40 references
  1. [9]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . arXiv preprint arXiv:1803.05457

    2018 arXiv

  2. [10]

    Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. https://aclanthology.org/D19-1600/ A span-extraction dataset for C hinese machine reading comprehension . In Proceedings of the Conference on Empirical Methods in Natural L...

    2019

  3. [11]

    Roberts Dar g is, Guntis B \= a rzdi n s , Inguna Skadi n a, Normunds Gr \= u z \= i tis, and Baiba Saul \= i te. 2024. Evaluating open-source LLM s in low-resource languages: Insights from L atvian high school exams. In Proceedings of the 4th International Conference on Natur...

    2024

  4. [12]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. https://arxiv.org/abs/2101.00027 The pile: An 800gb dataset of diverse text for language modeling . a...

    2020 arXiv

  5. [13]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. https://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning . arXiv preprint arXiv:2501.12948

    2025 arXiv

  6. [14]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring massive multitask language understanding . In Proceedings of the International Conference on Learning Representati...

    2021

  7. [15]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2016. https://arxiv.org/abs/1607.01759 Bag of tricks for efficient text classification . arXiv preprint arXiv:1607.01759

    2016 arXiv

  8. [16]

    Buckley, Jason Phang, Samuel R

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023. https://proceedings.mlr.press/v202/korbak23a/korbak23a.pdf Pretraining language models with human preferences . In Proceedings of th...

    2023

  9. [17]

    Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. https://papers.nips.cc/paper_files/paper/2023/hash/d49042a5d49818711c401d34172f9900-Abstract-Datasets_and_Benchmarks.html MADLAD-400: A mul...

    2023

  10. [18]

    Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. https://arxiv.org/abs/1910.07475 Mlqa: Evaluating cross-lingual extractive question answering . arXiv preprint arXiv:1910.07475

    2019 arXiv

  11. [19]

    Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Yitzhak Gadre, Hritik Bansal, Etash Guha, Sedrick Scott Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee F. Chen, Suchin Gururangan, Mitchell Wortsman, Alon ...

    2024

  12. [20]

    a in \"o Hatanp \

    Risto Luukkonen, Jonathan Burdge, Elaine Zosa, Aarne Talman, Ville Komulainen, V \"a in \"o Hatanp \"a \"a , Peter Sarlin, and Sampo Pyysalo. 2024. https://arxiv.org/abs/2404.01856 Poro 34b and the blessing of multilinguality . arXiv preprint arXiv:2404.01856

    2024 arXiv

  13. [21]

    Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A

    Ian Magnusson, Nguyen Tai, Ben Bogin, David Heineman, Jena D. Hwang, Luca Soldaini, Akshita Bhagia, Jiacheng Liu, Dirk Groeneveld, Oyvind Tafjord, Noah A. Smith, Pang Wei Koh, and Jesse Dodge. 2025. https://arxiv.org/abs/2504.11393 Datadecide: How to predict best pretraining d...

    2025 arXiv

  14. [22]

    Tamzeed Mahfuz, Satak Kumar Dey, Ruwad Naswan, Hasnaen Adil, Khondker Salman Sayeed, and Haz Sameen Shahgir. 2025. https://aclanthology.org/2025.coling-main.79.pdf Too late to train, too early to use? a study on necessity and viability of low-resource B engali LLM s . In Proce...

    2025

  15. [23]

    Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. 2019. https://www.aclweb.org/anthology/W19-4612 Neural A rabic question answering . In Proceedings of the Fourth Arabic Natural Language Processing Workshop. Association for Computational Linguistics

    2019

  16. [24]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. http://www.lrec-conf.org/proceedings/lrec-coling-2024/pdf/2024.main-1.377.pdf C ultura X : A cleaned, enormous, and multilingual dataset for ...

    2024

  17. [25]

    Raffel, Leandro von Werra, and Thomas Wolf

    Guilherme Penedo, Hynek Kydl \' cek, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin A. Raffel, Leandro von Werra, and Thomas Wolf. 2024 a . https://proceedings.neurips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_T...

    2024

  18. [26]

    Guilherme Penedo, Hynek Kydlíček, Vinko Sabolčec, Bettina Messmer, Negar Foroutan, Martin Jaggi, Leandro von Werra, and Thomas Wolf. 2024 b . https://doi.org/10.57967/hf/3744 Fineweb2: A sparkling update with 1000s of languages

    2024 doi

  19. [27]

    Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/hash/fa3ed726cc5073b9c31e3e49a807789c-Abstract-Data...

    2023

  20. [28]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. https://jmlr.org/papers/volume21/20-074/20-074.pdf Exploring the limits of transfer learning with a unified text-to-text transformer . Journal of...

    2020

  21. [29]

    Noveen Sachdeva, Benjamin Coleman, Wang-Cheng Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. 2024. https://arxiv.org/abs/2402.09668 How to train data-efficient llms . arXiv preprint arXiv:2402.09668

    2024 arXiv

  22. [30]

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. 2024. https://arxiv.org/abs/2409.10173 jina-embeddings-v3: Multilingual embeddings with task...

    2024 arXiv

  23. [31]

    Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. https://arxiv.org/abs/2412.02595 Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset . arXiv preprint ...

    2024 arXiv

  24. [32]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, ...

    2025 arXiv

  25. [33]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  26. [34]

    Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher R \' e , Irina Rish, and Ce Zhang....

    2024

  27. [35]

    Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. 2024. https://openreview.net/forum?id=GLGYYqPwjy Qurating: Selecting high-quality data for training language models . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-...

    2024

  28. [36]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al - Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the Conference of the North American Chapter of the Association fo...

    2021

  29. [37]

    Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. 2024. https://arxiv.org/abs/2412.04506 Arctic-embed 2.0: Multilingual retrieval without compromise . arXiv preprint arXiv:2412.04506

    2024 arXiv

  30. [38]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)

    2019

  31. [39]

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference ...

    2024

  32. [40]

    Ranchi Zhao, Zhen Leng Thai, Yifan Zhang, Shengding Hu, Jie Zhou, Yunqi Ba, Jie Cai, Zhiyuan Liu, and Maosong Sun. 2024. Decoratelm: Data engineering through corpus rating, tagging, and editing with language models. In EMNLP , pages 1401--1418. Association for Computational Li...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.