Pith. sign in

REVIEW 3 major objections 5 minor 48 references

CamemBERT 2.0: A Smarter French Language Model Aged to Perfection

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CamemBERTav2, a French language model trained with Replaced Token Detection on a 275B-token corpus from 2024, beats its 2019 predecessors by several points on named-entity recognition, question answering, and text classification, and…

desk verdict Useful new French encoders with a solid evaluation, but the abstract's plural 'vastly outperform' is contradicted by CamemBERTv2's regressions on FQuAD and XNLI. read the letter →

arxiv 2411.08868 v1 pith:X47RQZ3Y submitted 2024-11-13 cs.CL

Wissam Antoun , Francis Kulumba , Rian Touchent , Éric de la Clergerie , Benoît Sagot , Djamé Seddah This is my paper · ORCID
classification cs.CL
keywords FrenchlanguagemodelCamemBERTDeBERTaV3ReplacedTokenDetectiontokenizertemporalconceptdriftnamedentityrecognitionquestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that temporal concept drift has made the 2019-era French encoder CamemBERT outdated, and that retraining on a much larger, more recent corpus restores state-of-the-art performance. The authors present CamemBERTv2, an MLM-based RoBERTa-style model, and CamemBERTav2, an RTD-based DeBERTaV3-style model, trained on 275B tokens with a modernized tokenizer and longer 1024-token contexts. The central result is that both new models 'vastly outperform' their predecessors, with CamemBERTav2 leading on French NER (93.40 F1), FQuAD question answering (83.04 F1), and text classification, and matching the specialized CamemBERT-bio on biomedical NER (73.98 vs 73.96 F1). If these results hold, French NLP gains an efficient encoder that stays current with contemporary language while remaining a drop-in replacement for the widely used CamemBERT.

What carries the argument

The load-bearing mechanism is Replaced Token Detection (RTD) as implemented in DeBERTaV3 with gradient-disentangled embedding sharing: during pre-training, a small generator proposes replacement tokens and the model learns to classify each position as original or replaced, so the loss gradients flow from every token rather than only from masked ones. This is paired with a two-stage schedule that first trains at 512-token sequence length and then continues at 1024 tokens, an updated WordPiece tokenizer that treats French elisions (l', lorsqu') as single tokens, splits numbers into at most two-digit tokens, and adds newline, tab, and emoji tokens, and a 9x larger corpus (275B tokens vs 32B). The RTD objective is what lets CamemBERTav2 reach peak performance in one epoch, giving it a sample-efficiency advantage over the MLM-trained CamemBERTv2, which needs three epochs at a 40% masking rate.

What would settle it

Re-run CamemBERT and CamemBERTa under the fine-tuning protocol in Table 7 on the same splits and seeds; if either old model then matches or exceeds CamemBERTav2's FQuAD F1 of 83.04 or FTB NER F1 of 93.40, the claim that the new data, tokenizer, and RTD objective drive the gains would fail. Alternatively, an audit of the CulturaX French subset for duplicated or mislabeled documents would test the corpus-cleanliness assumption.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that scaling up and refreshing a French encoder's pre-training data, tokenizer, and objective produces large, consistent downstream gains. CamemBERTav2 trained for one epoch on 265B tokens from CulturaX's French subset plus HALvesting and Wikipedia reaches 93.40 F1 on FTB named-entity recognition, 83.04 F1 and 64.29 exact match on FQuAD 1.0, and 95.63 / 93.06 / 84.82 accuracy on CLS, PAWS-X, and XNLI — each several points above CamemBERT and CamemBERTa. On five biomedical NER tasks, CamemBERTav2 scores 73.98 F1, statistically tied with CamemBERT-bio (73.96), despite never seeing biomedical data; on a radicalization NER task, CamemBERTv2 and CamemBERTav2 improve over CamemBERT by 3.28 and 5.35 F1 points. POS tagging and dependency parsing show only marginal gains, which the paper reads as a sign that those benchmarks are near saturation for current encoder architectures.

Load-bearing premise

The load-bearing premise is that the 265B-token French slice of CulturaX, filtered by language IDs, URL blocklists, and perplexity scores, is clean and representative enough to help rather than bias the models, and that the older models' published scores are directly comparable to the new models' under identical fine-tuning.

Editorial extensions

If this is right

  • French NLP systems can replace CamemBERT with CamemBERTav2 as a drop-in encoder and gain roughly 3 F1 points on NER, 2 points on QA F1, and 1-3 points on text classification without changing task code.
  • CamemBERTav2's near-parity with CamemBERT-bio on biomedical NER suggests that a large, fresh general-domain corpus can substitute for domain-specific continual pre-training in at least some specialized settings.
  • The tokenizer changes — two-digit number splitting, elision-preserving tokens, and emoji support — should improve handling of dates, arithmetic, and social-media text, though the paper only evaluates these indirectly.
  • The near-saturation of POS and dependency parsing scores implies that further progress on such structural tasks will require architectural changes, task-specific designs, or new benchmarks rather than simply more pre-training data.
  • Because all intermediate checkpoints are released, practitioners can fine-tune from an epoch count that matches their compute budget instead of always using the final checkpoint.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper attributes gains to 'temporal concept drift,' but it never isolates when the new corpus helps: a controlled comparison on documents from before vs after 2020 would show whether the gains come from newer vocabulary or simply from more diverse data.
  • The biomedical 'near-parity' is aggregate: on MEDLINE the gap to CamemBERT-bio is 0.44 F1 (68.21 vs 67.77), while on CAS2 CamemBERT-bio leads by 0.65 F1 (82.50 vs 81.85); readers should check per-dataset numbers before assuming the general model fully replaces specialists.
  • The hyperparameter comparison with published predecessors may be optimistic for the new models, because the paper details fine-tuning search only for CamemBERTv2; re-running the old models under the exact Table 7 protocol would test whether the headline gains are partly tuning artifacts.
  • A natural extension would be to evaluate the new tokenizer's numeric and emoji handling on arithmetic reasoning and noisy user-generated text, since those capabilities are motivated but not directly measured here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces two French encoder models, CamemBERTv2 (RoBERTa-style, MLM objective) and CamemBERTav2 (DeBERTaV3-style, RTD objective), trained on a substantially larger French corpus assembled from CulturaX, Wikipedia, and HALvesting, with a new WordPiece tokenizer, longer context length, and a two-stage pre-training protocol. The models are evaluated on general-domain French tasks (POS tagging, dependency parsing, FTB NER, FQuAD QA, and FLUE classification) and on domain-specific NER (medical datasets and a radicalization counter-NER dataset), with multi-seed results and standard deviations. The authors claim that both updated models 'vastly outperform' their predecessors and release all models and intermediate checkpoints on Hugging Face.

Significance. If the stated results hold, this is a useful contribution to French NLP: it provides updated, openly available encoder models that address temporal concept drift, and it offers an interesting comparison of MLM versus RTD pre-training at comparable scale and data. The multi-seed evaluation with confidence intervals is a strength, as is the inclusion of domain-specific medical and counter-radicalization benchmarks and the public release of fine-tuned and intermediate checkpoints. However, the paper's headline claim is broader than its own evidence, and the fine-tuning protocol is incompletely documented, which weakens the reproducibility and fairness of the comparison in its current form.

major comments (3)
  1. [Abstract and §5 Conclusion] The abstract and conclusion state that the updated models 'vastly outperform their predecessors' as a plural claim covering both CamemBERTv2 and CamemBERTav2. This is contradicted by the paper's own results: Table 2 shows CamemBERTv2 at 80.39 F1 / 61.35 EM on FQuAD versus 80.98 / 62.51 for CamemBERT, and Table 3 shows CamemBERTv2 at 81.75 XNLI versus 81.95 for CamemBERT. On POS tagging and dependency parsing (Table 1), differences are within one standard deviation for several datasets. A defensible claim would restrict the 'vastly outperform' wording to CamemBERTav2, or explicitly qualify it as task-dependent with CamemBERTv2 improving on domain-specific NER and some classification tasks while regressing on QA and XNLI.
  2. [§4.1 and Table 7] The paper states that hyper-parameter tuning was performed on all models and datasets, but Table 7 is titled 'Hyperparameter Search During Fine-tuning of CamemBERTv2' and only reports one search grid. No equivalent details are given for CamemBERTav2 or for the baseline models CamemBERT, CamemBERTa, and CamemBERT-bio. It is therefore unclear whether the same hyperparameters were applied to all models, whether per-model search was performed and only summarized, or whether baseline numbers were taken from prior publications. This is load-bearing for the comparison's fairness and for reproducibility. The authors should either provide per-model search configurations or explicitly state that the Table 7 grid was reused, and justify that choice.
  3. [§3.1] The temporal-drift motivation rests on the assumption that the 265B-token French subset of CulturaX is a clean and representative sample of contemporary French. The paper does not report any audit of this corpus: no temporal distribution, no domain breakdown, no analysis of the effect of the CulturaX filtering steps (language filters, URL blocklists, perplexity filters), and no overlap/contamination check against the downstream evaluation sets. Given that the central 'up-to-date' claim depends on the corpus, the authors should provide at least basic corpus statistics and a contamination check, or temper the temporal-drift narrative accordingly.
minor comments (5)
  1. [§1] The sentence 'struggle to understand or generate accurate responses' is imprecise for encoder-only models like CamemBERT, which do not generate text.
  2. [Table 6] The table header reads 'CamemBERTa and CamemBERT 2.0' but the models described are CamemBERTav2 and CamemBERTv2; the naming should be consistent.
  3. [References] The reference for RoBERTa is given as 'Liu, 2019' in the text and 'Liu et al., 2020' in the bibliography; these should be unified.
  4. [Throughout] There are several typographical inconsistencies, including 'emojies', 'reseach', and inconsistent capitalization of 'CulturaX'/'Culturax'.
  5. [§4.1] The phrase 'for comparison with his model' should be 'their model' for consistency and inclusivity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the new models are trained on an independently assembled corpus and evaluated on external French NLP benchmarks; the self-citations are comparability and training-design choices, not fitted predictions.

full rationale

The paper's derivation chain is empirical and open: it assembles a 275B-token French corpus from CulturaX, Wikipedia and the HALvesting corpus, trains CamemBERTav2 (DeBERTaV3/RTD) and CamemBERTv2 (RoBERTa/MLM), then evaluates both on externally defined French benchmarks (FQuAD, FLUE, UD treebanks, FTB-NER, QUAERO/CAS/E3C, Counter-NER). No downstream benchmark score is used as a training signal, no fitted parameter is renamed as a prediction, and no equation defines one reported result in terms of another. The self-citations are not load-bearing in a circular sense: Antoun et al. (2023) is cited for splits and for the empirical observation that MLM models benefit from multiple epochs, and Touchent and de la Clergerie (2024) is cited for biomedical splits, hyperparameters and the CamemBERT-bio baseline; in all cases the resulting claims are checked by fresh measurements on external datasets. The abstract's plural claim that both models 'vastly outperform' their predecessors is overstated given CamemBERTv2's numerical regressions on FQuAD and XNLI, but that is an internal evidence-consistency problem, not a circularity problem.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or parameters. The free parameters are standard training choices, and assumptions are inherited from prior work and corpus construction.

free parameters (3)
  • Masking probability = 40% for CamemBERTv2, 20% for CamemBERTav2
    Chosen based on prior work (Wettig et al. 2023) rather than new experiments; 40% is adopted from literature.
  • Upsampling factor for Wikipedia = 10x
    Chosen by hand; no ablation provided.
  • Two-digit number splitting = yes
    A design choice hypothesized to help arithmetic; not empirically validated.
assumptions (3)
  • domain assumption CulturaX French subset is a clean and representative sample of contemporary French.
    The paper relies on CulturaX's filtering without auditing it; if biased, downstream scores may not generalize.
  • domain assumption Reusing the same fine-tuning splits and hyperparameters as prior work (Antoun et al. 2023, Touchent and de la Clergerie 2024) yields fair comparisons.
    This assumes the prior protocols are optimal or at least fair for the new models.
  • domain assumption RTD objective is more sample-efficient than MLM and one epoch is sufficient for CamemBERTav2.
    Supported by prior work, but no ablation in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CamemBERT 2.0: A Smarter French Language Model Aged to Perfection." pith.science (2026). https://pith.science/paper/X47RQZ3Y

@misc{pith2026241108868,
  author       = {Pith},
  title        = {Pith review of: CamemBERT 2.0: A Smarter French Language Model Aged to Perfection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X47RQZ3Y}},
  note         = {Machine review of arXiv:2411.08868}
}
read the original abstract

French language models, such as CamemBERT, have been widely adopted across industries for natural language processing (NLP) tasks, with models like CamemBERT seeing over 4 million downloads per month. However, these models face challenges due to temporal concept drift, where outdated training data leads to a decline in performance, especially when encountering new topics and terminology. This issue emphasizes the need for updated models that reflect current linguistic trends. In this paper, we introduce two new versions of the CamemBERT base model-CamemBERTav2 and CamemBERTv2-designed to address these challenges. CamemBERTav2 is based on the DeBERTaV3 architecture and makes use of the Replaced Token Detection (RTD) objective for better contextual understanding, while CamemBERTv2 is built on RoBERTa, which uses the Masked Language Modeling (MLM) objective. Both models are trained on a significantly larger and more recent dataset with longer context length and an updated tokenizer that enhances tokenization performance for French. We evaluate the performance of these models on both general-domain NLP tasks and domain-specific applications, such as medical field tasks, demonstrating their versatility and effectiveness across a range of use cases. Our results show that these updated models vastly outperform their predecessors, making them valuable tools for modern NLP systems. All our new models, as well as intermediate checkpoints, are made openly available on Huggingface.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 22 canonical work pages

  1. [1]

    Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Beno \^ t Sagot. 2022. https://aclanthology.org/2022.lrec-1.463 Towards a cleaner document-oriented multilingual crawled corpus . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4344--4355, Marseille, France. European Language Resources Association

    work page 2022

  2. [2]

    Julien Abadji, Pedro Javier Ortiz Su \'a rez, Laurent Romary, and Beno \^i t Sagot. 2021. https://doi.org/10.14618/ids-pub-10468 Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus . Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick, 12 July 2021 (Online-Event...

    doi:10.14618/ids-pub-10468 2021

  3. [3]

    Anne Abeill \'e , Lionel Cl \'e ment, and Alexandra Kinyon. 2000. http://www.lrec-conf.org/proceedings/lrec2000/pdf/230.pdf Building a treebank for F rench . In Proceedings of the Second International Conference on Language Resources and Evaluation ( LREC ' 00) , Athens, Greece. European Language Resources Association (ELRA)

    work page 2000

  4. [4]

    Oshin Agarwal and Ani Nenkova. 2022. https://doi.org/10.1162/tacl_a_00497 Temporal effects on pre-trained models for language processing tasks . Transactions of the Association for Computational Linguistics, 10:904--921

    doi:10.1162/tacl_a_00497 2022

  5. [5]

    Akani, R

    E. Akani, R. Gemignani, and R. Abrougui. 2023. https://doi.org/10.1049/icp.2023.0986 Enebert: a state-of-the-art language model trained on a corpus of texts generated from the set of dso activities . In 27th International Conference on Electricity Distribution (CIRED 2023), volume 2023, pages 2903--2907

    work page arXiv 2023

  6. [6]

    Wissam Antoun, Beno \^ t Sagot, and Djam \'e Seddah. 2023. Data-efficient french language modeling with camemberta. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada. Association for Computational Linguistics

    work page 2023

  7. [7]

    Marie Candito and Beno \^ t Crabb \'e . 2009. https://aclanthology.org/W09-3821 Improving generative statistical parsing with semi-supervised word clustering . In Proceedings of the 11th International Conference on Parsing Technologies ( IWPT ' 09) , pages 138--141, Paris, France. Association for Computational Linguistics

    work page 2009

  8. [8]

    Marie Candito, Guy Perrier, Bruno Guillaume, Corentin Ribeyre, Karën Fort, Djamé Seddah, and Eric De La Clergerie. 2014. Deep syntax annotation of the sequoia french treebank. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), Reykjavik, Iceland. European Language Resources Association (ELRA)

    work page 2014

Show all 48 references
  1. [9]

    Marie Candito and Djam \'e Seddah. 2012. https://aclanthology.org/F12-2024 Le corpus sequoia : annotation syntaxique et exploitation pour l ' adaptation d ' analyseur par pont lexical (the sequoia corpus : Syntactic annotation and use for a parser lexical domain adaptation met...

    2012

  2. [10]

    Oralie Cattan, Christophe Servan, and Sophie Rosset. 2021. https://hal.science/hal-03336060 On the Usability of Transformers-based models for a French Question-Answering task . In Recent Advances in Natural Language Processing (RANLP) , Varna, Bulgaria

    2021

  3. [11]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. https://openreview.net/pdf?id=r1xMH1BtvB ELECTRA : Pre-training text encoders as discriminators rather than generators . In ICLR

    2020

  4. [12]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  5. [13]

    Martin d'Hoffschmidt, Maxime Vidal, Wacim Belblidia, and Tom Brendlé. 2020. https://arxiv.org/abs/2002.06071 FQuAD: French Question Answering Dataset . arXiv e-prints, arXiv:2002.06071

    2020 arXiv

  6. [14]

    Simon Gabay, Pedro Ortiz Suarez, Alexandre Bartz, Alix Chagu \'e , Rachel Bawden, Philippe Gambette, and Beno \^ t Sagot. 2022. https://aclanthology.org/2022.lrec-1.359 From F re EM to d ' A lem BERT : a large corpus and a language model for early M odern F rench . In Proceedi...

    2022

  7. [15]

    Gemignani, E

    R. Gemignani, E. Akani, J.P. Delrieux, and A. Sayouti Souleymane. 2023. https://doi.org/10.1049/icp.2023.1011 Hape: optimizing customer relation by automatic task distribution using constrained optimization and natural language processing . In 27th International Conference on ...

    2023

  8. [16]

    Cyril Grouin, Natalia Grabar, Vincent Claveau, and Thierry Hamon. 2019. https://doi.org/10.18653/v1/W19-5029 Clinical case reports for NLP . In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 273--282, Florence, Italy. Association for Computational Linguistics

    2019 doi

  9. [17]

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021 a . https://arxiv.org/abs/2111.09543 Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing . Preprint, arXiv:2111.09543

    2021 arXiv

  10. [18]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021 b . https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced bert with disentangled attention . In International Conference on Learning Representations

    2021

  11. [19]

    Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022. https://doi.org/10.18653/v1/2022.bigscience-1.1 Lifelong pretraining: Continually adapting language models to emerging corpora . In Proceedings of BigScience Episode ...

    2022 doi

  12. [20]

    Moussa Kamal Eddine, Antoine Tixier, and Michalis Vazirgiannis. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.740 BART hez: a skilled pretrained F rench sequence-to-sequence model . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, ...

    2021 doi

  13. [21]

    Francis Kulumba, Wissam Antoun, Guillaume Vimont, and Laurent Romary. 2024. https://arxiv.org/abs/2407.20595 Harvesting textual and structured data from the hal publication repository . Preprint, arXiv:2407.20595

    2024 arXiv

  14. [22]

    Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, B \'e atrice Daille, and Pierre-Antoine Gourraud. 2023. https://doi.org/10.18653/v1/2023.acl-long.896 D r BERT : A robust pre-trained model in F rench for biomedical and clinical domains . In Proceed...

    2023 doi

  15. [23]

    Anne Lacheret, Sylvain Kahane, Julie Beliao, Anne Dister, Kim Gerdes, Jean-Philippe Goldman, Nicolas Obin, Paola Pietrandrea, and Atanas Tchobanov. 2014. https://doi.org/10.1051/shsconf/20140801305 Rhapsodie: un Treebank annot \'e pour l' \'e tude de l'interface syntaxe-prosod...

    2014

  16. [24]

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. https://openreview.net/forum?id=H1eA7AEtvS Albert: A lite bert for self-supervised learning of language representations . In International Conference on Learning Representations

    2020

  17. [25]

    Tommasone, Baptiste Pannier, Fran c ois Boniface, Am \'e lie Chatelain, Alessandro Cappelli, Iacopo Poli, and Djam \'e Seddah

    Julien Launay, E.l. Tommasone, Baptiste Pannier, Fran c ois Boniface, Am \'e lie Chatelain, Alessandro Cappelli, Iacopo Poli, and Djam \'e Seddah. 2022. https://aclanthology.org/2022.lrec-1.455 PAG nol: An extra-large F rench generative model . In Proceedings of the Thirteenth...

    2022

  18. [26]

    Hang Le, Lo \" c Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabb \'e , Laurent Besacier, and Didier Schwab. 2020. https://aclanthology.org/2020.lrec-1.302 F lau BERT : Unsupervised language model pre-training for F renc...

    2020

  19. [27]

    Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. 2019. https://arxiv.org/abs/1912.05372 Flaubert: Unsupervised language model pre-training for french . Preprint, arXiv...

    2019 arXiv

  20. [28]

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. https://doi.org/10.18653/v1/2020.acl-main.703 BART : Denoising sequence-to-sequence pre-training for natural language generation, translatio...

    2020 doi

  21. [29]

    Yinhan Liu. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  22. [30]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://openreview.net/forum?id=SyxS0T4tvS Ro \ bert \ a: A robustly optimized \ bert \ pretraining approach

    2020

  23. [31]

    Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022. https://doi.org/10.18653/v1/2022.acl-demo.25 T ime LM s: Diachronic language models from T witter . In Proceedings of the 60th Annual Meeting of the Association for Comput...

    2022 doi

  24. [32]

    Bernardo Magnini, Bego \ n a Altuna, Alberto Lavelli, Manuela Speranza, and Roberto Zanoli. 2020. https://api.semanticscholar.org/CorpusID:229293442 The e3c project: Collection and annotation of a multilingual corpus of clinical cases . Proceedings of the Seventh Italian Confe...

    2020

  25. [33]

    Louis Martin, Benjamin Muller, Pedro Javier Ortiz Su \'a rez, Yoann Dupont, Laurent Romary, \'E ric de la Clergerie, Djam \'e Seddah, and Beno \^ t Sagot. 2020. https://doi.org/10.18653/v1/2020.acl-main.645 C amem BERT : a tasty F rench language model . In Proceedings of the 5...

    2020 doi

  26. [34]

    a ckstr \

    Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar T \"a ckstr \"o m, Claudia Bedini, N \'u ria Bertomeu Castell \'o , and Jungmee Lee. 2013. https://aclanthology.org/P13-2017 U niversa...

    2013

  27. [35]

    Vincent Micheli, Martin d ' Hoffschmidt, and Fran c ois Fleuret. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.632 On the importance of pre-training data volume for compact language models . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro...

    2020 doi

  28. [36]

    Martin M \" u ller and Florian Laurent. 2022. https://arxiv.org/abs/2202.03371 Cedille: A large autoregressive french language model . Preprint, arXiv:2202.03371

    2022 arXiv

  29. [37]

    Aur \'e lie N \'e v \'e ol, Cyril Grouin, Jeremy Leixa, Sophie Rosset, and Pierre Zweigenbaum. 2014. The quaero french medical corpus: a ressource for medical entity recognition and normalization. In Bio text-mining workshop (BioTextM 2014), page 7p, Reykjavik, Iceland

    2014

  30. [38]

    Rossi, and Thien Huu Nguyen

    Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. https://arxiv.org/abs/2309.09400 Culturax: A cleaned, enormous, and multilingual dataset for large language models in 167 languages . Preprin...

    2023 arXiv

  31. [39]

    Pedro Javier Ortiz Su \'a rez , Beno \^i t Sagot, and Laurent Romary. 2019. https://doi.org/10.14618/ids-pub-9021 Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures . Proceedings of the Workshop on Challenges in the Management of Large...

    2019 doi

  32. [40]

    Arij Riabi, Menel Mahamdi, Virginie Mouilleron, and Djam \'e Seddah. 2024. https://aclanthology.org/2024.privatenlp-1.13 Cloaked classifiers: Pseudonymization strategies on sensitive classification tasks . In Proceedings of the Fifth Workshop on Privacy in Natural Language Pro...

    2024

  33. [41]

    Arij Riabi, Beno \^ t Sagot, and Djam \'e Seddah. 2021. https://doi.org/10.18653/v1/2021.wnut-1.47 Can character-based language models improve downstream task performances in low-resource and noisy language scenarios? In Proceedings of the Seventh Workshop on Noisy User-genera...

    2021 doi

  34. [42]

    Beno \^ t Sagot, Marion Richard, and Rosa Stern. 2012. https://aclanthology.org/F12-2050 Annotation r \'e f \'e rentielle du corpus arbor \'e de P aris 7 en entit \'e s nomm \'e es (referential named entity annotation of the P aris 7 F rench T ree B ank) [in F rench] . In Proc...

    2012

  35. [43]

    Djam \'e Seddah, Benoit Sagot, Marie Candito, Virginie Mouilleron, and Vanessa Combet. 2012. https://aclanthology.org/C12-1149 The F rench S ocial M edia B ank: a treebank of noisy user generated content . In Proceedings of COLING 2012 , pages 2441--2458, Mumbai, India. The CO...

    2012

  36. [44]

    Rian Touchent and \'E ric de la Clergerie. 2024. https://aclanthology.org/2024.lrec-main.241 C amem BERT -bio: Leveraging continual pre-training for cost-effective models on F rench biomedical data . In Proceedings of the 2024 Joint International Conference on Computational Li...

    2024

  37. [45]

    Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. https://doi.org/10.18653/v1/2023.eacl-main.217 Should you mask 15 \ In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985--3000, Dubrovnik, Cr...

    2023 doi

  38. [46]

    Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. https://doi.org/10.18653/v1/2021.naacl-main.41 m T 5: A massively multilingual pre-trained text-to-text transformer . In Proceedings of the 2021 Conferenc...

    2021 doi

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.