{"id":"58bff4c0-f8e5-4c1a-a5c7-616d7ad63bb8","arxiv_id":"2411.08868","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CamemBERTv2 and CamemBERTav2, trained on 275B tokens with an improved tokenizer, outperform prior French encoder models on several downstream tasks.","lead":"Researchers trained two updated French language models, CamemBERTv2 and CamemBERTav2, on a much larger and more recent text corpus than the original CamemBERT. The models show clear gains on several French NLP benchmarks, especially question answering and medical text tasks, and are released openly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's plural claim that both new models 'vastly outperform' predecessors is contradicted by CamemBERTv2's regression on FQuAD and XNLI; the strongest evidence only supports CamemBERTav2.","rationale":"The reader's verdict is CONDITIONAL, and my concern supports that conditionality without moving the verdict. The most load-bearing issue is not the un-audited corpus or hyperparameter comparability, though both are plausible secondary risks; it is the mismatch between the paper's headline claim and its own reported numbers. The central claim, as stated in the abstract and conclusion, covers both models, yet CamemBERTv2 shows no gain or a regression on FQuAD, XNLI, and most POS/parsing scores. Because the contradiction is internal to the reported tables, it does not depend on external assumptions about the corpus or on rerunning baselines. The fix is a scope revision: attribute the 'vastly outperform' language to CamemBERTav2, and describe CamemBERTv2's improvements as task-dependent. This preserves the paper's contribution and model releases, so the appropriate outcome remains a conditional acceptance pending revision, matching the reader's verdict.","tokens_in":11814,"tokens_out":5010,"duration_ms":46609,"concrete_test":"Compute, from Tables 1-3, the sign and significance (mean difference versus pooled standard error) of CamemBERTv2 versus CamemBERT on every task. If v2 is worse on FQuAD and XNLI while not being better on POS/parsing, the abstract and conclusion must be revised to avoid the plural 'vastly outperform' claim. This is an analytic check requiring no new experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2 (FQuAD) reports CamemBERTv2 at 80.39 F1 / 61.35 EM versus CamemBERT at 80.98 / 62.51, so the v2 model is numerically worse on both metrics. Table 3 (FLUE) reports CamemBERTv2 at 81.75 XNLI versus 81.95 for CamemBERT, again worse. Table 1 shows no consistent gain on POS/dependency parsing, where differences are within one standard deviation. The abstract nevertheless states that 'these updated models vastly outperform their predecessors,' and the conclusion repeats the plural claim. This is an internal inconsistency in the paper's own evidence. The reader's weakest_assumption about corpus and tuning is legitimate, but it is not needed to see this problem: even granting the baselines and hyperparameters, the plural claim fails for CamemBERTv2 on several public benchmarks. The defensible claim is narrower: CamemBERTav2 consistently outperforms, and CamemBERTv2 improves on domain-specific NER and some classification tasks while regressing on QA and XNLI. The paper should either restrict 'vastly outperform' to av2 or qualify it as task-dependent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces two French encoder models, CamemBERTv2 (RoBERTa-style, MLM objective) and CamemBERTav2 (DeBERTaV3-style, RTD objective), trained on a substantially larger French corpus assembled from CulturaX, Wikipedia, and HALvesting, with a new WordPiece tokenizer, longer context length, and a two-stage pre-training protocol. The models are evaluated on general-domain French tasks (POS tagging, dependency parsing, FTB NER, FQuAD QA, and FLUE classification) and on domain-specific NER (medical datasets and a radicalization counter-NER dataset), with multi-seed results and standard deviations. The authors claim that both updated models 'vastly outperform' their predecessors and release all models and intermediate checkpoints on Hugging Face.","tokens_in":12087,"tokens_out":3522,"duration_ms":35275,"significance":"If the stated results hold, this is a useful contribution to French NLP: it provides updated, openly available encoder models that address temporal concept drift, and it offers an interesting comparison of MLM versus RTD pre-training at comparable scale and data. The multi-seed evaluation with confidence intervals is a strength, as is the inclusion of domain-specific medical and counter-radicalization benchmarks and the public release of fine-tuned and intermediate checkpoints. However, the paper's headline claim is broader than its own evidence, and the fine-tuning protocol is incompletely documented, which weakens the reproducibility and fairness of the comparison in its current form.","major_comments":[{"comment":"The abstract and conclusion state that the updated models 'vastly outperform their predecessors' as a plural claim covering both CamemBERTv2 and CamemBERTav2. This is contradicted by the paper's own results: Table 2 shows CamemBERTv2 at 80.39 F1 / 61.35 EM on FQuAD versus 80.98 / 62.51 for CamemBERT, and Table 3 shows CamemBERTv2 at 81.75 XNLI versus 81.95 for CamemBERT. On POS tagging and dependency parsing (Table 1), differences are within one standard deviation for several datasets. A defensible claim would restrict the 'vastly outperform' wording to CamemBERTav2, or explicitly qualify it as task-dependent with CamemBERTv2 improving on domain-specific NER and some classification tasks while regressing on QA and XNLI.","section":"Abstract and §5 Conclusion"},{"comment":"The paper states that hyper-parameter tuning was performed on all models and datasets, but Table 7 is titled 'Hyperparameter Search During Fine-tuning of CamemBERTv2' and only reports one search grid. No equivalent details are given for CamemBERTav2 or for the baseline models CamemBERT, CamemBERTa, and CamemBERT-bio. It is therefore unclear whether the same hyperparameters were applied to all models, whether per-model search was performed and only summarized, or whether baseline numbers were taken from prior publications. This is load-bearing for the comparison's fairness and for reproducibility. The authors should either provide per-model search configurations or explicitly state that the Table 7 grid was reused, and justify that choice.","section":"§4.1 and Table 7"},{"comment":"The temporal-drift motivation rests on the assumption that the 265B-token French subset of CulturaX is a clean and representative sample of contemporary French. The paper does not report any audit of this corpus: no temporal distribution, no domain breakdown, no analysis of the effect of the CulturaX filtering steps (language filters, URL blocklists, perplexity filters), and no overlap/contamination check against the downstream evaluation sets. Given that the central 'up-to-date' claim depends on the corpus, the authors should provide at least basic corpus statistics and a contamination check, or temper the temporal-drift narrative accordingly.","section":"§3.1"}],"minor_comments":[{"comment":"The sentence 'struggle to understand or generate accurate responses' is imprecise for encoder-only models like CamemBERT, which do not generate text.","section":"§1"},{"comment":"The table header reads 'CamemBERTa and CamemBERT 2.0' but the models described are CamemBERTav2 and CamemBERTv2; the naming should be consistent.","section":"Table 6"},{"comment":"The reference for RoBERTa is given as 'Liu, 2019' in the text and 'Liu et al., 2020' in the bibliography; these should be unified.","section":"References"},{"comment":"There are several typographical inconsistencies, including 'emojies', 'reseach', and inconsistent capitalization of 'CulturaX'/'Culturax'.","section":"Throughout"},{"comment":"The phrase 'for comparison with his model' should be 'their model' for consistency and inclusivity.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid model-release contribution with a clear experimental setup, but the abstract and conclusion overstate the evidence for CamemBERTv2, and the fine-tuning protocol needs explicit per-model documentation. These issues are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a practical model-release paper, not a research breakthrough. You get two new French encoders—CamemBERTav2 (DeBERTaV3/RTD) and CamemBERTv2 (RoBERTa/MLM)—trained on a much larger, more recent corpus (275B tokens vs 32B) with a new tokenizer that handles newlines, tabs, emojis, and two-digit numbers. The models are released with intermediate checkpoints, and the evaluation covers general and medical NER, QA, classification, and parsing with multiple seeds and standard deviations. That is genuinely useful for the French NLP community. The medical NER result, where av2 matches a specialized biomedical model, is the most interesting finding.\n\nThe main problem is the overclaim. The abstract and conclusion say both models 'vastly outperform' their predecessors, but the paper's own tables contradict that for v2. On FQuAD, CamemBERTv2 scores 80.39 F1 / 61.35 EM versus CamemBERT's 80.98 / 62.51—worse. On XNLI, v2 gets 81.75 versus 81.95. POS and dependency parsing differences are within one standard deviation. The defensible claim is narrower: av2 consistently improves across tasks, while v2 improves on domain-specific NER and some classification but regresses on QA and XNLI. That is an internal inconsistency that needs fixing.\n\nThere is also an incomplete reporting issue. The paper says hyperparameter tuning was performed on all models, but Table 7 only gives the search details for v2, and the text implies those settings were then applied to av2. That may disadvantage av2, or it may not—but the lack of detail makes it hard to judge. The reliance on CulturaX without auditing its filtering is a minor concern, not a fatal one; it is a standard corpus and the improvements are mostly plausible.\n\nThe paper is honest about saturation in parsing, which I appreciate. The core results are reproducible and the models are worth having. But the writing needs to align the claims with the data. This deserves a serious referee—send it to peer review, with a request for a revised abstract and fuller hyperparameter reporting.","headline":"Useful new French encoders with a solid evaluation, but the abstract's plural 'vastly outperform' is contradicted by CamemBERTv2's regressions on FQuAD and XNLI.","tokens_in":12593,"tokens_out":2204,"would_cite":false,"duration_ms":20334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CamemBERTav2, a French language model trained with Replaced Token Detection on a 275B-token corpus from 2024, beats its 2019 predecessors by several points on named-entity recognition, question answering, and text classification, and…","keywords":["French language model","CamemBERT","DeBERTaV3","Replaced Token Detection","tokenizer","temporal concept drift","named entity recognition","question answering"],"falsifier":"Re-run CamemBERT and CamemBERTa under the fine-tuning protocol in Table 7 on the same splits and seeds; if either old model then matches or exceeds CamemBERTav2's FQuAD F1 of 83.04 or FTB NER F1 of 93.40, the claim that the new data, tokenizer, and RTD objective drive the gains would fail. Alternatively, an audit of the CulturaX French subset for duplicated or mislabeled documents would test the corpus-cleanliness assumption.","tokens_in":11677,"feed_emoji":"🇫🇷","tokens_out":8732,"duration_ms":64910,"temperature":0.7,"pith_summary":"This paper claims that temporal concept drift has made the 2019-era French encoder CamemBERT outdated, and that retraining on a much larger, more recent corpus restores state-of-the-art performance. The authors present CamemBERTv2, an MLM-based RoBERTa-style model, and CamemBERTav2, an RTD-based DeBERTaV3-style model, trained on 275B tokens with a modernized tokenizer and longer 1024-token contexts. The central result is that both new models 'vastly outperform' their predecessors, with CamemBERTav2 leading on French NER (93.40 F1), FQuAD question answering (83.04 F1), and text classification, and matching the specialized CamemBERT-bio on biomedical NER (73.98 vs 73.96 F1). If these results hold, French NLP gains an efficient encoder that stays current with contemporary language while remaining a drop-in replacement for the widely used CamemBERT.","feed_headline":"CamemBERTav2 tops NER, QA, and text classification for French","feed_subtitle":"The DeBERTaV3-based model gains 3+ F1 points on question answering and nears a biomedical specialist on medical NER.","key_machinery":"The load-bearing mechanism is Replaced Token Detection (RTD) as implemented in DeBERTaV3 with gradient-disentangled embedding sharing: during pre-training, a small generator proposes replacement tokens and the model learns to classify each position as original or replaced, so the loss gradients flow from every token rather than only from masked ones. This is paired with a two-stage schedule that first trains at 512-token sequence length and then continues at 1024 tokens, an updated WordPiece tokenizer that treats French elisions (l', lorsqu') as single tokens, splits numbers into at most two-digit tokens, and adds newline, tab, and emoji tokens, and a 9x larger corpus (275B tokens vs 32B). The RTD objective is what lets CamemBERTav2 reach peak performance in one epoch, giving it a sample-efficiency advantage over the MLM-trained CamemBERTv2, which needs three epochs at a 40% masking rate.","core_discovery":"On the paper's own terms, the discovery is that scaling up and refreshing a French encoder's pre-training data, tokenizer, and objective produces large, consistent downstream gains. CamemBERTav2 trained for one epoch on 265B tokens from CulturaX's French subset plus HALvesting and Wikipedia reaches 93.40 F1 on FTB named-entity recognition, 83.04 F1 and 64.29 exact match on FQuAD 1.0, and 95.63 / 93.06 / 84.82 accuracy on CLS, PAWS-X, and XNLI — each several points above CamemBERT and CamemBERTa. On five biomedical NER tasks, CamemBERTav2 scores 73.98 F1, statistically tied with CamemBERT-bio (73.96), despite never seeing biomedical data; on a radicalization NER task, CamemBERTv2 and CamemBERTav2 improve over CamemBERT by 3.28 and 5.35 F1 points. POS tagging and dependency parsing show only marginal gains, which the paper reads as a sign that those benchmarks are near saturation for current encoder architectures.","pith_inferences":["The paper attributes gains to 'temporal concept drift,' but it never isolates when the new corpus helps: a controlled comparison on documents from before vs after 2020 would show whether the gains come from newer vocabulary or simply from more diverse data.","The biomedical 'near-parity' is aggregate: on MEDLINE the gap to CamemBERT-bio is 0.44 F1 (68.21 vs 67.77), while on CAS2 CamemBERT-bio leads by 0.65 F1 (82.50 vs 81.85); readers should check per-dataset numbers before assuming the general model fully replaces specialists.","The hyperparameter comparison with published predecessors may be optimistic for the new models, because the paper details fine-tuning search only for CamemBERTv2; re-running the old models under the exact Table 7 protocol would test whether the headline gains are partly tuning artifacts.","A natural extension would be to evaluate the new tokenizer's numeric and emoji handling on arithmetic reasoning and noisy user-generated text, since those capabilities are motivated but not directly measured here."],"forward_implications":["French NLP systems can replace CamemBERT with CamemBERTav2 as a drop-in encoder and gain roughly 3 F1 points on NER, 2 points on QA F1, and 1-3 points on text classification without changing task code.","CamemBERTav2's near-parity with CamemBERT-bio on biomedical NER suggests that a large, fresh general-domain corpus can substitute for domain-specific continual pre-training in at least some specialized settings.","The tokenizer changes — two-digit number splitting, elision-preserving tokens, and emoji support — should improve handling of dates, arithmetic, and social-media text, though the paper only evaluates these indirectly.","The near-saturation of POS and dependency parsing scores implies that further progress on such structural tasks will require architectural changes, task-specific designs, or new benchmarks rather than simply more pre-training data.","Because all intermediate checkpoints are released, practitioners can fine-tune from an epoch count that matches their compute budget instead of always using the final checkpoint."],"supporting_citations":[{"why":"Defines the original CamemBERT model that this paper retrains and beats.","marker":"(Martin et al., 2020)"},{"why":"Introduces CamemBERTa, the direct predecessor and the source of evaluation splits and prior hyperparameters.","marker":"(Antoun et al., 2023)"},{"why":"Supplies the DeBERTaV3 architecture with gradient-disentangled embedding sharing used by CamemBERTav2.","marker":"(He et al., 2021a)"},{"why":"Defines the Replaced Token Detection objective that gives CamemBERTav2 its training signal.","marker":"(Clark et al., 2020)"},{"why":"Defines RoBERTa, the base architecture and MLM setup for CamemBERTv2.","marker":"(Liu, 2019)"},{"why":"Provides the WordPiece tokenization algorithm adopted for the new tokenizer.","marker":"(Devlin et al., 2019)"},{"why":"Provides the 265B-token French subset of CulturaX that forms the bulk of the new pre-training data.","marker":"(Nguyen et al., 2023)"},{"why":"Sets the biomedical NER datasets, splits, and hyperparameters used for the domain-specific comparison, and supplies the CamemBERT-bio model that the new models are compared against.","marker":"(Touchent and de la Clergerie, 2024)"},{"why":"Motivates the 40% masking rate used to train CamemBERTv2.","marker":"(Wettig et al., 2023)"}],"fun_headline_variants":["CamemBERTav2: refreshed French model beats old on NER, QA, classification","New CamemBERTav2 gains 3+ F1 on French QA over previous model","French NLP upgrade: CamemBERTav2 matches biomedical expert on medical NER","CamemBERT gets smarter: av2 surpasses predecessors on French tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 265B-token French slice of CulturaX, filtered by language IDs, URL blocklists, and perplexity scores, is clean and representative enough to help rather than bias the models, and that the older models' published scores are directly comparable to the new models' under identical fine-tuning.","fun_headline_variants_meta":{"raw":{"variants":["CamemBERTav2: refreshed French model beats old on NER, QA, classification","New CamemBERTav2 gains 3+ F1 on French QA over previous model","French NLP upgrade: CamemBERTav2 matches biomedical expert on medical NER","CamemBERT gets smarter: av2 surpasses predecessors on French tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000995,"raw_usage":{"total_tokens":4272,"prompt_tokens":1059,"completion_tokens":3213,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":3124}},"tokens_in":675,"tokens_out":3213,"duration_ms":19450,"temperature":1.0,"reasoning_tokens":3124,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:14:15.185604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run CamemBERT and CamemBERTa under the fine-tuning protocol in Table 7 on the same splits and seeds; if either old model then matches or exceeds CamemBERTav2's FQuAD F1 of 83.04 or FTB NER F1 of 93.40, the claim that the new data, tokenizer, and RTD objective drive the gains would fail. Alternatively, an audit of the CulturaX French subset for duplicated or mislabeled documents would test the corpus-cleanliness assumption.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces CamemBERTa, the direct predecessor and the source of evaluation splits and prior hyperparameters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sets the biomedical NER datasets, splits, and hyperparameters used for the domain-specific comparison, and supplies the CamemBERT-bio model that the new models are compared against."}],"review_version":1}