{"id":"f4fbfd1a-9fbc-4123-8408-2bd2e407632c","arxiv_id":"2606.22722","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Continued pretraining of ModernBERT on curated Portuguese data produces moBERTo, which reports top results on Portuguese retrieval reranking and PLUE-PT benchmarks.","lead":"moBERTo is created by taking the ModernBERT encoder and continuing its pretraining on a large Portuguese corpus of 60 billion tokens. A smart generalist might read it to see practical trade-offs when adapting modern encoders to new languages, including tokenizer choice and long-context handling.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Corpus filtering and benchmark contamination risk not fully ruled out","rationale":"The reader's weakest assumption is exactly the load-bearing point; the full-text placeholder does not alter it because the abstract already flags the corpus and the evaluation as the unverified foundation. No other internal inconsistency (architecture, ablations, or tokenizer claims) appears more critical once the data-integrity premise is granted.","tokens_in":1795,"tokens_out":324,"duration_ms":17615,"concrete_test":"Using the publicly released pretraining dataset, compute 13-gram overlap (or document-level Jaccard) against the test splits of the three retrieval benchmarks and PLUE-PT; if any test document or >0.5 % of its tokens appear in the pretraining corpus, retrain or evaluate the best moBERTo variant on a decontaminated version of those splits and report the change in nDCG@10.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (best average reranking nDCG@10 on three Portuguese retrieval benchmarks plus best PLUE-PT score) rests on the assumption that the 12 B token FineWeb2-derived corpus, after educational/STEM filtering, contains no test-set leakage and is representative enough that continued pretraining genuinely improves the ModernBERT checkpoint rather than memorizing benchmark artifacts. The abstract states the corpus release and the ablation that continued pretraining beats from-scratch, but provides no quantitative overlap statistics, no description of the classifier training data or decision thresholds, and no decontamination protocol for the retrieval test sets.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces moBERTo, a Portuguese adaptation of ModernBERT-base obtained via continued pretraining on a 12-billion-token corpus curated from FineWeb2 and filtered using educational and STEM classifiers (5 epochs over 60 billion tokens total). The architecture is preserved, including rotary embeddings, alternating local-global attention, and flash attention. The best variant combines a Portuguese tokenizer, subword-matching embedding transfer, and long-context post-training at 8192 tokens. It reports the highest average reranking nDCG@10 on three Portuguese retrieval benchmarks plus best PLUE-PT results, with ablations showing continued pretraining outperforms from-scratch training, tokenizer effects on tasks, and benefits of long-context post-training. Model weights and the pretraining dataset are released publicly.","tokens_in":1898,"tokens_out":672,"duration_ms":16700,"significance":"If the empirical claims hold after addressing data-quality details, the work provides a competitive open encoder for Portuguese NLP pipelines, demonstrates practical benefits of continued pretraining for preserving long-context capabilities, and supplies reusable resources (model and 12B-token dataset). The ablations on tokenizer adaptation and long-context phases offer transferable insights for language-specific adaptation of modern encoders. The release of both weights and data strengthens reproducibility.","major_comments":[{"comment":"Data curation section: the manuscript provides no quantitative overlap statistics between the filtered 12B-token corpus and the three Portuguese retrieval test sets, nor any decontamination protocol. This is load-bearing for the central claim of highest nDCG@10, as leakage could artifactually inflate the reported gains.","section":"Data curation / corpus filtering"},{"comment":"Corpus filtering description: no details are given on the training data, architecture, or decision thresholds of the educational and STEM classifiers used to filter FineWeb2. Without these, it is impossible to assess whether the corpus is representative or whether filtering introduces unintended biases that affect downstream retrieval and PLUE-PT results.","section":"Corpus construction"},{"comment":"Evaluation and ablation sections: the abstract and results claim continued pretraining is \"strongly preferable\" to from-scratch training, yet no hyperparameter budgets, data schedules, or statistical significance tests (with error bars) are reported for the from-scratch baseline, undermining the ablation's ability to support the headline performance claims.","section":"Ablation studies"}],"minor_comments":[{"comment":"The abstract states \"60 billion tokens (5 epochs over a 12-billion-token corpus)\"; clarify whether the 60B figure refers to total tokens seen or a different quantity, and ensure this is consistent in the methods section.","section":"Abstract"},{"comment":"Table or figure reporting nDCG@10 averages should include per-benchmark breakdowns and variance estimates to allow readers to judge whether the \"highest average\" is driven by one outlier benchmark.","section":"Results tables"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's fit for a methods-oriented NLP venue is reasonable given the ablation depth and release, but the absence of decontamination details raises a standard data-integrity concern that should be resolved before acceptance."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback. We address each major comment below and indicate the revisions that will be incorporated to improve transparency and reproducibility.","responses":[{"response":"We agree this is a substantive concern for validating the retrieval results. The original manuscript omitted explicit overlap statistics and decontamination details. Since the full pretraining corpus is publicly released, we will add a new subsection reporting quantitative overlap percentages with each of the three Portuguese retrieval test sets along with the decontamination protocol used during curation.","revision_made":"yes","referee_comment":"[Data curation / corpus filtering] Data curation section: the manuscript provides no quantitative overlap statistics between the filtered 12B-token corpus and the three Portuguese retrieval test sets, nor any decontamination protocol. This is load-bearing for the central claim of highest nDCG@10, as leakage could artifactually inflate the reported gains."},{"response":"We acknowledge that the manuscript lacks sufficient detail on the classifiers. In the revised version we will expand the corpus construction section to specify the training data, model architectures, and exact decision thresholds employed for both the educational and STEM classifiers applied to FineWeb2.","revision_made":"yes","referee_comment":"[Corpus construction] Corpus filtering description: no details are given on the training data, architecture, or decision thresholds of the educational and STEM classifiers used to filter FineWeb2. Without these, it is impossible to assess whether the corpus is representative or whether filtering introduces unintended biases that affect downstream retrieval and PLUE-PT results."},{"response":"The from-scratch baseline was trained under a matched token budget and similar hyperparameter schedule to the continued-pretraining runs, but these details and statistical tests were not reported. We will revise the ablation section to include the hyperparameter budgets, data schedules, error bars, and significance tests comparing the from-scratch and continued-pretraining models.","revision_made":"yes","referee_comment":"[Ablation studies] Evaluation and ablation sections: the abstract and results claim continued pretraining is \"strongly preferable\" to from-scratch training, yet no hyperparameter budgets, data schedules, or statistical significance tests (with error bars) are reported for the from-scratch baseline, undermining the ablation's ability to support the headline performance claims."}],"tokens_in":1548,"tokens_out":500,"duration_ms":24291,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that moBERTo gives the field a usable open Portuguese encoder model trained from the ModernBERT checkpoint, and the ablations make a reasonable case for why continued pretraining plus tokenizer swap plus long-context post-training beats the obvious baselines.\n\nWhat stands out is the release of both weights and the 12B-token training corpus on Hugging Face. The paper shows continued pretraining preserves long-context behavior better than training from scratch, that a Portuguese tokenizer helps token-level tasks but hurts retrieval, and that an extra 8k-context phase lifts reranking and NER. Those comparisons are straightforward and the numbers line up with the claimed best average nDCG@10 on the three retrieval sets plus top PLUE-PT score.\n\nThe soft spots are mostly around the data side. The abstract mentions educational and STEM classifiers on FineWeb2 but gives no overlap statistics with the test sets, no threshold details, and no decontamination protocol. Without those, it is hard to rule out that some of the retrieval gains come from reduced leakage rather than genuine language adaptation. The evaluation section also lacks error bars or significance tests on the reported deltas, which makes the ablation claims harder to weigh.\n\nThis paper is for groups that need a strong Portuguese encoder right now or that want a worked example of adapting an existing modern encoder to another language. It is not advancing new methods, but the empirical recipe and the public artifacts are worth having. I would send it to peer review; the core claims are testable once the full methods and data splits are checked, and the resource itself has clear downstream value.","headline":"A practical Portuguese encoder from continued pretraining on ModernBERT, with clear ablations but light on corpus decontamination details.","tokens_in":2378,"tokens_out":391,"would_cite":false,"duration_ms":11160,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Continued pretraining of ModernBERT on a large Portuguese corpus creates an encoder that leads on retrieval and NLU benchmarks.","keywords":["Portuguese NLP","encoder models","continued pretraining","information retrieval","named entity recognition","language model adaptation","long-context modeling"],"falsifier":"A model trained from scratch on the identical corpus achieving equal or higher nDCG@10 on the three retrieval sets and higher PLUE-PT scores would undermine the claimed advantage of continued pretraining.","tokens_in":2701,"feed_emoji":"🤖","tokens_out":670,"duration_ms":24922,"temperature":0.7,"pith_summary":"The paper demonstrates that adapting an existing English encoder model through further training on Portuguese text produces competitive results for information retrieval, document classification, named entity recognition, and natural language understanding. It preserves the base architecture while testing variants that include a language-specific tokenizer and extended context training. A sympathetic reader would care because encoder models continue to serve production pipelines, and this method shows a lower-cost route to language-specific versions than training entirely new models. The ablations indicate continued pretraining maintains long-context strengths more reliably than starting from random weights.","feed_headline":"Continued pretraining yields top Portuguese encoder","feed_subtitle":"ModernBERT adapted on 12 billion Portuguese tokens leads retrieval reranking and PLUE-PT results.","key_machinery":"Continued pretraining of the ModernBERT checkpoint on a filtered Portuguese corpus, optionally combined with tokenizer adaptation and a dedicated long-context post-training phase.","core_discovery":"moBERTo is obtained by continued pretraining of ModernBERT-base on 60 billion tokens drawn from a 12-billion-token Portuguese corpus filtered from FineWeb2 using educational and STEM classifiers. The strongest variant applies a Portuguese tokenizer with subword-matching embedding transfer plus a long-context post-training stage at 8192 tokens, delivering the highest average reranking nDCG@10 across three Portuguese retrieval benchmarks and the best scores on PLUE-PT. Ablations establish that continued pretraining outperforms from-scratch training for long-context retention, that tokenizer adaptation aids token-level tasks while sometimes harming retrieval, and that the extra long-context pha","pith_inferences":["The same continued-pretraining recipe could produce usable encoders for additional languages that possess comparable filtered web corpora.","Modern encoder architectures may transfer across languages more efficiently through targeted adaptation than through full retraining from random initialization.","Production systems needing both retrieval and classification in non-English settings could adopt this style of adaptation to reduce compute costs."],"forward_implications":["Continued pretraining preserves long-context capabilities more effectively than training from scratch.","Tokenizer adaptation improves results on token-level tasks such as NER while sometimes lowering long-context retrieval performance.","A separate long-context post-training phase at 8192 tokens further raises reranking and NER scores.","Encoder-only models remain competitive with larger decoder-only models on discriminative tasks."],"fun_headline_variants":["moBERTo from continued pretraining of ModernBERT on Portuguese","60 billion token continued pretraining for moBERTo Portuguese model","moBERTo uses Portuguese tokenizer with subword embedding transfer","moBERTo long context post training reaches 8192 tokens"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The 12-billion-token Portuguese corpus is representative of real usage and the evaluation benchmarks measure practical performance without hidden biases or leakage.","fun_headline_variants_meta":{"raw":{"variants":["moBERTo from continued pretraining of ModernBERT on Portuguese","60 billion token continued pretraining for moBERTo Portuguese model","moBERTo uses Portuguese tokenizer with subword embedding transfer","moBERTo long context post training reaches 8192 tokens"]},"model":"grok-4.3","cost_usd":0.007704,"raw_usage":{"total_tokens":3574,"prompt_tokens":770,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":77037000,"prompt_tokens_details":{"text_tokens":770,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2736,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":770,"tokens_out":68,"duration_ms":18865,"temperature":1.0,"reasoning_tokens":2736,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T10:05:53.403926+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A model trained from scratch on the identical corpus achieving equal or higher nDCG@10 on the three retrieval sets and higher PLUE-PT scores would undermine the claimed advantage of continued pretraining.","supporting_citations":[],"review_version":1}