{"id":"016cf17d-15d1-4b40-b855-7615d3ab309b","arxiv_id":"2607.15209","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"VEXMLM, an XLM-R variant with 30,000 Ge'ez-derived subwords and continued pretraining, improves Amharic/Tigrinya QA by 21 EM points and SA accuracy by 3 points over XLM-R.","lead":"The paper extends XLM-R with Ge'ez-script subword tokens and continued pretraining, reporting big gains on Amharic and Tigrinya question answering and sentiment analysis. A generalist would read it to see if vocabulary expansion plus continued pretraining can rescue low-resource, non-Latin languages in large multilingual models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation table contradicts its own text: the claimed +7.1 point gain from continued pretraining is arithmetically impossible given the four rows in Table 5.","rationale":"The reader identifies several issues (embedding-initialization ambiguity, reviewer note left in text, missing macro-F1) and issues a CONDITIONAL verdict. I agree with CONDITIONAL, but the most load-bearing technical concern is the internal contradiction in the ablation table and its surrounding text. The paper's central claim is that the two-stage training strategy (vocabulary expansion + mean init + continued pretraining) yields large gains, and §5.3 is the only place where the components are isolated. The claimed +7.1 for continued pretraining is contradicted by the table's own numbers: 96.1→97.3 is +1.2, 97.3→97.8 is +0.5, and 97.8→98.2 is +0.4. The text claims random init gives +1.2 (consistent with the table), mean init gives +2.3 over random (contradicted by +0.5), and continued pretraining gives +7.1 (contradicted by +0.4). This is not a trivial typo; the discrepancy changes which component is credited with the gains. If the table is correct, the conclusion 'continued MLM pretraining contributes most' is unsupported, and the entire mechanism story weakens. If the text is correct, the table is misreported and the baseline differs from what is shown. Either way, the published document cannot support the stated attribution. This is more concrete than the abstract-vs-Eq.2 initialization mismatch because either initialization variant can be checked against code, whereas the ablation inconsistency directly undermines the paper's own evidence for the headline mechanism. I would keep CONDITIONAL because the issues are fixable by releasing/running code and correcting numbers, and the main QA/SA results may survive—but they cannot be accepted as-is.","tokens_in":14890,"tokens_out":1937,"duration_ms":13994,"concrete_test":"Inspect the released code (github.com/hailaykidu/VEXMLM) and rerun the Tigrinya NER ablation: (a) reproduce Table 5 by evaluating the four configurations at matched checkpoints; (b) if the table is correct, recompute the incremental gains and update §5.3 to match; (c) if the text's +7.1 is correct, the table's baseline or row ordering is wrong. Also verify whether the reported 98.2 full-model result corresponds to Table 3's Tigrinya OOV 98.2; if so, the ablation rows after random/mean init are redundant with the main result and the +7.1 claim requires a different baseline than 96.1. This single re-run settles whether the component attribution in the central claim is accurate.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim—that continued MLM pretraining is the largest single contributor (+7.1 points) and that the full pipeline reaches 98.2% OOV accuracy—is not supported by the paper's own ablation table. Table 5 reports: XLM-R baseline 96.1, +VocabExp (Random Init) 97.3, +VocabExp (Mean Init) 97.8, +Continued Pretraining 98.2. The increments are: random init +1.2, mean init +0.5, continued pretraining +0.4. But §5.3 claims random init gives +1.2, mean init yields a further +2.3 over random, and continued pretraining provides the largest single gain (+7.1). These numbers cannot all be true. The claimed +7.1 for continued pretraining would require a baseline near 91.1, not 96.1. The text also says full VEXMLM achieves the best OOV accuracy at 98.2, which matches the table's last row, but the component attributions are inconsistent. Because this ablation is the only direct evidence isolating the contribution of the method's novel components, the contradiction means the paper's mechanism story—that continued pretraining drives the gains—is not currently supported by the reported data. The downstream QA/SA headline numbers could still be real, but the attribution of those gains to specific pipeline components is unverifiable from the manuscript as written.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VEXMLM, an XLM-R variant whose vocabulary is extended with 30,000 SentencePiece subword tokens trained on Amharic and Tigrinya. New embeddings are claimed to be initialized by averaging source embeddings; the model is then trained with continued masked language modeling and fine-tuned on QA, NER, and sentiment analysis. The authors report large gains over XLM-R on Amharic/Tigrinya QA (EM 0.66→0.87), sentiment accuracy (0.77→0.80), and average NER OOV-token accuracy (81.4%→94.3% across 11 languages), and they claim transfer to 19 African languages. An ablation on Tigrinya NER is intended to attribute the gains to vocabulary expansion, mean initialization, and continued pretraining.","tokens_in":15234,"tokens_out":7621,"duration_ms":63410,"significance":"If the reported results are reproducible, the paper addresses a genuine problem: Latin-script-centric tokenizers fracture Ge'ez-script languages, and targeted vocabulary extension plus continued pretraining is a practical and inexpensive fix. The public code/model/demo links are a strength, and the evaluation spans a useful set of African languages and tasks. However, the manuscript currently contains a direct numerical inconsistency in the central ablation, a conflict between the abstract and Eq. (2) for embedding initialization, and an empty appendix table. These issues prevent the reader from verifying the paper's main attribution claim and must be fixed before the contribution can be accepted.","major_comments":[{"comment":"The ablation text and Table 5 are arithmetically inconsistent. The table rows (96.1, 97.3, 97.8, 98.2) imply increments of +1.2, +0.5, and +0.4 for random init, mean init, and continued pretraining, respectively. The text, however, claims mean init gives +2.3 over random init and continued pretraining gives the largest single gain of +7.1. The +7.1 value is not derivable from any row pair in this table; it would require a baseline near 91.1, which is the Amharic OOV number in Table 3, not the Tigrinya baseline. Because Table 5 is the only direct evidence isolating component contributions, the central claim that continued pretraining is the most important component is currently unsupported. Please correct the text or table and report the actual component-wise increments.","section":"§5.3, Table 5"},{"comment":"The description of embedding initialization is internally inconsistent. The abstract states that new embeddings are initialized by averaging the embeddings of constituent subwords under XLM-R's original tokenizer, which is the WECHSEL-style constituent averaging. In contrast, Eq. (2) defines each new embedding as the mean of all source embeddings in V_s. These are different procedures, and the behavior of the released model depends on which one was implemented. This ambiguity affects both reproducibility and the interpretation of the ablation. Please specify the exact algorithm and align the prose, equations, and code.","section":"Abstract; §3.3, Eq. (2)"},{"comment":"The appendix promises macro-F1 scores for NER 'alongside accuracy,' but Table 7 contains only a header and the text 'Full results pending model outputs.' This is an incomplete result section, not a finished evaluation. Since §7 explicitly says accuracy should be considered together with macro-F1, the omission is load-bearing for the NER claims. The table must be populated before the manuscript can be considered complete.","section":"Appendix B, Table 7"},{"comment":"The paper's 19-language framing conflicts with the fact that vocabulary augmentation targets only Ge'ez-script Amharic and Tigrinya. The largest OOV gains in Table 3 occur in non-Ge'ez languages (e.g., Swahili +15.0, Kinyarwanda +8.5), while the target languages gain only +1.1 and +2.1. Section 5.4 and §7 acknowledge these gains are a by-product of continued pretraining, not of the vocabulary component. The unedited note following §6 also tells the authors to clarify exactly this point. As written, the paper presents a 19-language average as evidence for a Ge'ez-specific intervention. Please report Ge'ez-script and non-Ge'ez-script results separately and align the claims with the acknowledged mechanism.","section":"§5.4/§7 and note following §6"}],"minor_comments":[{"comment":"The caption says 'not directly comparable to the overall NER accuracy reported in Table 3,' but the table itself is numbered Table 3. The reference should point to the downstream NER table (e.g., Table 4 or the appendix macro-F1 table).","section":"Table 3"},{"comment":"The parity interpretation is inconsistent. The text says values closer to 1.0 indicate more equitable tokenization, but then calls VEXMLM's very low Tigrinya parity (0.27) 'desirable' because it is compact. If compactness is the goal, the metric should be defined and interpreted as efficiency, not parity/equity.","section":"§5.1, Table 2"},{"comment":"The last row of Table 5 is labeled '+ Continued Pretraining,' while §4.5 defines the fourth configuration as 'VEXMLM (Full): Mean init + continued pretraining + fine-tuning.' Clarify whether the last row includes all previous components or only continued pretraining on top of the mean-initialized model.","section":"§4.5, Table 5"},{"comment":"The language table lists 19 languages, but the NER OOV analysis in Table 3 covers only 11 and the appendix macro-F1 table is empty. Please make clear, for each task, how many languages are actually evaluated and why the remaining languages are excluded.","section":"Appendix A, Table 6"},{"comment":"Some references are malformed. For example, the XLM-R entry appears as 'Kartikay Conneau, Alexis Workshop Khandelwal...', and the Wang et al. (2019) entry contains duplicated text. Please clean up the reference list.","section":"References"},{"comment":"The unedited internal note beginning 'One accuracy point worth double-checking before you finalize...' should be removed. It is not scholarly prose and should not appear in a submission.","section":"After §6"}],"recommendation":"major_revision","confidential_remarks":"The arithmetic inconsistency in §5.3 is severe enough that I would ask the authors to provide the raw experimental logs for the ablation and downstream tables before a final decision. Also, TIGQA, one of the two QA benchmarks, was authored by the first author; I do not consider this circular because the dataset is published, but the authors should disclose this relationship explicitly in the data description. Finally, the presence of an unedited author note after Section 6 suggests the manuscript was not proofread before submission; please ensure this is removed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper has a solid core idea but is not ready for publication. The authors extend XLM-R with 30k Ge'ez-script subwords for Amharic and Tigrinya, then continue MLM pretraining and fine-tune on QA, NER, SA. They report big QA gains (EM 0.87 vs 0.66) and a 12.9-point average OOV accuracy improvement on NER. The approach is a combination of known techniques — vocabulary expansion, WECHSEL-style embedding initialization, continued pretraining — applied to a new script family, and that combination is sensible and worth testing.\n\nWhat's good: the evaluation spans 19 languages, includes intrinsic tokenization metrics, and the comparison with Glot500 is informative. The authors also acknowledge that their vocabulary expansion only directly helps Ge'ez-script languages and that cross-lingual gains likely come from continued pretraining. That honesty is a point in their favor.\n\nBut there are serious problems. The ablation table (Table 5) reports XLM-R baseline 96.1, +vocab random 97.3, +mean 97.8, +continued pretraining 98.2. The text in §5.3 claims random init gives +1.2, mean init +2.3 over random, and continued pretraining +7.1. The actual increments are +1.2, +0.5, and +0.4. The +7.1 claim is arithmetically incompatible with the table. This is the only isolation of the continued-pretraining contribution, so the mechanism story — that continued pretraining drives the gains — is unsupported by the reported data. Second, the abstract describes initialization as averaging the embeddings of constituent subwords, while Eq. (2) defines it as the mean of all source embeddings. Those are different procedures. The downstream claims depend on which was actually used. Third, Table 7, meant to report NER macro-F1, is empty ('Full results pending model outputs'). And there's a reviewer note left in the conclusion of the manuscript ('One accuracy point worth double-checking before you finalize...'). These are not cosmetic problems; they mean the paper as submitted does not support its own claims.\n\nThe QA and SA headline numbers could still be real. The issues are fixable. But the current manuscript is not reliable, and a referee would have to do the authors' clean-up work. The evaluation on TIGQA, a dataset authored by the first author, is a minor concern but not disqualifying given it's published.\n\nIf this comes across your desk, send it back with a clear request: reconcile the ablation text with the table, specify the initialization procedure and verify it against the released code, fill in Table 7, and strip out the reviewer note. Then it could be a decent empirical contribution for the multilingual NLP community. As is, I would not trust the component attributions, but the core question is worth engaging with. I'd accept it for peer review only with the expectation of major revision.","headline":"Sensible vocabulary-expansion idea and plausible headline numbers, but the ablation table contradicts its own text and the method is described two different ways — needs major revision before it can be trusted.","tokens_in":15755,"tokens_out":2951,"would_cite":false,"duration_ms":21913,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Extending XLM-R with 30,000 Ge'ez-script subwords and continued masked-language-model training raises question-answering exact match from 0.66 to 0.87 for Amharic and Tigrinya, and improves OOV-token NER accuracy from 81.4% to 94.3% on aver","keywords":["Ge'ez script","Amharic","Tigrinya","vocabulary expansion","subword tokenization","multilingual language models","low-resource African languages","continued pretraining"],"falsifier":"Inspect the released code's embedding initialization. If the new embeddings are set to the mean of all original 250K embeddings, Eq. (2) is what ran; if they are set to the average of constituent subword embeddings, the paper's stated method is not the one that produced the numbers, and the Table 5 ablation labeled 'Mean Init' does not test what it claims.","tokens_in":14743,"feed_emoji":"🌍","tokens_out":9661,"duration_ms":67528,"temperature":0.7,"pith_summary":"The paper claims that the performance gap of multilingual language models on Ge'ez-script languages can be substantially closed by a targeted, low-cost adaptation: expanding XLM-R's vocabulary with 30,000 script-specific subwords and continuing masked-language-model training on Amharic and Tigrinya text before task fine-tuning. On question answering this raises exact match from 0.66 to 0.87 and F1 from 0.78 to 0.90; on sentiment analysis accuracy rises from 0.77 to 0.80; and on named-entity recognition, accuracy on out-of-vocabulary tokens increases from 81.4% to 94.3% on average across 11 African languages. The authors argue the gains carry over to 17 other African languages that received no vocabulary expansion, because continued pretraining strengthens general morphological representations. A sympathetic reader would care because the approach is cheap, backward-compatible with the original model, and offers a template for other underserved writing systems.","feed_headline":"30K new tokens lift Ge'ez-script Q&A to 87% exact match","feed_subtitle":"Gains hold across 17 other African languages, showing script-focused tokenization narrows the low-resource gap.","key_machinery":"The central object is the extended tokenizer and embedding matrix: a SentencePiece tokenizer trained on curated Amharic and Tigrinya corpora contributes 30,000 new subword tokens (50K units for Tigrinya, 32K for Amharic, deduplicated against the original 250K), and each new embedding is initialized as the mean of all source embeddings to keep the new tokens aligned with the existing representation space. The second moving part is two-stage training: continued masked language modeling over the extended vocabulary, which the ablations identify as the largest driver of gains, followed by task-specific fine-tuning that updates only the head and embedding layer. The mechanism as a whole mitigates","core_discovery":"VEXMLM is an XLM-R variant whose original 250K-token vocabulary is extended with 30,000 Ge'ez-script subwords produced by language-specific SentencePiece tokenizers for Amharic and Tigrinya. New token embeddings are initialized to the centroid of the source embedding space (Eq. 2), then the whole model is trained with the masked-language-model objective on monolingual corpora before being fine-tuned on QA, NER, and sentiment tasks. The reported results: 87.0 EM / 90.0 F1 on Amharic/Tigrinya QA vs. 66.0 / 78.0 for the base XLM-R; 80.0% sentiment accuracy vs. 77.0%; and average OOV-token NER accuracy rising from 81.4% to 94.3% across 11 languages. Ablations on Tigrinya NER indicate that contin","pith_inferences":["The paper's own ablations imply that continued MLM training – not the script-specific vocabulary itself – is responsible for most of the cross-lingual gains; a natural next experiment would isolate vocabulary expansion from continued pretraining on a non-Ge'ez language to confirm that attribution.","Because the 30K-token expansion is script-agnostic in spirit, the same recipe should transfer to other high-OOV abugida or syllabary scripts (e.g., Devanagari, Bengali, Thai), where Latin-centric tokenizers cause similar fragmentation.","The unresolved initialization ambiguity (Eq. 2 vs. constituent averaging) means the reported mean-init advantage could be an artifact of the implementation; re-running the ablation with both initializations on the released code would settle which mechanism actually helps convergence."],"forward_implications":["QA exact match on Amharic/Tigrinya rises from 0.66 to 0.87 and F1 from 0.78 to 0.90, a 21-point EM gain over the unmodified base model.","Sentiment accuracy improves from 0.77 to 0.80, and OOV-token NER accuracy from 81.4% to 94.3% on average across 11 languages.","The 30,000-token expansion preserves all original vocabulary entries, so existing languages suffer no regression in coverage.","Ablations show continued MLM pretraining is the largest single driver of the OOV gains, with mean initialization helping convergence relative to random initialization.","The design is parameter-efficient: vocabulary grows from 250K to 280K and parameters from 279M to 301M."],"fun_headline_variants":["Ge'ez vocab extension lifts Amharic/Tigrinya QA to 87 EM","30k new tokens make XLM-R stronger on 19 African languages","OOV-token NER accuracy improves from 81% to 94% with Ge'ez tokens","Script-wise tokenizer boosts sentiment accuracy to 80% for Ge'ez","Transferable gains: Ge'ez vocab extension aids 17 other languages"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole performance story rests on the training pipeline being exactly as described, but the paper gives two different recipes for initializing the new embeddings: the abstract says each token's embedding is the average of its constituent subwords' embeddings, while Section 3.3 Eq. (2) says every new token gets the mean of all 250K source embeddings; the reported results cannot be attributed to a uniquely specified method until that is resolved.","fun_headline_variants_meta":{"raw":{"variants":["Ge'ez vocab extension lifts Amharic/Tigrinya QA to 87 EM","30k new tokens make XLM-R stronger on 19 African languages","OOV-token NER accuracy improves from 81% to 94% with Ge'ez tokens","Script-wise tokenizer boosts sentiment accuracy to 80% for Ge'ez","Transferable gains: Ge'ez vocab extension aids 17 other languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001051,"raw_usage":{"total_tokens":4401,"prompt_tokens":1041,"completion_tokens":3360,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":785,"completion_tokens_details":{"reasoning_tokens":3252}},"tokens_in":785,"tokens_out":3360,"duration_ms":20570,"temperature":1.0,"reasoning_tokens":3252,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:49:45.069565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the released code's embedding initialization. If the new embeddings are set to the mean of all original 250K embeddings, Eq. (2) is what ran; if they are set to the average of constituent subword embeddings, the paper's stated method is not the one that produced the numbers, and the Table 5 ablation labeled 'Mean Init' does not test what it claims.","supporting_citations":[],"review_version":1}