{"id":"9429ee66-14c8-45dc-aeef-070bb55afe9f","arxiv_id":"2507.04569","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Nile-Chat models for dual-script Egyptian Arabic beat strong baselines on newly translated benchmarks, but the evaluation may be inflated by training/eval data overlap and Claude-generated script data.","lead":"This paper presents Nile-Chat, a family of large language models tuned for Egyptian Arabic in both Arabic and Latin scripts, along with a set of newly translated Egyptian evaluation benchmarks. The authors claim large accuracy gains over leading multilingual models, especially on Latin-script tasks, but the evaluation pipeline raises concerns about benchmark contamination and dependence on a single translation model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark train/test overlap is unverified: the SFT mix explicitly contains a 'Benchmarks Training Set' (Ben-Ar/Ben-Ltn), yet the paper never states these are disjoint from the EgyptianBench test splits used in Tables 1-2.","rationale":"The reader's verdict is CONDITIONAL with high correctness risk, and my independent reading reaches the same condition. The central claim is strong and checkable: the models, datasets, and evaluation harness are publicly released, which is real independent support. However, the weakest point in the argument is precisely the evaluation pipeline. Figure 2 places benchmark-related data inside the training mixture via Ben-Ar and Ben-Ltn, and Section 6 does not document a train/test split for the eight translated benchmarks. The only explicitly held-out split is the 10% reserved in Section 3.2.3 for translation and transliteration data. If benchmark items used in evaluation also appear in SFT, the reported gains are inflated. The proposed overlap test is feasible with the public artifacts and would settle the question. I would not reject the paper on this basis, but acceptance should be conditioned on demonstrating disjointness and on correcting the misleading '14.4% gain' language to percentage points. This does not change the reader's conditional verdict, so I mark the verdict as unchanged.","tokens_in":20976,"tokens_out":3523,"duration_ms":37218,"concrete_test":"Download EgyptianBench and Egyptian-SFT-Mixture from the released repositories. Normalize both Arabic-script and Latin-script text (strip diacritics, normalize alef/teh marbuta, map Arabizi digits to their Arabic equivalents), then compute exact and near-duplicate overlap between every evaluation benchmark's prompts and options and the Ben-Ar/Ben-Ltn subsets of the SFT mixture using MinHash with a Jaccard threshold of 0.8 or higher. Also inspect the dataset cards for Ben-Ar/Ben-Ltn to confirm whether they are the training splits of EgyptianBench and which split Tables 1 and 2 used. If overlap is zero and the splits are explicitly disjoint, the concern is resolved; if any evaluation item appears in training, recompute Tables 1 and 2 after excluding those items and report the resulting margins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Nile-Chat models 'significantly outperform' strong baselines on Egyptian benchmarks—requires that EgyptianBench measures capabilities the models were not trained on. The paper does not establish this. In Figure 2, the instruction mixture includes components labeled 'Ben-Ar (3.97%)' and 'Ben-Ltn (2.33%)', and the caption expands 'Ben' as 'Benchmarks Training Set'. Section 6 describes the benchmarks as newly translated from English MMLU, HellaSwag, PIQA, WinoGrande, OpenBookQA, and RACE, but it never states whether the items in Ben-Ar/Ben-Ltn are disjoint from the items scored in Tables 1 and 2. The only explicit held-out split mentioned anywhere is in Section 3.2.3, which reserves 10% of the translation/transliteration data for evaluation; no analogous split is documented for the eight translated benchmarks. If Ben-Ar and Ben-Ltn contain training splits of those benchmarks, and if the evaluation used the full benchmark or a test split drawn from the same Claude-generated translation pipeline, then the model was trained on items that are identical or near-identical to evaluation items. In that case the reported margins—for example, +20.2 absolute points over Qwen2.5-14B on Latin HellaSwag—would reflect memorization rather than transferable Egyptian Arabic ability. A secondary issue is that the headline '14.4% performance gain' is the average of absolute percentage-point differences across five Latin benchmarks, not a relative gain; it should be reported as percentage points.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Nile-Chat, a family of Egyptian Arabic LLMs (dense 4B and 12B variants plus a 3x4B-A6B MoE built with the Branch-Train-MiX strategy) that support both Arabic script and Latin-script (Arabizi/Franco-Arabic). The authors describe the construction of dual-script continual pre-training, instruction-tuning, and DPO alignment data, including Claude-based transliteration and translation. They also introduce EgyptianBench, a set of translated English benchmarks (MMLU, HellaSwag, PIQA, WinoGrande, OpenBookQA, RACE) in Arabic and Latin scripts, along with translation and transliteration tasks. The central empirical claim is that Nile-Chat models outperform strong multilingual and Arabic-specific baselines, with the 12B model reported to give a 14.4% performance gain over Qwen2.5-14B-Instruct on Latin-script benchmarks.","tokens_in":21299,"tokens_out":6190,"duration_ms":62441,"significance":"If the evaluation is clean, this is a valuable contribution: it is the first dual-script Egyptian Arabic LLM family, the resources (models, datasets, benchmark harness) are publicly released, and the application of Branch-Train-MiX to script-specialized experts is a reasonable and reusable idea. The promise of open weights and an open evaluation fork is a concrete strength. However, the empirical significance depends on the validity of the newly introduced benchmarks and on the claimed margins. As written, the paper does not establish that the evaluation items are disjoint from the 'Benchmarks Training Set' components in the SFT mixture, and it relies on Claude for both training-data generation and benchmark translation. These issues are load-bearing for the central claim and need to be resolved before the reported results can be taken at face value.","major_comments":[{"comment":"The SFT mixture in Figure 2 contains components labeled 'Ben-Ar (3.97%)' and 'Ben-Ltn (2.33%)', with the caption defining 'Ben' as 'Benchmarks Training Set'. Section 6 introduces the eight EgyptianBench benchmarks without stating which splits were used for evaluation and without stating whether the Ben-Ar/Ben-Ltn items are disjoint from the evaluated items. The only explicit held-out split described anywhere is the 10% reservation for translation/transliteration data in Section 3.2.3; no analogous statement is made for the translated benchmarks. This is load-bearing because the abstract's claim that Nile-Chat models 'significantly outperform' baselines requires that the benchmarks measure held-out ability rather than memorization. Please specify the exact sources and split definitions for Ben-Ar/Ben-Ltn, state explicitly whether the EgyptianBench test items were excluded from all training stages (including the DPO data), and report results on any uncontaminated subset if overlap is found.","section":"Figure 2, Section 6"},{"comment":"Claude is used to transliterate pre-training data, to translate/transliterate SFT data, to translate all eight evaluation benchmarks, and to serve as the judge for EgyptianAlpacaEval. The models are therefore evaluated on text generated by the same system used to produce their training data. This does not by itself invalidate the results, but it creates a style and format confound: a model trained on Claude-produced Egyptian may score higher on Claude-translated benchmarks without being better on independent Egyptian text. Please add an evaluation on human-authored or independently sourced Egyptian Arabic data (for example, existing Egyptian dialect benchmarks or native-speaker-validated samples) and report how the translated-benchmark results compare with that independent evaluation.","section":"Sections 3.1, 3.2.2, 6, 7"},{"comment":"The headline '14.4% performance gain' is the average of absolute percentage-point differences across the five Latin-script benchmarks in Table 2: (20.22 + 12.23 + 6.57 + 14.37 + 18.61)/5. This is not a relative gain, and the abstract and Section 7 should say '14.4 percentage points' or report a properly computed relative improvement. The same issue appears in Section 7, where Nile-Chat-4B is said to achieve 'relative gains of 1.2%' on EgyptianPIQA; Table 1 shows a difference of 1.2 absolute points (67.30 vs 66.10), which is 1.8% relative. Please adopt one clearly defined reporting convention and relabel all such numbers as percentage-point improvements unless relative gains are intended.","section":"Abstract, Section 7"},{"comment":"The statement that 'Nile-Chat models consistently outperform competitive baselines including LLaMa, ALLaM, Jais, and Qwen2.5 across all Egyptian-specific benchmarks' is too strong as written. In Table 1, Nile-Chat-4B trails ALLaM-7B-Instruct-preview on EgyptianMMLU (50.25 vs 60.08) and EgyptianHellaSwag (55.92 vs 57.29), among other rows. The claim should be qualified to the 12B model or to specific parameter-size categories, and 'consistently outperform all baselines' should be replaced with a claim that matches the table.","section":"Section 7, Abstract"}],"minor_comments":[{"comment":"The text says 'roughly 25%' of the data is represented in Latin script, but the stated figures (255M Latin words out of a 1.15B-word corpus) give about 22%. Please reconcile this or state the exact percentage.","section":"Section 3.1"},{"comment":"The code-switching detection heuristic (at least one English word and less than 35% English words) is introduced without any validation. A short analysis of its precision or a few examples of false positives/negatives would strengthen this part.","section":"Section 3.3"},{"comment":"The caption says 'hatched regions represent parts in Latin script,' but the hatching is not clearly visible in the figure's rendered form. Consider adding explicit per-slice labels or a more legible legend.","section":"Section 6, Figure 2"},{"comment":"The phrase 'significantly outperform' is used without any statistical testing or confidence intervals. Given that some margins are small (for example, 1.2 points on one benchmark), please either add uncertainty estimates or soften the wording to avoid implying formal significance.","section":"Section 7"},{"comment":"The Limitations section acknowledges heavy reliance on Claude for translating training data but does not mention that the evaluation benchmarks were also translated with Claude. This should be disclosed in the limitations as well.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The benchmark train/test overlap issue is the key technical concern. If the authors cannot demonstrate that Ben-Ar/Ben-Ltn and the EgyptianBench evaluation items are disjoint, the reported gains could be substantially inflated. I would not recommend rejection because the resources are public and the issue is fixable in principle, but the revision must include a clear data statement, an independent evaluation, and corrected metric reporting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is that someone finally built an LLM family for Egyptian Arabic that treats Arabizi/Latin script as a first-class input rather than as noise to be normalized. The BTX-with-script-specialists approach is a natural extension of Sukhbaatar et al. rather than a breakthrough, but it is applied sensibly, and the dense 4B and 12B models also look well-trained. The released models, the Egyptian-SFT-Mixture, and the EgyptianBench harness are genuinely useful resources for a dialect with over 100M speakers, and the paper deserves credit for being transparent about the composition of its training mixture.\n\nWhat the paper does well: careful data curation, a clear illustration of the mixture in Figure 2, evaluation across multiple understanding and generation tasks in both scripts, and a Limitations section that openly acknowledges heavy reliance on Claude for translation and transliteration. That last point is honest, but it also exposes the main weakness.\n\nThe soft spot is load-bearing. Figure 2 includes components labeled Ben-Ar and Ben-Ltn, expanded in the caption as \"Benchmarks Training Set.\" The paper never states that these components are disjoint from the test splits scored in Tables 1 and 2. The only explicit held-out split mentioned anywhere is the 10% reservation for translation and transliteration data in Section 3.2.3; there is no analogous statement for the eight translated benchmarks. If Ben-Ar and Ben-Ltn contain training splits of EgyptianMMLU, EgyptianHellaSwag, and the rest, and if the evaluation used the full benchmark or a test split drawn from the same Claude-generated pipeline, then the +18 to +20 point margins on Latin HellaSwag and PIQA would reflect memorization rather than transferable Egyptian Arabic ability. This is not a proven flaw, but the burden is on the authors to document the disjointness. The fact that Claude generated both the SFT translations and the benchmark translations makes the potential circularity worse, because the evaluation distribution matches the training distribution in style even when the items differ.\n\nA smaller but real issue: the \"14.4% performance gain\" reported in the abstract is the average of absolute percentage-point differences across five Latin benchmarks, not a relative gain. That should be restated as percentage points.\n\nThis paper is for researchers working on dialectal Arabic NLP and on script handling in LLMs. It deserves a serious referee, but the referee should require the authors to clarify the benchmark train/test split, report standard deviations, and ideally add a small human evaluation on authentic Arabizi before the headline numbers are taken at face value. I would not cite the central result until that clarification arrives, but I would bring the paper to a reading group to discuss the contamination issue.","headline":"Useful dual-script Egyptian Arabic models, but the reported gains hinge on an unverified benchmark train/test split that the paper must resolve before the numbers can be trusted.","tokens_in":21869,"tokens_out":2259,"would_cite":false,"duration_ms":28149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Egyptian Arabic LLMs now handle Arabizi as a first-class script","keywords":["Egyptian Arabic","Arabizi","Latin-script Arabic","dual-script language models","Mixture-of-Experts","Branch-Train-MiX","dialectal LLM","language adaptation"],"falsifier":"Run an independent evaluation on Arabizi and Arabic-script texts written by native Egyptian speakers that were never part of Nile-Chat's training data or its benchmark design; if the 12B model then no longer beats Qwen2.5-14B-Instruct on the Latin-script tasks, the reported 14.4% advantage would not generalize.","tokens_in":20773,"feed_emoji":"🗣️","tokens_out":9885,"duration_ms":89900,"temperature":0.7,"pith_summary":"Egyptian Arabic, the Arabic variety with over 100 million speakers, is routinely written in two scripts: standard Arabic script and a Latin-based form called Arabizi or Franco-Arabic. This paper claims that a language model family built specifically for Egyptian Arabic can learn both scripts natively, and that doing so beats general multilingual Arabic models at their own game. The authors introduce Nile-Chat-4B, Nile-Chat-12B, and Nile-Chat-3x4B-A6B, the last merging Arabic-script and Latin-script experts into a Mixture-of-Experts model using the Branch-Train-MiX strategy. On newly released Egyptian benchmarks spanning understanding, translation, and transliteration, the models report consistent gains over LLaMa, Jais, ALLaM, and Qwen, including a 14.4% improvement over Qwen2.5-14B-Instruct on Latin-script tasks. The work matters because no prior LLM has treated a single Arabic dialect's two written forms as equally native.","feed_headline":"Egyptian Arabic LLMs now handle Arabizi as a first-class script","feed_subtitle":"They route Arabic- and Latin-script Egyptian to separate experts and report beating Qwen by 14.4% on Latin tasks.","key_machinery":"The load-bearing mechanism is Branch-Train-MiX (BTX) applied to script-specialized experts. BTX recycles already-trained models into one MoE by taking each model's feed-forward layers as experts, averaging the attention and embedding layers into a shared backbone, and adding a token-level router; here the two experts are separately pre-trained on Arabic-script and Latin-script Egyptian, and a third expert is the original base model. The router's dynamic assignment lets Arabic words take the Arabic expert path and Latin-script words take the Latin path, including Arabizi's digit substitutions for Arabic letters, so each script gets dedicated capacity without doubling the inference cost of a dense 12B model. This is what carries the argument that script specialization, not just dialect data, explains the reported gains.","core_discovery":"The paper's central claim is that script is a first-class axis of language-model adaptation: the same dialect spoken by Egyptians deserves distinct parameter-level expertise for its Arabic-script and Latin-script forms, and a Mixture-of-Experts (MoE) architecture is the right scaffold for that separation. Concretely, the authors continually pre-train two branches of a base model on Arabic-script and Latin-script Egyptian data, then use Branch-Train-MiX (BTX) to merge their feed-forward layers as experts in a single MoE, with a trainable router deciding which expert handles each token. After instruction-tuning on a newly assembled 1.85M-example Egyptian dataset and a Direct Preference Optimization (DPO) alignment stage, the resulting models report the highest scores among the compared models on the Egyptian evaluation suite, with the 12B dense model leading every Arabic-script benchmark and the MoE models standing out on generation and Latin-script tasks. The paper thus frames itself as demonstrating a repeatable methodology for adapting LLMs to dual-script languages, rather than a one-off model release.","pith_inferences":["I infer the same script-expert recipe would transfer to other dual-script languages such as Hindi, Serbian, or Kazakh, since the specialization axis is script rather than dialect.","If the benchmark gains hold under independent scrutiny, the implication is that dialect-specific script modeling may be worth more than additional generic parameters for under-served languages.","A direct test would compare the MoE's routing decisions against a human annotation of script purity, such as whether tokens containing Arabizi digits route to the Latin expert; the paper does not report such a check.","Because much of the Latin-script data was machine-translated rather than collected from real Arabizi, a native-speaker preference study would reveal how much naturalness ceiling remains."],"forward_implications":["Users can prompt in Arabizi or in Arabic script and expect fluent Egyptian Arabic in return, because both scripts are treated as native inputs and outputs.","The 12B model reports the top score on every Arabic-script benchmark in the suite, including a 14.4% relative gain over Qwen2.5-14B-Instruct on the Latin-script subset.","The MoE models activate only 6B parameters yet match or exceed the dense 12B on translation, transliteration, and Latin-script understanding, suggesting script routing is an efficient route to capability.","Existing Arabic models' weak Latin-script scores imply that Arabizi has been an underserved input, and the released evaluation suite gives a standard way to measure progress.","All models, data, and evaluation harness are released publicly, so other dual-script languages can reuse the pipeline."],"supporting_citations":[{"why":"Supplies Branch-Train-Mix, the method that merges the script-specialized experts into a unified MoE.","marker":"(Sukhbaatar et al., 2024)"},{"why":"Supplies the Gemma-3 base models that Nile-Chat adapts for Egyptian.","marker":"(Team et al., 2025)"},{"why":"Jais is a primary Arabic baseline that Nile-Chat compares against.","marker":"(Sengupta et al., 2023)"},{"why":"ALLaM is a leading Arabic-English baseline that the Egyptian benchmarks measure against.","marker":"(Bari et al., 2024)"},{"why":"Provides ArabicMMLU-egy, one of the two source pools for the EgyptianMMLU benchmark.","marker":"(Mousi et al., 2025)"},{"why":"Provides Belebele, the reading-comprehension benchmark whose Egyptian subset is used directly.","marker":"(Bandarkar et al., 2023)"},{"why":"Provides the Egyptian Forums Corpus, a main pre-training source and the basis for transliteration data.","marker":"(Qarah, 2024)"},{"why":"Establishes the dialectal-model paradigm that Nile-Chat extends to dual-script support.","marker":"(Shang et al., 2025)"},{"why":"Supplies the TÜLU-v3 instruction mix that is translated into Egyptian for SFT.","marker":"(Lambert et al., 2024)"},{"why":"Supplies DPO, the alignment method used in the final training stage.","marker":"(Rafailov et al., 2023)"}],"fun_headline_variants":["Arabic-Latin MoE splits Egyptian dialect into script experts","Egyptian dialect gets MoE: separate experts for Arabizi","Branch-train-mix: Egyptian LLMs master both scripts","Dual-script Egyptian LLMs beat Qwen by 14.4%","Script-specialized experts for Egyptian Arabic and Arabizi"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation benchmarks are an honest yardstick: the benchmark material mixed into the instruction-tuning data (labeled 'Ben-Ar' and 'Ben-Ltn' in the paper's Figure 2) does not overlap with the held-out test sets used for the reported scores, and the machine-translated Arabic and Latin-script examples represent how Egyptians actually write.","fun_headline_variants_meta":{"raw":{"variants":["Arabic-Latin MoE splits Egyptian dialect into script experts","Egyptian dialect gets MoE: separate experts for Arabizi","Branch-train-mix: Egyptian LLMs master both scripts","Dual-script Egyptian LLMs beat Qwen by 14.4%","Script-specialized experts for Egyptian Arabic and Arabizi"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000765,"raw_usage":{"total_tokens":3398,"prompt_tokens":953,"completion_tokens":2445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2357}},"tokens_in":569,"tokens_out":2445,"duration_ms":16746,"temperature":1.0,"reasoning_tokens":2357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:44:57.321916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an independent evaluation on Arabizi and Arabic-script texts written by native Egyptian speakers that were never part of Nile-Chat's training data or its benchmark design; if the 12B model then no longer beats Qwen2.5-14B-Instruct on the Latin-script tasks, the reported 14.4% advantage would not generalize.","supporting_citations":[],"review_version":1}