Pith. sign in

REVIEW 2 major objections 7 minor 10 references

A 518M Arabic-specialized open language model beats every same-size open model and fits in 398 MB.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 23:40 UTC pith:EMUMHWJE

load-bearing objection First open sub-1B Arabic-specialized decoder with full weights and a clean, reproducible pipeline; engineering release, not a methods paper. the 2 major comments →

arxiv 2605.28827 v1 pith:EMUMHWJE submitted 2026-04-10 cs.CL cs.LG

RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment

Jaber Jaber , Osama Jaber This is my paper
classification cs.CL cs.LG
keywords Arabic language modelsvocabulary injectionsub-1B LLMsedge deploymentweight soupdirect preference optimizationtokenizer fertilityGGUF quantization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Arabic open language models have split into two camps that leave edge devices empty: tiny multilingual models that barely treat Arabic specially, and large Arabic-specialized models that need servers. This paper claims that careful assembly of known techniques is enough to close that gap. Starting from a 0.5B multilingual base, the authors inject more than 27,000 new Arabic tokens, continue pretraining on half a billion Arabic tokens, fine-tune on Arabic instructions with response-only loss, apply preference optimization, and average three checkpoints. The resulting 518-million-parameter model is the smallest open Arabic-specialized decoder with public weights. On three standard Arabic benchmarks it leads every open model of similar size, ties a model three times larger on one reasoning task, and recovers about two-thirds of a 9B Arabic model’s average score at roughly one-eighteenth the size. Quantized, it occupies 398 MB and runs at hundreds of tokens per second, making Arabic-specialized language modeling practical on phones and laptops.

Core claim

The authors establish that a fully open 518M Arabic-specialized decoder, produced by vocabulary injection plus continued pretraining, response-masked instruction tuning, preference optimization and weight-soup merging on a Qwen2.5-0.5B base, reaches 35.9 percent mean accuracy on COPA-ar, Arabic HellaSwag and ArabicMMLU—beating every same-class open model, matching a 1.5B model on COPA-ar, and recovering 67 percent of a 9B Arabic model’s mean at 1/18 the parameters—while quantizing to 398 MB.

What carries the argument

Mean-subtoken vocabulary injection: 27,032 new Arabic tokens are added to the base tokenizer and each new embedding row is initialized as the average of the old embeddings of its sub-token decomposition, which lowers Arabic fertility from 2.18 to 1.80 tokens per word and supplies the geometry for subsequent continued pretraining and post-training.

Load-bearing premise

That half a billion Modern-Standard-Arabic Wikipedia tokens plus the listed public instruction and preference sets are enough to turn a multilingual 0.5B base into a genuinely Arabic-specialized model whose three-task scores will hold up beyond that narrow data mixture.

What would settle it

Re-run the identical three lm-evaluation-harness tasks (COPA-ar, Arabic HellaSwag, ArabicMMLU) with the same limit=200 and chat-template settings on the released weights; if the mean falls below the open 0.5B baselines or the COPA-ar score no longer matches the 1.5B competitor, the central performance claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper presents RightNow-Arabic-0.5B-Turbo, a 518M-parameter Arabic-specialized decoder built from Qwen2.5-0.5B by injecting 27,032 Arabic tokens (mean-subtoken initialization), continued pretraining on 504M Arabic Wikipedia tokens, response-masked SFT on 129k instruction pairs, DPO on 6.75k preference pairs, and a 50/25/25 weight soup of DPO/SFT/pretrain checkpoints. Under a fixed lm-evaluation-harness protocol on COPA-ar, Arabic HellaSwag, and ArabicMMLU (limit=200), the merged model reports 35.9% mean accuracy, outperforming same-class open models (Qwen2.5-0.5B-Instruct, Falcon-H1-0.5B), tying Falcon-H1-1.5B on COPA-ar at one-third the size, and recovering ~67% of SILMA-9B’s mean at ~1/18 the parameters. The authors release bf16/int8/GGUF weights (down to 398 MB q4_k_m), 5,555 lines of training/export code, and benchmark scripts, and document edge throughput via llama.cpp.

Significance. If the reported head-to-head numbers and open release hold, the work fills a concrete gap: there is currently no downloadable sub-1B Arabic-specialized decoder with public weights, while 7B–70B Arabic models are server-bound and sub-1B multilingual models under-allocate Arabic vocabulary and data. The contribution is primarily an engineering and resource result rather than a new training algorithm; its value lies in careful orchestration (tokenizer fertility reduction, memmap packing, response-only SFT, soup merge, GGUF export) plus full reproducibility. Strengths that should count in the assessment include the complete open pipeline and intermediate checkpoints, identical evaluation methodology across baselines, and unusually candid reporting that DPO barely moved the model and that ArabicMMLU still favors the multilingual base. The scoped claim—that this is the smallest open Arabic-specialized decoder that wins same-class comparisons under the stated protocol—is therefore useful for edge Arabic deployment research even if absolute knowledge scores remain limited by scale and data.

major comments (2)
  1. [Section 5 / Table 3] Section 5 (Evaluation methodology) and Table 3: all three tasks are scored with limit=200 and no bootstrap/standard-error estimates. Relative same-class rankings (especially the 1.8-point mean edge over Qwen2.5-0.5B-Instruct and the exact 58.4% COPA-ar tie with Falcon-H1-1.5B) are therefore hard to assess for sampling noise. Because the central claim is comparative under a fixed protocol, either full-task evaluation (particularly for ArabicMMLU’s 14,575 items) or confidence intervals / multiple seeds would make the reported margins load-bearing rather than suggestive.
  2. [Sections 3.5, 5.2 / Table 4] Sections 3.5 and 5.2 / Table 4: the final soup (DPO 0.5, SFT 0.25, pretrain 0.25) is selected by benchmarking seven merge variants on the same three Arabic tasks later used for the main claim. The gain over the DPO endpoint is only +0.44 mean points and is concentrated on HellaSwag-ar. This is mild selection on the evaluation suite; the paper should either hold out one task for merge selection, report the soup as a post-hoc choice with the DPO endpoint as the primary result, or add an external Arabic task not used for selection.
minor comments (7)
  1. [Abstract / Section 3.1] Abstract and Section 1 state fertility reduction of 17.3%; Section 3.1 also reports 14.7% on a different 1000-word sample. Clarify which sample is primary and keep one number in the abstract.
  2. [Tables 3–4] Table 3 vs Table 4: small numerical differences for the same soup (e.g., HellaSwag-ar 26.0% vs 25.33%) are attributed to a separate evaluation pass. Prefer a single unified evaluation run for all reported numbers, or state the seed/config that produces Table 3 as canonical.
  3. [Section 6 / Abstract] Section 6 correctly flags MSA-only pretraining and ~1 token/parameter continued pretraining. Consider elevating a one-sentence dialect/knowledge caveat into the abstract or introduction so readers do not over-read “Arabic-specialized.”
  4. [Figure 2] Figure 2 Pareto plot is helpful; ensure axis labels and the “edge-deployable” band remain legible in grayscale print and that the dashed frontier is defined (convex hull of reported points?).
  5. [Section 2] Related Work: SILMA is listed without a citation or release date; if no paper exists, state “no accompanying paper” once and give the Hugging Face identifier for reproducibility.
  6. [Section 3.6 / Table 6] GGUF note that 144 of 290 tensors fall back to higher-bit k-quants (effective ~6.45 bpw for q4_k_m) is important for the 398 MB claim; consider putting effective bpw in Table 6.
  7. [Abstract / Table 6] Minor consistency: abstract “635 tokens/s” vs Table 6 generation 634.9 tok/s for q4_k_m; round consistently.

Circularity Check

1 steps flagged

No significant circularity: external public benchmarks and open weights; only mild selection of the weight-soup on the same three eval tasks.

specific steps
  1. fitted input called prediction [Section 3.5 Weight soup merging; Section 5.2 Merge ablation / Table 4]
    "We produce 7 merge variants: linear interpolations of DPO and pretrain at t∈{0.3,0.5,0.7}, SLERP at t∈{0.3,0.5}, linear DPO/SFT at t=0.5, and a 50/25/25 soup of DPO/SFT/pretrain. Each merge is benchmarked on the same 3 Arabic tasks (Section 5.1); the 50/25/25 soup wins and becomes the final checkpoint. ... The soup gains +0.44 absolute points over the DPO endpoint alone."

    The headline merged model is selected by maximizing mean accuracy on the same three tasks later used to claim 35.9% mean and same-class wins. The +0.44 gain is therefore partly a selection effect on the reported metric, not an independent out-of-sample prediction. The absolute scores vs external baselines remain non-circular; this is only mild model-selection circularity.

full rationale

This is an engineering systems paper, not a first-principles derivation. The central claims (518M Arabic-specialized open decoder; 35.9% mean on COPA-ar / Arabic HellaSwag / ArabicMMLU; same-class wins; COPA-ar tie with Falcon-H1-1.5B; 398 MB q4_k_m edge build) are measured against independently released models and public lm-evaluation-harness tasks under a stated protocol (limit=200, apply_chat_template, acc_norm preferred). Vocabulary injection, continued pretraining, SFT with response-only masking, DPO, and GGUF export are standard techniques applied to public data; none of the reported accuracies is a fitted parameter renamed as a prediction. The only mild circularity is that the final soup (DPO 0.5 / SFT 0.25 / pretrain 0.25) was chosen by benchmarking seven merge variants on the same three tasks that define the headline mean; that is ordinary model selection, not a self-definitional reduction of the absolute scores or the same-class comparisons. No self-citation load-bearing uniqueness theorem, no ansatz smuggled via author-prior work, and no renaming of a known result as a new derivation. Score 1.5 reflects that single non-load-bearing selection step; the paper is otherwise self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The work is an engineering pipeline, not a derivation from first principles. Load-bearing choices are standard ML training assumptions plus a handful of hand-chosen hyperparameters (learning rates, merge coefficients, token count). No new physical or mathematical entities are postulated; the model itself is the product of applying known methods to public data.

free parameters (4)
  • soup merge weights (DPO 0.5 / SFT 0.25 / pretrain 0.25)
    Chosen by exhaustive search over 7 merge variants on the same three evaluation tasks that define the final claim; not derived from theory.
  • number of injected Arabic tokens (27,032)
    Outcome of training a 32k SentencePiece model then deduplicating against Qwen vocab; the target size is a design choice.
  • pretrain peak LR 2e-4, SFT 2e-5, DPO 5e-7
    Hand-tuned schedule values that control the entire training trajectory.
  • DPO beta = 0.1
    Standard but free hyperparameter of the preference loss.
axioms (4)
  • domain assumption Mean-subtoken embedding initialization (average of old sub-piece embeddings) is a valid way to place new Arabic tokens without destroying the base model’s geometry.
    Invoked in Algorithm 1 and Section 3.1; taken from WECHSEL / ALLaM / Kuwain literature without new proof.
  • domain assumption Continued pretraining on Modern Standard Arabic Wikipedia alone, followed by public instruction/preference mixes, produces a model that can be fairly called “Arabic-specialized.”
    Stated throughout Sections 3–5; the paper itself notes dialect and knowledge limitations in Section 6.
  • domain assumption lm-evaluation-harness scores with apply_chat_template, limit=200, and acc_norm are a sufficient proxy for Arabic capability ranking.
    Section 5 evaluation methodology; no human evaluation or broader task suite is provided.
  • domain assumption Linear weight averaging of checkpoints that share architecture is a safe and beneficial post-training step.
    Section 3.5 cites Wortsman et al. model soups; used after DPO failed to move the loss.

pith-pipeline@v1.1.0-grok45 · 17828 in / 3092 out tokens · 32917 ms · 2026-07-12T23:40:47.887809+00:00 · methodology

0 comments
read the original abstract

Open Arabic large language models split into two classes: sub-1B multilingual models that treat Arabic as an afterthought (Qwen2.5-0.5B, Falcon-H1-0.5B), and 7B-70B Arabic-specialized models that require a server to run (Jais, AceGPT, ALLaM, SILMA). The one published attempt at a sub-2B Arabic-specialized model, Kuwain-1.5B, never released its weights. We present RightNow-Arabic-0.5B-Turbo, a 518M-parameter Arabic-specialized decoder LLM built on Qwen2.5-0.5B. The pipeline adds 27,032 Arabic tokens via mean-subtoken initialization, continues pretraining on 504M Arabic tokens on 8xH100 with FSDP, FlashAttention varlen packing, and Liger fused kernels, then applies supervised fine-tuning on 129,116 Arabic instruction pairs with response-only loss masking, direct preference optimization on 6,750 Arabic preference pairs, and weight soup merging across three checkpoints. On three lm-evaluation-harness Arabic benchmarks (COPA-ar, Arabic HellaSwag, ArabicMMLU) the merged model reaches 35.9% mean accuracy, beats every same-class open model, ties Falcon-H1-1.5B on COPA-ar (58.4%) at one-third the size, and recovers 67% of SILMA-9B's mean at 1/18 the parameters. The edge build quantizes to 398 MB (q4_k_m) and delivers 635 tokens/s at batch size 1 on a single H100 via llama.cpp. All code (5,555 lines across 25 scripts), weights (bf16, int8, and four GGUF quantizations), and benchmark scripts are released at https://huggingface.co/RightNowAI/RightNow-Arabic-0.5B-Turbo.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

10 extracted references · 9 linked inside Pith

  1. [1]

    Saiful Bari, Yazeed Alnumay, Norah A

    M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, et al. ALLaM: Large Language Models for Arabic and English.arXiv preprint arXiv:2407.15390,

    Pith/arXiv arXiv

  2. [2]

    A framework for few-shot language model evaluation.EleutherAI open-source software, 2021.https://github.com/EleutherAI/lm-evaluation-harness

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and Eric Tang. A framework for few-shot language model evaluation.Eleut...

    Pith/arXiv arXiv 2021

  3. [3]

    Liger Kernel: Efficient Triton kernels for LLM training.arXiv preprint arXiv:2410.10989,

    Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger Kernel: Efficient Triton kernels for LLM training.arXiv preprint arXiv:2410.10989,

    Pith/arXiv arXiv

  4. [4]

    AceGPT, Localizing Large Language Models in Arabic.arXiv preprint arXiv:2309.12053,

    Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu. AceGPT, Localizing Large Language Models in Arabic.arXiv preprint arXiv:2309.12053,

    Pith/arXiv arXiv

  5. [5]

    ArabicMMLU: Assessing massive multitask language understanding in Arabic.arXiv preprint arXiv:2402.12840,

    Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. ArabicMMLU: Assessing massive multitask language understanding in Arabic.arXiv preprint arXiv:2402.12840,

    Pith/arXiv arXiv

  6. [6]

    Mutarjim: Advancing bidirectional Arabic-English translation with a small language model.arXiv preprint arXiv:2505.17894,

    Khalil Hennara, Muhammad Hreden, Mohamed Motaism Hamed, Zeina Aldallal, Sara Chrouf, and Safwan AlModhayan. Mutarjim: Advancing bidirectional Arabic-English translation with a small language model.arXiv preprint arXiv:2505.17894,

    Pith/arXiv arXiv

  7. [7]

    WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models

    Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL),

    2022

  8. [8]

    Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Qwen Team: An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115,

    Pith/arXiv arXiv

  9. [9]

    Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models.arXiv preprint arXiv:2308.16149,

    Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassi...

    Pith/arXiv arXiv

  10. [10]

    TinyLlama: An open-source small language model.arXiv preprint arXiv:2401.02385,

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. TinyLlama: An open-source small language model.arXiv preprint arXiv:2401.02385,

    Pith/arXiv arXiv