REVIEW 5 major objections 6 minor 15 references
Sadeed: Advancing Arabic Diacritization Through Small Language Model
T0 review · 5 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Sadeed, a 1.5B fine-tuned Arabic model, claims state-of-the-art Word Error Rate on the Fadel diacritization test set and introduces a fresh benchmark.
desk verdict Useful benchmark and cleaned dataset, but the model's SOTA claim is an artifact of scoring Sadeed on a corrected test set without re-running baselines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Sadeed itself, a 1.5B-parameter decoder-only language model fine-tuned from Kuwain, with diacritization reformulated as a question-answering task: the model receives a system prompt and a non-diacritized input, and it generates the diacritized text. The second piece of machinery is post-processing alignment: raw model output is aligned to the original input with the Needleman-Wunsch algorithm, which removes inserted words, restores omitted words, and replaces altered words with their non-diacritized originals, so the final output is a faithful diacritized version of the input. The third piece is the data pipeline: roughly 53 million words of Tashkeela and ATB-3 text, cleaned with normalization rules, filtered to keep fully diacritized sentences, and de-overlapped against the Fadel test set.
What would settle it
Re-run the prior systems (SUKOUN, D3, PTCAD, Fadel) on the released corrected Fadel test set with the same 'excluding no diacritic, without case endings' metric. If any of them posts a Word Error Rate at or below Sadeed's 1.7115, the central claim fails. A second check: verify the iltiqa' as-sakinayn phonological rule used to alter the test set against a standard Arabic grammar reference; if the rule is disputed or was applied inconsistently, the corrected benchmark is not a valid comparison instrument.
Extended reading notes
Core claim
The paper's central claim is that Arabic diacritization can be solved at small scale by fine-tuning a decoder-only 1.5B Arabic language model on clean, overlap-free data, with the task framed as question-answering. On the original Fadel test set, Sadeed's best Word Error Rate is 1.7955 without case endings, and 1.7115 on the paper's phonologically corrected version of the same set, both measured while excluding characters that carry no diacritic. The paper reports that on the corrected set Sadeed's WER is lower than SUKOUN, D3, PTCAD, and the original Fadel model, and it argues this is the fair comparison because prior models were trained on data overlapping the test set. On its new SadeedDiac-25 benchmark, Sadeed trails Claude 3.7 Sonnet but outperforms all evaluated open Arabic models, and it is competitive with GPT-4 and Gemini-Flash 2.0. The authors identify hallucinations, especially on non-Arabic words, as the main source of Sadeed's remaining errors.
Load-bearing premise
The load-bearing assumption is that the phonologically corrected version of the Fadel test set is the right measure: the paper beats prior published models on Word Error Rate only after it changes the test set by applying a rule for adjacent consonants, and if that correction is not accepted or not applied to the comparison models, the state-of-the-art claim does not hold.
Editorial extensions
If this is right
- A 1.5B fine-tuned model can outperform far larger and proprietary systems on Classical Arabic diacritization, making the task accessible to teams with modest compute.
- Training on the released cleaned set avoids the Fadel/Abbad overlap, so future WER comparisons on the Fadel test set are not inflated by leakage.
- Existing results from models trained on both Fadel and Abbad data need re-examination, since over a third of the Fadel test set appears verbatim in Abbad training data.
- SadeedDiac-25 provides a fresh, expert-reviewed benchmark that mixes Classical and Modern Standard Arabic, giving future models a contamination-free evaluation instrument.
- Modern Standard Arabic remains the bottleneck: Sadeed's lower scores on MSA-heavy data point to training-data scarcity rather than model architecture as the limiting factor.
Reading between the lines
- Editorial inference: the same fine-tune-plus-alignment recipe should transfer to other tasks that restore omitted annotation layers, such as Hebrew niqqud or phonetic transcription from plain text, where a generative model can hallucinate and sequence alignment can repair the output.
- Editorial inference: if the phonological correction to the Fadel test set becomes standard practice, then the 'excluding no diacritic' WER without case endings is the metric most likely to be adopted for reporting, since it isolates the core diacritization decision from case-ending debates.
- Editorial inference: a testable extension of the paper's MSA argument is to fine-tune Sadeed on an MSA-heavy subset and measure WikiNews WER; if the gap to the in-domain-trained model closes, then data mix rather than capacity explains the current MSA shortfall.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Sadeed, a 1.5-billion-parameter decoder-only model for Arabic diacritization, fine-tuned from the Kuwain base model on a newly cleaned version of the Tashkeela corpus combined with ATB-3 (about 1.04 million samples, roughly 53 million words). It also introduces SadeedDiac-25, a 1,200-paragraph expert-reviewed benchmark spanning Classical and Modern Standard Arabic, and reports an overlap analysis indicating substantial leakage between the Fadel and Abbad benchmark splits. The evaluation claims state-of-the-art word error rate (WER) on the Fadel benchmark (Table 6), competitive performance versus proprietary large language models on SadeedDiac-25 (Table 8), and competitive results on WikiNews (Section 7.2). The key evidence for the headline claim is Table 6, where Sadeed's best numbers are reported on a phonologically 'corrected' version of the Fadel test labels, while all baseline rows are taken from prior publications scored on the original labels.
Significance. The contributions are potentially useful even if the headline claims are scaled back. The released training dataset, the SadeedDiac-25 benchmark with expert review, the Fadel/Abbad overlap analysis, and the public evaluation code all support reproducibility, and the paper is transparent about hallucination rates and about Sadeed's weakness on Modern Standard Arabic. If the WER state-of-the-art claim were valid, the paper would also be significant in showing that a 1.5B fine-tuned SLM can surpass much larger and dedicated diacritization systems with modest compute. Unfortunately, as detailed below, the two headline claims are not supported by the evidence as presented: the Fadel comparison is asymmetric, and Table 8 contradicts the claim of competitiveness with proprietary models. The manuscript's value currently lies mainly in its resources and analysis rather than in its model-evaluation claims.
major comments (5)
- [§7.1, Table 6] The central WER state-of-the-art claim rests on an asymmetric comparison. The 'Sadeed-Fadel corrected' rows are scored on a version of the Fadel test set that the authors modified with the same phonological rule (adjusting the vowelization of the first consonant in iltiqa' assakinayn) that was applied to Sadeed's training data in Section 3, whereas the D3, SUKOUN, PTCAD, and Fadel baseline rows match published values and there is no indication that these baselines were re-scored on the corrected set. On the original Fadel test set, Sadeed's best WER (1.7955, excluding no-diacritic, without case endings) is worse than SUKOUN's 1.77, and the 1.7115 value that motivates the state-of-the-art claim is obtained only on the corrected set. The DER comparison shows the same pattern: Sadeed beats SUKOUN only in the without-case-ending columns, while SUKOUN's with-case-ending DER (1.16-1.23) is better than Sadeed's (1.58-1.68) on the original set. The sentence 'Sadeed achieves state-of-the-art (SOTA) performance on the Fadel dataset in terms of Word Error Rate' is therefore not supported for the original benchmark. The authors should re-run all baselines on both the original and the corrected test sets with their own evaluation code, or restrict the comparison to a configuration in which all models are scored under identical conditions.
- [§5, §7.1] The Sadeed scores in Table 6 are computed after a hallucination-correction stage that has no counterpart in the baseline numbers. Section 5 describes aligning the raw generation to the input with the Needleman-Wunsch algorithm, removing added words, restoring missing words, and replacing altered words with their non-diacritized versions, leaving some words undiacritized in the final output. The WER and DER therefore measure a post-processed output, and the 'excluding no diacritic' columns — exactly the columns in which Sadeed leads — are those that exclude the words left undiacritized by this procedure. The paper neither reports the fraction of test words left undiacritized after correction nor gives raw uncorrected scores, so even the 'Sadeed-Fadel original' rows are not strictly comparable to the baseline results under a common operational definition of WER.
- [Abstract, §7.3, Table 8] The abstract's claim that Sadeed 'achieves competitive results compared to proprietary large language models' is contradicted by the paper's own Table 8. On SadeedDiac-25, Sadeed's WER is 13.74 with case endings and 9.92 without, versus 4.67 and 2.31 for Claude 3.7 Sonnet and 5.27 (with case endings) for GPT-4; the gap is roughly threefold and is larger than the 7.19 hallucination-related WER points that the paper itself identifies as Sadeed's main weakness. The abstract, introduction, and Section 7.3 should be revised to state that Sadeed is the best of the evaluated open-weight models but substantially behind the best proprietary systems.
- [Table 8] The GPT-4 row of Table 8 is internally inconsistent and appears to contain a transcription error. The without-case-ending WER (10.9274) is reported as roughly double the with-case-ending WER (5.2719), and the DER value (3.8645) is identical in both columns; no other row in the table shows this pattern, and under the standard definitions removing case endings cannot increase WER. All entries of Table 8, and the evaluation logs behind them, should be re-verified before the benchmark results can be considered reliable.
- [§4] The novelty claim for SadeedDiac-25 is overstated. The paper states that 'this dataset remains entirely novel to existing models' because the diacritization was performed in-house, but 600 of the 1,200 benchmark paragraphs are drawn from the Fadel test set, a public benchmark since 2019, and the paper's own Section 6.1 shows that the Fadel test set overlaps substantially with the Abbad training data (865 Fadel-test samples appear identically in the Abbad train set). The novelty claim should be restricted to the 454 originally curated paragraphs, and the possibility that the Fadel-derived portion is familiar to some evaluated models should be acknowledged.
minor comments (6)
- [§7.1] The sentence 'we systematically the vowelization of the first consonant based on standard Arabic phonological rules' is missing a main verb and should read, for example, 'we systematically adjusted the vowelization'.
- [Page 1 footnote, Figure 2] The paper is inconsistent in spelling the model name: the footnote on page 1 and the caption of Figure 2 use 'Sadded', while the title, abstract, and body use 'Sadeed'; the spelling should be unified.
- [References] The bibliography lists Darwish et al. 2017a and 2017b as two entries with identical titles, venues, and page ranges, but they are cited for different purposes in Sections 4 and 7.2; the duplicate should be re-keyed to the correct publication, and the citation in Section 7.2 should be checked for consistency with Table 7, which attributes the FRRNN results to Darwish et al. [2020].
- [§3] After the filter that 'removed examples containing three or more words with partial diacritics', the statement that 'only 10.7% of the sentences in the dataset contain at most two partially diacritized words' is logically inconsistent, since all remaining sentences satisfy that condition; the intended statistic should be reworded or recomputed.
- [Table 8] The eight metric columns in Table 8 would benefit from clearer visual grouping of the 'With Case Ending' and 'Without Case Ending' blocks, because the current layout makes it easy to misassign values across the two blocks.
- [Tables 1, 3, Appendix C] Several Arabic examples show broken glyph or ligature rendering in the manuscript (for example, Table 1 and Figure 3); the camera-ready version should verify the typesetting of diacritized Arabic.
Circularity Check
Fadel SOTA claim is obtained by scoring Sadeed on a phonologically corrected test set built with the same rule used in its training, while published baselines are scored on the original labels.
-
self definitional
[Section 3 (Diacritization Dataset, preprocessing) and Section 7.1 (Evaluation on Fadel Benchmark), Table 6]
"To resolve this inconsistency, we automatically adjusted the vowelization of the first consonant based on standard Arabic phonological rules. ... To address this limitation, we systematically the vowelization of the first consonant based on standard Arabic phonological rules. This refined and phonologically consistent version of the Fadel test set has been made publicly available to facilitate accurate and reliable benchmarking."
The same phonological rule used to normalize Sadeed's training data is applied to the Fadel test set before the SOTA claim is made. Table 6 lists only 'Sadeed-Fadel corrected' and previously published baseline numbers (SUKOUN, D3, PTCAD, Fadel) that were evaluated on the original test-set labels; no baseline is re-run on the corrected labels. On the original Fadel set, Sadeed's best excluding-no-diacritic WER without case endings is 1.7955, which is worse than SUKOUN's 1.77; Sadeed becomes SOTA only on the corrected set (1.7115). The comparison thus reduces to scoring Sadeed against labels rewritten to match its own training convention, while the competing models are scored against different, unmodified ground truth.
full rationale
The paper's central Fadel SOTA claim is partially circular because the evaluation instrument is altered by the authors using the same normalization rule that shaped Sadeed's training data, and the baseline numbers are not recomputed under that alteration. This is a specific, quotable reduction: the corrected test set is constructed from the same phonological rule as the training pipeline, and the headline WER comparison is asymmetric. The other contributions, including SadeedDiac-25, the overlap analysis, and the released dataset, are independently constructed and externally checkable, so they do not exhibit circularity. The self-citation of Kuwain is not load-bearing for the empirical comparisons, since Sadeed's performance is measured rather than assumed; at most it makes the base model's details harder to verify. The abstract's 'competitive results compared to proprietary large language models' is contradicted by Table 8 (Sadeed 13.74 WER vs Claude 3.7's 4.67), but that is a correctness or evidence problem rather than a circularity problem. Overall, the partial circularity in the Fadel benchmark comparison warrants a score of 6 rather than a higher score, because the model and dataset work still have independent content.
Assumptions & free parameters
free parameters (8)
- max_undiacritized_words_per_sample =
2
- max_partial_diacritics_words_per_sample =
2
- overlap_similarity_threshold =
0.5
- benchmark_modern_standard_arabic_fraction =
0.5
- training_epochs =
3
- learning_rate =
5e-6
- batch_size =
1024
- max_sequence_length =
512
assumptions (5)
- domain assumption Tashkeela corpus and ATB-3 are reliable sources of diacritized Arabic text.
- domain assumption The iltiqa' assakinayn phonological rule can be applied automatically to normalize both training and test data without introducing linguistic errors.
- domain assumption The two-stage expert review process yields correct ground truth for SadeedDiac-25.
- domain assumption Needleman-Wunsch alignment is a valid method for removing model hallucinations without altering the model's diacritization decisions.
- domain assumption Kuwain 1.5B is a suitable pretrained Arabic SLM for fine-tuning to diacritization.
Cite this review
Pith. "Pith review of Sadeed: Advancing Arabic Diacritization Through Small Language Model." pith.science (2026). https://pith.science/paper/LD7SQQ4S
@misc{pith2026250421635,
author = {Pith},
title = {Pith review of: Sadeed: Advancing Arabic Diacritization Through Small Language Model},
year = {2026},
howpublished = {\url{https://pith.science/paper/LD7SQQ4S}},
note = {Machine review of arXiv:2504.21635}
}
read the original abstract
Arabic text diacritization remains a persistent challenge in natural language processing due to the language's morphological richness. In this paper, we introduce Sadeed, a novel approach based on a fine-tuned decoder-only language model adapted from Kuwain 1.5B Hennara et al. [2025], a compact model originally trained on diverse Arabic corpora. Sadeed is fine-tuned on carefully curated, high-quality diacritized datasets, constructed through a rigorous data-cleaning and normalization pipeline. Despite utilizing modest computational resources, Sadeed achieves competitive results compared to proprietary large language models and outperforms traditional models trained on similar domains. Additionally, we highlight key limitations in current benchmarking practices for Arabic diacritization. To address these issues, we introduce SadeedDiac-25, a new benchmark designed to enable fairer and more comprehensive evaluation across diverse text genres and complexity levels. Together, Sadeed and SadeedDiac-25 provide a robust foundation for advancing Arabic NLP applications, including machine translation, text-to-speech, and language learning tools.
Figures
Reference graph
Works this paper leans on
-
[7]
Arabic diacritization: Stats, rules, and hacks
Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. Arabic diacritization: Stats, rules, and hacks. In Nizar Habash, Mona Diab, Kareem Darwish, Wassim El-Hajj, Hend Al-Khalifa, Houda Bouamor, Nadi Tomeh, Mahmoud El-Haj, and Wajdi Zaghouani (eds.), Proceedings of the Third Arabic Natural Language Processing Workshop , pp. 9–17, Valencia, Spain, April 2017a....
-
[8]
Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model
URLhttps://arxiv.org/abs/2002.01207. Mona Diab, Mahmoud Ghoneim, and Nizar Habash. Arabic diacritization in the context of statis- tical machine translation. InProceedings of Machine Translation Summit XI: Papers ,
work page Pith review arXiv 2002
-
[10]
Kuwain 1.5 b: An arabic slm via language injection
Khalil Hennara, Sara Chrouf, Mohamed Motaism Hamed, Zeina Aldallal, Omar Hadid, and Safwan AlModhayan. Kuwain 1.5 b: An arabic slm via language injection. arXiv preprint arXiv:2504.15120,
-
[11]
doi: https://doi.org/10.1016/j.eswa.2024.123416
ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa.2024.123416. URL https://www.sciencedirect.com/science/article/pii/S0957417424002811. Vladimir Likic. The needleman-wunsch algorithm for sequence alignment.Lecture given at the 7th Melbourne Bioinformatics Course, Bi021 Molecular Science and Biotechnology Institute, Univer- sity of Melbourne , pp. 1–46,
arXiv 2024
- [12]
-
[13]
Abderrahman Skiredj and Ismail Berrada
URL https://arxiv.org/abs/2308.16149. Abderrahman Skiredj and Ismail Berrada. Arabic text diacritization in the age of transfer learning: Token classification is all you need,
-
[14]
Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Need
URLhttps://arxiv.org/abs/2401.04848. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118 ,
-
[15]
doi: 10.1016/j.dib.2017.01.011. 19 Appendix A Training Details The fine-tuning process forSadeed employed the standard next-token prediction methodology, with the system prompt and embedding tokens masked. Given the compact nature of theKuwain model, we designed a concise yet effective fine-tuning regimen. Table 9 presents the detailed hyperparam- eters u...
Show all 15 references
-
[2007]
Arabic text diacriti- zation using deep neural networks, 2019a
Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. Arabic text diacriti- zation using deep neural networks, 2019a. URLhttps://arxiv.org/abs/1905.01965. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. Neural arabic text dia- critizatio...
1905 arXiv
-
[2016]
doi: 10.18653/v1/N16-3003
Association for Computational Linguistics. doi: 10.18653/v1/N16-3003. URL https://aclanthology.org/N16-3003. Abdulmohsen Al-Thubaity, Atheer Alkhalifa, Abdulrahman Almuhareb, and Waleed Alsanie. Ara- bic diacritization using bidirectional long short-term memory neural networks...
-
[2017]
doi: https://doi.org/10.1016/j.procs.20 17.10.106
ISSN 1877-0509. doi: https://doi.org/10.1016/j.procs.20 17.10.106. URL https://www.sciencedirect.com/science/article/pii/S1877050917321634. Arabic Computational Linguistics. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, ...
-
[2020]
ISBN 978-3-030-45439-5
Springer International Publishing. ISBN 978-3-030-45439-5. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. Farasa: A fast and furious segmenter for Arabic. In John DeNero, Mark Finlayson, and Sravana Reddy (eds.),Proceedings of the 2016 Conference of the Nort...
2016
-
[2021]
doi: 10.1109/ACCESS.2021.312297
2021
-
[2023]
URLhttps://arxiv.org/ abs/2306.03557. M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan AlRashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. Allam: Large language models for arabic and english. ...
-
[2024]
Badr AlKhamissi, Muhammad N ElNokrashy, and Mohamed Gabr
URL https://arxiv.org/abs/2407.03236. Badr AlKhamissi, Muhammad N ElNokrashy, and Mohamed Gabr. Deep diacritization: Efficient hierarchical recurrence for improved arabic diacritization.arXiv preprint arXiv:2011.00538,
2011 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.