{"id":"135263dd-9cde-4c97-9b2b-079c25813070","arxiv_id":"2504.21635","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"The authors claim that Sadeed, a fine-tuned 1.5B Arabic SLM, reaches state-of-the-art word error rates on the Fadel benchmark and is competitive with proprietary models, while releasing a new benchmark and dataset.","lead":"This paper fine-tunes a small Arabic-language model, Sadeed, to add diacritical marks to Arabic text, and it releases a cleaned training set and a new benchmark called SadeedDiac-25. A generalist might read it to judge whether a compact model can rival far larger systems on Arabic diacritization, and whether existing benchmarks for that task are trustworthy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Corrected Fadel test set is applied only to Sadeed; the published baselines are scored on the original labels, so the Table 6 WER SOTA claim is an artifact of an asymmetric evaluation.","rationale":"The reader's weakest assumption correctly identifies the load-bearing flaw: the phonologically corrected Fadel test set is the only configuration in Table 6 where Sadeed surpasses SUKOUN on WER, yet the correction is applied asymmetrically. The paper introduces the correction in Section 7.1, but presents no evidence that SUKOUN, D3, or PTCAD were evaluated on the corrected labels. Because Sadeed's training pipeline already applies the same phonological normalization, the corrected test set is effectively aligned with Sadeed's inductive biases, making the comparison unfair. The reader's additional observation that Table 8 contradicts the 'competitive with proprietary LLMs' claim is also well grounded: on Sadeed's own benchmark, Claude 3.7, GPT-4, and Gemini-Flash 2.0 all post substantially lower WER than Sadeed. These two issues together undermine the central claim that a 1.5B fine-tuned SLM achieves state-of-the-art results on Fadel and competitive results with proprietary models. The paper's contributions are not purely negative: the cleaned training corpus, the SadeedDiac-25 benchmark, and the overlap analysis between Fadel and Abbad are useful resources that can support fairer evaluation. But the headline model results require either re-running baselines on the corrected test set or abandoning the SOTA claim. Since the reader already reached REJECT with moderate confidence, and this stress-test confirms the most load-bearing concern, no change to the verdict is needed.","tokens_in":23460,"tokens_out":3747,"duration_ms":39346,"concrete_test":"Obtain the released phonologically corrected Fadel test set and re-run SUKOUN (and, if feasible, D3 and PTCAD) using the same inference and Needleman-Wunsch alignment protocol applied to Sadeed. If SUKOUN's WER on the corrected set is at or below Sadeed's 1.7115, or if Sadeed's original-set WER is not reproduced at 1.7955, the Table 6 SOTA assertion collapses. The authors' public GitHub repository should make this a direct check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central SOTA claim rests entirely on the rows labeled 'Sadeed-Fadel corrected' in Table 6. Section 7.1 states that the authors modified the Fadel test set by 'systematically' changing the vowelization of the first consonant in iltiqa' assakinayn cases according to standard Arabic phonological rules. This is the same type of normalization applied to Sadeed's training data in Section 3. The comparable rows for SUKOUN, D3, and PTCAD are taken from previous papers that evaluated on the original Fadel labels; no baseline was re-run on the corrected set. Thus Sadeed is compared to models scored against a different, unmodified ground truth, while Sadeed is scored against a label set rewritten to match its own training conventions. On the original Fadel test set, Sadeed's best WER (1.7955, excluding no-diacritic, without case endings) is worse than SUKOUN's 1.77; only on the corrected set does Sadeed reach 1.7115 and claim leadership. The secondary contribution, 'competitive results compared to proprietary large language models,' is also contradicted by Table 8, where Sadeed's WER on SadeedDiac-25 is 13.74 versus Claude 3.7's 4.67 and GPT-4's 5.27. The released dataset, benchmark, and overlap analysis are useful, but the headline model claims are not supported by the evidence as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Sadeed, a 1.5-billion-parameter decoder-only model for Arabic diacritization, fine-tuned from the Kuwain base model on a newly cleaned version of the Tashkeela corpus combined with ATB-3 (about 1.04 million samples, roughly 53 million words). It also introduces SadeedDiac-25, a 1,200-paragraph expert-reviewed benchmark spanning Classical and Modern Standard Arabic, and reports an overlap analysis indicating substantial leakage between the Fadel and Abbad benchmark splits. The evaluation claims state-of-the-art word error rate (WER) on the Fadel benchmark (Table 6), competitive performance versus proprietary large language models on SadeedDiac-25 (Table 8), and competitive results on WikiNews (Section 7.2). The key evidence for the headline claim is Table 6, where Sadeed's best numbers are reported on a phonologically 'corrected' version of the Fadel test labels, while all baseline rows are taken from prior publications scored on the original labels.","tokens_in":23753,"tokens_out":16616,"duration_ms":147142,"significance":"The contributions are potentially useful even if the headline claims are scaled back. The released training dataset, the SadeedDiac-25 benchmark with expert review, the Fadel/Abbad overlap analysis, and the public evaluation code all support reproducibility, and the paper is transparent about hallucination rates and about Sadeed's weakness on Modern Standard Arabic. If the WER state-of-the-art claim were valid, the paper would also be significant in showing that a 1.5B fine-tuned SLM can surpass much larger and dedicated diacritization systems with modest compute. Unfortunately, as detailed below, the two headline claims are not supported by the evidence as presented: the Fadel comparison is asymmetric, and Table 8 contradicts the claim of competitiveness with proprietary models. The manuscript's value currently lies mainly in its resources and analysis rather than in its model-evaluation claims.","major_comments":[{"comment":"The central WER state-of-the-art claim rests on an asymmetric comparison. The 'Sadeed-Fadel corrected' rows are scored on a version of the Fadel test set that the authors modified with the same phonological rule (adjusting the vowelization of the first consonant in iltiqa' assakinayn) that was applied to Sadeed's training data in Section 3, whereas the D3, SUKOUN, PTCAD, and Fadel baseline rows match published values and there is no indication that these baselines were re-scored on the corrected set. On the original Fadel test set, Sadeed's best WER (1.7955, excluding no-diacritic, without case endings) is worse than SUKOUN's 1.77, and the 1.7115 value that motivates the state-of-the-art claim is obtained only on the corrected set. The DER comparison shows the same pattern: Sadeed beats SUKOUN only in the without-case-ending columns, while SUKOUN's with-case-ending DER (1.16-1.23) is better than Sadeed's (1.58-1.68) on the original set. The sentence 'Sadeed achieves state-of-the-art (SOTA) performance on the Fadel dataset in terms of Word Error Rate' is therefore not supported for the original benchmark. The authors should re-run all baselines on both the original and the corrected test sets with their own evaluation code, or restrict the comparison to a configuration in which all models are scored under identical conditions.","section":"§7.1, Table 6"},{"comment":"The Sadeed scores in Table 6 are computed after a hallucination-correction stage that has no counterpart in the baseline numbers. Section 5 describes aligning the raw generation to the input with the Needleman-Wunsch algorithm, removing added words, restoring missing words, and replacing altered words with their non-diacritized versions, leaving some words undiacritized in the final output. The WER and DER therefore measure a post-processed output, and the 'excluding no diacritic' columns — exactly the columns in which Sadeed leads — are those that exclude the words left undiacritized by this procedure. The paper neither reports the fraction of test words left undiacritized after correction nor gives raw uncorrected scores, so even the 'Sadeed-Fadel original' rows are not strictly comparable to the baseline results under a common operational definition of WER.","section":"§5, §7.1"},{"comment":"The abstract's claim that Sadeed 'achieves competitive results compared to proprietary large language models' is contradicted by the paper's own Table 8. On SadeedDiac-25, Sadeed's WER is 13.74 with case endings and 9.92 without, versus 4.67 and 2.31 for Claude 3.7 Sonnet and 5.27 (with case endings) for GPT-4; the gap is roughly threefold and is larger than the 7.19 hallucination-related WER points that the paper itself identifies as Sadeed's main weakness. The abstract, introduction, and Section 7.3 should be revised to state that Sadeed is the best of the evaluated open-weight models but substantially behind the best proprietary systems.","section":"Abstract, §7.3, Table 8"},{"comment":"The GPT-4 row of Table 8 is internally inconsistent and appears to contain a transcription error. The without-case-ending WER (10.9274) is reported as roughly double the with-case-ending WER (5.2719), and the DER value (3.8645) is identical in both columns; no other row in the table shows this pattern, and under the standard definitions removing case endings cannot increase WER. All entries of Table 8, and the evaluation logs behind them, should be re-verified before the benchmark results can be considered reliable.","section":"Table 8"},{"comment":"The novelty claim for SadeedDiac-25 is overstated. The paper states that 'this dataset remains entirely novel to existing models' because the diacritization was performed in-house, but 600 of the 1,200 benchmark paragraphs are drawn from the Fadel test set, a public benchmark since 2019, and the paper's own Section 6.1 shows that the Fadel test set overlaps substantially with the Abbad training data (865 Fadel-test samples appear identically in the Abbad train set). The novelty claim should be restricted to the 454 originally curated paragraphs, and the possibility that the Fadel-derived portion is familiar to some evaluated models should be acknowledged.","section":"§4"}],"minor_comments":[{"comment":"The sentence 'we systematically the vowelization of the first consonant based on standard Arabic phonological rules' is missing a main verb and should read, for example, 'we systematically adjusted the vowelization'.","section":"§7.1"},{"comment":"The paper is inconsistent in spelling the model name: the footnote on page 1 and the caption of Figure 2 use 'Sadded', while the title, abstract, and body use 'Sadeed'; the spelling should be unified.","section":"Page 1 footnote, Figure 2"},{"comment":"The bibliography lists Darwish et al. 2017a and 2017b as two entries with identical titles, venues, and page ranges, but they are cited for different purposes in Sections 4 and 7.2; the duplicate should be re-keyed to the correct publication, and the citation in Section 7.2 should be checked for consistency with Table 7, which attributes the FRRNN results to Darwish et al. [2020].","section":"References"},{"comment":"After the filter that 'removed examples containing three or more words with partial diacritics', the statement that 'only 10.7% of the sentences in the dataset contain at most two partially diacritized words' is logically inconsistent, since all remaining sentences satisfy that condition; the intended statistic should be reworded or recomputed.","section":"§3"},{"comment":"The eight metric columns in Table 8 would benefit from clearer visual grouping of the 'With Case Ending' and 'Without Case Ending' blocks, because the current layout makes it easy to misassign values across the two blocks.","section":"Table 8"},{"comment":"Several Arabic examples show broken glyph or ligature rendering in the manuscript (for example, Table 1 and Figure 3); the camera-ready version should verify the typesetting of diacritized Arabic.","section":"Tables 1, 3, Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The main claims are fixable in principle, since the authors control the evaluation protocol and could re-run the open baselines on the corrected Fadel test set; if they cannot obtain the D3/SUKOUN/PTCAD checkpoints, the appropriate outcome is a substantial downgrade of the claims rather than their retention in modified form. I would also ask the editor to verify, at revision time, that the released evaluation code reproduces Tables 6-8, since the internally inconsistent GPT-4 row raises questions about the evaluation pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: the benchmark and cleaned dataset are real contributions; the model's headline SOTA claim does not survive contact with the paper's own tables. The paper is worth engaging for the resources, not for Sadeed's reported performance.\n\nNew: SadeedDiac-25 with MSA/CA split and expert review; Sadeed_Tashkeela cleaned corpus (~53M words) with overlap removal; and a quantified overlap analysis between Fadel and Abbad, showing 68% of the Fadel test set has similarity >0.5 with Abbad's training set. That is useful, and they shipped code and data.\n\nWhere it goes soft: the central WER SOTA on Fadel rests entirely on a phonologically corrected test set. The correction, which is the same iltiqa' assakinayn rule used in training-data cleaning, is applied only to Sadeed's evaluation; the published baselines were not re-run on the corrected set. On the original Fadel labels, Sadeed's best WER is 1.7955, worse than SUKOUN's 1.77. So the SOTA claim is an artifact of asymmetric evaluation. The 'competitive with proprietary models' claim is also not supported: on SadeedDiac-25, Claude 3.7 gets WER 4.67, Sadeed 13.74, and 7.19 of its 9.92 WER without case endings comes from hallucinated words. That is honest in the sense that they report it, but it undercuts the abstract's framing.\n\nAlso minor: Table 7 on WikiNews shows Sadeed solidly behind Darwish 2020. The paper acknowledges the MSA weakness, so that is a real limitation, not a hidden one.\n\nThe benchmark itself has a wrinkle: 600 of its 1200 paragraphs come from the Fadel test set, so it is not a fully independent CA evaluation even if the MSA portion is new. The expert-review process is described but no inter-annotator agreement numbers are given, so the curation quality is taken on faith.\n\nBottom line: the dataset and benchmark work deserve a serious referee and probably a citation. The model evaluation needs a major rework: re-run baselines on the corrected test set, or drop the SOTA claim. This should go to peer review with a request for revision, not a desk reject.","headline":"Useful benchmark and cleaned dataset, but the model's SOTA claim is an artifact of scoring Sadeed on a corrected test set without re-running baselines.","tokens_in":24393,"tokens_out":1649,"would_cite":true,"duration_ms":16459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sadeed, a 1.5B fine-tuned Arabic model, claims state-of-the-art Word Error Rate on the Fadel diacritization test set and introduces a fresh benchmark.","keywords":["Arabic diacritization","small language model","fine-tuning","Tashkeela corpus","benchmark contamination","Word Error Rate","Classical Arabic","Modern Standard Arabic"],"falsifier":"Re-run the prior systems (SUKOUN, D3, PTCAD, Fadel) on the released corrected Fadel test set with the same 'excluding no diacritic, without case endings' metric. If any of them posts a Word Error Rate at or below Sadeed's 1.7115, the central claim fails. A second check: verify the iltiqa' as-sakinayn phonological rule used to alter the test set against a standard Arabic grammar reference; if the rule is disputed or was applied inconsistently, the corrected benchmark is not a valid comparison instrument.","tokens_in":23198,"feed_emoji":"🕌","tokens_out":8040,"duration_ms":73875,"temperature":0.7,"pith_summary":"Arabic diacritization restores the short vowels and other marks that ordinary written Arabic leaves out, and it matters for speech synthesis, machine translation, and language learning tools. This paper claims that Sadeed, a fine-tuned 1.5B parameter language model, achieves the best reported Word Error Rate on the standard Fadel test set for Classical Arabic, particularly when the metric ignores characters that were never diacritized, and that it does so with modest computational resources. The authors also argue that earlier high scores are inflated by overlap between the Fadel and Abbad training and test splits, and they release a cleaned 53-million-word training set plus a new expert-reviewed benchmark, SadeedDiac-25, spanning Classical and Modern Standard Arabic. If these claims hold, a compact task-specific model can compete with proprietary large language models on this task.","feed_headline":"A 1.5B Arabic model tops the Fadel diacritization benchmark","feed_subtitle":"Fine-tuned Sadeed beats prior systems on the corrected Fadel test set and ships a cleaned dataset plus a fresh benchmark.","key_machinery":"The load-bearing object is Sadeed itself, a 1.5B-parameter decoder-only language model fine-tuned from Kuwain, with diacritization reformulated as a question-answering task: the model receives a system prompt and a non-diacritized input, and it generates the diacritized text. The second piece of machinery is post-processing alignment: raw model output is aligned to the original input with the Needleman-Wunsch algorithm, which removes inserted words, restores omitted words, and replaces altered words with their non-diacritized originals, so the final output is a faithful diacritized version of the input. The third piece is the data pipeline: roughly 53 million words of Tashkeela and ATB-3 text, cleaned with normalization rules, filtered to keep fully diacritized sentences, and de-overlapped against the Fadel test set.","core_discovery":"The paper's central claim is that Arabic diacritization can be solved at small scale by fine-tuning a decoder-only 1.5B Arabic language model on clean, overlap-free data, with the task framed as question-answering. On the original Fadel test set, Sadeed's best Word Error Rate is 1.7955 without case endings, and 1.7115 on the paper's phonologically corrected version of the same set, both measured while excluding characters that carry no diacritic. The paper reports that on the corrected set Sadeed's WER is lower than SUKOUN, D3, PTCAD, and the original Fadel model, and it argues this is the fair comparison because prior models were trained on data overlapping the test set. On its new SadeedDiac-25 benchmark, Sadeed trails Claude 3.7 Sonnet but outperforms all evaluated open Arabic models, and it is competitive with GPT-4 and Gemini-Flash 2.0. The authors identify hallucinations, especially on non-Arabic words, as the main source of Sadeed's remaining errors.","pith_inferences":["Editorial inference: the same fine-tune-plus-alignment recipe should transfer to other tasks that restore omitted annotation layers, such as Hebrew niqqud or phonetic transcription from plain text, where a generative model can hallucinate and sequence alignment can repair the output.","Editorial inference: if the phonological correction to the Fadel test set becomes standard practice, then the 'excluding no diacritic' WER without case endings is the metric most likely to be adopted for reporting, since it isolates the core diacritization decision from case-ending debates.","Editorial inference: a testable extension of the paper's MSA argument is to fine-tune Sadeed on an MSA-heavy subset and measure WikiNews WER; if the gap to the in-domain-trained model closes, then data mix rather than capacity explains the current MSA shortfall."],"forward_implications":["A 1.5B fine-tuned model can outperform far larger and proprietary systems on Classical Arabic diacritization, making the task accessible to teams with modest compute.","Training on the released cleaned set avoids the Fadel/Abbad overlap, so future WER comparisons on the Fadel test set are not inflated by leakage.","Existing results from models trained on both Fadel and Abbad data need re-examination, since over a third of the Fadel test set appears verbatim in Abbad training data.","SadeedDiac-25 provides a fresh, expert-reviewed benchmark that mixes Classical and Modern Standard Arabic, giving future models a contamination-free evaluation instrument.","Modern Standard Arabic remains the bottleneck: Sadeed's lower scores on MSA-heavy data point to training-data scarcity rather than model architecture as the limiting factor."],"supporting_citations":[{"why":"Source of the Kuwain 1.5B base model that Sadeed is fine-tuned from; supplies the pretrained weights and the language-injection pretraining approach.","marker":"Hennara et al. [2025]"},{"why":"Introduces the Fadel test set and the original Tashkeela cleaning methodology; primary benchmark and baseline.","marker":"Fadel et al. [2019a]"},{"why":"Provides the Tashkeela corpus, the raw material for the cleaned 53-million-word training set.","marker":"Zerrouki & Balla [2017]"},{"why":"Supplies ATB-3, the MSA portion of the training data.","marker":"Maamouri et al. [2008]"},{"why":"Reports the SUKOUN baseline that Sadeed claims to beat on the corrected Fadel test set.","marker":"Kharsa et al. [2024]"},{"why":"PTCAD, the comparison model whose Fadel/Abbad training setup exemplifies the data contamination the paper quantifies.","marker":"Skiredj & Berrada [2024]"},{"why":"Defines the Abbad benchmark used in the overlap analysis and criticized for sentence truncation.","marker":"Abbad & Xiong [2020]"},{"why":"Creates the CATT benchmark and model whose punctuation removal and labeling errors the paper analyzes.","marker":"Alasmary et al. [2024]"},{"why":"Provides the WikiNews in-domain baseline that Sadeed does not surpass, used to argue for more MSA training data.","marker":"Darwish et al. [2020]"},{"why":"The Needleman-Wunsch alignment algorithm used to repair hallucinated model output during inference.","marker":"Likic [2008]"}],"fun_headline_variants":["1.5B Arabic model tops corrected Fadel benchmark","Small Arabic model beats prior diacritization systems","Sadeed: tiny fine-tune rivals big LLMs in diacritization","New benchmark exposes true leaders in Arabic diacritization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the phonologically corrected version of the Fadel test set is the right measure: the paper beats prior published models on Word Error Rate only after it changes the test set by applying a rule for adjacent consonants, and if that correction is not accepted or not applied to the comparison models, the state-of-the-art claim does not hold.","fun_headline_variants_meta":{"raw":{"variants":["1.5B Arabic model tops corrected Fadel benchmark","Small Arabic model beats prior diacritization systems","Sadeed: tiny fine-tune rivals big LLMs in diacritization","New benchmark exposes true leaders in Arabic diacritization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3035,"prompt_tokens":980,"completion_tokens":2055,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":1983}},"tokens_in":596,"tokens_out":2055,"duration_ms":16847,"temperature":1.0,"reasoning_tokens":1983,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:57:42.583536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the prior systems (SUKOUN, D3, PTCAD, Fadel) on the released corrected Fadel test set with the same 'excluding no diacritic, without case endings' metric. If any of them posts a Word Error Rate at or below Sadeed's 1.7115, the central claim fails. A second check: verify the iltiqa' as-sakinayn phonological rule used to alter the test set against a standard Arabic grammar reference; if the rule is disputed or was applied inconsistently, the corrected benchmark is not a valid comparison instrument.","supporting_citations":[],"review_version":1}