SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation System

Adel Ammar; Omer Nacar; Serry Sibaee; Wadii Boulila; Yasser Al-Habashi

REVIEW 3 major objections 6 minor 14 references

SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation System

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned AraT5v2 models translate between Modern Standard Arabic and the Syrian dialect in both directions; GPT-4.1 rates the MSA-to-Shami output 4.01 out of 5 on accuracy, authenticity, and fluency.

desk verdict A useful model release undermined by a one-directional evaluation: the MSA-to-Shami score is plausible, but the Shami-to-MSA direction is never measured, so the 'bidirectional' claim is unsupported. read the letter →

arxiv 2508.02268 v1 pith:NNXK7D42 submitted 2025-08-04 cs.CL

Serry Sibaee , Omer Nacar , Yasser Al-Habashi , Adel Ammar , Wadii Boulila This is my paper

classification cs.CL

keywords SyriandialectModernStandardArabicmachinetranslationdiglossiadialectalNLPAraT5v2NabradatasetGPT-4.1evaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to build and test a bidirectional machine translation system for the Syrian dialect (Shami) and Modern Standard Arabic (MSA), a language pair left underserved because Arabic's formal standard and everyday dialects differ sharply. The authors fine-tune two AraT5v2-base-1024 models, one for each direction, on the Nabra corpus of authentic Syrian Arabic, then evaluate on 1,500 unseen sentence pairs from the MADAR Damascus subset. Their central reported result is an average holistic quality score of 4.01 out of 5.0 for MSA-to-Shami output, assigned by GPT-4.1 on criteria of semantic accuracy, dialectal authenticity, and fluency. If the score reflects real quality, the system would give content localization, social-media understanding, and educational tools a practical way to work in Syrian Arabic rather than MSA alone.

What carries the argument

The load-bearing component is AraT5v2-base-1024, a T5-style encoder-decoder pre-trained on Arabic text, which frames translation as text-to-text generation and supports sequences up to 1,024 tokens. The paper fine-tunes it for 22 epochs on the Nabra corpus, chosen for real-world sources (social media, scripts, lyrics, proverbs) and geographic coverage of Syrian sub-dialects, then evaluates with GPT-4.1 as an automated holistic judge scoring 0 to 5 on semantic accuracy, dialectal authenticity, and fluency. The same architecture is fine-tuned separately for each direction, making the system bidirectional while keeping each model specialized.

What would settle it

Have native Syrian Arabic speakers rate the same 1,500 MSA-to-Shami outputs on the paper's three criteria and compare their average and per-item scores with GPT-4.1's; an average well below 4.01, or low agreement with the model judge on individual items, would directly falsify the central quality claim.

Watch

Extended reading notes

Core claim

The paper's central claim is that a deliberately specialized recipe—an Arabic-only text-to-text Transformer fine-tuned on a small but authentic dialect corpus—produces high-fidelity Syrian dialect translations that a strong general-purpose judge rates as 'very good' to 'excellent' on average. SHAMI-MT and SHAMI-MT-2MSA are the two released directions of the same foundation model. On the MSA-to-Shami direction, GPT-4.1 awarded 4.01/5 across the full 1,500-sentence blind MADAR set; the authors attribute the remaining failures mainly to disagreements with the reference translation and to very short idiomatic phrases that resist surface-level translation. They argue this evidence, together with qualitative examples, shows the models exceed what generic multilingual models deliver for dialectal Arabic, and that releasing the checkpoints gives the community a reusable resource for Syrian dialect NLP.

Load-bearing premise

The quality claim stands or falls on whether GPT-4.1's holistic ratings are a valid and unbiased measure of Syrian dialect translation; the paper gives no human ratings or other metric to calibrate those scores.

Editorial extensions

If this is right

If the 4.01/5 score holds up, MSA-to-Shami translation is usable for subtitles, app interfaces, and marketing copy aimed at Syrian audiences, replacing stiff MSA with natural dialect.
The Shami-to-MSA model can help researchers and institutions normalize large volumes of Syrian social-media text into MSA for analysis and archiving.
Because both checkpoints are released, other groups can fine-tune further or adapt the recipe to nearby Levantine dialects without starting from scratch.
The identified failure modes—reference mismatches and ultra-short idiomatic phrases—define concrete targets for better training data and evaluation design.
The same fine-tuning recipe may transfer to other dialects if comparable authentic corpora exist, supporting the paper's claim that specialization beats generic multilinguality for dialect work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the headline number is only as strong as GPT-4.1's judgment; a human-rated subsample of the same 1,500 outputs would likely move the score, and should be the first replication check.
Editorial inference: because the evaluation uses reference-based scoring with an LLM judge, the 4.01 figure may capture fluency and dialect style better than BLEU, but it does not by itself prove the model handles out-of-domain or conversational code-switched Syrian text.
Editorial inference: a natural extension is to use one direction to generate weakly labeled training data for the other, or for adjacent Levantine dialects, which could lower the cost of expanding coverage.
Editorial inference: connecting these text models to speech recognition and synthesis, as the paper lists as future work, would turn the translation pair into spoken-language support for Syrian Arabic; the paper does not demonstrate that integration.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

One thing to know before you read this: the 'bidirectional' claim is not supported by the evidence. Section 4 evaluates only MSA-to-Shami. The Shami-to-MSA model gets two illustrative lines in Figure 2 but no scores, no error bars, no analysis. So the 4.01/5.0 headline applies to one direction, and the conclusion's 'rigorous evaluation on the MADAR benchmark confirms robust performance' goes beyond what was actually measured.

That said, the paper is not worthless. Fine-tuning AraT5v2 on Nabra is a standard recipe, but the authors release two models, including a Shami-to-MSA direction that is absent from their own prior work. They also use an authentic, regionally diverse dialect corpus, and their qualitative section acknowledges failure modes, including some where the reference is arguably the weaker translation. That honesty earns credit.

The soft spots are substantial. The missing reverse-direction evaluation is load-bearing: you cannot claim a bidirectional system without measuring both directions. The LLM judge (GPT-4.1) is unvalidated—no human ratings, no baseline, no error bars, and the exact prompt is undisclosed. The low-scoring examples in Table 2 actually cast doubt on the judge: a model output that preserves the temporal focus of the source while the reference shifts to spatial gets a 2, and a faithful if less colloquial rendering gets a 2. That looks like reference-similarity scoring, not translation quality. The selection of 1,500 MADAR sentences is also under-specified: why 1,500, and how were they drawn?

The central argument—that this is a high-fidelity, rigorously evaluated bidirectional system—does not hold as written. But it is fixable. The authors can measure the Shami-to-MSA direction, calibrate the judge against human judgments, add a baseline (even zero-shot AraT5v2), and describe the test-set sampling. If they do that, the resource becomes genuinely useful for a low-resource dialect pair.

I would not cite the 4.01 as evidence of quality; I would cite the model release if I worked on Syrian Arabic MT. For a reading group, maybe, if you want a case study in evaluation pitfalls. It deserves a serious referee—the topic is underserved and the models are public—but it needs major revision, not acceptance.

Referee Report

3 major / 6 minor

Summary. The paper introduces SHAMI-MT, a pair of fine-tuned AraT5v2 models intended for bidirectional machine translation between Modern Standard Arabic (MSA) and the Syrian dialect. The authors train on the Nâbra corpus and evaluate the MSA-to-Shami direction on 1,500 sentences from the MADAR Damascus subset, reporting an average quality score of 4.01/5.0 assigned by GPT-4.1. The paper also presents qualitative examples of both directions, an error analysis of low-scoring outputs, and releases both models on Hugging Face.

Significance. If the reported results were fully supported, this would be a useful public resource for an underserved language pair: the authors release two Hugging Face models, evaluate on the widely used MADAR corpus, and provide a qualitative analysis of failure modes. The work addresses a real gap in dialectal Arabic NLP and explicitly describes limitations rather than claiming perfection. However, the central advertised capability is bidirectional translation, and the quantitative evaluation covers only the MSA-to-Shami direction; the Shami-to-MSA model is never measured. In addition, the sole headline metric comes from an uncalibrated GPT-4.1 judge whose scoring behavior appears to diverge from the paper's own qualitative assessments. The empirical contribution is therefore not yet substantiated as claimed.

major comments (3)

[Section 4 and Section 6] The reported 4.01/5.0 average is computed only for the MSA-to-Shami direction. The evaluation protocol states that GPT-4.1 is given the MSA input, the model's predicted Shami translation, and the Shami ground truth; there is no analogous quantitative evaluation for SHAMI-MT-2MSA, the Shami-to-MSA model. The Conclusion's statement that 'rigorous evaluation on the MADAR benchmark confirms the model's robust performance' therefore overreaches: the bidirectional claim requires a quantitative measurement of the reverse direction, for example by scoring SHAMI-MT-2MSA outputs against the MADAR MSA side with the same or a calibrated judge.
[Section 4 and Table 2] The GPT-4.1 judge is used as the sole basis for the headline number, but the exact evaluation prompt is not disclosed, and the judge is not calibrated against human ratings or established MT metrics such as BLEU, chrF, or COMET. More importantly, Table 2 includes cases where the judge assigns a score of 2 while the paper's own analysis says both the model output and the ground truth are plausible, such as 'Both are plausible interpretations, but differ in intent.' This suggests the judge may be penalizing divergence from the reference rather than measuring translation quality, which makes the absolute 4.01 score difficult to interpret. The authors should report the evaluation prompt and provide a human-judgment correlation study or a secondary metric on a random subset.
[Section 3 and Section 4] The selection of 1,500 sentence pairs from the MADAR Damascus subset is unexplained: there is no description of whether the sentences were randomly sampled, stratified, or filtered, and no confidence intervals or score distributions are reported. Furthermore, no baseline is provided—neither AraT5v2 without fine-tuning, the authors' previous Shami-MT model [14], nor a general-purpose Arabic MT system. Without a baseline and without distributional information, the single average score of 4.01 cannot be contextualized, and the characterization of the system as 'high-fidelity' is underdetermined.

minor comments (6)

[Abstract and Section 3] The corpus name is inconsistently spelled: 'Nabra' in the Abstract and 'Nâbra' in Section 3; please standardize the spelling throughout.
[Section 1] The SHAMIYAT collection URL is truncated with an ellipsis; please provide the complete URL so that the public resource is actually accessible.
[Figure 2] The bidirectional examples in Figure 2 are shown only in Arabic script without transliteration, glosses, or an English translation, which makes it difficult for readers who do not read Arabic to verify the claimed quality.
[References] Reference [7] is cited for AraT5v2, but the listed entry describes AraT5; please cite the correct AraT5v2 technical report or clarify the version and provenance of the model.
[Section 3] The description of AraT5v2 as 'state-of-the-art' should either be supported with a specific citation or softened, since the paper does not compare against other recent Arabic foundation models.
[Section 4] The scoring scale is described as '0 to 5' in Section 4 and 'out of 5.0' in the Abstract; please clarify whether integer scores were used and report the actual distribution of scores rather than only the mean.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported 4.01 score is an empirical LLM-judged measurement on a held-out external benchmark, not a derivation from the paper's own inputs.

full rationale

SHAMI-MT is an empirical fine-tuning study: two AraT5v2 models are trained on the Nabra dataset and evaluated on a held-out MADAR Damascus subset. There is no formal derivation, no equation chain, and no fitted parameter that is later renamed as a prediction, so the self-definitional and fitted-input patterns do not apply. The only self-references are [6], a prior shared-task paper cited as related work, and [14], an earlier Shami-MT paper by the same authors; [14] is listed in the bibliography but is not invoked as evidence in the body, so no load-bearing conclusion rests on it. The GPT-4.1 judge raises a validity concern (possible overlap with Arabic dialect text in training data, undisclosed prompt, no human calibration), and the paper's bidirectional claim outruns Section 4, which reports quantitative scores only for the MSA-to-Shami direction; those are correctness risks, not circularity. The 4.01 score is an independent measurement of outputs against an external corpus, so the central claim is not equivalent to its inputs by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the suitability of AraT5v2, the authenticity of Nabra, the representativeness of the selected MADAR subset, and the validity of GPT-4.1 as a judge; none are independently established in this paper.

free parameters (5)

Number of fine-tuning epochs = 22
Chosen by hand; not derived from data.
Batch size = 256
Chosen training hyperparameter.
Initial learning rate = 5e-5
Chosen training hyperparameter with cosine schedule.
MADAR test set size = 1,500 sentence pairs
Authors 'specifically selected' this subset; no sampling protocol given.
GPT-4.1 evaluation prompt = Not specified
The exact prompt and decoding settings are not provided, making the judge setup a hidden parameter.

assumptions (4)

domain assumption AraT5v2-base-1024 is a strong Arabic text encoder-decoder for dialectal translation
Adopted from [7] without independent verification in this paper.
domain assumption Nabra is a comprehensive, authentic Syrian dialect corpus
Trusted from [9]; no dataset analysis or sample statistics provided.
domain assumption MADAR Damascus subset is a gold-standard benchmark and is disjoint from Nabra
Assumed 'unseen' without a decontamination check for overlap with training data.
ad hoc to paper GPT-4.1 gives valid, unbiased quality scores for Syrian dialect translation
Introduced for this evaluation; no human raters or correlation with established metrics.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation System." pith.science (2026). https://pith.science/paper/NNXK7D42

@misc{pith2026250802268,
  author       = {Pith},
  title        = {Pith review of: SHAMI-MT: A Syrian Arabic Dialect to Modern Standard Arabic Bidirectional Machine Translation System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNXK7D42}},
  note         = {Machine review of arXiv:2508.02268}
}

read the original abstract

The rich linguistic landscape of the Arab world is characterized by a significant gap between Modern Standard Arabic (MSA), the language of formal communication, and the diverse regional dialects used in everyday life. This diglossia presents a formidable challenge for natural language processing, particularly machine translation. This paper introduces \textbf{SHAMI-MT}, a bidirectional machine translation system specifically engineered to bridge the communication gap between MSA and the Syrian dialect. We present two specialized models, one for MSA-to-Shami and another for Shami-to-MSA translation, both built upon the state-of-the-art AraT5v2-base-1024 architecture. The models were fine-tuned on the comprehensive Nabra dataset and rigorously evaluated on unseen data from the MADAR corpus. Our MSA-to-Shami model achieved an outstanding average quality score of \textbf{4.01 out of 5.0} when judged by OPENAI model GPT-4.1, demonstrating its ability to produce translations that are not only accurate but also dialectally authentic. This work provides a crucial, high-fidelity tool for a previously underserved language pair, advancing the field of dialectal Arabic translation and offering significant applications in content localization, cultural heritage, and intercultural communication.

Figures

Figures reproduced from arXiv: 2508.02268 by the authors.

**Figure 2.** Examples of bidirectional translation using SHAMI-MT [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 10 canonical work pages

[14]

Shami-MT : A Machine Translation from MSA to Syrian Dialect

Serry Sibaee and Omer Nacar. Shami-MT : A Machine Translation from MSA to Syrian Dialect. 2025. 7

work page 2025
[1]

From LLM to NMT: Advancing low-resource machine translation with claude,

M. Enis and M. Hopkins, “From LLM to NMT: Advancing low-resource machine translation with claude,” 2024, arXiv:2404.13813

arXiv 2024
[2]

Saeed, A. M. A. (2025). Machine Translation Evaluation between Arabic and English during 2020 to 2024: A Review Study. Arts for Linguistic & Literary Studies, 7(2), 665-678

work page 2025
[3]

Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic identification of arabic dialects in social media. SoMeRA ’14, page 35–40, New York, NY , USA. Association for Computing Machinery

work page 2014
[4]

Libovický, J., Rosa, R., & Fraser, A. (2019). How language-neutral is multilingual BERT?. arXiv preprint arXiv:1911.03310

arXiv 2019
[5]

& Raffel, C

Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., ... & Raffel, C. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934

arXiv 2020
[6]

(2024, August)

Nacar, O., Sibaee, S., Alharbi, A., Ghouti, L., & Koubaa, A. (2024, August). ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for Arabic Language Enhancement. In Proceedings of The Second Arabic Natural Language Processing Conference (pp. 748-753)

work page 2024
[7]

AraT5: Text-to-Text Transformers for Arabic Language Generation

El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. AraT5: Text-to-Text Trans- formers for Arabic Language Generation. arXiv preprint arXiv:2109.12068v4, 2022

work page Pith review arXiv 2022

Show all 14 references

[8]

& Liu, P

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140), 1-67

2020
[9]

Nâbra: Syrian Arabic dialects with morphological annotations

Amal Nayouf, Tymaa Hasanain Hammouda, Mustafa Jarrar, Fadi A Zaraket, and Mohamad-Bassam Kurdy. Nâbra: Syrian Arabic dialects with morphological annotations. arXiv preprint arXiv:2310.17315, 2023

2023 arXiv
[10]

& Oflazer, K

Bouamor, H., Habash, N., Salameh, M., Zaghouani, W., Rambow, O., Abdulrahim, D., ... & Oflazer, K. (2018, May). The MADAR Arabic dialect corpus and lexicon. In Proceedings of the eleventh international conference on language resources and evaluation (LREC 2018)

2018
[11]

A Morphologically-Aware Fine-Grained Arabic to English SMT System

Salam Khalifa, Nizar Habash, and Houda Bouamor. A Morphologically-Aware Fine-Grained Arabic to English SMT System. In Proceedings of the Second Workshop on Arabic Corpora and Processing Tools , pages 58–65, 2016

2016
[12]

Pre-training Bert on Arabic Tweets: Practical Considerations

Ahmed Abdelali, Sabit Hassan, Hamdy Mubarak, Kareem Darwish, and Younes Samih. Pre-training Bert on Arabic Tweets: Practical Considerations. arXiv preprint arXiv:2102.10684, 2021

2021 arXiv
[13]

Towards building arabic paraphrasing benchmark

Marwah Alian, Arafat Awajan, Ahmad Al-Hasan, and Raeda Akuzhia. Towards building arabic paraphrasing benchmark. In Proceedings of the Second International conference on Data Science E-learning and Information Systems, pages 1–5, 2019

2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [14]

Shami-MT : A Machine Translation from MSA to Syrian Dialect

Serry Sibaee and Omer Nacar. Shami-MT : A Machine Translation from MSA to Syrian Dialect. 2025. 7

work page 2025

[2] [1]

From LLM to NMT: Advancing low-resource machine translation with claude,

M. Enis and M. Hopkins, “From LLM to NMT: Advancing low-resource machine translation with claude,” 2024, arXiv:2404.13813

arXiv 2024

[3] [2]

Saeed, A. M. A. (2025). Machine Translation Evaluation between Arabic and English during 2020 to 2024: A Review Study. Arts for Linguistic & Literary Studies, 7(2), 665-678

work page 2025

[4] [3]

Fatiha Sadat, Farnazeh Kazemi, and Atefeh Farzindar. 2014. Automatic identification of arabic dialects in social media. SoMeRA ’14, page 35–40, New York, NY , USA. Association for Computing Machinery

work page 2014

[5] [4]

Libovický, J., Rosa, R., & Fraser, A. (2019). How language-neutral is multilingual BERT?. arXiv preprint arXiv:1911.03310

arXiv 2019

[6] [5]

& Raffel, C

Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., ... & Raffel, C. (2020). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934

arXiv 2020

[7] [6]

(2024, August)

Nacar, O., Sibaee, S., Alharbi, A., Ghouti, L., & Koubaa, A. (2024, August). ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for Arabic Language Enhancement. In Proceedings of The Second Arabic Natural Language Processing Conference (pp. 748-753)

work page 2024

[8] [7]

AraT5: Text-to-Text Transformers for Arabic Language Generation

El Moatez Billah Nagoudi, AbdelRahim Elmadany, and Muhammad Abdul-Mageed. AraT5: Text-to-Text Trans- formers for Arabic Language Generation. arXiv preprint arXiv:2109.12068v4, 2022

work page Pith review arXiv 2022