Pith. sign in

REVIEW 4 major objections 7 minor 31 references

Arabic Hate Speech Identification and Masking in Social Media using Deep Learning Models and Pre-trained Models Fine-tuning

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fine-tuned QARiB model with AraBERT preprocessing reaches 92% Macro F1 on Arabic offensive tweet detection, edging past the best SemEval-2020 score, and the paper proposes hate-word masking as machine translation.

desk verdict New masking-as-MT task with an uninterpretable BLEU result; the detection part is plausible but incremental. read the letter →

arxiv 2507.23661 v1 pith:6VOMHDAT submitted 2025-07-31 cs.CL

Salam Thabet Doghmash , Motaz Saad This is my paper · ORCID
classification cs.CL
keywords DeeplearningHatespeechdetectionmaskingTransformerArabicNLPOffensivelanguageBLEUQARiB
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tackles two Arabic social-media problems in one pipeline: detecting offensive or hateful tweets, and "cleaning" a tweet by replacing each hate word with as many stars as the word has letters. For detection, the authors fine-tune and compare a set of deep and transformer models on the SemEval-2020 Arabic Offensive Language Identification dataset, reporting that QARiB with AraBERT preprocessing reaches 92% Macro F1 and 95% accuracy, a result they say beats the best score published in the shared task. For cleaning, they reframe masking as a machine translation problem: a transformer sequence-to-sequence model receives a dirty sentence and is trained to emit the same sentence with the bad words starred. Their best masking model reaches a 1-gram BLEU score of 0.30, which they describe as good compared with state-of-the-art machine translation systems. The wider point a sympathetic reader should take is that a single pre-trained Arabic language model can carry both tasks, and that text cleaning as translation is a new direction worth testing more rigorously.

What carries the argument

The load-bearing object for detection is QARiB, a BERT-style Arabic language model pre-trained on Twitter, combined with the AraBERT preprocessing pipeline that normalizes letters, strips punctuation and diacritics, removes repeated characters and non-Arabic words, and tokenizes with WordPiece before fine-tuning. That combination is what carries the 92% Macro F1 claim. For masking, the central mechanism is a sequence-to-sequence transformer built from an encoder-decoder pair with TextVectorization layers, L2 regularization, early stopping, and the '[start]' and '[end]' control tokens; it treats the dirty sentence as a source language and the star-masked sentence as a target language, and it is evaluated with the BLEU metric.

What would settle it

Run the same 401 test pairs through a copy-the-input baseline and through a lexicon-based star replacer, and compare their 1-gram BLEU scores with the transformer's 0.30; if either baselines matches or beats 0.30, the claimed masking quality is not established.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that fine-tuning the Arabic pre-trained language model QARiB, after applying the AraBERT preprocessing pipeline, yields a Macro F1 score of 92% and accuracy of 95% on the SemEval-2020 Arabic offensive language detection test set, surpassing the 90.17% Macro F1 of the best system in that shared task under the same data split. For the second task, the paper claims to open a new research direction by treating hate speech masking as a machine translation problem; a transformer encoder-decoder trained on a parallel corpus of dirty and star-masked sentence pairs produces a best 1-gram BLEU of 0.30 with a 4,783-pair training set and a 12,000-word vocabulary. The authors present this BLEU figure as a good result compared with state-of-the-art machine translation systems, and note that neither popular pre-trained hate-speech models nor zero-shot classifiers came close to QARiB on detection.

Load-bearing premise

The masking claim rests on treating a 0.30 one-gram BLEU score against one person's star-masked targets as a good result, yet no simple baseline—such as copying the input unchanged—is reported, and many test sentences contain no hate words and are therefore unchanged in the reference.

Editorial extensions

If this is right

  • If the detection result holds, fine-tuning one Arabic tweet-pretrained BERT-style model with normalization preprocessing beats every system entered in the SemEval-2020 Arabic offensive language task, including ensembles and multi-task models.
  • The masking-as-translation setup gives a concrete way to produce star-masked "clean" text automatically, meaning moderation pipelines could generate sanitized versions of a post rather than just flagging it.
  • The authors report that increasing the parallel corpus from 3,183 to 4,783 pairs raises the 1-gram BLEU from 0.29 to 0.30, which they take as evidence that a larger corpus would improve the masking model further.
  • The poor results of the fine-tuned hate-speech-specific models (best 75% Macro F1) compared with QARiB suggest that general Arabic language models fine-tuned on the target task outperform models pre-trained for toxicity on this dataset.
  • Because the same data split as the shared task is used, the 92% figure is directly comparable with the published 90.17% best result.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the BLEU 0.30 result is not interpretable without a trivial baseline; since a large share of the test sentences contain no hate words and appear unchanged in the target, a system that simply copies its input could plausibly score close to or above 0.30 on 1-gram BLEU.
  • Editorial inference: the detection claim is about a single Twitter dataset from one shared task; the same model could lose ground on other Arabic dialects, platforms, or newer slang, so the headline number should be read as in-domain rather than general-purpose.
  • Editorial inference: the paper's own future-work plan—larger parallel corpus and publishing the hate-word lexicon—is the natural experiment to test whether masking-as-translation actually learns to mask, because a lexicon-based baseline would then be available for comparison.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. This paper addresses two tasks for Arabic social media text: (1) detection of offensive/hate speech using deep learning and transformer models on the SemEval-2020 Arabic offensive language dataset, and (2) a newly proposed task of masking hate words by replacing them with stars, framed as a sequence-to-sequence machine translation problem. For detection, the authors report that QARiB with AraBERT-style preprocessing achieves a Macro F1 of 92% and accuracy of 95%, which they claim outperforms the best published SemEval-2020 result. For masking, they construct a parallel corpus from the SemEval-2020 data by manually star-masking offensive words, train a Transformer seq2seq model, and report a best BLEU-1 score of 0.30, which they characterize as a good result relative to state-of-the-art MT systems.

Significance. If the detection result were rigorously established, it would be a useful improvement for Arabic offensive language identification. The masking task is a potentially novel practical contribution for content moderation, and the paper provides an initial dataset and open-source code. However, both central claims are currently under-supported: the detection comparison with SemEval-2020 is confounded by the release date of QARiB, preprocessing differences, and lack of statistical significance testing, while the masking evaluation lacks baselines, annotation reliability analysis, and basic test-set statistics, making the '0.30 BLEU-1 is good' claim uninterpretable. The paper's positive aspects are its use of a public benchmark, explicit experimental descriptions, and a GitHub repository with related materials.

major comments (4)
  1. [§4.6, Tables 10–11; also Abstract and Conclusion] The masking evaluation reports only BLEU scores for the proposed model and provides no baselines. Because the task is identity-plus-masking (the target equals the source except that offensive words are replaced by stars), a trivial copy baseline that outputs the input unchanged will achieve perfect BLEU-1 on every sentence that contains no hate word, and the corpus is dominated by such sentences (2791 out of 4383 pairs are listed as not hate in Table 11). Without reporting an identity baseline, a lexicon-based replacement baseline, or at least the proportion of test sentences that actually contain masked words, the BLEU-1 value of 0.30 cannot be interpreted as evidence of masking quality. In addition, the phrase 'a good result compared with the state of the art MT systems' is misleading because standard MT evaluation uses cumulative BLEU (typically up to 4-gram) and MT source and target are not near-identical, whereas here only 1-gram BLEU is used and the source and target are almost the same string.
  2. [§3.5.1, Table 5; Conclusion] The claim that QARiB with AraBERT preprocessing 'outperformed the best results that are published in the SemEval-2020 shared task' is not a controlled comparison. QARiB (reference [25]) was released after the event, so the SemEval participants could not use it; the paper does not retrain the ALAMIHamza AraBERT system under the same preprocessing, data splits, and hyperparameter conditions, and it reports only a single run with no seed variance or significance test. Moreover, Section 3.2 applies a preprocessing pipeline that removes non-Arabic characters and normalizes letters; if the authors evaluate on these preprocessed tweets rather than the original SemEval-2020 test inputs, the comparison with the published leaderboard is not valid because the test input is no longer identical. To support the 'outperform' statement, the authors should run both models under identical conditions over multiple seeds, or explicitly soften the claim to 'on the same test set split'.
  3. [§4.1] The parallel corpus used for the masking task was prepared by a single volunteer ('a volunteer and prepared the second part from the parallel corpus'). The paper provides no annotation guidelines, no second annotation, and no inter-annotator agreement measure, yet this corpus serves as both the training supervision and the evaluation ground truth. Because the decision of which words count as 'bad' and how to mask them is subjective, the quality and consistency of these annotations are load-bearing for the reported BLEU scores. The authors should provide a detailed annotation protocol, a sample of annotated examples, and ideally an agreement measure such as Cohen's kappa, or at minimum explicitly acknowledge this as a pilot limitation.
  4. [§4.6, Tables 10–11] The masking model is tested on only 401 sentences (Table 9), and the best BLEU-1 difference between configurations (0.30 vs 0.29) is within the range expected from random variation on such a small test set. The paper does not report confidence intervals, bootstrap estimates, or significance tests. In addition, the 'Not-HS size' column in Tables 10–11 reports the total number of non-hate sentences in the whole dataset, not the composition of the test set; the paper should report how many of the 401 test sentences contain at least one masked word. This statistic is necessary to interpret the BLEU numbers and to assess the copy-baseline concern raised in the previous comment.
minor comments (7)
  1. [Abstract, §4.6, Conclusion] The phrase 'a good result compared with the state of the art MT systems' is vague; please specify which systems are used as a reference or remove the comparison, since no MT baselines are actually run in the paper.
  2. [Introduction, Conclusion] The novelty claims ('there is no research which addresses the problem of cleaning such texts' and 'first experiment in Arabic hate speech masking') should be supported by a literature search that includes text detoxification and offensive-language-removal work; if the claim is specifically 'first for Arabic', that should be stated and justified.
  3. [§3.1] The SemEval-2020 Task 12 dataset is labeled for offensive language, not strictly hate speech; the paper should clarify the relationship between the two concepts and avoid using the terms interchangeably.
  4. [§3.4.3] There is a typo: 'ReLO' should be 'ReLU'.
  5. [Tables 4–8] The table column layout for the macro-average scores is confusing; please separate the per-class columns from the macro-average columns with a clear header or a vertical line.
  6. [§4.5] Reference [29] is a paper on dialogue metrics, not the canonical BLEU implementation; please cite and specify the exact BLEU implementation (e.g., NLTK's sentence_bleu), including tokenization and smoothing parameters.
  7. [Availability of data and materials] Please state explicitly whether the newly annotated parallel corpus is released in the GitHub repository; the current text says materials are available but also says details can be provided upon request, which is ambiguous for a central new resource.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; detection is anchored to an external benchmark and masking is standard supervised learning.

full rationale

The detection claim (QARiB + AraBERT, Macro F1 92%, accuracy 95%) is evaluated on the SemEval-2020 OffensEval Arabic test split and compared with published shared-task results, so it reduces to an external benchmark rather than to the paper's own assumptions. The masking claim is trained and tested on a parallel corpus in which a volunteer manually star-replaced hate words; this is ordinary supervised sequence-to-sequence learning, and the BLEU score is computed against held-out reference sentences, not against a fitted parameter or a self-cited theorem. There is no equation in the paper that defines a predicted quantity in terms of an input fit, and no load-bearing self-citation: the cited prior work on AraBERT preprocessing is by other authors and is used as a baseline, not as a forced premise. The absence of an identity/copy baseline or lexicon baseline makes the reported BLEU-1 of 0.30 uninterpretable as evidence of good masking quality, and the comparison with machine translation systems is questionable, but this is a lack of control in evaluation design rather than circularity. Accordingly the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claims rest on several assumptions: the identification task uses offensive language labels as a proxy for hate speech; the masking task relies on annotations from a single volunteer; and BLEU is used as the evaluation metric without baselines. There are no invented physical or conceptual entities introduced by the paper. The free parameters (vocabulary size, dataset size, preprocessing steps) are chosen by hand or by test-set performance, which affects the reported results.

free parameters (3)
  • vocabulary_size = 12000
    Selected as the configuration giving the highest BLEU-1 on the test set among five vocabulary sizes in Table 11, which is a form of test-set tuning without a separate validation selection.
  • dataset_size = 4383
    The larger of two corpus sizes compared in Section 4.6; the authors choose it as the best, but the text also says 4783 pairs in Section 4.2, creating ambiguity.
  • preprocessing choices = letter normalization, remove non-Arabic words, remove repeating characters
    Ad hoc preprocessing steps in Section 3.2 that affect tokenization and model performance; they are not systematically evaluated or justified.
assumptions (3)
  • domain assumption SemEval-2020 offensive language labels are treated as hate speech labels
    Section 1 and Section 3.1 use the offensive language dataset for 'hate speech detection' without justifying that offensive equals hate speech.
  • domain assumption The single volunteer's masking annotations are correct ground truth
    Section 4.1 describes one volunteer masking bad words; no inter-annotator agreement, validation, or quality checks are reported.
  • domain assumption BLEU is an appropriate evaluation metric for the masking task
    Section 4.5 applies BLEU without discussing its suitability for a task where most output tokens are identical to input tokens and the only changes are a small number of 'star' replacements.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Arabic Hate Speech Identification and Masking in Social Media using Deep Learning Models and Pre-trained Models Fine-tuning." pith.science (2026). https://pith.science/paper/6VOMHDAT

@misc{pith2026250723661,
  author       = {Pith},
  title        = {Pith review of: Arabic Hate Speech Identification and Masking in Social Media using Deep Learning Models and Pre-trained Models Fine-tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6VOMHDAT}},
  note         = {Machine review of arXiv:2507.23661}
}
read the original abstract

Hate speech identification in social media has become an increasingly important issue in recent years. In this research, we address two problems: 1) to detect hate speech in Arabic text, 2) to clean a given text from hate speech. The meaning of cleaning here is replacing each bad word with stars based on the number of letters for each word. Regarding the first problem, we conduct several experiments using deep learning models and transformers to determine the best model in terms of the F1 score. Regarding second problem, we consider it as a machine translation task, where the input is a sentence containing dirty text and the output is the same sentence with masking the dirty text. The presented methods achieve the best model in hate speech detection with a 92\% Macro F1 score and 95\% accuracy. Regarding the text cleaning experiment, the best result in the hate speech masking model reached 0.3 in BLEU score with 1-gram, which is a good result compared with the state of the art machine translation systems.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 21 canonical work pages

  1. [25]

    CoRR abs/2102.10684 (2021) https://arxiv.org/abs/2102.10684

    Abdelali, A., Hassan, S., Mubarak, H., Darwish, K., Samih, Y.: Pre-training BERT on arabic tweets: Practical considerations. CoRR abs/2102.10684 (2021) https://arxiv.org/abs/2102.10684

    arXiv 2021

  2. [1]

    ACM Computing Surveys (CSUR) 51(4), 1–30 (2018)

    Fortuna, P., Nunes, S.: A survey on automatic detection of hate speech in text. ACM Computing Surveys (CSUR) 51(4), 1–30 (2018)

    work page 2018

  3. [2]

    (2017) LATEX template Arabic Hate Speech Identification and Masking 21

    Fortuna, P.: Automatic detection of hate speech in text: an overview of the topic and dataset annotation with hierarchical classes. (2017) LATEX template Arabic Hate Speech Identification and Masking 21

    work page 2017

  4. [3]

    In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp

    Basile, V., Bosco, C., Fersini, E., Nozza, D., Patti, V., Rangel Pardo, F.M., Rosso, P., Sanguinetti, M.: SemEval-2019 task 5: Multilingual detection of hate speech against immigrants and women in Twitter. In: Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 54–63. Association for Computational Linguistics, Minneapolis, Minnesota...

    doi:10.18653/v1/s19-2007 2019

  5. [4]

    25–35 (2019)

    Davidson, T., Bhattacharya, D., Weber, I.: Racial bias in hate speech and abusive language detection datasets, pp. 25–35 (2019). https://doi.org/ 10.18653/v1/W19-3504

    doi:10.18653/v1/w19-3504 2019

  6. [5]

    Zampieri, M., Nakov, P., Rosenthal, S., Atanasova, P., Karadzhov, G., Mubarak, H., Derczynski, L., Pitenis, Z., ltekin, .: SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)

    work page 2020

  7. [6]

    Language and Computers, 171–189 (2004)

    Teubert, W.: Units of meaning, parallel corpora, and their implications for language teaching. Language and Computers, 171–189 (2004)

    work page 2004

  8. [7]

    In: Proceedings of the Third Workshop on Abusive Language Online, pp

    Mulki, H., Haddad, H., Bechikh Ali, C., Alshabani, H.: L-HSAB: A Levan- tine Twitter dataset for hate speech and abusive language. In: Proceedings of the Third Workshop on Abusive Language Online, pp. 111–118. Asso- ciation for Computational Linguistics, Florence, Italy (2019). https://doi. org/10.18653/v1/W19-3512. https://aclanthology.org/W19-3512

    work page doi:10.18653/v1/w19-3512 2019

Show all 31 references
  1. [8]

    2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 69–76 (2018)

    Albadi, N., Kurdi, M., Mishra, S.: Are they our brothers? analysis and detection of religious hate speech in the arabic twittersphere. 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM), 69–76 (2018)

    2018

  2. [9]

    In: Proceedings of the First Workshop on Abusive Language Online, pp

    Mubarak, H., Darwish, K., Magdy, W.: Abusive language detection on Arabic social media. In: Proceedings of the First Workshop on Abusive Language Online, pp. 52–56. Association for Computational Linguistics, Vancouver, BC, Canada (2017). https://doi.org/10.18653/v1/W17-3008. h...

    2017 doi

  3. [10]

    In: 2018 International Confer- ence on Control, Electronics, Renewable Energy and Communications (ICCEREC), pp

    Saksesi, A.S., Nasrun, M., Setianingsih, C.: Analysis text of hate speech detection using recurrent neural network. In: 2018 International Confer- ence on Control, Electronics, Renewable Energy and Communications (ICCEREC), pp. 242–248 (2018). https://doi.org/10.1109/ICCEREC. ...

    2018

  4. [11]

    IEEE Transactions on Knowledge and Data Engineering 33(4), 1802–1806 (2021)

    Qiang, J., Wu, X.: Unsupervised statistical text simplification. IEEE Transactions on Knowledge and Data Engineering 33(4), 1802–1806 (2021). https://doi.org/10.1109/TKDE.2019.2947679 LATEX template 22 Arabic Hate Speech Identification and Masking

    2021

  5. [12]

    In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp

    Alami, H., Ouatik El Alaoui, S., Benlahbib, A., En-nahnahi, N.: LISAC FSDM-USMBA team at SemEval-2020 task 12: Overcoming AraBERT’s pretrain-finetune discrepancy for Arabic offensive language identification. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp...

    2020 doi

  6. [13]

    In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp

    Hassan, S., Samih, Y., Mubarak, H., Abdelali, A.: ALT at SemEval- 2020 task 12: Arabic and English offensive language identification in social media. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp. 1891–1897. International Committee for Computational Lin...

    2020 doi

  7. [14]

    1448–1455 (2020)

    Wang, S., Liu, J., Ouyang, X., Sun, Y.: Galileo at semeval-2020 task 12: Multi-lingual learning for offensive language identification using pre-trained language models, pp. 1448–1455 (2020). https://doi.org/10. 18653/v1/2020.semeval-1.189

    2020

  8. [15]

    2054–2059 (2020)

    Safaya, A., Abdullatif, M., Yuret, D.: Kuisail at semeval-2020 task 12: Bert-cnn for offensive speech identification in social media, pp. 2054–2059 (2020). https://doi.org/10.18653/v1/2020.semeval-1.271

    2020 doi

  9. [16]

    In: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pp

    Keleg, A., El-Beltagy, S.R., Khalil, M.: ASU OPTO at OSACT4 - offensive language detection for Arabic text. In: Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pp. 66–70. Euro- pean Languag...

    2020

  10. [17]

    Mnassri, K., Rajapaksha, P., Farahbakhsh, R., Crespi, N.: Hate Speech and Offensive Language Detection using an Emotion-aware Shared Encoder (2023)

    2023

  11. [18]

    In: 2019 Sixth Interna- tional Conference on Social Networks Analysis, Management and Security (SNAMS), pp

    Mohaouchane, H., Mourhir, A., Nikolov, N.S.: Detecting offensive lan- guage on arabic social media using deep learning. In: 2019 Sixth Interna- tional Conference on Social Networks Analysis, Management and Security (SNAMS), pp. 466–471 (2019). https://doi.org/10.1109/SNAMS.201...

    2019 doi

  12. [19]

    CoRR abs/2102.05708 (2021) https://arxiv.org/abs/2102.05708

    Husain, F., Uzuner, ¨O.: Transfer learning approach for arabic offensive language detection system - bert-based model. CoRR abs/2102.05708 (2021) https://arxiv.org/abs/2102.05708

    2021 arXiv

  13. [20]

    Procedia LATEX template Arabic Hate Speech Identification and Masking 23 Computer Science 142, 174–181 (2018)

    Alakrot, A., Murray, L., Nikolov, N.S.: Dataset construction for the detec- tion of anti-social behaviour in online communication in arabic. Procedia LATEX template Arabic Hate Speech Identification and Masking 23 Computer Science 142, 174–181 (2018). https://doi.org/10.1016/j...

    2018 doi

  14. [21]

    Applied Sciences 10(23) (2020)

    Alshalan, R., Al-Khalifa, H.: A deep learning approach for automatic hate speech detection in the saudi twittersphere. Applied Sciences 10(23) (2020). https://doi.org/10.3390/app10238614

    2020 doi

  15. [22]

    In: Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - ICPRAM,, pp

    Faris., H., Aljarah., I., Habib., M., Castillo., P.A.: Hate Speech Detec- tion Using Word Embedding and Deep Learning in the Arabic Lan- guage Context. In: Proceedings of the 9th International Conference on Pattern Recognition Applications and Methods - ICPRAM,, pp. 453–

  16. [23]

    Alyafeai, Z., Al-shaibani, M., Ghaleb, M., Ahmad, I.: Evaluating Various Tokenizers for Arabic Text Classification

  17. [24]

    Procedia Computer Science 117, 256–265 (2017)

    Mohammad, A.B., Eissa, K., El-Beltagy, S.: Aravec: A set of arabic word embedding models for use in arabic nlp. Procedia Computer Science 117, 256–265 (2017). https://doi.org/10.1016/j.procs.2017.10.117

    2017 doi

  18. [26]

    7088–7105 (2021)

    Abdul-Mageed, M., Elmadany, A., Nagoudi, E.M.B.: Arbert & marbert: Deep bidirectional transformers for arabic, pp. 7088–7105 (2021). https: //doi.org/10.18653/v1/2021.acl-long.551

    2021 doi

  19. [27]

    https://doi.org/10.3115/ 1073083.1073135

    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for auto- matic evaluation of machine translation (2002). https://doi.org/10.3115/ 1073083.1073135

    2002

  20. [28]

    In: Proceedings of Machine Translation Summit IX: Papers, New Orleans, USA (2003)

    Coughlin, D.: Correlating automated and human assessments of machine translation quality. In: Proceedings of Machine Translation Summit IX: Papers, New Orleans, USA (2003). https://aclanthology.org/2003. mtsummit-papers.9

    2003

  21. [29]

    CoRR abs/1706.09799 (2017) https://arxiv.org/abs/1706

    Sharma, S., Asri, L.E., Schulz, H., Zumer, J.: Relevance of unsuper- vised metrics in task-oriented dialogue for evaluating natural language generation. CoRR abs/1706.09799 (2017) https://arxiv.org/abs/1706. 09799

    2017 arXiv

  22. [275]

    https://aclanthology.org/2020.semeval-1.275

    2020

  23. [460]

    https://doi.org/10.5220/0008954004530460

    SciTePress, ??? (2020). https://doi.org/10.5220/0008954004530460. INSTICC

    2020 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.