REVIEW 4 major objections 4 minor 31 references
From Arabic Text to Puzzles: LLM-Driven Development of Arabic Educational Crosswords
T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Fine-tuning an open 8-billion-parameter model on a new Arabic clue dataset achieves a 78.86% top human rating for crossword clue quality.
desk verdict A useful Arabic clue dataset and fine-tuned models that deserve peer review, but the 'puzzle generator' framing and the single-annotator human eval are both overclaims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Arabic-Clue-Instruct dataset: 14,497 text-keyword-category entries and 54,196 Arabic clues produced by GPT-4-Turbo through a SELF-INSTRUCT-style prompting approach, meaning a stronger model generates the training examples from carefully crafted prompts that ask for concise, wordplay-friendly clues without repeating the answer. The dataset supplies both the training signal for fine-tuning and the reference for ROUGE comparison. The fine-tuning recipes are the second mechanism: LoRA on Llama3-8B-Instruct with r=32, alpha=64, and total batch size 128, and fine-tuning of GPT-3.5-Turbo, both over three epochs. The evaluation machinery is a five-level human rating system (A through E) designed to catch contextual relevance, answer leakage, and hallucination; this rating system, not raw ROUGE, carries the paper's main quality claim.
What would settle it
Have a panel of independent native Arabic speakers rate the same sample of clues from the fine-tuned Llama3-8B model using the paper's A-E rubric; if the average 'A' share falls well below 78.86% or inter-annotator agreement is low, the headline quality claim does not replicate. A cheaper check is to measure how often the fine-tuned model reproduces GPT-4-Turbo clues nearly verbatim, since the training data was generated by that model.
Extended reading notes
Core claim
The central discovery is that fine-tuning on a purpose-built Arabic clue dataset makes general-purpose LLMs competent crossword-clue writers for Arabic educational content. Using 14,497 filtered Arabic Wikipedia articles across 20 categories, the authors had GPT-4-Turbo generate 54,196 clues, with 67.5% of a 200-sample human evaluation rated 'A'. Fine-tuning GPT-3.5-Turbo and Llama3-8B-Instruct on this data raised ROUGE similarity to GPT-4-generated clues and, more importantly, lifted Llama3-8B's human 'A' rating from 36.02% to 78.86%, while GPT-3.5 went from 28.47% to 54.33%. The paper claims this demonstrates both the effectiveness of fine-tuning and the quality of the released Arabic-Clue-Instruct dataset as an educational resource.
Load-bearing premise
The load-bearing premise is that GPT-4-Turbo's clues are good enough to serve as both training signal and evaluation reference; this is checked by a single native Arabic speaker on only 200 samples, so systematic errors or dialect bias in those clues would be inherited by the fine-tuned models and would make the ROUGE comparisons against GPT-4 meaningless as quality evidence.
Editorial extensions
If this is right
- Teachers can input any Arabic educational text and obtain custom clue-answer pairs without needing a reference corpus or manual puzzle construction.
- The Arabic-Clue-Instruct dataset becomes a reusable resource for training and benchmarking Arabic generation models beyond crosswords.
- The fine-tuning recipe, using stronger-model-generated data followed by parameter-efficient tuning of an open 8B model, is a transferable pattern for other under-resourced languages.
- The reported 78.86% 'A' rate implies that an open 8B model can approach proprietary-model clue quality after relatively cheap fine-tuning.
Reading between the lines
- Editorial inference: the paper's single-native-speaker evaluation on 200 samples is the natural stress point; a multi-annotator replication with agreement statistics would tell whether the 78.86% 'A' rate is stable.
- Editorial inference: because ROUGE-L scores are near zero and the fine-tuned models occasionally add outside knowledge, as the paper's own analysis notes, a faithfulness or answer-containment filter could reduce the remaining low-rated clues.
- Editorial inference: the method's dependence on GPT-4-Turbo ground truth means its ceiling is set by GPT-4's clue quality; a version trained on expert-authored or human-validated clues only would test whether that ceiling binds.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Arabic-Clue-Instruct, a dataset of 14,497 Arabic Wikipedia article introductions (54,196 entries) with GPT-4-Turbo-generated crossword clues, and fine-tunes GPT-3.5-Turbo and Llama3-8B-Instruct on this dataset to generate Arabic educational crossword clues from input texts. The authors report automatic ROUGE evaluations (Tables 2 and 3) and a human evaluation (Table 4) in which the fine-tuned Llama3-8B achieves 78.86% of clues rated 'A'. The stated broader contribution is an 'Arabic crossword puzzle generator from a given text,' with a sample grid shown in Figure 7. The authors make the dataset and models publicly available.
Significance. If the clue-generation quality is established, the contribution is a useful open resource for Arabic educational technology: a reasonably large clue dataset, fine-tuned open-weight models, and reproducible code. The paper addresses a genuine gap in Arabic crossword clue generation and provides concrete artifact releases (GitHub, HuggingFace). However, the current evidence is substantially narrower than the claims. The paper actually demonstrates clue generation, not crossword-grid construction, and its central quality claim rests on a single-annotator evaluation and on a circular automatic metric. With careful revision, the resource and the clue-generation pipeline could become a solid contribution.
major comments (4)
- [Section 1, Section 3.2, Section 4.3, Figure 7] The abstract and title describe an 'Arabic crossword puzzle generator from a given text,' but the methodology and experiments cover only clue generation. No algorithm is presented for constructing a crossword grid, placing answers, enforcing intersections, or selecting clues to fit a grid; the example puzzle in Figure 7 appears to be manually assembled. This mismatch undermines the central claim. Either provide an actual grid-construction component and evaluate it, or explicitly reframe the contribution as clue generation for Arabic educational crosswords.
- [Section 4.3, Table 4] The headline result that fine-tuned Llama3-8B achieves 78.86% 'A' ratings is based on a single native Arabic-speaking evaluator, with no inter-annotator agreement metric, no blinding, and no evidence that the rubric was applied consistently across sessions or across models. The fine-tuning improvement is the paper's core quantitative claim, so this evaluation needs to be reproduced with at least two independent annotators, with agreement reported (e.g., Cohen's kappa) and with explicit handling of potential evaluator bias.
- [Section 3.1, Section 4.2, Tables 2 and 3] The automatic evaluation is circular: the fine-tuning data were generated by GPT-4-Turbo, and Table 3 measures ROUGE similarity between the fine-tuned models' outputs and the same GPT-4-Turbo outputs. High similarity therefore indicates imitation of the teacher model, not independent clue quality. In addition, Table 2 reports a mean ROUGE-L of 0.0278 between source text and GPT-4-Turbo clues and calls this a 'substantial connection,' but such a tiny score is effectively negligible; the paper itself admits that ROUGE is not a reliable measure for this task. Please remove or heavily qualify these ROUGE-based quality claims, and evaluate with independent references, such as human-authored clues or task-specific semantic metrics.
- [Section 6, Limitations] The limitations section mentions dialect coverage and category flexibility, but it does not acknowledge the two threats to validity that matter most for the paper's claims: the single-annotator design of the human evaluation and the absence of any actual crossword-grid-construction component. The limitations discussion and the stated scope of the contribution should be revised accordingly.
minor comments (4)
- [Section 3.1] There is an internal inconsistency in the filtering description: the text says keywords outside the 3-to-20-character limit are removed, but later the same section states that keyword length is restricted to between 2 and 20 characters. Please unify these numbers.
- [Section 4.3] The sentence beginning 'Llama3-8B-Instruct emerges as the top performer...' is grammatically tangled and unclear; please rewrite it to state the comparison cleanly.
- [References] Entries [Zeinalipour et al., 2024a] and [Zeinalipour et al., 2024b] appear to refer to the same paper with the same title and arXiv identifier; one should be removed or differentiated.
- [Appendix A] In the example tables, the annotation text mixes Arabic and English in the 'clue' and 'rating' columns; for reproducibility, the paper should state clearly which language the annotator used when assigning ratings, and whether the Arabic or English translations were presented.
Circularity Check
The ROUGE-based evaluation in Table 3 is circular because fine-tuning targets and the comparison reference are both GPT-4-Turbo clues; the human-evaluation headline is independent but rests on a single annotator.
-
fitted input called prediction
[Section 4.2, Table 3; training data created in Section 3.1]
"We assessed the similarity between various sets of clues generated by different models, as presented in Table 3, and those generated by the GPT-4-Turbo model on a test set containing 200 educational contexts. This assessment was performed using ROUGE scores. The results reveal that the fine-tuned Llama3-8b-Instruct and GPT-3.5-Turbo model achieves a higher similarity toGPT-4-Turbo."
The Arabic-Clue-Instruct training set was generated by GPT-4-Turbo, as stated in Section 3.1: 'we utilized the capabilities of GPT-4-Turbo, generating a diverse set of at least three clues per Wikipedia entry.' Fine-tuning optimizes the student models to imitate those GPT-4 outputs. Therefore, measuring ROUGE of the fine-tuned models against GPT-4-Turbo clues is a direct check of how well the students fit the teacher's output distribution. The finding that fine-tuning increases similarity to GPT-4 is entailed by the training objective and cannot independently validate clue quality. The human evaluation in Section 4.3 is independent, but the Table 3 ROUGE claim of 'effectiveness of fine-tuning' reduces by construction to imitation of the training-data generator.
full rationale
The paper's headline claim, that fine-tuned Llama3-8B reaches 78.86% 'A' ratings, rests on human evaluation in Section 4.3, which is logically independent of the GPT-4 training labels and is not circular. However, the automated evaluation in Section 4.2 is partially circular: the models are fine-tuned on GPT-4-Turbo-generated clues and are then scored by ROUGE similarity to GPT-4-Turbo clues. High similarity to the teacher is the expected outcome of the training objective, so this specific evidence does not independently support clue quality. The paper itself warns that ROUGE 'is not a reliable metric,' which mitigates but does not eliminate the circularity. No load-bearing self-citation or imported uniqueness theorem was found; the many Zeinalipour et al. references are context, not the argument. The single-annotator human evaluation is a validity and reproducibility limitation (no inter-annotator agreement, no blinding), but it is not a circularity step under the definitions used here. Overall, one evaluation claim reduces by construction, while the central human-evaluated result retains independent content, giving partial circularity.
Assumptions & free parameters
free parameters (3)
- Minimum article length =
50 words
- Keyword length bounds =
3 to 20 characters
- Number of clues per article =
at least 3
assumptions (4)
- domain assumption GPT-4-Turbo generates high-quality Arabic clues suitable as ground truth
- domain assumption Arabic Wikipedia introductory sections are suitable educational contexts
- domain assumption A single native Arabic speaker's ratings are reliable ground truth
- domain assumption ROUGE-L is a meaningful similarity measure for clue evaluation
Cite this review
Pith. "Pith review of From Arabic Text to Puzzles: LLM-Driven Development of Arabic Educational Crosswords." pith.science (2026). https://pith.science/paper/J2CSPO2N
@misc{pith2026250111035,
author = {Pith},
title = {Pith review of: From Arabic Text to Puzzles: LLM-Driven Development of Arabic Educational Crosswords},
year = {2026},
howpublished = {\url{https://pith.science/paper/J2CSPO2N}},
note = {Machine review of arXiv:2501.11035}
}
read the original abstract
We present an Arabic crossword puzzle generator from a given text that utilizes advanced language models such as GPT-4-Turbo, GPT-3.5-Turbo and Llama3-8B-Instruct, specifically developed for educational purposes, this innovative generator leverages a meticulously compiled dataset named Arabic-Clue-Instruct with over 50,000 entries encompassing text, answers, clues, and categories. This dataset is intricately designed to aid in the generation of pertinent clues linked to specific texts and keywords within defined categories. This project addresses the scarcity of advanced educational tools tailored for the Arabic language, promoting enhanced language learning and cognitive development. By providing a culturally and linguistically relevant tool, our objective is to make learning more engaging and effective through gamification and interactivity. Integrating state-of-the-art artificial intelligence with contemporary learning methodologies, this tool can generate crossword puzzles from any given educational text, thereby facilitating an interactive and enjoyable learning experience. This tool not only advances educational paradigms but also sets a new standard in interactive and cognitive learning technologies. The model and dataset are publicly available.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
Bhavna Arora and NS Kumar. 2019. Automatic keyword extraction and crossword generation tool for indian languages: Seekh. In 2019 IEEE Tenth International Conference on Technology for Education (T4E), pages 272--273. IEEE
work page 2019
-
[2]
Yolanda Dita Bella and Endang Mastuti Rahayu. 2023. The improving of the student’s vocabulary achievement through crossword game in the new normal era. Edunesia: Jurnal Ilmiah Pendidikan, 4(2):830--842
work page 2023
-
[3]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901
2020
-
[4]
Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691
arXiv 2023
-
[5]
Sunita M Dol. 2017. Gpbl: An effective way to improve critical thinking and problem solving skills in engineering education. J Engin Educ Trans, 30(3):103--13
work page 2017
-
[6]
Dzulfikri Dzulfikri. 2016. Application-based crossword puzzles: Players’ perception and vocabulary retention. Studies in English Language and Education, 3(2):122--133
work page 2016
-
[7]
Jennifer Esteche, Romina Romero, Luis Chiruzzo, and Aiala Ros \'a . 2017. Automatic definition extraction and crossword generation from spanish news text. CLEI Electronic Journal, 20(2)
work page 2017
-
[8]
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685
arXiv 2021
Show all 31 references
-
[9]
Serap Kaynak, Sibel Erg \"u n, and Ay s e Karada s . 2023. The effect of crossword puzzle activity used in distance education on nursing students’ problem-solving and clinical decision-making skills: A comparative study. Nurse Education in Practice, 69:103618
2023
-
[10]
Shane T Mueller and Elizabeth S Veinott. 2018. Testing the effectiveness of crossword games on immediate and delayed memory for scientific vocabulary and concepts. In CogSci
2018
-
[11]
RS Nickerson. 1977. Crossword puzzles and lexical memory. In Attention and performance VI, pages 699--718. Routledge
1977
-
[12]
Wiwat Orawiwatnakul. 2013. Crossword puzzles as a learning tool for vocabulary development. Electronic Journal of Research in Education Psychology, 11(30):413--428
2013
-
[13]
Bali Ranaivo-Malan c on, Terrin Lim, Jacey-Lynn Minoi, and Amelia Jati Robert Jupit. 2013. Automatic generation of fill-in clues and answers from raw texts for crosswords. In 2013 8th International Conference on Information Technology in Asia (CITA), pages 1--5. IEEE
2013
-
[14]
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, p...
2020
-
[15]
Leonardo Rigutini, Michelangelo Diligenti, Marco Maggini, and Marco Gori. 2008. A fully automatic crossword generator. In 2008 Seventh International Conference on Machine Learning and Applications, pages 362--367. IEEE
2008
-
[16]
Leonardo Rigutini, Michelangelo Diligenti, Marco Maggini, and Marco Gori. 2012. Automatic generation of crossword puzzles. International Journal on Artificial Intelligence Tools, 21(03):1250014
2012
-
[17]
Mircea cel Batran
Corina Sandiuc and Alina Balagiu. 2020. The use of crossword puzzles as a strategy to teach maritime english vocabulary. Scientific Bulletin" Mircea cel Batran" Naval Academy, 23(1):236A--242
2020
-
[18]
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
2023 arXiv
-
[19]
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560
2022 arXiv
-
[20]
Elizabeth Yuriev, Ben Capuano, and Jennifer L Short. 2016. Crossword puzzles for chemistry education: learning goals beyond vocabulary. Chemistry education research and practice, 17(3):532--554
2016
-
[21]
Peyman Zamani, Somayeh Biparva Haghighi, and Majid Ravanbakhsh. 2021. The use of crossword puzzles as an educational tool. Journal of Advances in Medical Education & Professionalism, 9(2):102
2021
-
[23]
Kamyar Zeinalipour, Achille Fusco, Asya Zanollo, Marco Maggini, and Marco Gori. 2024 b . Harnessing llms for educational content-driven italian crossword generation. arXiv preprint arXiv:2411.16936
2024 arXiv
-
[24]
Kamyar Zeinalipour, Tommaso Iaquinta, Giovanni Angelini, Leonardo Rigutini, Marco Maggini, and Marco Gori. 2023 a . Building bridges of knowledge: Innovating education with automated crossword generation. In 2023 International Conference on Machine Learning and Applications (I...
2023
-
[25]
Kamyar Zeinalipour, Yusuf G \"o kberk Kepti g , Marco Maggini, Leonardo Rigutini, and Marco Gori. 2024 c . A turkish educational crossword puzzle generator. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innova...
2024
-
[26]
Kamyar Zeinalipour, Yusuf Gökberk Keptiğ, Marco Maggini, and Marco Gori. 2024 d . Automating turkish educational quiz generation using large language models. https://arxiv.org/abs/2406.03397
2024 arXiv
-
[27]
Kamyar Zeinalipour, Mohamed Saad, Marco Maggini, and Marco Gori. 2023 b . Arabicros: Ai-powered arabic crossword puzzle generation for educational applications. In Proceedings of ArabicNLP 2023, pages 288--301
2023
-
[28]
Kamyar Zeinalipour, Asya Zanollo, Giovanni Angelini, Leonardo Rigutini, Marco Maggini, Marco Gori, et al. 2023 c . Italian crossword generator: Enhancing education through interactive word puzzles. arXiv preprint arXiv:2311.15723
2023 arXiv
-
[29]
Victor Samuel Zirawaga, Adeleye Idowu Olusanya, and Tinovimbanashe Maduku. 2017. Gaming in education: Using games as a support tool to teach history. Journal of Education and Practice, 8(15):55--64
2017
-
[30]
Andrea Zugarini, Kamyar Zeinalipour, Surya Sai Kadali, Marco Maggini, Marco Gori, and Leonardo Rigutini. 2024. Clue-instruct: Text-based clue generation for educational crossword puzzles. arXiv preprint arXiv:2404.06186
2024 arXiv
-
[31]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[32]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.