REVIEW 4 major objections 6 minor 1 cited by
Harnessing LLMs for Educational Content-Driven Italian Crossword Generation
T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Fine-tuning LLMs on a new Italian clue dataset sharply improves crossword clue generation.
desk verdict Useful Italian clue dataset and fine-tuned models, but the headline 'significant improvements' rests on a circular automatic metric and an underspecified human evaluation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is the Italian-Clue-Instruct dataset together with the prompt templates used to create it. Each dataset entry pairs an Italian Wikipedia context with a keyword, the future crossword answer, and clues written by GPT-4o. The prompts force a sequence of rewriting operations: resolve all pronouns, split the text into independent sentences, choose three sentences that best characterize the keyword, and rephrase those sentences as clues that never contain the keyword or any part of it. Four prompt templates enforce four structures: no format constraint, a bare noun phrase with no determiner, a definite determiner phrase headed by a definite article, and a copular sentence with elliptical subject (for example, 'è una salsa piccante tipica della Tunisia' for the answer 'Harissa'). The same data then fine-tunes the two small models via LoRA, with the checkpoint of minimum loss among the three training epochs selected for evaluation.
What would settle it
Have multiple independent raters blindly compare base and fine-tuned clues on a few hundred contexts under the paper's five-level scale and compute inter-annotator agreement; if the fine-tuned advantage shrinks or disappears, the improvement claim is not supported. Alternatively, run a controlled experiment where human solvers must identify the answer from each clue; if fine-tuned clues do not lead to higher solving accuracy than base-model clues, the educational benefit is not confirmed.
Extended reading notes
Core claim
The central discovery is that a purpose-built instruction dataset can transfer GPT-4o's clue-writing ability to much smaller, open-weights models. The authors built Italian-Clue-Instruct by taking the opening sections of Italian Wikipedia articles, filtering them, extracting keywords, and prompting GPT-4o with four distinct prompt templates that force the generated clue into a specified syntactic structure. They then fine-tuned Mistral-7B and Llama3-8b with LoRA on about 15,000 of these clues. After fine-tuning, Mistral's ROUGE-1 score against GPT-4o references rose from 0.342 to 0.611 and Llama's from 0.258 to 0.552, with similar jumps in ROUGE-2 and ROUGE-L. A human evaluation of 100 contexts with three clues per context found that the fine-tuned models received mostly 'A' ratings—coherent, valid clues matching context, answer, and structure—while the base models received many lower ratings. The paper presents this as evidence that the dataset is an effective teaching signal for educational Italian crossword generation.
Load-bearing premise
The evaluation rests on the assumption that GPT-4o's clue style is the right gold standard: the models are trained on GPT-4o clues and then scored by how closely they match GPT-4o clues, so the reported improvements are improvements in imitation, not necessarily in educational usefulness.
Editorial extensions
If this is right
- Teachers and students can generate customized Italian crossword puzzles directly from course texts using models that run locally without API costs.
- The four clue structures allow puzzles to vary in syntactic difficulty, connecting the generator to psycholinguistic findings about which structures are harder to process.
- The public release of the dataset gives Italian computational linguistics a benchmark for educational clue generation, a resource that did not exist before.
- The demonstrated recipe—distill a strong model's clues into a small instruction set and fine-tune open models on it—generalizes to other languages and puzzle types.
- The fine-tuned models narrow the quality gap with proprietary GPT-4o on this specific task, making high-quality Italian clue generation more accessible.
Reading between the lines
- The gains are measured as similarity to GPT-4o, so they show successful imitation rather than proven educational benefit; a blind comparison against human-written clues or a test of solver accuracy would evaluate the pedagogical claim directly.
- The syntactic control suggests a testable extension: measure human solvers' accuracy and response time on the four clue types to see whether the predicted processing differences actually appear in crossword solving.
- The human evaluation used a single rater on 100 contexts; replicating it with several raters and reporting inter-annotator agreement would establish whether the rating scale is reproducible.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Italian-Clue-Instruct, a dataset of GPT-4o-generated Italian crossword clues derived from Wikipedia articles, organized into four clue types (unrestricted, bare noun phrases, definite determiner phrases, and copular sentences). The authors fine-tune Mistral-7B-Instruct-v0.3 and Llama3-8b-Instruct with LoRA on this dataset and evaluate the resulting models using ROUGE scores against GPT-4o references and a human rating study. The central claim is that fine-tuning yields 'significant improvements' in the models' ability to generate educational crossword clues from Italian texts. The paper also describes the dataset curation pipeline, prompt templates, and an example generated crossword.
Significance. If the central claim is substantiated, the Italian-Clue-Instruct dataset and the fine-tuned models would be a useful resource for Italian educational crossword generation, an area with few dedicated tools. The paper explicitly releases the dataset and models, which is a concrete contribution, and the linguistic grounding of the four syntactic clue types is a strength. However, the headline improvement claim currently rests on a circular automatic evaluation (ROUGE against GPT-4o, which also generated the training data) and an underspecified human evaluation, so the evidential value of the paper's experiments is limited until those issues are addressed.
major comments (4)
- [Section 4, Table 1] The automatic evaluation is circular with respect to the central claim. The fine-tuned models were trained on GPT-4o-generated clues (Section 3, 'Generation of Educational Italian Clues') and then evaluated with ROUGE scores computed against GPT-4o-generated references on a test set of 200 contexts. Matching the teacher distribution is exactly what the fine-tuning procedure optimizes, so the reported ROUGE gains (e.g., Mistral-7B ROUGE-1 from 0.342 to 0.611) largely confirm imitation of GPT-4o rather than an improvement in clue quality. The paper should either explicitly reframe these scores as a fidelity-to-teacher measure or provide a non-circular evaluation (for example, human judgments or a metric with human-written references) before claiming 'significant improvements'.
- [Section 4, 'Evaluation Results with the human evaluator' and Figure 5] The human evaluation is too underspecified to support the conclusion of significant improvements. It covers 100 contexts with 3 clues each across four models, but the text does not unambiguously state the number of raters ('a native Italian speaker, master student of linguistics, and PhD student in linguistics' can be read as one person or three), does not say whether raters were blind to model identity, reports no inter-annotator agreement, and provides no statistical test comparing the base and fine-tuned rating distributions. Given the small counts in categories A and B, the visible differences in Figure 5 could easily arise from sampling noise. The paper should report the full rating setup, agreement measures, and significance tests, and should avoid the phrase 'significant improvements' without such support.
- [Section 3, 'Evaluating quality of the Italian-Clue-Instruct Dataset'] The dataset-quality evaluation is not robust enough to support the claim that the majority of GPT-4o-generated clues are high quality. The ROUGE-based assessment (average ROUGE-1, ROUGE-2, ROUGE-L of 0.159, 0.114, and 0.146) is acknowledged by the authors to be unreliable for semantic quality, and the human evaluation is described only as a randomly chosen subset of 100 articles with no inter-annotator agreement or detailed rater instructions. Figure 4 should be accompanied by the rating protocol, the number of raters and their agreement, and confidence intervals or a test of the distribution before the dataset's quality is asserted.
- [Section 3, 'Italian-Clue-Instruct Data Collection Methodology'] The methodology states that GPT-4o generation was performed 'with human validation for accuracy,' but the only human validation described later is the evaluation of 100 sampled articles in Section 3. If this 100-article subset is the intended validation, the paper should say so explicitly; if there was additional manual validation of the full dataset, its scale and procedure should be reported. As written, the claim of human validation is unsupported.
minor comments (6)
- [Abstract and Section 3] The abstract says the dataset comprises 'over 30,000 entries' while Section 3 reports '15,000 unique clues'; these numbers should be reconciled.
- [Figure 1 caption] The caption says 'using GPT-4 Turbo' while the text and prompts consistently refer to GPT-4o; the inconsistency should be corrected.
- [Section 3, 'Formulation of Various Prompts'] There is a typo: 'prompt engeneering' should be 'prompt engineering'.
- [Section 1, last paragraph] The sentence 'a new dataset called 1 has been compiled' contains a placeholder footnote reference where the dataset name should appear.
- [Section 3, 'Formulation of Various Prompts'] The phrase 'Apart we aimed to elicit three specific types of clue' is missing a word and should read 'Apart from this, we aimed to elicit three specific types of clues'.
- [Figure 2 caption] The phrase 'Token Distribution of unique Content' and the duplicated 'Number of tokens' axis label should be cleaned up for readability.
Circularity Check
The main quantitative evaluation reduces to self-imitation: GPT-4o wrote both the fine-tuning targets and the ROUGE reference clues, so the reported gains largely measure fidelity to the teacher distribution.
-
fitted input called prediction
[Section 3 ('Generation of Educational Italian Clues' and 'Overview of the Italian-Clue-Instruct Dataset'); Section 4 ('Evaluation Results with the Automatic Metrics'); Table 1]
"Guided by the self-instruct framework [27], we devised a method to automate the generation of educational crossword clues in Italian, harnessing the power of LLMs. Central to our approach is the sophisticated GPT-4o... we leveraged the capabilities of GPT-4o, generating a minimum of three diverse clues per Wikipedia article... We evaluated the resemblance between various sets of clues produced by different models (details shown in Table 1) and those generated by the GPT-4o model on a test set of 200 educational contexts. This evaluation was done using ROUGE scores."
The automatic evaluation is circular by construction: the GPT-4o model is both the source of the fine-tuning targets (its clues form the Italian-Clue-Instruct training set used to fine-tune Mistral and Llama) and the reference against which the fine-tuned models are scored (ROUGE against GPT-4o-generated clues on the test contexts). Fine-tuning directly optimizes the student models to reproduce GPT-4o's clue distribution, so a higher ROUGE score against GPT-4o references primarily confirms successful imitation of the teacher, not an independent improvement in educational clue quality.
full rationale
Most of the pipeline is not circular: the Italian Wikipedia data collection is external, the four syntactic clue styles are analyzed independently, and the human rating protocol is a genuine attempt at independent quality assessment. The single clear circular step is the automatic evaluation: since GPT-4o generated both the fine-tuning clues (Section 3) and the ROUGE reference clues (Section 4, Table 1), the reported ROUGE improvements (Mistral ROUGE-L 0.261 -> 0.556; Llama ROUGE-L 0.198 -> 0.501) mostly measure how well the fine-tuned models learned to imitate their teacher distribution. This is a partial circularity, not a total one, because the paper also presents a human evaluation of 100 contexts (Figure 5) that does not use GPT-4o as its gold standard. That human evaluation, however, is too underspecified to fully break the circularity: the paper does not clearly state the number of raters, whether they were blind to model identity, any inter-annotator agreement, or a significance test, so the independent evidence is weaker than the conclusion requires. The self-citations to the authors' prior crossword-generation work are descriptive background and are not load-bearing here. Taking the automatic metric and the underspecified human evaluation together, the central claim is partly forced by the training/evaluation design, warranting a 6 rather than a higher score.
Assumptions & free parameters
free parameters (4)
- LoRA hyperparameters (rank r=16, alpha=32) =
r=16, alpha=32
- Article filtering thresholds =
min 50 words; keyword length 3-20 chars; no numerals/special chars
- Inference sampling parameters =
temperature=0.1, top_p=0.95, top_k=50
- Checkpoint selection rule =
second of three epochs (minimum loss)
assumptions (4)
- domain assumption Italian Wikipedia introductory sections provide suitable educational content for clue generation.
- domain assumption GPT-4o-generated clues are high-quality enough to serve as training targets and as evaluation references.
- ad hoc to paper ROUGE overlap with the input text is a meaningful measure of clue adequacy.
- domain assumption The syntactic clue types (determiner phrase, bare noun phrase, copular sentence) can be reliably elicited and recognized by the models and raters.
Cite this review
Pith. "Pith review of Harnessing LLMs for Educational Content-Driven Italian Crossword Generation." pith.science (2026). https://pith.science/paper/IK4IJNFU
@misc{pith2026241116936,
author = {Pith},
title = {Pith review of: Harnessing LLMs for Educational Content-Driven Italian Crossword Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/IK4IJNFU}},
note = {Machine review of arXiv:2411.16936}
}
read the original abstract
In this work, we unveil a novel tool for generating Italian crossword puzzles from text, utilizing advanced language models such as GPT-4o, Mistral-7B-Instruct-v0.3, and Llama3-8b-Instruct. Crafted specifically for educational applications, this cutting-edge generator makes use of the comprehensive Italian-Clue-Instruct dataset, which comprises over 30,000 entries including diverse text, solutions, and types of clues. This carefully assembled dataset is designed to facilitate the creation of contextually relevant clues in various styles associated with specific texts and keywords. The study delves into four distinctive styles of crossword clues: those without format constraints, those formed as definite determiner phrases, copular sentences, and bare noun phrases. Each style introduces unique linguistic structures to diversify clue presentation. Given the lack of sophisticated educational tools tailored to the Italian language, this project seeks to enhance learning experiences and cognitive development through an engaging, interactive platform. By meshing state-of-the-art AI with contemporary educational strategies, our tool can dynamically generate crossword puzzles from Italian educational materials, thereby providing an enjoyable and interactive learning environment. This technological advancement not only redefines educational paradigms but also sets a new benchmark for interactive and cognitive language learning solutions.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 1 Pith paper
-
From Arabic Text to Puzzles: LLM-Driven Development of Arabic Educational Crosswords
Fine-tuned Llama3-8B scores 78.86% top human ratings for Arabic crossword clues, but evaluations rely on a single annotator and the tool does not output a crossword grid.
Reference graph
Works this paper leans on
-
[1]
W. Orawiwatnakul, Crossword puzzles as a learn- ing tool for vocabulary development, Electronic Journal of Research in Education Psychology 11 (2013) 413–428
work page 2013
-
[2]
Y. D. Bella, E. M. Rahayu, The improving of the student’s vocabulary achievement through cross- word game in the new normal era, Edunesia: Jurnal Ilmiah Pendidikan 4 (2023) 830–842
work page 2023
-
[3]
D. Dzulfikri, Application-based crossword puzzles: Players’ perception and vocabulary retention, Stud- ies in English Language and Education 3 (2016) 122– 133
work page 2016
-
[4]
R. Nickerson, Crossword puzzles and lexical mem- ory, in: Attention and performance VI, Routledge, 1977, pp. 699–718
work page 1977
- [5]
-
[6]
C. Sandiuc, A. Balagiu, The use of crossword puz- zles as a strategy to teach maritime english vocabu- lary, Scientific Bulletin" Mircea cel Batran" Naval Academy 23 (2020) 236A–242
work page 2020
- [7]
-
[8]
S. T. Mueller, E. S. Veinott, Testing the effective- ness of crossword games on immediate and delayed memory for scientific vocabulary and concepts., in: CogSci, 2018
work page 2018
Show all 46 references
-
[9]
V. S. Zirawaga, A. I. Olusanya, T. Maduku, Gaming in education: Using games as a support tool to teach history., Journal of Education and Practice 8 (2017) 55–64
2017
-
[10]
Zamani, S
P. Zamani, S. B. Haghighi, M. Ravanbakhsh, The use of crossword puzzles as an educational tool, Journal of Advances in Medical Education & Pro- fessionalism 9 (2021) 102
2021
-
[11]
S. M. Dol, Gpbl: An effective way to improve critical thinking and problem solving skills in engineering education, J Engin Educ Trans 30 (2017) 103–13
2017
-
[12]
Ranaivo-Malançon, T
B. Ranaivo-Malançon, T. Lim, J.-L. Minoi, A. J. R. Jupit, Automatic generation of fill-in clues and an- swers from raw texts for crosswords, in: 2013 8th International Conference on Information Technol- ogy in Asia (CITA), IEEE, 2013, pp. 1–5
2013
-
[13]
Rigutini, M
L. Rigutini, M. Diligenti, M. Maggini, M. Gori, A fully automatic crossword generator, in: 2008 Sev- enth International Conference on Machine Learn- ing and Applications, IEEE, 2008, pp. 362–367
2008
-
[14]
Rigutini, M
L. Rigutini, M. Diligenti, M. Maggini, M. Gori, Au- tomatic generation of crossword puzzles, Inter- national Journal on Artificial Intelligence Tools 21 (2012) 1250014
2012
-
[15]
Esteche, R
J. Esteche, R. Romero, L. Chiruzzo, A. Rosá, Au- tomatic definition extraction and crossword gen- eration from spanish news text, CLEI Electronic Journal 20 (2017)
2017
-
[16]
Arora, N
B. Arora, N. Kumar, Automatic keyword extraction and crossword generation tool for indian languages: Seekh, in: 2019 IEEE Tenth International Confer- ence on Technology for Education (T4E), IEEE, 2019, pp. 272–273
2019
-
[17]
Zeinalipour, T
K. Zeinalipour, T. Iaquinta, G. Angelini, L. Rigutini, M. Maggini, M. Gori, Building bridges of knowl- edge: Innovating education with automated cross- word generation, in: 2023 International Conference on Machine Learning and Applications (ICMLA), IEEE, 2023, pp. 1228–1236
2023
-
[18]
Zeinalipour, A
K. Zeinalipour, A. Zanollo, G. Angelini, L. Rigutini, M. Maggini, M. Gori, et al., Italian crossword generator: Enhancing education through interac- tive word puzzles, arXiv preprint arXiv:2311.15723 (2023)
2023 arXiv
-
[19]
Zeinalipour, M
K. Zeinalipour, M. Saad, M. Maggini, M. Gori, Ara- bicros: Ai-powered arabic crossword puzzle gener- ation for educational applications, in: Proceedings of ArabicNLP 2023, 2023, pp. 288–301
2023
-
[20]
Zeinalipour, Y
K. Zeinalipour, Y. G. Keptiğ, M. Maggini, L. Rigutini, M. Gori, A turkish educational crossword puzzle generator, in: International Conference on Artificial Intelligence in Education, Springer, 2024, pp. 226– 233
2024
-
[21]
Zugarini, K
A. Zugarini, K. Zeinalipour, S. S. Kadali, M. Mag- gini, M. Gori, L. Rigutini, Clue-instruct: Text-based clue generation for educational crossword puzzles, arXiv preprint arXiv:2404.06186 (2024)
2024 arXiv
-
[22]
Chierchia, Reference to kinds across language, Natural language semantics 6 (1998) 339–405
G. Chierchia, Reference to kinds across language, Natural language semantics 6 (1998) 339–405
1998
-
[23]
Link, The logical analysis of plurals and mass terms: A lattice theoretical approach, Meaning, Use, and Interpretation of Language/Walter de Gruyter (1983)
G. Link, The logical analysis of plurals and mass terms: A lattice theoretical approach, Meaning, Use, and Interpretation of Language/Walter de Gruyter (1983)
1983
-
[24]
Longobardi, Reference and proper names: A theory of n-movement in syntax and logical form, Linguistic inquiry (1994) 609–665
G. Longobardi, Reference and proper names: A theory of n-movement in syntax and logical form, Linguistic inquiry (1994) 609–665
1994
-
[25]
Zamparelli, Layers in the determiner phrase, Routledge, 2014
R. Zamparelli, Layers in the determiner phrase, Routledge, 2014
2014
-
[26]
Moro, Copular sentences, The Blackwell com- panion to syntax (2006) 1–23
A. Moro, Copular sentences, The Blackwell com- panion to syntax (2006) 1–23
2006
-
[27]
Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, H. Hajishirzi, Self-instruct: Aligning language model with self generated instructions, arXiv preprint arXiv:2212.10560 (2022)
2022 arXiv
-
[28]
Brown, B
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, et al., Language models are few-shot learners, Advances in neural information process- ing systems 33 (2020) 1877–1901
2020
-
[29]
Touvron, T
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and effi- cient foundation language models, arXiv preprint arXiv:2302.13971 (2023)
2023 arXiv
-
[30]
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adap- tation of large language models, arXiv preprint arXiv:2106.09685 (2021)
2021 arXiv
-
[31]
Rasley, S
J. Rasley, S. Rajbhandari, O. Ruwase, Y. He, Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters, in: Proceedings of the 26th ACM SIGKDD Interna- tional Conference on Knowledge Discovery & Data Mining, 2020, pp. 3505–3506
2020
-
[32]
Dao, Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.08691 (2023)
T. Dao, Flashattention-2: Faster attention with better parallelism and work partitioning, arXiv preprint arXiv:2307.08691 (2023). A. Appendix You are a crossword expert. Generate concise and clever clues in Italian for educational crossword puzzles based on a specified Keyword...
2023 arXiv
-
[36]
Make sure that the keyword remains absent from the clues
Generate short and clever crossword clues in Italian from the selected sentences. Make sure that the keyword remains absent from the clues. If the Keyword is not the subject of the sentence, make sure that it is substituted with an appropriate clitic, possessive or demonstrati...
-
[40]
Make sure the output is in the requested format and do not include the whole process in the output, but only the clues
Compile these clues into a list formatted as follows: [clue1, clue2, clue3] into a JSON file under the key: 'clues'. Make sure the output is in the requested format and do not include the whole process in the output, but only the clues. Figure 6: Illustration of the prompt use...
-
[44]
Make sure that the keyword remains absent from the clues
Generate short and clever crossword clues in Italian from the selected sentences. Make sure that the keyword remains absent from the clues. Each clue must have the syntax of a bare noun phrase (zero determiner): the root node of each clue must be a common or proper noun and it...
-
[48]
Make sure the output is in the requested format and do not include the whole process in the output, but only the clues
Compile these clues into a list formatted as follows: [clue1, clue2, clue3] into a JSON file under the key: 'clues'. Make sure the output is in the requested format and do not include the whole process in the output, but only the clues. Figure 7: Illustration of the prompt use...
-
[52]
Make sure that the keyword remains absent from the clues
Generate short and clever crossword clues in Italian from the selected sentences. Make sure that the keyword remains absent from the clues. Each clue must have the syntax of a determiner phrase with the definite article (followed by a noun and possibly adjectives). It can be f...
-
[56]
Make sure the output is in the requested format and do not include the whole process in the output, but only the clues
Compile these clues into a list formatted as follows: [clue1, clue2, clue3] into a JSON file under the key: 'clues'. Make sure the output is in the requested format and do not include the whole process in the output, but only the clues. Figure 8: Illustration of the prompt use...
-
[57]
Substitute every pronoun in the text with full phrases expressing their referents
-
[58]
Split the text into small independent sentences that could be understood out of context
-
[59]
Try to select sentences from different parts of the Text
Pinpoint three concise sentences that contain the Keyword and best characterize the keyword. Try to select sentences from different parts of the Text
-
[60]
è <clue>
Generate short and clever crossword clues in Italian from the selected sentences. Make sure that the keyword remains absent from the clues. Each clue must be a copular sentence, in which the keyword constitutes the subject. The syntax of each clue then must corresponds to a co...
-
[61]
Ensure that each clue functions as a description or definition of the keyword rather than a query, focusing on details about the keyword
-
[62]
Make sure that the clues are relevant and that they are sufficient to identify the keyword
Make sure that each clue's information can be traced back to the text. Make sure that the clues are relevant and that they are sufficient to identify the keyword. Make sure that the keyword does not appear in the clues. Make sure that any part of the keyword is not present in ...
-
[63]
Select only the three best clues for educational purposes
-
[64]
Make sure the output is in the requested format and do not include the whole process in the output, but only the clues
Compile these clues into a list formatted as follows: [clue1, clue2, clue3] into a JSON file under the key: 'clues'. Make sure the output is in the requested format and do not include the whole process in the output, but only the clues. Figure 9: Illustration of the copular se...
2017
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.