{"id":"a349592f-2514-4d92-8db8-ddb55e555d10","arxiv_id":"2501.11035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned Llama3-8B scores 78.86% top human ratings for Arabic crossword clues, but evaluations rely on a single annotator and the tool does not output a crossword grid.","lead":"An Arabic crossword clue generator is trained by fine-tuning GPT-3.5 and Llama-3 on a newly created dataset of over 54,000 GPT-4-generated clues from Arabic Wikipedia articles. The authors report that fine-tuning sharply raises human-rated clue quality, but the system generates clues rather than the crossword grid itself.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-annotator human evaluation is the sole support for the headline 78.86% 'A' rating; without multi-annotator validation the central claim is unsubstantiated.","rationale":"The reader correctly identifies the dataset's GPT-4 ground-truth quality and the single-speaker validation as a weak point, but the more load-bearing concern is that the primary evidence for the central claim—the fine-tuned models' high 'A' ratings—comes from that same single-annotator evaluation. If the evaluation is unreliable, the 78.86% figure is not supported regardless of GPT-4 clue quality. This is an internal validity issue rather than a disagreement with external consensus, and it is directly testable by a multi-annotator study. The paper does provide public models and a dataset, which is real evidence, but the specific quantitative claim of superiority after fine-tuning is not independently verified. I therefore keep the CONDITIONAL verdict, with the condition being independent validation of the human evaluation. No change is needed to the reader's verdict category, but the justification is sharpened.","tokens_in":14621,"tokens_out":2558,"duration_ms":27048,"concrete_test":"Recruit at least two independent native Arabic speakers with no affiliation to the project. Present them with the same 200 evaluation contexts used in Section 4.3, with model outputs anonymized and shuffled across all models (base and fine-tuned), and ask them to apply the same A–E rubric. Compute inter-annotator agreement (e.g., Cohen's kappa or Krippendorff's alpha). Then compare the average independent 'A' percentage for Llama3-8B FT against the reported 78.86%. If the independent raters' average is substantially lower (e.g., below 60%) or agreement is poor (kappa < 0.4), the headline claim fails. This test directly assesses whether the single-annotator evaluation is reproducible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's strongest claim—that fine-tuned Llama3-8B produces high-quality Arabic clues (78.86% 'A' in Section 4.3, Table 4)—rests entirely on human evaluation performed by a single native Arabic speaker (Sections 3.1 and 4.3). There is no inter-annotator agreement metric, no blinding to model identity, and no evidence that the annotator's ratings are consistent or unbiased. The same annotator also rated GPT-4 clues in Section 3.1, and the five-level rubric, while explicit, leaves room for subjective judgment (e.g., 'slight discrepancies', 'vague connection'). A single annotator can systematically favor outputs that mimic the training distribution or apply different standards across sessions. If the annotator is an author or otherwise invested in the result, the 78.86% figure is particularly suspect. The fine-tuned model's improvement over its base and over GPT-4 (reported separately) is the quantitative core of the contribution; if independent raters do not reproduce a similar 'A' percentage, the central claim loses its empirical foundation. This concern is distinct from the reader's point about GPT-4 ground-truth quality: even if the training clues are excellent, the measured effectiveness of fine-tuning is only as trustworthy as the measurement instrument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Arabic-Clue-Instruct, a dataset of 14,497 Arabic Wikipedia article introductions (54,196 entries) with GPT-4-Turbo-generated crossword clues, and fine-tunes GPT-3.5-Turbo and Llama3-8B-Instruct on this dataset to generate Arabic educational crossword clues from input texts. The authors report automatic ROUGE evaluations (Tables 2 and 3) and a human evaluation (Table 4) in which the fine-tuned Llama3-8B achieves 78.86% of clues rated 'A'. The stated broader contribution is an 'Arabic crossword puzzle generator from a given text,' with a sample grid shown in Figure 7. The authors make the dataset and models publicly available.","tokens_in":14857,"tokens_out":3784,"duration_ms":40587,"significance":"If the clue-generation quality is established, the contribution is a useful open resource for Arabic educational technology: a reasonably large clue dataset, fine-tuned open-weight models, and reproducible code. The paper addresses a genuine gap in Arabic crossword clue generation and provides concrete artifact releases (GitHub, HuggingFace). However, the current evidence is substantially narrower than the claims. The paper actually demonstrates clue generation, not crossword-grid construction, and its central quality claim rests on a single-annotator evaluation and on a circular automatic metric. With careful revision, the resource and the clue-generation pipeline could become a solid contribution.","major_comments":[{"comment":"The abstract and title describe an 'Arabic crossword puzzle generator from a given text,' but the methodology and experiments cover only clue generation. No algorithm is presented for constructing a crossword grid, placing answers, enforcing intersections, or selecting clues to fit a grid; the example puzzle in Figure 7 appears to be manually assembled. This mismatch undermines the central claim. Either provide an actual grid-construction component and evaluate it, or explicitly reframe the contribution as clue generation for Arabic educational crosswords.","section":"Section 1, Section 3.2, Section 4.3, Figure 7"},{"comment":"The headline result that fine-tuned Llama3-8B achieves 78.86% 'A' ratings is based on a single native Arabic-speaking evaluator, with no inter-annotator agreement metric, no blinding, and no evidence that the rubric was applied consistently across sessions or across models. The fine-tuning improvement is the paper's core quantitative claim, so this evaluation needs to be reproduced with at least two independent annotators, with agreement reported (e.g., Cohen's kappa) and with explicit handling of potential evaluator bias.","section":"Section 4.3, Table 4"},{"comment":"The automatic evaluation is circular: the fine-tuning data were generated by GPT-4-Turbo, and Table 3 measures ROUGE similarity between the fine-tuned models' outputs and the same GPT-4-Turbo outputs. High similarity therefore indicates imitation of the teacher model, not independent clue quality. In addition, Table 2 reports a mean ROUGE-L of 0.0278 between source text and GPT-4-Turbo clues and calls this a 'substantial connection,' but such a tiny score is effectively negligible; the paper itself admits that ROUGE is not a reliable measure for this task. Please remove or heavily qualify these ROUGE-based quality claims, and evaluate with independent references, such as human-authored clues or task-specific semantic metrics.","section":"Section 3.1, Section 4.2, Tables 2 and 3"},{"comment":"The limitations section mentions dialect coverage and category flexibility, but it does not acknowledge the two threats to validity that matter most for the paper's claims: the single-annotator design of the human evaluation and the absence of any actual crossword-grid-construction component. The limitations discussion and the stated scope of the contribution should be revised accordingly.","section":"Section 6, Limitations"}],"minor_comments":[{"comment":"There is an internal inconsistency in the filtering description: the text says keywords outside the 3-to-20-character limit are removed, but later the same section states that keyword length is restricted to between 2 and 20 characters. Please unify these numbers.","section":"Section 3.1"},{"comment":"The sentence beginning 'Llama3-8B-Instruct emerges as the top performer...' is grammatically tangled and unclear; please rewrite it to state the comparison cleanly.","section":"Section 4.3"},{"comment":"Entries [Zeinalipour et al., 2024a] and [Zeinalipour et al., 2024b] appear to refer to the same paper with the same title and arXiv identifier; one should be removed or differentiated.","section":"References"},{"comment":"In the example tables, the annotation text mixes Arabic and English in the 'clue' and 'rating' columns; for reproducibility, the paper should state clearly which language the annotator used when assigning ratings, and whether the Arabic or English translations were presented.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper is probably best treated as a demonstration-and-resource paper rather than as a breakthrough in cross-lingual NLP. The main resource contributions (dataset and fine-tuned models) are genuine and the appendix examples show plausible clue quality, but the evaluation methodology and the exaggerated 'puzzle generator' framing would need to be fixed before the paper can be accepted. The issues are fixable within the manuscript's scope: reduce the claims to clue generation, add multi-annotator human evaluation, and drop or reinterpret the circular ROUGE evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the real contribution is Arabic-Clue-Instruct and the two fine-tuned models, both released. The paper is not, as the title promises, a crossword puzzle generator; it generates clues. And the headline 78.86% 'A' rating rests entirely on one native Arabic speaker's judgment, with no inter-annotator agreement or blinding. I'd trust the dataset more than the evaluation.\n\nWhat's new: the dataset (54k clue-keyword-text triples across 20 categories) is the first Arabic resource of its kind, and fine-tuning Llama3-8B and GPT-3.5 on it is a sensible, clearly described pipeline. The authors are transparent that ROUGE is not a semantic metric, and they ship code, data, and model weights. That's real substance.\n\nWhere it's soft: the gap between the abstract and the system. Figure 7 is a crossword grid, but there's no grid-generation algorithm; clues are produced and the grid appears hand-assembled. That's a scoping problem, not a fatal one.\n\nThe circularity in the automatic eval is real: fine-tuning targets are GPT-4-Turbo outputs, and Table 3 measures similarity to those same outputs. High ROUGE after fine-tuning mostly demonstrates imitation. The paper acknowledges ROUGE's limits, but the framing still implies quality.\n\nThe human eval is the bigger issue. One annotator, no agreement metric, no blinding to model identity. The appendix actually shows why this matters: for the Despicable Me example, 'A trilogy of cartoons' is rated A, but the text says four films. That suggests a lenient or inconsistent annotator. If that rating style generalizes, the 78.86% figure is not reliable.\n\nThe dataset itself is only spot-checked on 200 samples by the same type of annotation, so its quality is plausible but not confirmed. The limitations section is honest about dialect and category coverage, which is nice.\n\nWho should read it: anyone building educational clue generation for low-resource languages, and Arabic NLP folks who can use the dataset. It deserves a serious referee, but with major revision: multi-annotator evaluation with agreement statistics, a clearer separation of clue generation from puzzle assembly, and a toned-down abstract. I'd send it back for another round if I were the editor.","headline":"A useful Arabic clue dataset and fine-tuned models that deserve peer review, but the 'puzzle generator' framing and the single-annotator human eval are both overclaims.","tokens_in":15427,"tokens_out":2812,"would_cite":true,"duration_ms":27761,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning an open 8-billion-parameter model on a new Arabic clue dataset achieves a 78.86% top human rating for crossword clue quality.","keywords":["Arabic crossword generation","educational crosswords","large language models","Arabic-Clue-Instruct dataset","fine-tuning","Arabic NLP","clue generation","gamified learning"],"falsifier":"Have a panel of independent native Arabic speakers rate the same sample of clues from the fine-tuned Llama3-8B model using the paper's A-E rubric; if the average 'A' share falls well below 78.86% or inter-annotator agreement is low, the headline quality claim does not replicate. A cheaper check is to measure how often the fine-tuned model reproduces GPT-4-Turbo clues nearly verbatim, since the training data was generated by that model.","tokens_in":14411,"feed_emoji":"🧩","tokens_out":8745,"duration_ms":84717,"temperature":0.7,"pith_summary":"This paper tries to establish that a relatively small, open Arabic language model can turn educational texts into crossword clues good enough for classroom use. The authors built Arabic-Clue-Instruct, a dataset of over 50,000 entries pairing filtered Arabic Wikipedia passages with keywords, categories, and GPT-4-Turbo-generated clues, and then fine-tuned Llama3-8B-Instruct and GPT-3.5-Turbo on it. In human evaluation, fine-tuned Llama3-8B received the top 'A' rating on 78.86% of generated clues, up from 36.02% before fine-tuning, and fine-tuned GPT-3.5 more than doubled its A-rate from 28.47% to 54.33%. The authors argue this gives educators and students a reusable, open pipeline for generating Arabic crossword puzzles from their own texts, along with a dataset for further Arabic clue-generation work.","feed_headline":"Fine-tuned Llama3 earns 79% top rating on Arabic crossword clues","feed_subtitle":"A 54,000-clue dataset lets teachers turn Arabic Wikipedia texts into custom classroom crosswords.","key_machinery":"The load-bearing object is the Arabic-Clue-Instruct dataset: 14,497 text-keyword-category entries and 54,196 Arabic clues produced by GPT-4-Turbo through a SELF-INSTRUCT-style prompting approach, meaning a stronger model generates the training examples from carefully crafted prompts that ask for concise, wordplay-friendly clues without repeating the answer. The dataset supplies both the training signal for fine-tuning and the reference for ROUGE comparison. The fine-tuning recipes are the second mechanism: LoRA on Llama3-8B-Instruct with r=32, alpha=64, and total batch size 128, and fine-tuning of GPT-3.5-Turbo, both over three epochs. The evaluation machinery is a five-level human rating system (A through E) designed to catch contextual relevance, answer leakage, and hallucination; this rating system, not raw ROUGE, carries the paper's main quality claim.","core_discovery":"The central discovery is that fine-tuning on a purpose-built Arabic clue dataset makes general-purpose LLMs competent crossword-clue writers for Arabic educational content. Using 14,497 filtered Arabic Wikipedia articles across 20 categories, the authors had GPT-4-Turbo generate 54,196 clues, with 67.5% of a 200-sample human evaluation rated 'A'. Fine-tuning GPT-3.5-Turbo and Llama3-8B-Instruct on this data raised ROUGE similarity to GPT-4-generated clues and, more importantly, lifted Llama3-8B's human 'A' rating from 36.02% to 78.86%, while GPT-3.5 went from 28.47% to 54.33%. The paper claims this demonstrates both the effectiveness of fine-tuning and the quality of the released Arabic-Clue-Instruct dataset as an educational resource.","pith_inferences":["Editorial inference: the paper's single-native-speaker evaluation on 200 samples is the natural stress point; a multi-annotator replication with agreement statistics would tell whether the 78.86% 'A' rate is stable.","Editorial inference: because ROUGE-L scores are near zero and the fine-tuned models occasionally add outside knowledge, as the paper's own analysis notes, a faithfulness or answer-containment filter could reduce the remaining low-rated clues.","Editorial inference: the method's dependence on GPT-4-Turbo ground truth means its ceiling is set by GPT-4's clue quality; a version trained on expert-authored or human-validated clues only would test whether that ceiling binds."],"forward_implications":["Teachers can input any Arabic educational text and obtain custom clue-answer pairs without needing a reference corpus or manual puzzle construction.","The Arabic-Clue-Instruct dataset becomes a reusable resource for training and benchmarking Arabic generation models beyond crosswords.","The fine-tuning recipe, using stronger-model-generated data followed by parameter-efficient tuning of an open 8B model, is a transferable pattern for other under-resourced languages.","The reported 78.86% 'A' rate implies that an open 8B model can approach proprietary-model clue quality after relatively cheap fine-tuning."],"supporting_citations":[{"why":"It supplies the SELF-INSTRUCT-style data-generation approach and the A-E rating criteria used to build and assess the Arabic clues.","marker":"(Wang et al., 2022)"},{"why":"It is the earlier Arabic crossword system that this work extends from few-shot prompting to fine-tuning with a dedicated dataset.","marker":"(Zeinalipour et al., 2023b)"},{"why":"It provides the English Clue-Instruct dataset and clue-generation method that the Arabic dataset adapts.","marker":"(Zugarini et al., 2024)"},{"why":"It supplies LoRA, the parameter-efficient fine-tuning technique used for Llama3-8B-Instruct.","marker":"(Hu et al., 2021)"},{"why":"It is the base-model family from which the open Llama3-8B-Instruct model is drawn.","marker":"(Touvron et al., 2023)"},{"why":"It is cited as the foundation for the GPT-3-class models underlying GPT-3.5-Turbo.","marker":"(Brown et al., 2020)"}],"fun_headline_variants":["Arabic clue dataset lifts Llama3 to 79% human-rated quality","54k Arabic clues turn Llama3 into crossword clue writer","Fine-tuning on Arabic clues boosts Llama3 A-rating to 79%","Arabic-Clue-Instruct dataset: Llama3 clue quality jumps to 79%","From Arabic text to puzzles: LLM tool for classroom crosswords"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4-Turbo's clues are good enough to serve as both training signal and evaluation reference; this is checked by a single native Arabic speaker on only 200 samples, so systematic errors or dialect bias in those clues would be inherited by the fine-tuned models and would make the ROUGE comparisons against GPT-4 meaningless as quality evidence.","fun_headline_variants_meta":{"raw":{"variants":["Arabic clue dataset lifts Llama3 to 79% human-rated quality","54k Arabic clues turn Llama3 into crossword clue writer","Fine-tuning on Arabic clues boosts Llama3 A-rating to 79%","Arabic-Clue-Instruct dataset: Llama3 clue quality jumps to 79%","From Arabic text to puzzles: LLM tool for classroom crosswords"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":2933,"prompt_tokens":916,"completion_tokens":2017,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1919}},"tokens_in":532,"tokens_out":2017,"duration_ms":15541,"temperature":1.0,"reasoning_tokens":1919,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:41:09.041081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of independent native Arabic speakers rate the same sample of clues from the fine-tuned Llama3-8B model using the paper's A-E rubric; if the average 'A' share falls well below 78.86% or inter-annotator agreement is low, the headline quality claim does not replicate. A cheaper check is to measure how often the fine-tuned model reproduces GPT-4-Turbo clues nearly verbatim, since the training data was generated by that model.","supporting_citations":[{"cited_title":"Clue-Instruct: Text-Based Clue Generation for Educational Crossword Puzzles","cited_arxiv_id":"2404.06186","evidence_quote":"It provides the English Clue-Instruct dataset and clue-generation method that the Arabic dataset adapts."}],"review_version":1}