REVIEW 4 major objections 4 minor 31 references
What Makes Cryptic Crosswords Challenging for LLMs?
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Zero-shot LLMs solve at most 11.4% of cryptic crossword clues, far below human solvers.
desk verdict Solid zero-shot benchmark on cryptic crosswords, but the paper's own confusion matrices contradict its central 'why' claim about wordplay over/under-prediction. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the decomposition of a cryptic clue into a definition part and a wordplay part, with wordplay classified into five named types: anagram, assemblage, container, hidden word, and double definition. This decomposition supplies the paper's auxiliary tasks: extract the definition, classify the wordplay type under prompts that add explanations, examples, or the answer, and explain a given clue-answer pair. Differences in accuracy across these sub-tasks are used to identify which component of clue-solving is the bottleneck. The key comparison is between the all-inclusive prompt and the same prompt with the definition supplied, which isolates the contribution of definition extraction.
What would settle it
Re-annotate the 200 wordplay-type clues with at least two independent cryptic-crossword solvers and compute agreement; if agreement is low, or if a larger balanced sample changes which types are over- and under-predicted, the paper's localization of the difficulty to wordplay-type identification collapses. A smaller check: run the wordplay-type prompt on clues that contain no indicator words; if accuracy stays high, the surface-cue explanation is wrong.
Extended reading notes
Core claim
The central claim is that LLMs' poor performance on cryptic crosswords stems from a specific compositional failure rather than from a general lack of language knowledge. The evidence is a zero-shot evaluation on the Guardian and Times for the Times clue sets plus three auxiliary tasks: definition extraction, wordplay-type classification with increasing prompt information, and explanation of given clue-answer pairs. ChatGPT, the strongest model, solves only 11.4% of clues on the large set; Gemma2 and LLaMA3 remain in the 2-5% range. Giving the model the definition improves solving accuracy, while adding general wordplay guidelines does not. In wordplay-type detection, all three models over-predict anagram and hidden word and under-predict assemblage, and their explanations show operations applied to the wrong letters or words, indicating that the models name wordplay categories without reliably executing them.
Load-bearing premise
The load-bearing premise is that the 200 clues the authors annotated for wordplay type, with 40 clues per type and no reported inter-annotator agreement, are correct and representative enough to measure what models can and cannot detect.
Editorial extensions
If this is right
- Providing the definition explicitly improves solving accuracy (for ChatGPT, from 13.4% to 16.2% on the 1,000-clue sample), so better definition extraction should transfer directly to better solving.
- Wordplay-type classification stays at or below 44.5% even when the answer is given, meaning the bottleneck is recognizing how clue parts fit together, not producing the final word.
- All models over-predict anagram and hidden word and under-predict assemblage, suggesting they rely on surface indicator words rather than true letter-level analysis.
- Because models fail to break down the composite task on their own, the paper's suggested routes—chain-of-thought, curriculum learning, and mixture-of-experts—are concrete next steps, though untested here.
Reading between the lines
- If the bottleneck is definition extraction and wordplay-type identification, then prompt designs that force the model to mark the definition-wordplay boundary before answering should produce larger gains than scaling model size alone.
- The over-prediction pattern predicts a testable effect: clues written without typical indicator words (e.g., 'confused', 'in') should lower wordplay-type accuracy even further, because models would lack the surface cues they currently rely on.
- The single-clue setup removes the help of crossing letters; a natural extension is to feed LLMs a partially filled grid and measure how much accuracy rises, which would quantify how much of the human advantage comes from grid context.
- Because the 200-clue wordplay dataset has no inter-annotator agreement reported, the type-level over- and under-prediction findings should be treated as provisional until re-annotated by multiple solvers.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper evaluates three LLMs (Gemma2-9B-it, Meta-Llama-3-8B-Instruct, and gpt-3.5-turbo) on cryptic crossword clue solving under zero-shot prompting, reporting accuracies between 2.1% and 11.4% on the Guardian-based test set and between 3.3% and 16.2% (with definition provided) on a 1000-clue sample from the Times for the Times blog. It then studies three auxiliary tasks—definition extraction (best 41.2% accuracy), wordplay-type detection (best 44.5% accuracy), and qualitative explanation analysis—to understand why the models struggle. The paper releases its code and the small annotated wordplay-type dataset.
Significance. The benchmark numbers are a useful, reproducible reference point: they confirm that current LLMs remain far below human experts and amateurs on this task, and the definition-extraction results identify a clear, robust weakness (best 41.2% on the 1000-clue sample). If the wordplay-type and explanation analyses were adequately supported, the paper would provide actionable diagnostic information for future work. However, the wordplay-type evidence is based on a small, unvalidated annotation set and is presented in a way that contradicts the paper's own confusion matrices; the explanation analysis lacks any coding protocol. These two pillars of the 'why' contribution are therefore not yet reliable, though they are in principle fixable.
major comments (4)
- [§5.2.2 and Appendix C (Figures C1–C3)] The sentence in §5.2.2 that 'all 3 models over-predict some types (anagram and hidden word) while under-predicting others (assemblage)' is not consistent with the confusion matrices in Appendix C. For LLaMA3 (Figure C1), the predicted 'anagram' column sums to 10/200, which is a strong under-prediction, while the predicted 'hidden word' column sums to about 135/200, an over-prediction. For ChatGPT (Figure C3), the predicted 'hidden word' column sums to 21/200, again an under-prediction. Only Gemma (Figure C2) shows the pattern described in the text. The discussion should be corrected to describe each model separately, and any claims of a shared bias should be removed or supported by an appropriate aggregate analysis.
- [§3.3 and §5.2.2] The wordplay-type dataset contains only 200 examples (40 per type) annotated by the authors without reported inter-annotator agreement. For a binomial proportion at n=200, the standard error is about 3.5 percentage points; the observed gap between ChatGPT (44.5%) and Gemma (43.5%) is therefore within sampling noise, and even the gap between ChatGPT and LLaMA3 should be reported with confidence intervals. Additionally, because the dataset is balanced by construction, the over/under-prediction analysis compares model predictions against a uniform prior that does not reflect the natural distribution of wordplay types; the reported 'over-prediction of anagram' for Gemma may simply reflect the dataset's equal-weight design. The paper should either sample from the natural distribution, weight the evaluation accordingly, or report per-type precision and recall with uncertainty intervals. These results are the quantitative core of the paper's 'why' contribution and need stronger support.
- [§5.2.3] The explanation analysis is entirely qualitative. No annotation scheme, rubric, or inter-annotator agreement is described, and the five summary claims (e.g., that LLaMA3 works only at the synonym level or that double-definition clues are easiest to explain) are not accompanied by counts, examples of the coding decisions, or a defined sampling procedure. To support the paper's conclusion that models' inability to explain operations is one of the reasons for poor solving performance, this subsection should provide a coding protocol and at minimum quantitative breakdowns (e.g., fractions of explanations judged as correct, partially correct, or wrong per wordplay type), or be repositioned as preliminary anecdotal evidence.
- [§5.1] The statement that 'providing the models with the definition improves their performance' is only partially supported. On the 1000-clue subset (Table 1 rows 3–4), ChatGPT improves by 2.8 points and Gemma by 1.7 points, but LLaMA3 improves by only 0.5 points; no significance testing is reported. On the full 28,476-clue set (rows 1–2), switching from the base prompt to the all-inclusive prompt decreases Gemma's accuracy from 4.8% to 2.4% and LLaMA3's from 2.2% to 2.1%, so the general claim does not hold across the main dataset. The claim should be restricted to the specific 1000-clue subset or supported by a proper comparison that controls for prompt and dataset.
minor comments (4)
- [Figure E2 and surrounding text] The word 'definiton' appears in the prompt text in Figure E2 and in the running text of §5.1; fix the typo to 'definition'.
- [Appendix A] The evaluated model is identified as 'gpt3.5-turbo' in Appendix A but is called 'ChatGPT' throughout the main text; state explicitly that 'ChatGPT' refers to gpt-3.5-turbo, and include the exact model version and access date to facilitate reproducibility.
- [Appendix C] The captions of Figures C1–C3 do not specify matrix orientation; add a sentence in the main text or captions indicating that rows are true labels and columns are predicted labels to prevent misreading.
- [Table B1] The example for 'assemblage' would benefit from a brief annotation explaining how 'Bitter initially, but extremely enjoyable refreshment' decomposes into 'B' + 'eer'; as written, it is not transparent how the example illustrates the wordplay type.
Circularity Check
No circularity: the paper is an empirical benchmark and descriptive error analysis, with no fitted parameters or derivation chain that reduces to its inputs.
full rationale
The paper makes no mathematical derivation or modeling claim that could reduce to its own inputs. Its central results are direct measurements of LLM accuracy on two external datasets (Guardian and Times for the Times) and on a small author-annotated wordplay-type dataset. The wordplay-type dataset is used as ground truth for a classification task, and reporting accuracy against one's own annotations is standard evaluation practice, not circularity, because no parameter is fitted to the labels and no quantity is defined in terms of the output being predicted. The authors' citation of their own prior work (Sadallah et al., 2025) is contextual, reporting that other LLM evaluations also show low accuracy; the present benchmark results are computed independently in this paper and do not depend on that citation. The auxiliary tasks (definition extraction, wordplay detection, explanation extraction) are descriptive probes into model behavior rather than predictions derived from assumptions. The paper explicitly acknowledges small dataset size and limited model coverage in its Limitations section, which is a validity concern, not a circularity concern. No equation, fitted parameter, uniqueness claim, or ansatz is smuggled in via citation. Therefore the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The Times for the Times dataset's marked definitions are accurate and the 1,000-clue sample preserves the distribution of definition length and position.
- domain assumption The 200-clue small explanatory dataset is correctly and consistently annotated with wordplay types by the authors.
- domain assumption Solving one clue at a time is a meaningful proxy for the real grid-solver task.
- domain assumption Human performance baselines (99% experts, 74% amateurs) from the cited survey literature are reliable.
Cite this review
Pith. "Pith review of What Makes Cryptic Crosswords Challenging for LLMs?." pith.science (2026). https://pith.science/paper/QATNLBGK
@misc{pith2026241209012,
author = {Pith},
title = {Pith review of: What Makes Cryptic Crosswords Challenging for LLMs?},
year = {2026},
howpublished = {\url{https://pith.science/paper/QATNLBGK}},
note = {Machine review of arXiv:2412.09012}
}
read the original abstract
Cryptic crosswords are puzzles that rely on general knowledge and the solver's ability to manipulate language on different levels, dealing with various types of wordplay. Previous research suggests that solving such puzzles is challenging even for modern NLP models, including Large Language Models (LLMs). However, there is little to no research on the reasons for their poor performance on this task. In this paper, we establish the benchmark results for three popular LLMs: Gemma2, LLaMA3 and ChatGPT, showing that their performance on this task is still significantly below that of humans. We also investigate why these models struggle to achieve superior performance. We release our code and introduced datasets at https://github.com/bodasadallah/decrypting-crosswords.
Figures
Reference graph
Works this paper leans on
-
[1]
Tri Dao. 2023. https://arxiv.org/abs/2307.08691 FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning . Preprint, arXiv:2307.08691
arXiv 2023
-
[2]
Robin Deits. 2015. https://github.com/rdeits/cryptics Cryptics
work page 2015
-
[3]
Avia Efrat, Uri Shaham, Dan Kilman, and Omer Levy. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.344 Cryptonite: A Cryptic Crossword Benchmark for Extreme Ambiguity in Language . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4186--4192, Online and Punta Cana, Dominican Republic. Association for Comput...
-
[4]
Liana Ermakova, Anne-Gwenn Bosser, Adam Jatowt, and Tristan Miller. 2023. https://doi.org/10.1145/3539618.3591885 The JOKER Corpus: English-French Parallel Data for Multilingual Wordplay Recognition . In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, page 2796–2806, New York, NY,...
arXiv 2023
-
[5]
Kathryn J. Friedlander and Philip A. Fine. 2016. https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2016.00567/full The grounded expertise components approach in the novel area of cryptic crossword solving . Frontiers
arXiv 2016
-
[6]
Kathryn J. Friedlander and Philip A. Fine. 2020. Fluid Intelligence is Key to Successful Cryptic Crossword Solving . Journal of Expertise, 3(2):101--132
work page 2020
-
[7]
KJ Friedlander and PA Fine. 2009. Expertise in cryptic crossword performance: an exploratory survey . In Proceedings of the International Symposium on Performance Science, Auckland, eds A. Williamon, S. Pretty, and R. Buck (Utrecht: European Association of Conservatoires (AEC)), pages 279--284
work page 2009
-
[8]
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2022. https://arxiv.org/abs/2211.15841 MegaBlocks: Efficient Sparse Training with Mixture-of-Experts . Preprint, arXiv:2211.15841
arXiv 2022
Show all 31 references
-
[9]
Team Gemma, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...
2024 arXiv
-
[11]
He He, Nanyun Peng, and Percy Liang. 2019. https://doi.org/10.18653/v1/N19-1172 Pun Generation with Surprise . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and...
2019 doi
-
[12]
Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. 1991. https://doi.org/10.1162/neco.1991.3.1.79 Adaptive Mixture of Local Experts . Neural Computation, 3:78--88
1991 doi
-
[13]
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...
2024 arXiv
-
[14]
Littman, Greg A
Michael L. Littman, Greg A. Keim, and Noam Shazeer. 2002. https://doi.org/10.1016/S0004-3702(01)00114-X A probabilistic approach to solving crossword puzzles . Artificial Intelligence, 134(1):23--55
2002 doi
-
[15]
Fuli Luo, Shunyao Li, Pengcheng Yang, Lei Li, Baobao Chang, Zhifang Sui, and Xu Sun. 2019. https://doi.org/10.18653/v1/D19-1336 Pun- GAN : Generative Adversarial Network for Pun Generation . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process...
2019 doi
-
[16]
Tim Moorey. 2018. How to Crack Cryptic Crosswords . Collins Puzzles
2018
-
[17]
OpenAI. 2021. ChatGPT . Technical report, OpenAI
2021
-
[18]
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...
2024 arXiv
-
[19]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer . Preprint, arXiv:1910.10683
2023 arXiv
-
[20]
Josh Rozner, Christopher Potts, and Kyle Mahowald. 2021. https://proceedings.neurips.cc/paper/2021/hash/5f1d3986fae10ed2994d14ecd89892d7-Abstract.html Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLP . In Advances in Neural Information P...
2021
-
[21]
Abdelrahman Sadallah, Daria Kotova, and Ekaterina Kochmar. 2025. https://arxiv.org/abs/2403.12094 Are llms good cryptic crossword solvers? Preprint, arXiv:2403.12094
2025 arXiv
-
[22]
Soumadeep Saha, Sutanoya Chakraborty, Saptarshi Saha, and Utpal Garain. 2024. https://arxiv.org/abs/2406.09043 Language Models are Crossword Solvers . Preprint, arXiv:2406.09043
2024 arXiv
-
[23]
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and N. Sebe. 2021. https://api.semanticscholar.org/CorpusID:231709290 Curriculum Learning: A Survey . International Journal of Computer Vision, 130:1526 -- 1565
2021
-
[24]
Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://openreview.net/forum?id=1PL1NIMMrw Self-Consistency Improves Chain of Thought Reasoning in Language Models . In The Eleventh International Conferenc...
2023
-
[25]
Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. https://arxiv.org/abs/2206.07682 Emergen...
2022 arXiv
-
[26]
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. https://arxiv.org/abs/2201.11903 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models . Preprint, arXiv:2201.11903
2023 arXiv
-
[27]
Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. 2023. https://arxiv.org/abs/2305.14688 ExpertPrompting: Instructing Large Language Models to be Distinguished Experts . Preprint, arXiv:2305.14688
2023 arXiv
-
[28]
Griffiths, Yuan Cao, and Karthik Narasimhan
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2305.10601 Tree of Thoughts: Deliberate Problem Solving with Large Language Models . Preprint, arXiv:2305.10601
2023 arXiv
-
[29]
Andrea Zugarini, Thomas Röthenbacher, Kai Klede, Marco Ernandes, Björn Eskofier, and Dario Zanca. 2023. https://ceur-ws.org/Vol-3596/paper54.pdf Die Rätselrevolution : Automated German Crossword Solving . In Proceedings of the 9th Italian Conference on Computational Linguistics
2023
-
[30]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[31]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
-
[32]
Pun Generation with Surprise
OpenAI. 2023. https://cdn.openai.com/papers/gpt-4.pdf Gpt-4 techincal report . @misc llama3modelcard, title= The Llama 3 Herd of Models , author= Aaron Grattafiori and Abhimanyu Dubey and Abhinav Jauhri and Abhinav Pandey and Abhishek Kadian and Ahmad Al-Dahle and Aiesha Letma...
2023 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.