{"id":"d46f47af-a0d2-418f-9f99-9d34d2102e8c","arxiv_id":"2412.09012","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs solve cryptic crossword clues with at most 11.4% accuracy, far below human experts, and fail mainly at definition extraction and wordplay type identification.","lead":"This paper benchmarks three LLMs on cryptic crossword solving and analyzes where they fail. It finds ChatGPT reaches only 11.4% accuracy versus about 99% for expert humans, with the main difficulties being identifying the definition and the wordplay type.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The wordplay-type analysis, central to the 'why' contribution, rests on 200 unvalidated annotations; the paper's confusion-matrix summary (Sec. 5.2.2) also contradicts its own figures, so the localization claims are not yet reliable.","rationale":"I agree with the reader's weakest assumption: the wordplay-type dataset is the load-bearing weak point. The main solving-accuracy result (2-11%) is robust and well-supported by the released code and data; the paper is transparent about limitations and uses appropriate zero-shot prompts. However, the central 'why' contribution—that difficulties localize to definition extraction, wordplay detection, and explanation—depends heavily on the auxiliary tasks. The definition-extraction numbers (best 41.2%) are on 1,000 sampled clues and lack a stated scoring metric, but the wordplay-detection experiment is the shakiest because it uses only 200 self-annotated examples. The paper's own text misreads its confusion matrices, which is a concrete sign that the analysis is not being carefully validated. A re-annotation study and bootstrap CIs would settle whether the reported patterns are real. Until then, the verdict should remain conditional: the headline benchmark is solid, but the explanatory claims need verification.","tokens_in":14765,"tokens_out":15544,"duration_ms":150851,"concrete_test":"Have two independent expert cryptic solvers annotate the full 200-clue dataset; compute Cohen's kappa and recompute per-type confusion matrices. Then bootstrap the 200 samples (10k resamples) to obtain 95% CIs for overall accuracy and per-type recall, and check whether the Section 5.2.2 over/under-prediction pattern survives. If kappa < 0.7 or CIs overlap chance, the wordplay-detection results should be reported as inconclusive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2.2's conclusion that 'all 3 models over-predict some types (anagram and hidden word) while under-predicting others (assemblage)' is not supported by the confusion matrices in Appendix C. Figure C1 shows LLaMA3 predicts anagram only 10/200, i.e., strongly under-predicts it; Figure C3 shows ChatGPT predicts hidden word 21/200, also under-predicted. Only Gemma's matrix matches the text's pattern (over-predicts anagram at 101/200 and hidden at 43/200). This inconsistency indicates the small-sample analysis is being read loosely. More fundamentally, the 200-example dataset (40 per type, no inter-annotator agreement) has per-type standard error ~7.9%; the reported between-model differences (e.g., 44.5% vs 43.5% for ChatGPT vs Gemma) are within noise, and the balanced 40-per-type design does not reflect the natural distribution of wordplay types in the source corpus, so the over/under-prediction framing conflates model bias with test-set base rates. Since this experiment is the quantitative backbone of the paper's 'why' contribution, the localization claims need stronger support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper evaluates three LLMs (Gemma2-9B-it, Meta-Llama-3-8B-Instruct, and gpt-3.5-turbo) on cryptic crossword clue solving under zero-shot prompting, reporting accuracies between 2.1% and 11.4% on the Guardian-based test set and between 3.3% and 16.2% (with definition provided) on a 1000-clue sample from the Times for the Times blog. It then studies three auxiliary tasks—definition extraction (best 41.2% accuracy), wordplay-type detection (best 44.5% accuracy), and qualitative explanation analysis—to understand why the models struggle. The paper releases its code and the small annotated wordplay-type dataset.","tokens_in":14983,"tokens_out":7951,"duration_ms":67738,"significance":"The benchmark numbers are a useful, reproducible reference point: they confirm that current LLMs remain far below human experts and amateurs on this task, and the definition-extraction results identify a clear, robust weakness (best 41.2% on the 1000-clue sample). If the wordplay-type and explanation analyses were adequately supported, the paper would provide actionable diagnostic information for future work. However, the wordplay-type evidence is based on a small, unvalidated annotation set and is presented in a way that contradicts the paper's own confusion matrices; the explanation analysis lacks any coding protocol. These two pillars of the 'why' contribution are therefore not yet reliable, though they are in principle fixable.","major_comments":[{"comment":"The sentence in §5.2.2 that 'all 3 models over-predict some types (anagram and hidden word) while under-predicting others (assemblage)' is not consistent with the confusion matrices in Appendix C. For LLaMA3 (Figure C1), the predicted 'anagram' column sums to 10/200, which is a strong under-prediction, while the predicted 'hidden word' column sums to about 135/200, an over-prediction. For ChatGPT (Figure C3), the predicted 'hidden word' column sums to 21/200, again an under-prediction. Only Gemma (Figure C2) shows the pattern described in the text. The discussion should be corrected to describe each model separately, and any claims of a shared bias should be removed or supported by an appropriate aggregate analysis.","section":"§5.2.2 and Appendix C (Figures C1–C3)"},{"comment":"The wordplay-type dataset contains only 200 examples (40 per type) annotated by the authors without reported inter-annotator agreement. For a binomial proportion at n=200, the standard error is about 3.5 percentage points; the observed gap between ChatGPT (44.5%) and Gemma (43.5%) is therefore within sampling noise, and even the gap between ChatGPT and LLaMA3 should be reported with confidence intervals. Additionally, because the dataset is balanced by construction, the over/under-prediction analysis compares model predictions against a uniform prior that does not reflect the natural distribution of wordplay types; the reported 'over-prediction of anagram' for Gemma may simply reflect the dataset's equal-weight design. The paper should either sample from the natural distribution, weight the evaluation accordingly, or report per-type precision and recall with uncertainty intervals. These results are the quantitative core of the paper's 'why' contribution and need stronger support.","section":"§3.3 and §5.2.2"},{"comment":"The explanation analysis is entirely qualitative. No annotation scheme, rubric, or inter-annotator agreement is described, and the five summary claims (e.g., that LLaMA3 works only at the synonym level or that double-definition clues are easiest to explain) are not accompanied by counts, examples of the coding decisions, or a defined sampling procedure. To support the paper's conclusion that models' inability to explain operations is one of the reasons for poor solving performance, this subsection should provide a coding protocol and at minimum quantitative breakdowns (e.g., fractions of explanations judged as correct, partially correct, or wrong per wordplay type), or be repositioned as preliminary anecdotal evidence.","section":"§5.2.3"},{"comment":"The statement that 'providing the models with the definition improves their performance' is only partially supported. On the 1000-clue subset (Table 1 rows 3–4), ChatGPT improves by 2.8 points and Gemma by 1.7 points, but LLaMA3 improves by only 0.5 points; no significance testing is reported. On the full 28,476-clue set (rows 1–2), switching from the base prompt to the all-inclusive prompt decreases Gemma's accuracy from 4.8% to 2.4% and LLaMA3's from 2.2% to 2.1%, so the general claim does not hold across the main dataset. The claim should be restricted to the specific 1000-clue subset or supported by a proper comparison that controls for prompt and dataset.","section":"§5.1"}],"minor_comments":[{"comment":"The word 'definiton' appears in the prompt text in Figure E2 and in the running text of §5.1; fix the typo to 'definition'.","section":"Figure E2 and surrounding text"},{"comment":"The evaluated model is identified as 'gpt3.5-turbo' in Appendix A but is called 'ChatGPT' throughout the main text; state explicitly that 'ChatGPT' refers to gpt-3.5-turbo, and include the exact model version and access date to facilitate reproducibility.","section":"Appendix A"},{"comment":"The captions of Figures C1–C3 do not specify matrix orientation; add a sentence in the main text or captions indicating that rows are true labels and columns are predicted labels to prevent misreading.","section":"Appendix C"},{"comment":"The example for 'assemblage' would benefit from a brief annotation explaining how 'Bitter initially, but extremely enjoyable refreshment' decomposes into 'B' + 'eer'; as written, it is not transparent how the example illustrates the wordplay type.","section":"Table B1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest and reproducible in its benchmark component, and it does not suffer from circular reasoning or invented constructs. The core problem is that the paper's diagnostic claims go beyond what the evidence supports: the wordplay-type analysis is internally inconsistent with the appendix matrices, statistically underpowered, and built on unvalidated annotations. In its current form the paper is best described as a benchmark plus preliminary diagnostic observations; a revision that either strengthens the diagnostics or explicitly rebrands them as preliminary would bring the contribution in line with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The benchmark results are the real contribution: 28,476 Guardian clues, three LLMs, simple prompts, and a clean comparison showing ChatGPT at 11.4% against T5 fine-tuned at 21.8%. That is a useful, reproducible data point. The definition-extraction and explanation analyses are also worth having, and releasing code and data is the right move. The paper is honest about its limited model set and dataset sizes in the Limitations section.\n\nThe soft spot is exactly where the reader put it: the wordplay-type analysis. The 200-example dataset—40 per type, no inter-annotator agreement, balanced rather than naturalistic—has per-type standard error around 8%, so the differences between models (e.g., 44.5% vs. 43.5%) are noise. That alone would make me cautious, but the paper then makes a stronger claim that is simply false on its own evidence. Section 5.2.2 says \"all 3 models over-predict some types (anagram and hidden word) while under-predicting others (assemblage).\" The confusion matrices in Appendix C show only Gemma fits that pattern. LLaMA3 under-predicts anagram (4 of 40 true anagrams) and never predicts assemblage or double definition. ChatGPT over-predicts container (97 times) and under-predicts hidden word (21 total). This is not a minor wording issue; it is the quantitative backbone of the \"why\" contribution, and the summary misreads the data.\n\nThe stress-test note is right. The fix is straightforward: correct the text, report per-type confidence intervals, and either add IAA for the 200 annotations or frame the wordplay detection as exploratory. The 1,000-clue subsample matching only on definition length and position is also weaker than the paper implies, but that mainly affects the strength of the definition-extraction comparisons, not the main solving numbers.\n\nWho gets value from this? People working on LLM reasoning over structured language manipulation, and crossword/NLP folks who want a clean comparison against prior T5 results. It deserves a serious referee—the benchmark and task decomposition are useful enough—but the wordplay summary must be corrected before publication. I would send it to review with a request for major revision on that section.","headline":"Solid zero-shot benchmark on cryptic crosswords, but the paper's own confusion matrices contradict its central 'why' claim about wordplay over/under-prediction.","tokens_in":15516,"tokens_out":2064,"would_cite":false,"duration_ms":22490,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot LLMs solve at most 11.4% of cryptic crossword clues, far below human solvers.","keywords":["cryptic crosswords","large language models","wordplay detection","definition extraction","zero-shot prompting","explanation extraction","NLP benchmark"],"falsifier":"Re-annotate the 200 wordplay-type clues with at least two independent cryptic-crossword solvers and compute agreement; if agreement is low, or if a larger balanced sample changes which types are over- and under-predicted, the paper's localization of the difficulty to wordplay-type identification collapses. A smaller check: run the wordplay-type prompt on clues that contain no indicator words; if accuracy stays high, the surface-cue explanation is wrong.","tokens_in":14557,"feed_emoji":"🧩","tokens_out":9003,"duration_ms":77212,"temperature":0.7,"pith_summary":"This paper is trying to establish, with evidence, why large language models are so bad at cryptic crosswords rather than just documenting that they are. Across Gemma2, LLaMA3, and ChatGPT, zero-shot clue-solving accuracy ranges from 2.1% to 11.4% on a 28,476-clue Guardian test set, against human experts who reach up to 99%. By splitting the task into three sub-tasks, the authors localize the difficulty: extracting the definition from a clue (best 41.2%), identifying the wordplay type (best 44.5%), and correctly explaining the wordplay operations. The paper concludes that models cannot spontaneously decompose a clue into its definition and wordplay halves, and that instruction-style prompts alone do not fix this.","feed_headline":"Best LLM cracks just 11.4% of cryptic crosswords","feed_subtitle":"The gap is not language ability but spotting the definition and the wordplay type.","key_machinery":"The carrying mechanism is the decomposition of a cryptic clue into a definition part and a wordplay part, with wordplay classified into five named types: anagram, assemblage, container, hidden word, and double definition. This decomposition supplies the paper's auxiliary tasks: extract the definition, classify the wordplay type under prompts that add explanations, examples, or the answer, and explain a given clue-answer pair. Differences in accuracy across these sub-tasks are used to identify which component of clue-solving is the bottleneck. The key comparison is between the all-inclusive prompt and the same prompt with the definition supplied, which isolates the contribution of definition extraction.","core_discovery":"The central claim is that LLMs' poor performance on cryptic crosswords stems from a specific compositional failure rather than from a general lack of language knowledge. The evidence is a zero-shot evaluation on the Guardian and Times for the Times clue sets plus three auxiliary tasks: definition extraction, wordplay-type classification with increasing prompt information, and explanation of given clue-answer pairs. ChatGPT, the strongest model, solves only 11.4% of clues on the large set; Gemma2 and LLaMA3 remain in the 2-5% range. Giving the model the definition improves solving accuracy, while adding general wordplay guidelines does not. In wordplay-type detection, all three models over-predict anagram and hidden word and under-predict assemblage, and their explanations show operations applied to the wrong letters or words, indicating that the models name wordplay categories without reliably executing them.","pith_inferences":["If the bottleneck is definition extraction and wordplay-type identification, then prompt designs that force the model to mark the definition-wordplay boundary before answering should produce larger gains than scaling model size alone.","The over-prediction pattern predicts a testable effect: clues written without typical indicator words (e.g., 'confused', 'in') should lower wordplay-type accuracy even further, because models would lack the surface cues they currently rely on.","The single-clue setup removes the help of crossing letters; a natural extension is to feed LLMs a partially filled grid and measure how much accuracy rises, which would quantify how much of the human advantage comes from grid context.","Because the 200-clue wordplay dataset has no inter-annotator agreement reported, the type-level over- and under-prediction findings should be treated as provisional until re-annotated by multiple solvers."],"forward_implications":["Providing the definition explicitly improves solving accuracy (for ChatGPT, from 13.4% to 16.2% on the 1,000-clue sample), so better definition extraction should transfer directly to better solving.","Wordplay-type classification stays at or below 44.5% even when the answer is given, meaning the bottleneck is recognizing how clue parts fit together, not producing the final word.","All models over-predict anagram and hidden word and under-predict assemblage, suggesting they rely on surface indicator words rather than true letter-level analysis.","Because models fail to break down the composite task on their own, the paper's suggested routes—chain-of-thought, curriculum learning, and mixture-of-experts—are concrete next steps, though untested here."],"supporting_citations":[{"why":"Supplies the 142,380-clue Guardian dataset and the naive split whose 28,476-example test set is used for the main solving evaluation, plus the curriculum-learning T5 baseline of 21.8%.","marker":"Rozner et al. (2021)"},{"why":"Introduces the Cryptonite benchmark and the fine-tuned T5 baseline of 7.6% that frames the comparison with transformer models.","marker":"Efrat et al. (2021)"},{"why":"Provides the human performance reference: expert solvers reach up to 99% and amateurs 74% accuracy.","marker":"Friedlander and Fine (2009)"},{"why":"Provides the 7.3% rule-based solver baseline included in the comparison table.","marker":"Deits (2015)"},{"why":"Earlier LLM evaluation that this paper extends by analyzing why models fail rather than only reporting low accuracy.","marker":"Sadallah et al. (2025)"},{"why":"Earlier LLM study using chain-of-thought and self-consistency; its different subset explains why its 20.85% result is not directly compared.","marker":"Saha et al. (2024)"}],"fun_headline_variants":["Best LLM cracks only 11.4% of cryptic crosswords—here's why","LLMs fail cryptic crosswords by misnaming wordplay, not lacking language","Cryptic crosswords reveal LLMs' gap: they spot words but not clues","LLMs over-predict anagrams and hidden words in cryptic clues","Why LLMs solve few cryptic clues: definition and wordplay elude them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 200 clues the authors annotated for wordplay type, with 40 clues per type and no reported inter-annotator agreement, are correct and representative enough to measure what models can and cannot detect.","fun_headline_variants_meta":{"raw":{"variants":["Best LLM cracks only 11.4% of cryptic crosswords—here's why","LLMs fail cryptic crosswords by misnaming wordplay, not lacking language","Cryptic crosswords reveal LLMs' gap: they spot words but not clues","LLMs over-predict anagrams and hidden words in cryptic clues","Why LLMs solve few cryptic clues: definition and wordplay elude them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000411,"raw_usage":{"total_tokens":2071,"prompt_tokens":832,"completion_tokens":1239,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":1135}},"tokens_in":448,"tokens_out":1239,"duration_ms":11676,"temperature":1.0,"reasoning_tokens":1135,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:20:18.864748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the 200 wordplay-type clues with at least two independent cryptic-crossword solvers and compute agreement; if agreement is low, or if a larger balanced sample changes which types are over- and under-predicted, the paper's localization of the difficulty to wordplay-type identification collapses. A smaller check: run the wordplay-type prompt on clues that contain no indicator words; if accuracy stays high, the surface-cue explanation is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 142,380-clue Guardian dataset and the naive split whose 28,476-example test set is used for the main solving evaluation, plus the curriculum-learning T5 baseline of 21.8%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the human performance reference: expert solvers reach up to 99% and amateurs 74% accuracy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the 7.3% rule-based solver baseline included in the comparison table."}],"review_version":1}