{"id":"2c010a09-ef25-4ef8-b1ed-8747da6aa358","arxiv_id":"2608.04670","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ProverbIT shows that large language models can complete Italian proverbs but often fail to select 'none of the above' when the exact ending is absent, revealing a gap between memorized knowledge and discriminative reasoning.","lead":"This paper introduces ProverbIT, a benchmark of 100 Italian proverb questions where the correct answer is never listed and models must say 'none of the others.' Across 13 AI models, completing the proverbs is easy, but choosing the right 'none of the above' option is hard, and even reasoning models often mention the true ending in their internal thinking yet still pick a wrong answer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Without a human baseline, the benchmark's assumption that E is uniquely correct is untested; if some synonym distractors are actually acceptable proverb variants, the reported 'dramatic drop' overstates model failure and undermines the memorization-vs-understanding conclusion.","rationale":"The reader's weakest assumption is precisely the load-bearing point. The paper provides a new dataset and a clear empirical pattern, and the CoT traces showing models naming the true ending yet choosing a wrong option are genuinely interesting. That said, the headline interpretation depends on the validity of the E-only-answer premise. The paper even acknowledges the issue in §3.2 by stating 'we expect models to always answer E,' but expectation is not evidence. Example 1 from the supplementary materials illustrates why: 'ciò che brilla' is a semantic near-equivalent of the canonical 'quel che luccica,' and a native speaker could reasonably judge it as a correct completion. The absence of a human baseline makes Table 2 uninterpretable as evidence about 'semantic understanding'—the model may be choosing an acceptable paraphrase rather than failing negative reasoning. The proposed human-rating and item-filtering check would settle this. If the drop persists after removing ambiguous items, the paper's conclusion is supported; if not, the central claim needs substantial revision. I therefore agree with the reader's CONDITIONAL verdict and see no reason to change it.","tokens_in":13584,"tokens_out":4069,"duration_ms":44377,"concrete_test":"Run a human baseline study: recruit 10-20 native Italian speakers (not authors), present each of the 100 ProverbIT items in the same multiple-choice format (randomized option order), and ask them to select the best completion or E. Compute human accuracy on E and, crucially, item-level agreement. If human E-selection is below ~90% on any item, remove or rewrite that item; recompute Table 2 after excluding items that fail a pre-registered 'unambiguously wrong distractors' criterion (e.g., >2 of 15 raters choose a non-E option). If the post-filtered model accuracy still shows a large drop, the concern is resolved; if not, the central claim is an artifact of canonical-wording strictness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (abstract; §5) is that LLMs that fluently complete proverbs fail to choose 'None of the others' when the true ending is absent, and that this reveals reliance on memorized patterns rather than semantic understanding. This conclusion requires that every ProverbIT item has exactly one correct answer, E, and that a knowledgeable human would also select E. That premise is asserted in §3.1 ('we expect models to always answer E') but never validated: no human baseline is reported. The supplementary examples show the risk. In Example 1, 'Non è tutto oro...' offers B 'ciò che brilla' for the canonical 'quel che luccica'; semantically this is 'all that glitters is not gold' and a native speaker may well accept it as a variant. Example 3, 'Il gioco è bello quando...' with B 'è breve' for the canonical 'dura poco' is similarly a close paraphrase. If even a fraction of the 100 items admit acceptable alternative completions, the base-task accuracy numbers in Table 2 are not 'failure to detect absence' but disagreement about canonical wording. Without item-level human judgment, the abstract's 'performance drops dramatically' and the conclusion that models lack 'deeper semantic understanding' are not established. The claim could still be true, but its central measurement is currently unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ProverbIT, a new benchmark of 100 Italian proverb-completion multiple-choice questions in which the correct ending is deliberately absent and option E ('None of the other answers') is the intended answer. Thirteen LLMs and LRMs are evaluated on three tasks: open completion, multiple-choice with the true ending added, and the base ProverbIT task with no correct ending. The central reported finding is that models complete proverbs accurately (67-96%) but drop sharply on the base task (4-86%), with error analysis showing a bias toward literal-synonym distractors. Chain-of-thought analysis of DeepSeek R1 and Qwen 3 indicates that models often mention the correct proverb ending during reasoning yet still select a wrong option, and sometimes produce answers inconsistent with their own reasoning.","tokens_in":13731,"tokens_out":2894,"duration_ms":31942,"significance":"If the finding survives validation, it is a useful contribution: ProverbIT addresses an under-resourced language and a culturally embedded phenomenon, the dataset is publicly released, and the 'easy to complete, hard to choose' result offers a concrete test bed for the memorization-versus-understanding debate in LLMs. The paper's strengths include the manual construction of distractors with a clear taxonomy, the inclusion of both LRMs and smaller local models, and the detailed CoT analysis with quantified phenomena such as answer/reasoning mismatch. However, the central interpretation depends on an untested premise about item correctness, and the completion metric is described ambiguously; these issues must be resolved before the benchmark's conclusions can be accepted.","major_comments":[{"comment":"The claim that 'Since all provided endings are completely invented and thus incorrect, we expect models to always answer E' is not validated by any human baseline. The supplementary examples show that some distractors are semantically plausible variants of the canonical ending (e.g., 'ciò che brilla' for 'quel che luccica', 'è breve' for 'dura poco'), so a native speaker might accept them as legitimate completions. Without item-level human judgment, the 'dramatic drop' in base-task accuracy could reflect disagreement about canonical wording rather than failure to detect absence. I request a human baseline on all 100 items (or at least a subset with inter-annotator agreement) to establish that E is uniquely correct per item.","section":"§3.2 and §4.3"},{"comment":"The completion metric is ambiguous as stated: 'If the edit distance exceeds a threshold of 0.8 in at least two out of three runs, we consider the answer correct.' Since the implementation is from difflib, the value is likely a similarity ratio (higher = more similar), not an edit distance; as written, the sentence implies a larger edit distance indicates a correct answer, which would inflate completion accuracy. The threshold must be clarified (whether it is SequenceMatcher.ratio() > 0.8) and justified, because the completion results are the 'easy to complete' side of the paper's central contrast.","section":"§4.1.1"},{"comment":"No confidence intervals or significance tests are reported. With 100 items and three runs per condition, the differences between models (e.g., GPT o3 at 86% vs. Gemini 2.5 Pro at 77%) and between conditions may not be reliable. I recommend reporting bootstrap confidence intervals or applying McNemar's test per model for the base versus base+true-ending comparison, at minimum.","section":"§4.1 and Table 2"},{"comment":"It is unclear what happens to option E when the true ending is added. Does the true ending replace E, or is it added as a sixth option while E remains? The interpretation of the base+true-ending results changes substantially depending on whether models can still choose 'None of the others' in that condition. Please specify the option structure.","section":"§4.1, Base + true ending Task"}],"minor_comments":[{"comment":"The two Claude models are both labeled 'Claude Sonnet 4' in Table 2; one should be labeled 'Claude Sonnet 4 Thinking' to match Table 1 and the text.","section":"Table 2"},{"comment":"The sentence 'we set a reasonable thinking budget of 2000 tokens for o3, Sonnet 4, and Gemini 2.5 Pro' is followed by 'only 22 out of 600 CoTs from these two models exceeded the 2000-token limit'; the connection is confusing because the 600 CoTs come from DeepSeek R1 and Qwen 3, which were given unlimited budgets.","section":"§4.3.1"},{"comment":"Reference [39] contains a formatting typo in the title: '\" my answer is c\": First-token probabilities...' has mismatched quotes.","section":"References"},{"comment":"The figure captions for Figures 1 and 2 would benefit from stating explicitly that the 'correct proverb ending' is never present among the options A-D, so the dotted-line mentions demonstrate recall without recognition of absence; this is stated in the text but not in the captions.","section":"Figure captions"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical evaluation rather than a theoretical contribution, and its main risk is overclaiming from an unvalidated benchmark. The human-baseline issue is the primary blocker; without it, the central 'memorization versus understanding' conclusion is not established. The metric ambiguity in §4.1.1 is also load-bearing and should be fixed. The dataset release and CoT analysis are genuine strengths that make revision worthwhile."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, ProverbIT is a real asset: a public, 100-item Italian proverb benchmark with a systematic distractor typology, and a clean 13-model evaluation. Second, the central claim—models complete proverbs fine but fall apart when the true ending is absent—is probably right in direction, but the paper overstates it because the ground truth (E is always uniquely correct) is never checked against human judgment.\n\nThe genuinely new bits are the benchmark itself and the CoT mention-tracing for DeepSeek R1 and Qwen 3. The finding that these models often spell out the correct proverb ending in their reasoning and still select a wrong option is concrete and not in prior work. The result also coheres with Liu et al.'s MAPS and Salido et al.'s 'None of the others', so the phenomenon has independent plausibility.\n\nThe soft spot is load-bearing, not cosmetic. The authors assert in §3.1 that they expect models to always answer E because all invented endings are incorrect, but no human baseline is reported. Some of the supplementary items show why that matters. In Example 1, B ('ciò che brilla') is a natural paraphrase of the canonical 'quel che luccica'; in Example 3, B ('è breve') is close to 'dura poco'. If native speakers accept even a fraction of these as legitimate variants, the Table 2 drop partly reflects disagreement about canonical wording rather than failure to detect absence. The abstract's conclusion about 'memorized patterns rather than deeper semantic understanding' rides on that untested assumption.\n\nSecondary issues: no significance tests or confidence intervals on three-run majority votes, and the completion metric's 0.8 edit-distance threshold is under-specified (normalized? character-level?). These are fixable. The authors also could have randomized option order, which they mention as future work.\n\nWho it is for: people working on figurative-language evaluation, multilingual benchmarks, and negative reasoning. It deserves a serious referee. The dataset and CoT analysis are worth publishing, but the headline conclusion needs item-level human ratings and more careful statistical reporting before it should be accepted. Sent to review with expectation of major revision.","headline":"Useful new Italian proverb benchmark and a plausible completion/selection gap, but the missing human baseline keeps the headline conclusion unproven.","tokens_in":14390,"tokens_out":2551,"would_cite":false,"duration_ms":27595,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AI models ace proverb completion, yet fail when the right ending is absent.","keywords":["Italian proverbs","proverb completion","multiple-choice benchmark","large language models","chain-of-thought analysis","figurative language","negative reasoning","cultural knowledge"],"falsifier":"Run the same 100 ProverbIT items with a panel of native Italian speakers under the same prompt; if they choose a listed option, especially the synonym, on a substantial share of items, the benchmark's assumption that E is the only correct answer fails, and the model performance gap would then reflect exact-wording recall rather than lack of understanding.","tokens_in":1462,"feed_emoji":"🧠","tokens_out":1563,"duration_ms":80904,"temperature":0.7,"pith_summary":"The paper introduces ProverbIT, a set of 100 Italian multiple-choice questions built so that none of the four listed endings completes the proverb correctly; the only sound answer is \"None of the others.\" Across 13 language models, the authors find that nearly every model can freely complete the same proverbs, yet accuracy collapses when the correct ending is removed from the choices, with some models falling to 4-6%. Chain-of-thought traces from two reasoning models show the correct proverb ending appearing inside their reasoning even when they go on to choose a wrong option, usually a literal synonym. The authors' conclusion is that current models lean on memorized surface patterns rather than a semantic grasp of culturally grounded expressions.","feed_headline":"AI models ace proverb completion, fail if the right ending is absent","feed_subtitle":"A 100-item Italian benchmark drops accuracy from over 90% to as low as 4% when none of the listed endings is correct.","key_machinery":"The load-bearing object is the ProverbIT item format, in which the true ending is never one of the options: four manually constructed distractors (A assonant, B literal synonym, C inverse, D trivial) appear alongside E, \"None of the other answers.\" This shifts the task from pattern completion to negative discrimination, and it is what makes the performance gap interpretable. The second piece of machinery is the chain-of-thought mention-tracing applied to DeepSeek R1 and Qwen 3, which counts how often the true ending and each option are named during reasoning; this is what exposes the disconnect between knowing the correct ending and choosing it.","core_discovery":"The central finding is a knowledge-versus-discrimination gap: proverb completion succeeds, but selecting the absence of the correct continuation fails. The benchmark removes the true ending from every question, replacing it with four constructed distractors (assonant, literal-synonym, inverse, and trivial) plus option E, \"None of the other answers.\" On this base task, accuracy ranges from GPT o3's 86% down to Gemma 3's 4%, while the same models score much higher on direct completion and on a version where the true ending is added back as an option; some models lose tens of percentage points and one drops from roughly 75-88% to 6%. The paper interprets this as evidence that models rely on memorized linguistic patterns rather than deeper semantic understanding, and that even reasoning models only partially compensate. Error analysis shows a strong preference for the literal-synonym option, and CoT analysis of DeepSeek R1 and Qwen 3 reveals that both models frequently state the correct proverb ending while still failing to recognize that it is absent from the choices.","pith_inferences":["If native Italian speakers were given the same 100 questions and did not choose E on every item, the benchmark would be measuring exact-wording recall rather than proverb understanding; that human calibration is the paper's untested premise.","A direct follow-up could ask models to explain the meaning of a proverb after answering E, or to select the best paraphrase; this would separate recognition of absence from genuine semantic representation.","The same construction, removing the correct answer and adding \"None of the others,\" could be applied to idioms, catchphrases, and formulaic expressions in other low-resource languages; the paper's pattern suggests a general test for memorized routine versus understanding.","The bias toward the synonym option may be amplified by the letter E being the \"None\" answer; randomizing option positions, which the paper cites as future work, would show whether the effect is semantic or positional."],"forward_implications":["Completion scores should not be read as evidence that a model understands a proverb; a model can complete it fluently and still fail to reject wrong alternatives.","Multiple-choice evaluations should include a \"none of the others\" condition, otherwise a model can score well by recognizing surface patterns without verifying the answer's semantic correctness.","Reasoning traces are not reliable evidence of a model's conclusion: models can state the correct ending and still output a different letter.","Literal-synonym distractors are the main failure attractor, so model behavior on cultural expressions is better described as lexical matching than as meaning comparison.","Language-specific benchmarks like ProverbIT are needed because English-centric tests can hide these gaps."],"supporting_citations":[{"why":"The MAPS dataset on multicultural proverbs is the closest prior work; it provides the evidence that proverb memorization does not guarantee understanding, which ProverbIT extends to Italian with a no-correct-answer format.","marker":"[23]"},{"why":"The list of the 200 most common Italian proverbs from which the 100 ProverbIT items were selected.","marker":"[25]"},{"why":"The \"None of the others\" technique for distinguishing reasoning from memorization in multiple-choice benchmarks, the design principle behind the base task.","marker":"[37]"},{"why":"Documentation that reasoning models' final answers can diverge from their CoT conclusions, used to interpret the observed answer mismatches.","marker":"[38]"}],"fun_headline_variants":["Proverb test: models complete, can't tell when ending is missing","LLMs know proverb endings but fail when correct one is absent","Why LLMs ace proverb completion but miss missing endings","Proverb IQ: models memorize endings, fail absence test"],"cache_read_input_tokens":16384,"weakest_assumption_plain":"The load-bearing premise is that every invented ending is clearly wrong for a knowledgeable native speaker, so that \"None of the others\" is the only reasonable choice; because no human baseline is reported, a reader cannot verify that premise, and a few synonym options are close enough to blur it.","fun_headline_variants_meta":{"raw":{"variants":["Proverb test: models complete, can't tell when ending is missing","LLMs know proverb endings but fail when correct one is absent","Why LLMs ace proverb completion but miss missing endings","Proverb IQ: models memorize endings, fail absence test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000922,"raw_usage":{"total_tokens":3968,"prompt_tokens":977,"completion_tokens":2991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":2921}},"tokens_in":593,"tokens_out":2991,"duration_ms":22680,"temperature":1.0,"reasoning_tokens":2921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:17:59.382278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 100 ProverbIT items with a panel of native Italian speakers under the same prompt; if they choose a listed option, especially the synonym, on a substantial share of items, the benchmark's assumption that E is the only correct answer fails, and the model performance gap would then reflect exact-wording recall rather than lack of understanding.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The MAPS dataset on multicultural proverbs is the closest prior work; it provides the evidence that proverb memorization does not guarantee understanding, which ProverbIT extends to Italian with a no-correct-answer format."},{"cited_title":"Caramagna, I 200 proverbi italiani più belli e famosi (con significato), 2025","cited_arxiv_id":null,"evidence_quote":"The list of the 200 most common Italian proverbs from which the 100 ProverbIT items were selected."}],"review_version":1}