{"id":"414dd162-0a23-4365-8855-d307125a6509","arxiv_id":"2411.14486","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark of 675 unsolvable questions finds that leading LLMs often fail to admit ignorance, scoring 62-68% even when 'I don't know' is the only correct choice.","lead":"The paper builds a 675-question 'impossible test' where the only correct multiple-choice answer is 'I don't know', and runs twelve open-source plus several closed LLMs on it. It reports that even the best models choose the uncertainty option only 62-68% of the time, and that GPT-4 says 'I don't know' more often on harder problems than on easier ones.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The scoring design is degenerate: 'I don't know' is the correct answer on all 675 items, so a trivial always-IDK baseline scores 100% and reported accuracy measures response bias, not calibrated uncertainty.","rationale":"The reader's weakest_assumption identifies the same core problem: because 'I don't know' is always correct, the test cannot distinguish epistemic humility from a degenerate strategy. This is the load-bearing flaw in the paper's argument. The paper itself states that 'by design, a perfect score is a consensus vote of ignorance across all 675 questions,' which confirms that the metric is a single-option accuracy. No control condition or penalty for unnecessary uncertainty admission exists, so the 62-68% figures and the inverse-difficulty trend are not evidence about calibrated uncertainty recognition. The dataset construction and the attempt to measure 'I don't know' behavior are useful contributions, but the central claim as stated is unsupported. This supports the reader's REJECT verdict, so verdict_should_be is REJECT.","tokens_in":13056,"tokens_out":3828,"duration_ms":38832,"concrete_test":"Run the released 675-item evaluation with a zero-parameter baseline that always selects the 'I don't know' option, and report its score. If the baseline achieves ~100%, then every model's accuracy in Figures 1-4 is just its IDK-selection rate; the central claim should be re-scored with a control set of solvable questions where IDK is marked wrong, so that a model must distinguish unsolvable from solvable to do well.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's scoring design makes the central metric uninterpretable. Every one of the 675 questions has 'I don't know' as the unique correct option, so accuracy is exactly the model's rate of selecting that option. A trivial policy that always chooses 'I don't know' scores 100%, and any model with a prior bias toward that option will score highly regardless of its actual awareness of the problem's unsolvability. The abstract's headline numbers (62-68%) are therefore base rates of option choice, not measures of calibrated epistemic humility. The inverse-difficulty claim inherits the same problem: GPT-4o-generated difficulty ranks need only correlate with features that make models less likely to pick 'I don't know' (e.g., question length, domain, phrasing) to produce the reported trend, and the absence of any solvable control questions or penalty for over-acknowledgment means there is no way to separate 'correctly recognizing an impossible problem' from 'always saying I don't know.' The dataset may be a useful resource, but as evaluated it cannot support the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a dataset of 675 supposedly unsolvable questions, organized into 49 categories, with 'I don't know' as the only correct answer. Twelve open-source and three closed-source LLMs are evaluated on their accuracy in selecting that option. The authors report that the best models score in the 62-68% range and claim an inverse relationship between problem difficulty and accuracy, with GPT-4 more likely to acknowledge uncertainty on harder problems (35.8%) than on easier ones (20.0%). The dataset is released on GitHub. The paper interprets these findings as evidence about LLM epistemic humility and as a contribution to AGI evaluation.","tokens_in":13237,"tokens_out":4566,"duration_ms":44537,"significance":"If the central claims held, the dataset would be a useful resource for studying model calibration about unknowability, and the inverse-difficulty result would be an intriguing behavioral observation. The paper ships a reproducible prompt structure, a public dataset, and a clear single-answer scoring rule, which are commendable. However, as evaluated, the headline metric conflates response bias with calibrated uncertainty, the difficulty axis is generated by the very model family under test, and the direction of the difficulty-accuracy relationship is described inconsistently. These issues are load-bearing and preclude interpreting the results as evidence about epistemic humility.","major_comments":[{"comment":"The scoring design is degenerate: because 'I do not know' is the correct answer for all 675 items, a trivial policy that always selects that option scores 100%, yet the best models are reported at 62-68%. The reported 'accuracy' is therefore simply the model's base rate of choosing that option, not a measure of calibrated uncertainty. The paper never reports the always-IDK baseline, nor does it include any solvable control questions or a penalty for over-acknowledgment, so the central metric cannot separate genuine recognition of unsolvability from a response bias. This affects every result in Figures 1-4 and the abstract's headline percentages.","section":"Evaluation and Scoring (page 7); Tables 1 and 5"},{"comment":"The difficulty rankings that support the inverse-difficulty claim were generated by GPT-4o during the reformulation process, and the tested models include GPT-4. This is circular: the 'relationship' is between GPT-4's answer choices and GPT-4o's labels, with no independent ground truth. The paper should validate the difficulty labels against human raters or another model family and control for confounds such as question length, domain, and phrasing, which may correlate with both the labels and the probability of selecting 'I don't know.'","section":"Dataset Development (Methods, 'To enhance the interpretability...') "},{"comment":"The paper states the central claim as an 'inverse relationship between problem difficulty and model accuracy,' but the example given in the abstract (higher rates on more challenging problems, 35.8%, versus simpler ones, 20.0%) and the Results section ('examination accuracy increases with problem difficulty') describe a positive relationship. The direction of the claimed effect must be defined and stated consistently; otherwise the headline finding is ambiguous.","section":"Abstract and Results (Figures 1-3)"},{"comment":"The premise that every question is 'fundamentally unsolvable' and that 'I don't know' is the only correct answer is not externally validated. For example, Table 1 asks to construct a muon collider, which is a hard engineering challenge but not a logical impossibility, and Table 3 asks to confirm Legendre's conjecture, which is an open problem rather than a demonstrably unsolvable one. The paper should provide per-question or per-category justification of unsolvability, or at least acknowledge that the dataset contains a mix of unsolved and unsolvable problems, because the validity of the ground truth is load-bearing for the accuracy metric.","section":"Dataset Development and Tables 1, 3, 4"}],"minor_comments":[{"comment":"The text says 'twelve state-of-the-art LLMs' and later describes three closed models, but Table 2 lists twelve open-source models. The total number of evaluated models and the naming of the closed models (e.g., exact versions of Claude, GPT, Gemini) should be stated explicitly.","section":"Introduction and Table 2"},{"comment":"The figure captions are telegraphic and omit axis labels, legends, and model names. For example, Figure 3 lacks a clear definition of the y-axis (accuracy vs. rate of 'I don't know' selections) and the x-axis (difficulty categories). Please clarify the figures so the reader can map the results to the text.","section":"Figures 1, 2, and 3"},{"comment":"The GitHub repository link in reference [26] is unlikely to be stable and contains three separate URLs; please deposit the dataset in a permanent archive (e.g., Zenodo) and cite a versioned DOI.","section":"References [26]"},{"comment":"No error bars, standard deviations, or statistical tests are reported, and there is no indication of the number of runs per model. Given that the claimed difficulty effect is small (20.0% vs. 35.8%), the authors should report confidence intervals or bootstrapped estimates for at least the headline comparisons.","section":"Results"}],"recommendation":"major_revision","confidential_remarks":"The dataset may be a useful starting resource, but the evaluation as designed cannot support the paper's central claims. The always-IDK baseline issue is fundamental: it changes the interpretation of every reported accuracy figure, and the difficulty-axis circularity further weakens the secondary claim. A major revision that adds control questions, baselines, independent difficulty validation, and a consistent statement of the difficulty-accuracy relationship could make the contribution publishable, but the full set of experiments would need to be redone. If the authors prefer not to invest that effort, the paper might be better suited to a workshop venue as a dataset proposal rather than a full empirical study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core problem with this paper is in the scoring. Because \"I don't know\" is the only correct answer on all 675 items, accuracy is just the model's rate of choosing that option. A trivial always-IDK policy scores 100%. The reported 62-68% figures are therefore base rates of option choice, not measures of calibrated uncertainty. The stress-test note is correct, and it undercuts the abstract's claims.\n\nThat said, there is real value here. The null-dataset concept is a fresh twist on hallucination benchmarks, and the 675-question compilation across 49 categories from Wikipedia's unsolved-problems lists is a concrete, reproducible resource. The worked examples are genuinely informative: models often craft elaborate rationales to land on a confident wrong answer rather than admit ignorance. The paper is also honest about the design intent, and the discussion of related AGI evaluation work is fair.\n\nThe soft spots are in proportion to the central metric. Besides the degenerate baseline, the inverse-difficulty relationship is built on GPT-4o's own difficulty labels and only shown for GPT-4, so it is circular. There are no error bars, no statistical tests, and no solvable control questions to separate \"correctly recognizes an impossible problem\" from \"always says IDK.\" The paper itself lists \"standardization of impossibility criteria\" as future work, which is a hint that the unsolvability of the 675 items is assumed, not verified.\n\nWho is this for? Someone building calibration or hallucination benchmarks could mine the dataset, and the paper is a useful cautionary example of why a degenerate baseline can wreck an otherwise clever design. But as an evaluation of LLM uncertainty, the conclusions do not hold.\n\nI would send this to peer review rather than desk reject: a serious referee can demand the obvious fixes—add solvable controls, penalize unnecessary uncertainty admission, use independent difficulty labels, and report a trivial baseline. The idea and the resource justify a major revision. I would not cite the empirical claims in my own work.","headline":"A promising null-dataset idea and a concrete resource, but the scoring design makes the headline numbers uninterpretable.","tokens_in":13793,"tokens_out":2479,"would_cite":false,"duration_ms":23841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a new benchmark of 675 deliberately unsolvable problems, where the only correct answer is 'I don't know,' shows that even the best large language models admit ignorance only about two-thirds of the time and guess…","keywords":["impossible test","unsolvable problems","epistemic humility","large language models","uncertainty acknowledgment","null dataset","AI evaluation","hallucination"],"falsifier":"Have a panel of domain experts independently attempt to answer a random sample of, say, 100 questions with full access to the literature; if any question receives a defensible correct solution, the null-set assumption is broken and the reported accuracy numbers would have to be recomputed.","tokens_in":12811,"feed_emoji":"🤔","tokens_out":9941,"duration_ms":82223,"temperature":0.7,"pith_summary":"This paper introduces a new evaluation dataset, the 'impossible test': 675 deliberately unsolvable problems spanning fields from mathematics to philosophy, each presented as a multiple-choice question where the only correct answer is 'I don't know.' Testing twelve large language models, the authors find that even the best performers select the correct admission of ignorance only 62–68% of the time. They also report a pattern they call an inverse relationship between difficulty and accuracy: GPT-4 acknowledges uncertainty on 35.8% of the harder questions but only 20.0% of the easier ones, suggesting that models are more prone to fabricate plausible answers when a problem looks tractable. The paper argues that recognizing the boundary of one's knowledge is a distinct capability, and that this null dataset offers a practical probe for that capability in AGI evaluation.","feed_headline":"Top models admit ignorance 62-68% on unsolvable questions","feed_subtitle":"A new 675-question benchmark reveals that even the best LLMs often guess rather than admit uncertainty.","key_machinery":"The key machinery is the curated null dataset itself: 675 questions reformulated from Wikipedia's lists of unsolved problems into graduate-level multiple-choice items, each with four plausible-but-incorrect distractors and one randomly placed 'I don't know' option scored as correct. The difficulty axis is supplied by GPT-4o's own rankings (medium versus extreme), and the central metric is simply the fraction of times a model selects the abstention option.","core_discovery":"The central claim is that a benchmark built from genuinely unsolvable problems—where 'I don't know' is the only right answer—can reveal whether a machine knows the limits of its own knowledge, and that current LLMs largely fail this test. The paper reports that across twelve models, the best scores fell between 62% and 68% accuracy, meaning that in roughly a third of cases the models exchanged a correct abstention for a confident but wrong distractor. The authors describe the difficulty–accuracy pattern as an inverse relationship, with GPT-4 showing higher rates of uncertainty acknowledgment on more challenging problems (35.8%) than on simpler ones (20.0%). Their interpretation is that models are more inclined to speculate when a problem appears easy, and they recommend that uncertainty recognition be included as a core component of any practical AGI evaluation.","pith_inferences":["The finding implies a testable design principle: training a model to abstain on deliberately impossible questions may also improve its calibration on ordinary difficult questions, though the paper does not test this transfer.","A natural extension would be to mix impossible questions with merely very hard but solvable ones in a single evaluation, to see whether models can learn to distinguish 'unsolved' from 'unanswerable.'","If the relationship between perceived difficulty and overconfidence holds generally, it suggests that a model's tendency to guess is driven more by the surface appearance of tractability than by its internal uncertainty—an interpretation the paper hints at but does not prove.","The dataset's reliance on Wikipedia's lists means it will decay as open problems get solved; keeping the benchmark alive would require periodic expert review to add newly impossible or still-open problems."],"forward_implications":["If LLMs are to be trusted in high-stakes roles, they need explicit abstention mechanisms; this test shows they cannot be relied on to say 'I don't know' spontaneously.","Current AI benchmarks that reward only correct answers may conceal a systematic failure mode: confident guessing on problems the model cannot solve.","The difficulty–accuracy relationship suggests that calibration training should target easy-looking problems, where models are most overconfident.","The dataset offers a moving benchmark: as models improve, their accuracy on the impossible test should rise, giving a concrete—if narrow—measure of progress toward AGI."],"supporting_citations":[{"why":"Supplies the 675-question 'impossible test' dataset and the recorded model answers; all reported accuracy figures derive from it.","marker":"[26]"},{"why":"Provides the Wikipedia lists of unsolved problems from which the questions were curated.","marker":"[27]"},{"why":"Establishes the 'I don't know' response as a marker of intelligence and optimal trust, the conceptual basis for scoring.","marker":"[50]"},{"why":"The 'Humanity's Last Exam' benchmark for hard but solvable problems, which this null dataset is explicitly contrasted with.","marker":"[45]"},{"why":"Hernández-Orallo and Dowe's universal intelligence test, the theoretical framework this work claims to extend empirically.","marker":"[12]"},{"why":"Hallucination leaderboards that motivate the measurement of fabricated answers in LLMs.","marker":"[48]"},{"why":"Documented data contamination in closed-source LLM evaluations, which the new dataset's design tries to avoid.","marker":"[53]"}],"fun_headline_variants":["AI more likely to guess when it thinks the problem is easy","Even top models fail to say 'I don't know' on unsolvable quiz","New test shows AI overconfidence on impossible questions","AI's blind spot: admitting ignorance on unsolvable problems","Why AI guesses wrong on questions that have no right answer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's validity hinges on every one of the 675 questions being genuinely unsolvable, so that 'I don't know' is the only defensible correct answer and a model that always selects it would score perfectly regardless of its reasoning.","fun_headline_variants_meta":{"raw":{"variants":["AI more likely to guess when it thinks the problem is easy","Even top models fail to say 'I don't know' on unsolvable quiz","New test shows AI overconfidence on impossible questions","AI's blind spot: admitting ignorance on unsolvable problems","Why AI guesses wrong on questions that have no right answer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000324,"raw_usage":{"total_tokens":1832,"prompt_tokens":970,"completion_tokens":862,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":775}},"tokens_in":586,"tokens_out":862,"duration_ms":9920,"temperature":1.0,"reasoning_tokens":775,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:55:07.248707+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a panel of domain experts independently attempt to answer a random sample of, say, 100 questions with full access to the literature; if any question receives a defensible correct solution, the null-set assumption is broken and the reported accuracy numbers would have to be recomputed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 675-question 'impossible test' dataset and the recorded model answers; all reported accuracy figures derive from it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Wikipedia lists of unsolved problems from which the questions were curated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The 'Humanity's Last Exam' benchmark for hard but solvable problems, which this null dataset is explicitly contrasted with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Hernández-Orallo and Dowe's universal intelligence test, the theoretical framework this work claims to extend empirically."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Hallucination leaderboards that motivate the measurement of fabricated answers in LLMs."}],"review_version":1}