Pith. sign in

REVIEW 4 major objections 4 minor 61 references

The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a new benchmark of 675 deliberately unsolvable problems, where the only correct answer is 'I don't know,' shows that even the best large language models admit ignorance only about two-thirds of the time and guess…

desk verdict A promising null-dataset idea and a concrete resource, but the scoring design makes the headline numbers uninterpretable. read the letter →

arxiv 2411.14486 v1 pith:VJQA6Y47 submitted 2024-11-20 cs.CL cs.AI

David Noever , Forrest McKee This is my paper · ORCID
classification cs.CLcs.AI
keywords impossibletestunsolvableproblemsepistemichumilitylargelanguagemodelsuncertaintyacknowledgmentnulldatasetAIevaluationhallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces a new evaluation dataset, the 'impossible test': 675 deliberately unsolvable problems spanning fields from mathematics to philosophy, each presented as a multiple-choice question where the only correct answer is 'I don't know.' Testing twelve large language models, the authors find that even the best performers select the correct admission of ignorance only 62–68% of the time. They also report a pattern they call an inverse relationship between difficulty and accuracy: GPT-4 acknowledges uncertainty on 35.8% of the harder questions but only 20.0% of the easier ones, suggesting that models are more prone to fabricate plausible answers when a problem looks tractable. The paper argues that recognizing the boundary of one's knowledge is a distinct capability, and that this null dataset offers a practical probe for that capability in AGI evaluation.

What carries the argument

The key machinery is the curated null dataset itself: 675 questions reformulated from Wikipedia's lists of unsolved problems into graduate-level multiple-choice items, each with four plausible-but-incorrect distractors and one randomly placed 'I don't know' option scored as correct. The difficulty axis is supplied by GPT-4o's own rankings (medium versus extreme), and the central metric is simply the fraction of times a model selects the abstention option.

What would settle it

Have a panel of domain experts independently attempt to answer a random sample of, say, 100 questions with full access to the literature; if any question receives a defensible correct solution, the null-set assumption is broken and the reported accuracy numbers would have to be recomputed.

Watch

Extended reading notes

Core claim

The central claim is that a benchmark built from genuinely unsolvable problems—where 'I don't know' is the only right answer—can reveal whether a machine knows the limits of its own knowledge, and that current LLMs largely fail this test. The paper reports that across twelve models, the best scores fell between 62% and 68% accuracy, meaning that in roughly a third of cases the models exchanged a correct abstention for a confident but wrong distractor. The authors describe the difficulty–accuracy pattern as an inverse relationship, with GPT-4 showing higher rates of uncertainty acknowledgment on more challenging problems (35.8%) than on simpler ones (20.0%). Their interpretation is that models are more inclined to speculate when a problem appears easy, and they recommend that uncertainty recognition be included as a core component of any practical AGI evaluation.

Load-bearing premise

The benchmark's validity hinges on every one of the 675 questions being genuinely unsolvable, so that 'I don't know' is the only defensible correct answer and a model that always selects it would score perfectly regardless of its reasoning.

Editorial extensions

If this is right

  • If LLMs are to be trusted in high-stakes roles, they need explicit abstention mechanisms; this test shows they cannot be relied on to say 'I don't know' spontaneously.
  • Current AI benchmarks that reward only correct answers may conceal a systematic failure mode: confident guessing on problems the model cannot solve.
  • The difficulty–accuracy relationship suggests that calibration training should target easy-looking problems, where models are most overconfident.
  • The dataset offers a moving benchmark: as models improve, their accuracy on the impossible test should rise, giving a concrete—if narrow—measure of progress toward AGI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The finding implies a testable design principle: training a model to abstain on deliberately impossible questions may also improve its calibration on ordinary difficult questions, though the paper does not test this transfer.
  • A natural extension would be to mix impossible questions with merely very hard but solvable ones in a single evaluation, to see whether models can learn to distinguish 'unsolved' from 'unanswerable.'
  • If the relationship between perceived difficulty and overconfidence holds generally, it suggests that a model's tendency to guess is driven more by the surface appearance of tractability than by its internal uncertainty—an interpretation the paper hints at but does not prove.
  • The dataset's reliance on Wikipedia's lists means it will decay as open problems get solved; keeping the benchmark alive would require periodic expert review to add newly impossible or still-open problems.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces a dataset of 675 supposedly unsolvable questions, organized into 49 categories, with 'I don't know' as the only correct answer. Twelve open-source and three closed-source LLMs are evaluated on their accuracy in selecting that option. The authors report that the best models score in the 62-68% range and claim an inverse relationship between problem difficulty and accuracy, with GPT-4 more likely to acknowledge uncertainty on harder problems (35.8%) than on easier ones (20.0%). The dataset is released on GitHub. The paper interprets these findings as evidence about LLM epistemic humility and as a contribution to AGI evaluation.

Significance. If the central claims held, the dataset would be a useful resource for studying model calibration about unknowability, and the inverse-difficulty result would be an intriguing behavioral observation. The paper ships a reproducible prompt structure, a public dataset, and a clear single-answer scoring rule, which are commendable. However, as evaluated, the headline metric conflates response bias with calibrated uncertainty, the difficulty axis is generated by the very model family under test, and the direction of the difficulty-accuracy relationship is described inconsistently. These issues are load-bearing and preclude interpreting the results as evidence about epistemic humility.

major comments (4)
  1. [Evaluation and Scoring (page 7); Tables 1 and 5] The scoring design is degenerate: because 'I do not know' is the correct answer for all 675 items, a trivial policy that always selects that option scores 100%, yet the best models are reported at 62-68%. The reported 'accuracy' is therefore simply the model's base rate of choosing that option, not a measure of calibrated uncertainty. The paper never reports the always-IDK baseline, nor does it include any solvable control questions or a penalty for over-acknowledgment, so the central metric cannot separate genuine recognition of unsolvability from a response bias. This affects every result in Figures 1-4 and the abstract's headline percentages.
  2. [Dataset Development (Methods, 'To enhance the interpretability...') ] The difficulty rankings that support the inverse-difficulty claim were generated by GPT-4o during the reformulation process, and the tested models include GPT-4. This is circular: the 'relationship' is between GPT-4's answer choices and GPT-4o's labels, with no independent ground truth. The paper should validate the difficulty labels against human raters or another model family and control for confounds such as question length, domain, and phrasing, which may correlate with both the labels and the probability of selecting 'I don't know.'
  3. [Abstract and Results (Figures 1-3)] The paper states the central claim as an 'inverse relationship between problem difficulty and model accuracy,' but the example given in the abstract (higher rates on more challenging problems, 35.8%, versus simpler ones, 20.0%) and the Results section ('examination accuracy increases with problem difficulty') describe a positive relationship. The direction of the claimed effect must be defined and stated consistently; otherwise the headline finding is ambiguous.
  4. [Dataset Development and Tables 1, 3, 4] The premise that every question is 'fundamentally unsolvable' and that 'I don't know' is the only correct answer is not externally validated. For example, Table 1 asks to construct a muon collider, which is a hard engineering challenge but not a logical impossibility, and Table 3 asks to confirm Legendre's conjecture, which is an open problem rather than a demonstrably unsolvable one. The paper should provide per-question or per-category justification of unsolvability, or at least acknowledge that the dataset contains a mix of unsolved and unsolvable problems, because the validity of the ground truth is load-bearing for the accuracy metric.
minor comments (4)
  1. [Introduction and Table 2] The text says 'twelve state-of-the-art LLMs' and later describes three closed models, but Table 2 lists twelve open-source models. The total number of evaluated models and the naming of the closed models (e.g., exact versions of Claude, GPT, Gemini) should be stated explicitly.
  2. [Figures 1, 2, and 3] The figure captions are telegraphic and omit axis labels, legends, and model names. For example, Figure 3 lacks a clear definition of the y-axis (accuracy vs. rate of 'I don't know' selections) and the x-axis (difficulty categories). Please clarify the figures so the reader can map the results to the text.
  3. [References [26]] The GitHub repository link in reference [26] is unlikely to be stable and contains three separate URLs; please deposit the dataset in a permanent archive (e.g., Zenodo) and cite a versioned DOI.
  4. [Results] No error bars, standard deviations, or statistical tests are reported, and there is no indication of the number of runs per model. Given that the claimed difficulty effect is small (20.0% vs. 35.8%), the authors should report confidence intervals or bootstrapped estimates for at least the headline comparisons.

Circularity Check

1 steps flagged · score 4.0 of 10

The benchmark's headline 'accuracy' is, by construction, the rate of selecting the 'I don't know' option, and the paper renames this single rate as accuracy, epistemic humility, and resistance to hallucination.

  1. renaming known result [Introduction, 'Novel Framework Motivations' section: 'By leveraging a randomized multiple-choice structure...']
    "By leveraging a randomized multiple -choice structure that includes a deliberate “I don’t know” option [26], the framework quantifies AGI’s performance in terms of accuracy, epistemic humility, and resistance to test shenanigans and hallucination."

    The scoring design fixes 'I don't know' as the unique correct answer for every item: 'By design, the correct answer to all the questions in the test must be either “humans do not know that” or “it is currently impossible to solve that.”' Therefore a model's accuracy is, by construction, its rate of choosing the IDK option. The three headline constructs—accuracy, epistemic humility, and resistance to hallucination—are all operationalized by the same binary choice, so claims like 'the best models scored in 62-68% accuracy ranges for admitting the problem solution was unknown' reduce to the base rate of IDK selection. This is a renamed response rate presented as a multi-faceted evaluation, not an independent derivation of ability.

full rationale

The paper's main empirical numbers (62-68% accuracy, inverse difficulty-accuracy trend) are direct reports of how often models chose the pre-specified 'I don't know' answer; the dataset is fully specified and the questions come from Wikipedia's unsolved-problem lists, so the self-citation of the authors' GitHub dataset [26] is not load-bearing. The circularity is limited to the framing: 'accuracy,' 'epistemic humility,' and 'resistance to hallucination' are all the same measured variable, so the paper does not provide independent evidence for the distinct constructs it claims to quantify. The GPT-4o-generated difficulty rankings are a methodological validity concern—the difficulty axis comes from a sibling model of the tested GPT-4—but they are not a formal reduction of the result to its inputs. Overall, this is a partial, definitional circularity in metric interpretation rather than a fully forced derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its central measure is the rate at which models select a fixed 'I don't know' option, which depends entirely on the authors' labeling of questions as unsolvable and on the generated distractor set.

assumptions (4)
  • domain assumption All 675 problems are genuinely unsolvable or unknowable
    The dataset is compiled from Wikipedia lists of unsolved problems; 'unsolvability' is not independently verified, and some listed problems may have been solved or are ill-posed. This assumption is load-bearing because the correct answer is defined as 'I don't know'.
  • domain assumption Selecting 'I don't know' in the MCQ indicates epistemic humility rather than a trivial strategy
    A model that always picks 'I don't know' gets 100% accuracy, so the metric conflates genuine uncertainty with task exploitation. The paper does not control for this.
  • ad hoc to paper The GPT-4o difficulty labels are valid measures of problem difficulty
    Difficulty labels were generated by GPT-4o during reformulation and then used to analyze GPT-4's accuracy, introducing a self-referential difficulty axis.
  • domain assumption The five-option MCQ with random placement of 'I don't know' is a fair probe of model behavior
    No baseline such as random guessing (20%) or an 'always say I don't know' policy is reported, so above-chance scores could reflect recognition of the phrase rather than understanding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz." pith.science (2026). https://pith.science/paper/VJQA6Y47

@misc{pith2026241114486,
  author       = {Pith},
  title        = {Pith review of: The Impossible Test: A 2024 Unsolvable Dataset and A Chance for an AGI Quiz},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJQA6Y47}},
  note         = {Machine review of arXiv:2411.14486}
}
read the original abstract

This research introduces a novel evaluation framework designed to assess large language models' (LLMs) ability to acknowledge uncertainty on 675 fundamentally unsolvable problems. Using a curated dataset of graduate-level grand challenge questions with intentionally unknowable answers, we evaluated twelve state-of-the-art LLMs, including both open and closed-source models, on their propensity to admit ignorance rather than generate plausible but incorrect responses. The best models scored in 62-68% accuracy ranges for admitting the problem solution was unknown in fields ranging from biology to philosophy and mathematics. We observed an inverse relationship between problem difficulty and model accuracy, with GPT-4 demonstrating higher rates of uncertainty acknowledgment on more challenging problems (35.8%) compared to simpler ones (20.0%). This pattern indicates that models may be more prone to generate speculative answers when problems appear more tractable. The study also revealed significant variations across problem categories, with models showing difficulty in acknowledging uncertainty in invention and NP-hard problems while performing relatively better on philosophical and psychological challenges. These results contribute to the growing body of research on artificial general intelligence (AGI) assessment by highlighting the importance of uncertainty recognition as a critical component of future machine intelligence evaluation. This impossibility test thus extends previous theoretical frameworks for universal intelligence testing by providing empirical evidence of current limitations in LLMs' ability to recognize their own knowledge boundaries, suggesting new directions for improving model training architectures and evaluation approaches.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 52 canonical work pages

  1. [1]

    Cook, J. (2024), OpenAI’s 5 Levels Of ‘Super AI’ (AGI To Outperform Human Capability) , Forbes Magazine, https://www.forbes.com/sites/jodiecook/2024/07/16/openais-5-levels-of-super-ai-agi-to- outperform-human-capability/

    work page 2024

  2. [2]

    Kurshan, E. (2023). Systematic AI Approach for AGI: Addressing Alignment, Energy, and AGI Grand Challenges. arXiv preprint arXiv:2310.15274

    work page Pith review arXiv 2023

  3. [3]

    Landgrebe, J., & Smith, B. (2019). There is no artificial general intelligence. arXiv preprint arXiv:1906.05833

    work page Pith review arXiv 2019

  4. [4]

    Chang, E. Y. (2024). Unlocking the Wisdom of Large Language Models: An Introduction to The Path to Artificial General Intelligence. arXiv preprint arXiv:2409.01007

    work page Pith review arXiv 2024

  5. [5]

    Evaluating the Potential of Leading Large Language Models in Reasoning Biology Questions

    Gong, X., Holmes, J., Li, Y., Liu, Z., Gan, Q., Wu, Z., ... & Yan, Y. (2023). Evaluating the potential of leading large language models in reasoning biology questions. arXiv preprint arXiv:2311.07582

    work page Pith review arXiv 2023

  6. [6]

    Akpan, M. (2024). Have We Reached AGI? Comparing ChatGPT, Claude, and Gemini to Human Literacy and Education Benchmarks. arXiv preprint arXiv:2407.09573

    work page Pith review arXiv 2024

  7. [7]

    (2024, August)

    Wang, Z. (2024, August). Causalbench: A comprehensive benchmark for evaluating causal reasoning capabilities of large language models. In Proceedings of the 10th SIGHAN Workshop on Chinese Language Processing (SIGHAN-10) (pp. 143-151)

    work page 2024

  8. [8]

    Rohrer, B. (2010). Accelerating progress in Artificial General Intelligence: Choosing a benchmark for natural world interaction. Journal of Artificial General Intelligence, 2(1), 1 -28

    work page 2010

Show all 61 references
  1. [9]

    & Zhang, Y

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., ... & Zhang, Y. (2023). Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712

    2023 arXiv

  2. [10]

    Fjelland, R. (2020). Why general artificial intelligence will not be realized. Humanities and Social Sciences Communications, 7(1), 1-9

    2020

  3. [11]

    Ilić, D., & Gignac, G. E. (2024). Evidence of interrelated cognitive-like capabilities in large language models: Indications of artificial general intelligence or achievement? Intelligence, 106, 101858

    2024

  4. [12]

    Hernández-Orallo, J., & Dowe, D. L. (2010). Measuring universal intelligence: Towards an anytime intelligence test. Artificial Intelligence, 174(18), 1508-1539

    2010

  5. [13]

    Zhai, X., Nyaaba, M., & Ma, W. (2024). Can generative AI and ChatGPT outperform humans on cognitive - demanding problem-solving tasks in science? Science & Education, 1-22

    2024

  6. [14]

    Williams, A. E. (2023). Has OpenAI achieved artificial general intelligence in ChatGPT. Artificial Intelligence and Applications, February 1-15

    2023

  7. [15]

    J., Thompson, J., Baber, C., Stanton, N

    McLean, S., Read, G. J., Thompson, J., Baber, C., Stanton, N. A., & Salmon, P. M. (2023). The risks associated with Artificial General Intelligence: A systematic review. Journal of Experimental & Theoretical Artificial Intelligence, 35(5), 649-663

    2023

  8. [16]

    A., Faisal, S., Akbar, A., Haider, U., & Hajjami, S

    Fahad, M., Basri, T., Hamza, M. A., Faisal, S., Akbar, A., Haider, U., & Hajjami, S. E. (2024). The Benefits and Risks of Artificial General Intelligence (AGI). In Artificial General Intelligence (AGI) Security: Smart Applications and Sustainable Technologies (pp. 27-52). Sing...

    2024

  9. [17]

    Bringsjord, S., & Licato, J. (2012). Psychometric artificial general intelligence: the Piaget -MacGuyver room. In Theoretical foundations of artificial general intelligence (pp. 25 -48). Paris: Atlantis Press

    2012

  10. [18]

    Calais, P., Franco, G., Nikas, T., Tang, Z., Crovella, M., Meira Jr, W., & Terzi, E. (2024). Beyond accuracy: understanding the performance of LLMs on exams designed for humans

    2024

  11. [19]

    Williams, C. V. (2024). Bracing for Impact: Revising Legal Writing Assessments Ahead of the Collision of Generative AI and the NextGen Bar Exam. Legal Writing: J. Legal Writing Inst., 28, 1

    2024

  12. [20]

    (2024) Google DeepMind CEO on AGI, OpenAI and Beyond – MWC 2024, AI Business, https://aibusiness.com/nlp/google-deepmind-ceo-on-agi-openai-and-beyond-mwc-2024

    Wodecki, B., Yao, D. (2024) Google DeepMind CEO on AGI, OpenAI and Beyond – MWC 2024, AI Business, https://aibusiness.com/nlp/google-deepmind-ceo-on-agi-openai-and-beyond-mwc-2024

    2024

  13. [21]

    Sam Altman: First question for AGI | Lex Fridman Podcast, March 19, 2024, https://www.youtube.com/watch?v=hnldbcH2UV4

    Altman, S. Sam Altman: First question for AGI | Lex Fridman Podcast, March 19, 2024, https://www.youtube.com/watch?v=hnldbcH2UV4

    2024

  14. [22]

    (2023), Google DeepMind’s Six Level of AGI, AI Business, https://aibusiness.com/ml/what- exactly-is-artificial-general-intelligence-ask-deepmind-

    Brodsky, S. (2023), Google DeepMind’s Six Level of AGI, AI Business, https://aibusiness.com/ml/what- exactly-is-artificial-general-intelligence-ask-deepmind-

    2023

  15. [23]

    (2024), Sam Altman and Mike Sievert Fireside Chat at T-Mobile Capital Markets Day 2024, https://www.youtube.com/watch?v=0sQe4iF3aUM

    Sievert, M., Altman, S. (2024), Sam Altman and Mike Sievert Fireside Chat at T-Mobile Capital Markets Day 2024, https://www.youtube.com/watch?v=0sQe4iF3aUM

    2024

  16. [24]

    (2023), How Generative AI Can Augment Human Creativity, Harvard Business Review, https://hbr.org/2023/07/how-generative-ai-can-augment-human- creativity

    Eapen, T., Finkenstadt, D., Folk, J., Venkataswamy, L. (2023), How Generative AI Can Augment Human Creativity, Harvard Business Review, https://hbr.org/2023/07/how-generative-ai-can-augment-human- creativity

    2023

  17. [25]

    (2024), Creativity Under Turing Test Constraints, AGI Leap Summit, https://www.youtube.com/live/P5m4X_yHLMo?t=3845s

    Noever, D. (2024), Creativity Under Turing Test Constraints, AGI Leap Summit, https://www.youtube.com/live/P5m4X_yHLMo?t=3845s

    2024

  18. [26]

    Noever, D., McKee, F. (2024), https://github.com/reveondivad/certify/blob/main/claude-3p5-sonnet.csv https://github.com/reveondivad/certify/blob/main/gpt-4.csv https://github.com/reveondivad/certify/blob/main/impossible_test.csv

    2024

  19. [27]

    Wikipedia contributors. (n.d.). Lists of unsolved problems. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/Lists_of_unsolved_problems (accessed November 2024)

    2024

  20. [28]

    Wikipedia contributors. (n.d.). List of unsolved problems in mathematics. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_mathematics (accessed November 2024)

    2024

  21. [29]

    Wikipedia contributors. (n.d.). List of unsolved problems in physics. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_physics (accessed November 2024)

    2024

  22. [30]

    Wikipedia contributors. (n.d.). List of unsolved problems in neuroscience. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_neuroscience (accessed November 2024)

    2024

  23. [31]

    Wikipedia contributors. (n.d.). Unsolved problems in medicine. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/Unsolved_problems_in_medicine (accessed November 2024)

    2024

  24. [32]

    Wikipedia contributors. (n.d.). Unsolved problems in biology. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_biology (accessed November 2024)

    2024

  25. [33]

    Wikipedia contributors. (n.d.). List of unsolved problems in statistics. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_statistics (accessed November 2024)

    2024

  26. [34]

    Wikipedia contributors. (n.d.). List of unsolved problems in computer science. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_computer_science (accessed November 2024)

    2024

  27. [35]

    Wikipedia contributors. (n.d.). List of NP-complete problems. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_NP-complete_problems (accessed November 2024)

    2024

  28. [36]

    Wikipedia contributors. (n.d.). List of philosophical problems. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_philosophical_problems (accessed November 2024)

    2024

  29. [37]

    Wikipedia contributors. (n.d.). List of paradoxes. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_paradoxes (accessed November 2024)

    2024

  30. [38]

    Wikipedia contributors. (n.d.). List of undecidable problems. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_undecidable_problems (accessed November 2024)

    2024

  31. [39]

    Wikipedia contributors. (n.d.). List of unsolved problems in fair division. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_fair_division (accessed November 2024)

    2024

  32. [40]

    Wikipedia contributors. (n.d.). List of unsolved problems in economics. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_economics (accessed November 2024)

    2024

  33. [41]

    Wikipedia contributors. (n.d.). List of ciphertexts. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_ciphertexts (accessed November 2024)

    2024

  34. [42]

    Wikipedia contributors. (n.d.). Undeciphered writing systems. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/Undeciphered_writing_systems (accessed November 2024)

    2024

  35. [43]

    Wikipedia contributors. (n.d.). List of hypothetical technologies. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_hypothetical_technologies (accessed November 2024)

    2024

  36. [44]

    Wikipedia contributors. (n.d.). List of emerging technologies. In Wikipedia, The Free Encyclopedia. Retrieved from https://en.wikipedia.org/wiki/List_of_emerging_technologies (accessed November 2024)

    2024

  37. [45]

    Center for AI Safety (2024), Humanity's Last Exam, https://www.safe.ai/blog/humanitys-last-exam

    2024

  38. [46]

    & Hassabis, D

    Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., ... & Hassabis, D. (2021). Highly accurate protein structure prediction with AlphaFold. nature, 596(7873), 583 -589

    2021

  39. [47]

    Hossain, M. (2014). Breakthroughs with competition-based innovation: the X Prize Foundation. Journal of Organization Design, 3(3), 30-36

    2014

  40. [48]

    Vectara (2024), Hallucination Leaderboard, https://github.com/vectara/hallucination-leaderboard

    2024

  41. [49]

    & Shi, S

    Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., ... & Shi, S. (2023). Siren's song in the AI ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219

    2023 arXiv

  42. [50]

    Bastounis, A., Campodonico, P., van der Schaar, M., Adcock, B., & Hansen, A. C. (2024). On the consistent reasoning paradox of intelligence and optimal trust in AI: The power of 'I don't know'. arXiv preprint arXiv:2408.02357

    2024 arXiv

  43. [51]

    I don’t know

    Shashikumar, S. P., Wardi, G., Malhotra, A., & Nemati, S. (2021). Artificial intelligence sepsis prediction algorithm learns to say, “I don’t know”. NPJ digital medicine, 4(1), 134

    2021

  44. [52]

    Borji, A. (2023). A categorical archive of chatgpt failures. arXiv preprint arXiv:2302.03494

    2023 arXiv

  45. [53]

    Balloccu, S., Schmidtová, P., Lango, M., & Dušek, O. (2024). Leak, cheat, repeat: Data contamination and evaluation malpractices in closed source LLMs. arXiv preprint arXiv:2402.03927

    2024 arXiv

  46. [54]

    K., Zhao, D., & Bing, L

    Zhao, R., Zhang, W., Chia, Y. K., Zhao, D., & Bing, L. (2024). Auto Arena of LLMs: Automating LLM Evaluations with Agent Peer-battles and Committee Discussions. arXiv preprint arXiv:2405.20267

    2024 arXiv

  47. [55]

    Instructor (2024), Python Package Index, https://pypi.org/project/instructor/

    2024

  48. [56]

    F., Frazier, P., Oishi, S., & Kaler, M

    Steger, M. F., Frazier, P., Oishi, S., & Kaler, M. (2006). The meaning in life questionnaire: assessing the presence of and search for meaning in life. Journal of counseling psychology, 53(1), 80

    2006

  49. [57]

    Li, W., Li, L., Xiang, T., Liu, X., Deng, W., & Garcia, N. (2024). Can multiple -choice questions really be useful in detecting the abilities of LLMs? arXiv preprint arXiv:2403.17752

    2024 arXiv

  50. [58]

    M., & Shen, Z

    Myrzakhan, A., Bsharat, S. M., & Shen, Z. (2024). Open-LLM-Leaderboard: From Multi-choice to Open- style Questions for LLMs Evaluation, Benchmark, and Arena. arXiv preprint arXiv:2406.07545

    2024 arXiv

  51. [59]

    Plaat, A., Wong, A., Verberne, S., Broekens, J., van Stein, N., & Back, T. (2024). Reasoning with large language models, a survey. arXiv preprint arXiv:2407.11511

    2024

  52. [60]

    Turing, A. M. (1954). Solvable and unsolvable problems. London: Penguin Books

    1954

  53. [61]

    C., & Barrington, D

    Sindall, R. C., & Barrington, D. J. (2020). Fail fast, fail forward, fail openly: the need to share failures in development. Journal of Trial & Error, 1(1), 6-8. APPENDIX A: Distribution of Multiple-Choice Questions (MCQ) Distractors APPENDIX B: Defining Distractor Types with ...

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.