Pith. sign in

REVIEW 5 major objections 3 minor 70 references

UQ: Assessing Language Models on Unsolved Questions

T0 review · 5 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that evaluating language models on genuinely unsolved real-world questions yields a benchmark that is both hard and realistic, and that the top model still fails most of it, passing only 15%.

desk verdict A genuinely new benchmarking paradigm on unsolved questions, but the 15% pass rate needs calibrated validators before the central claim is established. read the letter →

arxiv 2508.17580 v1 pith:3IC26KLM submitted 2025-08-25 cs.CL cs.AIcs.LG

Fan Nie , Ken Ziyu Liu , Zihao Wang , Rui Sun , Wei Liu , Weijia Shi , Huaxiu Yao , Linjun Zhang
show 6 more authors
Andrew Y. Ng James Zou Sanmi Koyejo Yejin Choi Percy Liang Niklas Muennighoff
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords unsolvedquestionslanguagemodelevaluationbenchmarkgenerator-validatorgapStackExchangeasynchronouscommunityverificationfrontiermodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes evaluating language models not on exam-style benchmarks with known answers but on genuinely unanswered questions drawn from real human askers. It introduces UQ, a 500-question testbed collected from Stack Exchange and spanning CS theory, mathematics, sci-fi, and history, together with a validation protocol that combines generated candidate answers, LLM judges, and human review. Its central finding is that the strongest model passes UQ-validation on only 15% of questions, while preliminary human verification has already confirmed some correct answers among the passes. If the paradigm works, it gives the field a benchmark whose difficulty cannot be gamed by training-data exposure and whose solutions carry direct real-world value.

What carries the argument

The load-bearing mechanism is the generator-validator gap: candidate answers are produced by a generator model and then screened by separate validator models, composed into compound strategies, so that the checkers do not simply share the generator's blind spots and can pre-select candidates for expert human review. Around this sit three components: UQ-Dataset, the filtered collection pipeline that turns raw unsolved Stack Exchange posts into well-defined hard questions; UQ-Validators, the compound validation strategies that supply the evaluation signal when no ground-truth answer exists; and UQ-Platform, an open platform for asynchronous expert verification. The 15% pass rate is the output of the validator stage, with human verification reserved for the surviving candidates.

What would settle it

Collect a cohort of UQ questions that later receive human-accepted answers on the original sites, and score validator-passing model answers on those questions: if the pass rate on later-solved questions is no higher than on questions that remain unsolved, the validator signal is not tracking correctness.

Watch

Extended reading notes

Core claim

The central claim is that unsolved real-world questions form a viable evaluation paradigm that resolves the difficulty-realism tension: they are hard by nature and realistic because they arise from people actually seeking answers. Concretely, the paper curates 500 unsolved Stack Exchange questions, filters them through rule-based checks, LLM judges, and human review to ensure they are well-defined and difficult, and evaluates models asynchronously with validator-assisted screening followed by community verification. The headline result is that the top model passes UQ-validation on only 15% of questions, and early human verification has found correct answers among those that passed. The authors frame this as a path for assessing frontier models on open-ended challenges where success pushes the frontier of human knowledge.

Load-bearing premise

The benchmark assumes that an unsolved Stack Exchange question is well-defined and answerable, and that a candidate answer passing the validator screening is actually correct rather than merely persuasive.

Editorial extensions

If this is right

  • If the paradigm holds, benchmark difficulty can be sourced from human ignorance rather than constructed by test designers, so future models cannot memorize their way to high scores.
  • A model that solves an UQ question produces an answer that is already known to be wanted by a real asker, so progress on the benchmark translates into tangible utility.
  • The validation pipeline, rather than a static answer key, becomes the scoring instrument; as validators improve, the same question set can be re-scored asynchronously.
  • The 15% pass rate becomes a baseline for measuring whether future frontier models are actually expanding the space of answerable questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If UQ were scaled up with automated tracking of which validator-passing answers later receive community acceptance, the benchmark could double as a live measure of whether validator quality tracks true correctness; this is an extension the paper leaves implicit.
  • The same generator-validator gap mechanism could be ported to domains without a built-in community, such as internal enterprise question queues, where 'unsolved' status is defined by the absence of a trusted answer.
  • A testable extension is to compare model pass rates on questions that later receive human-written accepted answers against those that remain unsolved: if the rates do not differ, the validator signal is measuring plausibility rather than correctness.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper proposes a new evaluation paradigm, UQ, in which language models are assessed on 500 unsolved questions curated from Stack Exchange. The authors contribute three artifacts: UQ-Dataset and its collection pipeline (rule-based filters, LLM judges, and human review), UQ-Validators (compound validation strategies built on the generator-validator gap), and UQ-Platform for asynchronous community verification. The central empirical claim is that the top model passes UQ-validation on only about 15% of questions, with preliminary human verification confirming some of those passes as correct answers. The paper argues that because the questions are unsolved and naturally arising, the benchmark is 'difficult and realistic by construction' and offers a path toward evaluating frontier models on open-ended, real-world problems.

Significance. If the claims are established, UQ would be a genuinely useful complement to static benchmarks: it has ecological validity, an asynchronous community-verification mechanism, and a concrete public artifact. The paper is also commendable for releasing the dataset and platform, and for making a falsifiable quantitative claim (the 15% pass rate) that can be re-measured. However, the significance currently rests on an unvalidated proxy for correctness: since the questions are unsolved, there is no ground truth against which the validator precision can be checked, and the reported pass rate may partly reflect agreement with the validator rather than model ability. The strength of the contribution therefore depends on additional calibration evidence that is not yet reported.

major comments (5)
  1. [Abstract] The headline result, 'The top model passes UQ-validation on only 15% of questions,' is reported without any uncertainty estimate or statistical framing. For a 500-question testbed, even a simple 95% confidence interval would be informative, and the manuscript should also report the raw counts of passes, fails, and abstentions. Without this, readers cannot distinguish a true 15% ability level from a noisy estimate.
  2. [UQ-Validators and Abstract] Validator precision is uncalibrated, and this is load-bearing. Because every UQ question is unsolved by construction, 'passing UQ-validation' is defined by the paper's own compound validator strategies; there is no external ground truth. The abstract states that 'preliminary human verification has already identified correct answers among those that passed,' but that is existence evidence only. The paper needs a blind, independent review of a random sample of validator passes and validator fails, with inter-annotator agreement reported, to estimate precision and recall. Without such calibration, the 15% figure may measure agreement with the validator rather than genuine problem-solving ability.
  3. [Abstract / Contribution (1)] The claim that UQ is 'difficult and realistic by construction' is an assertion, not a measured property. The curation pipeline uses rule-based filters, LLM judges, and human review to ensure questions are 'well-defined and difficult,' but no validation statistics are reported for these filters (e.g., agreement between LLM-judge screening and human review, or pass-rate comparisons against existing benchmarks). Since the curation filters are themselves LLM-judge outputs, the 'difficult' claim inherits the same circularity as the validator: it needs an independent anchor such as human difficulty ratings or a comparison set of solved questions with known answer distributions.
  4. [Abstract / generator-validator gap] The generator-validator gap is the central methodological assumption of UQ-Validators, but the paper provides no evidence that this gap is a reliable correctness signal on unsolved questions. If the generator and validator share failure modes (sycophancy, style preference, hallucination, or reward hacking), a wrong answer can pass and a right answer can fail. The manuscript should specify each validator strategy, report the agreement and disagreement rates among them, and show that disagreement correlates with human judgments on a calibration set drawn from questions with known or later-confirmed answers.
  5. [Results / comparison baselines] The paper does not report how the same models perform on existing solved benchmarks under the same evaluation protocol. Without such a baseline, the statement that the 15% pass rate demonstrates 'difficulty' is not quantified: a low pass rate is equally consistent with validator over-strictness or with questions that are unanswerable or ill-posed despite curation. At minimum, the authors should report pass rates on a matched set of solved Stack Exchange questions processed through the same validator pipeline.
minor comments (3)
  1. [Throughout] The manuscript contains many typographical and rendering artifacts in the provided text, including garbled characters and repeated fragments; the authors should carefully proofread the final version.
  2. [Tables] Several tables appear without clear captions or explicit units; for example, the pass-rate tables should state the number of questions per category and the number of validator strategies applied.
  3. [Related work] The paper would benefit from an explicit comparison to prior benchmark-construction efforts that use human-LLM verification pipelines, such as human feedback for difficult evaluations, to clarify the novelty of the generator-validator gap approach.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: UQ's central claims are empirical benchmark-construction claims whose validity depends on external human verification, not on input-equivalent derivations.

full rationale

The paper's core claim is that assessing models on unsolved questions yields a difficult and realistic evaluation paradigm, and that the top model passes its UQ-validation on only 15% of questions. These are empirical claims about a constructed testbed, not mathematical derivations from a set of first principles, so the standard circularity patterns do not directly apply. The phrase 'UQ is difficult and realistic by construction' is explicit about the construction process; it does not disguise an input as a derived result. Likewise, 'passes UQ-validation' is openly an operational metric defined by the paper's compound validation strategies, and the paper does not claim this metric is logically equivalent to ground-truth correctness. The independent signal comes from 'preliminary human verification,' which the paper cites as evidence that some validator-positive answers are genuinely correct. That evidence is preliminary and not a full precision calibration, but the absence of a complete ground-truth set raises validity concerns rather than circularity under the hard rules: no equation is shown to reduce to its own inputs, no fitted parameter is renamed as a prediction, and no load-bearing self-citation chain is invoked. The 15% figure should be interpreted as agreement with UQ-Validators, and its scientific interpretation depends on validator precision, but that is a correctness-risk consideration, not a circularity finding. Overall, the paper is self-contained relative to the claims it actually makes, and the potential weakness is external calibration of the validator, not a definitional or self-referential derivation.

Assumptions & free parameters 2 free parameters · 4 assumptions · 3 invented entities

The central benchmark depends on the curation pipeline and the validators. Exact threshold values, judge prompts, and validator reliability metrics are not visible in the abstract, so the ledger entries carry null fitted values. The listed assumptions are load-bearing because there is no external answer key for unsolved questions.

free parameters (2)
  • Curated-question filter thresholds (rule-based and LLM-judge criteria)
    The abstract says the pipeline combines rule-based filters, LLM judges, and human review to ensure questions are well-defined and difficult, but it does not state the specific thresholds. These hand-tuned choices determine which 500 questions enter the benchmark and therefore shape the 15% pass rate.
  • Validator pass criteria (compound validator agreement thresholds)
    UQ-Validators pre-screen candidate solutions, but the exact criteria that define a 'pass' are not given in the abstract. This directly controls the headline result of 15% of questions passed.
assumptions (4)
  • domain assumption Unsolved Stack Exchange questions, after filtering, are well-defined and answerable.
    The benchmark assumes that a curated unsolved question has a correct or resolvable answer and is not simply abandoned or ill-posed. The abstract states the pipeline ensures well-defined and difficult questions, but gives no independent evidence of answerability.
  • domain assumption A candidate answer that passes validator screening is correct, or at least highly likely correct.
    UQ-Validators provide the evaluation signal for questions without known answers. The abstract says human verification is preliminary and only for some passed questions, so the full 500-question pass rate rests on validator judgments.
  • domain assumption LLM judges can reliably identify well-defined, difficult questions and can validate candidate answers.
    The curation pipeline uses LLM judges for question quality, and the validation strategy uses LLM-based validators. The abstract presents these as tools without reporting their accuracy against an external benchmark.
  • ad hoc to paper The generator-validator gap is a reliable signal on these unsolved questions.
    The validation strategy relies on the idea that judging answers is easier than generating them. If validators are no more reliable than generators for this question set, passes do not indicate correctness.
invented entities (3)
  • UQ-Dataset (500 unsolved Stack Exchange questions) independent evidence
    purpose: Serves as the benchmark/testbed for evaluating language models on real, open-ended questions.
    The dataset is released at https://uq.stanford.edu, so it can be inspected independently. However, the abstract gives no snapshot hash or download details, so the degree of independent reproducibility is unverified.
  • UQ-Validators (compound validation strategies)
    purpose: Provides automated pre-screening and candidate-answer validation for questions that have no known ground truth.
    The validator is a methodological artifact defined in this paper. It has no independent falsifiable handle beyond the preliminary human-verification examples mentioned in the abstract.
  • UQ-Platform (community verification platform) independent evidence
    purpose: Enables experts to collectively verify questions and solutions over time.
    The platform is publicly accessible at the same URL, so its existence and contents can be checked externally, though the abstract does not describe its governance or verification UI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UQ: Assessing Language Models on Unsolved Questions." pith.science (2026). https://pith.science/paper/3IC26KLM

@misc{pith2026250817580,
  author       = {Pith},
  title        = {Pith review of: UQ: Assessing Language Models on Unsolved Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3IC26KLM}},
  note         = {Machine review of arXiv:2508.17580}
}
read the original abstract

Benchmarks shape progress in AI research. A useful benchmark should be both difficult and realistic: questions should challenge frontier models while also reflecting real-world usage. Yet, current paradigms face a difficulty-realism tension: exam-style benchmarks are often made artificially difficult with limited real-world value, while benchmarks based on real user interaction often skew toward easy, high-frequency problems. In this work, we explore a radically different paradigm: assessing models on unsolved questions. Rather than a static benchmark scored once, we curate unsolved questions and evaluate models asynchronously over time with validator-assisted screening and community verification. We introduce UQ, a testbed of 500 challenging, diverse questions sourced from Stack Exchange, spanning topics from CS theory and math to sci-fi and history, probing capabilities including reasoning, factuality, and browsing. UQ is difficult and realistic by construction: unsolved questions are often hard and naturally arise when humans seek answers, thus solving them yields direct real-world value. Our contributions are threefold: (1) UQ-Dataset and its collection pipeline combining rule-based filters, LLM judges, and human review to ensure question quality (e.g., well-defined and difficult); (2) UQ-Validators, compound validation strategies that leverage the generator-validator gap to provide evaluation signals and pre-screen candidate solutions for human review; and (3) UQ-Platform, an open platform where experts collectively verify questions and solutions. The top model passes UQ-validation on only 15% of questions, and preliminary human verification has already identified correct answers among those that passed. UQ charts a path for evaluating frontier models on real-world, open-ended challenges, where success pushes the frontier of human knowledge. We release UQ at https://uq.stanford.edu.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 24 canonical work pages

  1. [1]

    Piqa: Reasoning about physical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence , pages 7432--7439, 2020

    work page 2020

  2. [2]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \'e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021

    arXiv 2021

  3. [3]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning , 2024

    work page 2024

  4. [4]

    Arc prize 2024: Technical report

    Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Arc prize 2024: Technical report. arXiv preprint arXiv:2412.04604 , 2024

    arXiv 2024

  5. [5]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021

    arXiv

  6. [6]

    Chain-of-verification reduces hallucination in large language models

    Shikhar Dhuliawala, Oriol Vinyals, Mikel Artetxe, Michael Auli, Joaquin Quiñonero-Candela, Liam Dugan, Vishrav Chaudhary, Edward Grefenstette, Dzmitry Bahdanau, Sumit Chaudhuri, Wenda Liu, and Amelia Glaese. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495 , 2023

    arXiv 2023

  7. [7]

    Alpacafarm: A simulation framework for methods that learn from human feedback

    Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387 , 2023

    arXiv 2023

  8. [8]

    Are we done with mmlu? arXiv preprint arXiv:2406.04127 , 2024

    Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint arXiv:2406.04127 , 2024

    arXiv 2024

Show all 70 references
  1. [9]

    Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai, 2024

    Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, Olli Järviniemi, Matthew Barnett, Robert Sandler, Matej Vrzala, Jaime Sevilla, Qiuyu Ren, Elizabeth Pratt, L...

    2024

  2. [10]

    Great models think alike and this undermines ai oversight

    Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. Great models think alike and this undermines ai oversight. arXiv preprint arXiv:2502.04313 , 2025

    2025 arXiv

  3. [11]

    Measuring coding challenge competence with apps

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 , 2021

    2021 arXiv

  4. [12]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 , 2020

    2009 arXiv

  5. [13]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 , 2021

    2021 arXiv

  6. [14]

    Math-perturb: Benchmarking llms' math reasoning abilities against hard perturbations

    Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, et al. Math-perturb: Benchmarking llms' math reasoning abilities against hard perturbations. arXiv preprint arXiv:2502.06453 , 2025

    2025 arXiv

  7. [15]

    Exploring and mitigating adversarial manipulation of voting-based leaderboards

    Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Christopher A Choquette-Choo, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Ziyu Liu, et al. Exploring and mitigating adversarial manipulation of voting-based leaderboards. arXiv pr...

    2025 arXiv

  8. [16]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  9. [17]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974 , 2024

    2024 arXiv

  10. [18]

    Lm vs lm: Detecting factual errors in language models using language models

    Saurav Jain, Zhengbao Guo, William W Cohen, and Graham Neubig. Lm vs lm: Detecting factual errors in language models using language models. arXiv preprint arXiv:2212.10511 , 2023

    2023 arXiv

  11. [19]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024

    2024

  12. [20]

    Verdict: A library for scaling judge-time compute

    Nimit Kalra and Leonard Tang. Verdict: A library for scaling judge-time compute. arXiv preprint arXiv:2502.18018 , 2025

    2025

  13. [21]

    Llm-as-an-interviewer: Beyond static testing through dynamic llm evaluation, 2025

    Eunsu Kim, Juyoung Suk, Seungone Kim, Niklas Muennighoff, Dongkwan Kim, and Alice Oh. Llm-as-an-interviewer: Beyond static testing through dynamic llm evaluation, 2025

    2025

  14. [22]

    Prometheus: Inducing fine-grained evaluation capability in language models, 2024

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models, 2024

    2024

  15. [23]

    Prometheus 2: An open source language model specialized in evaluating other language models, 2024

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models, 2024

    2024

  16. [24]

    Scaling evaluation-time compute with reasoning models as process evaluators, 2025

    Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gashteovski, Carolin Lawrence, Julia Hockenmaier, Graham Neubig, and Sean Welleck. Scaling evaluation-time compute with reasoning models as process evaluators, 2025

    2025

  17. [25]

    Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  18. [26]

    The measurement of observer agreement for categorical data

    J Richard Landis and Gary G Koch. The measurement of observer agreement for categorical data. biometrics , pages 159--174, 1977

    1977

  19. [27]

    Flask: Fine-grained language model evaluation based on alignment skill sets

    Seonghyeon Lee, Doyoung Kim, Jamin Hwang, Minjoon Lee, Seokhwan Hwang, and Hannaneh Hajishirzi Kyunghyun Cho Kang Lee. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928 , 2023

    2023 arXiv

  20. [28]

    Holistic evaluation of language models

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 , 2022

    2022 arXiv

  21. [29]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20050 , 2023

    2023 arXiv

  22. [30]

    Wildbench: Benchmarking llms with challenging tasks from real users in the wild

    Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770 , 2024

    2024 arXiv

  23. [31]

    Evaluating verifiability in generative search engines

    Nelson F Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. arXiv preprint arXiv:2304.09848 , 2023

    2023 arXiv

  24. [32]

    The lean 4 theorem prover and programming language

    Leonardo de Moura and Sebastian Ullrich. The lean 4 theorem prover and programming language. In International Conference on Automated Deduction , pages 625--635. Springer, 2021

    2021

  25. [33]

    2024 aime i

    Mathematical Association of America. 2024 aime i. https://artofproblemsolving.com/wiki/index.php/2024_AIME_I, 2024. Accessed: 2025-05-13

    2024

  26. [34]

    List of open problems in sublinear algorithms

    List of Open Problems in Sublinear Algorithms. List of open problems in sublinear algorithms. https://sublinear.info/, 2025

    2025

  27. [35]

    Introducing deep research

    OpenAI. Introducing deep research. Access: 2025-03-15

    2025

  28. [36]

    Introducing OpenAI o1

    OpenAI . Introducing OpenAI o1 . https://openai.com/o1/, September 2024. Accessed: 2025-06-29

    2024

  29. [37]

    Introducing OpenAI o3 and o4‑mini

    OpenAI . Introducing OpenAI o3 and o4‑mini . https://openai.com/index/introducing-o3-and-o4-mini/, April 2025. Accessed: 2025-06-29

    2025

  30. [38]

    Llm evaluators recognize and favor their own generations

    Arjun Panickssery, Samuel Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems , 37:68772--68802, 2024

    2024

  31. [39]

    For better or worse, benchmarks shape a field

    David Patterson. For better or worse, benchmarks shape a field. Communications of the ACM , 55, 2012

    2012

  32. [40]

    Kilt: a benchmark for knowledge intensive language tasks

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, et al. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252 , 2020

    2009 arXiv

  33. [41]

    Humanity's last exam, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, et al. Humanity's last exam, 2025

    2025

  34. [42]

    Squad: 100,000+ questions for machine comprehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 , 2016

    2016 arXiv

  35. [43]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling , 2024

    2024

  36. [44]

    Measurement to meaning: A validity-centered framework for ai evaluation

    Olawale Salaudeen, Anka Reuel, Ahmed Ahmed, Suhana Bedi, Zachary Robertson, Sudharsan Sundar, Ben Domingue, Angelina Wang, and Sanmi Koyejo. Measurement to meaning: A validity-centered framework for ai evaluation. arXiv preprint arXiv:2505.10573 , 2025

    2025 arXiv

  37. [45]

    The leaderboard illusion

    Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D'Souza, Sayash Kapoor, Ahmet \"U st \"u n, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah Smith, et al. The leaderboard illusion. arXiv preprint arXiv:2504.20879 , 2025

    2025 arXiv

  38. [46]

    Stack Exchange

    Stack Exchange . Stack Exchange . https://stackexchange.com/. https://stackexchange.com/

  39. [47]

    Terminal‑Bench : A benchmark for ai agents in terminal environments

    Stanford & Laude Collaborators . Terminal‑Bench : A benchmark for ai agents in terminal environments. https://www.tbench.ai/, 2025. Accessed: 2025-07-15

    2025

  40. [48]

    Stop evaluating ai with human tests, develop principled, ai-specific tests instead

    Tom S \"u hr, Florian E Dorner, Olawale Salaudeen, Augustin Kelava, and Samira Samadi. Stop evaluating ai with human tests, develop principled, ai-specific tests instead. arXiv preprint arXiv:2507.23009 , 2025

    2025 arXiv

  41. [49]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines, 2025

    P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, David Ma, Yuanshe...

    2025

  42. [50]

    Superglue: A stickier benchmark for general-purpose language understanding systems

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537 , 2019

    1905 arXiv

  43. [51]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018

    2018 arXiv

  44. [52]

    Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024

    2024

  45. [53]

    Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization

    Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087 , 2023

    2023 arXiv

  46. [54]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference on Neural Information Pr...

    2024

  47. [55]

    Self-preference bias in llm-as-a-judge

    Koki Wataoka, Tsubasa Takahashi, and Ryokan Ri. Self-preference bias in llm-as-a-judge. arXiv preprint arXiv:2410.21819 , 2024

    2024 arXiv

  48. [56]

    Measuring short-form factuality in large language models

    Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368 , 2024

    2024 arXiv

  49. [57]

    Browsecomp: A simple yet challenging benchmark for browsing agents

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516 , 2025

    2025 arXiv

  50. [58]

    Livebench: A challenging, contamination-free llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314 , 2024

    2024 arXiv

  51. [59]

    Pride and prejudice: Llm amplifies self-bias in self-refinement

    Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li, and William Yang Wang. Pride and prejudice: Llm amplifies self-bias in self-refinement. arXiv preprint arXiv:2402.11436 , 2024

    2024 arXiv

  52. [60]

    Jimenez, Alex L

    John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024

    2024

  53. [61]

    tau -bench: A benchmark for tool-agent-user interaction in real-world domains

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045 , 2024

    2024 arXiv

  54. [62]

    Justice or prejudice? quantifying biases in llm-as-a-judge

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736 , 2024

    arXiv

  55. [63]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019

    1905 arXiv

  56. [64]

    A careful examination of large language model performance on grade school arithmetic

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, et al. A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Processing Systems ...

    2024

  57. [65]

    Challenges in trustworthy human evaluation of chatbots

    Wenting Zhao, Alexander M Rush, and Tanya Goyal. Challenges in trustworthy human evaluation of chatbots. arXiv preprint arXiv:2412.04363 , 2024

    2024 arXiv

  58. [66]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685 , 2023

    2023 arXiv

  59. [67]

    Agieval: A human-centric benchmark for evaluating foundation models

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364 , 2023

    2023 arXiv

  60. [68]

    Lima: Less is more for alignment

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 , 2023

    2023 arXiv

  61. [69]

    Reinforcing general reasoning without verifiers

    Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21493 , 2025

    2025 arXiv

  62. [70]

    Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2025

    Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Ya...

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.