Pith. sign in

REVIEW 2 major objections 2 minor 1 cited by

Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information

T0 review · 2 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large reasoning models that handle well-defined math problems ably nevertheless fail to ask for missing information when problems are underspecified, according to a new evaluation dataset, and the ability to ask is only partially learnable

desk verdict The abstract and the full text are different papers; the evaluation claim is unverifiable from what is submitted. read the letter →

arxiv 2508.11252 v1 pith:UMCJKFGF submitted 2025-08-15 cs.AI cs.CLcs.IR

Youcheng Huang , Bowen Qin , Chen Huang , Duanyu Feng , Xi Yang , Wenqiang Lei This is my paper
classification cs.AIcs.CLcs.IR
keywords largereasoningmodelsclarifyingquestionsunderspecifiedproblemsproactiveinformationseekingbenchmarkdatasetsupervisedfine-tuningmathhallucination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that large reasoning models, despite their strength on fully specified math problems, do not proactively ask for clarification when a problem lacks necessary information. To test this, it introduces a new dataset of two types of incomplete problems across diverse contexts and evaluates current large reasoning models on them. The evaluation reveals a systematic inability to ask for information, along with observable behaviors of overthinking and hallucination. Supervised fine-tuning can induce some asking behavior but does not fully close the gap. If the paper is right, existing benchmarks overstate the readiness of these models for real-world requests, where information is often missing or ambiguous.

What carries the argument

The evaluation is carried by a new dataset of incomplete math problems, organized into two types of underspecification with diverse contexts. This dataset defines the target behavior: when a problem lacks sufficient information, the correct response is to ask for what is missing rather than to solve or guess. It turns the ability to ask for information into a measurable capability separate from problem-solving skill, and provides the basis for evaluating and fine-tuning large reasoning models.

What would settle it

Manually re-annotate the dataset to see whether a large share of problems can actually be solved under a reasonable assumption by human solvers; if so, the reported inability to ask would be a dataset artifact rather than a model deficiency. Alternatively, show one large reasoning model that, when explicitly instructed to ask for missing information, asks informative questions at a human-comparable rate on the same problems; that would confine the paper's inability claim to a particular prompting or training regime.

Watch

Extended reading notes

Core claim

The central claim is that large reasoning models lack the ability to proactively ask for information when presented with problems that do not contain sufficient information to be solved. The paper grounds this claim in a new dataset of two types of incomplete problems with diverse contexts. On this evaluation, the models tend to answer, guess, or reason from unstated assumptions rather than request the missing details, and their responses exhibit overthinking and hallucination. The paper also reports that supervised fine-tuning on such incomplete problems can improve asking behavior to a degree, but the ability remains incomplete. This positions 'asking for information' as a distinct capabil

Load-bearing premise

The evaluation rests on the assumption that the incomplete problems are genuinely underspecified in the way real requests are, and that the scoring rubric recognizes useful clarifying questions rather than rewarding generic filler; the abstract reports no dataset statistics, rubric details, or human baseline, so this premise cannot be checked from the paper's own description.

Editorial extensions

If this is right

  • Current benchmarks that use only well-defined problems overstate how ready large reasoning models are for real user requests, which are frequently incomplete or ambiguous.
  • Proactive asking can be treated as a separate, trainable capability rather than an automatic byproduct of strong reasoning.
  • Supervised fine-tuning on underspecified problems can produce some asking behavior, suggesting the ability is at least partially learnable with the right data.
  • Overthinking and hallucination are concrete failure modes that surface when models face underspecified problems, giving future work specific targets to measure and reduce.
  • Evaluation of intelligent agents should include tasks where the correct action is to request information, not merely to produce an answer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step would be to test whether asking behavior transfers across domains such as code generation, medical diagnosis, or customer support, or whether it remains confined to math word problems unless trained broadly.
  • If supervised fine-tuning only partially teaches asking, a plausible hypothesis is that reinforcement learning with a reward for informative clarifying questions, or an explicit ask-then-solve protocol, would yield larger gains than fine-tuning alone.
  • The reported hallucination behavior suggests a measurable extension: compare the frequency of fabricated constraints or invented assumptions before and after fine-tuning, to see whether asking training reduces hallucination or merely relocates it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The submission consists of an abstract claiming a systematic evaluation of Large Reasoning Models (LRMs) on incomplete mathematics problems, followed by a full text that is actually a nuclear-structure paper, arXiv:2508.11251v2 [nucl-th], 'Low-lying level structures in 162Lu'. The abstract's central claim is that LRMs fail to proactively ask for information when problems lack sufficient information, and that this inability is revealed by a new dataset of two types of incomplete problems. The submitted body text contains no dataset description, no model list, no evaluation protocol, no metrics, no baseline, and no results related to LRMs. Consequently, the central empirical claim and the benchmark on which it rests are entirely unsupported in the submitted manuscript.

Significance. The high-level evaluation goal is timely and potentially important: if LRMs perform well on well-posed math problems but systematically fail to ask clarifying questions on underspecified problems, this would be a meaningful limitation for deployment and a useful benchmark contribution. However, the submitted text provides no methods or results that could support such a claim. No dataset, rubric, model, or evaluation artifact is present to assess. The significance of the claimed finding therefore cannot be evaluated beyond the abstract's assertion.

major comments (2)
  1. [Full Text / overall manuscript] The body of the manuscript is arXiv:2508.11251v2 [nucl-th], 'Low-lying level structures in 162Lu', which is a nuclear-structure study with no content on LRMs, incomplete math problems, dataset construction, or evaluation results. The abstract and the full text are from different papers. The load-bearing premise that a systematic evaluation was performed is therefore entirely unsupported in the submitted text. This is not a local typographical or formatting issue; the evidence for the central claim is absent.
  2. [Abstract] Even if the abstract is considered alone, it does not specify dataset size, model list, prompting protocol, evaluation metrics, or a human/expert baseline. The phrase 'our systematical evaluation of LRMs reveals their inability in proactively asking for information' cannot be checked without the underlying dataset and rubric. In particular, the ground truth for when asking is the correct behavior, and the criteria for what counts as a useful clarifying question, are undefined. Without these details, the reported inability could be an artifact of rubric design; the manuscript provides no way to rule this out.
minor comments (2)
  1. [Abstract] Grammar and wording: 'we proposes' should be 'we propose'; 'systematical' is non-standard (use 'systematic'); 'responding users' requests' is unclear and should be rephrased, e.g., 'in responding to users' requests'.
  2. [Header/Abstract] The arXiv identifier in the abstract context (2508.11252) does not match the header of the supplied full text (arXiv:2508.11251v2 [nucl-th]). This mismatch is consistent with the wrong full text being attached.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity demonstrable; supplied full text is an unrelated nuclear-physics paper, so the abstract's AI-evaluation claim has no derivation chain to inspect.

full rationale

The manuscript as submitted contains an abstract about Large Reasoning Models failing to ask for missing information, but the full text is an unrelated nuclear-structure paper, 'Low-lying level structures in 162Lu' (arXiv:2508.11251v2). There is no dataset, no rubric, no LRM evaluation, and no result connecting the abstract's central claim to any computation in the full text. That is an absence of support, not a demonstrated circularity: no equation, fitted parameter, or self-citation in the supplied text reduces the claimed 'inability to ask' to the authors' own operationalization, so the quote-and-reduction requirement cannot be met. The nuclear-physics content itself is a non-circular empirical application of the well-tested Two Quasiparticle Rotor Model: it takes experimental 1qp proton/neutron energies from neighboring isotopes/isotones as inputs (Eq. 1) and produces new assignments for 162Lu, including the 1.9 min isomer (Jπ=6+, Ex≈160 keV), which is not present in the adopted data and is explicitly flagged as requiring further experimental validation. The confirmations of the tentative (4−) isomer and the 166Lu/168Lu comparisons are validations against external adopted data rather than by-construction equivalences; the adopted values are not used as outputs of the same fit. Under the hard rule prohibiting speculation about author intent and requiring a specific quoted reduction, no circular step can be identified in the supplied text.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The abstract reveals no fitted numbers, so the free-parameter ledger is empty. The load-bearing premises are domain assumptions about what intelligent behavior requires, about the dataset's validity as a test of underspecification, about the benchmark landscape, about contamination, and about supervised fine-tuning as a learnability probe. No new physical or conceptual entities are postulated; the dataset is a measurement instrument, not an entity in the ledger sense. All five assumptions are unverifiable from the abstract alone.

assumptions (5)
  • domain assumption A genuine intelligent agent should ask for information when problems lack it, and this is a separately evaluable ability.
    Quoted in the abstract ('a genuine intelligent agent should not only solve problems... but also be able to ask for information'); it is the value premise that makes question-asking the benchmark's target behavior.
  • domain assumption The proposed dataset is a valid operationalization of incomplete problems with diverse contexts.
    Abstract says 'a new dataset consisting of two types of incomplete problems with diverse contexts'; whether these categories capture genuine under-specification is assumed, not demonstrated in the abstract.
  • domain assumption Existing benchmarks evaluate exclusively well-defined problems, so a gap exists.
    Abstract's opening claim ('evaluation... exclusively on well-defined problems'); the paper's motivation depends on this characterization of the benchmark landscape, which the abstract supports with no citations.
  • domain assumption The evaluated LRMs had no prior exposure to the new benchmark (no contamination).
    Every fresh-benchmark evaluation assumes the test items were not in pretraining data; the abstract does not mention any contamination check.
  • domain assumption Supervised fine-tuning is an appropriate probe for whether the question-asking ability can be learned.
    Abstract: 'highlight the potential and challenges of supervised fine-tuning in learning such ability'; the interpretation of the SFT results depends on this, and the details are absent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information." pith.science (2026). https://pith.science/paper/UMCJKFGF

@misc{pith2026250811252,
  author       = {Pith},
  title        = {Pith review of: Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UMCJKFGF}},
  note         = {Machine review of arXiv:2508.11252}
}
read the original abstract

Large Reasoning Models (LRMs) have demonstrated remarkable problem-solving abilities in mathematics, as evaluated by existing benchmarks exclusively on well-defined problems. However, such evaluation setup constitutes a critical gap, since a genuine intelligent agent should not only solve problems (as a math quiz solver), but also be able~to ask for information when the problems lack sufficient information, enabling proactivity in responding users' requests. To bridge such gap, we proposes a new dataset consisting of two types of incomplete problems with diverse contexts. Based on the dataset, our systematical evaluation of LRMs reveals their inability in proactively asking for information. In addition, we uncover the behaviors related to overthinking and hallucination of LRMs, and highlight the potential and challenges of supervised fine-tuning in learning such ability. We hope to provide new insights in developing LRMs with genuine intelligence, rather than just solving problems.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MIRA-Math: A Benchmark for Minimal Information Requesting and Mathematical Reasoning

    cs.AI 2026-07 conditional novelty 6.0 of 10

    MIRA-Math introduces a 2,310-instance benchmark isolating the ability of LLMs to request a single missing atomic fact needed to solve an underdetermined mathematical problem and then integrate it into an exact answer.

Reference graph

Works this paper leans on

62 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    van Eemeren, Rob Grootendorst, Sally Jackson, Scott Jacobs, Agnes van Rees, Francisca Snoeck Henkemans, Eveline T

    Frans H. van Eemeren, Rob Grootendorst, Sally Jackson, Scott Jacobs, Agnes van Rees, Francisca Snoeck Henkemans, Eveline T. Feteris, Erik C. W. Krabbe, Agnes Verbiest, Charles Arthur Willard, and John Hayden Woods. What is reasoning? what is an argument?*. 2001

    work page 2001

  2. [2]

    Dictionary of philosophy

    Peter Adam Angeles. Dictionary of philosophy. 1981

    work page 1981

  3. [3]

    McCarthy and P.J

    J. McCarthy and P.J. Hayes. Some philosophical problems from the standpoint of artificial intelligence. In Bonnie Lynn Webber and Nils J. Nilsson, editors, Readings in Artificial Intelligence , pages 431--450. Morgan Kaufmann, 1981

    work page 1981

  4. [4]

    Programs with common sense, 1959

    John McCarthy. Programs with common sense, 1959

    work page 1959

  5. [5]

    Newell and H

    A. Newell and H. Simon. The logic theory machine--a complex information processing system. IRE Transactions on Information Theory , 2(3):61--79, 1956

    work page 1956

  6. [6]

    Openai o1 system card

    Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720 , 2024

    arXiv 2024

  7. [7]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 , 2025

    arXiv 2025

  8. [8]

    2024 aime i

    AoPS Online. 2024 aime i. URL: https://artofproblemsolving.com/wiki/index.php/2024\_AIME\_I , 2025

    work page 2024

Show all 62 references
  1. [9]

    Let's verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations , 2024

    2024

  2. [10]

    Omni- MATH : A universal olympiad level mathematic benchmark for large language models

    Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang, Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni- MATH : A universa...

    2025

  3. [11]

    Open r1: A fully open reproduction of deepseek-r1, January 2025

    Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025

    2025

  4. [12]

    Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models

    Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat-Seng Chua. Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models. In Proceedings of the 62nd Annual Meeting of the Ass...

    2024

  5. [13]

    Rethinking conversational agents in the era of llms: Proactivity, non-collaborativity, and beyond

    Yang Deng, Wenqiang Lei, Minlie Huang, and Tat-Seng Chua. Rethinking conversational agents in the era of llms: Proactivity, non-collaborativity, and beyond. In SIGIR-AP , pages 298--301, 2023

    2023

  6. [14]

    Li, Been Kim

    Zi Wang Belinda Z. Li, Been Kim. Questbench: Can llms ask the right question to acquire information in reasoning tasks? arXiv preprint arXiv:2503.22674 , 2025

    2025

  7. [15]

    Reasoning attack: Inducing llm to never-end thinking, 2025

    Kunpeng Ning Jiayu Yao. Reasoning attack: Inducing llm to never-end thinking, 2025. https://github.com/PKU-YuanGroup/Reasoning-Attack/tree/master

    2025

  8. [16]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  9. [17]

    s1: Simple test-time scaling

    Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Cand \`e s, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393 , 2025

    2025 arXiv

  10. [18]

    Bespoke-stratos Labs

    B. Bespoke-stratos Labs. The unreasonable effectiveness of reasoning distillation. https://hf.co/bespokelabs/Bespoke-Stratos-32B, 2025. Accessed: 2025-01-22

    2025

  11. [19]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021

    2021 arXiv

  12. [20]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022

    2022 arXiv

  13. [21]

    Sky-t1: Train your own o1 preview model within \ 450

    NovaSky Team. Sky-t1: Train your own o1 preview model within \ 450. https://novasky-ai.github.io/posts/sky-t1, 2025. Accessed: 2025-01-09

    2025

  14. [22]

    Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284 , 2025

    Haotian Xu, Xing Wu, Weinong Wang, Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, et al. Redstar: Does scaling long-cot data unlock better slow-reasoning systems? arXiv preprint arXiv:2501.11284 , 2025

    2025 arXiv

  15. [23]

    Qwen3: Think deeper, act faster

    Qwen Team. Qwen3: Think deeper, act faster. https://qwenlm.github.io/blog/qwen3/, 2025. Accessed: 2025-04-29

    2025

  16. [24]

    Claude 3.7 sonnet and claude code

    Antropic. Claude 3.7 sonnet and claude code. https://www.anthropic.com/news/claude-3-7-sonnet, 2025. Accessed: 2025-02-25

    2025

  17. [25]

    Grok 3 beta — the age of reasoning agents

    xAI. Grok 3 beta — the age of reasoning agents. URL: https://x.ai/blog/grok-3 , 2025

    2025

  18. [26]

    A survey on llm-as-a-judge

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 , 2024

    2024 arXiv

  19. [27]

    The lessons of developing process reward models in mathematical reasoning

    Zhenru Zhang, Chujie Zheng, Yangzhen Wu, Beichen Zhang, Runji Lin, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. The lessons of developing process reward models in mathematical reasoning. arXiv preprint arXiv:2501.07301 , 2025

    2025 arXiv

  20. [28]

    Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290 , 2025

    2025 arXiv

  21. [29]

    Do not think that much for 2+ 3=? on the overthinking of o1-like llms

    Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187 , 2024

    2024 arXiv

  22. [30]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Informatio...

    2025

  23. [31]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Associat...

    2023

  24. [32]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015

    2015 arXiv

  25. [33]

    Math-Verify: Math Verification Library

    Hynek Kydlíček. Math-Verify: Math Verification Library

  26. [34]

    Openrlhf: An easy-to-use, scalable and high-performance rlhf framework

    Jian Hu, Xibin Wu, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143 , 2024

    2024 arXiv

  27. [35]

    Interference and inhibition in cognition and behavior: Unifying themes for educational psychology

    Frank N Dempster and Alice J Corkill. Interference and inhibition in cognition and behavior: Unifying themes for educational psychology. Educational Psychology Review , 11:1--88, 1999

    1999

  28. [36]

    How students “unpack” the structure of a word problem: Graphic representations and problem solving

    Kellah Edens and Ellen Potter. How students “unpack” the structure of a word problem: Graphic representations and problem solving. School Science and Mathematics , 108(5):184--196, 2008

    2008

  29. [37]

    Renae Low and R. Over. Detection of missing and irrelevant information within algebraic story problems. British Journal of Educational Psychology , 59:296--305, 1989

    1989

  30. [38]

    Metacognition: A literature review

    Emily R Lai. Metacognition: A literature review. 2011

    2011

  31. [39]

    Strategies for improving learner metacognition in health professional education

    Melissa S Medina, Ashley N Castleberry, and Adam M Persky. Strategies for improving learner metacognition in health professional education. American journal of pharmaceutical education , 81(4):78, 2017

    2017

  32. [40]

    Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars

    Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.01307 , 2025

    2025 arXiv

  33. [41]

    A survey of deep active learning

    Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR) , 54(9):1--40, 2021

    2021

  34. [42]

    Deep bayesian active learning with image data

    Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning , pages 1183--1192. PMLR, 2017

    2017

  35. [43]

    Reinforcement learning: An introduction , volume 1

    Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction , volume 1. MIT press Cambridge, 1998

    1998

  36. [44]

    Partially observable task and motion planning with uncertainty and risk awareness

    Aidan Curtis, George Matheos, Nishad Gothoskar, Vikash Mansinghka, Joshua Tenenbaum, Tom \'a s Lozano-P \'e rez, and Leslie Pack Kaelbling. Partially observable task and motion planning with uncertainty and risk awareness. arXiv preprint arXiv:2403.10454 , 2024

    2024 arXiv

  37. [45]

    Combined task and motion planning under partial observability: An optimization-based approach

    Camille Phiquepal and Marc Toussaint. Combined task and motion planning under partial observability: An optimization-based approach. In 2019 International Conference on Robotics and Automation (ICRA) , pages 9000--9006. IEEE, 2019

    2019

  38. [46]

    The communicative function of ambiguity in language

    Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cognition , 122(3):280--291, 2012

    2012

  39. [47]

    The puzzle of ambiguity

    Thomas Wasow, Amy Perfors, and David Beaver. The puzzle of ambiguity. Morphology and the web of grammar: Essays in memory of Steven G. Lapointe , pages 265--282, 2005

    2005

  40. [48]

    Semantic ambiguity within and across languages: An integrative review

    Tamar Degani and Natasha Tokowicz. Semantic ambiguity within and across languages: An integrative review. Quarterly Journal of Experimental Psychology , 63(7):1266--1303, 2010

    2010

  41. [49]

    What computers can’t do: The limits of artificial intelligence

    Hubert L Dreyfus. What computers can’t do: The limits of artificial intelligence. 1972

    1972

  42. [50]

    Foundational challenges in assuring alignment and safety of large language models

    Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932 , 2024

    2024 arXiv

  43. [51]

    How to enable effective cooperation between humans and nlp models: A survey of principles, formalizations, and beyond

    Chen Huang, Yang Deng, Wenqiang Lei, Jiancheng Lv, Tat-Seng Chua, and Jimmy Xiangji Huang. How to enable effective cooperation between humans and nlp models: A survey of principles, formalizations, and beyond. arXiv preprint arXiv:2501.05714 , 2025

    2025 arXiv

  44. [52]

    Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations , 2023

    2023

  45. [53]

    Ambigqa: Answering ambiguous open-domain questions

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645 , 2020

    2004 arXiv

  46. [54]

    Chatshop: Interactive information seeking with language agents

    Sanxing Chen, Sam Wiseman, and Bhuwan Dhingra. Chatshop: Interactive information seeking with language agents. arXiv preprint arXiv:2404.09911 , 2024

    2024 arXiv

  47. [55]

    Style: Improving domain transferability of asking clarification questions in large language model powered conversational agents

    Yue Chen, Chen Huang, Yang Deng, Wenqiang Lei, Dingnan Jin, Jia Liu, and Tat-Seng Chua. Style: Improving domain transferability of asking clarification questions in large language model powered conversational agents. In Findings of the Association for Computational Linguistics...

    2024

  48. [56]

    Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration

    Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. In Findings of the Association for Computational Linguistics: EMNLP 2023...

    2023

  49. [57]

    Clam: Selective clarification for ambiguous questions with generative language models

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Clam: Selective clarification for ambiguous questions with generative language models. arXiv preprint arXiv:2212.07769 , 2022

    2022 arXiv

  50. [58]

    Selectively answering ambiguous questions

    Jeremy Cole, Michael Zhang, Dan Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively answering ambiguous questions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages 530--543, 2023

    2023

  51. [59]

    Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling

    Pawe Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I \ n igo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in...

    2018

  52. [60]

    Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset

    Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 8689--8696, 2020

    2020

  53. [61]

    We need to consider disagreement in evaluation

    Valerio Basile, Michael Fell, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, Massimo Poesio, and Alexandra Uma. We need to consider disagreement in evaluation. Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future , 2021

    2021

  54. [62]

    Everyone’s voice matters: Quantifying annotation disagreement using demographic information

    Ruyuan Wan, Jaehyung Kim, and Dongyeop Kang. Everyone’s voice matters: Quantifying annotation disagreement using demographic information. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 14523--14530, 2023

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.