REVIEW 6 cited by
Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Resolving ambiguities through interaction is a hallmark of natural language, and modeling this behavior is a core challenge in crafting AI assistants. In this work, we study such behavior in LMs by proposing a task-agnostic framework for resolving ambiguity by asking users clarifying questions. Our framework breaks down this objective into three subtasks: (1) determining when clarification is needed, (2) determining what clarifying question to ask, and (3) responding accurately with the new information gathered through clarification. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. For the first subtask, we present a novel uncertainty estimation approach, intent-sim, that determines the utility of querying for clarification by estimating the entropy over user intents. Our method consistently outperforms existing uncertainty estimation approaches at identifying predictions that will benefit from clarification. When only allowed to ask for clarification on 10% of examples, our system is able to double the performance gains over randomly selecting examples to clarify. Furthermore, we find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs. Together, our work lays foundation for studying clarifying interactions with LMs.
Forward citations
Cited by 6 Pith papers
-
HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation
VLAC-Cut-guided multi-robot HITL post-training reaches 80–95% success and 1.7–4.2× throughput over the base VLA, outperforming HITL-only under the same human budget.
-
Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspective
Transformers systematically deviate from the Bayes-optimal predictor under high-ambiguity contexts on a new HMM benchmark, and a Monte Carlo predictor that decouples task inference from token prediction partly closes ...
-
SG-CoT: An Ambiguity-Aware Robotic Planning Framework using Scene Graph Representations
SG-CoT grounds an LLM planner's chain-of-thought in a scene graph via iterative API queries, improving ambiguity detection and clarification in simulated manipulation, though its success metric credits any clarifying ...
-
Referential ambiguity and clarification requests: comparing human and LLM behaviour
Humans seldom ask clarification questions for referential ambiguity, while LLMs ask them more often, and reasoning prompts increase LLM question frequency and relevance.
-
Extending Epistemic Uncertainty Beyond Parameters Would Assist in Designing Reliable LLMs
Bayesian Modeling of Experiments is proposed as a unifying framework for quantifying and reducing the many sources of uncertainty in LLM deployments, beyond abstention.
- Exploring the Usage of Generative AI for Group Project-Based Offline Art Courses in Elementary Schools
Discussion (0). Continue with ORCID to comment.