REVIEW 4 cited by
ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ)
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This document presents a detailed description of the challenge on clarifying questions for dialogue systems (ClariQ). The challenge is organized as part of the Conversational AI challenge series (ConvAI3) at Search Oriented Conversational AI (SCAI) EMNLP workshop in 2020. The main aim of the conversational systems is to return an appropriate answer in response to the user requests. However, some user requests might be ambiguous. In IR settings such a situation is handled mainly thought the diversification of the search result page. It is however much more challenging in dialogue settings with limited bandwidth. Therefore, in this challenge, we provide a common evaluation framework to evaluate mixed-initiative conversations. Participants are asked to rank clarifying questions in an information-seeking conversations. The challenge is organized in two stages where in Stage 1 we evaluate the submissions in an offline setting and single-turn conversations. Top participants of Stage 1 get the chance to have their model tested by human annotators.
Forward citations
Cited by 4 Pith papers
-
Personalized Deep Research Query Refinement with Graph-Scaffolded Evidence Grounding
G-STEER uses an Intent Elicitation Graph and evidence-state tracking to train a query refiner that routes between memory retrieval, user clarification, and stopping, improving personalized deep research outcomes.
-
One More Turn, Less Regret: A Regret-Based Multi-Turn Benchmark for LLMs' Clarification Policies
RegretBench evaluates LLM clarification as a sequential policy under hidden intent, showing that final accuracy alone misses large differences in interaction efficiency and stopping quality.
-
Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions
A fine-tuned classifier and question generator let a small coding assistant detect under-specified prompts and ask for clarification, which users rated better than a baseline in a small study.
-
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
LLMs alone annotate search clarifications unreliably; adding confidence-based selective human review cuts effort 24-45% in simulation, but the evaluation is partly built from the ground truth it predicts.
Discussion (0). Sign in to comment.