REVIEW 6 cited by
ALFA: Aligning LLMs to Ask Good Questions A Case Study in Clinical Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) often fail to ask effective questions under uncertainty, making them unreliable in domains where proactive information-gathering is essential for decision-making. We present ALignment via Fine-grained Attributes, (ALFA) a framework that improves LLM question-asking by (i) decomposing the notion of a "good" question into a set of theory-grounded attributes (e.g., clarity, relevance), (ii) controllably synthesizing attribute-specific question variations, and (iii) aligning models via preference-based optimization to explicitly learn to ask better questions along these fine-grained attributes. Focusing on clinical reasoning as a case study, we introduce the MediQ-AskDocs dataset, composed of 17k real-world clinical interactions augmented with 80k attribute-specific preference pairs of follow-up questions, as well as a novel expert-annotated interactive healthcare QA task to evaluate question-asking abilities. Models aligned with ALFA reduce diagnostic errors by 56.6% on MediQ-AskDocs compared to SoTA instruction-tuned LLMs, with a question-level win-rate of 64.4% and strong generalizability. Our findings suggest that explicitly guiding question-asking with structured, fine-grained attributes offers a scalable path to improve LLMs, especially in expert application domains.
Forward citations
Cited by 6 Pith papers
-
Asking Questions the Right Way: A Multi-Agent Conversational System for Prompt Formulation in Complex Task Resolution
An eight-agent question-asking system that front-loads intent clarification produced more complete prompts, higher-rated outputs, and single-turn task completion in a four-person pilot, with unstable effect sizes.
-
MedUPS: Towards Diagnostic Assistance in Uncommon Medical Cases with Large Language Models
Mid-stream alignment on next-step clinical decisions (MedUPS) improves LLM next-step accuracy on uncommon cases, but the size of the gain depends substantially on which LLM judge does the scoring.
-
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
A clinician-validated benchmark for patient-facing health AI agents shows that even frontier models fail triage in up to a quarter of realistic tool-using conversations.
-
Beyond Passive Critical Thinking: Fostering Proactive Questioning to Enhance Human-AI Collaboration
A training method using reinforcement learning and answerability heuristics lets small language models actively ask for missing math details and then solve problems, raising accuracy on the new GSM-MC benchmark from 0...
-
PrefPalette: Personalized Preference Modeling with Latent Attributes
Decomposing text into latent attributes and learning community-specific attribute weights improves preference prediction on Reddit and yields interpretable community profiles.
-
The Curious Language Model: Strategic Test-Time Information Acquisition
CuriosiTree is a greedy tree-search policy that lets LLMs select cost-effective information-gathering actions at test time, outperforming baselines in a simulated clinical diagnosis environment.
Discussion (0). Sign in to comment.