On the Utility of Prediction Sets in Human-AI Teams
read the original abstract
Research on human-AI teams usually provides experts with a single label, which ignores the uncertainty in a model's recommendation. Conformal prediction (CP) is a well established line of research that focuses on building a theoretically grounded, calibrated prediction set, which may contain multiple labels. We explore how such prediction sets impact expert decision-making in human-AI teams. Our evaluation on human subjects finds that set valued predictions positively impact experts. However, we notice that the predictive sets provided by CP can be very large, which leads to unhelpful AI assistants. To mitigate this, we introduce D-CP, a method to perform CP on some examples and defer to experts. We prove that D-CP can reduce the prediction set size of non-deferred examples. We show how D-CP performs in quantitative and in human subject experiments ($n=120$). Our results suggest that CP prediction sets improve human-AI team performance over showing the top-1 prediction alone, and that experts find D-CP prediction sets are more useful than CP prediction sets.
This paper has not been read by Pith yet.
Forward citations
Cited by 5 Pith papers
-
Set-Valued Policy Learning
The paper develops set-valued policies and conformal policy learning methods that output treatment sets with marginal coverage guarantees for robust decision-making under uncertainty.
-
What Types of Human-AI Teams Exist?
Analysis of 53 human-AI team papers yields five distinct clusters (AI Assistant, Ad-hoc Dependency, Ad-hoc Forced Dependency, Paired Equanimity, Group Equanimity) based on psychological team characteristics.
-
Strategic Decision Support for AI Agents
The paper introduces an optimization framework for AI agents to strategically seek support, proving a threshold policy on support value and providing an online algorithm to control missed-support error without distrib...
-
Medical Model Synthesis Architectures: A Case Study
MedMSA framework retrieves knowledge via language models then builds formal probabilistic models to produce uncertainty-weighted differential diagnoses from symptoms.
-
PLACO: A Multi-Stage Framework for Cost-Effective Performance in Human-AI Teams
PLACO is a multi-stage framework that extends Bayesian combination of human and model labels to achieve cost-effective high performance in human-AI teams.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.