REVIEW 5 cited by
Evaluating Human-AI Collaboration: A Review and Methodological Framework
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The use of artificial intelligence (AI) in working environments with individuals, known as Human-AI Collaboration (HAIC), has become essential in a variety of domains, boosting decision-making, efficiency, and innovation. Despite HAIC's wide potential, evaluating its effectiveness remains challenging due to the complex interaction of components involved. This paper provides a detailed analysis of existing HAIC evaluation approaches and develops a fresh paradigm for more effectively evaluating these systems. Our framework includes a structured decision tree which assists to select relevant metrics based on distinct HAIC modes (AI-Centric, Human-Centric, and Symbiotic). By including both quantitative and qualitative metrics, the framework seeks to represent HAIC's dynamic and reciprocal nature, enabling the assessment of its impact and success. This framework's practicality can be examined by its application in an array of domains, including manufacturing, healthcare, finance, and education, each of which has unique challenges and requirements. Our hope is that this study will facilitate further research on the systematic evaluation of HAIC in real-world applications.
Forward citations
Cited by 5 Pith papers
-
When Models Know More Than They Can Explain: Quantifying Knowledge Transfer in Human-AI Collaboration
Model benchmark performance only weakly predicts how well people learn from AI explanations, with notable outliers across code and math.
-
More Human or More AI? Visualizing Human-AI Collaboration Disclosures in Journalistic News Production
Disclosure visualization format systematically shifts readers' perceptions of human vs AI contribution: role-based timelines amplify perceived AI role in mostly human articles, while task-based timelines make mostly A...
-
Toward Resilient Human-AI Collaboration: A Lifecycle Taxonomy of Sociotechnical Risks and Cascading Failures
A literature synthesis maps human-AI collaboration failures into six interacting risk clusters arranged along a four-stage lifecycle.
-
HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualization
HiLSVA shows that a human-in-the-loop LLM agent system can help novices and experts complete scientific visualization tasks, while human oversight adds measurable execution time.
-
Interaction as Intelligence: Deep Research With Human-AI Partnership
A human-in-the-loop deep research system with transparent, interruptible interaction is claimed to outperform commercial baselines, but the evidence is weakened by small samples and biased instructions.
Discussion (0). Continue with ORCID to comment.