A clinical validity protocol using indices from 2x2 contingency tables classifies LLM confidence signals as Valid, Indeterminate, or Invalid, with valid profiles showing positive correlation to accuracy and invalid ones negative.
arXiv preprint arXiv:2603.09309 (2026)
5 Pith papers cite this work. Polarity classification is still indexing.
5
Pith papers citing it
citation-role summary
background 2
citation-polarity summary
years
2026 5roles
background 2representative citing papers
Validity indices adapted from clinical assessment classify four frontier LLMs as construct-level invalid on metacognitive probes, with valid models showing positive item-sensitive confidence (r=.18) while invalid ones show the opposite (r=-.20).
Meta-d′ and signal detection theory give comparable measures of AI metacognitive sensitivity and risk-sensitive decision regulation, demonstrated on three LLMs.
The validity screen classifies LLM confidence signals as Valid, Indeterminate, or Invalid, and these labels predict selective prediction AUROC with Valid models averaging 0.624 and Invalid models 0.357 across 20 LLMs.
citing papers explorer
-
Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals
A clinical validity protocol using indices from 2x2 contingency tables classifies LLM confidence signals as Valid, Indeterminate, or Invalid, with valid profiles showing positive correlation to accuracy and invalid ones negative.
-
Before You Interpret the Profile: Validity Scaling for LLM Metacognitive Self-Report
Validity indices adapted from clinical assessment classify four frontier LLMs as construct-level invalid on metacognitive probes, with valid models showing positive item-sensitive confidence (r=.18) while invalid ones show the opposite (r=-.20).
-
Measuring the metacognition of AI
Meta-d′ and signal detection theory give comparable measures of AI metacognitive sensitivity and risk-sensitive decision regulation, demonstrated on three LLMs.
-
Concurrent Criterion Validation of a Validity Screen for LLM Confidence Signals via Selective Prediction
The validity screen classifies LLM confidence signals as Valid, Indeterminate, or Invalid, and these labels predict selective prediction AUROC with Valid models averaging 0.624 and Invalid models 0.357 across 20 LLMs.
- Process Supervision of Confidence Margin for Calibrated LLM Reasoning