A recipe that maps stakeholder risk tolerance to a confidence level and then uses statistical methods, with conformal prediction performing best, to set thresholds on LLM evaluation metrics.
Schroedinger's Threshold: When the AUC doesn't predict Accuracy
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The Area Under Curve measure (AUC) seems apt to evaluate and compare diverse models, possibly without calibration. An important example of AUC application is the evaluation and benchmarking of models that predict faithfulness of generated text. But we show that the AUC yields an academic and optimistic notion of accuracy that can misalign with the actual accuracy observed in application, yielding significant changes in benchmark rankings. To paint a more realistic picture of downstream model performance (and prepare a model for actual application), we explore different calibration modes, testing calibration data and method.
citation-role summary
background 1
citation-polarity summary
fields
stat.ML 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
How to Choose a Threshold for an Evaluation Metric for Large Language Models
A recipe that maps stakeholder risk tolerance to a confidence level and then uses statistical methods, with conformal prediction performing best, to set thresholds on LLM evaluation metrics.