Product-of-token-probabilities with Platt or isotonic rescaling is a strong, cheap confidence signal for text-to-SQL, beating minimum-token pooling but matching or losing to self-check on large Llama models.
TCE: A Test-Based Approach to Measuring Calibration Error
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
This paper proposes a new metric to measure the calibration error of probabilistic binary classifiers, called test-based calibration error (TCE). TCE incorporates a novel loss function based on a statistical test to examine the extent to which model predictions differ from probabilities estimated from data. It offers (i) a clear interpretation, (ii) a consistent scale that is unaffected by class imbalance, and (iii) an enhanced visual representation with repect to the standard reliability diagram. In addition, we introduce an optimality criterion for the binning procedure of calibration error metrics based on a minimal estimation error of the empirical probabilities. We provide a novel computational algorithm for optimal bins under bin-size constraints. We demonstrate properties of TCE through a range of experiments, including multiple real-world imbalanced datasets and ImageNet 1000.
fields
cs.DB 1years
2024 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities
Product-of-token-probabilities with Platt or isotonic rescaling is a strong, cheap confidence signal for text-to-SQL, beating minimum-token pooling but matching or losing to self-check on large Llama models.