REVIEW 2 cited by
Posterior calibration and exploratory analysis for natural language processing models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Many models in natural language processing define probabilistic distributions over linguistic structures. We argue that (1) the quality of a model' s posterior distribution can and should be directly evaluated, as to whether probabilities correspond to empirical frequencies, and (2) NLP uncertainty can be projected not only to pipeline components, but also to exploratory data analysis, telling a user when to trust and not trust the NLP analysis. We present a method to analyze calibration, and apply it to compare the miscalibration of several commonly used models. We also contribute a coreference sampling algorithm that can create confidence intervals for a political event extraction task.
Forward citations
Cited by 2 Pith papers
-
Rationales Are Not Silver Bullets: Measuring the Impact of Rationales on Model Performance and Reliability
Rationale-augmented finetuning can hurt accuracy while improving calibration, with the sizes of both effects tied linearly to task difficulty.
-
Clustered Calibration: Representation-Aware Probability Calibration via Learned Subpopulations
Clustered Calibration groups samples by learned representations and calibrates each cluster separately, with a new cluster-binned ECE claimed to rank models by both calibration and AUC.
Discussion (0). Sign in to comment.