Pith. sign in

REVIEW 3 major objections 1 minor 38 references

Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data

T0 review · 3 major / 1 minor · reviewed 2026-06-26 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Default confidence interval methods for precision, recall and F1 often fall well below nominal coverage in small-sample text classification tasks.

desk verdict The paper's simulations show Wald and basic bootstrap intervals under-cover for F1 in small-sample high-performance text classification, with Agresti-Coull/Wilson and a new pseudo-count bootstrap doing better, plus effective-N adjustments needed for nested data. read the letter →

arxiv 2606.26422 v1 pith:HRW3GNNV submitted 2026-06-24 cs.AI

Kylie Anglin This is my paper · ORCID
classification cs.AI
keywords confidenceintervalstextclassificationperformancemetricsbootstrapnesteddataprecisionrecallF1largelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper evaluates confidence interval methods for performance metrics such as recall, precision, and F1 when supervised models or large language models classify text. It targets conditions typical in social science applications: small to moderate sample sizes, infrequent constructs, and texts nested within individuals. Simulations demonstrate that the Wald interval and basic percentile bootstrap frequently produce coverage far below the nominal 95 percent level. Accuracy improves with Agresti-Coull, Wilson, Clopper-Pearson intervals and a novel pseudo-count regularized bootstrap, with the latter especially useful for F1. For nested data, both effective sample size and appropriate degrees of freedom must be adjusted; the hierarchical bootstrap outperforms the cluster bootstrap when individuals contribute a moderate number of texts.

What carries the argument

Simulation evaluation of analytic and bootstrap confidence intervals for binary classification metrics, with a novel pseudo-count regularized bootstrap introduced to stabilize F1 intervals.

What would settle it

An empirical check on a large collection of real labeled text datasets in which the true performance parameters can be treated as known, testing whether the recommended intervals achieve coverage rates close to 95 percent across repeated random subsamples.

Watch

Extended reading notes

Core claim

Across simulations, default methods such as the Wald interval and the basic percentile bootstrap are the least accurate, with coverage sometimes far below the nominal 95% level. Accuracy is improved with the use of Agresti-Coull, Wilson, Clopper-Pearson, and a novel pseudo-count regularized bootstrap (which is particularly relevant to the calculation of F1). When texts are nested within individuals, adjustment for both effective N and the appropriate degrees of freedom is necessary for producing accurate analytic intervals. Among bootstrap intervals, the hierarchical bootstrap is more accurate than the cluster bootstrap when individuals produce a moderate number of texts but overly conservat

Load-bearing premise

The simulation conditions and data-generating processes accurately represent the typical use cases in social science text classification, including small to moderate sample sizes, infrequent constructs, and texts nested within individuals.

Editorial extensions

If this is right

  • Reporting of uncertainty around classifier performance metrics becomes more reliable when adjusted intervals replace default methods.
  • Study designs for text classification should incorporate larger validation samples when constructs are rare.
  • Hierarchical bootstrap or adjusted analytic intervals should be used when texts are nested within individuals.
  • Pseudo-count regularization offers a practical improvement for F1 intervals in high-performance or small-sample regimes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Validation studies of large language models on modest labeled sets would benefit from switching to these intervals to avoid understated uncertainty.
  • The same coverage problems likely appear in other imbalanced classification settings outside text, such as medical diagnosis or fraud detection.
  • Design-stage sample-size planning could be informed by the simulation results to target acceptable interval width before data collection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 1 minor

Summary. The paper evaluates confidence interval methods for performance metrics (precision, recall, F1) from text classifiers, including LLMs, under conditions typical of social science applications: small-to-moderate sample sizes, low-prevalence constructs, and texts nested within individuals. Simulations demonstrate that Wald intervals and basic percentile bootstrap exhibit undercoverage (sometimes well below nominal 95%), while Agresti-Coull, Wilson, Clopper-Pearson, and a proposed pseudo-count regularized bootstrap achieve better coverage; for nested data, both effective sample size and degrees-of-freedom adjustments are required for analytic intervals, and hierarchical bootstrap outperforms cluster bootstrap except at very low texts-per-individual.

Significance. If the simulation rankings hold under realistic conditions, the work supplies actionable guidance for uncertainty reporting in ML-based measurement, addressing a common gap in validity assessment. The explicit treatment of nesting and the novel regularized bootstrap constitute concrete contributions that could raise standards for transparency in applied NLP.

major comments (3)
  1. [Section 4, Appendix B] Section 4 and Appendix B: The data-generating processes fix prevalence, intra-class correlation, and performance levels within narrow ranges under binomial or simple hierarchical models. No external validation or sensitivity checks against published social-science corpora are reported; if real datasets exhibit stronger within-individual dependence, zero-inflated errors, or performance near 1.0 with small effective N, the reported coverage gaps and method rankings could reverse, undermining the central recommendation.
  2. The pseudo-count regularization parameter is listed as a free parameter with no default selection rule or sensitivity analysis provided in the main text or appendices; because the novel bootstrap's reported superiority depends on this choice, its practical advantage over Agresti-Coull/Wilson remains incompletely characterized.
  3. [Section 4] The claim that effective-N plus df adjustment is necessary for nested analytic intervals rests on the specific simulation design; without reporting coverage under alternative dependence structures (e.g., varying ICC or zero-inflated Bernoulli), the necessity statement is tied to the chosen DGPs rather than shown to be robust.
minor comments (1)
  1. Notation for effective sample size and the exact formula for the pseudo-count bootstrap should be stated explicitly in the main text rather than deferred to appendices.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive comments. The feedback highlights important considerations regarding the scope of our simulations and the characterization of the proposed method. We address each point below and indicate where revisions will be made to clarify limitations and strengthen the practical guidance.

read point-by-point responses

  1. Referee: Section 4 and Appendix B: The data-generating processes fix prevalence, intra-class correlation, and performance levels within narrow ranges under binomial or simple hierarchical models. No external validation or sensitivity checks against published social-science corpora are reported; if real datasets exhibit stronger within-individual dependence, zero-inflated errors, or performance near 1.0 with small effective N, the reported coverage gaps and method rankings could reverse, undermining the central recommendation.

    Authors: We agree that the simulations employ controlled DGPs and do not directly benchmark against specific published corpora. The design isolates the impact of sample size, prevalence, and nesting under conditions representative of social-science text classification tasks. We will revise the discussion section to explicitly state these scope limitations and recommend that users perform dataset-specific checks. Additional sensitivity runs varying ICC more broadly and introducing zero-inflation can be added to Appendix B to test robustness of the observed rankings. revision: partial

  2. Referee: The pseudo-count regularization parameter is listed as a free parameter with no default selection rule or sensitivity analysis provided in the main text or appendices; because the novel bootstrap's reported superiority depends on this choice, its practical advantage over Agresti-Coull/Wilson remains incompletely characterized.

    Authors: We accept this criticism. The revised manuscript will designate a default regularization value (selected via a simple cross-validation heuristic on held-out folds) and include a new appendix subsection with sensitivity plots across a range of pseudo-count values, demonstrating that the coverage advantage persists for moderate choices while noting when performance converges to the unregularized bootstrap. revision: yes

  3. Referee: The claim that effective-N plus df adjustment is necessary for nested analytic intervals rests on the specific simulation design; without reporting coverage under alternative dependence structures (e.g., varying ICC or zero-inflated Bernoulli), the necessity statement is tied to the chosen DGPs rather than shown to be robust.

    Authors: The simulations already vary ICC across a range of values typical in nested text data and include both binomial and hierarchical Bernoulli generators. The necessity of the dual adjustment is shown to hold across these conditions. We will rephrase the relevant claim in Section 4 to emphasize that it applies within the studied dependence structures and add a brief note that extrapolation to markedly different processes (e.g., strong zero-inflation) would require further investigation. revision: partial

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: claims rest on independent simulation coverage checks

full rationale

The paper's central claims concern the relative accuracy of CI methods (Wald, Agresti-Coull, Wilson, Clopper-Pearson, percentile bootstrap, pseudo-count bootstrap, hierarchical bootstrap) for precision/recall/F1 under small-sample, high-performance, and nested-text conditions. These claims are supported by Monte Carlo simulations that generate data under binomial or hierarchical models, compute intervals with each method, and measure empirical coverage against the known true performance parameter. No equation or derivation reduces a reported coverage gap or ranking to a fitted parameter taken from the same data; the simulation DGPs and performance levels are chosen independently of any fitted result in the paper. No self-citation is invoked as a uniqueness theorem or load-bearing premise. The evaluation is therefore self-contained against external benchmarks (nominal 95% coverage) and does not exhibit any of the enumerated circularity patterns.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

The work relies on standard statistical assumptions for binomial proportions and bootstrap resampling. The novel method introduces a regularization parameter whose specific value and selection process are not detailed in the abstract. No new entities are postulated.

free parameters (1)
  • pseudo-count regularization parameter
    Introduced in the novel bootstrap for stabilizing F1 estimates; value not specified in abstract.
assumptions (2)
  • domain assumption Classifier performance metrics behave as binomial or multinomial proportions under the simulated conditions.
    Invoked implicitly for all analytic interval methods evaluated.
  • domain assumption Nested texts within individuals induce clustering that reduces effective sample size.
    Central to the nested data adjustment recommendations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data." pith.science (2026). https://pith.science/paper/HRW3GNNV

@misc{pith2026260626422,
  author       = {Pith},
  title        = {Pith review of: Estimating Uncertainty in Classifier Performance with Applications to Large Language Models and Nested Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HRW3GNNV}},
  note         = {Machine review of arXiv:2606.26422}
}
read the original abstract

Researchers increasingly use text classification--supervised models or large language models--to measure constructs from natural language, providing metrics such as recall and precision as evidence of their validity. Yet, though these metrics are point estimates subject to sampling variation, measures of uncertainty are inconsistently reported alongside them. Further, when they are reported, they are often estimated with methods that are not appropriate when relevant labelled datasets are small or performance is high. To increase and improve confidence interval reporting in the field, this paper evaluates confidence interval methods for performance metrics under conditions typical of social science text classification: small to moderate sample sizes, infrequent constructs, and texts nested within individuals. Across simulations, default methods such as the Wald interval and the basic percentile bootstrap are the least accurate, with coverage sometimes far below the nominal 95% level. Accuracy is improved with the use of Agresti-Coull, Wilson, Clopper-Pearson, and a novel pseudo-count regularized bootstrap (which is particularly relevant to the calculation of F1). When texts are nested within individuals, we demonstrate that adjustment for both effective N and the appropriate degrees of freedom is necessary for producing accurate analytic intervals. Among bootstrap intervals, the hierarchical bootstrap is more accurate than the cluster bootstrap when individuals produce a moderate number of texts but overly conservative when individuals produce only a few. By providing guidance to the field on appropriate interval estimation, we aim to improve the transparency of machine learning applications, and to encourage greater attention to the validation sample size at the design stage.

Figures

Figures reproduced from arXiv: 2606.26422 by the authors.

Figure 1
Figure 1. Coverage Rates for Interval Calculation Methods As expected, the Wald interval demonstrates undesirable properties. At higher proportions and sample sizes below 50, coverage is often below 0.9 with some coverage rates reaching 0.6 and 8 [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Coverage Rates for the Percentile Bootstrap and Pseudo-Count Regularized Bootstrap (λ = 1) at a Prevalence of 0.10 10 [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Relationship between Confidence Interval Half-Width and Sample Size [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Small Corpus Coverage (∼3 Texts Per Individual) [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Moderate Corpus Coverage (∼30 Texts Per Individual) 16 [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Half-Width and Coverage Comparison for the Small-Corpus Design (∼3 Texts Per Person) 17 [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: displays the relationship between the number of texts used to estimate performance and the half width of an adjusted 95% Agresti-Coull confidence interval, holding the performance rate at 0.90. As expected, confidence interval width increases with ICC values, and the i…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 10 canonical work pages

  1. [1]

    Leakage and the reproducibility crisis inmachine-learning-basedscience.Patterns2023;4:100804–100804

    Kapoor, Sayash and Narayanan, Arvind , year =. Leakage and the reproducibility crisis in machine-learning-based science , volume =. Patterns , publisher =. doi:https://doi.org/10.1016/j.patter.2023.100804 , number =

    work page doi:10.1016/j.patter.2023.100804 2023

  2. [2]

    FotiosFitsilis, MariaKamilaki, BasilisGatos, VassilisKatsouros, andGeorgeMikros

    Construct validity in psychological tests. , volume =. Psychological Bulletin , author =. 1955 , note =. doi:10.1037/h0040957 , number =

    work page doi:10.1037/h0040957 1955

  3. [3]

    Harvard University Press, Cambridge (2015); Selbst, A.D., et al.: Fairness and abstraction in sociotechnical systems

    Mitchell, Margaret and Wu, Simone and Zaldivar, Andrew and Barnes, Parker and Vasserman, Lucy and Hutchinson, Ben and Spitzer, Elena and Raji, Inioluwa Deborah and Gebru, Timnit , year =. Model cards for model reporting , booktitle =. doi:10.1145/3287560.3287596 , publisher =

    work page doi:10.1145/3287560.3287596

  4. [4]

    Statistical power analysis for the behavioral sciences , isbn =

    Cohen, Jacob , year =. Statistical power analysis for the behavioral sciences , isbn =

  5. [5]

    National Center for Special Education Research , author =

    Statistical power analysis in education research (. National Center for Special Education Research , author =. 2000 , keywords =. doi:10.1037/e599842011-001 , abstract =

    work page doi:10.1037/e599842011-001 2000

  6. [6]

    , editor =

    Shaffer, David Williamson and Ruis, Andrew R. , editor =. How. 2021 , keywords =

    2021

  7. [7]

    Educational Evaluation and Policy Analysis , author =

    Intraclass correlation values for planning group-randomized trials in education , volume =. Educational Evaluation and Policy Analysis , author =. 2007 , pages =. doi:https://doi.org/10.3102/0162373707299 , number =

    work page doi:10.3102/0162373707299 2007

  8. [8]

    and Stewart, Brandon M

    Grimmer, Justin and Roberts, Margaret E. and Stewart, Brandon M. , year =. Text as data:

Show all 38 references
  1. [9]

    Proceedings of the 2020 conference on fairness, accountability, and transparency , author =

    Algorithmic realism: expanding the boundaries of algorithmic thought , url =. Proceedings of the 2020 conference on fairness, accountability, and transparency , author =. 2020 , pages =

    2020

  2. [10]

    Gender shades:

    Buolamwini, Joy and Gebru, Timnit , year =. Gender shades:. Conference on fairness, accountability and transparency , publisher =

  3. [11]

    James, Gareth and Witten, Daniela and Hastie, Trevor and Tibshirani, Robert , year =. An

  4. [12]

    and Grossmann, Igor , year =

    Stavropoulos, Alexander and Crone, Damien L. and Grossmann, Igor , year =. Shadows of wisdom:. doi:https://doi.org/10.3758/s13428-024-02441-0 , journal =

    doi

  5. [13]

    Proceedings of the National Academy of Sciences , publisher =

    Gilardi, Fabrizio and Alizadeh, Meysam and Kubli, Maël , year =. Proceedings of the National Academy of Sciences , publisher =. doi:https://doi.org/10.1073/pnas.230501612 , number =

    doi

  6. [14]

    , month = dec, year =

    Anglin, Kylie L. , month = dec, year =. Addressing threats to validity in supervised machine learning:. AERA Open , publisher =. doi:10.1177/23328584241303495 , abstract =

    doi

  7. [15]

    , year =

    Gwet, Kilem L. , year =. Handbook of inter-rater reliability:

  8. [16]

    arXiv preprint arXiv:2508.10223 , author =

    A. arXiv preprint arXiv:2508.10223 , author =. 2025 , file =

    2025

  9. [17]

    , year =

    Agresti, Alan and Coull, Brent A. , year =. Approximate is better than “exact” for interval estimation of binomial proportions , volume =. The American Statistician , publisher =

  10. [18]

    Survey sampling , publisher =

    Kish, Leslie , year =. Survey sampling , publisher =

  11. [19]

    Easterling, Robert , year =. There's. The American Statistician , publisher =

  12. [20]

    and Pearson, Egon S

    Clopper, Charles J. and Pearson, Egon S. , year =. The use of confidence or fiducial limits illustrated in the case of the binomial , volume =. Biometrika , publisher =

  13. [21]

    Monographs on statistics and applied probability , author =

    An introduction to the bootstrap , volume =. Monographs on statistics and applied probability , author =. 1993 , pages =

    1993

  14. [22]

    Model evaluation, model selection, and algorithm selection in machine learning , journal =

    Raschka, Sebastian , year =. Model evaluation, model selection, and algorithm selection in machine learning , journal =

  15. [23]

    Evaluation metrics and statistical tests for machine learning , volume =

    Rainio, Oona and Teuho, Jarmo and Klén, Riku , year =. Evaluation metrics and statistical tests for machine learning , volume =. Scientific Reports , publisher =

  16. [24]

    Nonparametric standard errors and confidence intervals , volume =

    Efron, Bradley , year =. Nonparametric standard errors and confidence intervals , volume =. canadian Journal of Statistics , publisher =

  17. [25]

    Why question machine learning evaluation methods , volume =

    Japkowicz, Nathalie , year =. Why question machine learning evaluation methods , volume =

  18. [26]

    and Huang, Jin and Zhang, Harry , year =

    Ling, Charles X. and Huang, Jin and Zhang, Harry , year =. Conference of the canadian society for computational studies of intelligence , publisher =

  19. [27]

    and Reisenbichler, Emily S

    Han, Gang and Schell, Michael J. and Reisenbichler, Emily S. and Guo, Bohong and Rimm, David L. , year =. Determination of the number of observers needed to evaluate a subjective test and its application in two. Statistics in medicine , publisher =

  20. [28]

    Journal of the Royal Statistical Society: Series B (Methodological) , author =

    A. Journal of the Royal Statistical Society: Series B (Methodological) , author =. 1988 , note =

    1988

  21. [29]

    WIREs Comp Stat , author =

    Bootstrap , volume =. WIREs Comp Stat , author =. 2011 , note =

    2011

  22. [30]

    Evaluating confidence interval methods for binomial proportions in clustered surveys , volume =

    Dean, Natalie and Pagano, Marcello , year =. Evaluating confidence interval methods for binomial proportions in clustered surveys , volume =. Journal of Survey Statistics and Methodology , publisher =

  23. [31]

    Contemporary Clinical Trials , author =

    Comparison of methods for estimating the intraclass correlation coefficient for binary responses in cancer prevention cluster randomized trials , volume =. Contemporary Clinical Trials , author =. 2012 , keywords =. doi:10.1016/j.cct.2012.05.004 , abstract =

    2012 doi

  24. [32]

    and Graubard, Barry I

    Korn, Edward L. and Graubard, Barry I. , year =. Confidence intervals for proportions with small expected number of positive counts estimated from survey data , volume =. Survey Methodology , publisher =

  25. [33]

    Leveraging large language models to identify microcounseling skills in psychotherapy transcripts , doi =

    Hammerfald, Karin and Schmidt, Fabian and Vlassov, Vladimir and Haaland Jahren, Henrik and Solbakken, Ole André , year =. Leveraging large language models to identify microcounseling skills in psychotherapy transcripts , doi =. Psychotherapy Research , publisher =

  26. [34]

    Proceedings of the fourth workshop on computational linguistics and clinical psychology—from linguistic signal to clinical reality , author =

    Monitoring tweets for depression to detect at-risk users , url =. Proceedings of the fourth workshop on computational linguistics and clinical psychology—from linguistic signal to clinical reality , author =. 2017 , pages =

    2017

  27. [35]

    Neurons, behavior, data analysis and theory , author =

    Application of the hierarchical bootstrap to multi-level data in neuroscience , volume =. Neurons, behavior, data analysis and theory , author =. 2020 , pages =

    2020

  28. [36]

    and Welsh, Alan H

    Field, Christopher A. and Welsh, Alan H. , year =. Bootstrapping clustered data , volume =. Journal of the Royal Statistical Society Series B: Statistical Methodology , publisher =

  29. [37]

    Bernoulli , volume =

    Resampling and exchangeable arrays , author =. Bernoulli , volume =. 2000 , file =

    2000

  30. [38]

    REVSTAT-Statistical Journal , author =

    Interval estimators for a binomial proportion:. REVSTAT-Statistical Journal , author =. 2008 , note =

    2008

Pith tools

Reviewed June 26, 2026 · model on record in the stance chip above.