{"id":"c5df44f6-54f0-4962-abb7-bcf22697e023","arxiv_id":"2606.26422","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"Simulations show standard CI methods underperform for classifier metrics in small and nested datasets, while Agresti-Coull, Wilson, Clopper-Pearson, and a new pseudo-count regularized bootstrap perform better, with specific adjustments needed for nested structures.","lead":"The paper evaluates confidence interval methods for text classifier performance metrics like precision and recall under small samples, rare constructs, and nested data. A smart generalist might read it to learn how to report uncertainty more reliably when using machine learning to measure things from text.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Simulation DGPs may not capture real-world nesting structures or performance regimes in social science text classification","rationale":"The reader correctly flags the simulation-to-reality mapping as the weakest link; the full text confirms that all quantitative claims derive from those simulations without external calibration, so the concern remains load-bearing and the UNVERDICTED stance is appropriate until the check is performed.","tokens_in":1833,"tokens_out":340,"duration_ms":10846,"concrete_test":"Extract the exact simulation parameters (N, prevalence, ICC, performance targets) from §4 and Appendix B; sample 20 recent social-science text-classification papers, compute their empirical N, prevalence, ICC, and observed F1; re-run the paper’s simulation code with those empirical moments as targets and check whether the coverage ordering of the six interval methods changes by more than 5 percentage points.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Agresti-Coull/Wilson/Clopper-Pearson and the pseudo-count bootstrap outperform Wald/percentile bootstrap, and that effective-N + df adjustments are required for nested data—rests entirely on the simulation design. The paper generates data under binomial or simple hierarchical models with fixed prevalence, fixed intra-class correlation, and performance levels drawn from a narrow range. If real labelled datasets exhibit stronger within-individual dependence, zero-inflated or heavy-tailed error patterns, or performance near 1.0 with very small effective N, the reported coverage gaps and relative rankings can reverse. Section 4 and Appendix B describe the DGPs but provide no external validation against published social-science corpora.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper evaluates confidence interval methods for performance metrics (precision, recall, F1) from text classifiers, including LLMs, under conditions typical of social science applications: small-to-moderate sample sizes, low-prevalence constructs, and texts nested within individuals. Simulations demonstrate that Wald intervals and basic percentile bootstrap exhibit undercoverage (sometimes well below nominal 95%), while Agresti-Coull, Wilson, Clopper-Pearson, and a proposed pseudo-count regularized bootstrap achieve better coverage; for nested data, both effective sample size and degrees-of-freedom adjustments are required for analytic intervals, and hierarchical bootstrap outperforms cluster bootstrap except at very low texts-per-individual.","tokens_in":1991,"tokens_out":561,"duration_ms":12953,"significance":"If the simulation rankings hold under realistic conditions, the work supplies actionable guidance for uncertainty reporting in ML-based measurement, addressing a common gap in validity assessment. The explicit treatment of nesting and the novel regularized bootstrap constitute concrete contributions that could raise standards for transparency in applied NLP.","major_comments":[{"comment":"Section 4 and Appendix B: The data-generating processes fix prevalence, intra-class correlation, and performance levels within narrow ranges under binomial or simple hierarchical models. No external validation or sensitivity checks against published social-science corpora are reported; if real datasets exhibit stronger within-individual dependence, zero-inflated errors, or performance near 1.0 with small effective N, the reported coverage gaps and method rankings could reverse, undermining the central recommendation.","section":"Section 4, Appendix B"},{"comment":"The pseudo-count regularization parameter is listed as a free parameter with no default selection rule or sensitivity analysis provided in the main text or appendices; because the novel bootstrap's reported superiority depends on this choice, its practical advantage over Agresti-Coull/Wilson remains incompletely characterized.","section":null},{"comment":"The claim that effective-N plus df adjustment is necessary for nested analytic intervals rests on the specific simulation design; without reporting coverage under alternative dependence structures (e.g., varying ICC or zero-inflated Bernoulli), the necessity statement is tied to the chosen DGPs rather than shown to be robust.","section":"Section 4"}],"minor_comments":[{"comment":"Notation for effective sample size and the exact formula for the pseudo-count bootstrap should be stated explicitly in the main text rather than deferred to appendices.","section":null}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central claims are simulation-dependent; the absence of any real-data validation or external corpus comparison is a scope issue that the authors should address before publication."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive comments. The feedback highlights important considerations regarding the scope of our simulations and the characterization of the proposed method. We address each point below and indicate where revisions will be made to clarify limitations and strengthen the practical guidance.","responses":[{"response":"We agree that the simulations employ controlled DGPs and do not directly benchmark against specific published corpora. The design isolates the impact of sample size, prevalence, and nesting under conditions representative of social-science text classification tasks. We will revise the discussion section to explicitly state these scope limitations and recommend that users perform dataset-specific checks. Additional sensitivity runs varying ICC more broadly and introducing zero-inflation can be added to Appendix B to test robustness of the observed rankings.","revision_made":"partial","referee_comment":"Section 4 and Appendix B: The data-generating processes fix prevalence, intra-class correlation, and performance levels within narrow ranges under binomial or simple hierarchical models. No external validation or sensitivity checks against published social-science corpora are reported; if real datasets exhibit stronger within-individual dependence, zero-inflated errors, or performance near 1.0 with small effective N, the reported coverage gaps and method rankings could reverse, undermining the central recommendation."},{"response":"We accept this criticism. The revised manuscript will designate a default regularization value (selected via a simple cross-validation heuristic on held-out folds) and include a new appendix subsection with sensitivity plots across a range of pseudo-count values, demonstrating that the coverage advantage persists for moderate choices while noting when performance converges to the unregularized bootstrap.","revision_made":"yes","referee_comment":"The pseudo-count regularization parameter is listed as a free parameter with no default selection rule or sensitivity analysis provided in the main text or appendices; because the novel bootstrap's reported superiority depends on this choice, its practical advantage over Agresti-Coull/Wilson remains incompletely characterized."},{"response":"The simulations already vary ICC across a range of values typical in nested text data and include both binomial and hierarchical Bernoulli generators. The necessity of the dual adjustment is shown to hold across these conditions. We will rephrase the relevant claim in Section 4 to emphasize that it applies within the studied dependence structures and add a brief note that extrapolation to markedly different processes (e.g., strong zero-inflation) would require further investigation.","revision_made":"partial","referee_comment":"The claim that effective-N plus df adjustment is necessary for nested analytic intervals rests on the specific simulation design; without reporting coverage under alternative dependence structures (e.g., varying ICC or zero-inflated Bernoulli), the necessity statement is tied to the chosen DGPs rather than shown to be robust."}],"tokens_in":1538,"tokens_out":585,"duration_ms":18669,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that common default intervals for precision, recall, and F1 often fall short on coverage in the small-to-moderate samples and low-prevalence settings that social scientists face with text classifiers or LLMs. The paper's simulations back Agresti-Coull, Wilson, Clopper-Pearson, and especially their pseudo-count regularized bootstrap as more reliable, and they show that nested texts require both effective sample size and degrees-of-freedom fixes. The hierarchical bootstrap also comes out ahead of cluster bootstrap in moderate nesting but turns conservative with very few texts per person.\n\nWhat is new is the pseudo-count regularized bootstrap aimed at F1, which extends standard proportion methods in a way that directly addresses the zero-one problems common with these metrics. The work does a clean job of setting up simulations that match the target use case rather than generic stats problems, and it gives practitioners concrete comparisons they can act on.\n\nThe soft spot is that the whole case rests on the simulation design. The DGPs are binomial or simple hierarchical with fixed prevalence and ICC, and there is no external check against actual labeled social-science corpora. If real data have heavier tails, stronger dependence, or performance near 1.0 with tiny effective N, the coverage gaps and method rankings could shift. The regularization parameter also lacks detailed selection guidance.\n\nThis is for applied researchers in computational social science who report classifier metrics and want defensible uncertainty. A reader facing nested LLM annotations would get usable advice. It deserves peer review because the practical problem is clear and the simulation evidence is focused, even if real-data validation would make the claims stronger.","headline":"The paper's simulations show Wald and basic bootstrap intervals under-cover for F1 in small-sample high-performance text classification, with Agresti-Coull/Wilson and a new pseudo-count bootstrap doing better, plus effective-N adjustments needed for nested data.","tokens_in":2482,"tokens_out":424,"would_cite":false,"duration_ms":18685,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Default confidence interval methods for precision, recall and F1 often fall well below nominal coverage in small-sample text classification tasks.","keywords":["confidence intervals","text classification","performance metrics","bootstrap","nested data","precision recall F1","large language models"],"falsifier":"An empirical check on a large collection of real labeled text datasets in which the true performance parameters can be treated as known, testing whether the recommended intervals achieve coverage rates close to 95 percent across repeated random subsamples.","tokens_in":2718,"feed_emoji":"📊","tokens_out":725,"duration_ms":15485,"temperature":0.7,"pith_summary":"This paper evaluates confidence interval methods for performance metrics such as recall, precision, and F1 when supervised models or large language models classify text. It targets conditions typical in social science applications: small to moderate sample sizes, infrequent constructs, and texts nested within individuals. Simulations demonstrate that the Wald interval and basic percentile bootstrap frequently produce coverage far below the nominal 95 percent level. Accuracy improves with Agresti-Coull, Wilson, Clopper-Pearson intervals and a novel pseudo-count regularized bootstrap, with the latter especially useful for F1. For nested data, both effective sample size and appropriate degrees of freedom must be adjusted; the hierarchical bootstrap outperforms the cluster bootstrap when individuals contribute a moderate number of texts.","feed_headline":"Default intervals under-cover for text classifier metrics","feed_subtitle":"Wald and basic bootstrap often miss 95 percent coverage with small samples or rare labels; adjusted methods restore accuracy","key_machinery":"Simulation evaluation of analytic and bootstrap confidence intervals for binary classification metrics, with a novel pseudo-count regularized bootstrap introduced to stabilize F1 intervals.","core_discovery":"Across simulations, default methods such as the Wald interval and the basic percentile bootstrap are the least accurate, with coverage sometimes far below the nominal 95% level. Accuracy is improved with the use of Agresti-Coull, Wilson, Clopper-Pearson, and a novel pseudo-count regularized bootstrap (which is particularly relevant to the calculation of F1). When texts are nested within individuals, adjustment for both effective N and the appropriate degrees of freedom is necessary for producing accurate analytic intervals. Among bootstrap intervals, the hierarchical bootstrap is more accurate than the cluster bootstrap when individuals produce a moderate number of texts but overly conservat","pith_inferences":["Validation studies of large language models on modest labeled sets would benefit from switching to these intervals to avoid understated uncertainty.","The same coverage problems likely appear in other imbalanced classification settings outside text, such as medical diagnosis or fraud detection.","Design-stage sample-size planning could be informed by the simulation results to target acceptable interval width before data collection."],"forward_implications":["Reporting of uncertainty around classifier performance metrics becomes more reliable when adjusted intervals replace default methods.","Study designs for text classification should incorporate larger validation samples when constructs are rare.","Hierarchical bootstrap or adjusted analytic intervals should be used when texts are nested within individuals.","Pseudo-count regularization offers a practical improvement for F1 intervals in high-performance or small-sample regimes."],"fun_headline_variants":["Wald and bootstrap miss 95 percent coverage for metrics","Agresti-Coull and Wilson fix under-coverage in classifiers","Hierarchical bootstrap needed for nested text data","Pseudo-count method improves F1 interval accuracy"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The simulation conditions and data-generating processes accurately represent the typical use cases in social science text classification, including small to moderate sample sizes, infrequent constructs, and texts nested within individuals.","fun_headline_variants_meta":{"raw":{"variants":["Wald and bootstrap miss 95 percent coverage for metrics","Agresti-Coull and Wilson fix under-coverage in classifiers","Hierarchical bootstrap needed for nested text data","Pseudo-count method improves F1 interval accuracy"]},"model":"grok-4.3","cost_usd":0.0036,"raw_usage":{"total_tokens":1934,"prompt_tokens":773,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":35999500,"prompt_tokens_details":{"text_tokens":773,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1101,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":773,"tokens_out":60,"duration_ms":10309,"temperature":1.0,"reasoning_tokens":1101,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T01:15:12.874853+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"An empirical check on a large collection of real labeled text datasets in which the true performance parameters can be treated as known, testing whether the recommended intervals achieve coverage rates close to 95 percent across repeated random subsamples.","supporting_citations":[],"review_version":1}