Pith. sign in

REVIEW 5 minor 50 references

This paper proves that certifying small missed relevant mass in a high-recall filter is possible only by auditing the excluded pool, and that a simple excluded-pool audit is minimax rate-optimal, with exact finite-sample certificates.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 07:19 UTC pith:YZYXWZ2Z

load-bearing objection A solid theory paper: the new minimax lower bound for excluded-pool auditing is the real contribution, and the proofs hold up under close reading.

arxiv 2607.21480 v1 pith:YZYXWZ2Z submitted 2026-07-23 cs.LG stat.ML

Finite-Sample Coverage Audits for High-Recall Candidate Generation: Certification and Learning-Theoretic Design

Martin Anthony , Kaveh Salehzadeh Nobari This is my paper
classification cs.LG stat.ML
keywords high-recall candidate generationmissed mass certificationaudit label complexityexcluded-pool samplingfinite-sample confidence boundshypergeometric inversionminimax optimalitydesign-certification separation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

High-recall candidate generation—the first filter that decides which items pass to review—loses every relevant item it misses. This paper asks how many audit labels are needed to certify that the missed relevant mass is small. It proves three things: labels drawn only from inside the candidate set cannot certify any nontrivial missed-mass bound; any valid audit in a fixed corpus must inspect on the order of N0/m excluded-pool items, even if adaptive and allowed to label the whole included pool; and a simple zero-count excluded-pool audit matches that rate, making excluded-pool auditing minimax optimal. The paper then turns this principle into exact binomial and hypergeometric certificates for missed mass, two-pool recall, simultaneous prefixes, and stress tests, all under one discipline: fix the generator family and audit rule before opening certification labels.

Core claim

The central discovery is a label-complexity characterisation of missed-mass certification. In the finite-corpus model, certifying M0(g) ≤ m−1 with confidence 1−δ when the excluded pool truly contains no relevant items requires, in expectation, at least (1−β−δ)N0/m excluded-pool items to be inspected, regardless of adaptivity or included-pool labelling (Theorem 13). The non-adaptive zero-count audit—sample uniformly from the excluded pool and certify if no relevant item appears—attains the same Θ(N0/m) rate up to constants (Corollary 15). The matching upper and lower bounds make excluded-pool auditing not merely convenient but provably optimal. On the construction side, the paper develops exa

What carries the argument

The argument turns on the decomposition r(g) = p0(g)·η(g): the missed relevant mass equals the excluded-pool mass times the relevance rate inside the excluded pool. Since p0(g) is observable without labels, certification reduces to bounding η(g), which requires sampling the excluded pool. Validity comes from exact binomial tail inversion with random effective sample sizes handled by a conditional binomial lemma, and hypergeometric inversion for sampling without replacement. The lower bound uses a planted-set coupling: compare a zero-miss labelling with a labelling that makes a random m-subset of the excluded pool relevant; any audit that cannot distinguish them must have inspected enough dis

Load-bearing premise

The certificates and the lower bound all assume the audit oracle returns true relevance labels with no noise; if labels are noisy, the planted-set coupling no longer guarantees identical audit trajectories, so the Ω(N0/m) lower bound is not established.

What would settle it

With N0=1000 excluded items, m=10, and β=δ=0.05, Theorem 13 predicts any valid noiseless audit must inspect at least 90 excluded items on average. A concrete falsifier: run an adaptive auditor that inspects only 50 excluded items but also labels included items, and check whether it certifies M0≤9 in zero-miss corpora with probability ≥0.95; succeeding would refute the theorem. Alternatively, introduce 1% independent label noise in the excluded pool and see whether any procedure achieves o(N0/m) average inspections while remaining valid—if it does, the noiseless assumption is carrying the resul

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In a fixed corpus, a zero-count excluded-pool audit of n0 items certifies M0(g) ≤ M_U(0, δ/|G|) missed relevant items with no asymptotic approximation; 300 excluded-pool labels in the paper's example certify missed mass below one percent.
  • An audit that labels only included items cannot support any missed-mass or recall claim; reporting a recall value requires excluded-pool sampling or a fully reviewed included pool plus a denominator bound.
  • Two-pool auditing converts the missed-mass certificate into a lower bound on recall by combining an excluded-pool upper bound with an included-pool lower bound.
  • A pre-specified family of nested prefix generators can be certified simultaneously from one shared reference sample, and a fixed target allows a fixed-sequence stopping rule that keeps each test at level δ rather than paying a δ/M multiplicity penalty.
  • Design-stage selection from a class (finite, VC, sparse unions) can be separated from certification, so the audit labels never influence which generator is chosen.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The Ω(N0/m) lower bound gives a concrete cost–benefit trade-off: halving the tolerated missed relevant count m roughly doubles the required excluded-pool labels, so certification targets can be priced before auditing begins.
  • The same planted-set lower bound plausibly transfers to other coverage problems—for instance, certifying that a learned retriever does not miss query-relevant passages—where the 'excluded pool' is the set of candidates not returned; the minimax rate should be identical.
  • In noisy labelling environments, the paper's own extension shows the certificate must be widened by a known sensitivity floor; deployments with unmeasured label noise should treat any certificate as optimistic and plan adjudication.
  • The fixed-sequence stopping result suggests a practical pre-registration protocol: pick the prefix cut points and the target before opening labels, share one reference sample across prefixes, and report the least burdensome certified prefix.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 5 minor

Summary. The paper studies finite-sample auditing of the missed relevant mass of a fixed high-recall candidate generator. It first shows that labels from the included pool alone cannot certify a nontrivial missed-mass bound (Proposition 3), then proves a finite-corpus lower bound: any valid audit that certifies fewer than m missed relevant items with high probability in the zero-miss case must inspect Ω(N0/m) excluded-pool items on average, even under adaptivity (Theorem 13). A non-adaptive zero-count excluded-pool audit matches this rate, giving a minimax optimality statement (Corollary 15). The rest of the paper develops exact binomial and hypergeometric certificates for missed mass, two-pool recall, pre-specified prefix families, shared-reference designs, fixed-target stopping without multiplicity penalty, auditable coverage amplification, stress-test cluster certificates, and a design-stage layer using standard PAC/VC/Neyman-Pearson bounds. All guarantees are explicitly conditional on the candidate generator or generator family being fixed before certification labels are examined (Assumption 1) and on noiseless audit labels (Assumption 2).

Significance. If the results hold, this is a significant contribution to the theory of coverage audits for candidate generation. The lower bound in Theorem 13 and the matching upper bound in Corollary 15 are the first label-complexity characterisation for missed-mass certification in the zero-miss regime, and they establish that excluded-pool auditing is minimax rate-optimal rather than merely convenient. The exact finite-sample certificates (Theorems 7, 8, 10, 17, 19, 24, 29) are useful and are proved cleanly from standard binomial/hypergeometric inversion. The paper is disciplined about the design-certification separation and clearly states its scope: the lower bound does not extend to noisy labels, and Section 9 only partially extends the i.i.d. excluded-pool certificate to noise. The proofs are standard but carefully executed; I found no load-bearing technical error. The numerical worked example is reproducible from the stated formulas and helps make the practical workflow concrete.

minor comments (5)
  1. [Assumption 2 / §9] The noiseless-label assumption is explicit, and Section 9 correctly notes that the lower bound Theorem 13 and the recall/stress-test certificates are not established under label noise. This is a scope limitation, not a flaw, but the abstract's phrasing 'how many audit labels are needed' could be read more broadly than the noiseless-oracle model. I suggest adding one sentence to the abstract or introduction stating that the optimality result assumes an exact labelling oracle.
  2. [§4.1, paragraph after Theorem 7] The Hoeffding upper bound p0(g) ≤ \hat p0(g) + sqrt(log(2|G|/δ)/(2m)) should be truncated at 1. As written, the displayed upper confidence expression can exceed 1 when the excluded pool is large and the sample is small. The theorem statement is unaffected, but the text should explicitly cap the bound at one.
  3. [Theorem 8 / Corollary 18] The hypergeometric inversion M_U(k, α; N, n) is used with n = n_eff = 0 in the shared-reference design, where the proof says it returns the trivial bound N. Please state the convention M_U(k, α; N, 0) = N in the theorem/corollary statement itself rather than only in the proof, so readers do not have to infer it.
  4. [§5.2 / Remark 25] The sampling requirement for Theorem 24 and Proposition 26 is important: positives obtained by reviewing only the included pool are not distributed as P(· | Y=1) and do not satisfy the hypothesis. This caveat is currently in a remark after the theorem; consider moving it to a 'Sampling requirement' paragraph immediately after the theorem statement to prevent misapplication.
  5. [§7.5, Remark 39] The remark on precision mentions that a lower bound on B(g) is needed but does not give the corresponding finite-sample construction. A one-sentence reference to the same Hoeffding bound used after Theorem 7 would make this self-contained.

Circularity Check

0 steps flagged

No significant circularity: the central lower and upper bounds are derived independently from stated assumptions, and the only same-author citation is illustrative, not load-bearing.

full rationale

The central claim (Theorem 13 and Corollary 15) is not circular. Theorem 13 is a lower bound proved by a planted-set coupling: it starts from two explicit assumptions (validity when M0(g) ≥ m, and success when M0(g) = 0), couples the zero-miss labelling with a random planted m-subset, and derives E_A T ≥ (1−β−δ) N0/m. The proof does not presuppose the form of any excluded-pool audit or the N0/m rate. The matching upper bound (Remark 14 / Corollary 15) is a separate construction: the non-adaptive zero-count audit's error probability is bounded directly by the hypergeometric tail inequality (N0−m choose n0)/(N0 choose n0) ≤ (1−n0/N0)^m, and its sample size n0 = N0(1−δ^{1/m}) is then shown to be O((N0/m) log(1/δ)). Thus the lower and upper bounds are derived from independent arguments that meet only in the final Θ statement. Proposition 3 is also an indistinguishability argument valid for every measurable relevance indicator, not a restatement of its conclusion. The audit certificates in Sections 4–6 are exact Clopper–Pearson or hypergeometric inversions applied to the stated sampling designs; no fitted parameter is renamed as a prediction, and the worked example in Section 8 is a deterministic calculation. The only same-author citation, [4], is used in Section 6.1 solely as an illustrative stress-test generator: 'A particularly relevant instance is the two-model semantic-adversarial framework of [4].' It is not used to justify any theorem, and the other overlapping citation, [3], is a standard textbook used for standard PAC bounds. Assumptions 1 and 2 are explicit validity and noiseless-label conditions, and Section 9 candidly states that the noisy-label extension is limited; these are limitations, not circular dependencies. No equation in the paper is equivalent to its inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 6 axioms · 0 invented entities

The central claim rests on the explicit Assumptions 1 and 2, the fixed-corpus/i.i.d. population model, and the conditional-i.i.d. positive sampling in Sections 5-6. No free parameters are fitted to data; the audit sizes and error allocations are chosen by the analyst rather than estimated.

axioms (6)
  • domain assumption Assumption 1 (Design-Certification Separation): the candidate generator or pre-specified family and the audit rule are fixed before certification labels are seen.
    Section 2.3. Every certificate's sampling distribution is only valid under this pre-registration; if violated, the bound is void. The lower bound also assumes the audit procedure is fixed independently of the labelling.
  • domain assumption Assumption 2 (Noiseless Audit Labels): the audit oracle returns true relevance labels with no noise.
    Section 2.3. All theorems except the Section 9 sensitivity-floor extension assume exact labels; the minimax lower bound (Thm. 13) uses identity of observed labels under the two coupled labellings.
  • domain assumption Population model: (X,F,P), measurable φ*, and candidate generator g fixed; in the finite-corpus model, corpus, labels, and generator are fixed and randomness is only in the audit sample.
    Section 2. The probability statements refer only to audit/design randomness, not to the corpus or labels.
  • domain assumption For Sections 5-6, audited positives are conditionally i.i.d. from P(·|Y=1) given the realised count.
    Remark 25: positives sampled only from the included pool do not satisfy this and would invalidate the binomial law for the residual counts.
  • standard math For Theorem 13, the audit is non-anticipating with an almost surely finite stopping time; the random seed R and revealed labels determine the trajectory.
    Section 4.5. This is the formal definition of the adaptive audit used in the lower bound; it is a standard formalization.
  • standard math Standard measurability and uniform-convergence conditions for infinite VC classes in Section 7.
    Footnote in Section 7: suprema of empirical processes are assumed measurable; VC uniform convergence bounds are used as black boxes.

pith-pipeline@v1.3.0-alltime-deepseek · 34577 in / 20200 out tokens · 176197 ms · 2026-08-01T07:19:11.379571+00:00 · methodology

0 comments
read the original abstract

An initial high-recall stage in an empirical pipeline decides which items pass to later review, labelling, or modelling, and relevant items it misses are lost to every subsequent stage. We study how many audit labels are needed to certify, with finite-sample validity, that this missed relevant mass is small, and our main results characterise the label complexity of this problem. We first show that no procedure using only labels from inside the candidate set can certify any non-trivial bound on the missed mass: the audit must sample the excluded pool, the only region where unrecovered relevant items can lie. We then prove a matching finite-corpus lower bound. Any valid audit that certifies fewer than $m$ missed relevant items with high probability when none are present, even if adaptive and permitted to label the entire included pool, must inspect on the order of $N_0/m$ excluded-pool labels. Excluded-pool auditing is therefore minimax rate-optimal, not merely convenient, for missed-mass certification in the zero-miss regime. Building on this characterisation, we develop an exact finite-sample toolkit, using binomial and hypergeometric inversion rather than asymptotic approximation, that certifies missed mass, converts it to recall through a two-pool design, certifies pre-specified families of nested candidate generators simultaneously, and produces stress-test certificates against declared perturbation mechanisms. These certificates can be paired with observable review burden to select the least burdensome pre-specified candidate generator meeting a missed-mass target. Every guarantee holds under one discipline: the candidate generator, or the pre-specified family from which it is selected, and the audit rule are fixed before the certification labels are examined.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    A. N. Angelopoulos, S. Bates, E. J. Cand` es, M. I. Jordan, and L. Lei. Learn then test: Calibrating predictive algorithms to achieve risk control.Annals of Applied Statistics, 19(2):1641–1662, 2025. 41

    2025

  2. [2]

    A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster. Conformal risk control. In International Conference on Learning Representations (ICLR), 2024

    2024

  3. [3]

    Anthony and P

    M. Anthony and P. L. Bartlett.Neural Network Learning: Theoretical Foundations. Cam- bridge University Press, 1999

    1999

  4. [4]

    Generalised Eigenvalue Geometry of Semantic Adversarial Attacks

    M. Anthony and K. Salehzadeh Nobari. Generalised eigenvalue geometry of semantic ad- versarial attacks.arXiv preprintarXiv:2606.19212, 2026.https://doi.org/10.48550/ arXiv.2606.19212

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.19212 2026

  5. [5]

    P. Bauer. Multiple testing in clinical trials.Statistics in Medicine, 10(6):871–890, 1991

    1991

  6. [6]

    Blumer, A

    A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K. Warmuth. Learnability and the Vapnik– Chervonenkis dimension.Journal of the ACM, 36(4):929–965, 1989

    1989

  7. [7]

    M. W. Callaghan and F. M¨ uller-Hansen. Statistical stopping criteria for automated screen- ing in systematic reviews.Systematic Reviews, 9:Article 273, 2020

    2020

  8. [8]

    Cannon, J

    A. Cannon, J. Howse, D. Hush, and C. Scovel. Learning with the Neyman–Pearson and min-max criteria. Technical Report LA-UR-02-2951, Los Alamos National Laboratory, 2002

    2002

  9. [9]

    C. J. Clopper and E. S. Pearson. The use of confidence or fiducial limits illustrated in the case of the binomial.Biometrika, 26(4):404–413, 1934

    1934

  10. [10]

    G. V. Cormack and M. R. Grossman. Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. InProceedings of the 37th International ACM SIGIR Conference (SIGIR 2014), pp. 153–162, 2014

    2014

  11. [11]

    G. V. Cormack and M. R. Grossman. Autonomy and reliability of continuous active learning for technology-assisted review.arXiv preprintarXiv:1504.06868, 2015.https://doi.org/ 10.48550/arXiv.1504.06868

    Pith/arXiv arXiv doi:10.48550/arxiv.1504.06868 2015

  12. [12]

    G. V. Cormack and M. R. Grossman. Engineering quality and reliability in technology- assisted review. InProceedings of the 39th International ACM SIGIR Conference (SIGIR 2016), pp. 75–84, 2016

    2016

  13. [13]

    G. V. Cormack and M. Mojdeh. Machine learning for information retrieval: TREC 2009 Web, Relevance Feedback and Legal Tracks. InProceedings of the 18th Text REtrieval Conference (TREC 2009), 2009

    2009

  14. [14]

    Devroye and G

    L. Devroye and G. L. Wise. Detection of abnormal behavior via nonparametric estimation of the support.SIAM Journal on Applied Mathematics, 38(3):480–488, 1980

    1980

  15. [15]

    Ehrenfeucht, D

    A. Ehrenfeucht, D. Haussler, M. Kearns, and L. Valiant. A general lower bound on the number of examples needed for learning.Information and Computation, 82(3):247–261, 1989

    1989

  16. [16]

    Eisenstat and D

    D. Eisenstat and D. Angluin. The VC dimension ofk-fold union.Information Processing Letters, 101(5):181–184, 2007

    2007

  17. [17]

    Facet-Level Tracing of Evidence Uncertainty and Hallucination in RAG

    P. Elchafei, M. Swain, S. Masoudian, and M. Schedl. Facet-level tracing of evidence uncertainty and hallucination in RAG.arXiv preprintarXiv:2604.09174, 2026.https: //doi.org/10.48550/arXiv.2604.09174

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.09174 2026

  18. [18]

    Freund and R

    Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting.Journal of Computer and System Sciences, 55(1):119–139, 1997. 42

    1997

  19. [19]

    M. R. Grossman and G. V. Cormack. The Grossman–Cormack glossary of technology- assisted review.Federal Courts Law Review, 7(1):1–34, 2013

    2013

  20. [20]

    M. R. Grossman and G. V. Cormack. Vetting and validation of AI-enabled tools for elec- tronic discovery. In J. Presser, J. Beatson, and G. Chan (eds.),Litigating Artificial Intelli- gence, chapter 13. Emond Publishing, 2021

    2021

  21. [21]

    J. A. Hanley and A. Lippman-Hand. If nothing goes wrong, is everything all right? Inter- preting zero numerators.Journal of the American Medical Association, 249(13):1743–1745, 1983

    1983

  22. [22]

    J. J. Heckman. Sample selection bias as a specification error.Econometrica, 47(1):153–161, 1979

    1979

  23. [23]

    S. R. Howard, A. Ramdas, J. McAuliffe, and J. Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences.Annals of Statistics, 49(2):1055–1080, 2021

    2021

  24. [24]

    J. C. Hsu and R. L. Berger. Stepwise confidence intervals without multiplicity adjustment for dose-response and toxicity studies.Journal of the American Statistical Association, 94(446):468–482, 1999

    1999

  25. [25]

    K. K. G. Lan and D. L. DeMets. Discrete sequential boundaries for clinical trials. Biometrika, 70(3):659–663, 1983

    1983

  26. [26]

    Lease, G

    M. Lease, G. V. Cormack, A. T. Nguyen, T. A. Trikalinos, and B. C. Wallace. Systematic review is e-discovery in doctor’s clothing. InSIGIR 2016 MedIR Workshop, 2016

    2016

  27. [27]

    D. D. Lewis, E. Yang, and O. Frieder. Certifying one-phase technology-assisted reviews. InProceedings of the 30th ACM International Conference on Information and Knowledge Management (CIKM 2021), pp. 893–902, 2021

    2021

  28. [28]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K¨ uttler, M. Lewis, W. Yih, T. Rockt¨ aschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. InAdvances in Neural Information Processing Systems 33 (NeurIPS 2020), pp. 9459–9474, 2020

    2020

  29. [29]

    R. J. A. Little and D. B. Rubin. Statistical analysis with missing data. John Wiley & Sons, 2019

    2019

  30. [30]

    Liu and B

    A. Liu and B. D. Ziebart. Robust classification under sample selection bias.Advances in Neural Information Processing Systems, 27, 2014

    2014

  31. [31]

    Magdy and G

    W. Magdy and G. J. F. Jones. PRES: A score metric for evaluating recall-oriented informa- tion retrieval applications. InProceedings of the 33rd Annual International ACM SIGIR Conference (SIGIR 2010), pp. 611–618, 2010

    2010

  32. [32]

    Maurer, L

    W. Maurer, L. Hothorn, and W. Lehmacher. Multiple comparisons in drug clinical trials and preclinical assays: a priori ordered hypotheses. In J. Vollmar (ed.),Biometrie in der chemisch-pharmazeutischen Industrie, volume 6, pp. 3–18. Fischer, Stuttgart, 1995

    1995

  33. [33]

    B. K. Natarajan. On learning Boolean functions. InProceedings of the 19th Annual ACM Symposium on Theory of Computing (STOC 1987), pp. 296–304, 1987

    1987

  34. [34]

    G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. An analysis of approximations for maxi- mizing submodular set functions—I.Mathematical Programming, 14:265–294, 1978

    1978

  35. [35]

    W. Polonik. Measuring mass concentrations and estimating density contour clusters: an excess mass approach.Annals of Statistics, 23(3):855–881, 1995. 43

    1995

  36. [36]

    Rigollet and X

    P. Rigollet and X. Tong. Neyman–Pearson classification, convexity and stochastic con- straints.Journal of Machine Learning Research, 12:2831–2855, 2011

    2011

  37. [37]

    E. G. Schilling and D. V. Neubauer.Acceptance Sampling in Quality Control. Chapman and Hall/CRC, third edition, 2017

    2017

  38. [38]

    Scott and R

    C. Scott and R. Nowak. A Neyman–Pearson approach to statistical learning.IEEE Trans- actions on Information Theory, 51(11):3806–3819, 2005

    2005

  39. [39]

    Scott and R

    C. Scott and R. Nowak. Learning minimum volume sets.Journal of Machine Learning Research, 7:665–704, 2006

    2006

  40. [40]

    Sneyd and M

    A. Sneyd and M. Stevenson. Stopping criteria for technology-assisted reviews based on counting processes. InProceedings of the 44th International ACM SIGIR Conference (SI- GIR 2021), pp. 2293–2297, 2021

    2021

  41. [41]

    Sviridenko

    M. Sviridenko. A note on maximizing a submodular set function subject to a knapsack constraint.Operations Research Letters, 32(1):41–43, 2004

    2004

  42. [42]

    X. Tong. A plug-in approach to Neyman–Pearson classification.Journal of Machine Learn- ing Research, 14:3011–3040, 2013

    2013

  43. [43]

    A. B. Tsybakov. On nonparametric estimation of density level sets.Annals of Statistics, 25(3):948–969, 1997

    1997

  44. [44]

    F. Tuyl, R. Gerlach, and K. Mengersen. The rule of three, its variants and extensions. International Statistical Review, 77(2):266–275, 2009

    2009

  45. [45]

    W. Uhlmann. Vergleich der hypergeometrischen mit der Binomial-Verteilung.Metrika, 10(1):145–158, 1966

    1966

  46. [46]

    L. G. Valiant. A theory of the learnable.Communications of the ACM, 27(11):1134–1142, 1984

    1984

  47. [47]

    V. N. Vapnik and A. Y. Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities.Theory of Probability and its Applications, 16(2):264–280, 1971

    1971

  48. [48]

    W. Webber. Approximate recall confidence intervals.ACM Transactions on Information Systems, 31(1):Article 2, 2013

    2013

  49. [49]

    R. M. Willett and R. D. Nowak. Minimax optimal level-set estimation.IEEE Transactions on Image Processing, 16(12):2965–2979, 2007

    2007

  50. [50]

    E. Yang, D. D. Lewis, and O. Frieder. Heuristic stopping rules for technology-assisted review. InProceedings of the 21st ACM Symposium on Document Engineering (DocEng 2021), pp. 31:1–31:10, 2021. 44

    2021