Pith. sign in

REVIEW 2 major objections 5 minor 92 references

PPAT estimates model risk with far fewer labels by residualising the loss with black-box predictions while remaining unbiased and producing valid confidence intervals.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 08:57 UTC pith:K4Y2YSK5

load-bearing objection Clean residualized LURE + residual-aware acquisition with real theory and solid experiments; the CLT assumption is the only soft spot and it is secondary. the 2 major comments →

arxiv 2607.08347 v1 pith:K4Y2YSK5 submitted 2026-07-09 stat.ML cs.LG

Prediction-Powered Active Testing

Kianoosh Ashouritaklimi , Valentin Kilian , Daolang Huang , Tom Rainforth , Fran\c{c}ois Caron This is my paper
classification stat.ML cs.LG
keywords active testingrisk estimationcontrol variatesprediction-powered inferenceLURElabel-efficient evaluationasymptotic confidence intervals
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When labels are expensive but a strong black-box predictor is cheap to run on the whole test pool, most active-testing methods either ignore those predictions or treat them as biased pseudo-labels. This paper introduces Prediction-Powered Active Testing (PPAT): it folds the proxy losses into a control variate that residualises the true loss inside the unbiased LURE estimator. The result stays unbiased for any fixed strength of correction, can strictly cut variance when the proxy tracks the true loss, and comes with a matching acquisition rule that samples points expected to shrink the residualised variance. The authors also prove asymptotic normality, so one can attach confidence intervals that empirically hit the target coverage with fewer labels and narrower widths than random sampling, plain LURE, or surrogate-imputation baselines on tabular regression and image classification.

Core claim

Combining the unbiased LURE importance-weighted estimator with a prediction-powered control variate yields an estimator of finite-pool risk that remains unbiased for any fixed correction strength λ, reduces variance whenever λ lies in an open interval around the optimal control-variate coefficient, and admits both a myopic oracle proposal proportional to absolute residualised loss and a practical surrogate approximation. Under uniform overlap and mild moment conditions the estimator is asymptotically normal, so plug-in standard errors produce asymptotically valid confidence intervals that, in experiments, reach nominal coverage faster and with smaller width than existing active-testing metho

What carries the argument

The PPAT estimator: LURE applied to residualised losses ℓ_i − λ(˜ℓ_i − ˜R), equivalently ˆR_LURE − λ ˆC_LURE, where the centred proxy-risk term is a zero-mean control variate. This single identity simultaneously preserves unbiasedness, supplies the variance formula that guides λ, and redefines the acquisition scores.

Load-bearing premise

The accumulated conditional variance of the martingale increments must settle to a finite positive constant; without that stabilisation the central-limit theorem and the confidence intervals do not hold for fully adaptive sampling.

What would settle it

On a fixed test pool with a known true risk, run many independent PPAT trajectories and check whether the empirical coverage of the nominal 90 percent intervals fails to approach 90 percent as the label budget grows, or whether the residualised estimator has higher variance than plain LURE for λ values the theory predicts should reduce it.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Risk estimates for black-box models can be made more label-efficient without sacrificing unbiasedness whenever a cheap proxy predictor is available on the full test pool.
  • Acquisition should target residualised rather than raw loss; reusing LURE proposals is suboptimal once the control variate is present.
  • Asymptotically valid confidence intervals for active testing become available as a by-product, including for ordinary LURE as the special case λ = 0.
  • Fixed heuristics λ = 1 or ½, or an online plug-in of the variance-bound minimiser λ†, are practical defaults that recover most of the gain when the proxy is informative.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same residualisation-plus-LURE pattern could be dropped into other pool-based evaluation pipelines (stratified sampling, model selection, dense vision metrics) without redesigning the importance weights.
  • If the proxy itself is allowed to update with acquired labels, the theory still applies because only the centred proxy losses enter the control variate; this opens a natural online-proxy extension the paper only sketches.
  • Poor surrogate tracking hurts LURE more than PPAT because residualisation shrinks the scale of the weighted terms, suggesting PPAT is a useful hedge when uncertainty estimates are unreliable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Prediction-Powered Active Testing (PPAT): an unbiased risk estimator that applies the LURE importance weights of Farquhar et al. to residualized losses ℓ_i − λ(˜ℓ_i − ˜R), where ˜ℓ_i are losses under a fixed black-box proxy. This is a control-variate correction of LURE (Props. 3.1–3.2). The authors derive a myopic oracle proposal proportional to the absolute residualized loss and a practical RMS surrogate score (Prop. 3.3, Eq. 12–13), discuss fixed and online choices of λ (including a variance-bound target λ†, Prop. 3.4), and prove consistency and asymptotic normality under uniform overlap and moment conditions, yielding plug-in asymptotic CIs (Thm. 4.1). Experiments on UCI regression and CLIP-based image classification show lower median squared error than LURE, ASE, and random sampling, with CIs that reach nominal coverage faster and with smaller width.

Significance. If the results hold, PPAT is a clean and useful extension of active testing: it keeps finite-sample unbiasedness while systematically exploiting cheap black-box predictions that are already available in modern evaluation pipelines. The residualized acquisition rule is a genuine contribution beyond simply bolting PPI onto LURE. The appendix develops a triangular-array LURE framework with martingale CLT and studentized CIs that also specialize to ordinary LURE—valuable theory that prior active-testing work largely lacked. Empirically, gains appear on both tabular regression and higher-dimensional classification, and the method is complementary to ASE and to recent LLM active-testing work. Strengths include explicit unbiasedness for any fixed λ, a transparent control-variate variance formula, and extensive ablations (proxy, surrogate, estimator vs acquisition).

major comments (2)
  1. [§4, Thm. 4.1, §E.5.5] §4 / Thm. 4.1 and §E.5.5: Asymptotic normality and the CIs in Eq. (16) require that the predictable quadratic variation σ²_M(λ) converge in probability to a finite positive limit. The appendix correctly states that this does not follow from pool-level moment bounds for fully adaptive proposals and is supported only indirectly by coverage plots. The main text still presents the intervals as “asymptotically valid” without that caveat. Because CIs are a headline claim (abstract, §1, §5.4), the main text should state the assumption, note that it is the standard martingale-CLT stabilisation condition, and make clear that validity under adaptive Q_m is empirically supported rather than proved from first principles.
  2. [§5.2, Eq. (12)–(13)] §5.2 (classification) and §3.2: On CIFAR/Tiny-ImageNet, Random substantially beats LURE, which the authors attribute to a surrogate that poorly tracks true losses (and thus produces harmful importance weights). PPAT still wins, but its proposal Q^{PPAT}_{m,λ} is built from the same surrogate via a_{m,λ} (Eq. 12). The claim that residualization makes variance “less sensitive to errors in the surrogate” is plausible and consistent with the figures, yet it is not formalized and is not stress-tested when both proxy and surrogate are weak (e.g., λ=1 on Tiny-ImageNet is only comparable to Random). A short analysis or ablation that isolates surrogate misspecification under residualized vs raw-loss proposals would strengthen the central “estimator + tailored acquisition” package.
minor comments (5)
  1. [§3.3] §3.3: The online ˆλ_M update every k=100 labels is a free hyperparameter; a brief sensitivity plot (or statement that results are stable for k ∈ {50,100,200}) would help practitioners.
  2. [§5.2] Fig. 1–2: Median squared error is the primary metric; mean error (bias) is only in the appendix. A one-sentence pointer in §5.2 to the bias plots for plug-in PPAT and ASE would help readers interpret early-budget transients.
  3. [§2.1, §3.2, §D.1] §D.1–D.2: The justification for RMS rather than E[|·|] surrogate scores is good but buried; a short forward reference from §2.1/§3.2 would improve readability.
  4. [§6, §A] Related work: Active Measurement (Hamilton et al., 2025) and ASI (Zrnic & Candès, 2024a) are discussed; the distinction that PPAT uses a fixed proxy as a control variate while AM updates the predictor for the proposal is clear in the appendix—consider one sentence of that contrast in the main §6.
  5. [§2.1, §E.2] Notation: V_m in the main text vs V_{M,m} in the appendix is standard but occasionally confuses when M is both budget and triangular-array index; a brief note would help.

Circularity Check

0 steps flagged

No significant circularity: PPAT is LURE on residualized losses; unbiasedness, variance formulas, myopic oracle, and asymptotics are derived from first principles without self-definitional loops or load-bearing self-citations.

full rationale

The derivation chain is self-contained and non-circular. PPAT is defined as the LURE estimator applied to residualized losses ℓ_i − λ(ℓ̃_i − R̃) (Eq. 8); unbiasedness (Prop. 3.1 / Cor. E.7) follows immediately because the control has pool mean zero, so the residualized array has the same finite-pool mean R as the raw losses. The variance identity (Prop. 3.2 / E.8) is the ordinary control-variate expansion; λ⋆ is the usual covariance ratio, not fitted to force a claim. The myopic oracle (Prop. 3.3 / E.10–E.11) is obtained by minimizing the conditional second-moment of the importance-weighted residual under the same Lagrange argument used for ordinary LURE, then substituting the residualized array. λ† is the population least-squares coefficient that flattens residuals and minimizes an explicit upper bound (Prop. 3.4 / E.9); the plug-in λ̂_M is a LURE estimate of the numerator and is shown consistent and asymptotically normal (Thm. E.18 / Cor. E.19), not assumed. Asymptotic normality (Thm. 4.1) invokes a standard martingale CLT under uniform overlap (enforced by construction) and moment conditions on the fixed pool; the only non-trivial hypothesis is stabilization of the predictable quadratic variation, which the paper states openly does not follow from pool moments alone for fully adaptive proposals and supports only empirically via coverage. Citations to Farquhar et al. (LURE) and Angelopoulos et al. (PPI) are external prior art used as building blocks, not self-citations that close a uniqueness or ansatz loop. No quantity is defined in terms of the quantity it is claimed to predict, no parameter is fitted to data and then re-presented as an independent prediction of a closely related functional, and no uniqueness theorem is imported from the authors’ own prior work. The empirical gains are evaluations against external baselines, not forced by construction. Score 0 is therefore the correct, proportionate finding.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The paper rests on standard finite-population sampling theory, the existing LURE unbiasedness/variance results, and classical martingale CLTs. Free parameters are the control-variate strength λ (or its plug-in), the uniform-mixing ε, and the update frequency k for ˆλ. No new physical entities are postulated; the only invented objects are the PPAT estimator itself and the residualized acquisition score, both fully defined from existing quantities.

free parameters (3)
  • λ (control-variate coefficient) = 1, 0.5, or ˆλ_M (init 0.5, update every 100 labels)
    Fixed choices λ=1 or 0.5, or the online plug-in ˆλ_M targeting λ†; variance reduction holds only inside an interval determined by the unknown correlation, so the practical choice is a free design parameter.
  • ε (uniform mixing for positivity)
    Small ε∈(0,1) mixed into the proposal to enforce the uniform-overlap condition required by all asymptotic results; value not specified numerically in experiments.
  • k (ˆλ update frequency) = 100
    ˆλ is held fixed between blocks of k newly acquired labels to stabilize the residualized proposal; set to 100 in experiments.
axioms (5)
  • domain assumption LURE is unbiased for any admissible active proposal process and its variance admits the stated martingale decomposition (Farquhar et al. 2021).
    Used as the starting point for residualization; all PPAT unbiasedness and variance claims inherit from it (§2.1, §E.2–E.3).
  • standard math Martingale central-limit theorem for triangular arrays (Billingsley 1995, Thm. 35.12) under Lindeberg and predictable-quadratic-variation convergence.
    Invoked to obtain asymptotic normality of √M(ˆR_PPAT − R) (§E.5.2).
  • domain assumption Uniform overlap: Q_m(i) ≥ β/N for some β>0 on the remaining pool (enforced by ε-mixing).
    Required for bounded importance weights, L² consistency, and the CLT; stated as an assumption of Thm. 4.1 and enforced in practice.
  • domain assumption Finite-pool empirical moments of true and proxy losses remain O(1) (or o(M)) as N,M→∞ with N/M→α>1.
    Moment conditions for consistency and normality; mild because the pool is treated as fixed (§E.5.5).
  • ad hoc to paper Myopic (one-step) variance minimization is a suitable proxy for the intractable finite-horizon optimal proposal.
    Standard in active testing; used to derive the closed-form oracle Q⋆∝|residual| (Prop. 3.3).
invented entities (2)
  • PPAT residualized LURE estimator ˆR_PPAT(λ) independent evidence
    purpose: Unbiased, lower-variance risk estimator that subtracts a centered proxy-loss control variate inside LURE weights.
    Defined by Eq. (8); fully constructive from existing LURE and PPI ingredients; no external ontological claim.
  • Residual-aware acquisition score a_{m,λ}(i) independent evidence
    purpose: Surrogate approximation to the myopic oracle that samples points expected to reduce variance of the residualized estimator.
    Eq. (12)–(13); practical implementation of the oracle derived in Prop. 3.3.

pith-pipeline@v1.1.0-grok45 · 53508 in / 3411 out tokens · 41867 ms · 2026-07-10T08:57:55.960344+00:00 · methodology

0 comments
read the original abstract

Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive. To address this, we propose \textbf{Prediction--Powered Active Testing (PPAT)}, a novel label--efficient risk estimation framework that combines the unbiased LURE estimator \citep{farquhar2021statistical} with a prediction--powered control variate. Rather than using proxy predictions as biased pseudo--labels, PPAT uses them to residualise the loss, preserving unbiasedness while reducing variance. Beyond the estimator itself, PPAT also changes which points should be acquired: we derive oracle and practical surrogate--based acquisition rules tailored to reducing the variance of our estimator. Moreover, we establish asymptotic normality for PPAT, yielding asymptotically valid confidence intervals and thus a principled estimate of the uncertainty around our estimates. Across tabular regression and image--classification tasks, PPAT outperforms existing methods in risk estimation, while its confidence intervals attain the target coverage with substantially fewer labels and smaller widths.

Figures

Figures reproduced from arXiv: 2607.08347 by Daolang Huang, Fran\c{c}ois Caron, Kianoosh Ashouritaklimi, Tom Rainforth, Valentin Kilian.

Figure 1
Figure 1. Figure 1: Regression experiments comparing PPAT, LURE, ASE, and Random. Plots show the median squared error across 1000 trials. Tiny-ImageNet, where PPAT with λ = 1 performs comparably to Random. We further note that Random outperforms LURE on these datasets, likely a result of the surrogate poorly tracking the losses. Indeed, since LURE builds its proposal, and hence its importance weights, from the surrogate’s exp… view at source ↗
Figure 2
Figure 2. Figure 2: Classification experiments comparing PPAT, LURE, ASE, and Random. Plots show the median squared error across 1000 trials. Overall, these results indicate that PPAT improves risk estimation not only for tabular regression, but also for more challenging, higher–dimensional image-classification tasks. PPAT PPAT Acq. + LURE PPAT Acq. + Empirical 0 200 400 Number of Acquired Test Points 10 5 10 4 10 3 10 2 10 1… view at source ↗
Figure 3
Figure 3. Figure 3: Experiments on UCI and classification datasets studying the influence of our estimator. We compare [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Experiments on UCI and classification datasets studying the influence of our acquisition strategy. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Representative experiments on UCI and classification datasets for the coverage of [PITH_FULL_IMAGE:figures/full_fig_p011_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Plot of PPAT for a range of fixed λ values and λ † for the datasets Keggdirected and CIFAR-10. λ † is computed on the full pool. The plots report the median squared error over 1000 trials [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation with different proxy models comparing [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablation with different surrogate models comparing [PITH_FULL_IMAGE:figures/full_fig_p029_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Experiments on UCI datasets comparing PPAT with Random, ASE, LURE. Plots show the mean error (bias) across 1000 trials. Note that ASE results in biased estimates of the risk. Across Figs. 9 and 10, ASE exhibits persistent non-zero mean error, as expected: ASE estimates the risk through the surrogate predictive model, and any surrogate misspecification can therefore translate directly 29 [PITH_FULL_IMAGE:f… view at source ↗
Figure 10
Figure 10. Figure 10: Experiments on classification datasets comparing [PITH_FULL_IMAGE:figures/full_fig_p030_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Experiments on UCI and classification datasets studying the influence of our estimator. We [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Experiments on UCI and classification datasets studying the influence of our acquisition strategy. [PITH_FULL_IMAGE:figures/full_fig_p031_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Representative experiments on UCI and classification datasets for the mean widths of the confidence [PITH_FULL_IMAGE:figures/full_fig_p031_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Experiments on UCI and classification datasets studying the influence of our acquisition strategy. [PITH_FULL_IMAGE:figures/full_fig_p032_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Experiments on Keggdirected and CIFAR-10 comparing PPAT with Random, ASE, Proxy, LURE. Plots show the median squared error across 1000 trials. Note that ASE and Proxy result in biased estimates of the risk. We report this comparison on Keggdirected and CIFAR-10, representing one regression dataset and one classification dataset. We use the same setup as in Figs. 1 and 2 which is described in detail in §B.… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

92 extracted references · 92 canonical work pages · 18 internal anchors

  1. [1]

    Probability and Measure , author =

    work page

  2. [2]

    Greenwade

    George D. Greenwade. The C omprehensive T ex A rchive N etwork ( CTAN ). TUGBoat. 1993

    work page 1993

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Active surrogate estimators: An active learning approach to label-efficient model evaluation , author=. Advances in Neural Information Processing Systems , volume=. 2022 , pages=

    work page 2022

  4. [4]

    International Conference on Machine Learning , pages=

    Active testing: Sample-efficient model evaluation , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    work page 2021

  5. [5]

    arXiv preprint arXiv:2508.09093 , year=

    Scaling up active testing to large language models , author=. arXiv preprint arXiv:2508.09093 , year=

    work page arXiv

  6. [6]

    Active Statistical Inference

    Active statistical inference , author=. arXiv preprint arXiv:2403.03208 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    Science , volume=

    Prediction-powered inference , author=. Science , volume=. 2023 , publisher=

    work page 2023

  8. [8]

    On Statistical Bias In Active Learning: How and When To Fix It

    On statistical bias in active learning: How and when to fix it , author=. arXiv preprint arXiv:2101.11665 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    Active Learning Literature Survey , type =

    Settles, Burr , biburl =. Active Learning Literature Survey , type =

    work page

  10. [10]

    Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    Want to reduce labeling cost? GPT-3 can help , author=. Findings of the Association for Computational Linguistics: EMNLP 2021 , pages=

    work page 2021

  11. [11]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

    work page

  12. [12]

    2004 , publisher=

    Monte Carlo Methods in Financial Engineering , author=. 2004 , publisher=

    work page 2004

  13. [13]

    and Casella, G

    Robert, C.P. and Casella, G. , keywords =. Monte

    work page

  14. [14]

    PPI++: Efficient Prediction-Powered Inference

    Ppi++: Efficient prediction-powered inference , author=. arXiv preprint arXiv:2311.01453 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Robust Statistics

    Huber, Peter J. Robust Statistics. International Encyclopedia of Statistical Science. 2011. doi:10.1007/978-3-642-04898-2_594

    work page doi:10.1007/978-3-642-04898-2_594 2011

  16. [16]

    TabPFN-2.5: Advancing the State of the Art in Tabular Foundation Models

    Tabpfn-2.5: Advancing the state of the art in tabular foundation models , author=. arXiv preprint arXiv:2511.08667 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    UCI Machine Learning Repository

    Dheeru, Dua and Karra Taniskidou, Efi. UCI Machine Learning Repository. 2017

    work page 2017

  18. [18]

    Proceedings of the National Academy of Sciences , volume=

    Cross-prediction-powered inference , author=. Proceedings of the National Academy of Sciences , volume=. 2024 , publisher=

    work page 2024

  19. [19]

    Advances in Neural Information Processing Systems , volume=

    Stratified prediction-powered inference for effective hybrid evaluation of language models , author=. Advances in Neural Information Processing Systems , volume=

    work page

  20. [20]

    Rasmussen, Carl Edward and Williams, Christopher K. I. , biburl =

    work page

  21. [21]

    1996 , address =

    Rasmussen, Carl Edward , title =. 1996 , address =

    work page 1996

  22. [22]

    Benchmarking the Neural Linear Model for Regression

    Benchmarking the neural linear model for regression , author=. arXiv preprint arXiv:1912.08416 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  23. [23]

    Recommendations for Baselines and Benchmarking Approximate Gaussian Processes

    Recommendations for baselines and benchmarking approximate Gaussian processes , author=. arXiv preprint arXiv:2402.09849 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    , title =

    Tipping, Michael E. , title =. 2001 , issue_date =. doi:10.1162/15324430152748236 , journal =

    work page doi:10.1162/15324430152748236 2001

  25. [25]

    and Varoquaux, G

    Pedregosa, F. and Varoquaux, G. and Gramfort, A. and Michel, V. and Thirion, B. and Grisel, O. and Blondel, M. and Prettenhofer, P. and Weiss, R. and Dubourg, V. and Vanderplas, J. and Passos, A. and Cournapeau, D. and Brucher, M. and Perrot, M. and Duchesnay, E. , journal=. Scikit-learn: Machine Learning in

    work page

  26. [26]

    Learning Multiple Layers of Features from Tiny Images , url =

    Krizhevsky, Alex , biburl =. Learning Multiple Layers of Features from Tiny Images , url =

    work page

  27. [27]

    2015 , url=

    Tiny ImageNet Visual Recognition Challenge , author=. 2015 , url=

    work page 2015

  28. [28]

    DINOv2: Learning Robust Visual Features without Supervision

    Dinov2: Learning robust visual features without supervision , author=. arXiv preprint arXiv:2304.07193 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  29. [29]

    Advances in neural information processing systems , volume=

    Bootstrap your own latent-a new approach to self-supervised learning , author=. Advances in neural information processing systems , volume=

    work page

  30. [30]

    International conference on machine learning , pages=

    A simple framework for contrastive learning of visual representations , author=. International conference on machine learning , pages=. 2020 , organization=

    work page 2020

  31. [31]

    Artificial Intelligence Review , year=

    Deep transfer learning for image classification: a survey , author=. Artificial Intelligence Review , year=

    work page

  32. [32]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Do better imagenet models transfer better? , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    work page

  33. [33]

    Understanding intermediate layers using linear classifier probes

    Understanding intermediate layers using linear classifier probes , author=. arXiv preprint arXiv:1610.01644 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  34. [34]

    Advances in neural information processing systems , volume=

    Laplace redux-effortless bayesian deep learning , author=. Advances in neural information processing systems , volume=

    work page

  35. [35]

    MacKay, David J. C. , title =. Neural Computation , volume =. 1992 , month =. doi:10.1162/neco.1992.4.3.415 , url =

    work page doi:10.1162/neco.1992.4.3.415 1992

  36. [36]

    International Conference on Learning Representations , year=

    A Scalable Laplace Approximation for Neural Networks , author=. International Conference on Learning Representations , year=

    work page

  37. [37]

    Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

    Reimers, Nils and Gurevych, Iryna. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. 2019

    work page 2019

  38. [38]

    Proceedings of the 27th International Conference on International Conference on Machine Learning , pages =

    Sawade, Christoph and Landwehr, Niels and Bickel, Steffen and Scheffer, Tobias , title =. Proceedings of the 27th International Conference on International Conference on Machine Learning , pages =. 2010 , isbn =

    work page 2010

  39. [39]

    Active Estimation of F-Measures , url =

    Sawade, Christoph and Landwehr, Niels and Scheffer, Tobias , booktitle =. Active Estimation of F-Measures , url =

    work page

  40. [40]

    Sample Efficient Model Evaluation

    Sample efficient model evaluation , author=. arXiv preprint arXiv:2109.12043 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  41. [41]

    Active Evaluation of Classifiers on Large Datasets , year=

    Katariya, Namit and Iyer, Arun and Sarawagi, Sunita , booktitle=. Active Evaluation of Classifiers on Large Datasets , year=

    work page

  42. [42]

    and Carvalho, Vitor R

    Bennett, Paul N. and Carvalho, Vitor R. , title =. 2010 , isbn =. doi:10.1145/1871437.1871677 , booktitle =

    work page doi:10.1145/1871437.1871677 2010

  43. [43]

    2022 , issue_date =

    Liu, Peng and Wang, Lizhe and Ranjan, Rajiv and He, Guojin and Zhao, Lei , title =. 2022 , issue_date =. doi:10.1145/3510414 , journal =

    work page doi:10.1145/3510414 2022

  44. [44]

    Training Connectionist Networks with Queries and Selective Sampling , url =

    Atlas, Les and Cohn, David and Ladner, Richard , booktitle =. Training Connectionist Networks with Queries and Selective Sampling , url =

    work page

  45. [45]

    1988 , issue_date =

    Angluin, Dana , title =. 1988 , issue_date =. doi:10.1023/A:1022821128753 , journal =

    work page doi:10.1023/a:1022821128753 1988

  46. [46]

    Gemini: A Family of Highly Capable Multimodal Models

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  47. [47]

    6G Non-Terrestrial Networks Enabled Low-Altitude Economy: Opportunities and Challenges

    6G non-terrestrial networks enabled low-altitude economy: Opportunities and challenges , author=. arXiv preprint arXiv:2311.09047 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  48. [48]

    LLaMA: Open and Efficient Foundation Language Models

    Llama: Open and efficient foundation language models , author=. arXiv preprint arXiv:2302.13971 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  49. [49]

    International conference on machine learning , pages=

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models , author=. International conference on machine learning , pages=. 2023 , organization=

    work page 2023

  50. [50]

    International conference on machine learning , pages=

    Scaling up visual and vision-language representation learning with noisy text supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    work page 2021

  51. [51]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Sigmoid loss for language image pre-training , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

  52. [52]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Active bayesian assessment of black-box classifiers , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    work page

  53. [53]

    Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages=

    Classifier risk estimation under limited labeling resources , author=. Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages=. 2018 , organization=

    work page 2018

  54. [54]

    International Conference on Machine Learning , pages=

    Active testing: An efficient and robust framework for estimating accuracy , author=. International Conference on Machine Learning , pages=. 2018 , organization=

    work page 2018

  55. [55]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Benchmarking large language model capabilities for conditional generation , author=. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  56. [56]

    tinyBenchmarks: evaluating LLMs with fewer examples

    tinyBenchmarks: evaluating LLMs with fewer examples , author=. arXiv preprint arXiv:2402.14992 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  57. [57]

    2024 , url=

    Gayathri Saranathan and Mahammad Parwez Alam and James Lim and Suparna Bhattacharya and Soon Yee Wong and Martin Foltin and Cong Xu , booktitle=. 2024 , url=

    work page 2024

  58. [58]

    Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Anchor points: Benchmarking models with much fewer examples , author=. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    work page

  59. [59]

    ACM Transactions on Software Engineering and Methodology , volume=

    Actracer: Active testing of large language model via multi-stage sampling , author=. ACM Transactions on Software Engineering and Methodology , volume=. 2026 , publisher=

    work page 2026

  60. [60]

    European Conference on Computer Vision , pages=

    MetaAT: Active Testing for Label-Efficient Evaluation of Dense Recognition Tasks , author=. European Conference on Computer Vision , pages=. 2024 , organization=

    work page 2024

  61. [61]

    Actively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice , url =

    Yu, Dayou and Shi, Weishi and Yu, Qi , booktitle =. Actively Testing Your Model While It Learns: Realizing Label-Efficient Learning in Practice , url =

    work page

  62. [62]

    Label-Efficient Model Selection for Text Generation

    Label-efficient model selection for text generation , author=. arXiv preprint arXiv:2402.07891 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  63. [63]

    NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World , year=

    Active Model Selection: A Variance Minimization Approach , author=. NeurIPS 2023 Workshop on Adaptive Experimental Design and Active Learning in the Real World , year=

    work page 2023

  64. [64]

    GitHub repository , howpublished =

    Trefor Evans , title =. GitHub repository , howpublished =. 2022 , publisher =

    work page 2022

  65. [65]

    GitHub repository , publisher =

    TorchVision: PyTorch's Computer Vision library , author =. GitHub repository , publisher =

    work page

  66. [66]

    Adam: A Method for Stochastic Optimization

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  67. [67]

    International conference on artificial intelligence and statistics , pages=

    Improving predictions of Bayesian neural nets via local linearization , author=. International conference on artificial intelligence and statistics , pages=. 2021 , organization=

    work page 2021

  68. [68]

    Nature , year=

    Accurate predictions on small data with a tabular foundation model , author=. Nature , year=. doi:10.1038/s41586-024-08328-6 , publisher=

    work page doi:10.1038/s41586-024-08328-6

  69. [69]

    arXiv preprint arXiv:2507.01372 , year=

    Active Measurement: Efficient Estimation at Scale , author=. arXiv preprint arXiv:2507.01372 , year=

    work page arXiv

  70. [70]

    Cost-Optimal Active AI Model Evaluation

    Cost-optimal active ai model evaluation , author=. arXiv preprint arXiv:2506.07949 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  71. [71]

    Revisiting Active Sequential Prediction-Powered Mean Estimation

    Revisiting Active Sequential Prediction-Powered Mean Estimation , author=. arXiv preprint arXiv:2604.18569 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  72. [72]

    Sampling Techniques , author=

    work page

  73. [73]

    Robins and Andrea Rotnitzky and Lue Ping Zhao , journal =

    James M. Robins and Andrea Rotnitzky and Lue Ping Zhao , journal =. Estimation of Regression Coefficients When Some Regressors Are Not Always Observed , urldate =

    work page

  74. [74]

    Biometrics , author =

    Bang, Heejung and Robins, James M. , title =. Biometrics , volume =. 2005 , month =. doi:10.1111/j.1541-0420.2005.00377.x , url =

    work page doi:10.1111/j.1541-0420.2005.00377.x 2005

  75. [75]

    nature , volume=

    Array programming with NumPy , author=. nature , volume=. 2020 , publisher=

    work page 2020

  76. [76]

    Nature methods , volume=

    SciPy 1.0: fundamental algorithms for scientific computing in Python , author=. Nature methods , volume=. 2020 , publisher=

    work page 2020

  77. [77]

    Hunter, J. D. , Title =. Computing in Science & Engineering , Volume =

    work page

  78. [78]

    doi:10.5281/zenodo.3509134 , url =

    The pandas development team , title =. doi:10.5281/zenodo.3509134 , url =

    work page doi:10.5281/zenodo.3509134

  79. [79]

    SciPy 2010 , year =

    McKinney, Wes , title =. SciPy 2010 , year =. doi:10.25080/Majora-92bf1922-00a , url =

    work page doi:10.25080/majora-92bf1922-00a 2010

  80. [80]

    Proceedings of the 38th International Conference on Machine Learning , series =

    Learning Transferable Visual Models From Natural Language Supervision , author =. Proceedings of the 38th International Conference on Machine Learning , series =. 2021 , url =

    work page 2021

Showing first 80 references.