Pith. sign in

REVIEW 3 major objections 5 minor 81 references

Survey of Active Learning Hyperparameters: Insights from a Large-Scale Experimental Grid

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In active learning, the strategy implementation can matter more than the strategy itself, and about 4,000 varied runs can reproduce full-grid rankings.

desk verdict The field's largest AL hyperparameter grid, with code and raw data released, is worth refereeing; but the flagship 'implementation beats strategy' number (QUIRE at 44.5%) may be an artifact of zero-imputation and needs a complete-case check. read the letter →

arxiv 2506.03817 v1 pith:XI2EJMRM submitted 2025-06-04 cs.LG

Julius Gonsior , Tim Rie{ss} , Anja Reusch , Claudio Hartmann , Maik Thiele , Wolfgang Lehner This is my paper
classification cs.LG
keywords activelearninghyperparametergridquerystrategyimplementationreproducibilitybenchmarkdesignleaderboardrankingcorrelationtabularclassificationexperimentalsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to explain why active learning (AL) experiments so often contradict one another by measuring the influence of every major AL hyperparameter at once. The authors ran a grid of 4.6 million hyperparameter combinations, covering 92 tabular datasets, three classical learners, and 28 implementations of query strategies, and analyzed how each choice changes the resulting rankings. Their central claim is that the concrete implementation of an AL strategy can change results more than the underlying strategy itself, with two implementations of the same QUIRE strategy correlating at only 44.5%. They also argue that roughly 4,000 well-varied hyperparameter combinations are enough to reproduce the rankings of the full grid, making controlled, reproducible AL studies computationally feasible. If true, comparisons of AL strategies that ignore implementation and hyperparameter choices should not be trusted, and benchmark conclusions need to be re-examined.

What carries the argument

The paper's central object is a dense experimental grid over the hyperparameter tuple E = (S, D, T, I, b, L), with 92 datasets, 5 train-test splits, 20 start sets, 28 strategy implementations, 6 batch sizes, and 3 learner models, totaling 4,636,800 combinations and 4,560,876 completed runs. The argument is carried by three correlation instruments: metric-based heatmaps using Pearson correlation between aggregated metric vectors; queried-samples-based heatmaps using 1 - Jaccard similarity between the sets of samples selected for labeling; and leaderboard ranking-invariance heatmaps using two-sided Kendall's tau-b between strategy rankings across datasets. The first two measure whether outcomes and selected samples are the same, while the third measures whether conclusions change, which is the aspect that matters for reproducibility. The same machinery yields the grid-subsample analysis, where Spearman correlation between rankings from random subsets of growing size and rankings from the full grid identifies the roughly 4,000-combination stability threshold.

What would settle it

Take two implementations of the same strategy, for example the two QUIRE variants, and run them on a fresh collection of, say, 60 datasets from other domains including images and text; if their metric correlation is consistently above 0.8 rather than the reported 44.5%, the implementation-dominance claim would not generalize, and if the 4,000-combination stability threshold fails on those domains, the recommended study design would need revision.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that AL experiment outcomes are governed by the full tuple of hyperparameters, and that tuple cannot be collapsed to a single 'which strategy' choice. Using metric-based Pearson correlations, queried-sample Jaccard similarities, and leaderboard ranking-invariance correlations, the authors find that same-named strategies from different codebases can behave almost independently (QUIRE at 44.5% metric correlation), while strategies from the same framework correlate strongly (one framework's self-correlation was 89.8%), so implementation and framework design choices can outweigh the strategic principle. They consequently recommend reporting and varying implementations explicitly, and they show that a random subset of about 4,000 combinations drawn from a varied grid yields leaderboard correlations of 80 to 100 percent with the full 4.6-million-combination grid, while smaller subsets scatter widely. They also find that batch size and learner model are high-impact, that train-test split and start set can be fixed without loss, and that the F1-score aggregated by the full mean is the safest evaluation metric.

Load-bearing premise

The grid built from 92 tabular datasets, three classical learners, and 28 strategy implementations is assumed to be representative enough that the observed influence patterns and the roughly 4,000-combination sufficiency threshold generalize to active learning as a whole, including deep learning and non-tabular data, even though about 75,924 slow combinations were excluded and imputed as zero.

Editorial extensions

If this is right

  • If the central claim holds, any AL benchmark that varies strategies while fixing a single implementation is measuring a mix of strategy and implementation, so two papers using the same strategy in different codebases can reach opposite conclusions without either being wrong.
  • A study that draws a varied random grid of about 4,000 hyperparameter combinations can reproduce the full-grid leaderboard to 80 to 100 percent correlation, so reproducibility does not require millions of runs.
  • Batch size and learner model should be varied deliberately, with at least one small and one large batch size and the MLP as the safest single learner choice, while train-test splits and start sets can be fixed.
  • The full-mean F1-score is the recommended evaluation metric, because ramp-up-only or plateau-only aggregation can change conclusions and the phase boundary is dataset-dependent.
  • Strategy families such as uncertainty-based, diversity-based, and combined showed no consistent performance grouping, so recommendations should target concrete implementations and settings rather than family labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a cheap robustness test for future AL papers: before claiming strategy A beats strategy B, re-run both in at least two independent implementations and require the ordering to survive the implementation change; otherwise the result is about code, not method.
  • The 4,000-combination threshold could be turned into a practical bootstrap protocol: draw small varied subsets, check ranking stability, and stop growing the grid once correlation saturates, which would reduce benchmarking cost for new AL methods.
  • The finding that two random baselines selected the same samples yet produced different metrics (100 percent query overlap but only 78.5 percent metric correlation) implies that even a random baseline is implementation-sensitive through sample ordering, so baselines need their iteration order fixed and reported.
  • One open test is whether implementation dominance persists for deep neural network settings, where uncertainty estimation varies more across frameworks than for MLP, SVM, and RF; if it weakens, the conclusion is specific to classical learners on tabular data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a large-scale active learning (AL) benchmark: a hyperparameter grid of 4,636,800 combinations built from 92 tabular datasets, 28 AL strategy implementations from five frameworks, three learners, six batch sizes, five train-test splits, and 20 start sets. The authors record metrics and queried sample sets for the roughly 4.56 million combinations that completed within a 5-minute per-cycle limit, then quantify the influence of each hyperparameter via metric-based Pearson correlations, queried-samples Jaccard similarity, and leaderboard-ranking Kendall tau-b correlations. They conclude that the specific implementation of an AL strategy can affect performance more than the strategy itself (exemplified by a 44.5% correlation between two QUIRE implementations), that dataset diversity matters more than specific dataset choices, and that roughly 4,000 randomly drawn hyperparameter combinations already produce leaderboard rankings highly correlated with the full grid. They close with recommendations for designing reproducible AL evaluations.

Significance. If the central empirical claims survive scrutiny, this is one of the largest and most transparent AL benchmarking efforts to date, with the source code and raw results publicly released. The paper provides a useful mapping of which hyperparameters matter (batch size, learner model, strategy implementation) and which appear relatively inert (train-test splits, start sets), and the practical recommendation of a ~4,000-combination grid is actionable. The authors are also commendably explicit about their runtime limits, missing data, and the choice to fix aggregation metrics. However, the flagship claim about implementation beating strategy rests on correlations computed after zero-imputing a non-randomly missing subset of the grid, and the paper does not provide a complete-case check; this is a load-bearing internal-validity concern.

major comments (3)
  1. [Sec. IV-A and IV-G] The central claim that the QUIRE implementations correlate only 44.5% is computed on result vectors in which missing cells are filled with a fixed value of 0 (the interpolation strategy described in Sec. IV-B). Sec. IV-A states that the missing results "are mainly from the largest datasets and both implementations of the QUIRE AL strategy," so the missingness is concentrated in exactly the two implementations being compared. If the two implementations time out on different subsets of the large datasets, zero-imputation injects discordant pairs into the Pearson correlation and mechanically lowers the observed value. The paper must show that the low QUIRE correlation survives a complete-case analysis (restricting to hyperparameter combinations where both implementations completed) or under a more realistic imputation model; without this, the flagship example for "implementation beats strategy" is not secure.
  2. [Sec. IV-B3 and IV-J] The "4,000 combinations suffice" threshold in Fig. 11 is derived from the leaderboard ranking-invariance correlation between random subsets and the full grid, and the full grid itself contains the 75,924 zero-imputed missing cells. Because those missing cells are concentrated in the largest datasets and in QUIRE implementations, the subset-stability analysis may reflect the imputation pattern rather than the true structure of AL results. The authors should re-run the subsampling analysis on the complete-case portion of the grid (or with a missing-indicator included) to demonstrate that the 4,000-combination recommendation is robust to the treatment of missing results.
  3. [Sec. IV-G and Fig. 8] The conclusion that implementation choices dominate strategy identity is also supported by the average within-framework versus cross-framework correlations in Fig. 8, but the paper provides no uncertainty quantification for the differences between these averages. The reported standard deviations overlap considerably (e.g., a mean of 89.8% for ALI with the lowest standard deviation still leaves a wide range), so a permutation or bootstrap test is needed to establish that the within-framework correlation is significantly higher than the cross-framework baseline. This is a secondary support for the central claim, but it should be reported as a provisional observation until such a test is added.
minor comments (5)
  1. [Sec. IV-B2] The queried-samples heatmap formula is internally inconsistent: the text says "we subtract the Jaccard index from 1 so that 1 indicates full similarity," but the displayed expression is 1 - (sum of Jaccard indices)/|V|, which would be 0 for identical sets, while the diagonal of the displayed matrix shows 1. Please clarify whether the heatmaps plot J or 1-J, and correct the formula or the caption accordingly.
  2. [Sec. II-C] There are several typos, e.g., "complxeity" should be "complexity," "practicioners" should be "practitioners," and "framworks" should be "frameworks." A careful proofreading pass is needed.
  3. [Sec. IV-C] The sentence "we use the the F1-score" contains a duplicated article; it should read "we use the F1-score."
  4. [Fig. 7 and Fig. 8] The 28x28 strategy heatmaps and the framework correlation figure are nearly impossible to read at preprint resolution; the axis labels and cell values are illegible. Please provide vector graphics or a higher-resolution version, and consider an interactive or zoomable appendix.
  5. [Sec. II-D] The grid notation in Sec. II-D lists "c" and "M" in the hyperparameter sets but then states they are not part of the experimental grid; this is a notational muddle that should be clarified in the text rather than only in the table.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper reports measured correlations from a large empirical grid; the 'implementation beats strategy' claim is a direct observational finding, not a fitted or self-referential prediction.

full rationale

The paper's derivation chain is empirical rather than deductive: it defines an experiment grid, runs 4.56M combinations, and reports correlations among hyperparameter-conditioned result vectors. No parameter is fitted to a subset and then renamed a prediction. The headline finding in Sec IV-G ('the specific implementation of AL strategies can impact performance more than the underlying strategy itself') is a direct reading of the metric-based heatmap's observed correlations (e.g., the two QUIRE implementations at 44.5% correlation), not a quantity forced by the construction of the heatmap. The heatmap measures exactly this quantity: as stated in Sec IV-B, 'If two combinations produce similar ML metrics, they have a high correlation.' The subset-stability analysis in Sec IV-J uses the full grid as its own reference benchmark; the paper explicitly labels this as an assumption ('Under the assumption that our hyperparameter grid is large enough to include almost any other possible AL evaluation scenarios'), so it is a consistency check rather than a self-referential derivation. The main caveat relevant to the central claim is internal validity, not circularity: missing cells are imputed as 0 (Sec IV-B: 'interpolating the results using a fixed value of 0'), and Sec IV-A reports that missing results 'are mainly from the largest datasets and both implementations of the QUIRE AL strategy'; this could bias the QUIRE correlation, but that is a data-quality threat to an empirical measurement, not an equivalence between input and output. I therefore find no step where a claimed prediction reduces by construction to its inputs.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim is an empirical observation, not a derived formula; no free parameters are fitted to produce predictions. The main burden lies in the implicit assumption that the chosen grid and imputation rules capture enough of the AL design space.

assumptions (5)
  • domain assumption 92 tabular classification datasets and three classical learners (MLP, SVM, RF) are representative of active learning practice.
    Invoked throughout Section II-D and IV; all recommendations generalize from this grid to AL at large.
  • domain assumption The 28 strategy implementations from five frameworks fairly represent AL strategy design.
    Sec IV-G uses correlations among these implementations to conclude implementation matters more than strategy.
  • ad hoc to paper Missing results (runtime limits, implementation errors) can be imputed with a fixed value of 0 without distorting leaderboard rankings.
    Sec IV-B3: 'we are interpolating the results using a fixed value of 0, meaning the worst possible outcome for the ML metric.'
  • ad hoc to paper The 5-minute per-cycle runtime limit excludes only a negligible part of the grid.
    Sec IV-A states 75,924 combinations (1.6%) were excluded; this limits coverage of QUIRE on large datasets.
  • standard math Pearson correlation, Jaccard index and Kendall tau-b are appropriate similarity measures for this comparison.
    Sec IV-B introduces these as the quantification methods; they are standard, but their equivalence for this purpose is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Survey of Active Learning Hyperparameters: Insights from a Large-Scale Experimental Grid." pith.science (2026). https://pith.science/paper/XI2EJMRM

@misc{pith2026250603817,
  author       = {Pith},
  title        = {Pith review of: Survey of Active Learning Hyperparameters: Insights from a Large-Scale Experimental Grid},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XI2EJMRM}},
  note         = {Machine review of arXiv:2506.03817}
}
read the original abstract

Annotating data is a time-consuming and costly task, but it is inherently required for supervised machine learning. Active Learning (AL) is an established method that minimizes human labeling effort by iteratively selecting the most informative unlabeled samples for expert annotation, thereby improving the overall classification performance. Even though AL has been known for decades, AL is still rarely used in real-world applications. As indicated in the two community web surveys among the NLP community about AL, two main reasons continue to hold practitioners back from using AL: first, the complexity of setting AL up, and second, a lack of trust in its effectiveness. We hypothesize that both reasons share the same culprit: the large hyperparameter space of AL. This mostly unexplored hyperparameter space often leads to misleading and irreproducible AL experiment results. In this study, we first compiled a large hyperparameter grid of over 4.6 million hyperparameter combinations, second, recorded the performance of all combinations in the so-far biggest conducted AL study, and third, analyzed the impact of each hyperparameter in the experiment results. In the end, we give recommendations about the influence of each hyperparameter, demonstrate the surprising influence of the concrete AL strategy implementation, and outline an experimental study design for reproducible AL experiments with minimal computational effort, thus contributing to more reproducible and trustworthy AL research in the future.

Figures

Figures reproduced from arXiv: 2506.03817 by the authors.

Figure 1
Figure 1. Standard Active Learning Cycle AL cycle, the cycle iteration is indicated by t: Starting with a small initial labeled dataset D0 lab = {(xi , yi)} n i=0 of n samples xi ∈ X and corresponding labels yi ∈ Y, and a large pool of unlabeled data D0 unl = {xi}, xi ̸∈ D0 lab, an ML model, called the learner model L: X 7→ Y, is trained on the labeled data. A query strategy (blue box) S : Dunl −→ Q selects a batch of b unlab… view at source ↗
Figure 2
Figure 2. Showcasing the limits of learning curve plots [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Illustrating the different aggregation metrics [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Metric-based and leaderboard ranking invariance heatmaps for the basic as well as the aggregation metrics [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Heatmaps for the batch size hyperparameter [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Heatmaps for the learner model hyperparameter [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Heatmaps for the AL query strategy hyperparameter AL strategy [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Average correlation and standard deviation among the AL strategies [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Heatmaps for the hyperparameter train-test-split [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Leaderboard ranking invariance-based heatmap for the hyperparameter start set [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Hyperparameter combination amount simulations [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Average Runtime of all strategies over the complete hyperparameter grid, logarithmic scale [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Final Leaderboard of all datasets and strategies, full mean of class weighted F1-score, normalized per dataset for comparability, and interpolated [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

81 extracted references · 80 canonical work pages

  1. [1]

    Active Learning Literature Survey,

    B. Settles, “Active Learning Literature Survey,” University of Wisconsin- Madison Department of Computer Sciences, Technical Report, 2009

    2009

  2. [2]

    A web survey on the use of active learning to support annotation of text data,

    K. Tomanek and F. Olsson, “A web survey on the use of active learning to support annotation of text data,” inProceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing, E. Ringger, R. Haertel, and K. Tomanek, Eds. Boulder, Colorado: Association for Computational Linguistics, 2009, pp. 45–48

    work page 2009

  3. [3]

    Have LLMs Made Active Learning Obsolete? Surveying the NLP Community,

    J. Romberg, C. Schr ¨oder, J. Gonsior, K. Tomanek, and F. Olsson, “Have LLMs Made Active Learning Obsolete? Surveying the NLP Community,” 2025

    work page 2025

  4. [4]

    Active Learning for Spreadsheet Cell Classification,

    J. Gonsior, J. Rehak, M. Thiele, E. Koci, M. G ¨unther, and W. Lehner, “Active Learning for Spreadsheet Cell Classification,” inProceedings of the Workshops of the EDBT/ICDT 2020 Joint Conference, Copen- hagen, Denmark, March 30, 2020, ser. CEUR Workshop Proceedings, A. Poulovassilis, D. Auber, N. Bikakis, P. K. Chrysanthis, G. Pa- pastefanatos, M. A. Sha...

    work page 2020

  5. [5]

    DeepCA VE: An Interactive Analysis Tool for Automated Machine Learning,

    R. Sass, E. Bergman, A. Biedenkapp, F. Hutter, and M. Lindauer, “DeepCA VE: An Interactive Analysis Tool for Automated Machine Learning,” 2022

    work page 2022

  6. [6]

    Diminishing Uncertainty Within the Training Pool: Active Learning for Medical Image Segmentation,

    V . Nath, D. Yang, B. A. Landman, D. Xu, and H. R. Roth, “Diminishing Uncertainty Within the Training Pool: Active Learning for Medical Image Segmentation,”IEEE Transactions on Medical Imaging, vol. 40, no. 10, pp. 2534–2547, 2021

    work page 2021

  7. [7]

    Active Learning for Air Quality Station Location Recommendation,

    S. D. Narayanan, A. Agnihotri, and N. Batra, “Active Learning for Air Quality Station Location Recommendation,” inProceedings of the 7th ACM IKDD CoDS and 25th COMAD, ser. CoDS COMAD 2020. New York, NY , USA: Association for Computing Machinery, 2020, pp. 326– 327

    work page 2020

  8. [8]

    DECAL: DEployable Clinical Active Learning,

    Y .-y. Logan, M. Prabhushankar, and G. AlRegib, “DECAL: DEployable Clinical Active Learning,” 2022

    work page 2022

Show all 81 references
  1. [9]

    Using Active Learning for Assisted Short Answer Grading,

    J. Kishaan, M. Muthuraja, D. Nair, and P. G. Pl ¨oger, “Using Active Learning for Assisted Short Answer Grading,” inICML 2020 Workshop on Real World Experiment Design and Active Learning, 2020

    2020

  2. [10]

    Unsupervised Active Learning For Video Annotation,

    E. Demir, Z. Cataltepe, U. Ekmekci, M. Budnik, and L. Besacier, “Unsupervised Active Learning For Video Annotation,” inICML Active Learning Workshop 2015, Lille, France, 2015

    2015

  3. [11]

    A benchmark and comparison of active learning for logistic regression,

    Y . Yang and M. Loog, “A benchmark and comparison of active learning for logistic regression,”Pattern Recognition, vol. 83, pp. 401–415, 2018

    2018

  4. [12]

    Is margin all you need? An extensive empirical study of active learning on tabular data,

    D. Bahri, H. Jiang, T. Schuster, and A. Rostamizadeh, “Is margin all you need? An extensive empirical study of active learning on tabular data,” 2022

    2022

  5. [13]

    Revisiting uncertainty-based query strategies for active learning with transformers,

    C. Schr ¨oder, A. Niekler, and M. Potthast, “Revisiting uncertainty-based query strategies for active learning with transformers,” inFindings of the Association for Computational Linguistics: ACL 2022, S. Muresan, P. Nakov, and A. Villavicencio, Eds. Dublin, Ireland: Associati...

    2022

  6. [14]

    Learning active learning at the cross- roads? Evaluation and discussion,

    L. Desreumaux and V . Lemaire, “Learning active learning at the cross- roads? Evaluation and discussion,”CEUR Workshop Proceedings, vol. 2660, pp. 38–54, 2020

    2020

  7. [15]

    A comparative survey: Benchmarking for pool-based active learning,

    X. Zhan, H. Liu, Q. Li, and A. B. Chan, “A comparative survey: Benchmarking for pool-based active learning,” inProceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, Z. Zhou, Ed. ...

    2021

  8. [16]

    On the Fragility of Active Learners for Text Classification,

    A. Ghose and E. T. Nguyen, “On the Fragility of Active Learners for Text Classification,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Association for Computation...

    2024

  9. [17]

    Randomness is the Root of All Evil: More Reliable Evaluation of Deep Active Learning,

    Y . Ji, D. Kaestner, O. Wirth, and C. Wressnegger, “Randomness is the Root of All Evil: More Reliable Evaluation of Deep Active Learning,” in 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), ser. W ACV 2023. Waikoloa, HI, USA: IEEE, 2023, pp. 3932– 3941

    2023

  10. [18]

    ALPBench: A Benchmark for Active Learning Pipelines on Tabular Data,

    V . Margraf, M. Wever, S. Gilhuber, G. M. Tavares, T. Seidl, and E. H ¨ullermeier, “ALPBench: A Benchmark for Active Learning Pipelines on Tabular Data,” 2024

    2024

  11. [19]

    When Does Active Learning Work?

    L. P. G. Evans, N. M. Adams, and C. Anagnostopoulos, “When Does Active Learning Work?” inAdvances in Intelligent Data Analysis XII, A. Tucker, F. H¨oppner, A. Siebes, and S. Swift, Eds. Berlin, Heidelberg: Springer, 2013, pp. 174–185

    2013

  12. [20]

    Challenges of Reliable, Realistic and Comparable Active Learning Evaluation

    D. Kottke, A. Calma, D. Huseljic, G. M. Krempl, and B. Sick, “Challenges of Reliable, Realistic and Comparable Active Learning Evaluation.” inIAL@PKDD/ECML 2017, 2017, pp. 2–14

    2017

  13. [21]

    Em- pirical investigation of active learning strategies,

    D. Pereira-Santos, R. B. C. Prud ˆencio, and A. C. De Carvalho, “Em- pirical investigation of active learning strategies,”Neurocomputing, vol. 326–327, pp. 15–27, 2019

    2019

  14. [22]

    Active learning: An empirical study of common baselines,

    M. E. Ramirez-Loaiza, M. Sharma, G. Kumar, and M. Bilgic, “Active learning: An empirical study of common baselines,”Data Mining and Knowledge Discovery, vol. 31, no. 2, pp. 287–313, 2017

    2017

  15. [23]

    Rebuilding Trust in Active Learning with Actionable Metrics,

    A. Abraham and L. Dreyfus-Schmidt, “Rebuilding Trust in Active Learning with Actionable Metrics,”IEEE International Conference on Data Mining Workshops, ICDMW, vol. 2020-November, pp. 836–843, 2020

    2020

  16. [24]

    Re-Benchmarking Pool-Based Active Learning for Binary Classification,

    P.-Y . Lu, C.-L. Li, and H.-T. Lin, “Re-Benchmarking Pool-Based Active Learning for Binary Classification,” 2023

    2023

  17. [25]

    On initial pools for deep active learning,

    A. L. Chandra, S. V . Desai, C. Devaguptapu, and V . N. Balasubramanian, “On initial pools for deep active learning,” inNeurIPS 2020 Workshop on Pre-Registration in Machine Learning, ser. Proceedings of Machine Learning Research, L. Bertinetto, J. F. Henriques, S. Albanie, M. ...

    2020

  18. [26]

    An overview and a bench- mark of active learning for outlier detection with one-class classifiers,

    H. Trittenbach, A. Englhardt, and K. B ¨ohm, “An overview and a bench- mark of active learning for outlier detection with one-class classifiers,” Expert Systems with Applications, vol. 168, p. 114372, 2021

    2021

  19. [27]

    Sample Noise Impact on Active Learning,

    A. Abraham and L. Dreyfus-Schmidt, “Sample Noise Impact on Active Learning,”CEUR Workshop Proceedings, vol. 3079, pp. 80–88, 2021

    2021

  20. [28]

    Hitting the target: Stopping active learning at the cost-based optimum,

    Z. Pullar-Strecker, K. Dost, E. Frank, and J. Wicker, “Hitting the target: Stopping active learning at the cost-based optimum,”Machine Learning, pp. 1–19, 2022

    2022

  21. [29]

    A Comparative Survey of Deep Active Learning,

    X. Zhan, Q. Wang, K.-h. Huang, H. Xiong, D. Dou, and A. B. Chan, “A Comparative Survey of Deep Active Learning,” 2022

    2022

  22. [30]

    Active learning for logistic regression: An evaluation,

    A. I. Schein and L. H. Ungar, “Active learning for logistic regression: An evaluation,”Machine learning, vol. 68, no. 3, pp. 235–265, 2007

    2007

  23. [31]

    The UCI Machine Learning Repository,

    M. Kelly, R. Longjohn, and K. Nottingham, “The UCI Machine Learning Repository,” 2017. ARXIV PREPRINT: SURVEY OF ACTIVE LEARNING HYPERPARAMETERS 19

    2017

  24. [32]

    OpenML Bench- marking Suites,

    B. Bischl, G. Casalicchio, M. Feurer, P. Gijsbers, F. Hutter, M. Lang, R. Gomes Mantovani, J. van Rijn, and J. Vanschoren, “OpenML Bench- marking Suites,”Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, vol. 1, 2021

    2021

  25. [33]

    Learning active learning from data,

    K. Konyushkova, R. Sznitman, and P. Fua, “Learning active learning from data,” inAdvances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, I. Guyon, U. von Luxburg, S. Bengio,...

    2017

  26. [34]

    Comparing and Improving Active Learning Uncertainty Measures for Transformer Models by Discarding Outliers,

    J. Gonsior, C. Falkenberg, S. Magino, A. Reusch, C. Hartmann, M. Thiele, and W. Lehner, “Comparing and Improving Active Learning Uncertainty Measures for Transformer Models by Discarding Outliers,” Information Systems Frontiers, 2024

    2024

  27. [35]

    Imagenet: A large- scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li, “Imagenet: A large- scale hierarchical image database,” in2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA. IEEE Computer Society, 2009, pp. 248–255

    2009

  28. [36]

    Probabilistic Active Learning: Towards Combining Versatility, Optimality and Efficiency,

    G. Krempl, D. Kottke, and M. Spiliopoulou, “Probabilistic Active Learning: Towards Combining Versatility, Optimality and Efficiency,” in Discovery Science, S. D ˇzeroski, P. Panov, D. Kocev, and L. Todorovski, Eds. Cham: Springer International Publishing, 2014, pp. 168–179

    2014

  29. [37]

    ImitAL: Learned Active Learning Strategy on Synthetic Data,

    J. Gonsior, M. Thiele, and W. Lehner, “ImitAL: Learned Active Learning Strategy on Synthetic Data,” inDiscovery Science - 25th International Conference, DS 2022, Montpellier, France, October 10-12, 2022, Pro- ceedings, ser. Lecture Notes in Computer Science, P. Poncelet and D....

    2022

  30. [38]

    Let us know your decision: Pool-based active training of a generative classifier with the selection strategy 4DS,

    T. Reitmaier and B. Sick, “Let us know your decision: Pool-based active training of a generative classifier with the selection strategy 4DS,” Information Sciences, vol. 230, pp. 106–131, 2013

    2013

  31. [39]

    Active Learn- ing for Support Vector Machines with Maximum Model Change,

    W. Cai, Y . Zhang, S. Zhou, W. Wang, C. Ding, and X. Gu, “Active Learn- ing for Support Vector Machines with Maximum Model Change,” in Machine Learning and Knowledge Discovery in Databases, T. Calders, F. Esposito, E. H ¨ullermeier, and R. Meo, Eds. Berlin, Heidelberg: Springe...

    2014

  32. [40]

    A Novel Uncertainty Sampling Algorithm for Cost-Sensitive Multiclass Active Learning,

    K.-H. Huang and H.-T. Lin, “A Novel Uncertainty Sampling Algorithm for Cost-Sensitive Multiclass Active Learning,” in2016 IEEE 16th International Conference on Data Mining (ICDM), 2016, pp. 925–930

    2016

  33. [41]

    To Actively Initialize Active Learning,

    Y . Yang and M. Loog, “To Actively Initialize Active Learning,”Pattern Recognition, vol. 131, p. 108836, 2022

    2022

  34. [42]

    Individual Comparisons by Ranking Methods,

    F. Wilcoxon, “Individual Comparisons by Ranking Methods,”Biometrics Bulletin, vol. 1, no. 6, p. 80, 1945

    1945

  35. [43]

    Active learning by querying informa- tive and representative examples,

    S. Huang, R. Jin, and Z. Zhou, “Active learning by querying informa- tive and representative examples,” inAdvances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010...

    2010

  36. [44]

    Optimised probabilistic active learning (OPAL),

    G. Krempl, D. Kottke, and V . Lemaire, “Optimised probabilistic active learning (OPAL),”Machine learning, vol. 100, no. 2, pp. 449–476, 2015

    2015

  37. [45]

    Active learning using transductive sparse Bayesian regression,

    Y . Son and J. Lee, “Active learning using transductive sparse Bayesian regression,”Information Sciences, vol. 374, pp. 240–254, 2016

    2016

  38. [46]

    Statistical Comparisons of Classifiers over Multiple Data Sets,

    J. Dem ˇsar, “Statistical Comparisons of Classifiers over Multiple Data Sets,”J. Mach. Learn. Res., vol. 7, pp. 1–30, 2006

    2006

  39. [47]

    Kaggle: Your Machine Learning and Data Science Community

    “Kaggle: Your Machine Learning and Data Science Community.”

  40. [48]

    ALiPy: Active Learning in Python,

    Y .-P. Tang, G.-X. Li, and S.-J. Huang, “ALiPy: Active Learning in Python,” 2019

    2019

  41. [49]

    Libact: Pool-based Active Learning in Python,

    Y .-Y . Yang, S.-C. Lee, Y .-A. Chung, T.-E. Wu, S.-A. Chen, and H.-T. Lin, “Libact: Pool-based Active Learning in Python,” 2017

    2017

  42. [50]

    Google/active-learning,

    Y . Yang, “Google/active-learning,” Google, 2024

    2024

  43. [51]

    Scikit-activeml: A Library and Toolbox for Active Learning Algorithms,

    D. Kottke, M. Herde, T. P. Minh, A. Benz, P. Mergard, A. Roghman, C. Sandrock, and B. Sick, “Scikit-activeml: A Library and Toolbox for Active Learning Algorithms,” 2021

    2021

  44. [52]

    Small-text: Active learning for text classification in python,

    C. Schr ¨oder, L. M ¨uller, A. Niekler, and M. Potthast, “Small-text: Active learning for text classification in python,” inProceedings of the 17th Conference of the European Chapter of the Association for Computa- tional Linguistics: System Demonstrations, D. Croce and L. Sol...

    2023

  45. [53]

    Active learning by acquiring contrastive examples,

    K. Margatina, G. Vernikos, L. Barrault, and N. Aletras, “Active learning by acquiring contrastive examples,” inProceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Ca...

    2021

  46. [54]

    Active learning for convolutional neural networks: A core-set approach,

    O. Sener and S. Savarese, “Active learning for convolutional neural networks: A core-set approach,” in6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018

    2018

  47. [55]

    Dual Strategy Active Learning,

    P. Donmez, J. G. Carbonell, and P. N. Bennett, “Dual Strategy Active Learning,” inProceedings of the 18th European Conference on Machine Learning, ser. ECML ’07. Berlin, Heidelberg: Springer-Verlag, 2007, pp. 116–127

    2007

  48. [56]

    Active learning using pre- clustering,

    H. T. Nguyen and A. W. M. Smeulders, “Active learning using pre- clustering,” inMachine Learning, Proceedings of the Twenty-first In- ternational Conference (ICML 2004), Banff, Alberta, Canada, July 4- 8, 2004, ser. ACM International Conference Proceeding Series, C. E. Brodley...

    2004

  49. [57]

    Cold-start active learning through self-supervised language modeling,

    M. Yuan, H.-T. Lin, and J. Boyd-Graber, “Cold-start active learning through self-supervised language modeling,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), B. Webber, T. Cohn, Y . He, and Y . Liu, Eds. Online: Association f...

    2020

  50. [58]

    RALF: A reinforced active learning formulation for object class recognition,

    S. Ebert, M. Fritz, and B. Schiele, “RALF: A reinforced active learning formulation for object class recognition,” in2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012. IEEE Computer Society, 2012, pp. 3626–3633

    2012

  51. [59]

    Learning how to actively learn: A deep imitation learning approach,

    M. Liu, W. Buntine, and G. Haffari, “Learning how to actively learn: A deep imitation learning approach,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), I. Gurevych and Y . Miyao, Eds. Melbourne, Australia: As...

    2018

  52. [60]

    Query by committee,

    H. S. Seung, M. Opper, and H. Sompolinsky, “Query by committee,” in Proceedings of the Fifth Annual Workshop on Computational Learning Theory, ser. COLT92. Pittsburgh Pennsylvania USA: ACM, 1992, pp. 287–294

    1992

  53. [61]

    Query Learning Strategies Using Boosting and Bagging

    N. Abe and H. Mamitsuka, “Query Learning Strategies Using Boosting and Bagging.” inProceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24- 27, 1998, ser. ICML 1998. Morgan Kaufmann Publishers Inc., 1998, pp. 1–9

    1998

  54. [62]

    Active Learning for Regression Based on Query by Committee,

    R. Burbidge, J. J. Rowland, and R. D. King, “Active Learning for Regression Based on Query by Committee,” inIntelligent Data En- gineering and Automated Learning - IDEAL 2007, H. Yin, P. Tino, E. Corchado, W. Byrne, and X. Yao, Eds. Berlin, Heidelberg: Springer Berlin Heidelbe...

    2007

  55. [63]

    Entropy-based active learning for object recognition,

    A. Holub, P. Perona, and M. C. Burl, “Entropy-based active learning for object recognition,” in2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Anchorage, AK, USA: IEEE, 2008, pp. 1–8

    2008

  56. [64]

    A Sequential Algorithm for Training Text Classifiers

    D. D. Lewis and W. A. Gale, “A Sequential Algorithm for Training Text Classifiers.” inSIGIR 1994. Springer London, 1994, pp. 3–12

    1994

  57. [65]

    Mining the Web with active hidden Markov models,

    T. Scheffer, C. Decomain, and S. Wrobel, “Mining the Web with active hidden Markov models,” inProceedings 2001 IEEE International Conference on Data Mining, ser. ICDM-01. San Jose, CA, USA: IEEE Comput. Soc, 2001, pp. 645–646

    2001

  58. [66]

    Active learning to recognize multiple types of plankton,

    Tong Luo, K. Kramer, S. Samson, A. Remsen, D. Goldgof, L. Hall, and T. Hopkins, “Active learning to recognize multiple types of plankton,” inProceedings of the 17th International Conference on Pattern Recog- nition, 2004. ICPR 2004.Cambridge, UK: IEEE, 2004, pp. 478–481 V ol.3

    2004

  59. [67]

    Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering,

    S. Karamcheti, R. Krishna, L. Fei-Fei, and C. Manning, “Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th I...

    2021

  60. [68]

    VII. Note on regression and inheritance in the case of two parents,

    K. Pearson and F. Galton, “VII. Note on regression and inheritance in the case of two parents,”Proceedings of the Royal Society of London, vol. 58, no. 347-352, pp. 240–242, 1895

  61. [69]

    The Finley Affair: A Signal Event in the History of Forecast Verification,

    A. H. Murphy, “The Finley Affair: A Signal Event in the History of Forecast Verification,”Weather and Forecasting, vol. 11, no. 1, pp. 3– 20, 1996

    1996

  62. [70]

    A New Measure of Rank Correlation,

    M. G. Kendall, “A New Measure of Rank Correlation,”Biometrika, vol. 30, no. 1/2, pp. 81–93, 1938

    1938

  63. [71]

    The Proof and Measurement of Association between Two Things,

    C. Spearman, “The Proof and Measurement of Association between Two Things,”The American Journal of Psychology, vol. 15, no. 1, pp. 72– 101, 1904

    1904

  64. [72]

    A Mathematical Theory of Communication,

    C. E. Shannon, “A Mathematical Theory of Communication,”Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. ARXIV PREPRINT: SURVEY OF ACTIVE LEARNING HYPERPARAMETERS 20

    1948

  65. [73]

    A Survey of Active Learning for Text Classification using Deep Neural Networks,

    C. Schr ¨oder and A. Niekler, “A Survey of Active Learning for Text Classification using Deep Neural Networks,” 2020

    2020

  66. [74]

    Cartography active learning,

    M. Zhang and B. Plank, “Cartography active learning,” inFindings of the Association for Computational Linguistics: EMNLP 2021, M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Punta Cana, Dominican Republic: Association for Computational Linguistics, 2021, pp. 395– 406

    2021

  67. [75]

    Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning,

    A. Kirsch, J. van Amersfoort, and Y . Gal, “Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning,” inAdvances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-...

    2019

  68. [76]

    Bayesian Active Learning for Classification and Preference Learning,

    N. Houlsby, F. Husz ´ar, Z. Ghahramani, and M. Lengyel, “Bayesian Active Learning for Classification and Preference Learning,” 2011

    2011

  69. [77]

    Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study,

    A. Siddhant and Z. C. Lipton, “Deep Bayesian active learning for natural language processing: Results of a large-scale empirical study,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, E. Riloff, D. Chiang, J. Hockenmaier, and J. Tsuji...

    2018

  70. [78]

    Selection via proxy: Efficient data selection for deep learning,

    C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovec, and M. Zaharia, “Selection via proxy: Efficient data selection for deep learning,” in8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 20...

    2020

  71. [79]

    Active learning by learning,

    W. Hsu and H. Lin, “Active learning by learning,” inProceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, B. Bonet and S. Koenig, Eds. AAAI Press, 2015, pp. 2659–2665

    2015

  72. [80]

    Discovering General- Purpose Active Learning Strategies,

    K. Konyushkova, R. Sznitman, and P. Fua, “Discovering General- Purpose Active Learning Strategies,” 2019

    2019

  73. [81]

    Querying discriminative and representative samples for batch mode active learning,

    Z. Wang and J. Ye, “Querying discriminative and representative samples for batch mode active learning,” inThe 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013, I. S. Dhillon, Y . Koren, R. Ghani, T....

    2013

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.