Pith. sign in

REVIEW 4 major objections 4 minor 65 references

The paper claims that a retrieve-then-rank funnel with an adaptive exploit-explore switch finds rare target classes starting from a single positive example, using fewer labels than single-stage active learning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:53 UTC pith:YES2BLK3

load-bearing objection FunnelAL's precision-triggered retrieve-then-rank design is a real contribution, but the headline annotation-efficiency numbers rest on an oracle-based protocol that needs a fixed-budget rerun. the 4 major comments →

arxiv 2607.25276 v1 pith:YES2BLK3 submitted 2026-07-28 cs.CV cs.IRcs.LG

FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery

Reihaneh Rostami (RAIC Labs) , Brian Goodwin (RAIC Labs) This is my paper
classification cs.CV cs.IRcs.LG
keywords active learningsingle-class discoveryretrieve-then-rankrecommender systemsquery-by-committeeRankNetannotation efficiencylabel noise
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

FunnelAL is an active-learning system for finding all instances of a single target class in a large image corpus, starting from just one positive and one negative example. The paper argues that by decomposing sample selection into a retrieval stage that narrows the pool via embedding similarity and a ranking stage that alternates between exploiting confident positives (RankNet) and exploring uncertain boundary regions (Query-by-Committee), an annotation campaign finds more positives per label than conventional one-stage acquisition functions. The central claim is that this 'funnel' architecture, borrowed from recommender systems, makes annotation both cheaper per round and more efficient in total labels, and that an automatic precision-triggered switch between exploitation and exploration is what delivers the gains. On three benchmarks spanning fine-grained species, manufactured objects, and domain-shifted aerial imagery, FunnelAL reports the best final F1, the best area under the learning curve, and the fewest annotation rounds, and it degrades more gracefully than uncertainty-based baselines when annotators make labeling errors. The paper further claims that the newest single-class discovery methods (GAL and PF-MA) at best match its final accuracy, but only at consistently higher labeling cost.

Core claim

The central discovery is that the funnel decomposition itself—an explicit embedding-retrieval prefilter followed by a two-arm ranking stage—is what drives label efficiency in single-class discovery, and that the interaction between the arms should be adaptive rather than fixed. Starting from one positive and one negative seed, FunnelAL scores the corpus by a distance-weighted vote over the current positives' nearest neighbors to form a candidate set, then ranks that set with a RankNet pairwise ranker while batch precision stays high, growing the batch size as long as every sample is confirmed positive. When the rolling mean batch precision drops below a threshold (0.7 over three batches), th

What carries the argument

The central object is the multi-stage funnel: Stage 1 retrieval uses Distance-Weighted Vote Aggregation (DWVA) over embedding-based nearest-neighbor searches to cut the pool to a fixed-size candidate set; Stage 2 ranking uses a RankNet pairwise ranker as the exploitation arm and a four-member Query-by-Committee committee as the exploration arm; the precision trigger—a one-way transition when the mean precision of the last three batches falls below τ=0.7—moves the system from pure exploitation to a 60/40 RankNet-plus-QBC hybrid, and a batch-growth rule multiplies batch size by 1.2 after each fully-confirmed batch. The machinery's work is to keep computational cost proportional to the candidat

Load-bearing premise

The evaluation protocol sets the annotation budget from the true positive count and stops once all positives are found—information a real user would not have—so the reported label-efficiency and round-count advantages may not survive a protocol where budget and stopping are fixed without oracle knowledge.

What would settle it

Run the same three benchmarks with a fixed annotation budget for every class, no early termination, and no knowledge of the true positive count; if FunnelAL no longer leads on AULC or matches rounds, the central efficiency claim fails. A second falsifier is a real-annotator user study testing whether funnel recommendations and manual explore-exploit control save wall-clock time.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Annotation systems for single-class discovery can adopt the recommender funnel pattern and reach a given F1 with fewer total labels and fewer human rounds than fixed acquisition functions.
  • The precision-triggered transition self-calibrates per class, switching when easy positives are exhausted, and is far more reliable than a fixed positive-count schedule, which loses up to 16.5 F1 points and 30.8 AULC points on the bird benchmark.
  • Under annotator label noise at 5–15%, the system remains first or statistically tied for first, while boundary-seeking uncertainty baselines degrade two to three times faster.
  • Because all stages operate purely on embeddings, the method transfers to other modalities with a strong pretrained encoder, though the paper demonstrates this only on images.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported round-count savings depend on the paper's early-termination oracle; a fixed-budget comparison without oracle knowledge would likely reduce the round advantage, though the per-class recall trajectory gains suggest some label-efficiency benefit may persist.
  • The 60/40 exploit-explore split and threshold τ are fixed; making them per-class adaptive with contextual bandits, as the paper suggests, could extend the funnel's advantage to uneven class structures.
  • Mild label noise acted as accidental exploration for trapped baselines; this implies that adding a small amount of explicit diversity to any pure-exploitation method may reproduce the noise benefit without the errors, a testable design tweak.
  • The paper leaves the manual annotator-control affordance unevaluated; a user study would determine whether human judgment in choosing exploit versus explore beats the automated trigger.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. FunnelAL is a retrieve-then-rank active-learning system for single-class discovery. Starting from one positive and one negative seed, it (1) retrieves a candidate set via embedding-based DWVA-KNN, (2) ranks that set with a precision-triggered hybrid of RankNet exploitation and QBC exploration, and (3) updates positive/negative sets from annotator feedback. The paper evaluates on CUB-200-2011, FGVC-Aircraft, and UC Merced Land Use under a one-vs-rest protocol with DINOv2 embeddings, comparing to eleven ablations/baselines including recent single-class methods GAL and PF-MA. Headline claims are best final F1, best annotation efficiency (AULC), and fewest annotation rounds across all three benchmarks, with retained advantage under simulated annotator label noise.

Significance. If the empirical claims hold, FunnelAL is a practical and conceptually interesting contribution: it transfers the multi-stage funnel architecture from recommender systems to active learning and demonstrates consistent gains across fine-grained and domain-shifted benchmarks. The paper has clear strengths: three diverse datasets, a large number of one-vs-rest tasks, class-level paired Wilcoxon tests, careful ablations of the retrieval and ranking components, a computational cost analysis, a noise-robustness study, and an unusually candid limitations section. However, the central label-efficiency claim (AULC, rounds, B@90) rests on an evaluation protocol that uses ground-truth positive counts for the budget and ground-truth knowledge for early stopping. Until that protocol is replaced or supplemented by a fixed-budget, no-oracle comparison, the magnitude of the claimed efficiency advantage is not established for a deployable system.

major comments (4)
  1. [Section 4.3, Algorithm 1] The evaluation protocol uses oracle knowledge in two load-bearing places: B=min(3×n_pos,500) uses the true positive count, and each run terminates early as soon as all working-set positives are discovered. These choices directly shape the headline metrics: AULC is computed over the learning curve up to termination, B@90 and round counts are termination-based, and even final F1 is measured at different annotation counts across methods. Because FunnelAL discovers positives faster, it terminates earlier; slower baselines continue to the full budget, often labeling less-informative negatives, inflating FunnelAL's AULC and rounds advantage. The paper acknowledges the early-stop issue in Section 5.2, but all headline results still use this protocol. A fixed-budget rerun with a budget that does not depend on n_pos, and with no oracle early stopping (or a stopping rule based only on observed lab
  2. [Section 3.4, Section 5.2] The precision-triggered adaptive policy, identified as load-bearing in Table 2, depends on several free parameters: τ=0.7, w=3, γ=1.2, the 60/40 RankNet/QBC split, plus retrieval parameters α=2, K=512, C=5000, and k0=20. The paper states these are fixed across datasets and 'not tuned per dataset,' but this does not rule out selection on the collection of the three benchmarks, and no sensitivity analysis is reported. The paper itself says 'a systematic sweep of these values remains warranted' (Section 5.2). Given that the central claim is that the precision-triggered mechanism generalizes, a sensitivity analysis or a hold-one-benchmark-out parameter selection is necessary to show the results are not artifacts of particular values.
  3. [Table 1, Section 4.4.1, Abstract] The abstract and conclusion state that FunnelAL attains the best final F1 on all three benchmarks, but on CUB-200 the margin over PF-MA is 0.001 (0.881 vs 0.880) and on UC Merced it is 0.002 (0.952 vs 0.950). No paired Wilcoxon p-values are reported for these final-F1 comparisons, and the text later characterizes several such gaps as statistical ties (e.g., Section 4.4.2 reports p=0.31 for GAL on FGVC-Aircraft). The headline claim should be qualified as 'numerically best / statistically tied,' or the relevant p-values should be reported. This is not purely presentational: the unqualified wording appears in the abstract and duplicate claims in the conclusion.
  4. [Section 4.2] GAL is reimplemented using a 'full weight-vector impact appropriate for high-dimensional features,' and no code is provided for FunnelAL or the baselines. Since GAL is a recent and directly comparable single-class discovery method, an inaccurate or unverified reimplementation could bias the comparison. The paper should either release code for all methods or provide a verification of the reimplementation against official published results. Without this, readers cannot confirm that GAL is not disadvantaged by implementation choices.
minor comments (4)
  1. [Tables 1 and 3] No error bars or confidence intervals are shown for the macro-averaged metrics. Since some differences are very small (e.g., F1 margins of 0.001–0.002), adding per-class variance or reporting the full Wilcoxon test results for all headline pairs would help readers gauge stability.
  2. [Figures 4 and 5] The 'shared 90% bar' is described as 90% of the best strategy's final value, distinct from B@90's per-strategy target. This distinction is important and correct, but it would be clearer to define the dashed line explicitly in the caption of each figure rather than only in the text.
  3. [Equation (1)] The DWVA score s_j is the product of a retrieval count fraction and a distance weight. If a candidate is not retrieved by any positive, the first factor is zero and s_j=0, but the definition of the second factor (1/(1+d_bar_j)) when no positive retrieves the candidate is not specified. Please clarify the empty-set case.
  4. [Section 4.4.5] The computational cost comparison mixes timings from an A100 host and a CPU-only host. The paper justifies cross-host contrasts, but it would be more transparent to report the host for each row or to provide a single-host comparison for all strategies.

Circularity Check

0 steps flagged

No significant circularity: empirical system evaluated against external baselines; no fit-to-prediction or self-citation chain.

full rationale

I walked the paper's claimed derivation chain. FunnelAL is an empirical active-learning system; there is no derivation of a target quantity from a fitted parameter. Eq. (1) is the retrieval score, Eq. (2) the RankNet pairwise loss, Eq. (3) the committee entropy; these are standard components and none is defined in terms of the reported metrics (F1, AULC, Pos. Rec., B@90). The paper claims no uniqueness theorem and cites no prior work by the current authors (Rostami and Goodwin); all references are external baselines or standard methods, so there is no self-citation load-bearing chain. Hyperparameters (tau, w, gamma, 60/40, alpha, K, C) are fixed defaults chosen on the same benchmarks, which is a tuning/selection risk, not a circularity; Section 5.2 explicitly calls for a systematic sweep. The oracle-based budget B=min(3*n_pos,500) and early termination once all positives are found are acknowledged in Section 4.3/5.2 as unavailable in practice; this threatens external validity of the 'fewest rounds/AULC' claims but is not a reduction of the result to the method's own inputs. Comparisons are against externally reimplemented baselines with paired Wilcoxon tests, so the empirical claims are not forced by construction. Finding: no significant circularity.

Axiom & Free-Parameter Ledger

11 free parameters · 5 axioms · 0 invented entities

The central empirical claim rests on a modest number of hand-set hyperparameters (alpha, K, C, tau_g, tau, w, gamma, 60/40 split, k0, RankNet schedule) and an oracle-derived budget. The authors state tau, w, gamma were not tuned per dataset, and only a partial robustness check (K=256, C=1000) on UC Merced is reported; no systematic sensitivity sweep is provided. Evaluation also relies on ground-truth knowledge of positive count and early stopping, which is a separate load-bearing assumption.

free parameters (11)
  • DWVA distance exponent alpha = 2
    Eq. 1; chosen by authors to control distance weighting; no principled justification or sensitivity analysis.
  • Retrieval neighbor count K = 512
    Section 3.2; chosen for benchmark scale; robustness check with K=256 only on UC Merced.
  • Candidate set size C = 5,000
    Section 3.2; exceeds the UC Merced pool; robustness check only on UC Merced.
  • Transition threshold tau = 0.7
    Section 3.4; rolling mean precision below which Phase 1 ends; fixed default, not swept.
  • Growth threshold tau_g = 1.0
    Section 3.4; batch grows only while batch precision is exactly 1; fixed default.
  • Window size w = 3
    Section 3.4; mean over last 3 batches; fixed default.
  • Growth factor gamma = 1.2
    Section 3.4; batch growth multiplier; fixed default without sweep.
  • Hybrid exploit/explore split = 60% RankNet / 40% QBC
    Section 3.4; fixed ratio, not tuned per dataset.
  • Initial batch size k0 = 20
    Section 4.3; fixed across all strategies.
  • RankNet architecture and training schedule = d-128-128-1, 30 epochs, lr=1e-3, 4096 pairs/epoch
    Section 3.3; selected without reported sensitivity analysis.
  • Annotation budget B = min(3*n_pos, 500)
    Section 4.3; uses the true positive count in the working set, an oracle quantity unavailable in practice.
axioms (5)
  • domain assumption DINOv2 embeddings are informative enough that nearest-neighbor retrieval can surface positives, and the relative separability ordering of the three benchmarks tracks retrieval quality.
    Section 3.2 and Figure 3; if embeddings were uninformative, stage 1 would fail; the paper acknowledges retrieval collapses on FGVC and relies on ranking to compensate.
  • domain assumption One positive and one negative seed suffice to bootstrap RankNet and committee training.
    Section 3.1 cold start; only random seed repeats (5 per class) are varied, no systematic study of seed quality or out-of-distribution seeds; later flagged as a limitation.
  • domain assumption Batch precision computed from (possibly noisy) annotator labels is a reliable signal of diminishing returns for switching phases.
    Section 3.4 and 4.4.7; the trigger is demonstrated empirically but has no theoretical support; under noise it fires earlier by construction.
  • domain assumption The evaluation classifier (logistic regression on labeled positives plus random presumed negatives) is a valid proxy for the quality of the discovered label set.
    Section 3.5; PU-learning assumption; identical for all strategies, so fair for comparison but affects absolute realism.
  • domain assumption Ground-truth-based early stopping and budget (B=min(3n_pos,500)) do not qualitatively change the ranking of strategies versus an operational protocol without oracle knowledge.
    Section 4.3 and Algorithm 1; the authors use oracle quantities for efficiency and acknowledge that deployed systems need stopping criteria, but use them as an evaluation convenience.

pith-pipeline@v1.3.0-alltime-deepseek · 24558 in / 14850 out tokens · 141118 ms · 2026-08-01T02:53:21.613325+00:00 · methodology

0 comments
read the original abstract

We present FunnelAL, a retrieve-then-rank active learning system for single-class discovery, which adapts the multi-stage funnel architecture of industrial recommender systems to data annotation. Large-scale supervised learning faces two challenges: efficiently finding relevant samples in a massive corpus, and distinguishing true positives from visually confusable negatives when embeddings do not cleanly separate classes. Conventional active learning offers a principled framework for reducing annotation cost, yet it treats sample selection as a single-stage process that addresses neither challenge efficiently. FunnelAL decomposes the problem into cascaded stages. Starting from a single positive and negative example, the system iterates through: (1) embedding-based retrieval scoring that narrows the corpus to a manageable candidate set; (2) a precision-triggered ranking stage that exploits a learned ranker (RankNet) while batch precision remains high, then automatically blends in committee-based exploration (QBC) once returns diminish; and (3) feedback from the annotator's labels that refines both stages in subsequent iterations. We evaluate on three diverse image classification benchmarks. With a perfect annotator, FunnelAL attains the best final F1 on all three benchmarks, the best annotation efficiency (first in AULC), and the fewest annotation rounds. The most recent single-class discovery methods (GAL, PF-MA) at best match its final quality, and only at consistently higher labeling cost. Under annotator labeling errors at realistic rates, FunnelAL remains first or statistically tied for first while classical uncertainty-based methods degrade two to three times faster. Our work provides a concrete bridge between multi-stage recommender systems and active learning.

Figures

Figures reproduced from arXiv: 2607.25276 by Brian Goodwin (RAIC Labs), Reihaneh Rostami (RAIC Labs).

Figure 1
Figure 1. Figure 1: The FunnelAL pipeline. Solid edges carry data (seeds, candidate set [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: t-SNE [50] of the DINOv2 embeddings, illustrating the differing class structure of the three benchmarks (computed from the embeddings alone, with no annotation method involved). Top row: each dataset with some representative confusable classes highlighted (three gull, two crow, and three tern species for CUB-200; four Boeing variants for FGVC-Aircraft; the four least-separable land-use classes for UC Merce… view at source ↗
Figure 3
Figure 3. Figure 3: Embedding separability of the three benchmarks, [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Recall efficiency across all classes on CUB-200 and FGVC-Aircraft (UC Merced’s per-class deep dive is Figure 5). [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: UC Merced. (a) Test-set recall and (b) Positive recall vs. annotation budget for all strategies. The horizontal dashed line [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Final macro-F1 vs. annotator label-flip rate on all three benchmarks (same protocol, strategies, and repeats as Table 1; [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 6 linked inside Pith

  1. [1]

    Diego Antognini et al. 2021. Multi-Step Critiquing User Interface for Recom- mender Systems. InProceedings of the 15th ACM Conference on Recommender Systems (RecSys). ACM

    2021

  2. [2]

    Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal

    Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2020. Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. InProceedings of the 8th International Conference on Learning Representations (ICLR)

    2020

  3. [3]

    Leah Bar, Boaz Lerner, Nir Darshan, and Rami Ben-Ari. 2024. Active Learn- ing via Classifier Impact and Greedy Selection for Interactive Image Retrieval. Transactions on Machine Learning Research(2024)

    2024

  4. [4]

    Jessa Bekker and Jesse Davis. 2020. Learning from Positive and Unlabeled Data: A Survey.Machine Learning109, 4 (2020), 719–760

    2020

  5. [5]

    Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamil- ton, and Greg Hullender. 2005. Learning to Rank Using Gradient Descent. In Proceedings of the 22nd International Conference on Machine Learning (ICML). ACM, 89–96

    2005

  6. [6]

    Christopher J. C. Burges. 2010.From RankNet to LambdaRank to LambdaMART: An Overview. Technical Report MSR-TR-2010-82. Microsoft Research

    2010

  7. [7]

    Olivier Chapelle and Lihong Li. 2011. An Empirical Evaluation of Thompson Sampling. InAdvances in Neural Information Processing Systems (NeurIPS)

    2011

  8. [8]

    Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazaros Karydas, Anand Ra- jagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. 2021. Batch Active Learning at Scale. InAdvances in Neural Information Processing Systems (NeurIPS)

    2021

  9. [9]

    Berg, Robert Nowak, Roshan Sumbaly, Matei Zaharia, and I

    Cody Coleman, Edward Chou, Julian Katz-Samuels, Sean Culatana, Peter Bailis, Alexander C. Berg, Robert Nowak, Roshan Sumbaly, Matei Zaharia, and I. Zeki Yalniz. 2022. Similarity Search for Efficient Active Learning and Search of Rare Concepts. InProceedings of the AAAI Conference on Artificial Intelligence

    2022

  10. [10]

    Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. InProceedings of the 10th ACM Conference on Recommender Systems (RecSys). ACM, 191–198

    2016

  11. [11]

    Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. 2024. Vision Transformers Need Registers. InProceedings of the 12th International Conference on Learning Representations (ICLR)

    2024

  12. [12]

    Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets.Journal of Machine Learning Research7 (2006), 1–30

    2006

  13. [13]

    Ding et al. 2024. Learning to Rank for Active Learning via Multi-Task Bilevel Optimization. InProceedings of the 41st International Conference on Machine Learning (ICML)

    2024

  14. [14]

    Mehdi Elahi, Francesco Ricci, and Neil Rubens. 2016. A Survey of Active Learning in Collaborative Filtering Recommender Systems.Computer Science Review20 (2016), 29–50

    2016

  15. [15]

    Benoît Frénay and Michel Verleysen. 2014. Classification in the Presence of Label Noise: A Survey.IEEE Transactions on Neural Networks and Learning Systems25, 5 (2014), 845–869

    2014

  16. [16]

    Sebastian Seung, Eli Shamir, and Naftali Tishby

    Yoav Freund, H. Sebastian Seung, Eli Shamir, and Naftali Tishby. 1992. Informa- tion, Prediction, and Query by Committee. InAdvances in Neural Information Processing Systems (NeurIPS)

    1992

  17. [17]

    Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021. Advances and Challenges in Conversational Recommender Systems: A Survey.AI Open2 (2021), 100–126

    2021

  18. [18]

    Claudio Gentile, Shuai Li, and Giovanni Zappella. 2014. Online Clustering of Bandits. InProceedings of the 31st International Conference on Machine Learning (ICML)

    2014

  19. [19]

    Ran Gilad-Bachrach, Amir Navot, and Naftali Tishby. 2005. Query by Committee Made Real. InAdvances in Neural Information Processing Systems (NeurIPS)

    2005

  20. [20]

    Nirschl, and Serena Yeung-Levy

    Sanket Rajan Gupte, Josiah Aklilu, Jeffrey J. Nirschl, and Serena Yeung-Levy

  21. [21]

    Isabelle Guyon, Gavin Cawley, Gideon Dror, and Vincent Lemaire. 2011. Results of the Active Learning Challenge. InJMLR Workshop and Conference Proceedings: Active Learning and Experimental Design, Vol. 16. 19–45

    2011

  22. [22]

    Guy Hacohen, Avihu Dekel, and Daphna Weinshall. 2022. Active Learning on a Budget: Opposite Strategies Suit High and Low Budgets. InProceedings of the 39th International Conference on Machine Learning (ICML), Vol. 162. PMLR, 8175–8195

    2022

  23. [23]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-Scale Similarity Search with GPUs.IEEE Transactions on Big Data7, 3 (2021), 535–547

    2021

  24. [24]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. InInternational Conference on Learning Representations (ICLR)

    2015

  25. [25]

    Darius Lam, Richard Kuzma, Kevin McGee, Samuel Dooley, Michael Laielli, Matthew Klaric, Yaroslav Bulatov, and Brendan McCord. 2018. xView: Objects in Context in Overhead Imagery.arXiv preprint arXiv:1802.07856(2018)

    Pith/arXiv arXiv 2018

  26. [26]

    Meyer, and Iryna Gurevych

    Ji-Ung Lee, Christian M. Meyer, and Iryna Gurevych. 2020. Empowering Active Learning to Jointly Optimize System and User Demands. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL). ACL

    2020

  27. [27]

    Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020. Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems. InProceedings of the 13th International Conference on Web Search and Data Mining (WSDM). ACM

    2020

  28. [28]

    Gaston Lenczner, Adrien Chan-Hon-Tong, Bertrand Le Saux, Nicola Luminari, and Guy Le Besnerais. 2022. DIAL: Deep Interactive and Active Learning for Semantic Segmentation in Remote Sensing.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing15 (2022)

    2022

  29. [29]

    Pietro Lesci and Andreas Vlachos. 2024. AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics, 8445–8464

    2024

  30. [30]

    Li, Alex Tamkin, Noah Goodman, and Jacob Andreas

    Belinda Z. Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. 2025. Eliciting Human Preferences with Language Models.arXiv preprint arXiv:2310.11589 (2025). ICLR 2025

    Pith/arXiv arXiv 2025

  31. [31]

    Schapire

    Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010. A Contextual- Bandit Approach to Personalized News Article Recommendation. InProceedings of the 19th International Conference on World Wide Web (WWW). ACM, 661–670

    2010

  32. [32]

    Shuai Li, Alexandros Karatzoglou, and Claudio Gentile. 2016. Collaborative Filtering Bandits. InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM

    2016

  33. [33]

    Xu, Ouyu Lan, and Xiang Ren

    Bill Yuchen Lin, Dong-Ho Lee, Frank F. Xu, Ouyu Lan, and Xiang Ren. 2019. AlpacaTag: An Active Learning-based Crowd Annotation Framework for Se- quence Tagging. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL). ACL

    2019

  34. [34]

    Xiao Ma, Liqin Zhao, Guan Huang, Zhi Wang, Zelin Hu, Xiaoqiang Zhu, and Kun Gai. 2018. Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate. InProceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1137– 1140

    2018

  35. [35]

    2013.Fine-Grained Visual Classification of Aircraft

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013.Fine-Grained Visual Classification of Aircraft. Technical Report 1306.5151. arXiv

    Pith/arXiv arXiv 2013

  36. [36]

    Palmer, Craig Packer, and Jeff Clune

    Mohammad Sadegh Norouzzadeh, Anh Nguyen, Margaret Kosmala, Alexandra Swanson, Meredith S. Palmer, Craig Packer, and Jeff Clune. 2018. Automatically Identifying, Counting, and Describing Wild Animals in Camera-Trap Images with Deep Learning.Proceedings of the National Academy of Sciences115, 25 (2018), E5716–E5725

    2018

  37. [37]

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Lab...

    2024

  38. [38]

    Jiarui Qin, Jiachen Zhu, Bo Chen, Zhirong Liu, Weiwen Liu, Ruiming Tang, Rui Zhang, Yong Yu, and Weinan Zhang. 2022. RankFlow: Joint Optimization of Multi-Stage Cascade Ranking Systems as Flows. InProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM

    2022

  39. [39]

    Caleb Robinson, Anthony Ortiz, Kolya Malkin, Blake Elias, Andi Peng, Dan Morris, Bistra Dilkina, and Nebojsa Jojic. 2020. Human-Machine Collaboration for Fast Land Cover Mapping. InProceedings of the AAAI Conference on Artificial Intelligence

    2020

  40. [40]

    Rousseeuw

    Peter J. Rousseeuw. 1987. Silhouettes: A Graphical Aid to the Interpretation and Validation of Cluster Analysis.J. Comput. Appl. Math.20 (1987), 53–65

    1987

  41. [41]

    Neil Rubens, Mehdi Elahi, Masashi Sugiyama, and Dain Kaplan. 2015. Active Learning in Recommender Systems. InRecommender Systems Handbook(2nd ed.). Springer, Chapter 24

    2015

  42. [42]

    2012.Active Learning

    Burr Settles. 2012.Active Learning. Morgan & Claypool Publishers

    2012

  43. [43]

    Sebastian Seung, Manfred Opper, and Haim Sompolinsky

    H. Sebastian Seung, Manfred Opper, and Haim Sompolinsky. 1992. Query by Committee. InProceedings of the 5th Annual ACM Workshop on Computational Learning Theory (COLT). ACM, 287–294

    1992

  44. [44]

    Yi Su and Minmin Chen. 2023. Nonlinear Bandits Exploration for Recommen- dations. InProceedings of the 17th ACM Conference on Recommender Systems (RecSys). 1054–1057

    2023

  45. [45]

    Chi, and Minmin Chen

    Yi Su, Haokai Lu, Yuening Li, Liang Liu, Shuchao Bi, Ed H. Chi, and Minmin Chen

  46. [46]

    Chi, Cristos Goodrow, Su-Lin Wu, Lexi Baugher, and Minmin Chen

    Yi Su, Xiangyu Wang, Elaine Ya Le, Liang Liu, Yuening Li, Haokai Lu, Benjamin Lipshitz, Sriraj Badam, Lukasz Heldt, Shuchao Bi, Ed H. Chi, Cristos Goodrow, Su-Lin Wu, Lexi Baugher, and Minmin Chen. 2024. Long-Term Value of Explo- ration: Measurements, Findings and Algorithms. InProceedings of the 17th ACM International Conference on Web Search and Data Mi...

    2024

  47. [47]

    InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)

    Multi-Task Neural Linear Bandit for Exploration in Recommender Systems. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD). 5723–5730

  48. [48]

    Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. 2018. The HAM10000 Dataset, a Large Collection of Multi-Source Dermatoscopic Images of Common Pigmented Skin Lesions.Scientific Data5 (2018), 180161

    2018

  49. [49]

    Zhen Tan, Dawei Li, Song Wang, et al. 2024. Large Language Models for Data Annotation: A Survey.arXiv preprint arXiv:2402.13446(2024)

    Pith/arXiv arXiv 2024

  50. [50]

    Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE.Journal of Machine Learning Research9 (2008), 2579–2605

    2008

  51. [51]

    Kanevski, and Jordi Muñoz- Marí

    Devis Tuia, Michele Volpi, Loris Copa, Mikhail F. Kanevski, and Jordi Muñoz- Marí. 2011. A Survey of Active Learning Algorithms for Supervised Remote Sensing Image Classification.IEEE Journal of Selected Topics in Signal Processing 5, 3 (2011), 606–617

    2011

  52. [52]

    Wang et al . 2025. LCRON: Learning Cascade Ranking as One Network. In Proceedings of the 42nd International Conference on Machine Learning (ICML)

    2025

  53. [53]

    2011.The Caltech-UCSD Birds-200-2011 Dataset

    Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. 2011.The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-

    2011

  54. [54]

    California Institute of Technology

  55. [55]

    Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods.Biometrics Bulletin1, 6 (1945), 80–83

    1945

  56. [56]

    Qingzhong Wang, Haifang Li, Haoyi Xiong, Wen Wang, Jiang Bian, Yu Lu, Shuaiqiang Wang, Zhicong Cheng, Dawei Yin, and Dejing Dou. 2022. A Simple yet Effective Framework for Active Learning to Rank. InAdvances in Neural Information Processing Systems (NeurIPS)

    2022

  57. [57]

    Zhe Wang, Liqin Zhao, Biye Jiang, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai

  58. [58]

    Ofer Yehuda, Avihu Dekel, Guy Hacohen, and Daphna Weinshall. 2022. Active Learning Through a Covering Lens. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 35

    2022

  59. [59]

    Kawtar Zaher, Olivier Buisson, and Alexis Joly. 2026. Positive-First Most Am- biguous: A Simple Active Learning Criterion for Interactive Retrieval of Rare Categories.arXiv preprint arXiv:2603.24480(2026)

    Pith/arXiv arXiv 2026

  60. [60]

    Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2022. A Survey of Human-in-the-loop for Machine Learning.Future Genera- tion Computer Systems135 (2022), 364–381

    2022

  61. [61]

    Yi Yang and Shawn Newsam. 2010. Bag-of-Visual-Words and Spatial Extensions for Land-Use Classification.Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems(2010), 270–279

    2010

  62. [64]

    Jifan Zhang, Julian Katz-Samuels, and Robert Nowak. 2022. GALAXY: Graph- based Active Learning at the Extreme. InProceedings of the 39th International Conference on Machine Learning (ICML), Vol. 162. PMLR, 26223–26238

    2022

  63. [65]

    Dongruo Zhou, Lihong Li, and Quanquan Gu. 2020. Neural Contextual Bandits with UCB-based Exploration. InProceedings of the 37th International Conference on Machine Learning (ICML). 15

    2020

  64. [2020]

    InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (DLP-KDD Workshop)

    COLD: Towards the Next Generation of Pre-Ranking System. InProceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (DLP-KDD Workshop)

  65. [2024]

    14 FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery

    Revisiting Active Learning in the Era of Vision Foundation Models.arXiv preprint arXiv:2401.14555(2024). 14 FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery

    Pith/arXiv arXiv 2024