{"id":"cd88ec71-d843-463f-8ca4-a4d6872f3cc1","arxiv_id":"2607.25276","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":11,"one_line_summary":"A precision-triggered retrieve-then-rank active-learning pipeline achieves better label efficiency and F1 than recent single-class discovery baselines on three image benchmarks.","lead":"This paper proposes FunnelAL, an active-learning system that first retrieves a candidate set via embedding similarity and then ranks it with an exploitation ranker that switches to committee-based exploration when easy positives run out. In simulations on bird, aircraft, and aerial-image benchmarks, it reaches higher F1 with fewer labels than recent single-class discovery baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle-based budget and early stopping may inflate FunnelAL's label-efficiency lead; fixed-budget rerun needed.","rationale":"The reader's weakest assumption and verdict both center on the oracle-based protocol, and I agree that this is the most load-bearing concern. The paper is otherwise internally coherent: the ablations show the precision trigger is the active component, the noise study is thoughtful, and the limitations section is candid. The most serious soft spot is that the evaluation protocol uses ground-truth budget and stopping, and those choices feed directly into the metrics that support the paper's strongest claims (AULC, B@90, rounds). Addressing this with a fixed-budget, no-oracle-stopping rerun is the single most informative test. The reader's CONDITIONAL verdict already reflects this, so no change in verdict is needed; the headline should remain conditional until the test is run.","tokens_in":24828,"tokens_out":6157,"duration_ms":68127,"concrete_test":"Re-run the Table 1 protocol with (a) a fixed budget B = 500 for every class (no 3×n_pos scaling) and (b) no ground-truth early termination; instead use an observed-data stopping rule (e.g., stop when no new positives appear in the last w = 3 batches). Compare FunnelAL vs PF-MA and GAL on AULC, final F1, and rounds. If FunnelAL still leads under the fixed budget and estimated stopping, the oracle protocol is not load-bearing; if the lead shrinks or reverses, the headline claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 sets B = min(3×n_pos, 500) and stops each run as soon as all working-set positives are discovered, both using ground-truth knowledge unavailable in practice. These choices directly shape the headline metrics: AULC is measured over the learning curve up to termination, B@90 and 'fewest rounds' are termination-based, and even final F1 is evaluated at that point. Because FunnelAL discovers positives faster (higher Pos. Rec.), it terminates earlier, so its curve is measured over a shorter, high-F1 prefix; slower baselines continue to the full budget, often labeling less-informative negatives. The budget formula itself scales with n_pos, which can favor methods that exhaust small positive sets quickly. The paper candidly acknowledges the early-stop limitation in Section 5.2, but all headline results are reported under this oracle protocol. If FunnelAL's AULC/rounds advantage is partly an artifact of stopping exactly when it has found all positives, then the central 'annotation efficiency' claim is not yet established for a deployable system.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"FunnelAL is a retrieve-then-rank active-learning system for single-class discovery. Starting from one positive and one negative seed, it (1) retrieves a candidate set via embedding-based DWVA-KNN, (2) ranks that set with a precision-triggered hybrid of RankNet exploitation and QBC exploration, and (3) updates positive/negative sets from annotator feedback. The paper evaluates on CUB-200-2011, FGVC-Aircraft, and UC Merced Land Use under a one-vs-rest protocol with DINOv2 embeddings, comparing to eleven ablations/baselines including recent single-class methods GAL and PF-MA. Headline claims are best final F1, best annotation efficiency (AULC), and fewest annotation rounds across all three benchmarks, with retained advantage under simulated annotator label noise.","tokens_in":25117,"tokens_out":7373,"duration_ms":79979,"significance":"If the empirical claims hold, FunnelAL is a practical and conceptually interesting contribution: it transfers the multi-stage funnel architecture from recommender systems to active learning and demonstrates consistent gains across fine-grained and domain-shifted benchmarks. The paper has clear strengths: three diverse datasets, a large number of one-vs-rest tasks, class-level paired Wilcoxon tests, careful ablations of the retrieval and ranking components, a computational cost analysis, a noise-robustness study, and an unusually candid limitations section. However, the central label-efficiency claim (AULC, rounds, B@90) rests on an evaluation protocol that uses ground-truth positive counts for the budget and ground-truth knowledge for early stopping. Until that protocol is replaced or supplemented by a fixed-budget, no-oracle comparison, the magnitude of the claimed efficiency advantage is not established for a deployable system.","major_comments":[{"comment":"The evaluation protocol uses oracle knowledge in two load-bearing places: B=min(3×n_pos,500) uses the true positive count, and each run terminates early as soon as all working-set positives are discovered. These choices directly shape the headline metrics: AULC is computed over the learning curve up to termination, B@90 and round counts are termination-based, and even final F1 is measured at different annotation counts across methods. Because FunnelAL discovers positives faster, it terminates earlier; slower baselines continue to the full budget, often labeling less-informative negatives, inflating FunnelAL's AULC and rounds advantage. The paper acknowledges the early-stop issue in Section 5.2, but all headline results still use this protocol. A fixed-budget rerun with a budget that does not depend on n_pos, and with no oracle early stopping (or a stopping rule based only on observed lab","section":"Section 4.3, Algorithm 1"},{"comment":"The precision-triggered adaptive policy, identified as load-bearing in Table 2, depends on several free parameters: τ=0.7, w=3, γ=1.2, the 60/40 RankNet/QBC split, plus retrieval parameters α=2, K=512, C=5000, and k0=20. The paper states these are fixed across datasets and 'not tuned per dataset,' but this does not rule out selection on the collection of the three benchmarks, and no sensitivity analysis is reported. The paper itself says 'a systematic sweep of these values remains warranted' (Section 5.2). Given that the central claim is that the precision-triggered mechanism generalizes, a sensitivity analysis or a hold-one-benchmark-out parameter selection is necessary to show the results are not artifacts of particular values.","section":"Section 3.4, Section 5.2"},{"comment":"The abstract and conclusion state that FunnelAL attains the best final F1 on all three benchmarks, but on CUB-200 the margin over PF-MA is 0.001 (0.881 vs 0.880) and on UC Merced it is 0.002 (0.952 vs 0.950). No paired Wilcoxon p-values are reported for these final-F1 comparisons, and the text later characterizes several such gaps as statistical ties (e.g., Section 4.4.2 reports p=0.31 for GAL on FGVC-Aircraft). The headline claim should be qualified as 'numerically best / statistically tied,' or the relevant p-values should be reported. This is not purely presentational: the unqualified wording appears in the abstract and duplicate claims in the conclusion.","section":"Table 1, Section 4.4.1, Abstract"},{"comment":"GAL is reimplemented using a 'full weight-vector impact appropriate for high-dimensional features,' and no code is provided for FunnelAL or the baselines. Since GAL is a recent and directly comparable single-class discovery method, an inaccurate or unverified reimplementation could bias the comparison. The paper should either release code for all methods or provide a verification of the reimplementation against official published results. Without this, readers cannot confirm that GAL is not disadvantaged by implementation choices.","section":"Section 4.2"}],"minor_comments":[{"comment":"No error bars or confidence intervals are shown for the macro-averaged metrics. Since some differences are very small (e.g., F1 margins of 0.001–0.002), adding per-class variance or reporting the full Wilcoxon test results for all headline pairs would help readers gauge stability.","section":"Tables 1 and 3"},{"comment":"The 'shared 90% bar' is described as 90% of the best strategy's final value, distinct from B@90's per-strategy target. This distinction is important and correct, but it would be clearer to define the dashed line explicitly in the caption of each figure rather than only in the text.","section":"Figures 4 and 5"},{"comment":"The DWVA score s_j is the product of a retrieval count fraction and a distance weight. If a candidate is not retrieved by any positive, the first factor is zero and s_j=0, but the definition of the second factor (1/(1+d_bar_j)) when no positive retrieves the candidate is not specified. Please clarify the empty-set case.","section":"Equation (1)"},{"comment":"The computational cost comparison mixes timings from an A100 host and a CPU-only host. The paper justifies cross-host contrasts, but it would be more transparent to report the host for each row or to provide a single-host comparison for all strategies.","section":"Section 4.4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-organized and unusually honest about limitations, and the core idea is promising. The decisive issue is the oracle-based evaluation protocol: the central 'annotation efficiency / fewest rounds' claim is likely to change under a fixed-budget, no-early-stop protocol. I would be willing to reconsider after a revised evaluation, ideally with code release. The hyperparameter sensitivity and final-F1 significance issues are secondary but should also be addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper proposes a genuinely new layout for single-class discovery — an explicit retrieval funnel (DWVA-KNN) followed by a two-arm ranker (RankNet for exploitation, QBC for exploration), with a precision-triggered switch that fires when rolling batch precision drops. That adaptive transition, plus the recommender-funnel framing, is not in the SEALS/AnchorAL/GAL/PF-MA line. The paper is worth reading for that design alone.\n\nWhere it earns credit: the ablations are well constructed. The precision trigger is shown to be load-bearing — swapping it for a count-based trigger costs 16 pp F1 on CUB-200 — and the batch-growth rule is honestly shown to buy rounds, not accuracy. The noise study is a real strength; the observation that label noise acts as accidental exploration for single-stage baselines on CUB-200 is clever and well supported. The limitations section is candid about oracle-based stopping, out-of-distribution seeds, and the absence of a user study. Computations are profiled and the cost discussion is fair.\n\nThe soft spot is the evaluation protocol, and it is not minor. Budget is B = min(3 × n_pos, 500), and runs stop early as soon as all positives in the working set are found — both use ground-truth knowledge. That directly shapes the headline metrics: AULC is measured over the curve up to termination, B@90 and \"fewest rounds\" are termination-based, and even final F1 is evaluated at that stopping point. FunnelAL discovers positives faster, so it stops earlier, and its curve is a shorter, high-F1 prefix. A fixed-budget rerun with no early stopping is needed before the \"best annotation efficiency\" claim is credible. The paper acknowledges this in Section 5.2, but it still reports all headline numbers under the oracle protocol. Also: no error bars in Tables 1 and 3, no code release, and GAL is reimplemented with a modification (full weight-vector impact). The fixed hyperparameters (τ, γ, w, the 60/40 split) were chosen on the same benchmarks; the authors say they are not per-dataset tuned, but a sensitivity sweep is missing. These are addressable, not fatal.\n\nBottom line: the architecture is real, the experiments are mostly careful, and the final-F1 gaps to GAL/PF-MA are usually small; the large AULC advantage is exactly the metric most exposed to the protocol critique. This deserves a serious referee — definitely not a desk reject. I'd send it to review and ask for a fixed-budget evaluation, error bars, and code. I'd bring it to reading group; the protocol question is a good discussion.","headline":"FunnelAL's precision-triggered retrieve-then-rank design is a real contribution, but the headline annotation-efficiency numbers rest on an oracle-based protocol that needs a fixed-budget rerun.","tokens_in":25621,"tokens_out":3063,"would_cite":true,"duration_ms":32039,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a retrieve-then-rank funnel with an adaptive exploit-explore switch finds rare target classes starting from a single positive example, using fewer labels than single-stage active learning.","keywords":["active learning","single-class discovery","retrieve-then-rank","recommender systems","query-by-committee","RankNet","annotation efficiency","label noise"],"falsifier":"Run the same three benchmarks with a fixed annotation budget for every class, no early termination, and no knowledge of the true positive count; if FunnelAL no longer leads on AULC or matches rounds, the central efficiency claim fails. A second falsifier is a real-annotator user study testing whether funnel recommendations and manual explore-exploit control save wall-clock time.","tokens_in":24709,"feed_emoji":"🎯","tokens_out":8141,"duration_ms":75305,"temperature":0.7,"pith_summary":"FunnelAL is an active-learning system for finding all instances of a single target class in a large image corpus, starting from just one positive and one negative example. The paper argues that by decomposing sample selection into a retrieval stage that narrows the pool via embedding similarity and a ranking stage that alternates between exploiting confident positives (RankNet) and exploring uncertain boundary regions (Query-by-Committee), an annotation campaign finds more positives per label than conventional one-stage acquisition functions. The central claim is that this 'funnel' architecture, borrowed from recommender systems, makes annotation both cheaper per round and more efficient in total labels, and that an automatic precision-triggered switch between exploitation and exploration is what delivers the gains. On three benchmarks spanning fine-grained species, manufactured objects, and domain-shifted aerial imagery, FunnelAL reports the best final F1, the best area under the learning curve, and the fewest annotation rounds, and it degrades more gracefully than uncertainty-based baselines when annotators make labeling errors. The paper further claims that the newest single-class discovery methods (GAL and PF-MA) at best match its final accuracy, but only at consistently higher labeling cost.","feed_headline":"Funnel-style active learning finds rare classes with fewer labels","feed_subtitle":"One positive and one negative seed suffice; the system wins on F1 and annotation efficiency across three benchmarks.","key_machinery":"The central object is the multi-stage funnel: Stage 1 retrieval uses Distance-Weighted Vote Aggregation (DWVA) over embedding-based nearest-neighbor searches to cut the pool to a fixed-size candidate set; Stage 2 ranking uses a RankNet pairwise ranker as the exploitation arm and a four-member Query-by-Committee committee as the exploration arm; the precision trigger—a one-way transition when the mean precision of the last three batches falls below τ=0.7—moves the system from pure exploitation to a 60/40 RankNet-plus-QBC hybrid, and a batch-growth rule multiplies batch size by 1.2 after each fully-confirmed batch. The machinery's work is to keep computational cost proportional to the candidat","core_discovery":"The central discovery is that the funnel decomposition itself—an explicit embedding-retrieval prefilter followed by a two-arm ranking stage—is what drives label efficiency in single-class discovery, and that the interaction between the arms should be adaptive rather than fixed. Starting from one positive and one negative seed, FunnelAL scores the corpus by a distance-weighted vote over the current positives' nearest neighbors to form a candidate set, then ranks that set with a RankNet pairwise ranker while batch precision stays high, growing the batch size as long as every sample is confirmed positive. When the rolling mean batch precision drops below a threshold (0.7 over three batches), th","pith_inferences":["The reported round-count savings depend on the paper's early-termination oracle; a fixed-budget comparison without oracle knowledge would likely reduce the round advantage, though the per-class recall trajectory gains suggest some label-efficiency benefit may persist.","The 60/40 exploit-explore split and threshold τ are fixed; making them per-class adaptive with contextual bandits, as the paper suggests, could extend the funnel's advantage to uneven class structures.","Mild label noise acted as accidental exploration for trapped baselines; this implies that adding a small amount of explicit diversity to any pure-exploitation method may reproduce the noise benefit without the errors, a testable design tweak.","The paper leaves the manual annotator-control affordance unevaluated; a user study would determine whether human judgment in choosing exploit versus explore beats the automated trigger."],"forward_implications":["Annotation systems for single-class discovery can adopt the recommender funnel pattern and reach a given F1 with fewer total labels and fewer human rounds than fixed acquisition functions.","The precision-triggered transition self-calibrates per class, switching when easy positives are exhausted, and is far more reliable than a fixed positive-count schedule, which loses up to 16.5 F1 points and 30.8 AULC points on the bird benchmark.","Under annotator label noise at 5–15%, the system remains first or statistically tied for first, while boundary-seeking uncertainty baselines degrade two to three times faster.","Because all stages operate purely on embeddings, the method transfers to other modalities with a strong pretrained encoder, though the paper demonstrates this only on images."],"fun_headline_variants":["FunnelAL finds rare classes with fewer labels","Adaptive funneling finds rare classes with fewer labels","Retrieve, rank, discover rare classes with fewer labels","FunnelAL cuts label costs for rare-class discovery","One seed each finds rare classes with fewer labels"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation protocol sets the annotation budget from the true positive count and stops once all positives are found—information a real user would not have—so the reported label-efficiency and round-count advantages may not survive a protocol where budget and stopping are fixed without oracle knowledge.","fun_headline_variants_meta":{"raw":{"variants":["FunnelAL finds rare classes with fewer labels","Adaptive funneling finds rare classes with fewer labels","Retrieve, rank, discover rare classes with fewer labels","FunnelAL cuts label costs for rare-class discovery","One seed each finds rare classes with fewer labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001214,"raw_usage":{"total_tokens":4872,"prompt_tokens":822,"completion_tokens":4050,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":3975}},"tokens_in":566,"tokens_out":4050,"duration_ms":30076,"temperature":1.0,"reasoning_tokens":3975,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:53:21.613325+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same three benchmarks with a fixed annotation budget for every class, no early termination, and no knowledge of the true positive count; if FunnelAL no longer leads on AULC or matches rounds, the central efficiency claim fails. A second falsifier is a real-annotator user study testing whether funnel recommendations and manual explore-exploit control save wall-clock time.","supporting_citations":[],"review_version":1}