{"id":"b576e77e-10a6-4a33-8620-5f7c277e1dec","arxiv_id":"2506.03817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 4.6 million-experiment grid across 92 datasets and 28 strategy implementations shows that active learning hyperparameters, and especially the software library behind a strategy, can change which methods look best.","lead":"The authors ran over 4.6 million active learning experiments to test how every setup choice changes which query strategy wins. They found that the concrete software implementation of a strategy often matters more than the strategy itself, and that a few thousand well-chosen runs can already give stable rankings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sec IV-G's conclusion that implementation beats strategy rests on correlations computed after imputing missing results as 0; for QUIRE the missing cells are concentrated in exactly the compared implementations, so the 44.5% correlation may be an artifact of the imputation.","rationale":"Good faith reading: the paper is a large-scale empirical mapping of AL hyperparameter influence, with open data and code and an honest discussion of missingness. The central claim is plausible and the scale is a genuine contribution. The most load-bearing point, however, is not the breadth of the grid (which the reader emphasized) but the internal validity of the key exhibit: QUIRE's 44.5% correlation. The paper's own text confirms that the missing cells are concentrated in QUIRE and large datasets, and that missing values are replaced by 0 before computing correlations. Because the Pearson correlation is sensitive to the joint distribution of the two variables, imputing worst-case values for informative-missing cells in only one or both compared implementations can create or exaggerate dissimilarity. This is a concrete, mechanistically specified threat. The reader identified 0-imputation as a red flag but did not connect it to the flagship QUIRE exhibit; the weakest_assumption was instead external generalizability. My check would settle the internal threat directly. If the complete-case correlation is similar, the paper's central claim is supported within its domain; if it jumps, the claim needs re-scoping. The verdict remains CONDITIONAL as the reader proposed: the authors should rerun the key correlations without 0-imputation and report missingness per strategy before the headline conclusion is accepted.","tokens_in":75350,"tokens_out":5508,"duration_ms":54541,"concrete_test":"Recompute the metric-based Pearson correlation between QUIRE (libact) and QUIRE (scikit-activeml) using only complete cases: drop every hyperparameter combination where either implementation has a missing/errored/timeout result, and compute the correlation on the remaining observed pairs. Report the number of complete pairs and the missingness pattern per dataset for each implementation. Also re-run the metric-based heatmap for all 28 strategies with listwise deletion and compare to the zero-imputed version. If the QUIRE complete-case correlation rises to, say, 0.7 or higher (or if the two implementations are missing on disjoint datasets), the Sec IV-G conclusion is an artifact of the 0-imputation and should be revised; if it remains around 0.45, the claim survives this test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is stated in Sec IV-G: 'the specific implementation of AL strategies can impact performance more than the underlying strategy itself,' illustrated by the 44.5% metric-based correlation between the two QUIRE implementations (libact and scikit-activeml). This evidence is computed on result vectors in which missing grid cells are filled with 0, described in Sec IV-A/IV-B as an 'interpolation strategy.' The missing cells are not random: Sec IV-A states they 'are mainly from the largest datasets and both implementations of the QUIRE AL strategy,' and Sec III imposes a 5-minute-per-cycle runtime limit. If the two QUIRE implementations time out on different subsets of the large datasets (e.g., because of different kernel implementations or internal tolerances), then zero-imputation injects systematically discordant pairs into the Pearson correlation, mechanically lowering the observed correlation. Because the low QUIRE correlation is the flagship example for the implementation-beats-strategy conclusion, the conclusion is not secure unless the correlation survives a complete-case analysis. The paper also notes errors 'for almost all strategies' beyond runtime, which are handled the same way. This is an internal-validity threat to the central claim, independent of the generalizability question about deep learners or non-tabular data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a large-scale active learning (AL) benchmark: a hyperparameter grid of 4,636,800 combinations built from 92 tabular datasets, 28 AL strategy implementations from five frameworks, three learners, six batch sizes, five train-test splits, and 20 start sets. The authors record metrics and queried sample sets for the roughly 4.56 million combinations that completed within a 5-minute per-cycle limit, then quantify the influence of each hyperparameter via metric-based Pearson correlations, queried-samples Jaccard similarity, and leaderboard-ranking Kendall tau-b correlations. They conclude that the specific implementation of an AL strategy can affect performance more than the strategy itself (exemplified by a 44.5% correlation between two QUIRE implementations), that dataset diversity matters more than specific dataset choices, and that roughly 4,000 randomly drawn hyperparameter combinations already produce leaderboard rankings highly correlated with the full grid. They close with recommendations for designing reproducible AL evaluations.","tokens_in":75610,"tokens_out":5073,"duration_ms":50162,"significance":"If the central empirical claims survive scrutiny, this is one of the largest and most transparent AL benchmarking efforts to date, with the source code and raw results publicly released. The paper provides a useful mapping of which hyperparameters matter (batch size, learner model, strategy implementation) and which appear relatively inert (train-test splits, start sets), and the practical recommendation of a ~4,000-combination grid is actionable. The authors are also commendably explicit about their runtime limits, missing data, and the choice to fix aggregation metrics. However, the flagship claim about implementation beating strategy rests on correlations computed after zero-imputing a non-randomly missing subset of the grid, and the paper does not provide a complete-case check; this is a load-bearing internal-validity concern.","major_comments":[{"comment":"The central claim that the QUIRE implementations correlate only 44.5% is computed on result vectors in which missing cells are filled with a fixed value of 0 (the interpolation strategy described in Sec. IV-B).  Sec. IV-A states that the missing results \"are mainly from the largest datasets and both implementations of the QUIRE AL strategy,\" so the missingness is concentrated in exactly the two implementations being compared.  If the two implementations time out on different subsets of the large datasets, zero-imputation injects discordant pairs into the Pearson correlation and mechanically lowers the observed value.  The paper must show that the low QUIRE correlation survives a complete-case analysis (restricting to hyperparameter combinations where both implementations completed) or under a more realistic imputation model; without this, the flagship example for \"implementation beats strategy\" is not secure.","section":"Sec. IV-A and IV-G"},{"comment":"The \"4,000 combinations suffice\" threshold in Fig. 11 is derived from the leaderboard ranking-invariance correlation between random subsets and the full grid, and the full grid itself contains the 75,924 zero-imputed missing cells.  Because those missing cells are concentrated in the largest datasets and in QUIRE implementations, the subset-stability analysis may reflect the imputation pattern rather than the true structure of AL results.  The authors should re-run the subsampling analysis on the complete-case portion of the grid (or with a missing-indicator included) to demonstrate that the 4,000-combination recommendation is robust to the treatment of missing results.","section":"Sec. IV-B3 and IV-J"},{"comment":"The conclusion that implementation choices dominate strategy identity is also supported by the average within-framework versus cross-framework correlations in Fig. 8, but the paper provides no uncertainty quantification for the differences between these averages.  The reported standard deviations overlap considerably (e.g., a mean of 89.8% for ALI with the lowest standard deviation still leaves a wide range), so a permutation or bootstrap test is needed to establish that the within-framework correlation is significantly higher than the cross-framework baseline.  This is a secondary support for the central claim, but it should be reported as a provisional observation until such a test is added.","section":"Sec. IV-G and Fig. 8"}],"minor_comments":[{"comment":"The queried-samples heatmap formula is internally inconsistent: the text says \"we subtract the Jaccard index from 1 so that 1 indicates full similarity,\" but the displayed expression is 1 - (sum of Jaccard indices)/|V|, which would be 0 for identical sets, while the diagonal of the displayed matrix shows 1.  Please clarify whether the heatmaps plot J or 1-J, and correct the formula or the caption accordingly.","section":"Sec. IV-B2"},{"comment":"There are several typos, e.g., \"complxeity\" should be \"complexity,\" \"practicioners\" should be \"practitioners,\" and \"framworks\" should be \"frameworks.\"  A careful proofreading pass is needed.","section":"Sec. II-C"},{"comment":"The sentence \"we use the the F1-score\" contains a duplicated article; it should read \"we use the F1-score.\"","section":"Sec. IV-C"},{"comment":"The 28x28 strategy heatmaps and the framework correlation figure are nearly impossible to read at preprint resolution; the axis labels and cell values are illegible.  Please provide vector graphics or a higher-resolution version, and consider an interactive or zoomable appendix.","section":"Fig. 7 and Fig. 8"},{"comment":"The grid notation in Sec. II-D lists \"c\" and \"M\" in the hyperparameter sets but then states they are not part of the experimental grid; this is a notational muddle that should be clarified in the text rather than only in the table.","section":"Sec. II-D"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial empirical contribution and the authors have been admirably transparent about their data and code.  The main risk is that the zero-imputation of missing QUIRE results may be creating the headline finding; a complete-case check is essential before the implementation-beats-strategy claim is accepted.  If the authors add that analysis (and a small amount of uncertainty quantification for the correlation comparisons), the paper would be a strong candidate for acceptance in this venue.  The missingness concentration is explicitly acknowledged in Sec. IV-A, so the fix is within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the largest AL hyperparameter grid I have seen, openly documented, and the practical recommendations are useful. But the headline claim that implementation can beat strategy, at least the QUIRE 44.5% correlation in Sec IV-G, rests on zero-imputed missing cells. That specific number is not secure until they run a complete-case analysis.\n\nWhat is genuinely new: the scale (4.6M combinations, 4.56M completed) and the public release of code and raw results. That is real evidence and a useful community resource. Several findings look solid: start sets and train-test splits barely matter, batch size and learner model do, and the ~4,000-combination threshold for a stable leaderboard is concrete and actionable. The cross-framework clustering in Fig. 8 is a genuinely interesting observation that would be hard to get without this scale.\n\nThe soft spots, in proportion. The zero-imputation is the main one. The paper is transparent about it, but calling it 'interpolation' overstates the case: they fill missing results with 0, the worst possible outcome. Missingness is concentrated in the two QUIRE implementations and the largest datasets. If those two implementations time out on different subsets, the imputed zeros create discordant pairs that mechanically lower the Pearson correlation. The 44.5% is the flagship illustration for Sec IV-G, so the conclusion needs a sensitivity analysis without imputation before I would trust that number. I doubt the general direction is wrong — the framework clustering supports it — but the headline statistic is not secure.\n\nSecond, there are no statistical tests on correlation differences. With millions of rows, they need uncertainty bands or at least a resampling check. Third, the grid is classical tabular AL (92 datasets, MLP/SVM/RF); generalizing to deep learning and non-tabular data is a stretch, though the authors are mostly careful about this in Sec II-D.\n\nWho should read it: anyone building or using AL benchmarks, and anyone who wants a concrete argument for why AL comparisons are so contradictory. It is not a methods paper; it is an empirical resource.\n\nMy recommendation: send it to peer review. The scale and openness justify referee time. The referee should require: a complete-case correlation for the QUIRE pair, uncertainty estimates on the main correlations, and a recalibration of the Sec IV-G claim so it no longer leans on an imputed number.","headline":"The field's largest AL hyperparameter grid, with code and raw data released, is worth refereeing; but the flagship 'implementation beats strategy' number (QUIRE at 44.5%) may be an artifact of zero-imputation and needs a complete-case check.","tokens_in":76125,"tokens_out":2914,"would_cite":true,"duration_ms":29896,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In active learning, the strategy implementation can matter more than the strategy itself, and about 4,000 varied runs can reproduce full-grid rankings.","keywords":["active learning","hyperparameter grid","query strategy implementation","reproducibility","benchmark design","leaderboard ranking correlation","tabular classification","experimental grid search"],"falsifier":"Take two implementations of the same strategy, for example the two QUIRE variants, and run them on a fresh collection of, say, 60 datasets from other domains including images and text; if their metric correlation is consistently above 0.8 rather than the reported 44.5%, the implementation-dominance claim would not generalize, and if the 4,000-combination stability threshold fails on those domains, the recommended study design would need revision.","tokens_in":1925,"feed_emoji":"📊","tokens_out":2152,"duration_ms":85103,"temperature":0.7,"pith_summary":"This paper tries to explain why active learning (AL) experiments so often contradict one another by measuring the influence of every major AL hyperparameter at once. The authors ran a grid of 4.6 million hyperparameter combinations, covering 92 tabular datasets, three classical learners, and 28 implementations of query strategies, and analyzed how each choice changes the resulting rankings. Their central claim is that the concrete implementation of an AL strategy can change results more than the underlying strategy itself, with two implementations of the same QUIRE strategy correlating at only 44.5%. They also argue that roughly 4,000 well-varied hyperparameter combinations are enough to reproduce the rankings of the full grid, making controlled, reproducible AL studies computationally feasible. If true, comparisons of AL strategies that ignore implementation and hyperparameter choices should not be trusted, and benchmark conclusions need to be re-examined.","feed_headline":"Same AL strategy, different code: rankings barely correlate","feed_subtitle":"Same strategy in a different codebase can score 44.5% correlation. Roughly 4,000 varied runs stabilize rankings.","key_machinery":"The paper's central object is a dense experimental grid over the hyperparameter tuple E = (S, D, T, I, b, L), with 92 datasets, 5 train-test splits, 20 start sets, 28 strategy implementations, 6 batch sizes, and 3 learner models, totaling 4,636,800 combinations and 4,560,876 completed runs. The argument is carried by three correlation instruments: metric-based heatmaps using Pearson correlation between aggregated metric vectors; queried-samples-based heatmaps using 1 - Jaccard similarity between the sets of samples selected for labeling; and leaderboard ranking-invariance heatmaps using two-sided Kendall's tau-b between strategy rankings across datasets. The first two measure whether outcomes and selected samples are the same, while the third measures whether conclusions change, which is the aspect that matters for reproducibility. The same machinery yields the grid-subsample analysis, where Spearman correlation between rankings from random subsets of growing size and rankings from the full grid identifies the roughly 4,000-combination stability threshold.","core_discovery":"On the paper's own terms, the discovery is that AL experiment outcomes are governed by the full tuple of hyperparameters, and that tuple cannot be collapsed to a single 'which strategy' choice. Using metric-based Pearson correlations, queried-sample Jaccard similarities, and leaderboard ranking-invariance correlations, the authors find that same-named strategies from different codebases can behave almost independently (QUIRE at 44.5% metric correlation), while strategies from the same framework correlate strongly (one framework's self-correlation was 89.8%), so implementation and framework design choices can outweigh the strategic principle. They consequently recommend reporting and varying implementations explicitly, and they show that a random subset of about 4,000 combinations drawn from a varied grid yields leaderboard correlations of 80 to 100 percent with the full 4.6-million-combination grid, while smaller subsets scatter widely. They also find that batch size and learner model are high-impact, that train-test split and start set can be fixed without loss, and that the F1-score aggregated by the full mean is the safest evaluation metric.","pith_inferences":["This suggests a cheap robustness test for future AL papers: before claiming strategy A beats strategy B, re-run both in at least two independent implementations and require the ordering to survive the implementation change; otherwise the result is about code, not method.","The 4,000-combination threshold could be turned into a practical bootstrap protocol: draw small varied subsets, check ranking stability, and stop growing the grid once correlation saturates, which would reduce benchmarking cost for new AL methods.","The finding that two random baselines selected the same samples yet produced different metrics (100 percent query overlap but only 78.5 percent metric correlation) implies that even a random baseline is implementation-sensitive through sample ordering, so baselines need their iteration order fixed and reported.","One open test is whether implementation dominance persists for deep neural network settings, where uncertainty estimation varies more across frameworks than for MLP, SVM, and RF; if it weakens, the conclusion is specific to classical learners on tabular data."],"forward_implications":["If the central claim holds, any AL benchmark that varies strategies while fixing a single implementation is measuring a mix of strategy and implementation, so two papers using the same strategy in different codebases can reach opposite conclusions without either being wrong.","A study that draws a varied random grid of about 4,000 hyperparameter combinations can reproduce the full-grid leaderboard to 80 to 100 percent correlation, so reproducibility does not require millions of runs.","Batch size and learner model should be varied deliberately, with at least one small and one large batch size and the MLP as the safest single learner choice, while train-test splits and start sets can be fixed.","The full-mean F1-score is the recommended evaluation metric, because ramp-up-only or plateau-only aggregation can change conclusions and the phase boundary is dataset-dependent.","Strategy families such as uncertainty-based, diversity-based, and combined showed no consistent performance grouping, so recommendations should target concrete implementations and settings rather than family labels."],"supporting_citations":[{"why":"Defines the pool-based active-learning cycle and the notions of learner, query strategy, and batch that the experiment grid formalizes.","marker":"[1]"},{"why":"A large prior survey comparing many AL strategies whose inconclusive strategy rankings motivate the paper's search for a hyperparameter explanation.","marker":"[15]"},{"why":"An early study concluding that AL experiments have many influential hyperparameters and that random sampling is hard to beat, directly motivating the grid approach.","marker":"[19]"},{"why":"Frames the evaluation challenges of AL and supplies the dynamic ramp-up versus plateau aggregation idea that the authors adapt for their metrics.","marker":"[20]"},{"why":"Re-ran a published AL benchmark and found large discrepancies between implementations, serving as direct prior evidence for implementation sensitivity.","marker":"[24]"},{"why":"Used for arithmetic-mean aggregation of learning curves and for the runtime-limit values that determine which grid combinations are feasible.","marker":"[37]"},{"why":"One of the open-source libraries whose implementations of several query strategies are treated as distinct hyperparameter values.","marker":"[48]"},{"why":"Supplies one of the two QUIRE implementations whose low mutual correlation anchors the implementation-beats-strategy claim.","marker":"[49]"},{"why":"Supplies the second QUIRE implementation and several other strategy variants that enlarge the same-strategy comparison set.","marker":"[51]"},{"why":"Provides additional strategy implementations, including uncertainty variants and coreset methods, used in the cross-framework correlation analysis.","marker":"[52]"}],"fun_headline_variants":["Active learning results hinge more on code than strategy","AL strategy rankings flip depending on the codebase","Four million experiments show AL implementation dominates strategy","AL hyperparameter grid: implementation trumps strategy choice"],"cache_read_input_tokens":78336,"weakest_assumption_plain":"The grid built from 92 tabular datasets, three classical learners, and 28 strategy implementations is assumed to be representative enough that the observed influence patterns and the roughly 4,000-combination sufficiency threshold generalize to active learning as a whole, including deep learning and non-tabular data, even though about 75,924 slow combinations were excluded and imputed as zero.","fun_headline_variants_meta":{"raw":{"variants":["Active learning results hinge more on code than strategy","AL strategy rankings flip depending on the codebase","Four million experiments show AL implementation dominates strategy","AL hyperparameter grid: implementation trumps strategy choice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1532,"prompt_tokens":1012,"completion_tokens":520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":461}},"tokens_in":628,"tokens_out":520,"duration_ms":5285,"temperature":1.0,"reasoning_tokens":461,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:53:52.473513+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two implementations of the same strategy, for example the two QUIRE variants, and run them on a fresh collection of, say, 60 datasets from other domains including images and text; if their metric correlation is consistently above 0.8 rather than the reported 44.5%, the implementation-dominance claim would not generalize, and if the 4,000-combination stability threshold fails on those domains, the recommended study design would need revision.","supporting_citations":[{"cited_title":"A comparative survey: Benchmarking for pool-based active learning,","cited_arxiv_id":null,"evidence_quote":"A large prior survey comparing many AL strategies whose inconclusive strategy rankings motivate the paper's search for a hyperparameter explanation."},{"cited_title":"When Does Active Learning Work?","cited_arxiv_id":null,"evidence_quote":"An early study concluding that AL experiments have many influential hyperparameters and that random sampling is hard to beat, directly motivating the grid approach."},{"cited_title":"Challenges of Reliable, Realistic and Comparable Active Learning Evaluation","cited_arxiv_id":null,"evidence_quote":"Frames the evaluation challenges of AL and supplies the dynamic ramp-up versus plateau aggregation idea that the authors adapt for their metrics."},{"cited_title":"Re-Benchmarking Pool-Based Active Learning for Binary Classification,","cited_arxiv_id":null,"evidence_quote":"Re-ran a published AL benchmark and found large discrepancies between implementations, serving as direct prior evidence for implementation sensitivity."},{"cited_title":"ImitAL: Learned Active Learning Strategy on Synthetic Data,","cited_arxiv_id":null,"evidence_quote":"Used for arithmetic-mean aggregation of learning curves and for the runtime-limit values that determine which grid combinations are feasible."},{"cited_title":"ALiPy: Active Learning in Python,","cited_arxiv_id":null,"evidence_quote":"One of the open-source libraries whose implementations of several query strategies are treated as distinct hyperparameter values."},{"cited_title":"Libact: Pool-based Active Learning in Python,","cited_arxiv_id":null,"evidence_quote":"Supplies one of the two QUIRE implementations whose low mutual correlation anchors the implementation-beats-strategy claim."},{"cited_title":"Scikit-activeml: A Library and Toolbox for Active Learning Algorithms,","cited_arxiv_id":null,"evidence_quote":"Supplies the second QUIRE implementation and several other strategy variants that enlarge the same-strategy comparison set."},{"cited_title":"Small-text: Active learning for text classification in python,","cited_arxiv_id":null,"evidence_quote":"Provides additional strategy implementations, including uncertainty variants and coreset methods, used in the cross-framework correlation analysis."}],"review_version":1}