{"id":"335c21ae-cd75-49dc-9963-9bead4bf78f8","arxiv_id":"2506.01992","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"No single active learning query strategy wins across all frozen LLM embeddings and text tasks; strategy rankings depend on embedding quality, task, and initial pool selection.","lead":"This study tests how the quality of frozen LLM embeddings changes which active learning query strategy works best on text classification. It finds no universally best strategy: the winner depends on the embedding model and the task, with diversity-based initial selection helping early when embeddings are strong.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The title's causal claim that embedding quality dictates query-strategy success is inferred from MTEB rank, a proxy the authors themselves qualify, and Section 4.3 contains an unresolved IPS-protocol inconsistency (Figure 6 caption vs. text); the strong version of the claim is not yet established.","rationale":"The benchmark is genuinely useful: it releases an extensible framework, uses five seeds with paired analyses, and its core observation that query-strategy rankings vary across datasets and models is directly visible in the reported win-rate matrices and learning curves. That part of the 'no free lunch' message does not depend on MTEB. However, the title and abstract make a stronger causal claim: that embedding quality dictates strategy success. The evidence for that claim is correlational across seven models, with MTEB score as the only quality measure, and the paper itself flags MTEB's limited scope. The Figure 6 caption/text mismatch compounds the problem because it leaves unclear whether the quality-attribution results were produced with the IPS protocol the text says was used. These issues do not amount to fraud or to a fatal flaw; they mean the paper should be read as a conditional benchmark contribution whose headline causal interpretation needs a direct test. My proposed degradation experiment would settle whether quality per se moves strategy rankings. Consequently, I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":16700,"tokens_out":6846,"duration_ms":76763,"concrete_test":"Use a single high-performing embedding model (e.g., NV-Embed-v2) and create controlled quality degradations—PCA truncation of the frozen embeddings to 128, 512, and 1024 dimensions, plus an isotropic-noise condition at full dimension—while keeping the model, pooling, classifier, datasets, budgets, and seeds fixed. Run the full random-IPS and TypiClust-IPS AL loops under each degradation and test, e.g., by paired sign tests over seeds, whether the ranking of the seven query strategies changes monotonically as quality decreases. If the optimal-strategy ranking is invariant across degradation levels, the 'quality dictates strategy success' claim is unsupported; if it changes systematically, the MTEB-rank confound is secondary and the Figure 6 caption issue can be treated as typographical.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3, which carries the paper's main quality-attribution argument, states: 'We conduct all experiments with TypiClust as our best-performing IPS strategy.' Yet Figure 6, the figure cited for the 'higher-quality embeddings ... lead to higher overall accuracy' finding, is captioned 'with random IPS.' This is not cosmetic: if the quality-attribution plots were produced with random IPS, the claimed synergy between TypiClust and embedding quality is not cleanly evidenced; if the caption is wrong, the reporting is unreliable at exactly the point where the causal claim is made. Independently of that, 'embedding quality' is never measured directly. It is identified with MTEB leaderboard rank/score (Table 3), and the authors' Limitations concede MTEB is 'primarily a dynamic retrieval-focused benchmark' that may not generalize to AL. With only seven models, the observed variation in strategy rankings across models (Figures 5 and 7) is compatible with confounds such as model family, parameter count (110M to 7.8B), embedding dimension (768 to 4096), or pooling strategy, none of which is 'quality' in the AL-relevant sense. The paper convincingly shows that no single strategy wins everywhere; it does not yet show that embedding quality, as opposed to model identity or task, dictates which strategy wins.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a benchmark study of active learning (AL) on text classification using frozen LLM embeddings. The authors evaluate seven embedding models (five MTEB top-ranked LLMs and two baselines) on ten tasks from the ActiveGLAE benchmark, with seven query strategies and five seeds. They investigate two stages: informed initial pool selection (IPS) via diversity-based methods, and the full AL cycle. The main claims are that (1) TypiClust-based IPS provides early advantages with high-quality embeddings, (2) no single query strategy is universally superior, and (3) the optimal strategy depends on both the embedding model and the task, with Margin performing well on simpler tasks while Badge and Entropy are more robust across settings, and Badge benefits from higher-quality embeddings. The paper also releases an extensible framework built on scikit-activeml.","tokens_in":16912,"tokens_out":4305,"duration_ms":43909,"significance":"If the claims are established, this would be a useful reference benchmark for practitioners using frozen LLM embeddings in AL, and it would strengthen the already-known message that AL strategy choice is context-dependent. The study's strengths include a reasonably broad evaluation (10 datasets, 7 models, 5 seeds), paired seed-level analyses, clear description of the logistic-regression training protocol, and the release of code. The negative result for CoreSet as an IPS strategy and the demonstration that no universal winner exists are valuable empirical findings. However, the central causal claim that \"embedding quality dictates query strategy success\" rests on an internal inconsistency in the IPS protocol and on an operationalization of quality (MTEB rank) that is confounded with model architecture and scale.","major_comments":[{"comment":"The text in §4.3 states \"We conduct all experiments with TypiClust as our best-performing IPS strategy,\" but Figure 6, which is cited for the finding that higher-quality embeddings lead to higher overall accuracy, is captioned \"with random IPS.\" This is a load-bearing inconsistency: the quality-attribution analysis and the claimed synergy between TypiClust embeddings and AL performance depend on which IPS was used. Please clarify which IPS was used for Figures 5–7 and the associated text. If Figure 6 was produced with random IPS, then it does not by itself support the TypiClust-specific synergy claim; if the caption is wrong, the reporting is unreliable at exactly the point where the paper's central claim is made.","section":"Section 4.3, Figure 6"},{"comment":"The paper equates \"embedding quality\" with MTEB leaderboard rank (Table 3) and bases its central claim on this operationalization, yet the Limitations concede that MTEB is \"primarily a dynamic retrieval-focused benchmark\" that may not generalize to AL. With only seven models, differences in strategy performance are confounded with parameter count (110M–7.8B), embedding dimension (768–4096), pooling strategy, and model family (BERT-derived vs. LLM-based). The observed variation in strategy rankings across models is therefore not uniquely attributable to \"quality\" in the AL-relevant sense. To support the causal wording in the title and abstract, the authors would need controlled comparisons (e.g., checkpoints of the same architecture with different MTEB scores, or analyses controlling for model size) or should soften the claims from \"quality dictates\" to \"the embedding model, along with its MTEB score, is associated with strategy performance.\"","section":"Section 4.3, Table 3, Limitations"},{"comment":"The total labeling budget B is \"empirically determined\" per dataset by observing the convergence point of the BERT embedding with random sampling (Table 2). This exogenous choice may compress the AL curves of higher-quality embeddings, especially in later cycles, making differences among query strategies harder to detect for those models and potentially biasing the strategy rankings reported in Figures 5 and 7. Please justify this budget-selection rule with respect to fairness across embedding models, or include a sensitivity analysis with alternative budgets (e.g., a fixed budget across models or a budget scaled to each model's convergence).","section":"Section 3 (AL cycle budget)"}],"minor_comments":[{"comment":"There are inconsistent model-name spellings: footnote 2 refers to \"GWEN 2.5\" and Table 4 to \"QWEN 2.7-7B,\" while the main text and Table 3 use \"QWEN2.5-7B\"; please standardize the spelling.","section":"Footnote 2 and Table 4"},{"comment":"The caption refers to \"Banks77\" while the dataset is named \"Banking77\" in Table 2; please use one consistent name.","section":"Figure 2 caption"},{"comment":"Some runtime standard deviations are implausibly large relative to the reported means (e.g., TypiClust on QNLI: 01:37 ± 06:44; TypiClust on Yelp5: 03:50 ± 36:56); please verify the formatting or the values, as these look like unit or rounding errors.","section":"Table 5"},{"comment":"The phrase \"differences exceeding 0.05-0.10 %\" should likely read \"0.05–0.10 percentage points\" (or an absolute accuracy difference ), since the figure reports accuracy differences, not relative percentages.","section":"Section 4.2"},{"comment":"The phrase \"Figure 6 illustrates this at the example of the Banks77 and MNLI datasets\" should be reworded to \"using the example of\" or \"on the Banks77 and MNLI datasets.\"","section":"Section 4.3, first paragraph"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and the released framework are valuable, and the 'no universal winner' message is well supported. The main risks are the IPS-protocol inconsistency in §4.3 and the conflation of MTEB score with AL-relevant embedding quality; both need to be addressed before publication. The self-citation pattern (ActiveGLAE, scikit-activeml) is largely infrastructural and seems appropriate for this kind of benchmark paper. The manuscript is within scope for cs.CL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is the first text-domain benchmark I know that systematically varies frozen LLM embeddings across IPS and query strategies, and it is worth a serious referee. The core qualitative finding—no single strategy is universally best, and the winner depends on the embedding model and the task—is well supported by the data. TypiClust's early advantage with high-quality embeddings on multi-class tasks, and Badge's robustness, are genuinely new reference points for frozen-feature AL. The experimental design is careful: ten ActiveGLAE datasets, seven models, five seeds, paired seed-level comparisons, and logistic-regression probing that isolates the embedding contribution from fine-tuning dynamics. Releasing the framework is a real plus.\n\nThe soft spots are real but mostly not fatal. First, the Figure 6 caption says random IPS while Section 4.3 explicitly says all experiments use TypiClust. That is exactly the figure carrying the quality-attribution argument, so this is not cosmetic; it has to be corrected and the analyses rerun or clearly re-described. Second, 'embedding quality' is never measured directly. It is equated with MTEB rank, and the seven models vary in size, dimension, pooling, and family all at once. The authors themselves concede in the Limitations that MTEB is retrieval-focused. So the strong version of the title—quality dictates success—is an interpretation, not an established causal claim. The weaker, still useful, claim that strategy choice interacts with model and task is solid. Third, win-rate aggregates in Figures 4, 5, and 7 come without confidence intervals, and some appendix runtime standard deviations are implausible (QNLI TypiClust 01:37 ± 06:44 implies negative runtimes). Minor, but it makes one wonder about the rest of the appendix.\n\nThe paper deserves peer review. It is a solid empirical map for AL researchers and practitioners who use frozen embeddings. With the caption fixed, the MTEB proxy caveat made prominent in the conclusions, and some error-bar sanity checks, it would be a reliable reference.\n\nRecommendation: send it out, but expect substantial revision.","headline":"A genuinely useful frozen-embedding AL benchmark whose main qualitative claims hold, but the title's causal 'quality dictates' framing outruns the MTEB-based evidence and one key figure caption is inconsistent with the protocol.","tokens_in":17517,"tokens_out":1779,"would_cite":true,"duration_ms":17620,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No single active-learning query strategy wins across embeddings","keywords":["active learning","LLM embeddings","query strategies","initial pool selection","MTEB","text classification","frozen embeddings","no free lunch"],"falsifier":"Take the same ten datasets and query strategies, but order the embedding models by a different quality measure—for example, linear-probe accuracy on an unrelated held-out task or a cluster-separability score—and check whether the strategy rankings still sort by quality. A single low-MTEB model that matches high-MTEB models in Badge's robustness, or a high-MTEB model that shows Margin-style task spikes, would break the claimed link.","tokens_in":16431,"feed_emoji":"🎯","tokens_out":5313,"duration_ms":46144,"temperature":0.7,"pith_summary":"This paper argues that in deep active learning with frozen LLM embeddings, the best query strategy is not universal: it depends on the quality of the embedding model and on the task. The authors benchmark seven embedding models—five top-ranked on the MTEB leaderboard plus BERT and ModernBERT—across ten text classification tasks and eight query strategies over twenty AL cycles. They find that a diversity-based initial pool (TypiClust) gives a real early-iteration head start when combined with high-quality embeddings, that Margin sampling produces strong spikes on easier tasks, and that Entropy and Badge are the most robust across tasks and models. If these results hold, active-learning practitioners should treat strategy choice as context-dependent and re-evaluate it whenever they change the embedding backbone.","feed_headline":"No single active-learning strategy wins across embeddings","feed_subtitle":"On ten tasks and seven embedding models, Margin, Badge, and Entropy each take the lead depending on context.","key_machinery":"The machinery is a frozen-embedding active-learning loop: a pre-trained LLM maps each text to a single vector once; a logistic-regression head is retrained from scratch on the growing labeled pool each cycle; a query strategy picks the next batch; and accuracy over 20 cycles is the score. This isolates the embedding's contribution from fine-tuning dynamics. The named machinery includes the query strategies themselves—uncertainty methods (Margin, Entropy), diversity methods (CoreSet, ProbCover, TypiClust), and hybrids (Badge, DropQuery)—plus the initial-pool-selection choice (random vs. TypiClust). MTEB leaderboard rank is used as the operational measure of embedding quality, so the benchmark's comparisons are organized around that ranking.","core_discovery":"The central claim is that the effectiveness of an active-learning query strategy is conditioned on the representation it sits on. On the paper's benchmark, no strategy is universally superior: Margin wins most often on datasets with simple, separable classes (AGNews, DBPedia, TREC-6), while Badge and Entropy win on complex multi-class tasks such as Banks77, Yelp-5, and MNLI. The paper also claims that embedding quality modulates initialization: TypiClust, which selects diverse and representative points, outperforms random initial pools mainly when the pool is small (under roughly 300 instances) and the embeddings are high-quality, especially on fine-grained tasks. Higher MTEB-ranked embeddings generally yield faster convergence and larger TypiClust advantages inside the AL loop. The conclusion is that AL strategy choice is a contextual decision, and that a static leaderboard like MTEB is a helpful but incomplete predictor of AL usefulness.","pith_inferences":["As embedding models continue to improve, the balance among strategies may shift further toward hybrid methods like Badge, which exploit richer representations, and away from pure diversity sampling.","A practical rule of thumb can be tested: compute a cheap proxy for embedding structure (cluster purity or per-class separability) and use it to pre-select between Margin, Entropy, and Badge before running any AL loop.","The frozen-embedding setting is likely to generalize to other modalities that use fixed pretrained features, so similar quality-dependent strategy rankings may appear in vision or audio active learning, but this would need a separate benchmark to confirm.","If MTEB rank is a valid proxy, then one can predict relative AL gains without running AL by comparing embedding models on retrieval and classification benchmarks—this is a testable shortcut the paper does not itself evaluate."],"forward_implications":["On a small labeling budget, starting with a diversity-based pool like TypiClust is likely to pay off when the embedding model is strong; with weaker embeddings or larger initial pools, random selection gets competitive.","Margin sampling is a cheap, effective first choice on simple classification tasks, but for complex multi-class tasks Badge or Entropy should be preferred.","An active-learning evaluation that reports only a single aggregated winner is misleading; results should be broken down by embedding model and task.","Higher-quality embeddings do not just lift final accuracy—they make the AL process converge faster, so their added compute cost can be offset by fewer labeling rounds.","Static embedding benchmarks such as MTEB are useful for ranking models but should not be treated as a substitute for task-specific AL evaluation."],"supporting_citations":[{"why":"Supplies the MTEB leaderboard and quality scores that order the embedding models used throughout the benchmark.","marker":"[21]"},{"why":"Supplies the ten datasets, labeling budgets, and evaluation protocol that the benchmark is built on.","marker":"[28]"},{"why":"Introduces the frozen-feature AL paradigm and the DropQuery strategy that this work adapts to NLP.","marker":"[8]"},{"why":"Introduces TypiClust and the budget-dependent strategy argument that motivates the initial-pool and strategy analyses.","marker":"[9]"},{"why":"Defines the CoreSet diversity-based strategy that serves as both a query strategy and an IPS baseline.","marker":"[32]"},{"why":"Defines the Badge hybrid strategy whose robustness across tasks is one of the paper's central findings.","marker":"[1]"},{"why":"Provides the uncertainty-sampling definitions underpinning Margin and Entropy strategies.","marker":"[33]"},{"why":"Defines the ProbCover covering-based strategy included among the compared methods.","marker":"[41]"}],"fun_headline_variants":["Query strategy success hinges on LLM embedding quality","No universal active-learning winner across embeddings","Embedding quality dictates which query strategy shines","Active learning: best strategy varies with embedding strength","Context-dependent query strategies: embedding quality matters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's central conclusion rests on treating MTEB leaderboard rank as a faithful measure of embedding quality for active learning; if MTEB scores do not track the representation properties that query strategies actually exploit, then differences in strategy success cannot be attributed to 'embedding quality'.","fun_headline_variants_meta":{"raw":{"variants":["Query strategy success hinges on LLM embedding quality","No universal active-learning winner across embeddings","Embedding quality dictates which query strategy shines","Active learning: best strategy varies with embedding strength","Context-dependent query strategies: embedding quality matters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000156,"raw_usage":{"total_tokens":1203,"prompt_tokens":915,"completion_tokens":288,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":221}},"tokens_in":531,"tokens_out":288,"duration_ms":3165,"temperature":1.0,"reasoning_tokens":221,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:35:43.608355+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same ten datasets and query strategies, but order the embedding models by a different quality measure—for example, linear-probe accuracy on an unrelated held-out task or a cluster-separability score—and check whether the strategy rankings still sort by quality. A single low-MTEB model that matches high-MTEB models in Badge's robustness, or a high-MTEB model that shows Margin-style task spikes, would break the claimed link.","supporting_citations":[{"cited_title":"MTEB: Massive text embedding benchmark","cited_arxiv_id":null,"evidence_quote":"Supplies the MTEB leaderboard and quality scores that order the embedding models used throughout the benchmark."},{"cited_title":"Activeglae: A benchmark for deep active learning with transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the ten datasets, labeling budgets, and evaluation protocol that the benchmark is built on."},{"cited_title":"Revisiting active learning in the era of vision foundation models","cited_arxiv_id":null,"evidence_quote":"Introduces the frozen-feature AL paradigm and the DropQuery strategy that this work adapts to NLP."},{"cited_title":"Active learning on a budget: Opposite strategies suit high and low budgets","cited_arxiv_id":null,"evidence_quote":"Introduces TypiClust and the budget-dependent strategy argument that motivates the initial-pool and strategy analyses."},{"cited_title":"Active learning literature survey","cited_arxiv_id":null,"evidence_quote":"Provides the uncertainty-sampling definitions underpinning Margin and Entropy strategies."},{"cited_title":"Active learning through a covering lens","cited_arxiv_id":null,"evidence_quote":"Defines the ProbCover covering-based strategy included among the compared methods."}],"review_version":1}