{"id":"f35aa1be-3c99-4668-8e17-b2f8065d123e","arxiv_id":"2507.21482","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Weighted Task Diversity allocates the annotation budget across tasks in inverse proportion to the base model's average confidence, improving MMLU and AlpacaEval scores with up to 80% fewer labels.","lead":"This paper proposes a simple data-selection trick for fine-tuning large language models: spend the annotation budget across tasks in proportion to how unsure the base model is about each task, while keeping at least a few examples from every task. On two instruction-tuning datasets, models trained on the selected subset matched or beat models trained on the full dataset, cutting annotation costs by up to 80%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inverse-confidence allocation conflates uncertainty with output length: confidence is the product of token probabilities, so long-generation tasks dominate selection and the claimed uncertainty mechanism is untested.","rationale":"The paper's central empirical claim is that prioritizing low-confidence tasks improves label-efficient SFT, and that claim depends on conf(x) being a meaningful uncertainty measure. The product definition makes this false by construction for variable-length generations: two tasks with identical per-token confidence can differ by orders of magnitude in conf solely because of response length. The allocation patterns in Figures 4 and 6 match what a pure output-length rule would produce, so the experiments do not discriminate between 'uncertain tasks' and 'long-output tasks.' This matters for the strongest claim: the 4% MMLU advantage comes from the Dolly k=3K row, where the selected set is dominated by open QA and brainstorming. It is possible that oversampling long instructive answers genuinely helps downstream instruction-following or even MMLU, but that would be a different mechanism than the one the paper claims, and it would require a different justification. The reader identified the same assumption, and the paper's own Limitations section concedes that confidence may reflect pretraining exposure, but the length confound is more specific and directly testable by a length-normalization ablation. Secondary issues reinforce caution: the abstract's 'consistently at or above' claim is not supported by the paper's own BBH results (e.g., Table 1 at k=20K, DPP 40.68 vs WTD 39.96; at k=45K, FL(0.002) 41.68 vs WTD 40.86), and AlpacaEval figures lack error bars. These issues do not overturn the possible empirical value of the method, but they do mean the mechanism and the strong claims should not be accepted without the length-normalization check. Since the reader already assigned CONDITIONAL for essentially this assumption, I do not move the verdict.","tokens_in":14249,"tokens_out":4066,"duration_ms":50344,"concrete_test":"Recompute task-level scores using a length-normalized confidence, e.g., geometric mean token probability exp(mean_j log p_j) or average per-token negative log-likelihood, keeping all other parts of the pipeline fixed. Then rerun the Dolly k=3K and k=6K and FLAN k=30K experiments. If length-normalized Weighted Task Diversity no longer outperforms Task Diversity or Random on MMLU and AlpacaEval, the reported gains are attributable to upweighting long-output tasks rather than to confidence-calibrated uncertainty. As a secondary check, report the correlation between task-level mean confidence and mean generated response length for the base model; a high negative correlation (e.g., |r| > 0.9) would confirm that the current allocation is effectively a length-based rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 defines conf(x) as the product of token probabilities of the model's generated response. For any response with average per-token probability p < 1, this quantity equals p^m and decays exponentially in sequence length m. Task-level averages are therefore driven by how long the base model's generations are, not by calibrated uncertainty or task difficulty. This is not a hypothetical confound: the Dolly allocation in Figure 4 is ordered almost exactly by output length, with open QA receiving 2,288 samples, brainstorming 1,400, and general QA 1,275, while classification, closed QA, information extraction, and summarization receive 196, 75, 86, and 60. The FLAN allocation in Figure 6 shows the same pattern, with long-generation tasks such as math/algebra and QQP/MNLI receiving the largest shares. The abstract's headline result (MMLU 39.74 vs 35.33 in Table 2) is obtained with this allocation, so the experiments do not distinguish 'prioritize uncertain tasks' from 'prioritize long-output tasks.' The Limitations paragraph acknowledges that confidence may reflect pretraining exposure, but it does not address the length dependence that is built into the product definition, and no ablation with a length-normalized confidence is reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies one-batch label-efficient supervised finetuning: given an unlabeled prompt pool with task labels, select k prompts to annotate. It proposes Task Diversity (round-robin equal allocation across tasks) and Weighted Task Diversity (allocation proportional to the inverse of the base model's average task-level confidence, after a base allocation of five examples per task). Confidence for a prompt is the product of token probabilities of the base model's generated response. Experiments on LLaMA-2 7B use a 90K FLAN V2 subset and Dolly, with budgets 20K/30K/45K/90K and 3K/6K/13.5K, evaluating MMLU, BBH, and AlpacaEval. The main claims are that Weighted Task Diversity beats full-data training by about 4% MMLU on Dolly at 3K labels and that it generally matches or exceeds more complex baselines while saving up to 80% of labels.","tokens_in":14495,"tokens_out":7969,"duration_ms":86176,"significance":"If the empirical results hold, the contribution is practically useful: a simple, transparent task-level allocation rule using readily available task labels and one forward pass over the pool, with no embedding optimization or per-example score tuning, can produce strong SFT models under small annotation budgets. The paper reports means and standard errors over three seeds, includes multiple diversity and uncertainty baselines, and provides length-controlled AlpacaEval win rates, which are strengths. The main caveat is that the uncertainty mechanism underlying the method is not identified: the confidence score conflates output length with uncertainty, so the headline gains may be explained by a length bias rather than by a calibrated notion of task difficulty. With an added length-normalized ablation and a qualified abstract, the practical claim would be credible.","major_comments":[{"comment":"The definition of conf(x) as the product of token probabilities makes the measure exponentially sensitive to output length: for a response of length m with geometric mean per-token probability p, conf(x) = p^m. Task-level averages therefore rank tasks primarily by how long the base model's generations are, rather than by calibrated uncertainty or task difficulty. This is visible in the allocations: in Figure 4, Dolly's long-generation tasks receive 2288 (open QA), 1400 (brainstorming), and 1275 (general QA) samples, while classification, closed QA, information extraction, and summarization receive 196, 75, 86, and 60; Figure 6 shows the same pattern for FLAN. Consequently, Table 2's headline result (Weighted Task Diversity at k=3K, MMLU 39.74 vs. full-pool 35.33) does not separate 'prioritize low-confidence tasks' from 'prioritize long-output tasks.' The Limitations paragraph acknowledges that confidence may reflect pretraining exposure, but it does not address the length dependence that is built into the product definition, and no length-normalized confidence ablation is reported. Please report results with a per-token average log-probability or another length-normalized uncertainty score, or otherwise explicitly control for output length.","section":"Section 4.3"},{"comment":"The abstract's claim that the algorithm 'consistently performs at or above the level of the best existing methods' is contradicted by the BBH columns of Table 1. At k=20K, Weighted Task Diversity scores 39.96±0.52, below Min Margin (40.44±0.48) and DPP (40.68±0.50); at k=45K, FL(γ=0.002) reaches 41.68±0.34 and Task Diversity 41.10±0.53, both above Weighted Task Diversity's 40.86±0.15. Section 5.2 itself concedes that on BBH 'no single method consistently dominates.' The abstract should be qualified to the datasets and metrics where the claim actually holds (e.g., MMLU and AlpacaEval, or 'on most budgets'), rather than stated as a universal property.","section":"Abstract / Section 5.2"}],"minor_comments":[{"comment":"The text describes Dolly as a 15K dataset but the experiments use a 13.5K pool; please explain how the 13.5K subset was constructed and why it differs from the full dataset.","section":"Section 4.1 / Tables 2-3"},{"comment":"The clamping notation in the allocation formula is typeset ambiguously, with the bounds appearing as |X_t| and 5 without a clear order. Please define [·]_a^b explicitly and state which bound is lower and which is upper.","section":"Section 4.3"},{"comment":"The bar labels 'T ask' and 'WTD T ask Div' contain spacing artifacts; please correct these to 'Task' and 'WTD Task Div.'","section":"Figures 3 and 5"},{"comment":"The sentence about the 45K budget 'saving 50% annotation budget when compared to random sampling' is imprecise; the saving is relative to the 90K full-data model, not to random sampling per se.","section":"Section 5.2"},{"comment":"The paper does not mention a code or data-release plan; releasing the selected prompt indices and the finetuning code would substantially aid reproducibility and comparison.","section":"Reproducibility"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for a machine learning or NLP venue, and the self-citations to Bhatt et al. (2024) are natural given the shared framework. The main risk is the output-length confound in the confidence measure; if the authors can show that a length-normalized confidence score produces similar allocations and gains, the paper would be much stronger. The BBH results also need to be reconciled with the abstract's consistency claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2507.21482. First, it is a straightforward and practical paper: allocate a small base budget to every task, then spend the rest across tasks in proportion to the inverse of the base model's average confidence. Second, the headline MMLU result at a 3K budget on Dolly (39.74 vs. 35.33 for full data) is genuinely surprising and worth taking seriously. If it reproduces, the method saves 80% of labels with a better score than full-data SFT. That alone justifies a careful look.\n\nWhat is new: prior task-level active learning (Kung et al.) spends the whole budget on the hardest tasks. Adding a per-task floor and inverse-confidence proportional allocation is a legitimate, simple twist. The paper also compares many baselines, reports computational costs, and is honest about some limitations, including that confidence may reflect pretraining exposure.\n\nThe soft spots are real. The confidence score is the product of token probabilities, so it decays exponentially in generation length. The allocation plots confirm the worry: open QA, brainstorming, and general QA get thousands of samples while classification, closed QA, and summarization get tens. That could just as easily be 'select long outputs' as 'select uncertain tasks.' The reader's stress test is correct, and the paper does not run the obvious control, e.g., length-normalized confidence or per-token average. The abstract claim of 'consistently at or above the best existing methods' is also contradicted by the BBH column in Table 1, where Weighted Task Diversity is not consistently best. And the Dolly BBH results are downweighted post hoc, which weakens the narrative. No code is released, and AlpacaEval has no error bars.\n\nNone of this kills the paper. The method is simple, the MMLU effect is large, and even a 'length-driven' interpretation might still be practically useful—but then the story needs to change. The right fix is a length-normalized ablation and a more measured abstract. I would send this to a serious referee, not desk reject it; it needs a major revision, not dismissal. If you work on data selection or label-efficient SFT, bring it to a reading group: the bug is instructive and the empirical result is provocative.","headline":"A simple task-level inverse-confidence selection method with a promising MMLU result, but the confidence metric is confounded with output length and the abstract overclaims consistency.","tokens_in":15024,"tokens_out":1767,"would_cite":true,"duration_ms":22616,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a simple inverse-confidence task allocation can beat full-dataset LLM finetuning with up to 80% fewer labels.","keywords":["label-efficient supervised finetuning","task diversity","inverse confidence weighting","round-robin sampling","instruction tuning","data selection","one-batch active learning","uncertainty quantification"],"falsifier":"Run Weighted Task Diversity with confidence normalized by response length or computed as a per-token geometric mean, holding the budget, pool, and training recipe fixed; if the 3K-label MMLU advantage over full-data training disappears, the reported gain rests on the length confound rather than on task difficulty.","tokens_in":14054,"feed_emoji":"🎯","tokens_out":8970,"duration_ms":94216,"temperature":0.7,"pith_summary":"The paper tries to establish that task-level diversity, rather than prompt-level diversity in an embedding space, is the right organizing principle for label-efficient supervised finetuning of LLMs. Its Weighted Task Diversity rule gives every task a small base budget and then spends the remaining annotation budget on tasks in proportion to the inverse of the base model's average confidence, so low-confidence tasks receive more labels. On the Dolly dataset, models finetuned on 3K labels selected this way score 39.74 on MMLU versus 35.33 for models trained on the full 13.5K pool, a gain of more than 4 points with roughly 80% fewer labels. On FLAN V2, the same strategy matches full-data training at half the budget and achieves the highest AlpacaEval win rates. If correct, this means a cheap, reusable confidence signal can replace elaborate embedding-based and uncertainty-based selectors.","feed_headline":"80% fewer labels, 4% better MMLU via task-aware sampling","feed_subtitle":"Annotate more from tasks the base model is least sure about; this beats complex selectors and full-data training.","key_machinery":"The load-bearing object is the task-level inverse-confidence allocation $\\alpha_t \\propto 1/\\mathrm{conf}_t$, realized through a round-robin sampler. Task confidence is defined as $\\mathrm{conf}_t = \\frac{1}{|X_t|}\\sum_{x \\in X_t} \\prod_{j=1}^m g(y_j \\mid y_{<j}, x)$, the average over a task's prompts of the product of token probabilities of the base model's generated response. The allocation first gives every task a small base budget, then distributes the remainder in proportion to $1/\\mathrm{conf}_t$, clamped to each task's available prompts. The round-robin pass converts fractional allocations into an integer selection and enforces that small tasks are covered before the budget moves on. This machinery does two jobs at once: it guarantees task coverage, and it shifts labels toward tasks where the base model is most uncertain.","core_discovery":"The central claim is that a pretrained model's own confidence, averaged per task, is a usable signal for deciding which instruction-tuning prompts to pay humans to label. Under Weighted Task Diversity, each task first receives a small coverage floor (the paper uses 5 examples), and the remaining budget is split across tasks proportionally to $1/\\mathrm{conf}_t$, where $\\mathrm{conf}_t$ is the mean over prompts in task $t$ of the product of token probabilities of the base model's generated response; a round-robin procedure then selects examples uniformly within each task. The paper reports that on Dolly, this selection at a 3K budget reaches 39.74 MMLU versus 35.33 for training on the full 13.5K pool, that on FLAN V2 a 45K budget reaches the same MMLU as all 90K examples, and that the selected subsets win more often than full-data-trained models in GPT-4-judged AlpacaEval comparisons. These results are presented as evidence that task diversity plus uncertainty weighting is at least as powerful as, and much simpler than, embedding-based diversity methods.","pith_inferences":["Inference: an unnormalized product-of-token-probabilities confidence conflates task difficulty with response length and decoding choices; a length-normalized or per-token-geometric-mean version is a natural ablation the paper does not run.","Inference: the same allocation rule could be recycled across active-learning rounds, recomputing task confidence after each finetune; the paper studies only the one-batch setting.","Inference: the over-4-point gain over full-data training on Dolly might be partly a curation effect, since the 3K subset drops many redundant or lower-quality examples; a deduplicated pool would separate allocation value from data cleaning."],"forward_implications":["On the Dolly pool, a 3K-label Weighted Task Diversity selection produces an MMLU score of 39.74, higher than the 35.33 from training on all 13.5K labels, an annotation saving of roughly 80% with an accuracy gain.","On FLAN V2, the same strategy reaches full-data MMLU performance at a 45K budget, cutting the annotation load in half at matched accuracy.","Weighted Task Diversity and plain Task Diversity match or beat all tested baselines—random, entropy, confidence, margin, k-center, facility location, DPP, and ActiveIT—on AlpacaEval win rates at matched budgets.","The method needs no precomputed embeddings, no clustering, no ground-truth-based quality scores, and no careful kernel hyperparameters; task labels already present in instruction datasets suffice.","Because allocation happens before any annotation is collected, the rule is immediately usable in the one-batch active learning setting where ground-truth responses are unknown."],"supporting_citations":[{"why":"Supplies the label-efficient SFT problem setup, the confidence/entropy/margin scoring baselines, and the experimental-design framing the paper adopts.","marker":"Bhatt et al. (2024)"},{"why":"Provides ActiveIT, the task-level uncertainty baseline that spends its whole budget on the hardest tasks and that Weighted Task Diversity modifies with a coverage floor.","marker":"Kung et al. (2023)"},{"why":"Provides the FLAN V2 instruction-tuning collection whose task categories define the allocation units for the main experiments.","marker":"Longpre et al. (2023)"},{"why":"Provides the Dolly 15K dataset and its eight task categories, the source of the headline 3K-versus-13.5K MMLU comparison.","marker":"Conover et al. (2023)"},{"why":"Supplies LLaMA-2 7B, the base model whose confidence scores drive selection and which is finetuned in all experiments.","marker":"Touvron et al. (2023)"},{"why":"Supplies the DPP/deterministic D-optimal design baseline and the notion of incorporating quality scores into subset selection, which the paper replaces with confidence scores.","marker":"Wang et al. (2024)"},{"why":"Defines the MMLU benchmark used for the main accuracy comparisons.","marker":"Hendrycks et al. (2020)"},{"why":"Provides the AlpacaEval framework with GPT-4 Turbo as judge used for preference comparisons against full-data-trained models.","marker":"Li et al. (2023)"},{"why":"Supplies the uncertainty-sampling literature and the sequence-probability product used to define example-level confidence.","marker":"Settles (2009)"}],"fun_headline_variants":["Task confidence sampling cuts labels 80%, boosts MMLU 4%","Fewer labels, better LLM: pick tasks the model is unsure of","Uncertainty-based task selection: 80% less annotation, +4% MMLU","Label less, win more: sample tasks by model confidence","Simple inverse confidence sampling beats full-data finetuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method rests on treating the base model's average per-task confidence as a trustworthy signal of which tasks most need labels; if confidence mostly reflects pretraining exposure, output length, or decoding style rather than labeling value, the inverse-confidence allocation points at the wrong tasks.","fun_headline_variants_meta":{"raw":{"variants":["Task confidence sampling cuts labels 80%, boosts MMLU 4%","Fewer labels, better LLM: pick tasks the model is unsure of","Uncertainty-based task selection: 80% less annotation, +4% MMLU","Label less, win more: sample tasks by model confidence","Simple inverse confidence sampling beats full-data finetuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000856,"raw_usage":{"total_tokens":3750,"prompt_tokens":1007,"completion_tokens":2743,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":2657}},"tokens_in":623,"tokens_out":2743,"duration_ms":22109,"temperature":1.0,"reasoning_tokens":2657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:42:14.782561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Weighted Task Diversity with confidence normalized by response length or computed as a per-token geometric mean, holding the budget, pool, and training recipe fixed; if the 3K-label MMLU advantage over full-data training disappears, the reported gain rests on the length confound rather than on task difficulty.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Dolly 15K dataset and its eight task categories, the source of the headline 3K-versus-13.5K MMLU comparison."}],"review_version":1}