{"id":"4232964f-0662-4614-8561-9d1e369605a7","arxiv_id":"2505.14964","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper claims that curated 20 to 40 percent subsets of training data can match full-data models and that shared label errors can inflate validation scores.","lead":"The paper proposes smart-sizing, a method for training AI on a small, carefully chosen set of labels, and reports that 20 to 40 percent of a curated dataset can match or beat full-data training. It matters because high-stakes AI teams want cheaper annotation and reliable rare-event detection, but the experimental evidence here is too thinly described to verify.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 20%-vs-full-data result in §6.2 is not testable as reported: no dataset, model, selection algorithm, or leakage control is given, so the central claim rests on an uncontrolled protocol.","rationale":"The reader's verdict is REJECT with moderate confidence, and the weakest assumption identified is that the subset-selection procedure used only training-time information and did not leak validation labels. My stress-test pass reaches the same conclusion: the single most load-bearing condition for the paper's headline empirical claim is that the 20% subset was selected using only information that would be available in a real deployment of smart-sizing. That condition is not established anywhere in the manuscript. The text says the subsets were curated 'based on embedding space diversity and model disagreement' but gives no algorithm, no dataset names, no model details, no code, and no leakage controls. This is not a disagreement with consensus; it is an internal gap in the evidence chain. The paper could still be useful as a management framework, but as a scientific demonstration of the 20%-versus-full-data equivalence it is unreproducible. I also note that Section 6.4 claims the decision equation is 'validated by experimental evidence' even though that equation is a definitional threshold, and it cites workflow diagrams (Figures 1 and 2) as experimental demonstrations, which is an internal inconsistency in the argument. The proposed concrete test—a controlled public-benchmark replication with explicit separation of training and validation information—would settle whether the central claim is real or an artifact of leakage or a weak baseline. Since the reader already rejected on essentially these grounds, my verdict is UNCHANGED.","tokens_in":8781,"tokens_out":2219,"duration_ms":19763,"concrete_test":"Run a controlled replication on a public benchmark (e.g., CIFAR-10 or an ImageNet subset): (1) train a full-data baseline with a fixed architecture and compute budget; (2) compute embedding uniqueness and model disagreement using only the training set, with disagreement from a model trained on a small random seed subset, never touching validation labels; (3) select the top 20% by those scores; (4) train a model on that subset under the identical pipeline and budget; (5) repeat across five seeds and report mean and confidence interval. If the subset model is not within 5% of the full baseline, or if selection required validation labels, the §6.2 claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim (Section 6.2, Figures 5–6) is that training on the top 20% of samples ranked by 'embedding-based uniqueness' and 'model disagreement' achieves performance within 5% of full-data baselines. For this claim to be true, the selection procedure must use only training-time information. The text never specifies: (i) which dataset(s) were used, (ii) which model architecture or training pipeline produced the baseline, (iii) how uniqueness scores were computed, (iv) whether the model used to generate disagreement was trained on the full data, a subset, or validation examples, and (v) whether any validation labels or hold-out model evaluations influenced the choice of the 20% threshold or the ranking. Without this, the reported equivalence could be produced by selection leakage: if validation labels were used to pick the subset, or if the baseline was trained with a different budget or architecture, the result is not evidence for smart-sizing. Additionally, Figure 6's caption references 'six training epochs' but no error bars, repetitions, or seed variance are reported; the 'in some configurations outperformed' claim is not quantified. Section 6.4 then asserts that Figures 1 and 2 (workflow diagrams) 'demonstrate that labeling cost includes...' which is a non sequitur and suggests the experiments are being over-interpreted. The core empirical claim is therefore unsupported as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes \"smart-sizing,\" a framework for selectively labeling training data in high-stakes domains, and \"Adaptive Label Optimization (ALO)\" as its operational implementation. The authors claim that models trained on only 20–40% of a dataset, selected by embedding diversity and model disagreement, can match or exceed full-data baselines, and that label errors shared between training and validation sets can mask true generalization loss. The manuscript is primarily conceptual, with a decision rule ΔPerf(x)/C(x) > τ for when to label, and reports experiments in Section 6 that are described qualitatively.","tokens_in":9142,"tokens_out":5371,"duration_ms":46168,"significance":"If the empirical claims were substantiated, the framework would be valuable for resource-constrained, high-consequence AI applications where labeling budgets are tight and rare-class detection is critical. The paper correctly highlights a real problem—annotation throughput often trades off against data value—and it draws on a relevant body of literature including Ng, Maggio, Sambasivan, Kotian, and Northcutt. It also stresses an important and sometimes overlooked risk: shared train/validation label errors can inflate evaluation metrics. However, the central empirical claims are presented without datasets, model specifications, error bars, leakage controls, or code, and the decision rule is not operationalized. As a result, the significance of the paper's specific experimental contribution cannot currently be assessed.","major_comments":[{"comment":"The central empirical claim—that models trained on only 20% of data selected by \"embedding space diversity and model disagreement\" achieve performance within 5% of full-data baselines—is not reproducible from the information provided. The manuscript does not identify the datasets, model architectures, training pipelines, or the exact computation of \"embedding-based uniqueness\" and \"model disagreement.\" It also does not state whether the disagreement model was trained on the full dataset, a subset, or validation examples, or whether any validation labels influenced the selection threshold. Without these details, the reported equivalence could be an artifact of selection leakage or a weak baseline, so the claim cannot be assessed.","section":"§6.2, Figures 5–6"},{"comment":"The assertion that the experiments \"provide strong validation\" for the decision rule ΔPerf(x)/C(x) > τ is circular. The rule was introduced in §4 as a governing convention for when to label, but ΔPerf(x) and C(x) are never estimated for individual samples in any experiment. Figures 1 and 2 are workflow diagrams, not empirical results, and Figures 3 and 4 concern label-error distortion, not marginal utility per cost. Therefore the sentence \"This decision logic is no longer a theory... validated by experimental evidence\" is unsupported by the presented material.","section":"§6.4, Eq. (1)"},{"comment":"The label-error experiment is under-specified: no dataset, model, error-injection mechanism, or evaluation protocol is described, and no quantitative accuracy values are reported. The qualitative conclusion that shared train/validation errors inflate validation performance is already established in the cited literature (Northcutt et al.), so without specific new measurements this experiment does not advance the paper's empirical contribution.","section":"§6.1, Figures 3–4"},{"comment":"The decision rule and its components (ΔPerf(x), C(x), τ) are not operationalized. The manuscript gives a list of possible proxies (e.g., model uncertainty, annotator disagreement) but no explicit scoring function, calibration procedure, or threshold selection method. As a result, the proposed \"quantifiable method\" is not actually quantifiable as presented, and the ALO workflow cannot be implemented or evaluated from the text.","section":"§5.2"}],"minor_comments":[{"comment":"The caption mentions \"six training epochs\" but the figure is described without axis labels, error bars, or per-epoch values; please clarify the construction and add uncertainty estimates or repeated runs.","section":"Figure 6 caption"},{"comment":"Reference [5] (Nagase et al.) lacks place of publication and page numbers, and reference [6] (Li et al.) contains a malformed citation string with a \"contentReference\" tag; please fix these.","section":"References [5] and [6]"},{"comment":"The statement \"Figures 3 and 4 illustrate how such samples contributed disproportionately to model improvement\" is inaccurate because those figures address label-error effects, not per-sample contribution.","section":"§6.4"},{"comment":"These figures are workflow diagrams, but they appear in a way that may confuse readers into thinking they are empirical results; please add a note clarifying that they are schematics.","section":"Figures 1 and 2"}],"recommendation":"reject","confidential_remarks":"This manuscript reads more like a white paper or position statement than a research article for a machine learning venue. The empirical claims are asserted without the experimental details required for verification, and the validation of the decision rule is logically circular. If the journal publishes perspective pieces, the paper might be reconsidered in that format, but the current research-track submission is not sufficiently supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a readable, well-organized argument for treating training data curation as a performance-driven process, but the empirical claims that make it look like a research paper are not checkable as written. The 20%-vs-full-data result in §6.2 could be leakage, a weak baseline, or an artifact; the text gives you no way to tell.\n\nWhat it does well: the literature review is honest and mostly on point. It draws on Maggio et al. (2023) for data diet results, Northcutt et al. (2021) for label errors, and a set of active-learning and data-centric AI work. The smart-sizing/ALO framework is a reasonable synthesis of standard ideas: uncertainty sampling, disagreement, pre-labeling, SME review, and marginal-utility stopping. For a manager or program lead, it is a sensible way to think about annotation budgets and feedback loops. The structural barriers section (§3) is a fair description of how annotation pipelines often fail.\n\nWhere it falls down: the experiments. Section 6 never names a dataset, a model family, a training pipeline, a selection algorithm, or a leakage control. Figures 5 and 6 have no error bars, no repetitions, no seeds. The claim that smart-sized models \"even outperformed\" full-data models on rare-class recall is unquantified. The label-error experiment is a straightforward demonstration of a known result, but it too lacks the details that would make it independently reproducible.\n\nThe bigger logical problem is §6.4. The authors say Figures 1 and 2 \"demonstrate\" that labeling cost includes downstream risk, but those figures are workflow diagrams, not data. And they treat ΔPerf(x)/C(x) > τ as something the experiments \"validate.\" That rule is a convention for how to prioritize labeling, not a falsifiable prediction. Calling it validated is circular.\n\nThe paper would be more honest as a position paper or a practitioner guide, without the pretense of experimental validation. If the authors have real experiments, they need to ship the dataset and code; then the 20% claim could be tested. As it stands, I would not send this to a scientific peer-review process in its current form. It is not a scientific demonstration; it is a management manifesto with a few illustrative simulations.\n\nMy take: worth a skim for the framework, but not citable as evidence. The authors should be told to either provide the full empirical protocol or stop making quantitative claims.\n\nHope this helps.","headline":"A clear-eyed white paper on data curation whose experimental core is too underspecified to test; the framework is useful, the evidence is not.","tokens_in":9616,"tokens_out":2611,"would_cite":false,"duration_ms":22818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that carefully chosen 20 percent of training data can match full-dataset baselines in high-consequence AI, and that validation accuracy alone can hide label-error damage.","keywords":["smart-sizing","Adaptive Label Optimization","active learning","training data curation","rare class recall","label efficiency","annotation governance","label-error auditing"],"falsifier":"Run the same experiment with the subset-selection code locked to unlabeled embedding diversity and model disagreement only, train the full-data baseline under identical epochs, batch size, and compute budget, and evaluate both on a clean held-out set; if the 20-percent model is not within 5 percent of the full-data baseline, or if reproducing the gap requires validation information during selection, the paper's central claim is not supported.","tokens_in":8580,"feed_emoji":"🎯","tokens_out":11092,"duration_ms":89842,"temperature":0.7,"pith_summary":"This paper argues that in high-stakes AI, the bottleneck is not how many examples are labeled but which ones are. It introduces smart-sizing, a strategy for labeling only examples whose expected performance gain outweighs the labeling cost, and Adaptive Label Optimization (ALO), an iterative process that uses pre-labeling, expert review, disagreement tracking, and stopping rules to put that strategy into practice. The central experimental claim is that models trained on 20 to 40 percent of a dataset, chosen for diversity across the model's internal representation space and for model-annotator disagreement, match or beat full-dataset baselines, especially on rare-class recall and edge cases. A second set of experiments shows that when the same label errors are baked into both training and validation sets, validation accuracy can stay high even with 25 percent of labels wrong, hiding real generalization failure. The result matters because it points to a path toward more reliable models with far lower labeling budgets, and to a way of auditing data quality before deployment.","feed_headline":"Twenty percent of curated training data matches full-dataset models","feed_subtitle":"Smart-sized subsets preserve rare-class recall and edge-case generalization while cutting annotation cost.","key_machinery":"The load-bearing mechanism is the smart-sizing decision rule $\\Delta\\mathrm{Perf}(x) / C(x) > \\tau$, which says to label a sample only when the expected performance gain, estimated from model uncertainty, annotator-model disagreement, class rarity, and diversity in the model's learned representation space, exceeds the labeling cost such as expert time and re-labeling risk. Adaptive Label Optimization (ALO) is the named workflow that operationalizes this rule: a batch of candidates is pre-labeled, ambiguous or high-risk cases go to subject-matter experts, labels are compared with model predictions, the model is retrained, and embedding-space shifts are checked before deciding whether to continue. This ratio gives the paper's argument its testable content, and both experiment sets are interpreted through it: the 20-percent result is read as high $\\Delta\\mathrm{Perf}$ per unit cost, while label-error inflation is read as a hidden rise in cost that the rule would expose.","core_discovery":"The central claim is that label selection governs model risk more than label volume does. Concretely, the paper reports that a model trained on a 20 percent subset chosen by embedding-space diversity and model disagreement lands within 5 percent of full-dataset baselines and can outperform them on rare-class recall and edge-case generalization. It also reports that systematically wrong labels placed in both training and validation sets can keep validation accuracy high even when a quarter of the labels are incorrect, so apparent validation success is not evidence of real generalization. In the authors' framing, these findings establish smart-sizing: annotation should be an ongoing, feedback-driven decision loop in which each label is justified by expected performance gain and the process stops when marginal value falls below a threshold.","pith_inferences":["The paper leaves implicit that its diversity-plus-disagreement selection could be packaged into standard active-learning components, making smart-sizing a default behavior of annotation software rather than a bespoke process.","A cheap extension of the label-error findings: hold out a small clean hand-labeled sample, compare predictions against it, and treat a large gap versus the official validation set as evidence of systematic label corruption.","The experiments are limited to vision data, so a direct next test is whether the same 20-percent result transfers to text, audio, or tabular datasets where disagreement and representation diversity are measured differently.","A practical production test suggested by the decision rule is to log $\\Delta\\mathrm{Perf}(x)$ and $C(x)$ per labeling batch and watch when their ratio crosses $\\tau$, giving an organization an empirical stop signal."],"forward_implications":["Annotation budgets in high-consequence programs could shrink by 60 to 80 percent while model quality is preserved, freeing money and expert time for schema refinement and edge-case review.","Models built from smart-sized subsets may generalize better in the field than full-data models, since gains concentrate on rare classes and edge cases rather than on common patterns.","Validation accuracy should no longer be treated as proof of readiness unless label-error audits are in place; the experiments show validation curves can look healthy while true generalization is lost.","The rule $\\Delta\\mathrm{Perf}(x)/C(x) > \\tau$ gives labeling teams a defensible stopping point, replacing quota-driven annotation with a marginal-value threshold that can be tuned to mission risk.","Realizing the reported gains requires annotation tooling that displays disagreement, embedding diversity, and marginal label value in real time, not just tools that maximize labeling speed."],"supporting_citations":[{"why":"Supplies the premise that a small number of well-designed examples can teach a model, which is the conceptual foundation for smart-sizing.","marker":"[1]"},{"why":"Reports that models trained on about 30 percent of data retain roughly 95 percent of full-dataset accuracy, the quantitative baseline the paper sharpens to 20 percent.","marker":"[2]"},{"why":"Documents how early annotation errors propagate through high-stakes AI development, the mechanism behind the paper's label-error distortion experiments.","marker":"[3]"},{"why":"Provides a prior active-learning method that prioritizes examples by class-specific performance gaps, the closest existing precedent for ALO's target-aware selection.","marker":"[4]"},{"why":"Shows that pre-labeling combined with human correction accelerates annotation without quality loss, supporting ALO's pre-labeling and expert-review stages.","marker":"[7]"},{"why":"Gives evidence that benchmark validation sets contain systematic label errors, grounding the claim that validation accuracy can be misleading.","marker":"[9]"}],"fun_headline_variants":["Smart-sized data: 20% beats full set on rare cases","Less data, smarter labels: AI wins with 20% curation","Cut training set by 80% and still ace edge cases","Label quality over quantity: 20% data matches full","Ditch 80% of labels, keep edge-case accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Section 6.2 result stands on the assumption that the 20-percent subsets were chosen using only training-time signals, namely unlabeled embedding diversity and model disagreement, with no validation labels steering the selection, and that the full-data baseline was trained under the same pipeline and budget; the paper provides no algorithm, leakage controls, or baseline details to confirm either condition.","fun_headline_variants_meta":{"raw":{"variants":["Smart-sized data: 20% beats full set on rare cases","Less data, smarter labels: AI wins with 20% curation","Cut training set by 80% and still ace edge cases","Label quality over quantity: 20% data matches full","Ditch 80% of labels, keep edge-case accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1511,"prompt_tokens":868,"completion_tokens":643,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":556}},"tokens_in":484,"tokens_out":643,"duration_ms":5586,"temperature":1.0,"reasoning_tokens":556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:25:52.364408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiment with the subset-selection code locked to unlabeled embedding diversity and model disagreement only, train the full-data baseline under identical epochs, batch size, and compute budget, and evaluate both on a clean held-out set; if the 20-percent model is not within 5 percent of the full-data baseline, or if reproducing the gap requires validation information during selection, the paper's central claim is not supported.","supporting_citations":[{"cited_title":"Unbiggen AI","cited_arxiv_id":null,"evidence_quote":"Supplies the premise that a small number of well-designed examples can teach a model, which is the conceptual foundation for smart-sizing."},{"cited_title":"ML Models on a Data Diet","cited_arxiv_id":null,"evidence_quote":"Reports that models trained on about 30 percent of data retain roughly 95 percent of full-dataset accuracy, the quantitative baseline the paper sharpens to 20 percent."},{"cited_title":"Target-Aware Active Learning (TAAL)","cited_arxiv_id":null,"evidence_quote":"Provides a prior active-learning method that prioritizes examples by class-specific performance gaps, the closest existing precedent for ALO's target-aware selection."},{"cited_title":"Human-Machine Collaboration on Image Annotation","cited_arxiv_id":null,"evidence_quote":"Shows that pre-labeling combined with human correction accelerates annotation without quality loss, supporting ALO's pre-labeling and expert-review stages."}],"review_version":1}