{"id":"442992ac-55d6-4898-ad07-4d413e79b7ec","arxiv_id":"2412.06575","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DQE selects about half of a training set via greedy sampling plus similarity-based categorization of uncovered, difficult, and noisy samples, and reports improved text classification accuracy over full-data fine-tuning.","lead":"This paper proposes DQE, a data selection method that uses greedy sampling, nearest-neighbor similarity, and GPT-4o verification to build a half-size fine-tuning set for LLM text classification. On six public datasets, the method reports accuracy above full-data fine-tuning while using roughly half the training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DQE's noise-removal step depends on the assumption that similar texts with different labels are annotation errors; if many are legitimate near-boundary examples, the method deletes valid training data and the claimed half-data advantage may reflect selection bias rather than quality enhancement.","rationale":"The central claim is empirical: with about half the data, DQE improves accuracy over full-data fine-tuning on six public benchmarks. For that claim to hold as a general data-quality method, the 'noisy' label must actually identify annotation errors; otherwise the method's success would be an artifact of deleting a region of the input space. Section 3.3's premise that similar texts share a label is the exact point where the argument is least secure. I therefore focused the stress test on this premise rather than on the secondary K-Center-Greedy coverage assumption, which affects the definition of 'uncovered' but not the final selection in the same way. The reader's verdict already conditions acceptance on releasing code, multiple seeds, and stronger baselines; my concern adds a specific requirement to validate the noisy-pair judgments. I do not see an internal inconsistency in the pipeline, and the use of public datasets and a concrete selection algorithm is a point in the paper's favor. However, because the noisy-pair audit has not been done and the GPT-4o oracle is unmeasured, the conditional verdict should stand; I recommend UNCHANGED rather than a move to REJECT because the method is plausible and the missing evidence is obtainable.","tokens_in":11943,"tokens_out":8413,"duration_ms":99541,"concrete_test":"Manually audit all, or a stratified random sample of roughly 300 per dataset from, the pairs that DQE flagged as noisy. Using each dataset's original annotation guidelines, two independent annotators judge whether the two labels are both defensible for their respective texts or whether at least one label is a clear error. Compute the proportion of pairs where both annotators classify the disagreement as legitimate rather than a clear error, and measure agreement with GPT-4o's verdicts. If that proportion is substantial (e.g., above 20%) or GPT-4o agreement is low, the core noise premise in Section 3.3 fails and the method should be re-evaluated with a human-validated noise set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 defines noisy data using the premise that 'two highly similar data items have the same label': an incorrectly predicted unsampled item whose nearest neighbor has a different label is suspected of label noise and, after GPT-4o confirmation, removed from the final training set. This premise is load-bearing because the headline result is that a roughly half-size DQE set beats full-data training, and the mechanism claimed for that result is the removal of noisy data. The premise fails whenever semantic similarity does not imply class equivalence. In sentiment datasets and in AG News, legitimate near-boundary or multi-faceted examples can be very similar in embedding space yet carry different valid labels; a generic embedding model (all-mpnet-base-v2, not fine-tuned on the task) makes this more likely, not less. The paper provides no quantitative verification that the pairs flagged as noisy are actually mislabeled: Figure 4 shows a handful of examples, and the GPT-4o check is a proprietary, unmeasured oracle. If a substantial fraction of flagged pairs are valid boundary cases, DQE is not cleaning noise but deleting a region of the training distribution, which can raise test accuracy if that region is underrepresented in the test set while hurting generalization and undermining the 'data quality enhancement' interpretation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes DQE, a data-selection pipeline for fine-tuning large language models on text classification. The method first selects a diverse half of the training set using K-Center-Greedy on embeddings from all-mpnet-base-v2, fine-tunes Qwen2-7B-Instruct on that subset, and then labels misclassified unsampled examples as uncovered, difficult, or noisy by comparing each to its nearest neighbor in the full training set via cosine similarity. Uncovered and difficult examples are added to the final training set, while noisy examples are verified by GPT-4o and removed. On six public benchmarks the authors report that the resulting roughly half-size training set yields higher test accuracy than full-data fine-tuning and also surpasses greedy sampling on every task, leading them to claim better accuracy with nearly half the training time.","tokens_in":12203,"tokens_out":4893,"duration_ms":53944,"significance":"If established, the claim that a carefully selected half-size training set can outperform the full dataset for LLM fine-tuning would be practically important, and the paper's decomposition of selected data into uncovered, difficult, and noisy categories is a useful interpretive framing. The authors use standard public datasets and an open base model, and the core comparison across Full-Data, Greedy, and DQE is performed with the same base model and hyperparameters, which is a reasonable experimental design. However, the evidence is currently insufficient to support the central claim: all results come from single fine-tuning runs with no seed variation, the significance test in Table 3 is applied to per-sample predictions of two fixed models rather than across independent runs, and the load-bearing noise-detection assumption in Section 3.3 is not validated quantitatively. The absolute gains over Full-Data are small (0.32 to 1.13 percentage points), so without variance estimates they are consistent with training noise.","major_comments":[{"comment":"The statistical test does not support the claim that DQE reliably outperforms Full-Data. The t-test is computed on per-sample binary outcomes (1 for correct, 0 for incorrect) for two fixed models, so it can only describe a difference on this particular test set for these particular trained models. It does not account for variance across independent fine-tuning runs, and the paper reports only one run per condition. To establish the central claim, the authors should run multiple random seeds for each method and dataset, report mean accuracy and standard deviation, and use an appropriate test on the distribution of accuracy differences across runs.","section":"§4.4, Table 3"},{"comment":"The noise-detection step relies on the premise that two highly similar texts must have the same label, but this premise is not justified and is likely violated for legitimate near-boundary examples, especially in sentiment classification and AG News. The generic all-mpnet-base-v2 embedding is not task-specific, so valid opposite-label texts can have high cosine similarity. The paper provides only a handful of anecdotal examples in Figure 4 and an unmeasured GPT-4o verification, with no precision or recall analysis of the noise detector against human-annotated labels. Without this validation, the claim that DQE enhances quality by removing label noise is not established; the removed examples could be valid training data, and the reported accuracy gains could reflect a different selection effect.","section":"§3.3, Figure 4"},{"comment":"The reported improvements of DQE over Full-Data are small (0.32% for AG News, 0.33% for SST-5, 0.53% for CR, 0.54% for IMDb, 0.88% for SST-2, 1.13% for MR) and are presented without any measure of run-to-run variability. With a single training run per condition, these differences are within the range that seed variation can produce. Additionally, the paper does not provide a significance test for DQE versus Greedy, even though the narrative that DQE improves on greedy sampling is supported only by the point estimates in Table 2.","section":"§4.4, Table 2"},{"comment":"The comparison against the PapersWithCode baselines is not a controlled comparison: those numbers come from different model architectures, training procedures, and hyperparameter choices, so the statement that DQE 'has achieved state-of-the-art performance' is not supported by direct evidence. The same-setting comparison among Full-Data, Greedy, and DQE is the valid core of the paper, and the SOTA claim should either be removed or backed by identical-base-model baseline runs.","section":"§4.4, 'Baseline' column in Table 2"}],"minor_comments":[{"comment":"The description of K-Center-Greedy is imprecise: the statement that the algorithm 'iteratively selects the sample that is farthest from the current vector center' suggests a single center rather than the set of already selected points; the algorithm should be described as selecting the point with the largest distance to the nearest selected point.","section":"§3.2"},{"comment":"The phrase 'the second stage of DeepSpeed's data parallel training strategy' is ambiguous; it should say 'ZeRO stage 2' to be clear about the distributed training configuration.","section":"§4.3"},{"comment":"The p-value for SST-2 is formatted inconsistently as '< 0.0001 (6.14e-05)'; either report the exact p-value or use a uniform format across all rows.","section":"Table 3"},{"comment":"Several example texts in Figure 4 contain corrupted glyphs (for example, '濠濥' and similar artifacts) that make the examples difficult or impossible to read; clean English text should be used.","section":"Figure 4"},{"comment":"The claim that many test-set errors are actually correct predictions on mislabeled test examples is supported by only a few selected examples; a systematic random-sample annotation with counts and inter-annotator agreement would be needed to make this point quantitative.","section":"§5.2"},{"comment":"The paper would benefit from a statement on the availability of code and the exact selected data subsets; without these, the DQE procedure is not fully reproducible.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central idea is plausible and the paper is clearly written, but the experimental evidence for the headline claim is under-powered: single runs per condition and a misapplied per-sample t-test do not establish that the half-size DQE set reliably outperforms full-data training. The noise-detection premise is also load-bearing and unvalidated. I would be willing to reconsider after the authors add multi-seed experiments, proper cross-run statistical testing, and a quantitative evaluation of the precision of the noise detector on a human-annotated subsample. No concerns about scholarship or citation integrity beyond the unverified SOTA comparison against PapersWithCode baselines."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: a genuinely useful data-selection recipe with an honest write-up, but the headline claim rests on single runs and a misused significance test. I’d still send it to review—the method is elegant and the result is worth chasing.\n\nWhat’s actually new: combining K-Center-Greedy diversity sampling with a second error-driven pass, where the model fine-tuned on the sampled half forecasts the unsampled half, and those misses are split into uncovered, difficult, and noisy using nearest-neighbor label agreement, with GPT-4o as the final noise judge. That specific pipeline isn’t in the prior work they cite, and it’s a sensible way to turn a model’s blind spots into training data. The paper is clearly written, uses public datasets, a public base model, and contains a genuine limitations section.\n\nWhat it does well: the internal comparison is the useful part. DQE improves over Greedy on all six tasks, typically by a small but consistent margin, while keeping the set around half the original size. That is the kind of practical result people can build on. They also check for malformed outputs, which is often ignored.\n\nWhere it’s soft: the t-test in Table 3 is not testing what they claim. A t-test on per-sample binary outcomes compares two fixed checkpoints, not the variance across independent fine-tuning runs. You can’t conclude that one procedure is reliably better without multiple seeds. The 0.5–1 point advantage could be train-once luck. The GPT-4o noise check is a proprietary, unmeasured oracle; the stress-tester’s concern about near-boundary examples is fair, and while Figure 4 shows some likely mislabeled pairs, there’s no human audit to estimate the false-deletion rate. Missing code and data make the numbers uncheckable. The PapersWithCode baselines are from different model families, but the key comparisons are against Full-Data and Greedy, so that’s a minor issue.\n\nThe citation pattern is fine. They acknowledge relevant work on diversity (Sener & Savarese; MODS; D4; IFD) and don’t oversell novelty. The idea of using similarity as a proxy for label consistency is load-bearing, and they at least flag the limitation.\n\nRealistic recommendation: accept for peer review with major revision. The fixes are concrete: multiple seeds (report mean±std), a proper paired test across runs, release code/data, and a small human audit of GPT-4o’s noise flags. If those hold up, this becomes a standard recipe; if not, it’s still a clean negative result. I’d want it in the literature either way.","headline":"A practical half-data selection recipe with a plausible but not yet solid empirical claim; the idea is good, the evidence needs seeds and code.","tokens_in":12713,"tokens_out":4025,"would_cite":true,"duration_ms":42345,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A half-size training set, selected for diversity, beats full-data fine-tuning on every benchmark tested.","keywords":["data quality enhancement","text classification","large language models","fine-tuning","data selection","label noise","diversity sampling","uncovered difficult noisy data"],"falsifier":"Evaluate DQE on a dataset whose labels are hand-verified to be noise-free, or on a dataset with artificially injected label noise at known rates; if the method still removes a large proportion of clean examples or fails to recover the injected noise, then the nearest-neighbor disagreement heuristic is not isolating label noise, and the accuracy gains should be attributed to something other than noise removal.","tokens_in":11771,"feed_emoji":"🧠","tokens_out":8981,"duration_ms":78599,"temperature":0.7,"pith_summary":"The paper proposes a data quality enhancement (DQE) method for fine-tuning large language models on text classification. It first uses K-Center-Greedy sampling to pick a diverse half of the training set, fine-tunes on that half, and then predicts the unsampled half. Incorrectly predicted samples are split into three categories using cosine similarity to their nearest neighbor: uncovered, difficult, and noisy. Noisy samples are removed while uncovered and difficult samples are added to the final set, which stays around half the original size. The paper claims this DQE-selected set beats full-data fine-tuning on all six public benchmarks while saving nearly half the training time.","feed_headline":"Half the training data beats fine-tuning on all six benchmarks","feed_subtitle":"A diversity-first sample plus error triage cuts compute and lifts accuracy across six public benchmarks.","key_machinery":"The load-bearing mechanism is the three-way classification of incorrectly predicted samples based on the maximum cosine similarity to any other training sample. A sample is 'uncovered' if its nearest neighbor lies in the unsampled set and shares its label, 'difficult' if the nearest neighbor lies in the sampled set and shares its label, and 'noisy' if the nearest neighbor carries a different label. This single-nearest-neighbor rule, applied on top of K-Center-Greedy diversity sampling, determines which mispredicted samples are added back to the training set and which are removed. The 'noisy' candidates are further confirmed by GPT-4o before deletion.","core_discovery":"The central discovery is that a training set built from a diversity-greedy half, enriched with mispredicted samples whose nearest neighbor shares their label, and purged of samples whose nearest neighbor disagrees, yields higher test accuracy than training on the whole dataset. The authors attribute this to label noise: full-data fine-tuning is dragged down by mislabeled examples, which the greedy seed set partially avoids and DQE further removes. Across MR, CR, IMDb, SST-2, SST-5, and AG News, the DQE-selected model achieves the highest accuracy in every task, outperforming full-data training by margins up to 1.13 points, with statistically significant gains in four of six datasets. The paper also shows that some remaining test-set errors are themselves due to incorrect labels, so the true performance gap may be even wider.","pith_inferences":["The nearest-neighbor rule may be too aggressive on genuinely hard or near-boundary examples where similar texts legitimately carry different labels, so the method could discard valid data; a multi-neighbor vote or similarity threshold might improve robustness.","The noise-verification step depends on GPT-4o, which is a reproducibility concern; the paper does not test whether a smaller open-source verifier would give the same results.","Combining DQE with active learning or iterative re-sampling could use the 'uncovered' signal to guide further data collection, potentially extending the gains beyond a single pass.","The data-efficiency gain suggests that within a fixed compute budget one could double the model size or training epochs while keeping the same cost, potentially yielding even higher accuracy than reported."],"forward_implications":["Fine-tuning an LLM for text classification can use roughly half the training data and still outperform full-data training, cutting training time approximately in half.","The three-way error taxonomy gives a practical recipe for cleaning noisy public benchmarks without full manual re-annotation.","The method is model-agnostic and scalable: it needs only a vectorizer, a greedy sampler, and one fine-tuned model, so it applies to large datasets.","Because some test-set errors are themselves mislabeled, DQE models' true accuracy may be higher than the reported numbers on noisy test sets."],"supporting_citations":[{"why":"Supplies the K-Center-Greedy core-set selection algorithm that forms the diversity-sampled seed set.","marker":"Sener and Savarese (2018)"},{"why":"Phi-1.5 technical report, cited as evidence that data quality matters more than quantity, motivating the focus on data selection.","marker":"Li et al. (2023b)"},{"why":"The GPT-4 technical report; the GPT-4o model built on it is used as the noisy-data verifier in the selection pipeline.","marker":"Achiam et al. (2023)"},{"why":"The Qwen2 technical report; specifies the Qwen2-7B-Instruct base model that is fine-tuned in all experiments.","marker":"Yang et al. (2024)"},{"why":"Introduces the Instruction-Following Difficulty (IFD) metric, the closest prior data-selection method that DQE builds on and contrasts with.","marker":"Li et al. (2024a)"},{"why":"Provides the SST-2 and SST-5 datasets, two of the benchmarks used to test the claim of beating full-data fine-tuning.","marker":"Socher et al. (2013)"}],"fun_headline_variants":["Half the training data, full accuracy: diversity-first selection","Train on half, but smarter: error triage boosts LLM classification","Less data, more accuracy: LLM text classification gets a boost","Half data, half time, better accuracy: LLM text classification","Diversity-first data selection: cut data, clean noise, beat baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that any two highly similar text items should have the same label, so a nearest neighbor with a different label is treated as a labeling error; if that assumption fails for near-boundary or multi-faceted examples, valid data get discarded and the method's gains could reverse.","fun_headline_variants_meta":{"raw":{"variants":["Half the training data, full accuracy: diversity-first selection","Train on half, but smarter: error triage boosts LLM classification","Less data, more accuracy: LLM text classification gets a boost","Half data, half time, better accuracy: LLM text classification","Diversity-first data selection: cut data, clean noise, beat baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001205,"raw_usage":{"total_tokens":4947,"prompt_tokens":909,"completion_tokens":4038,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":3946}},"tokens_in":525,"tokens_out":4038,"duration_ms":26333,"temperature":1.0,"reasoning_tokens":3946,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:30:06.111467+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate DQE on a dataset whose labels are hand-verified to be noise-free, or on a dataset with artificially injected label noise at known rates; if the method still removes a large proportion of clean examples or fails to recover the injected noise, then the nearest-neighbor disagreement heuristic is not isolating label noise, and the accuracy gains should be attributed to something other than noise removal.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SST-2 and SST-5 datasets, two of the benchmarks used to test the claim of beating full-data fine-tuning."}],"review_version":1}