{"id":"2193cb19-4b2a-4d87-abef-ae053b8ba652","arxiv_id":"2412.10091","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A soft-label entropy computed from logits averaged over training epochs is proposed as a data-pruning importance metric for ReID, with integrated label correction and outlier removal.","lead":"This paper introduces a data pruning and cleaning method for object re-identification: it ranks training images by uncertainty computed from the model's predictions over the whole training run, then prunes easy images, fixes wrong labels, and removes outliers. It reports cutting up to 35% of training data on three ReID benchmarks with under 0.1% accuracy loss, which could cut storage and training cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 1 averages raw logits without per-epoch normalization, so late-epoch logit growth may dominate the soft label; the claimed full-training-history metric and the pruning/label-correction results built on it need a normalization check.","rationale":"The reader's weakest assumption is exactly the comparability of raw logits across epochs, and the paper provides no evidence that logit magnitudes are stationary during the 12-epoch estimation phase. The central claim of a more accurate and comprehensive metric depends on Eq. 1 faithfully summarizing the whole training history; logit scale growth directly threatens that. The empirical pruning results could still be valid as a late-epoch heuristic, but the paper's conceptual contribution and the label-correction guarantees would need to be restated. I do not see a reason to change the conditional verdict: the issue is addressable with a normalization ablation, and the paper otherwise provides reproducible code, four-seed runs, and clear comparisons. I also note the pipeline-order discrepancy between Sec. 5.5 and A.6 and the lack of significance testing around the '<0.1%' numbers, but the logit-averaging issue is more fundamental because it undermines the proposed metric itself.","tokens_in":18874,"tokens_out":6709,"duration_ms":77780,"concrete_test":"Re-run the MSMT17 pruning and the synthetic-noise label-correction experiments with per-epoch normalized logits before averaging (e.g., subtract per-epoch mean and divide by per-epoch standard deviation, or subtract per-epoch max), and compare against (i) the paper's raw-average results and (ii) a single-epoch soft label using only epoch-12 logits. If raw-average curves are statistically indistinguishable from the epoch-12-only curves, the trajectory is not being leveraged; if normalized-average curves differ by more than the reported <0.1% accuracy or change the outlier/label-correction lists substantially, Eq. 1 needs per-epoch normalization before the central claims can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. 1 defines the soft label as softmax of the arithmetic mean of raw logits z^(t)(x) over T=12 epochs. The paper's own Fig. 2 shows logit magnitudes growing from roughly 0 to about 7 over 13 epochs. Because softmax is monotone and the average is an unweighted sum, later epochs with much larger logits dominate the sum; early-epoch information is largely diluted. Consequently the 'soft label' is close to the final-epoch prediction, and the claimed benefit of 'fully exploiting the logit trajectory' (Sec. 3.2) is not actually realized by Eq. 1. This matters not only for the pruning metric but also for label correction (argmax(y~) != y triggers relabeling) and outlier removal (max(y~) <= delta), since both use the same averaged soft label. The 12-epoch sufficiency result (Fig. 9) and the logit-accumulation ablation (Figs. 10-11) are confounded: accumulating more epochs mostly adds later high-magnitude logits, so the observed improvement does not establish that the full trajectory is being used. If logit scale drift dominates, the proposed metric is essentially a final-epoch confidence/entropy heuristic, and the paper's central conceptual claim is unsupported, even if the empirical pruning numbers remain useful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a data pruning and purification framework for object re-identification. For each training sample, it records the logits of the classification head over T=12 training epochs, averages them, applies softmax to form a 'soft label', and uses the entropy of this soft label as the importance score. It then prunes low-entropy (easy) samples, corrects mislabeled samples when argmax of the soft label disagrees with the assigned label, and removes outliers whose maximum soft-label score is below a threshold delta. Experiments on Market1501, MSMT17, and VeRi with ResNet50 (and generalization to ResNet101/ViT) report that 5%, 30%, and 35% of samples can be removed with negligible (<0.1%) loss in accuracy, and the method also improves accuracy on synthetically noised datasets. The code and sample lists are publicly available.","tokens_in":19111,"tokens_out":5838,"duration_ms":58123,"significance":"If the empirical claims hold, the paper provides a practical, architecture-agnostic data pruning tool for ReID, a task where data pruning has been little studied. The method's estimated low cost (12 epochs, no multi-model ensembling) and the release of sample importance, mislabel, and outlier lists are concrete contributions. The paper includes a useful assessment of existing pruning baselines on ReID and demonstrates cross-architecture transfer of the learned ranking. However, the conceptual novelty rests on the untested claim that the soft label genuinely aggregates the full training trajectory, and the key comparisons contain a loss-function confound; these issues prevent the current version from fully supporting the stated claims.","major_comments":[{"comment":"The comparison between the proposed score and EL2N is confounded by the training loss. The proposed score is computed from logits of a model trained with cross-entropy plus triplet loss, while the EL2N baselines use cross-entropy only, as stated in Appendix A.2. With metric loss, the model produces different logit geometry, and the better pruning curves in Fig. 4 could reflect a better-trained model rather than a better importance metric. Please rerun EL2N with the same CE+triplet loss (or rerun the proposed method with CE only) and confirm that the gap remains.","section":"Sec. 5.3 / Appendix A.2"},{"comment":"Eq. 1 averages raw logits across epochs without any per-epoch normalization. The paper's own Fig. 2 shows logit magnitudes growing from roughly 0 to about 7 over 13 epochs. Since softmax is monotone, the averaged logits are dominated by the later high-magnitude epochs, so the 'soft label' is close to the final-epoch prediction. Consequently, the claimed benefit of 'fully exploiting the logit trajectory' is not established. The ablations in Figs. 10-11 compare accumulation frequencies (every 1/2/4/6 epochs) but are also explained by scale: adding more epochs mostly adds later large logits. The 12-epoch result in Fig. 9 is likewise not evidence for trajectory use. Please normalize logits per epoch (e.g., standardize or use temperature) before averaging, or directly compare against a 'last-epoch only' baseline to show what the accumulation adds.","section":"Sec. 3.3, Eq. 1, Fig. 2, Figs. 9-11"},{"comment":"The headline claim of removing 35%, 30%, and 5% of samples with <0.1% accuracy loss is presented without error bars in Fig. 4, although the caption reports means over four seeds. Without measures of variance on the pruning curves (and a paired comparison to the no-pruning baseline), the reader cannot assess whether the differences are within noise. Please add error bars or confidence intervals to all main pruning figures and report the per-seed accuracy differences against the no-pruning baseline.","section":"Fig. 4 / Sec. 5.3.1"}],"minor_comments":[{"comment":"There is a typo at the end of Appendix D.1: 'e.g. 10%, 20%.,' should read 'e.g., 10%, 20%.'","section":"Appendix D"},{"comment":"In the 'Differences in Datasets' paragraph, 'MSMT' is used once instead of 'MSMT17'; please make the dataset name consistent.","section":"Sec. 5.3.1"},{"comment":"The comparison with supervised prototypes replaces the original self-supervised learning with a supervised variant; this deviation should be acknowledged in the main text of Sec. 5.3.1 as well, not only in the appendix.","section":"Appendix A.2"},{"comment":"The phrase 'a logit value, presented on a logarithmic scale, represents the network's predicted probability of a particular class' is imprecise; logits are not generally log-probabilities. Please rephrase to avoid confusion.","section":"Sec. 3.3"},{"comment":"The abstract says the approach reduces importance-score estimation cost by 10 times, but Table 1 shows roughly 10x versus forgetting/prototypes and roughly 20x versus EL2N(20 models). Please clarify the reference point for the 10x claim.","section":"Abstract / Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical study with useful artifacts (code and sample lists), and the pruning results are potentially publishable. However, the loss-function confound in the EL2N baseline and the lack of logit normalization in Eq. 1 both bear on the central claims. If the authors can re-run the key comparisons under matched losses and either justify or fix the logit averaging, the contribution would be much stronger. The missing error bars in the main pruning figure should also be addressed for the quantitative headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper, but read it with the logit-scale concern in mind. The core idea is simple: average raw logits over 12 training epochs, apply softmax to get a 'soft label', and use its entropy as an importance score. That's a useful heuristic for ReID, where data pruning hasn't been systematically studied before. The empirical results are real: you can drop 35% of VeRi, 30% of MSMT17, and 5% of Market1501 with under 0.1% accuracy loss, and the ablation shows that accumulating logits every epoch works better than using only the final epoch. The efficiency gain—12 epochs vs 120 for forgetting scores—is practical and makes the method attractive.\n\nThat said, the paper's central claim that this 'fully leverages the logit trajectory' is overstated. Eq. 1 averages raw logits without any per-epoch normalization. Since logits grow from ~0 to ~7 during training, later epochs dominate the average. The soft label ends up being close to the last-epoch prediction, so the trajectory benefit is weaker than advertised. This isn't fatal—the empirical results still stand—but the authors should either normalize logits per epoch or average softmax probabilities, and temper the language in Sec. 3.2.\n\nThree other soft spots: (1) The EL2N baseline is trained with cross-entropy only, while the proposed method uses CE + triplet. That's a confounded comparison. Match the loss. (2) Fig. 4 shows no error bars on the pruning curves, so the '<0.1%' claim isn't supported by variance estimates. (3) The closest prior work, Dataset Cartography (Swayamdipta et al., 2020), is not cited. That's a real omission—it also uses training dynamics for pruning and should be compared. Minor: the pipeline order in Sec. 5.5 (outliers, then label correction, then pruning) conflicts with Appendix A.6 (outliers, pruning, correction).\n\nThe label-correction component is only validated on synthetic noise, not real mislabels. That's a caveat, not a dealbreaker. The code and data lists are promised, which helps reproducibility.\n\nBottom line: this is a practical, well-tested method for ReID data pruning. It deserves peer review, not desk rejection. Push the authors to fix the EL2N comparison, add error bars, cite/compare Dataset Cartography, and address the logit scale issue. Then it could be a solid contribution.","headline":"Useful, practical ReID data pruning with a real but fixable flaw: the logit-averaging metric overstates its use of training history.","tokens_in":19686,"tokens_out":4038,"would_cite":true,"duration_ms":39762,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Averaging a sample's logits over 12 training epochs and taking the entropy of that averaged soft label yields a single score that prunes redundant re-identification images, corrects mislabeled ones, and flags outliers.","keywords":["data pruning","object re-identification","soft label entropy","logit trajectory","label correction","outlier detection","sample importance","training dynamics"],"falsifier":"Compute the mean logit magnitude per epoch over the 12-epoch estimation window on MSMT17; if the scale drifts substantially (e.g., more than a factor of 2 between early and late epochs), recompute the soft labels after standardizing each epoch's logits to unit variance before averaging, and compare the resulting pruning ranking and accuracy-versus-pruning curves with the raw-average version. A material difference would falsify the claim that the raw average reflects the full training history.","tokens_in":18634,"feed_emoji":"🧹","tokens_out":14255,"duration_ms":117738,"temperature":0.7,"pith_summary":"This paper tries to establish that a single number, the entropy of a sample's time-averaged soft label, captures how important that sample is for training an object re-identification (ReID) model, and that the same soft label can repair mislabeled images and remove outliers. The method records classification logits at every epoch, averages them over the first 12 epochs, and applies softmax to get a soft label; its entropy ranks samples from easy to hard, and low-entropy samples can be pruned without hurting accuracy. On three ReID benchmarks the paper reports pruning 35% of VeRi, 30% of MSMT17, and 5% of Market1501 training data with under 0.1% loss in mean accuracy, while using only 10% of the training budget to compute the scores. The authors also show the same score works on two image-classification datasets, suggesting the mechanism generalizes beyond ReID.","feed_headline":"Entropy of averaged logits prunes 35% of re-ID images","feed_subtitle":"A 12-epoch soft-label average also corrects mislabeled samples and flags outliers, keeping accuracy loss under 0.1%.","key_machinery":"The time-averaged soft label $\\tilde{y} = \\sigma\\!\\bigl(\\tfrac{1}{T}\\sum_{t=1}^{T} z^{(t)}(x)\\bigr)$ — the softmax of the mean logit vector across $T=12$ training epochs — is the central object of the paper. Its entropy $H(\\tilde{y})$ provides the importance ranking, its argmax provides the corrected label for mislabeled samples, and its maximum class score acts as the outlier detector when it falls below a threshold $\\delta$. The 12-epoch window is what makes the approach cheap: importance is estimated in 10% of the usual 120-epoch training budget, a tenfold reduction in scoring cost relative to methods that need a full training run.","core_discovery":"The central claim is that the entropy of the time-averaged soft label is a more accurate and cheaper measure of sample importance than single-epoch error norms, forgetting events, or feature-space distances. For each image the paper records the classification-head logits at every epoch, averages them across $T=12$ epochs, and applies softmax to obtain $\\tilde{y} = \\sigma\\bigl(\\tfrac{1}{T}\\sum_{t=1}^{T} z^{(t)}(x)\\bigr)$; the importance score is the entropy $H(\\tilde{y}) = -\\sum_i p_i \\log_2 p_i$. High-entropy samples are treated as hard and valuable; low-entropy samples can be pruned. The same soft label drives purification: if $\\arg\\max(\\tilde{y}) \\neq y$ the image is re-labeled as $\\arg\\max(\\tilde{y})$, and if $\\max(\\tilde{y}) \\leq \\delta = 10\\%$ it is removed as an outlier. Integrated, the procedure removes easy samples, repairs mislabeled ones, and deletes outliers, yielding pruned datasets that train faster with under 0.1% mean-accuracy loss on three ReID benchmarks and, as an extension, on two classification benchmarks.","pith_inferences":["If logit magnitudes drift across epochs, the raw average in Eq. 1 is dominated by later epochs; a variant that standardizes each epoch's logits before averaging would test whether the claimed use of the full training dynamics is real, and would change the ranking only if the drift is large.","The entropy of the time-averaged soft label is effectively an uncertainty measure over the training trajectory, so the same quantity could serve as a curriculum ordering or sample-weighting signal during training, not just a one-shot pruning score.","Because the soft label is a full distribution over identities, the method could be repurposed to detect identity-level annotation errors or quantify inter-identity similarity, which the paper does not explore.","The outlier threshold is fixed at 10% of the maximum class score; a data-driven choice based on the distribution of max scores would make the method applicable when the noise rate is unknown."],"forward_implications":["On VeRi, MSMT17, and Market1501, the full pipeline (pruning plus label correction and outlier removal) removes 35%, 30%, and 5% of training samples respectively — with proportionally less training time — while the mean of rank-1 accuracy and mAP stays within 0.1% of the full-data model.","A ResNet50-derived importance ranking transfers to ResNet101 and ViT-B/16, so the pruning step does not need to be repeated when the architecture changes.","Injected label noise of 10% to 50% is largely neutralized by combining label correction (re-label to the argmax of the averaged soft label) with outlier removal (max soft-label score below the threshold), outperforming the AUM baseline on the same noisy settings.","Because the score is computed in 12 epochs rather than a full 120-epoch run, importance estimation costs about 15.8 minutes instead of 2.6 to 5.2 hours on MSMT17, a roughly tenfold saving.","The same importance score works for CIFAR-100 and CUB-200-2011 without pretrained backbones, indicating the mechanism is not specific to re-identification."],"supporting_citations":[{"why":"Defines the EL2N and GraNd baselines and the early-training protocol (12 epochs) that the entropy score is compared against; its 20-model variant supplies the high-cost alternative.","marker":"Paul et al. (2021)"},{"why":"Introduces the forgetting score and the easy-versus-hard sample framing that motivates difficulty-based pruning; supplies the full-training baseline for scoring cost.","marker":"Toneva et al. (2018)"},{"why":"Supplies the supervised-prototype pruning baseline and the random-pruning and no-pruning reference curves used in the pruning comparisons.","marker":"Sorscher et al. (2022)"},{"why":"Provides the AUM ranking metric and threshold-selection procedure used as the comparison baseline for identifying mislabeled samples.","marker":"Pleiss et al. (2020)"},{"why":"Defines the ReID training recipe (cross-entropy plus triplet loss, 120 epochs) whose first 12 epochs yield the logit trajectories and whose settings are used for all full-model trainings.","marker":"Luo et al. (2019)"},{"why":"Documents label noise and outliers in ReID datasets, motivating the data-purification component of the framework.","marker":"Yu et al. (2019)"},{"why":"Contributes the moving-average-loss curriculum learning idea, repurposed here as an additional pruning-metric baseline.","marker":"Zhou et al. (2020)"}],"fun_headline_variants":["Entropy of averaged logits prunes 35% of re-ID data","Re-ID data pruning via logit history cuts 35%, fixes labels","Averaged logit entropy: 35% re-ID data pruned, mislabels corrected","First data pruning for re-ID: 35% less data, 10x cheaper scores","Logit-history entropy prunes re-ID images, repairs mislabels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that raw logits from different epochs can be averaged directly without rescaling: if logit magnitudes drift during training, the average is dominated by the last epochs and the entropy ranking no longer represents the whole training history.","fun_headline_variants_meta":{"raw":{"variants":["Entropy of averaged logits prunes 35% of re-ID data","Re-ID data pruning via logit history cuts 35%, fixes labels","Averaged logit entropy: 35% re-ID data pruned, mislabels corrected","First data pruning for re-ID: 35% less data, 10x cheaper scores","Logit-history entropy prunes re-ID images, repairs mislabels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1398,"prompt_tokens":1043,"completion_tokens":355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":250}},"tokens_in":659,"tokens_out":355,"duration_ms":3953,"temperature":1.0,"reasoning_tokens":250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:22:47.565631+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the mean logit magnitude per epoch over the 12-epoch estimation window on MSMT17; if the scale drifts substantially (e.g., more than a factor of 2 between early and late epochs), recompute the soft labels after standardizing each epoch's logits to unit variance before averaging, and compare the resulting pruning ranking and accuracy-versus-pruning curves with the raw-average version. A material difference would falsify the claim that the raw average reflects the full training history.","supporting_citations":[{"cited_title":"Deep learning on a data diet: Finding important examples early in training","cited_arxiv_id":null,"evidence_quote":"Defines the EL2N and GraNd baselines and the early-training protocol (12 epochs) that the entropy score is compared against; its 20-model variant supplies the high-cost alternative."},{"cited_title":"Bag of tricks and a strong baseline for deep person re-identification","cited_arxiv_id":null,"evidence_quote":"Defines the ReID training recipe (cross-entropy plus triplet loss, 120 epochs) whose first 12 epochs yield the logit trajectories and whose settings are used for all full-model trainings."},{"cited_title":"Robust curriculum learning: from clean label detection to noisy label self-correction","cited_arxiv_id":null,"evidence_quote":"Contributes the moving-average-loss curriculum learning idea, repurposed here as an additional pruning-metric baseline."}],"review_version":1}