{"id":"80afa702-fd2f-492a-a13c-d5d6bee365f8","arxiv_id":"2411.15217","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LPLgrad trains a main classifier with an auxiliary loss-prediction module and queries samples by entropy gradient norm, yielding modest accuracy gains over existing active learning baselines.","lead":"This paper combines two existing active learning ideas: training an auxiliary model to predict loss, and selecting unlabeled images by the gradient norm of their entropy. It reports small accuracy gains over baselines on several image classification datasets, though the headline claim of order-of-magnitude improvement is not supported by its own tables.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'order of magnitude' accuracy claim is contradicted by the paper's own Table III, where LPLgrad's largest gain over the best baseline is roughly 2-3 percentage points and several differences are within one standard deviation or negative.","rationale":"I read the paper as claiming a new state-of-the-art active learning method, with the central assertion being the order-of-magnitude accuracy improvement over prior methods at small label budgets. That claim is the one that must be true for the paper to have its stated value. The paper's own Table III does not support it: the largest advantages are a few accuracy points, many are within error bars, and at least one reported point is negative. This is an internal inconsistency, not a disagreement with field consensus, so it is a load-bearing concern independent of any external benchmark. The reader's weakest_assumption focused on Eq. (2), which is also a genuine reproducibility defect, but I judge the empirical contradiction as the primary failure because it directly invalidates the headline contribution. I therefore agree with the REJECT verdict while only partially agreeing on which assumption is the most load-bearing. No code was supplied despite the GitHub claim, which makes independent verification harder, but the arithmetic from the reported table is already sufficient to establish the concern.","tokens_in":13856,"tokens_out":4982,"duration_ms":50204,"concrete_test":"Recompute from Table III, for every dataset and budget point, delta = LPLgrad_accuracy - max_baseline_accuracy, and compare delta to the pooled standard error implied by the reported +/- values (treating them as standard deviations over the 5 trials). Flag all points where |delta| < 1.96*sqrt(SE_LPL^2 + SE_baseline^2). Also test the 'order of magnitude' interpretation directly: for any baseline accuracy level reached by LPLgrad, check whether a baseline reaches that same accuracy with one-tenth or fewer labeled samples. If no significant delta exceeds roughly 3 accuracy points and no 10x budget advantage exists, the order-of-magnitude claim is unsupported by the paper's own evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central load-bearing concern is that the paper's headline empirical claim is falsified by its own reported results. The abstract and conclusion state that LPLgrad 'outperforms state-of-the-art methods by order of magnitude in terms of accuracy on a small number of labeled images,' but Table III shows only small, often non-significant margins. Concretely, at the high-budget comparisons: CIFAR-10 with 6,000 labels gives LPLgrad 86.2 vs. Ent-GradNorm 84.8 (+1.4 pp); CIFAR-100 with 12,500 labels gives 62.8 vs. 61.7 (+1.1 pp); Caltech-101 with 3,500 labels gives 56.6 vs. 54.4 (+2.2 pp); and SVHN with 6,000 labels gives 93.7 vs. LearningLoss 93.9 (-0.2 pp). Most entries overlap within the reported standard deviations from five trials. An 'order of magnitude' improvement would require either roughly a tenfold reduction in error rate at fixed budget or comparable accuracy with roughly tenfold fewer labels; no result in Table III or Figures 3-5 exhibits either property. This is not an external-consensus disagreement; it is an internal inconsistency between the claim and the data in the manuscript. The Eq. (2) defect identified by the reader is also real and blocks reproduction, but the table-level contradiction is the more direct failure of the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LPLgrad, an active learning method for image classification that combines two existing ideas: an auxiliary loss-prediction module trained jointly with the main model (following Yoo and Kweon [10]), and a querying strategy that selects unlabeled samples by the gradient norm of their output entropy (following Wang et al. [13]). The authors report experiments on CIFAR-10, CIFAR-100, SVHN, Caltech-101, and a disaster-image dataset, comparing against LearningLoss, Entropy, Ent-GradNorm, Coreset, and random sampling. They claim in the abstract and conclusion that LPLgrad outperforms state-of-the-art methods by an order of magnitude in accuracy on small label budgets, with comparable training and querying times.","tokens_in":14147,"tokens_out":5717,"duration_ms":57942,"significance":"If the central claim were true, LPLgrad would be a notable advance in low-budget active learning. The paper does provide useful empirical coverage: five datasets, five trials, an ablation study, and computational overhead comparisons. However, the headline claim is contradicted by the paper's own Table III, where the largest improvement over the best baseline is about 2-3 percentage points and several differences are within one standard deviation or negative. The training objective in Eq. (2) is also internally inconsistent with the pair-wise description in Section III.B, which prevents reproduction. The contribution is an incremental combination of two published methods, and the reported evidence does not support the stated significance.","major_comments":[{"comment":"The claim that LPLgrad 'outperforms state-of-the-art methods by order of magnitude in terms of accuracy' is contradicted by the experimental results in Table III. For example, at the largest budgets: CIFAR-10 with 6,000 labels gives 86.2 vs. 84.8 for Ent-GradNorm (+1.4 pp); CIFAR-100 with 12,500 labels gives 62.8 vs. 61.7 (+1.1 pp); SVHN with 6,000 labels gives 93.7 vs. 93.9 for LearningLoss (-0.2 pp); and Caltech-101 with 3,500 labels gives 56.6 vs. 54.4 (+2.2 pp). Most entries overlap within the reported standard deviations from five trials. An 'order of magnitude' improvement would require roughly a tenfold reduction in error rate at a fixed budget or comparable accuracy with roughly tenfold fewer labels; no result in Table III or Figures 3-5 exhibits either property. The text in Section IV.B also states that LPLgrad improves Caltech-101 accuracy by 'approximately 5%,' but the largest gap in Table III for Caltech-101 is 2.2 percentage points. This is an internal inconsistency between the paper's central claim and its own reported data.","section":"Abstract and Section V (Conclusion)"},{"comment":"Equation (2) does not implement the pair-wise loss-prediction ranking described in the text. The text states that the auxiliary model is trained on P/2 data pairs, comparing the differences between predicted losses and true losses for each pair. However, Eq. (2) as written is a per-sample margin loss: l_aux = (1/P) sum_i max(0, M - d_i * (l_aux,i - l_main,i)), where d_i is defined in Eq. (3) as max(0, l_main,i). There is no pair index, no pair construction, and d_i is a scalar that cannot encode the 'direction of the margin penalty' for a pair. Additionally, l_main is defined in Eq. (1) as an average over the batch, but it is used in Eq. (2) as a per-sample ground-truth loss. This ambiguity makes the auxiliary training objective unclear and blocks reproduction of the method.","section":"Section III.B, Eq. (2)"},{"comment":"The active learning schedule is underspecified, which prevents verification of the experimental claims. Table II lists B (query batch size), R (random subset size), and A (annotation budget) for high and low budget settings, but the number of AL rounds and the relationship between these quantities are not stated. Table III reports accuracy at cumulative budgets such as 3,000, 6,000, and 9,000 labels for CIFAR-10, implying three rounds of size 1,000 each, but Algorithm 2 says 'select B samples' and Figure 3/4 plot accuracy 'across AL rounds' without defining the number of rounds or how R is used in the selection. This lack of detail makes it impossible to reproduce the experiments or interpret the claimed gains.","section":"Section IV.A and Algorithm 2"}],"minor_comments":[{"comment":"Table I lists CIFAR-100 as having 10 classes and 70,000 images; CIFAR-100 has 100 classes, and both CIFAR-10 and CIFAR-100 have 60,000 images. The table also lists CDD as having 8,677 images, while the text in Section IV.A states that CDD has 10,733 images with a train/test split of 8,591/2,142. These inconsistencies should be corrected.","section":"Table I"},{"comment":"The paper states 'Our code is available at Github' but provides no URL or repository identifier, so the claimed code release cannot be accessed.","section":"Abstract and Section IV.A"},{"comment":"The notation l_main is overloaded: Eq. (1) defines it as a mean cross-entropy loss over the entire batch, while Eq. (2) uses l_main,i as a per-sample loss. Please use distinct symbols for the per-sample and batch-level losses, and define l_aux,i explicitly as the auxiliary model's per-sample prediction.","section":"Section III.B, Eq. (1)-(2)"},{"comment":"The four subfigures in Figure 6 are all labeled '(a)' in the caption; they should be (a), (b), (c), (d) to match the text references.","section":"Figure 6 caption"},{"comment":"Table IV repeats the column headers 'Train Time Querying Time' five times, but the values appear as a single continuous block. The table should be reformatted with dataset-specific column groups so that each training and querying time is clearly associated with the correct dataset.","section":"Table IV"},{"comment":"The text claims LPLgrad improves Caltech-101 accuracy by 'approximately 5%,' but Table III shows a gain of only 2.2 percentage points over the best baseline at 3,500 labels. This quantitative claim should be aligned with the reported numbers.","section":"Section IV.B"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is not supported by its own experimental results, and the training objective in Eq. (2) is internally inconsistent with the described method. The order-of-magnitude statement in the abstract and conclusion is a load-bearing overclaim that cannot be fixed by minor edits; the empirical advantage is at best marginal and largely within error bars. The paper would require a substantial repositioning (e.g., as an incremental combination of [10] and [13] with modest gains), corrected derivations, and a complete experimental protocol before it could be considered for publication. The scope fit for an IEEE Big Data venue is acceptable, but the technical soundness and reproducibility issues are disqualifying in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: LPLgrad is a straightforward combination of two published active learning components — the loss-prediction auxiliary module from Yoo & Kweon and the entropy gradient-norm query criterion from Wang et al. The combination is new, but neither component is modified in any substantive way. The abstract's claim that it \"outperforms state-of-the-art methods by order of magnitude in terms of accuracy\" is directly contradicted by the paper's own numbers: the largest gain over the best baseline in Table III is about 2.2 percentage points on Caltech-101, and several entries are within one standard deviation or negative. That is not an order of magnitude; it is not even a consistently significant margin.\n\nThe paper does have real strengths. The empirical study is wide: five datasets, two annotation-budget regimes, an ablation, and timing comparisons. The ablation suggests the auxiliary model adds a couple of points on most datasets, which is a plausible and mildly interesting result. The writing is clear and the experimental setup is described in enough detail that a motivated researcher could re-implement the pipeline — if the loss equation were fixed.\n\nBut there is a reproducibility defect that blocks that re-implementation. Equation (2) does not match the pair-wise ranking loss described in Section III-B. As written, it penalizes per-sample underestimates of the true loss, with d_i = max(0, lmain,i) doing no pair-ordering work. The text describes forming P/2 pairs and comparing predicted losses across a pair; the equation sums over P individual samples. This is not a minor typo; the training phase depends on that objective, and the method is not reproducible as stated. Also, the code link is claimed but no repository appears in the manuscript.\n\nThe theoretical justification for gradient-norm selection is borrowed from Wang et al. [13] and asserted here without additional proof. That is fine if the empirical results were strong, but they are not strong enough to carry the weight. On CIFAR-10 at 6k labels, LPLgrad gets 86.2 vs. 84.8 for Ent-GradNorm; on CIFAR-100 at 12.5k, 62.8 vs. 61.7; on SVHN at 6k, it actually loses to LearningLoss by 0.2 points. These are incremental differences, and several are within the reported standard deviations.\n\nMy take: the idea is plausible and the experiments are real, but the central claim is false and the training objective is broken as written. I would not cite this as a reliable result, but I would send it to peer review with a clear directive to fix the equation, correct the abstract, and release the code. A revised version that makes honest claims and has a working loss function could be a modest positive contribution. As submitted, it fails on its own terms.","headline":"The paper combines two known active learning methods into one pipeline, but the 'order of magnitude' accuracy claim is contradicted by the paper's own Table III, and Eq. (2) is too inconsistent to reproduce as written.","tokens_in":14673,"tokens_out":3743,"would_cite":false,"duration_ms":38686,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that querying by the gradient norm of output entropy, after joint training with a loss-prediction auxiliary model, makes active learning for image classification substantially more label-efficient.","keywords":["active learning","uncertainty sampling","gradient norm","loss prediction","entropy","image classification","annotation budget","auxiliary model"],"falsifier":"A direct check is to re-run the CIFAR-10 experiment at $B=3{,}000$ labeled images with the same ResNet-18 and hyperparameters as Table III: if LPLgrad does not reach roughly 72.4\\% accuracy and beat LearningLoss (71.3\\%) and Ent-GradNorm (68.0\\%), the central claim fails. A second check is to hold out random sample pairs and test whether the auxiliary model's equation (2) ranks predicted losses in the intended order; if it does not, the training phase is not doing the described job.","tokens_in":13615,"feed_emoji":"🎯","tokens_out":9065,"duration_ms":89807,"temperature":0.7,"pith_summary":"LPLgrad targets the expensive-label bottleneck in image classification by using both phases of active learning more aggressively. It trains a main classifier together with an auxiliary loss-prediction model, then selects unlabeled images whose output-entropy gradients have the largest norm. On CIFAR-10, CIFAR-100, SVHN, Caltech-101, and a disaster-image dataset, the paper reports accuracy gains over existing active learning baselines at small annotation budgets, with training and querying times comparable to or better than those baselines. If true, the result matters because it would let practitioners reach high classifier accuracy while paying for far fewer human annotations.","feed_headline":"Entropy-gradient scoring makes active learning label-efficient","feed_subtitle":"LPLgrad trains a loss-prediction side model, then queries the largest entropy-gradient images, needing fewer labels.","key_machinery":"The machinery is a two-phase loop. In the training phase, a main model $w_{\\mathrm{main}}$ (ResNet-18 in the experiments) is trained together with an auxiliary model $w_{\\mathrm{aux}}$ that takes feature maps from the main model's layers and predicts each sample's loss; the total loss is $L_{\\mathrm{total}} = l_{\\mathrm{main}} + l_{\\mathrm{aux}}$. In the querying phase, each unlabeled sample's softmax probabilities are converted to an entropy $H(P(y_i|x_i))$, this entropy is treated as a loss, and its gradient $\\nabla_{w_{\\mathrm{main}}} H_i$ is computed; the sample score is the Frobenius norm $g_i = \\|\\nabla_{w_{\\mathrm{main}}} H_i\\|_F$, and the $B$ largest scores are selected for labeling. The paper follows the theoretical result cited as [13] that selecting larger gradient norms reduces an upper bound on total loss.","core_discovery":"The central claim is that the loss-prediction module should be used only during training, not during querying, and that unlabeled samples should be scored by the Frobenius norm of the gradient of their output entropy with respect to the main model's parameters. The paper argues that jointly training the auxiliary loss-prediction model with the classifier improves feature learning, while the entropy-gradient norm captures the model's uncertainty more reliably than predicted losses alone. Across all reported datasets and budgets, LPLgrad is the best or near-best method in almost every round, with its largest advantages on imbalanced datasets such as Caltech-101 and the disaster dataset. The abstract goes further, claiming an improvement by an order of magnitude in accuracy on a small number of labeled images.","pith_inferences":["If the entropy-gradient query score is doing most of the work, the auxiliary model could be removed or replaced by a cheaper feature-learning regularizer, changing the algorithm's compute profile; this is not tested in the paper.","The same entropy-gradient query rule should transfer to any differentiable classifier with probabilistic outputs, such as detection or segmentation, but that transfer is untested.","The 'order of magnitude' phrasing reflects specific budget points in the tables, not a uniform gain; readers should compare relative accuracy at each annotation budget.","A natural testable extension is to combine the gradient-norm score with a diversity term, since pure uncertainty sampling is often vulnerable to outliers; the paper does not address this."],"forward_implications":["A classifier can reach a given accuracy with a smaller annotated pool, lowering annotation cost wherever labels are scarce.","Because the querying score is just one backward pass per candidate, LPLgrad can be dropped into existing ResNet-based pipelines without ensembles or expensive retraining schedules.","The largest reported gains occur on imbalanced datasets, suggesting the rule is especially useful when some classes are rare.","The ablation tables show consistent accuracy improvements from including the auxiliary model, roughly eight percentage points across datasets and annotation budgets.","Training and querying times stay comparable to or faster than standard uncertainty methods, so the label savings do not come with a large runtime penalty."],"supporting_citations":[{"why":"Supplies the loss-prediction module and joint main/auxiliary training setup that LPLgrad adapts.","marker":"[10]"},{"why":"Provides the theoretical rationale that samples with larger gradient norms reduce an upper bound on total loss.","marker":"[13]"},{"why":"The entropy-sampling baseline that LPLgrad extends and compares against.","marker":"[5]"},{"why":"The coreset diversity-sampling baseline used as a principal comparison.","marker":"[7]"},{"why":"Provides the CIFAR-10 and CIFAR-100 datasets used for the main evaluations.","marker":"[17]"},{"why":"Provides the SVHN dataset and the margin-sampling reference.","marker":"[6]"},{"why":"Provides the imbalanced Caltech-101 dataset used to demonstrate robustness.","marker":"[18]"},{"why":"Provides the real-world disaster dataset CDD used for additional validation.","marker":"[19]"},{"why":"Supplies the random-subset technique used to build the initial labeled pool and reduce redundancy.","marker":"[21]"}],"fun_headline_variants":["LPLgrad uses entropy gradient norms for active learning queries","Auxiliary model plus entropy gradient norm reduces labeling effort","Gradient norm of entropy picks samples needing labels in AL","Training loss predictor and scoring entropy gradients boost AL","Entropy-gradient active learning outperforms with fewer labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's training-phase contribution depends entirely on equation (2) correctly implementing the pairwise ranking of predicted versus true losses described in Section III-B; if that formula does not enforce the intended ordering, the auxiliary model may not actually learn to predict loss, and the reported accuracy gains from joint training would not follow.","fun_headline_variants_meta":{"raw":{"variants":["LPLgrad uses entropy gradient norms for active learning queries","Auxiliary model plus entropy gradient norm reduces labeling effort","Gradient norm of entropy picks samples needing labels in AL","Training loss predictor and scoring entropy gradients boost AL","Entropy-gradient active learning outperforms with fewer labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1479,"prompt_tokens":1009,"completion_tokens":470,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":392}},"tokens_in":625,"tokens_out":470,"duration_ms":5583,"temperature":1.0,"reasoning_tokens":392,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:17:59.044921+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to re-run the CIFAR-10 experiment at $B=3{,}000$ labeled images with the same ResNet-18 and hyperparameters as Table III: if LPLgrad does not reach roughly 72.4\\% accuracy and beat LearningLoss (71.3\\%) and Ent-GradNorm (68.0\\%), the central claim fails. A second check is to hold out random sample pairs and test whether the auxiliary model's equation (2) ranks predicted losses in the intended order; if it does not, the training phase is not doing the described job.","supporting_citations":[{"cited_title":"Boosting active learning via improving test performance,","cited_arxiv_id":null,"evidence_quote":"Provides the theoretical rationale that samples with larger gradient norms reduce an upper bound on total loss."},{"cited_title":"Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories,","cited_arxiv_id":null,"evidence_quote":"Provides the imbalanced Caltech-101 dataset used to demonstrate robustness."},{"cited_title":"Disaster images dataset, version 1,","cited_arxiv_id":null,"evidence_quote":"Provides the real-world disaster dataset CDD used for additional validation."}],"review_version":1}