{"id":"c91601a7-b694-42f7-bfb2-86eeea2ab064","arxiv_id":"2501.12191","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new margin-based loss, HEM, trains image classifiers that are more robust to unknown and adversarial inputs and better at continual learning and segmentation than cross-entropy-trained models.","lead":"This paper introduces a new margin-based loss function, called HEM, for training image classifiers. Across many architectures and datasets, HEM improves rejection of unknown and adversarial images, continual learning, and segmentation, with only a small loss in clean accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The margin heuristic µ = sqrt(2000 / Σ s_i) is calibrated on two CIFAR-10 conditions and is never validated on ImageNet or segmentation, yet it drives HEM's key OOD/robustness advantage; if it is off-target on those tasks the central claim weakens.","rationale":"Strength: The paper is unusually careful, with 18+ architectures, 8 datasets, 5+ tasks, released code, hyperparameters fixed from CE-optimized recipes, and multiple seeds. The error-combining modification to MM is intuitive and ablated (Table 3). The clean-accuracy deficit is small and honestly reported. These are real merits. Concern: The margin is the only hyperparameter, and the paper's own Figure 10 shows OOD rejection is not insensitive to it. The chosen M=2000 is calibrated on CIFAR-10 only and extrapolated to other datasets via a sample-count scaling law that is not derived or validated. For ImageNet and segmentation the resulting margins are much smaller (≈0.04 or less) than the 0.2–2 range explored in the pilot; whether those small margins are adequate for OOD separation is unknown. The paper explicitly defers better margin selection to future work, which is an admission that the heuristic is provisional. Since the headline advantages over CE are concentrated in OOD rejection and adversarial robustness, an unsuccessful margin transfer would materially weaken the central claim. This does not invalidate the work; it means the empirical claim should be accepted only conditionally on a margin-sensitivity check in a non-pilot setting. I agree with the reader's weakest assumption. The reader's conditionality is appropriate. No verdict change needed.","tokens_in":28276,"tokens_out":5855,"duration_ms":60240,"concrete_test":"Sweep M ∈ {200, 2000, 20000} for HEM on ImageNet with ResNet50 (or, for tractability, TinyImageNet with ResNet18), and for HEM+ on Cityscapes with the FPN/ResNet34 backbone, keeping all other training settings fixed as in the paper. For each M, report clean accuracy, unknown-class-rejection AUROC, AutoAttack DAR, and (for Cityscapes) mean IoU over at least three seeds. If M=2000 is not statistically indistinguishable from the best M for AUROC/DAR/IoU, then the claimed general-purpose advantage is contingent on an unvalidated margin heuristic; the paper should then either tune M per task or qualify the claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"HEM's central claim is that it outperforms CE on OOD rejection, adversarial robustness, imbalanced learning, continual learning, and segmentation. The only hyper-parameter is the margin. Appendix B.1 sets µ = sqrt(2000 / Σ_i s_i) based on a ResNet18/CIFAR-10 pilot, using two per-class sample counts (5000 and 50). Figure 10(b) shows that unknown-class-rejection AUROC (MSP) is strongly margin-dependent: large margins create overconfident MSP scores and hurt separation, while too-small margins degrade accuracy. The heuristic's only empirical anchors are the two CIFAR-10 points; no sweep is reported for any other dataset or task. Extrapolating to ImageNet (Σ s_i ≈ 1.28e6, so µ ≈ 0.04) and to pixel-level segmentation (where Σ s_i is enormous and HEM+ margins become very small) is unsupported. The paper's own closing (§4.2, Future work) admits there may be more optimal settings for the margins and suggests learning margins as future work. If M=2000 is not near-optimal for the large-scale or pixel-level settings, HEM's advantages—especially the headline OOD-rejection and adversarial-robustness gains—could shrink or disappear.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes high error margin (HEM) loss, a variant of multiclass margin loss in which per-sample errors below the mean are discarded and the mean of the remaining above-threshold errors is used as the loss, together with HEM+, a version with per-class margins scaled by class sample counts. The authors evaluate these losses against cross-entropy and several specialized losses on standard image classification (MNIST, CIFAR-10/100, TinyImageNet, ImageNet-1k; 14 architectures), unknown-class rejection, AutoAttack rejection, long-tailed/imbalanced learning, continual learning, and semantic segmentation. The headline claim is that HEM/HEM+ is a general-purpose replacement for cross-entropy: it loses only a small amount of clean and corrupt accuracy on balanced data while matching or improving on CE across the other tasks, and it often outperforms losses specialized for those tasks.","tokens_in":28672,"tokens_out":13525,"duration_ms":148164,"significance":"The empirical scope is a genuine strength: multiple architectures, five classification datasets including ImageNet-1k, several continual-learning strategies, three segmentation benchmarks, multiple trials with reported standard deviations, and public code. If the results hold, HEM+ is practically attractive because replacing CE with it is a simple change with no extra inference cost. The main caveat is that the method's only hyperparameter, the margin, is set by a heuristic calibrated on two CIFAR-10 conditions and is never validated in the large-scale or pixel-level settings where the headline OOD and adversarial-robustness advantages are claimed. The paper also overstates the segmentation result for plain HEM, which is supported only for the HEM+ variant.","major_comments":[{"comment":"The margin is the only free parameter of the proposed loss, and it is chosen from a pilot on ResNet18/CIFAR10: M=2000 is fixed so that µ=sqrt(2000/Σ_i s_i) gives µ=0.2 for full CIFAR10 and µ=2 for the 50-samples-per-class version. Fig. 10(b) shows that unknown-class-rejection AUROC under MSP is strongly margin-dependent, yet the same M is used for ImageNet1k (where µ is about 0.04) and for pixel-level semantic segmentation without any reported sweep or validation. Because the paper's main advantages over CE are precisely in OOD rejection and adversarial robustness, the unsupported extrapolation of the margin heuristic is load-bearing. I would need to see margin-sensitivity experiments for at least the large-scale classification and segmentation settings, or some other demonstration that the conclusions are stable across a range of M values; the paper's own admission in §5.2 that 'there may be more optimal settings for the margins' makes this validation necessary.","section":"§4.2, Appendix B.1, Fig. 10"},{"comment":"The abstract and conclusion state that HEM is more effective than CE for semantic segmentation, but the results in §5.4 show that plain HEM performs comparably to CE on segmentation and that only HEM+, whose per-class margins exploit class imbalance, is clearly superior. Since HEM and HEM+ coincide only for balanced data and the segmentation datasets are imbalanced, the supported claim is about HEM+, not about HEM. The abstract and summary should be revised to distinguish the two variants; otherwise the central 'general-purpose replacement' claim overstates the evidence.","section":"Abstract, §5.4, §6"},{"comment":"The evaluation reports means and standard deviations over trials but no paired comparisons, confidence intervals, or effect sizes for the loss comparisons. Several central claims rest on average differences over a modest number of conditions—for example, the continual-learning results in Fig. 7 and the segmentation results in Fig. 8—and without a paired analysis the reader cannot tell whether the advantages are consistent across architectures and datasets or driven by a few favorable conditions. Please report paired differences relative to CE on the same architecture/dataset, with confidence intervals or a signed-rank test, for the main pairwise comparisons.","section":"§5, Figs. 3, 7, 8"}],"minor_comments":[{"comment":"Equation (8) defines the loss per sample, but the batch-level aggregation and the handling of the case where no per-sample error exceeds the mean are not specified; please state the exact implementation (including any numerical safeguard for an empty numerator/denominator), since this affects the gradients.","section":"§4.1, Eq. (8)"},{"comment":"There are several typos that should be corrected: 'using using' and 'serveral' in §5.1.1, and 'PermutedMIST' in the appendix/Figure 7 discussion.","section":"§5.1.1, §A.3"},{"comment":"The reported MM and HEM values for the third and fourth rows do not appear to follow directly from Eqs. (6)–(8) with the stated µ=0.5; please verify the numerical examples or clarify how they were computed.","section":"Table 1"},{"comment":"For HEM+, the definition of s_i in the margin formula is clear for image classification but is not explicitly defined for pixel-level segmentation; please state whether s_i denotes the number of pixels of class i in the training set.","section":"§5.2, §5.4"}],"recommendation":"major_revision","confidential_remarks":"The contribution is largely empirical, and the risk is concentrated in the margin-setting heuristic rather than in the loss definition itself. I do not see a circularity problem in using the first author's earlier DAR metric, but an independent implementation of that metric would strengthen the adversarial-robustness evaluation. The main revision is feasible: add margin-sensitivity checks for the large-scale and segmentation settings, tighten the HEM versus HEM+ wording in the claims, and add paired statistical comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: HEM is a real new loss, not a re-packaging. The specific rule—zero out per-sample errors below the sample mean, then average only the above-zero errors—is not in the cited literature, and the paper's empirical case is unusually thorough: 14 architectures, 8 datasets, multiple trials with standard deviations, and code. The headline claim that HEM beats cross-entropy on OOD rejection, adversarial detection, continual learning, and segmentation while losing only about 1% clean accuracy is supported by the reported numbers. That is a practically useful result if it holds up.\n\nThe paper also does several things right. It uses training recipes optimized for CE, so HEM is if anything at a disadvantage. It runs an ablation separating the two modifications to MM loss. It compares against LogitNorm, logit-adjusted, DICE, and MM, and reports honest failures (LA beats HEM+ on imbalanced clean accuracy). The confusions that kill MM loss—averaging many zero errors, early loss near zero—are clearly explained.\n\nNow the soft spots, in proportion. The main one is the margin heuristic. M=2000 is fixed from a pilot on ResNet18/CIFAR10 with two per-class sample counts, and the same constant is then carried to ImageNet (margin ≈ 0.04) and to pixel-level segmentation where per-class counts are enormous and margins become tiny. Figure 10(b) shows OOD AUROC with MSP is strongly margin-dependent, so this is not a cosmetic detail. The paper itself admits better margin settings may exist and suggests learning margins as future work. This is a legitimate concern, but not a fatal flaw: clean accuracy in the pilot was flat across a wide margin range, and the HEM+ segmentation results are strong even with extrapolated margins. Still, one margin sweep on a large-scale task would have removed most of the doubt.\n\nA second omission: there are no comparisons with widely used large-margin softmax losses (ArcFace, CosFace) or focal loss. Those are the natural competitors for OOD rejection and imbalanced learning, and skipping them leaves the \"general-purpose replacement for CE\" claim less bulletproof than it could be.\n\nSome task-level claims rest on a modest number of conditions—continual learning advantage in 11/16 conditions, mixed CamVid backbone results—but the averaged picture is consistent. The paper is not circular; the self-cited DAR metric and margin fitting are transparent and the headline results are independent benchmarks.\n\nWho is this for? Vision practitioners choosing a cheap loss swap, and loss-function researchers. It deserves a serious referee. I would send it to review and ask for margin validation on at least one large-scale or segmentation task, plus the missing comparisons.","headline":"A genuinely new margin-loss variant with a broad, mostly careful empirical case; the fixed margin heuristic is the one soft spot that could bite.","tokens_in":29086,"tokens_out":1926,"would_cite":true,"duration_ms":23029,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HEM, a margin-based replacement for cross-entropy loss, claims better robustness, continual learning, and segmentation at about one percent clean-accuracy cost.","keywords":["deep learning","loss functions","cross-entropy","margin-based loss","unknown class rejection","imbalanced data","continual learning","semantic segmentation"],"falsifier":"Re-run the HEM-versus-cross-entropy comparisons with the margin chosen separately for each dataset and architecture, for example by cross-validating $\\mu$ over a small grid on a held-out slice of each training set, instead of fixing it by $\\sqrt{2000/\\sum_i s_i}$. If HEM's advantages on unknown-class rejection, continual learning, or segmentation shrink or reverse under per-task margin tuning, the claim that HEM is a general-purpose replacement for cross-entropy would be refuted; if the advantages persist, the fixed heuristic is not the load-bearing part.","tokens_in":28065,"feed_emoji":"🎯","tokens_out":8586,"duration_ms":76962,"temperature":0.7,"pith_summary":"This paper proposes replacing cross-entropy loss with a margin-based loss called high error margin (HEM) for training deep classifiers. The central claim is that HEM is more effective than cross-entropy across unknown-class rejection, adversarial robustness, learning with imbalanced data, continual learning, and pixel-level semantic segmentation, while giving up only about one percent clean accuracy on balanced image classification. The motivation is that cross-entropy keeps increasing confidence and rewriting weights even after a sample is correctly classified, which produces overconfident predictions and catastrophic forgetting; a margin loss stops updating once the correct class is sufficiently ahead. The paper evaluates HEM against cross-entropy and against specialist losses such as LogitNorm, logit-adjusted, and DICE, and concludes that HEM is a general-purpose replacement for cross-entropy that performs well on all tested tasks.","feed_headline":"A margin-based loss beats cross-entropy on five vision tasks","feed_subtitle":"HEM loss improves unknown-class rejection, continual learning, and segmentation while sacrificing about 1% clean accuracy.","key_machinery":"The machinery is the HEM loss, a revised multiclass margin (hinge) loss. For a sample with correct-class logit $y_l$, each wrong logit $y_i$ contributes an error $e_i = \\max(0, y_i - y_l + \\mu_i)$, and HEM then discards every error below the mean error for that sample and averages only the above-mean errors, with the threshold mean detached from the gradient graph. This keeps the loss focused on the largest errors at the start of training while preventing it from collapsing toward zero when only a few hard errors remain late in training. The margin $\\mu = \\sqrt{2000/\\sum_i s_i}$, where $s_i$ is the number of training samples in class $i$, sets the required gap between the correct and wrong logits; HEM+ gives each class its own margin $\\mu_i = \\sqrt{2000/(n s_i)}$ so minority classes demand a larger gap. Because the loss is zero once the margin is met, training stops pushing confidence up and stops overwriting previously learned weights.","core_discovery":"The central discovery is that the training failures of the older multiclass margin loss can be fixed by how errors are combined, and the fixed loss then matches cross-entropy on accuracy while being much better on robustness. Rather than averaging all per-logit errors, HEM keeps only errors above the mean and averages those, so the loss stays large while only a few hard errors remain and it does not vanish prematurely during training. The paper reports that HEM-trained networks reject unknown classes and AutoAttack perturbations far better than cross-entropy-trained networks, learn longer-tailed data and continual-learning sequences with less forgetting, and, with class-count-adjusted margins (HEM+), segment images more accurately than cross-entropy and DICE. HEM roughly ties the specialist LogitNorm loss on unknown-class rejection, trails only logit-adjusted loss on clean imbalanced accuracy, and beats all tested alternatives on the other tasks. The paper's conclusion is that HEM+ is a general-purpose loss that almost always performs better than cross-entropy and adds no computational or development cost.","pith_inferences":["Beyond the paper's image experiments, HEM+ should transfer to any classifier that emits logits, including text and audio models, because the loss depends only on final logits and per-class sample counts; the paper notes the possibility but does not test it.","Because HEM produces zero gradient for already-correct samples, a natural and testable extension is to skip or down-weight those samples in later training epochs to cut compute; the paper lists this as future work rather than demonstrating it.","The margin heuristic is the main free parameter; learning the margins from data, or annealing them during training, could close the small clean-accuracy gap to cross-entropy while preserving the robustness gains, though the paper does not explore this."],"forward_implications":["If the central claim holds, switching the default classification loss from cross-entropy to HEM+ should improve unknown-class rejection and adversarial-attack detection with little or no clean-accuracy cost.","A single general-purpose loss could replace task-specific losses: HEM roughly matches LogitNorm on unknown-class rejection, beats logit-adjusted loss on robustness and segmentation, and beats DICE on every tested task.","Because all training recipes in the study were optimized for cross-entropy, the roughly one-percent clean-accuracy gap is the disadvantage case; tuning hyperparameters for HEM should reduce or eliminate it.","For continual learning, HEM's tendency to stop updating weights for already-learned samples should reduce catastrophic forgetting relative to cross-entropy across different anti-forgetting strategies."],"supporting_citations":[{"why":"Defines the multiclass margin (hinge) loss that HEM modifies; its averaging of errors is the failure mode HEM fixes.","marker":"Crammer and Singer, 2002"},{"why":"Proposes LogitNorm loss, the specialist baseline for unknown-class rejection that HEM matches on that task.","marker":"Wei et al., 2022"},{"why":"Proposes logit-adjusted loss, the imbalanced-data baseline that HEM+ trails only on clean accuracy.","marker":"Menon et al., 2021"},{"why":"Introduces DICE loss, the segmentation specialist baseline that HEM+ outperforms on all tasks.","marker":"Milletari et al., 2016"},{"why":"Provides Maximum Softmax Probability confidence scoring and documents the overconfidence problem HEM addresses.","marker":"Hendrycks and Gimpel, 2017"},{"why":"Supplies AutoAttack, the adversarial attack used to measure HEM's detection-based robustness.","marker":"Croce and Hein, 2020"},{"why":"Defines the Detection Accuracy Rate metric used to evaluate adversarial sample rejection.","marker":"Spratling, 2023"},{"why":"Provides the common-corruptions benchmarks used for corrupt-image accuracy comparisons.","marker":"Hendrycks and Dietterich, 2019"}],"fun_headline_variants":["Margin-based loss matches CE accuracy, beats it on robustness","HEM loss beats cross-entropy on rejection, continual learning, segmentation","Fix for margin loss yields gains on unknown classes and segmentation","New margin loss competes with CE, beats specialists on several tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that one fixed rule for setting the margin, the gap between the correct class score and the wrong-class scores, works well across every dataset, architecture, and task tested; if the ideal gap differs by task, HEM's reported advantages could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Margin-based loss matches CE accuracy, beats it on robustness","HEM loss beats cross-entropy on rejection, continual learning, segmentation","Fix for margin loss yields gains on unknown classes and segmentation","New margin loss competes with CE, beats specialists on several tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1433,"prompt_tokens":1066,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":295}},"tokens_in":682,"tokens_out":367,"duration_ms":4202,"temperature":1.0,"reasoning_tokens":295,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:24:42.622687+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the HEM-versus-cross-entropy comparisons with the margin chosen separately for each dataset and architecture, for example by cross-validating $\\mu$ over a small grid on a held-out slice of each training set, instead of fixing it by $\\sqrt{2000/\\sum_i s_i}$. If HEM's advantages on unknown-class rejection, continual learning, or segmentation shrink or reverse under per-task margin tuning, the claim that HEM is a general-purpose replacement for cross-entropy would be refuted; if the advantages persist, the fixed heuristic is not the load-bearing part.","supporting_citations":[{"cited_title":"A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifiers","cited_arxiv_id":"2308.04137","evidence_quote":"Defines the Detection Accuracy Rate metric used to evaluate adversarial sample rejection."}],"review_version":1}