{"id":"d07b7e3e-b3d1-4118-befe-cfa8425579a1","arxiv_id":"1908.05569","paper_version":13,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A distance-based softmax loss trained with scaled logits, unscaled at inference, plus an entropy score, improves out-of-distribution detection without extra data, tuning, or accuracy loss.","lead":"This paper proposes replacing the standard SoftMax loss with a distance-based IsoMax loss and using the entropy of the network's outputs as a score, which improves detection of inputs the model has never seen. The method needs no extra data, no hyperparameter tuning, and no slower inference, making it an attractive low-cost reliability upgrade for trained classifiers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Entropic scale Es=10 was selected using OOD detection results, so the 'no hyperparameter validation' seamless claim is not yet established.","rationale":"The reader's weakest assumption identifies exactly the load-bearing weakness: Es=10 was chosen after observing OOD detection results, yet the paper claims no hyperparameter validation is required. This is the single most important threat to the central claim because the 'seamless' property is what differentiates the contribution from prior temperature-calibration methods such as ODIN. If Es is a hyperparameter that needs per-dataset tuning, the method is not seamless, and the headline 'state-of-the-art under severe restrictions' loses its defining feature. The paper's Figure 3 addresses within-dataset stability but not transfer to new datasets, so the reader's concern stands. I do not see an internal inconsistency or a more fundamental flaw: the empirical improvement of IsoMax+ES over SoftMax+ES is substantial and likely real, and the theoretical story, while loose, is not load-bearing. The lack of error bars is a secondary issue that would be worth addressing but is less central than the Es-selection protocol. The concrete test proposed would settle the concern cleanly by evaluating fixed Es=10 on held-out OOD sources and comparing to tuned Es; hence the reader's CONDITIONAL verdict remains appropriate and no change is needed.","tokens_in":12562,"tokens_out":13520,"duration_ms":142434,"concrete_test":"Pre-register Es=10 before any OOD evaluation. Train the same DenseNet and ResNet models on a held-out dataset (e.g., CIFAR-10, and ideally a subset of ImageNet) and evaluate on OOD sets not used in Fig. 2 or Table II (e.g., Places365, iNaturalist, or the full ImageNet). Compare fixed Es=10 AUROC against a version where Es is tuned per training/OOD pair using a validation split. If fixed Es=10 stays within 1% AUROC of the tuned Es on all held-out pairs, the selection-bias concern does not land; otherwise the seamless claim must be qualified or rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is the 'seamless' property: no hyperparameter validation, no outlier data, no accuracy drop. Sections V-A and V-B are internally inconsistent. In V-A, the authors state they 'decided to use Es=10' after observing in Fig. 2e that higher entropic scales improve OOD detection performance, using a DenseNet trained on SVHN with the same OOD sets (CIFAR10, TinyImageNet, LSUN) that later appear in Table II. In V-B, they claim 'validation of Es ... is not even possible because access to the OOD or outlier samples is not allowed.' This contradicts the selection procedure: Es was effectively tuned on OOD examples. Figure 3 shows stability of Es>=10 only on the same datasets and OOD sources, not on truly held-out data. If Es=10 does not transfer to new datasets or architectures, the method requires per-dataset tuning and fails to qualify as seamless; if it transfers, the concern is resolved. The strongest claim ('no hyperparameter tuning') therefore rests on an untested generalization, not on the current evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the SoftMax loss with the IsoMax loss, an isotropic distance-based loss with class prototypes, combined with an 'entropic scale' that is used during training and removed at inference, and an 'entropic score' (negative output entropy) for out-of-distribution detection. The authors claim this yields a seamless OOD detection method: no classification accuracy drop, no outlier data collection, no hyperparameter validation, and fast inference. They report experiments on CIFAR10, CIFAR100, and SVHN with DenseNet and ResNet architectures, showing improved OOD detection over SoftMax baselines in Table II and competitive or better numbers against non-seamless methods in Table III.","tokens_in":12770,"tokens_out":3279,"duration_ms":32158,"significance":"If the seamless claim holds, the IsoMax loss would be a valuable drop-in replacement giving free reliability improvements. The paper's strengths include a clear and practically motivated problem definition, a simple and reproducible method with released source code, comprehensive tables covering multiple datasets, architectures, OOD sources, and metrics, and an explicit separation between seamless and non-seamless evaluation. The reliance on a single entropic scale value selected with access to OOD data is, however, a load-bearing issue for the 'no hyperparameter validation' property, and the absence of error bars makes several numerical comparisons difficult to interpret.","major_comments":[{"comment":"The selection of Es=10 is performed using OOD detection results: Section V-A states that Fig. 2e, obtained with a DenseNet trained on SVHN and evaluated on the same OOD sets used later in Table II (CIFAR10, TinyImageNet, LSUN), shows that higher entropic scales improve OOD detection, and the authors 'decided to use Es=10.' Section V-B then claims that validation of Es is 'not even possible because access to the OOD or outlier samples is not allowed in seamless OOD detection.' These statements are internally inconsistent. The seamless property, which is a central claim of the paper, requires that Es be set without OOD data; the current evidence only shows that Es=10 works on the datasets and OOD sources used for its selection.","section":"V-A and V-B"},{"comment":"The stability analysis in Fig. 3, which reports that OOD performance is approximately stable for Es>=10 regardless of dataset and model, is performed on the same datasets and OOD sources that motivated the choice of Es=10. This is not a truly held-out test of the transfer of Es=10 to new datasets, architectures, or OOD sources. If Es must be tuned per dataset or architecture, the 'no hyperparameter validation' claim fails; if it transfers, the paper provides no evidence beyond the already-seen configurations. The authors should either provide a genuinely held-out validation (e.g., a dataset or model not used in any earlier figure or table) or explicitly soften the seamless claim.","section":"V-A, Fig. 3"},{"comment":"All reported metrics appear to come from single training runs, with no error bars, standard deviations, or multiple seeds. This matters concretely: in Table III, the differences between IsoMax+ES and Mahalanobis are less than 2 percentage points in most rows (e.g., DenseNet CIFAR10/SVHN AUROC 96.6 vs 97.6; ResNet CIFAR100/SVHN AUROC 85.3 vs 84.4), and in several rows IsoMax+ES is lower. Without variance estimates or significance testing, the claims that IsoMax+ES 'significantly improves' baseline performance and achieves state-of-the-art seamless results are not fully supported. The authors should report means and standard deviations over at least a few seeds, or otherwise justify why single runs suffice.","section":"Tables II and III"},{"comment":"The footnote to Table III acknowledges that validating Es using adversarial examples cannot produce significant gains and therefore the authors 'prefer to keep Es=10.' This is an admission that Es is treated as a hyperparameter that could in principle be validated. It strengthens the concern in the first major comment: the paper's own description of non-seamless methods repeatedly criticizes hyperparameter validation, yet the proposed method's single free parameter is selected using the test OOD sources. The distinction between seamless and non-seamless evaluation should be redrawn to make clear what is actually validated and on what data.","section":"V-D, Table III footnote"}],"minor_comments":[{"comment":"The phrase 'regardless of the out-of-distribution (out-dist)' is informal; consider spelling out 'out-distribution' and clarifying the caption's sentence structure.","section":"Fig. 2 caption"},{"comment":"The entry 'NA' for ACET is not explained in the caption or the text. Please state that ACET was not evaluated on DenseNet, or provide the reason.","section":"Table III caption"},{"comment":"The training details list dropout, weight decay, learning rate schedule, but do not specify whether standard data augmentation (e.g., random crops and flips for CIFAR) is used. This information is important for reproducibility.","section":"Section IV"},{"comment":"The caption and text refer to 'presumed computational cost and energy consumption rates,' but only inference delays are measured. The energy-consumption claim is inference and not direct measurement; please either measure energy or explicitly label it as an assumption.","section":"Table IV"},{"comment":"The choice to initialize all prototypes to the zero vector is stated as an observation without analysis. Adding a sentence on why zero initialization avoids the 'oscillations in performance' seen with Xavier or Kaiming initializations would help readers understand the stability behavior.","section":"Section III-C"}],"recommendation":"major_revision","confidential_remarks":"The core idea is simple and likely to be useful, and the experimental protocol is largely standard. The main obstacle to acceptance is that the 'no hyperparameter validation' property is not established, because the entropic scale Es=10 is selected using the very OOD sources used in the evaluation. This is fixable by reframing the claim or by providing a genuinely held-out validation. The lack of error bars is also a standard requirement that should be addressed. I do not see a fundamental correctness error; the issues are about evidence and framing rather than about the method's viability."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [colleague],\n\nThe headline: this is a genuinely useful baseline for OOD detection, but the 'seamless, no tuning' claim isn't fully earned because the entropic scale was chosen after looking at OOD results. The core idea is neat: replace the affine softmax logits with distances to trainable prototypes, and train with a large logit scale that is removed at inference, leaving a near-maximum-entropy distribution. As far as I know, that specific combination is new. The experiments are broad: two architectures, three training sets, three OOD sources, three metrics, plus a clean ablation separating isotropy from the entropy trick. The authors also honestly label the non-seamless comparison as unfair, and they release code. That part is all solid.\n\nThe main soft spot is exactly the one the stress-test note flags. Section V-A says they picked Es=10 because higher entropic scales improve OOD detection in Fig 2e; Section V-B then says validation of Es is 'not even possible' because OOD access is forbidden. That's inconsistent. The mitigating point is that Fig 3 shows performance is flat for Es >= 10 across their dataset/model combinations, so the chosen value isn't a tight fit. But those are the same OOD sources used in the main tables, not held-out data. If someone applies IsoMax to a new architecture or domain, we don't yet know Es=10 will work without checking. So the 'no hyperparameter validation' property is plausible, not demonstrated.\n\nSecond, there are no error bars or multiple seeds anywhere. Their margin over Mahalanobis is often under 2%, so the state-of-the-art claim would be more convincing with variance estimates. That said, for the purpose of showing a cheap improvement over the softmax baseline, the lack of error bars is a minor issue.\n\nThird, the conclusion leans on a causal claim that 'low OOD detection performance ... is due to the SoftMax loss drawbacks.' The ablation supports a correlation, not causation; and 'does not present any drawbacks compared to the SoftMax loss' is stronger than what is shown. The maximum-entropy explanation is a plausible story, not a derivation.\n\nWho is this for? Practitioners working on OOD detection. It's a free improvement in a common setting and deserves a serious referee. My recommendation: engage with it, but ask for a held-out check where Es=10 is fixed on one dataset and then applied without modification to new datasets or architectures, plus at least a few seeds to show the differences are stable.","headline":"Useful new OOD baseline, but the 'no tuning' claim rests on an entropic scale that was selected with OOD data in view.","tokens_in":13279,"tokens_out":4461,"would_cite":true,"duration_ms":38976,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that replacing the SoftMax loss with an isotropic distance-based IsoMax loss, scored by output entropy, detects out-of-distribution inputs with no accuracy drop, no extra data, and no hyperparameter tuning.","keywords":["out-of-distribution detection","IsoMax loss","entropic score","entropy maximization trick","maximum entropy principle","SoftMax loss anisotropy","seamless OOD detection","distance-based classification"],"falsifier":"Train identical architectures on a held-out dataset with SoftMax and IsoMax at $E_s=10$ and compare TNR@TPR95 on an OOD set the paper did not use; the paper predicts a consistent large gain, so a vanishing gain or an in-distribution mean entropy far below $\\log N$ would falsify the transfer claim.","tokens_in":12386,"feed_emoji":"🎯","tokens_out":11376,"duration_ms":96978,"temperature":0.7,"pith_summary":"The paper tries to establish that poor out-of-distribution (OOD) detection in deep classifiers is not a limitation of the models themselves but a side effect of the SoftMax loss, which produces anisotropic features and overconfident low-entropy outputs. It proposes replacing SoftMax with an IsoMax loss that depends only on distances to learned class prototypes, plus an entropic score that measures the negative entropy of the output probabilities. Under a self-imposed 'seamless' constraint—no accuracy drop, no outlier data, no hyperparameter tuning, and inference as fast as a normal forward pass—the paper reports large OOD detection gains on CIFAR10, CIFAR100, and SVHN with both DenseNet and ResNet. If the claim holds, it means a reliability improvement is available to any existing classifier at essentially zero cost. The key move is the entropy maximization trick: training with logits multiplied by a constant entropic scale that is removed before inference, so outputs stay high-entropy and therefore easier to separate from unknown inputs.","feed_headline":"A loss swap detects out-of-distribution images with no tuning","feed_subtitle":"Replacing the SoftMax loss with an isotropic distance loss improves OOD detection at no extra cost.","key_machinery":"The central object is the IsoMax loss, an isotropic, distance-only cross-entropy loss. With $\\|\\cdot\\|$ the Euclidean norm, $f_\\theta(x)$ the embedding, and $p_\\varphi^j$ the learnable prototype of class $j$, the training loss is $L_I(\\hat y^{(k)}|x) = -\\log \\frac{\\exp(-E_s \\|f_\\theta(x)-p_\\varphi^k\\|)}{\\sum_j \\exp(-E_s \\|f_\\theta(x)-p_\\varphi^j\\|)}$, where the entropic scale $E_s=10$ is applied during training and removed at inference. Removal turns the output distribution into the high-entropy form $\\frac{\\exp(-\\|f_\\theta(x)-p_\\varphi^i\\|)}{\\sum_j \\exp(-\\|f_\\theta(x)-p_\\varphi^j\\|)}$, and the negative entropy of that distribution is the entropic score used for detection. The mechanism aligns feature geometry with classification geometry and keeps posterior entropy near its maximum $\\log N$, so unknown inputs land far from the prototypes in a predictable way.","core_discovery":"The central claim is that IsoMax+ES—training with the IsoMax loss and scoring with the negative entropy of the output distribution—achieves state-of-the-art out-of-distribution detection under a set of constraints no current alternative meets at once: classification accuracy equal to SoftMax-trained models, no outlier or background data, no hyperparameter validation, and single-pass fast inference. In the paper's experiments on CIFAR10, CIFAR100, and SVHN with DenseNet and ResNet, replacing SoftMax with IsoMax consistently raises TNR@TPR95 and AUROC across every OOD set, with the largest gains coming from high-entropy training rather than from isotropy alone. The paper takes this as evidence that SoftMax loss anisotropy and overconfidence are the root causes of weak OOD detection, and that a distance-based loss producing near-maximum-entropy posteriors is the corrective.","pith_inferences":["Editorial inference: the fixed $E_s=10$ may need to grow with class count $N$, since the paper's saturation argument is demonstrated on small datasets; testing $E_s$ values on a 1000-class task would show whether the no-tuning property survives scale-up.","Editorial inference: because the loss is purely distance-based, the same prototype geometry could anchor open-set recognition or few-shot classifiers, tasks the paper does not study.","Editorial inference: the entropy maximization trick is tied to removing the scale at inference, and the paper reports difficulties with SoftMax and cosine logits; identifying which isotropic loss families tolerate the trick is an open follow-up."],"forward_implications":["A SoftMax-trained classifier can be turned into an OOD-aware one by retraining with IsoMax and scoring with the entropic score, with no architectural change and no extra inference cost.","The seamless task becomes a practical evaluation mode: OOD methods can be compared without accuracy-drop compensation, outlier data, or tuning-dependent results.","Non-seamless techniques such as input preprocessing, outlier exposure, and energy scores can be composed with IsoMax as a stronger starting point instead of SoftMax.","The observed correlation between higher posterior entropy and better OOD detection suggests that any training change pushing output entropy up without hurting accuracy is a candidate OOD improvement."],"supporting_citations":[{"why":"Provides the SoftMax maximum-probability baseline and the evaluation setup (datasets, metrics) that the seamless OOD detection comparison inherits.","marker":"[35]"},{"why":"Defines ODIN, the temperature-calibration plus input-preprocessing baseline whose hyperparameter validation motivates the seamless constraints.","marker":"[11]"},{"why":"Defines the Mahalanobis detector that serves as the strongest non-seamless comparison and supplies the detection accuracy metric.","marker":"[2]"},{"why":"Defines ACET, the adversarial-training baseline whose accuracy drop and slower training contrast with IsoMax.","marker":"[10]"},{"why":"Documents overconfident low-entropy predictions of modern networks, supporting the maximum-entropy critique of SoftMax.","marker":"[23]"},{"why":"Supplies the maximum entropy principle that justifies training toward high-entropy posterior distributions.","marker":"[24]"},{"why":"Defines the SoftMax loss whose anisotropy the IsoMax loss is designed to remove.","marker":"[22]"},{"why":"Shows the SoftMax loss produces separable but anisotropic features, motivating distance-based isotropic prototypes.","marker":"[1]"}],"fun_headline_variants":["IsoMax: drop-in loss for better OOD detection","One loss swap, no tuning, sharper OOD","Entropic score with IsoMax: seamless OOD","Swap SoftMax for IsoMax, keep accuracy, boost OOD","IsoMax loss: state-of-the-art OOD without extras"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The fixed entropic scale $E_s=10$, chosen after observing OOD detection results on SVHN, is assumed to transfer to all datasets and models without further validation; if a different $E_s$ is needed on a new dataset or architecture, the promised absence of hyperparameter validation fails.","fun_headline_variants_meta":{"raw":{"variants":["IsoMax: drop-in loss for better OOD detection","One loss swap, no tuning, sharper OOD","Entropic score with IsoMax: seamless OOD","Swap SoftMax for IsoMax, keep accuracy, boost OOD","IsoMax loss: state-of-the-art OOD without extras"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000125,"raw_usage":{"total_tokens":1092,"prompt_tokens":915,"completion_tokens":177,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":93}},"tokens_in":531,"tokens_out":177,"duration_ms":2503,"temperature":1.0,"reasoning_tokens":93,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:09:00.031125+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train identical architectures on a held-out dataset with SoftMax and IsoMax at $E_s=10$ and compare TNR@TPR95 on an OOD set the paper did not use; the paper predicts a consistent large gain, so a vanishing gain or an in-distribution mean entropy far below $\\log N$ would falsify the transfer claim.","supporting_citations":[{"cited_title":"A baseline for detecting misclassified and out-of-distribution examples in neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the SoftMax maximum-probability baseline and the evaluation setup (datasets, metrics) that the seamless OOD detection comparison inherits."},{"cited_title":"Enhancing the reliability of out-of- distribution image detection in neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines ODIN, the temperature-calibration plus input-preprocessing baseline whose hyperparameter validation motivates the seamless constraints."},{"cited_title":"A simple unified framework for detecting out-of-distribution samples and adversarial attacks,","cited_arxiv_id":null,"evidence_quote":"Defines the Mahalanobis detector that serves as the strongest non-seamless comparison and supplies the detection accuracy metric."},{"cited_title":"Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem,","cited_arxiv_id":null,"evidence_quote":"Defines ACET, the adversarial-training baseline whose accuracy drop and slower training contrast with IsoMax."},{"cited_title":"On calibration of modern neural networks,","cited_arxiv_id":null,"evidence_quote":"Documents overconfident low-entropy predictions of modern networks, supporting the maximum-entropy critique of SoftMax."},{"cited_title":"Information theory and statistical mechanics,","cited_arxiv_id":null,"evidence_quote":"Supplies the maximum entropy principle that justifies training toward high-entropy posterior distributions."},{"cited_title":"Large-margin softmax loss for convolutional neural networks.,","cited_arxiv_id":null,"evidence_quote":"Defines the SoftMax loss whose anisotropy the IsoMax loss is designed to remove."},{"cited_title":"A discriminative feature learning approach for deep face recognition,","cited_arxiv_id":null,"evidence_quote":"Shows the SoftMax loss produces separable but anisotropic features, motivating distance-based isotropic prototypes."}],"review_version":1}