REVIEW 4 major objections 5 minor 37 references
Taylor Outlier Exposure
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read TaylorOE replaces the OE regularization term with a truncated Taylor polynomial of $-\log p$, so that in-distribution contamination in a noisy auxiliary OOD dataset is suppressed and OOD detection training works without data cleaning.
desk verdict A useful extension of OE with a real mathematical error in the printed Taylor expansion — the empirical results may hold, but the mechanism as written does not. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the truncated Taylor polynomial $L_{\mathrm{toe}}(f(x)) = \frac{1}{K}\sum_{i=1}^K \sum_{n=1}^{t} \frac{(1-p_i)^n}{n!}$, which is the paper's replacement for the OE term $L_{\mathrm{oe}} = \frac{1}{K}\sum_i -\log p_i$. Taylor-expanding $-\log p$ around $p=1$ and keeping only the first $t$ terms yields a loss whose gradient is bounded and vanishes for samples with extremely confident predictions, so samples with very large logits (the ID contamination in the noisy auxiliary set) stop influencing training. The order $t$ is the control knob: low $t$ suppresses more of the high-confidence tail, high $t$ approximates OE. The argument rests on the empirical separation, established in Sec. 4.1 with a pretrained Wide Residual Network on CIFAR-10 versus 300K Random Images, that ID samples have larger $L_{\mathrm{oe}}$ values than clean OOD samples.
What would settle it
Run the Sec. 4.1 histogram analysis on an ID/OOD pair with different statistical texture, for example ImageNet as ID and a visually similar OOD set, using a pretrained ResNet; if the $L_{\mathrm{oe}}$ distribution for ID samples does not sit noticeably to the right of the OOD distribution, then a single truncation order $t$ cannot separate harmful from useful samples, and the reported FPR95 gains over OE should shrink or invert at low $\pi$.
Extended reading notes
Core claim
TaylorOE asserts that the failure mode of OE on noisy auxiliary data is the tail of the OE loss $L_{\mathrm{oe}} = \frac{1}{K}\sum_i -\log p_i$, which is dominated by classes whose predicted probability is near zero. Because in-distribution samples concentrate probability on one class, their $-\log p$ values are large across many classes, making their $L_{\mathrm{oe}}$ systematically larger than that of clean OOD samples (Sec. 4.1). Truncating the Taylor expansion $-\log p = \sum_{n=1}^{\infty} (1-p)^n/n!$ at a finite order $t$ gives $L_{\mathrm{toe}} = \frac{1}{K}\sum_i\sum_{n=1}^{t} (1-p_i)^n/n!$, a polynomial that closely matches $-\log p$ near $p=1$ but lies below it near $p=0$. With a suitable $t$, the gradient contributed by confident ID samples is suppressed or becomes zero, while clean OOD samples keep a regularization nearly identical to OE. The paper claims that, across noise ratios $\pi \in \{0.05, \ldots, 1.0\}$, TaylorOE consistently outperforms OE and WOODS (Tables 1–3), and that the regularizer can be dropped into other OE variants such as OOD resampling and OOD synthesis.
Load-bearing premise
The method's usefulness depends on the empirical claim, measured on one pretrained CIFAR-10 model, that in-distribution samples in the noisy auxiliary dataset have systematically larger negative-log-probability values than clean OOD samples; if that ordering is weak or reversed for another ID/OOD pair, the truncation suppresses useful OOD gradients as much as the harmful ID ones.
Editorial extensions
If this is right
- OE-style training becomes feasible directly on raw, uncurated auxiliary data, eliminating the cleaning step that OE currently requires.
- At high contamination ratios ($\pi=0.05$), TaylorOE improves average FPR95 over OE by about 16.7 points on CIFAR-10 and by comparable margins on CIFAR-100.
- Because $L_{\mathrm{toe}}$ is a drop-in replacement for $L_{\mathrm{oe}}$, it can be combined with other OE extensions, such as OOD resampling and OOD synthesis, without retraining or redesign.
- As the noise ratio rises (cleaner auxiliary data), the optimal truncation order $t$ increases, so TaylorOE automatically approaches plain OE performance at $\pi=1$.
- The relationship between $\pi$, the number of ID classes, and the optimal $t$ (Sec. 6) gives a practical tuning rule: use smaller $t$ for more contaminated or lower-class-count ID data, larger $t$ otherwise.
Reading between the lines
- A natural extension would be an adaptive, per-sample truncation order (or a soft gate on logits) instead of a global $t$, which could remove the reported sensitivity to $\pi$ and dataset.
- The same truncation idea could be applied to other per-sample regularizers whose magnitude is monotone in a confidence score (e.g., energy loss), turning any noisy auxiliary dataset into a usable training signal.
- The paper's key hypothesis is distributional rather than architectural; a cheap test is to measure the overlap of $L_{\mathrm{oe}}$ histograms on a new dataset pair before committing to training.
- Because TaylorOE and PolyLoss both arise from polynomial expansions of classification losses, the two strands could be unified into a single robust-OOD framework, though the paper does not make that connection.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Taylor Outlier Exposure (TaylorOE), a modification of Outlier Exposure (OE) for training with auxiliary OOD datasets contaminated by in-distribution (ID) samples. OE regularizes by minimizing the average negative log-probability (Eq. 3); TaylorOE replaces this penalty with a truncated Taylor expansion of -log p around p=1, with order t controlling how much the loss is suppressed for high-confidence, ID-like samples. The authors motivate the design by measuring that a pretrained model assigns larger Loe values to ID than to OOD samples (Sec. 4.1), and they evaluate on CIFAR-10/CIFAR-100 with six OOD test sets over noise ratios pi in {0.05,...,1.0}, comparing with OE and WOODS, and applying the idea to two OE extensions. They report that TaylorOE outperforms baselines especially at low pi, and they release code.
Significance. The practical problem (learning from noisy auxiliary OOD data without cleaning) is relevant, and the empirical study is broad: six test OOD sets, multiple contamination ratios, five seeds, and comparisons to OE, WOODS, and energy baselines. The paper also tests applicability to OOD resampling and synthesis, and it ships code. If the derivation and experiments are corrected, the idea of a polynomial regularizer with order-controlled suppression is a plausible and simple contribution. However, the current manuscript's central mechanism rests on an incorrect Taylor expansion, and until that is fixed the quantitative claims cannot be interpreted as evidence for the proposed explanation.
major comments (4)
- [Sec. 4.2, Eqs. (4)-(5)] The Taylor expansion of -log p about p=1 is -log p = sum_{n=1}^infty (1-p)^n / n, not sum (1-p)^n / n!. As printed, the infinite sum in Eq. (4) equals e^{1-p} - 1, which is bounded by e-1 approx 1.718 at p=0 and does not diverge like -log p. Consequently Ltoe in Eq. (5) does not converge to Loe as t tends to infinity, contradicting the sentence in Sec. 4.2 that the regularization "gradually converges to Loe". This is not a cosmetic typo: the polynomial coefficients determine the saturation behavior that Fig. 6 and the motivation in Sec. 4.1 rely on. The authors must correct Eqs. (4)-(5) and reconcile Fig. 5 and the convergence claim; if the experiments used coefficients 1/n!, then the reported results do not test the Taylor mechanism that the paper claims.
- [Secs. 5.1 and 6, Figs. 8-9] Under Eq. (5) with 1/n! coefficients, the terms with n>10 contribute less than about 10^{-6} for p in [0,1], so choosing t from {30,...,300} or {250,...,300} would have essentially no effect on Ltoe. Yet Figs. 8 and 9 report clear FPR95 changes across exactly these ranges, and Sec. 5.1 describes selecting t from {30,...,300}. This inconsistency suggests either the implementation uses different coefficients from those printed, or the reported t-sensitivity is due to something other than the Taylor truncation. Either way, the experiments as presented do not support the stated mechanism of order-based regularization.
- [Tables 1 and 3, Sec. 5.2] The paper reports five-seed averages without error bars or significance tests. In the low-noise regime (pi=0.9,1.0), many differences between OE and TaylorOE are extremely small (e.g., Table 1 at pi=1.0 gives AUROC 99.39 vs 99.41 for SVHN and 99.30 vs 99.31 for LSUN-R), so the abstract's claim of "consistently outperforms conventional methods" is not supported for those regimes. Adding standard deviations or a paired significance test is necessary, especially because Table 3 averages over six datasets and may hide variance.
- [Sec. 4.1, Fig. 3] The empirical premise that ID samples in the noisy auxiliary dataset have larger Loe (and hence need suppression) is demonstrated with a single pretrained WRN on CIFAR-10 against 300K Random Images. The method is then applied to CIFAR-100 and to different noise ratios and test OOD sets without showing that the separation persists in those settings. If the ordering of Loe between ID and OOD is weaker for another ID/OOD pair, the polynomial truncation may suppress useful OOD gradients rather than only noisy ID gradients. The authors should verify the separation on CIFAR-100, or at least discuss the risk, since Fig. 9 indicates that the optimal t changes with the ID dataset and class count.
minor comments (5)
- [Sec. 4.1] The section heading "Propeties of OE" should be "Properties of OE".
- [Sec. 4.2 and Figs. 4, 7] The captions and text refer to "information content" and to "Loe values" for TaylorOE; these should be clarified to distinguish the regularization value during training from the loss of the trained model.
- [Eq. (4)] The text should specify that the Taylor expansion is about p=1 for p in (0,1] and that the series diverges at p=0; this is important for the reader to understand the behavior near p=0.
- [Table 3] The pi column is repeated inside the table body, which is a formatting artifact that makes the table harder to read.
- [Figs. 8-9] The captions should state explicitly that lower FPR95 is better, as this is only implicit from the text.
Circularity Check
No circularity found; TaylorOE's reported gains are tested on held-out OOD datasets with hyperparameters chosen on a validation split, not by construction.
full rationale
I walked the derivation chain and found no step where a 'prediction' is equivalent to its inputs by construction or where the argument reduces to a self-citation. The proposed loss Ltoe in Eq. 5 is an explicit polynomial formula, and the method is evaluated on held-out OOD datasets (SVHN, LSUN, iSUN, etc.) after tuning t on a validation split, as stated in Sec. 5.1: 'the hyperparameters for each method were determined by splitting the training data into training and validation sets and tuning based on FPR95 values on the validation data.' This is standard model selection, not fitted-input-called-prediction. The motivation in Sec. 4.1 is empirical: the authors measure that ID samples in noisy OOD data tend to have larger -log p values for the pretrained model, and use that observation to motivate suppressing large per-class -log p values. That is evidence-based design, not circularity. There are no self-citations by the authors; references to Feng et al. and Leng et al. are external prior work on polynomial losses and are not load-bearing for the specific TaylorOE claim. One genuine issue exists but is not circularity: Eq. 4 states -log p = sum (1-p)^n / n!, whereas the correct Taylor expansion is sum (1-p)^n / n, so the claim that Ltoe converges to Loe as t grows is mathematically suspect. This is a correctness/verification concern, not a case of the result reducing to its own inputs; the empirical comparison could still hold for the implemented loss, but the paper should clarify or fix the coefficients. Overall, the central claim is self-contained and externally tested, so circularity score is 0.
Assumptions & free parameters
free parameters (1)
- Taylor order t =
varies: 20 for CIFAR-10 pi=0.1, 30 for CIFAR-10 pi=0.5, 300 for CIFAR-100 pi=0.1/0.5 (Figs. 8, 9)
assumptions (4)
- standard math Taylor expansion of -log p at p=1 converges with coefficients 1/n for n>=1
- domain assumption ID samples in the noisy auxiliary OOD dataset have larger Loe values than clean OOD samples under a pretrained ID model
- ad hoc to paper Truncating the polynomial at low order t suppresses high -log p values more than low ones, so ID samples are affected more than OOD samples
- domain assumption The noisy auxiliary OOD dataset follows the Huber contamination model D_noisy = (1-pi) D_in + pi D_aux_out
Cite this review
Pith. "Pith review of Taylor Outlier Exposure." pith.science (2026). https://pith.science/paper/BAXS5P6Q
@misc{pith2026241207219,
author = {Pith},
title = {Pith review of: Taylor Outlier Exposure},
year = {2026},
howpublished = {\url{https://pith.science/paper/BAXS5P6Q}},
note = {Machine review of arXiv:2412.07219}
}
read the original abstract
Out-of-distribution (OOD) detection is the task of identifying data sampled from distributions that were not used during training. This task is essential for reliable machine learning and a better understanding of their generalization capabilities. Among OOD detection methods, Outlier Exposure (OE) significantly enhances OOD detection performance and generalization ability by exposing auxiliary OOD data to the model. However, constructing clean auxiliary OOD datasets, uncontaminated by in-distribution (ID) samples, is essential for OE; generally, a noisy OOD dataset contaminated with ID samples negatively impacts OE training dynamics and final detection performance. Furthermore, as dataset scale increases, constructing clean OOD data becomes increasingly challenging and costly. To address these challenges, we propose Taylor Outlier Exposure (TaylorOE), an OE-based approach with regularization that allows training on noisy OOD datasets contaminated with ID samples. Specifically, we represent the OE regularization term as a polynomial function via a Taylor expansion, allowing us to control the regularization strength for ID data in the auxiliary OOD dataset by adjusting the order of Taylor expansion. In our experiments on the OOD detection task with clean and noisy OOD datasets, we demonstrate that the proposed method consistently outperforms conventional methods and analyze our regularization term to show its effectiveness. Our implementation code of TaylorOE is available at \url{https://github.com/fukuchan41/TaylorOE}.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Atom: Robustifying out-of-distribution detection using outlier mining
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution detection using outlier mining. In Machine Learning and Knowledge Dis- covery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part III 21, pages 430–445. Springer, 2021. 1, 2
work page 2021
-
[2]
Describing textures in the wild
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606–3613, 2014. 5
2014
-
[3]
Learning confi- dence for out-of-distribution detection in neural networks
Terrance DeVries and Graham W Taylor. Learning confi- dence for out-of-distribution detection in neural networks. arXiv preprint arXiv:1802.04865, 2018. 1, 2
arXiv 2018
-
[4]
Extremely simple activation shaping for out- of-distribution detection
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out- of-distribution detection. arXiv preprint arXiv:2209.09858,
-
[5]
V os: Learning what you don’t know by virtual outlier synthesis
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. V os: Learning what you don’t know by virtual outlier synthesis. In ICLR, 2022. 2
work page 2022
-
[6]
Dream the impossible: Outlier imagination with diffusion models
Xuefeng Du, Yiyou Sun, Jerry Zhu, and Yixuan Li. Dream the impossible: Outlier imagination with diffusion models. In NIPS, pages 60878–60901. Curran Associates, Inc., 2023. 2
work page 2023
-
[7]
Can cross entropy loss be robust to label noise? In IJCAI, pages 2206–2212, 2021
Lei Feng, Senlin Shu, Zhuoyi Lin, Fengmao Lv, Li Li, and Bo An. Can cross entropy loss be robust to label noise? In IJCAI, pages 2206–2212, 2021. 3
work page 2021
-
[8]
A baseline for detect- ing misclassified and out-of-distribution examples in neural networks
Dan Hendrycks and Kevin Gimpel. A baseline for detect- ing misclassified and out-of-distribution examples in neural networks. In ICLR, 2017. 1, 2, 4
work page 2017
Show all 37 references
-
[9]
Deep anomaly detection with outlier exposure
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In ICLR,
-
[10]
Scaling out-of-distribution detection for real-world settings
Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Joseph Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In ICLR, pages 8759–8773. PMLR,
-
[11]
On the impor- tance of gradients for detecting distributional shifts in the wild
Rui Huang, Andrew Geng, and Yixuan Li. On the impor- tance of gradients for detecting distributional shifts in the wild. In NIPS, pages 677–689. Curran Associates, Inc.,
-
[12]
Robust estimation of a location parameter
Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics: Methodology and distribution , pages 492–518. Springer, 1992. 3
1992
-
[13]
DOS: Diverse outlier sampling for out- of-distribution detection
Wenyu Jiang, Hao Cheng, MingCai Chen, Chongjun Wang, and Hongxin Wei. DOS: Diverse outlier sampling for out- of-distribution detection. In ICLR, 2024. 1, 2
2024
-
[14]
Training ood detectors in their natural habitats
Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In ICML, pages 10848–10865. PMLR, 2022. 2, 3, 5, 7
2022
-
[15]
Learning multiple layers of features from tiny images
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, Uni- versity of Toronto, Toronto, Ontario, 2009. 3, 5
2009
-
[16]
A simple unified framework for detecting out-of-distribution samples and adversarial attacks
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. InNIPS. Curran Associates, Inc., 2018. 1, 2
2018
-
[17]
Polyloss: A polynomial expansion perspective of classifica- tion loss functions
Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguelov. Polyloss: A polynomial expansion perspective of classifica- tion loss functions. In International Conference on Learning Representations, 2022. 3
2022
-
[18]
Background data resampling for outlier-aware classification
Yi Li and Nuno Vasconcelos. Background data resampling for outlier-aware classification. In CVPR, pages 13215– 13224. Computer Vision Foundation / IEEE, 2020. 1, 2, 3, 7, 8
2020
-
[19]
Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the re- liability of out-of-distribution image detection in neural net- works. In ICLR, 2018. 1, 2
2018
-
[20]
Mood: Multi- level out-of-distribution detection
Ziqian Lin, Sreya Dutta Roy, and Yixuan Li. Mood: Multi- level out-of-distribution detection. In CVPR, pages 15313– 15323, 2021. 1, 2
2021
-
[21]
Energy-based out-of-distribution detection
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NIPS, pages 21464–21475. Curran Associates, Inc., 2020. 1, 5, 6, 7
2020
-
[22]
Peer loss functions: Learning from noisy labels without knowing noise rates
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In ICML, pages 6226–6236. PMLR, 2020. 3
2020
-
[23]
SGDR: Stochastic gradi- ent descent with warm restarts
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradi- ent descent with warm restarts. In ICLR, 2017. 5
2017
-
[24]
Poem: Out-of- distribution detection with posterior sampling
Yifei Ming, Ying Fan, and Yixuan Li. Poem: Out-of- distribution detection with posterior sampling. In ICML, pages 15650–15665. PMLR, 2022. 1, 2
2022
-
[25]
Reading digits in natural images with unsupervised feature learning
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learn- ing, page 7. Granada, Spain, 2011. 5
2011
-
[26]
Gradient-regularized out-of- distribution detection
Sina Sharifi, Taha Entesari, Bardia Safaei, Vishal M Pa- tel, and Mahyar Fazlyab. Gradient-regularized out-of- distribution detection. In ECCV, pages 459–478. Springer,
-
[27]
Dice: Leveraging sparsification for out-of-distribution detection
Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV, pages 691–708. Springer, 2022. 1, 2
2022
-
[28]
Non- parametric outlier synthesis
Leitian Tao, Xuefeng Du, Jerry Zhu, and Yixuan Li. Non- parametric outlier synthesis. In ICLR, 2023. 2
2023
-
[29]
Out-of- distribution detection with implicit outlier transformation
Qizhou Wang, Junjie Ye, Feng Liu, Quanyu Dai, Marcus Ka- lander, Tongliang Liu, Jianye HAO, and Bo Han. Out-of- distribution detection with implicit outlier transformation. In ICLR, 2023. 1, 2
2023
-
[30]
Mitigating neural network overconfidence with logit normalization
Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML, pages 23631–23644. PMLR, 2022. 1, 2
2022
-
[31]
Turkergaze: Crowdsourcing saliency with webcam based eye tracking
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkel- stein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.06755, 2015. 2, 5
2015 arXiv
-
[32]
Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365, 2015. 5 9
2015 arXiv
-
[33]
Wide residual net- works
Sergey Zagoruyko and Nikos Komodakis. Wide residual net- works. In BMVC. BMV A Press, 2016. 3, 5
2016
-
[34]
Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments
Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In WACV, pages 5531–5540, 2023. 1
2023
-
[35]
Generalized cross entropy loss for training deep neural networks with noisy labels
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In NIPS. Curran Associates, Inc., 2018. 3
2018
-
[36]
Places: A 10 million image database for scene recognition
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 40(6):1452–1464, 2017. 5
2017
-
[37]
Diversified outlier ex- posure for out-of-distribution detection via informative ex- trapolation
Jianing Zhu, Yu Geng, Jiangchao Yao, Tongliang Liu, Gang Niu, Masashi Sugiyama, and Bo Han. Diversified outlier ex- posure for out-of-distribution detection via informative ex- trapolation. In NIPS, pages 22702–22734. Curran Asso- ciates, Inc., 2023. 1, 2, 3, 7, 8 10
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.