Pith. sign in

REVIEW 4 major objections 5 minor 54 references

Optimizing Latent Dimension Allocation in Hierarchical VAEs: Balancing Attenuation and Information Retention for OOD Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a dataset-specific optimal latent compression ratio r* exists for hierarchical VAEs under a fixed latent budget, and that tuning to it improves out-of-distribution detection.

desk verdict The empirical sweep makes a credible design point, but the existence proof for r* is invalid; this needs major revision. read the letter →

arxiv 2506.10089 v1 pith:GKL33CQV submitted 2025-06-11 cs.LG

Dane Williamson , Yangfeng Ji , Matthew Dwyer This is my paper
classification cs.LG
keywords out-of-distributiondetectionhierarchicalvariationalautoencoderlatentdimensionallocationgeometriccompressionratioinformationbottleneckmutualposteriorcollapse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that how a hierarchical VAE (a VAE whose latent variables are split into several layers of abstraction) divides a fixed latent-dimension budget among its layers is not a neutral design choice but a tunable quantity with a real optimum. It restricts allocations to a geometric progression governed by a single compression ratio r and proves, under mild regularity assumptions on an unknown per-layer utility function, that an optimal ratio r* exists. Empirically, the paper finds distinct dataset-specific optima—0.75 for FashionMNIST, 0.5 for MNIST, and 0.25 for CIFAR-10—and shows that tuning to r* improves AUROC, AUPRC, FPR80, and FPR95 over baseline allocations without changing the model size. This matters because OOD detection is safety-critical, and the proposed tuning offers a cheap, architecture-level way to improve it without new training objectives.

What carries the argument

The central object is the latent dimension ratio r with the geometric allocation formula li = b(1-r)$r^{{i-1}}$/(1-r^N), which compresses each successive latent layer by the same factor while exactly consuming the fixed budget b. This one-parameter family turns architecture search into a one-dimensional optimization over F(r) = sum_i f(li), and the proof that an optimum exists uses continuity and compactness of the domain (0,1] together with boundedness of f. The same formula also generates the layer sizes used in all experiments, so the theoretical and empirical claims rest on the same mechanism.

What would settle it

Train every integer allocation ($\ell^1$,$\ell^2$,l3) with $\ell^1$ ≥ $\ell^2$ ≥ l3 ≥ 1 and $\ell^1$ + $\ell^2$ + l3 = 32 on FashionMNIST and test on MNIST, KMNIST, and notMNIST; if any non-geometric triple beats the reported r* = 0.75 allocation (14→10→8) on AUROC or FPR95, then r* is not optimal in the full allocation space. A second check: with any explicit smooth concave f, maximize F(r) symbolically and compare its argmax to the empirical r*; disagreement shows that the claimed optimum depends on the unknown f.

Watch

Extended reading notes

Core claim

For a hierarchical VAE with N latent layers and a fixed total latent budget b, the paper parameterizes layer dimensionalities as li = b(1-r)$r^{{i-1}}$/(1-r^N), so that the entire architecture is described by one ratio r. Defining total efficacy as F(r) = sum_i f(li) for an unknown utility function f, the paper claims that under continuity, saturation for large li, and a finite limit as li goes to 0, F attains its maximum at some r* in (0,1]. Empirically, the paper reports that r* is dataset-specific and consistently outperforms baselines: FashionMNIST r* = 0.75 (14→10→8), MNIST r* = 0.5 (18→9→5), and CIFAR-10 r* = 0.25 (170→43→11), with gains across four OOD metrics and multiple OOD pairings per in-distribution dataset. Mutual-information estimates I(X; Z3) are low at r = 0.1, matching information-loss failures, and high but increasingly variable as r approaches 1, matching attenuation and overfitting, which the paper takes as evidence for the claimed tradeoff.

Load-bearing premise

The argument assumes the best latent allocation is a single compression factor shared by every adjacent pair of layers, and that the unknown per-layer utility function is continuous, saturates for large layers, and has a finite limit as layer size goes to zero.

Editorial extensions

If this is right

  • Under a fixed latent budget, sweeping r and selecting the dataset-specific r* improves OOD detection metrics over baseline allocations without increasing model size.
  • The optimal r* transfers across different OOD pairings for a given in-distribution dataset, so practitioners can tune once per dataset rather than per OOD task.
  • Extreme compression (r = 0.1) consistently causes information loss and poor detection, while relaxed compression (r near 1) raises mutual information without improving detection, so both extremes should be avoided.
  • Non-geometric heuristic allocations such as expand-then-compress, stable, and compress-then-expand underperform optimized geometric allocations, indicating that structured compression is doing real work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the existence proof applies to any continuous bounded f, so the theorem itself is broad but nearly content-free; the empirical r* values are the substantive result, and they are established only within the one-parameter geometric family.
  • My inference: r* likely tracks the difficulty of the OOD task—for far-OOD pairs almost any allocation succeeds, so the practical gains should concentrate on near-OOD discrimination.
  • My inference: the mutual-information estimator samples non-target latents from the prior, so the reported I(X; Z3) measures information flow through one specific generative path; a different estimator could change the attenuation picture.
  • My inference: relaxing the constant-r constraint to per-layer ratios or a learned allocation could improve performance further, which the paper itself leaves open.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript studies how the distribution of a fixed total latent dimensionality across layers of a hierarchical VAE affects out-of-distribution detection. It restricts allocations to a geometric progression with a single compression ratio r, defines a utility F(r)=Σ_i f(l_i) for an unspecified layer utility f, and claims (Proposition 1) that an optimal ratio r* exists on (0,1]. The empirical part sweeps r ∈ {0.1, 0.25, 0.5, 0.75} for FashionMNIST, MNIST, and CIFAR10 in-distribution sets, evaluates four OOD detection metrics with several OOD counterparts, and reports that dataset-specific r* values outperform the tested baseline and control allocations. It also estimates I(X;Z3) to motivate an information-bottleneck interpretation.

Significance. The empirical protocol has notable strengths: fixed latent budgets and depth for most configurations, multiple OOD pairings per in-distribution set, standard metrics with standard deviations, and non-geometric control configurations. If the results are reproducible, the paper would offer simple practical guidance for HVAE architecture design. However, the central theoretical result is not established: the proof of Proposition 1 is invalid (and the proposition is false under the stated assumptions), the claimed cross-pairing generalization of r* is circular given the way r* is selected, and a fixed-budget inconsistency in the CIFAR10 setting undermines the controlled comparison. These are load-bearing issues, so the paper requires substantial revision before the contributions can be accepted.

major comments (4)
  1. [Section 3.3 and Section 8, Eqs. (17)-(20) and (27)-(36)] The proof of Proposition 1 is invalid. The interval (0,1] is not compact, so the Extreme Value Theorem cannot be applied; in addition, the expression for li in Eq. (10) has denominator 1-r^N and is undefined at r=1, so F is not defined on the stated domain without a limit convention. The failure is not a technicality: under the paper's own assumptions the maximum can fail to exist. With N=2, b=2, and f(l)=(2-l)^2, the assumptions of Proposition 1 hold (f is continuous, f'(l)≤0 for l∈[1,2], and lim_{l→0} f(l)=4), but F(r)=4(1+r^2)/(1+r)^2 is strictly decreasing on (0,1], so its supremum 4 is approached as r→0+ and never attained. Proposition 1 as stated is therefore false, and the claimed existence guarantee must either be corrected with additional assumptions or removed.
  2. [Section 5.2 and Section 6.1] The claim that r* 'generalizes across OOD pairings' is circular. Section 5.2 defines r* for each in-distribution dataset as the ratio yielding the highest average performance across exactly the OOD pairings listed in the same section, and Section 6.1 then reports that this same r* is 'consistently optimal across its corresponding OOD pairings.' That statement is true by construction. A non-circular evaluation would select r* on one set of OOD datasets and test it on held-out OOD datasets, or would clearly separate model selection from evaluation; without such a split, the cross-pairing generalization claim is unsupported.
  3. [Section 4.3, Figure 5, and Table 3] The fixed-budget constraint for CIFAR10 is not met by the reported configurations. The text states b=228 for CIFAR10, but every compressed configuration in Figure 5 and Table 3 sums to 224 (e.g., 128+64+32, 170+43+11, 202+20+2, 97+73+54), and the control configuration 56→80→98 sums to 234. Since the paper's central comparison is explicitly 'under a fixed latent budget,' this discrepancy must be resolved: either the actual budget is 224 and the text is wrong, or the displayed configurations violate the stated budget and the corresponding results need to be rerun or re-reported.
  4. [Section 3.5 and Figure 6] The mutual information estimator is presented without any error analysis or validation, yet it is used to draw the paper's information-theoretic conclusions. Equation (23) replaces the intractable marginal log-likelihood with the ELBO, and the non-target latents are sampled from their priors; these substitutions introduce an unknown bias that is not bounded or compared against a tractable ground-truth case. Before the RQ3 claims about information loss and attenuation can be accepted, the paper should quantify the estimator's bias (e.g., on synthetic settings where I(X;Zi) is known) or substantially weaken the interpretation.
minor comments (5)
  1. [Sections 1 and 2.2] Several typos interrupt the exposition, including 'exhibitsuboptimal' and 'which limits. the model'; a careful proofread is needed.
  2. [Figure 2 caption] The caption says 'Only the top latent z3 is inferred from the posterior; lower latents are sampled from the prior,' which appears to contradict the hierarchical inference described in Eq. (2) and Figure 1; please clarify which inference scheme is used in the reconstructions.
  3. [Section 13.1.2] The notation 'dVar(I(X;Z3))' is not defined; specify whether this is the variance across runs or across samples.
  4. [Table 2] Table 2 contains malformed entries (e.g., '1.±0.' and truncated Omniglot columns) that must be repaired before publication.
  5. [Section 3.3, Eq. (13)] The paper should state explicitly that Proposition 1 concerns optimality only within the one-parameter geometric family defined by Eq. (10); the current wording 'optimal latent compression ratio' can be misread as a global optimum over all allocations.

Circularity Check

2 steps flagged · score 6.0 of 10

The claimed 'generalization across OOD pairings' is just the criterion used to select r*, and the existence proof for r* restates the definition of argmax.

  1. fitted input called prediction [Section 5.2 (RQ2) and Section 6.1 (Analysis); contribution bullet in Section 1]
    "We define r∗ as the compression ratio that yields the highest average performance across OOD pairings for a given ID dataset. ... Across all configurations evaluated, the empirically identified optimal ratio r∗ for a given ID dataset is consistently optimal across its corresponding OOD pairings. This suggests that r∗ captures dataset-specific structure that generalizes across distributional shifts."

    The quantity presented as a 'generalization' is the same quantity used to select r*. Section 5.2 defines r* as the best average over the OOD pairings, and Section 6.1 then reports that r* is optimal for those very pairings. No held-out OOD pairing or second-stage evaluation is introduced. The claim that the optimal configuration 'generalizes across OOD pairings' is therefore a restatement of the selection criterion, not an independent prediction.

  2. self definitional [Section 3.3, Eq. (13)/(17), Proposition 1; Section 3.4]
    "The optimal latent dimension ratio r∗ is then given by: r∗ = arg max_{r∈(0,1]} F (r). ... Proposition 1. An optimal latent compression ratio r∗ exists that maximizes total efficacy F (r) under a fixed budget b. ... Since F (r) is continuous on (0, 1], there exists r∗ ∈ (0, 1] such that F (r∗) = max_{r∈(0,1]} F (r)."

    F(r) is not an independently specified theoretical quantity; Section 3.4 says 'the latent utility function f is not known in closed form, we determine r∗ empirically' and 'We define the empirical optimum r∗ as the value that yields the best tradeoff.' The proposition thus proves only that an argmax of the chosen objective exists, which is exactly what 'arg max' means. The conclusion r* exists carries no content beyond the definition of r* as the best configuration in the search, so the 'theoretical guarantee' is a tautology relative to the empirical objective.

full rationale

The paper's empirical core is a legitimate grid sweep: fixed budgets, fixed depth, and measured OOD metrics across four r values and several control architectures. Those measurements are not circular. The circularity is concentrated in two places. First, the 'generalization across OOD pairings' claim (abstract, Section 1 contribution, Section 6.1) is drawn from the same OOD pairings that were averaged to select r* in Section 5.2; no independent OOD pairings or held-out evaluation are used, so the generalization statement reduces to the selection rule. Second, Proposition 1's existence proof is self-definitional: r* is defined as argmax of an unknown empirical function F, and the proof shows that an argmax exists (modulo the non-compactness flaw, which is a correctness issue rather than a circularity issue). That theorem adds no predictive information about which allocation is best. The paper does not rely on load-bearing self-citations; its references are to external prior work. Overall, the central 'optimal r*' narrative partially reduces to construction, while the underlying comparison data remain informative. Score 6 reflects the partial circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on an unspecified utility function f and a one-parameter geometric family of allocations. The proof of existence assumes properties of f that are not derived from the model, and the empirical search defines r* on the evaluation data. These are the main non-standard premises.

free parameters (4)
  • compression ratio r = r*=0.75 (FashionMNIST), 0.5 (MNIST), 0.25 (CIFAR10)
    Grid-searched over {0.1,0.25,0.5,0.75}; the 'optimal' value is selected by OOD detection metrics on the evaluation pairings.
  • latent budget b = 32 (grayscale), 228 (CIFAR10)
    Fixed from prior work [8]; a design choice, not optimized.
  • number of layers N = 3
    Fixed for all experiments, following prior work.
  • rounding rule for integer dimensions = round to nearest integer, adjust highest layers
    Implementation note in Section 3.1; affects exact layer sizes and can break the nominal geometric progression.
assumptions (5)
  • ad hoc to paper The latent utility function f is continuous in li
    Assumed in Proposition 1 without proof; f is never specified and could be discontinuous due to training noise.
  • ad hoc to paper f saturates: derivative of f with respect to li is at most 0 for sufficiently large li
    Assumed to ensure a maximum; contradicts the usual initial increase in performance with capacity.
  • ad hoc to paper The limit of f(li) as li goes to 0 equals gamma (an information loss floor)
    Assumed as a lower bound; no evidence for a specific limit.
  • domain assumption The optimal allocation lies in the one-parameter geometric family li = b(1-r)r^(i-1)/(1-r^N)
    The search is restricted to constant ratio r across layers; non-geometric allocations are only partially tested via a few controls.
  • domain assumption The mutual information estimator in Section 3.5 is a reliable approximation of I(X;Zi)
    Uses the ELBO as a surrogate for log p(x) and samples non-target latents from the prior; no bias or consistency analysis is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Latent Dimension Allocation in Hierarchical VAEs: Balancing Attenuation and Information Retention for OOD Detection." pith.science (2026). https://pith.science/paper/GKL33CQV

@misc{pith2026250610089,
  author       = {Pith},
  title        = {Pith review of: Optimizing Latent Dimension Allocation in Hierarchical VAEs: Balancing Attenuation and Information Retention for OOD Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GKL33CQV}},
  note         = {Machine review of arXiv:2506.10089}
}
abstract

Out-of-distribution (OOD) detection is a critical task in machine learning, particularly for safety-critical applications where unexpected inputs must be reliably flagged. While hierarchical variational autoencoders (HVAEs) offer improved representational capacity over traditional VAEs, their performance is highly sensitive to how latent dimensions are distributed across layers. Existing approaches often allocate latent capacity arbitrarily, leading to ineffective representations or posterior collapse. In this work, we introduce a theoretically grounded framework for optimizing latent dimension allocation in HVAEs, drawing on principles from information theory to formalize the trade-off between information loss and representational attenuation. We prove the existence of an optimal allocation ratio $r^{\ast}$ under a fixed latent budget, and empirically show that tuning this ratio consistently improves OOD detection performance across datasets and architectures. Our approach outperforms baseline HVAE configurations and provides practical guidance for principled latent structure design, leading to more robust OOD detection with deep generative models.

Figures

Figures reproduced from arXiv: 2506.10089 by the authors.

Figure 1
Figure 1. The encoder, qϕ(z|x), and decoder, pθ(x, z), models for a 3 level (L) HVAE. Solid lines represent stochastic dependencies in the data generation process, dashed lines indicate skip connections which assist in alleviating posterior collapse. In particular, we introduce a structured allocation bias that encourages meaningful information flow through all latent levels, which can help reduce the severity of posterior co… view at source ↗
Figure 2
Figure 2. Reconstructed CIFAR10 samples under varying latent compression ratios [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The optimal configuration for FashionMNIST-In is [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The optimal configuration for MNIST-In is [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: The configuration, r ∗ = 0.25 returns the optimal configuration of 170 → 43 → 11. This configuration is optimal across all metrics for CIFAR10 for all pairings except one. This is in the case of FPR95 for SVHN. It is worth noting that for CelebA out, the compressed con…
Figure 6
Figure 6. Figure 6: Mutual Information as r is varied in compressed configurations. Here we can [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 22 canonical work pages

  1. [1]

    W. Liu, X. Wang, J. D. Owens, Y. Li, Energy-based out-of-distribution de- tection, CoRR abs/2010.03759 (2020). arXiv:2010.03759. URL https://arxiv.org/abs/2010.03759

    arXiv 2020

  2. [2]

    Y. Hsu, Y. Shen, H. Jin, Z. Kira, Generalized ODIN: detecting out-of- distribution image without learning from out-of-distribution data, CoRR abs/2002.11297 (2020). arXiv:2002.11297. URL https://arxiv.org/abs/2002.11297 25

    work page Pith review arXiv 2020

  3. [3]

    Y. Song, N. Sebe, W. Wang, Rankfeat: Rank-1 feature removal for out-of- distribution detection (2022). arXiv:2209.08590. URL https://arxiv.org/abs/2209.08590

    work page Pith review arXiv 2022

  4. [4]

    Hendrycks, K

    D. Hendrycks, K. Gimpel, A baseline for detecting misclassified and out- of-distribution examples in neural networks, CoRR abs/1610.02136 (2016). arXiv:1610.02136. URL http://arxiv.org/abs/1610.02136

    arXiv 2016

  5. [5]

    D. P. Kingma, M. Welling, Auto-encoding variational bayes (2013). doi: 10.48550/ARXIV.1312.6114. URL https://arxiv.org/abs/1312.6114

    arXiv doi:10.48550/arxiv.1312.6114 2013

  6. [6]

    Vahdat, J

    A. Vahdat, J. Kautz, Nvae: A deep hierarchical variational autoencoder (2021). arXiv:2007.03898. URL https://arxiv.org/abs/2007.03898

    arXiv 2021

  7. [7]

    C. K. Sønderby, T. Raiko, L. Maaløe, S. K. Sønderby, O. Winther, Ladder variational autoencoders (2016). arXiv:1602.02282. URL https://arxiv.org/abs/1602.02282

    arXiv 2016

  8. [8]

    J. D. Havtorn, J. Frellsen, S. Hauberg, L. Maaløe, Hierarchical vaes know what they don’t know, CoRR abs/2102.08248 (2021). arXiv:2102.08248. URL https://arxiv.org/abs/2102.08248

    arXiv 2021

Show all 54 references
  1. [9]

    J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. A. DePristo, J. V. Dil- lon, B. Lakshminarayanan, Likelihood ratios for out-of-distribution detection (2019). arXiv:1906.02845. URL https://arxiv.org/abs/1906.02845

    2019 arXiv

  2. [10]

    Nalisnick, A

    E. Nalisnick, A. Matsukawa, Y. W. Teh, D. Gorur, B. Lakshminarayanan, Do deep generative models know what they don’t know? (2019). arXiv: 1810.09136

    2019 arXiv

  3. [11]

    Y. Li, C. Wang, X. Xia, T. Liu, x. miao, B. An, Out-of-distribution de- tection with an adaptive likelihood ratio on informative hierarchical vae, in: S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, A. Oh (Eds.), Ad- vances in Neural Information Processing Systems, Vol...

    2022

  4. [12]

    S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. J´ ozefowicz, S. Bengio, Generating sentences from a continuous space, CoRR abs/1511.06349 (2015). arXiv:1511.06349. URL http://arxiv.org/abs/1511.06349

    2015 arXiv

  5. [13]

    D. P. Kingma, T. Salimans, M. Welling, Improving variational inference with inverse autoregressive flow, CoRR abs/1606.04934 (2016). arXiv: 1606.04934. URL http://arxiv.org/abs/1606.04934

    2016 arXiv

  6. [14]

    X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, P. Abbeel, Variational lossy autoencoder, CoRR abs/1611.02731 (2016). arXiv:1611.02731. URL http://arxiv.org/abs/1611.02731

    2016 arXiv

  7. [15]

    A. B. Dieng, Y. Kim, A. M. Rush, D. M. Blei, Avoiding latent variable collapse with generative skip models (2019). arXiv:1807.04863

    2019 arXiv

  8. [16]

    Brock, J

    A. Brock, J. Donahue, K. Simonyan, Large scale gan training for high fidelity natural image synthesis (2019). arXiv:1809.11096. URL https://arxiv.org/abs/1809.11096

    2019 arXiv

  9. [17]

    J. Ho, A. Jain, P. Abbeel, Denoising diffusion probabilistic models, CoRR abs/2006.11239 (2020). arXiv:2006.11239. URL https://arxiv.org/abs/2006.11239

    2020 arXiv

  10. [18]

    Maaløe, M

    L. Maaløe, M. Fraccaro, V. Li´ evin, O. Winther, Biva: A very deep hierarchy of latent variables for generative modeling (2019). arXiv:1902.02102

    2019 arXiv

  11. [19]

    Sinha, A

    S. Sinha, A. B. Dieng, Consistency regularization for variational auto- encoders, CoRR abs/2105.14859 (2021). arXiv:2105.14859. URL https://arxiv.org/abs/2105.14859

    2021 arXiv

  12. [20]

    Bozkurt, B

    A. Bozkurt, B. Esmaeili, D. H. Brooks, J. G. Dy, J. van de Meent, Evaluating combinatorial generalization in variational autoencoders, CoRR abs/1911.04594 (2019). arXiv:1911.04594. URL http://arxiv.org/abs/1911.04594

    2019 arXiv

  13. [21]

    H. Xu, D. Luo, R. Henao, S. Shah, L. Carin, Learning autoencoders with relational regularization, CoRR abs/2002.02913 (2020). arXiv:2002.02913. URL https://arxiv.org/abs/2002.02913 27

    2020 arXiv

  14. [22]

    T. Ma, J. Chen, C. Xiao, Constrained generation of semantically valid graphs via regularizing variational autoencoders, CoRR abs/1809.02630 (2018). arXiv:1809.02630. URL http://arxiv.org/abs/1809.02630

    2018 arXiv

  15. [23]

    H. Wu, M. Flierl, Vector quantization-based regularization for autoencoders, Proceedings of the AAAI Conference on Artificial Intelligence 34 (04) (2020) 6380–6387. doi:10.1609/aaai.v34i04.6108. URL https://ojs.aaai.org/index.php/AAAI/article/view/6108

    2020 doi

  16. [24]

    Serr` a, D.´Alvarez, V

    J. Serr` a, D.´Alvarez, V. G´ omez, O. Slizovskaia, J. F. N´ u˜ nez, J. Luque, Input complexity and out-of-distribution detection with likelihood-based generative models, CoRR abs/1909.11480 (2019). arXiv:1909.11480. URL http://arxiv.org/abs/1909.11480

    2019 arXiv

  17. [25]

    Z. Wang, B. Dai, D. Wipf, J. Zhu, Further analysis of outlier detection with deep generative models (2020). arXiv:2010.13064. URL https://arxiv.org/abs/2010.13064

    2020 arXiv

  18. [26]

    L. H. Zhang, M. Goldstein, R. Ranganath, Understanding failures in out- of-distribution detection with deep generative models, CoRR abs/2107.06908 (2021). arXiv:2107.06908. URL https://arxiv.org/abs/2107.06908

    2021 arXiv

  19. [27]

    K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, CoRR abs/1512.03385 (2015). arXiv:1512.03385. URL http://arxiv.org/abs/1512.03385

    2015 arXiv

  20. [28]

    Krizhevsky, I

    A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, Commun. ACM 60 (6) (2017) 84–90. doi: 10.1145/3065386. URL https://doi.org/10.1145/3065386

    2017 doi

  21. [29]

    Lecun, Y

    Y. Lecun, Y. Bengio, G. Hinton, Deep learning, Nature 521 (7553) (2015) 436–444, publisher Copyright: © 2015 Macmillan Publishers Limited. All rights reserved. doi:10.1038/nature14539

    2015 doi

  22. [30]

    Recanatesi, M

    S. Recanatesi, M. Farrell, M. Advani, T. Moore, G. Lajoie, E. Shea-Brown, Dimensionality compression and expansion in deep neural networks, CoRR abs/1906.00443 (2019). arXiv:1906.00443. URL http://arxiv.org/abs/1906.00443 28

    2019 arXiv

  23. [31]

    K. G. Wilson, Renormalization group and critical phenomena. i. renormaliza- tion group and the kadanoff scaling picture, Phys. Rev. B 4 (1971) 3174–3183. doi:10.1103/PhysRevB.4.3174. URL https://link.aps.org/doi/10.1103/PhysRevB.4.3174

    1971 doi

  24. [32]

    L. P. Kadanoff, Scaling laws for ising models near Tc, Physics Physique Fizika 2 (1966) 263–272. doi:10.1103/PhysicsPhysiqueFizika.2.263. URL https://link.aps.org/doi/10.1103/PhysicsPhysiqueFizika.2. 263

    1966 doi

  25. [33]

    Mehta, D

    P. Mehta, D. J. Schwab, An exact mapping between the variational renormal- ization group and deep learning (2014). arXiv:1410.3831. URL https://arxiv.org/abs/1410.3831

    2014 arXiv

  26. [34]

    Tishby, F

    N. Tishby, F. C. Pereira, W. Bialek, The information bottleneck method (2000). arXiv:physics/0004057. URL https://arxiv.org/abs/physics/0004057

    2000 arXiv

  27. [35]

    Shamir, S

    O. Shamir, S. Sabato, N. Tishby, Learning and generalization with the information bottleneck, Theoretical Computer Science 411 (29) (2010) 2696–2711, algorithmic Learning Theory (ALT 2008). doi:https://doi.org/10.1016/j.tcs.2010.04.006. URL https://www.sciencedirect.com/scienc...

    2010 doi

  28. [36]

    A. A. Alemi, I. Fischer, J. V. Dillon, K. Murphy, Deep variational information bottleneck, CoRR abs/1612.00410 (2016). arXiv:1612.00410. URL http://arxiv.org/abs/1612.00410

    2016 arXiv

  29. [37]

    L. Ruff, R. A. Vandermeulen, N. G¨ ornitz, A. Binder, E. M¨ uller, K. M¨ uller, M. Kloft, Deep semi-supervised anomaly detection, CoRR abs/1906.02694 (2019). arXiv:1906.02694. URL http://arxiv.org/abs/1906.02694

    2019 arXiv

  30. [38]

    H. Xiao, K. Rasul, R. Vollgraf, Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, CoRR abs/1708.07747 (2017). arXiv:1708.07747. URL http://arxiv.org/abs/1708.07747

    2017 arXiv

  31. [39]

    Deng, The mnist database of handwritten digit images for machine learning research, IEEE Signal Processing Magazine 29 (6) (2012) 141–142

    L. Deng, The mnist database of handwritten digit images for machine learning research, IEEE Signal Processing Magazine 29 (6) (2012) 141–142. 29

    2012

  32. [40]

    Krizhevsky, Learning multiple layers of features from tiny images, Techni- cal Report, University of Toronto (2009) 32–33

    A. Krizhevsky, Learning multiple layers of features from tiny images, Techni- cal Report, University of Toronto (2009) 32–33. URL https://www.cs.toronto.edu/~kriz/learning-features-2009-TR. pdf

    2009

  33. [41]

    Netzer, T

    Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, A. Y. Ng, Reading digits in natural images with unsupervised feature learning, in: NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http://ufldl.stanford.edu/housenumbers/nips2011_ housenumbers.pdf

    2011

  34. [42]

    Bulatov, notMNIST Dataset, Available at http://yaroslavvb

    Y. Bulatov, notMNIST Dataset, Available at http://yaroslavvb. blogspot.com/2011/09/notmnist-dataset.html (2011)

    2011

  35. [43]

    B. M. Lake, R. Salakhutdinov, J. B. Tenenbaum, Human-level concept learning through probabilistic program induction, Science 350 (6266) (2015) 1332–1338. arXiv:https://www.science.org/doi/pdf/10.1126/science. aab3050, doi:10.1126/science.aab3050. URL https://www.science.org/do...

    2015 doi

  36. [44]

    Tishby, N

    N. Tishby, N. Zaslavsky, Deep learning and the information bottleneck prin- ciple, CoRR abs/1503.02406 (2015). arXiv:1503.02406. URL http://arxiv.org/abs/1503.02406

    2015 arXiv

  37. [45]

    Liang, Y

    S. Liang, Y. Li, R. Srikant, Principled detection of out-of-distribution exam- ples in neural networks, CoRR abs/1706.02690 (2017). arXiv:1706.02690. URL http://arxiv.org/abs/1706.02690

    2017 arXiv

  38. [46]

    Burda, R

    Y. Burda, R. B. Grosse, R. Salakhutdinov, Importance weighted autoen- coders, in: Y. Bengio, Y. LeCun (Eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://arxiv.org/abs...

    2016 arXiv

  39. [47]

    D. P. Kingma, J. Ba, Adam: A method for stochastic optimization (2017). arXiv:1412.6980. 30 Supplementary Material

    2017 arXiv

  40. [48]

    Given the total latent budget b and number of lay- ers N , the latent dimension li for layer i is: li = b · (1 − r) · ri−1 1 − rN

    Proofs Proof of Existence of r∗. Given the total latent budget b and number of lay- ers N , the latent dimension li for layer i is: li = b · (1 − r) · ri−1 1 − rN . (27) Define the objective function: F (r) = NX i=1 f (li) = NX i=1 f b · (1 − r) · ri−1 1 − rN . (28) Given the ...

  41. [49]

    Decoding Distributions In this section, we describe the different decoding distributions considered for the HV AES considered in our experiments (as well as prior work). 9.1. Bernoulli Distribution The Bernoulli distribution is used for modeling binary data. The decoder output...

  42. [50]

    Reduction Ratio Geometric Series 10.1. Derivation for li Given the first layer’s dimensionality, l1, we can express the dimension- ality of the i-th layer in a hierarchical structure where the dimensions of successive layers follow a geometric series. 10.1.1. Assumptions The d...

  43. [51]

    Evaluation 11.1. Metrics These metrics are widely adopted in the literature for OOD Detection: • FPR80 (False Positive Rate at 80% True Positive Rate): Mea- sures the rate of false positives when the true positive rate is 80%. • AUROC (Area Under the Receiver Operating Charact...

  44. [52]

    Training time for a single HV AE model on one Nvidia GTX 2080 Ti GPU was approximately 48 hours for grayscale images and > 200 hours for natural images

    Model Details The hyperparameter settings used to train the hierarchical variational autoencoders used for detection are listed in Table 1. Training time for a single HV AE model on one Nvidia GTX 2080 Ti GPU was approximately 48 hours for grayscale images and > 200 hours for ...

    2000

  45. [53]

    Additional Results 13.1. Mutual Information Relationship with r To illustrate the tradeoff between attenuation and information loss, we first explore the relationship between choice of r in the compressed configu- rations and the corresponding estimates of I(X; Z3). 13.1.1. I(...

    1937

  46. [54]

    Threats to Validity 14.1. Internal • Selection bias:The choice of specific dataset pairs (FashionMNIST/MNIST and CIF AR-10/SVHN) while similar to prior work, might not represent the full spectrum of possible in-distribution and out-of-distribution re- lationships. 40 14.2. Ext...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.