REVIEW 3 major objections 5 minor 21 references
Mixture of Balanced Information Bottlenecks for Long-Tailed Visual Recognition
T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read The paper argues that a loss combining balanced softmax cross-entropy with variational self-distillation implicitly optimizes the information bottleneck objective, and applying it at three network depths rivals mixture-of-experts accuracy o
desk verdict Solid empirical recipe for long-tailed recognition, but the information-bottleneck story doesn't survive contact with the math; the method may still work, but the paper's main claim needs reworking. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The pivotal identity is I(v;z) = I(z;y) + I(v;z|y), which splits the bottleneck objective into three sub-objectives. The load-bearing mechanism is the variational self-distillation loss, Loss3 = E[D_KL(q(y|v)||q(y|z))]: minimizing the KL divergence between the classifier outputs at v and z makes H(y|v) ≈ H(y|z), which the paper takes to force I(z;y) ≈ I(v;y) without explicit mutual-information estimation. Loss1 and Loss2 are balanced softmax cross-entropy losses that add log n_i to the logits and weight classes inversely to frequency, rebalancing the long-tailed label distribution. MBIB's structural argument is the data processing inequality I(v3;y) ≤ I(v2;y) ≤ I(v1;y), showing shallower rep
What would settle it
A direct test: train BIB on a synthetic long-tailed dataset whose label depends only on a few input dimensions, then measure I(v;z|y) and |I(v;y) − I(z;y)| as β grows. If the self-distillation loss can be driven near zero while z still retains label-independent noise from x (I(v;z|y) stays large), the claimed equivalence fails. Concretely, check whether increasing β continues to compress z's label-irrelevant dimensions after the KL term has already converged.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that minimizing LossBIB(v,z) = Loss1 + Loss2 + β·Loss3 implicitly solves the information bottleneck problem under a long-tailed label distribution. Under the assumption that encoder output v preserves all label-relevant information from x, the IB objective decomposes via I(v;z) = I(z;y) + I(v;z|y) into three sub-objectives: maximize I(v;y), maximize I(z;y), and minimize |I(v;y) − I(z;y)|. Loss1 and Loss2 are balanced softmax cross-entropy losses that maximize the first two terms; Loss3 is a variational self-distillation loss minimizing D_KL[q(y|v)||q(y|z)], which drives I(z;y) toward I(v;y). MBIB sums BIB losses at three depths, a·LossBIB(v1,z)
Load-bearing premise
The whole argument rides on the claim that driving two classifiers' predictions together is equivalent to driving the mutual information between their representations and the label together; if that equivalence needs conditions the paper does not state, Loss3 is not a valid information bottleneck proxy and what remains is an empirical loss combination.
Editorial extensions
If this is right
- Long-tailed recognition can be trained end-to-end in one stage: representation and classifier are learned jointly from a single loss, avoiding the representation/classifier decoupling used by two-stage methods.
- Single-expert models become a practical alternative to mixture-of-experts: the paper reports MBIB at 486K parameters and 27.93 GFLOPs versus 784K/40.69 G for SADE and 1018K/50.98 G for RIDE(4E), with competitive or better accuracy on iNaturalist 2018.
- Both rebalancing ingredients matter: ablations show removing logit adjustment costs 0.9% overall accuracy and removing class rebalancing costs 2.2%, so the balanced cross-entropy design is load-bearing, not decorative.
- The self-distillation term (β > 0) is what makes the loss a bottleneck: ablation results show the balanced terms alone (β = 0) underperform BIB, consistent with the claim that compressing label-irrelevant information helps long-tailed recognition.
Reading between the lines
- The paper's information-theoretic reading of Loss3 is one of several possible descriptions: the same objective can be seen as standard teacher-student self-distillation between two heads of one network. If the equivalence between KL matching and mutual-information matching fails in some regime (e.g., when v is a poor statistic of x), the method's empirical gains might still hold, but the IB interp
- The ablation on the number of observations v (2-v through 6-v) suggests a general diminishing-returns law for layer-wise auxiliary supervision: each added bottleneck helps until the optimization cost outweighs the extra label information. This is a testable design principle for other long-tailed architectures.
- The Appendix G result that adding MBIB to the SADE expert framework improves accuracy by 1.6% hints that the bottleneck mechanism and expert routing are complementary—one could combine them systematically, e.g., one BIB per expert with expert-specific weights.
- The class-dependent temperature Ti = (n_max/n_i)^γ couples distillation strength to class frequency; tuning γ per class rather than globally might extract more tail-class signal, an extension the paper does not explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Balanced Information Bottleneck (BIB) and Mixture of Balanced Information Bottlenecks (MBIB) for long-tailed visual recognition. BIB combines two balanced softmax cross-entropy losses (Loss1, Loss2) with a variational self-distillation loss (Loss3) and claims this implicitly optimizes the information bottleneck objective. MBIB extends BIB by summing BIB losses at three network depths, using intermediate representations to preserve label-related information. Experiments on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 report accuracy competitive with or better than many one-stage, two-stage, and mixture-of-experts baselines, along with an efficiency comparison.
Significance. If the theoretical claim were sound, BIB/MBIB would offer a useful single-expert alternative to MoE-style long-tailed methods, with lower compute and end-to-end training. The paper has empirical strengths: evaluations span three standard datasets and multiple backbones; ablations address loss components, re-balancing choices, and number of observations; efficiency is compared with MoE baselines; and Python code for the BIB loss is included in Appendix J. These strengths support the practical potential of the method. However, the information-bottleneck interpretation is the paper's central contribution and is not established: Theorem 3.2 contains a load-bearing equivalence that is false under the stated Markov structure, and the proposed loss contains no actual compression term. The empirical method may still be valuable, but the current framing overstates the theoretical guarantee.
major comments (3)
- [Sec. 3.2, Theorem 3.2, Eq. (3)] The proof claims that minimizing I(v;z|y) is equivalent to minimizing |I(v;y)-I(z;y)|, citing Tian et al. Under the natural Markov chain y→v→z, we have I(z;y|v)=0, so I(v;y)-I(z;y)=I(v;y|z), not I(v;z|y). These conditional mutual informations are not generally equal. For example, let Z be a noisy copy of Y and let V=(Z,N) with N independent of Y given Z. Then Z is sufficient for Y, giving I(v;y|z)=0 and |I(v;y)-I(z;y)|=0, while I(v;z|y)=H(z|y)>0. Thus Loss3, which matches q(y|v) and q(y|z), is not a valid proxy for the compression term I(v;z|y). Eq. (9) therefore contains no term that minimizes the bottleneck quantity I(v;z|y) or I(v;z), and the central claim that LossBIB implicitly optimizes the IB objective is unsupported. The same issue propagates to Eq. (11) and the MBIB motivation.
- [Sec. 3.2, Eq. (8) and implementation] The VSD loss actually used in experiments is not the KL divergence in Eq. (8) when gamma>0. Immediately after Eq. (8), class-dependent temperatures T_i=(n_max/n_i)^gamma are inserted into q(y|v) and q(y|z). For ImageNet-LT and iNaturalist, gamma=0.5, so the loss is a temperature-scaled, class-weighted KL. No information-theoretic derivation is given for this scaling, and it changes the conditional entropy and mutual-information relationships on which Theorem 3.2 relies. Either the theorem and loss need to be formulated for the temperature-scaled distributions, or the experiments need to be run with gamma=0 so that Eq. (8) matches the implementation.
- [Sec. 4.2, Figures 3 and 4] The hyperparameters beta, a, and b are chosen by looking at overall accuracy on the test sets (e.g., 'we take beta in {0,1,2,3,4,5} according to different datasets' and 'we selected a=0.1 and b=0.3 within these satisfactory configurations'). No validation split or selection protocol is described. If the reported numbers were selected using test-set accuracy, the state-of-the-art comparisons in Tables 1-3 are at risk of selection bias. The paper should either use a held-out validation set or explicitly report the selection procedure and its influence on the reported gains.
minor comments (5)
- [Sec. 1 / References] In the introduction, 'BBN (Wang et al., 2020), RIDE (Zhou et al., 2020)' appears to swap the citations: BBN is Zhou et al. (2020) and RIDE is Wang et al. (2020).
- [Eq. (8)] The notation 'q(y|v)[log q(y|v)-log q(y|z)]' is ambiguous; the expectation should be written as E_{q(y|v)}[log q(y|v)-log q(y|z)].
- [Appendix C] The text says 'We assume v can retain all the information in x, but the information will still be lost from x to v.' This directly contradicts Assumption 3.1, which states I(v;y)=I(x;y). Please reconcile the assumption with the acknowledged information loss.
- [Table 6] The efficiency comparison reports Params and FLOPs but does not specify the backbone or training/inference configuration for each method. Since RIDE and SADE use different backbones in the main experiments, clarify that the comparison is apples-to-apples.
- [Appendix J] The code hard-codes beta=5 and label_weighting=0.1 rather than taking them as arguments. This makes it hard to reproduce the hyperparameter search described in Sec. 4.2.
Circularity Check
No significant circularity: the central claims are empirical, and the information-theoretic framing, while relying on an external citation, does not reduce to its inputs by construction.
full rationale
The paper's headline claims are empirical: BIB and MBIB improve accuracy on long-tailed benchmarks relative to baselines. The loss (Eq. 9) is a weighted sum of two balanced softmax cross-entropy losses and a KL-divergence self-distillation term; no fitted parameter is later presented as a prediction of that same parameter. The information-theoretic justification in Theorem 3.2 rests on an equivalence attributed to Tian et al. (2021) — an external reference, not a self-citation. Whether that equivalence is correct is a question of mathematical validity, not circularity: the paper does not define the IB objective as its own loss, nor does any equation reduce the stated 'prediction' to the training inputs by construction. The MBIB objective (Eq. 11) is a linear combination of BIB losses, again an empirical design. The absence of any self-citation chain and the presence of external benchmarks (CIFAR-100-LT, ImageNet-LT, iNaturalist 2018) with reported test accuracies further support a non-circular finding. The main caveat is that the cited equivalence is not proved in the paper and may be false; that is a correctness/rigor concern, not circularity.
Assumptions & free parameters
free parameters (5)
- beta =
selected from {0,1,2,3,4,5} per dataset
- a =
0.1
- b =
0.3
- m =
0.1
- gamma =
0 for CIFAR-100-LT, 0.5 for ImageNet-LT and iNaturalist
assumptions (4)
- ad hoc to paper Assumption 3.1: the observation v extracted by the encoder retains all label-relevant information, I(v;y)=I(x;y).
- domain assumption Minimizing I(v;z|y) is equivalent to minimizing |I(v;y)-I(z;y)|, and minimizing KL(q(y|v)||q(y|z)) makes H(y|v) approximate H(y|z).
- standard math The variables follow a Markov chain y -> x -> v -> z, and variational distributions q(y|v), q(y|z), q(z|x), r(z) can approximate the true distributions.
- standard math Data processing inequality: I(v3;y) <= I(v2;y) <= I(v1;y) for intermediate representations of a CNN.
Cite this review
Pith. "Pith review of Mixture of Balanced Information Bottlenecks for Long-Tailed Visual Recognition." pith.science (2026). https://pith.science/paper/PA2UMU32
@misc{pith2026250901804,
author = {Pith},
title = {Pith review of: Mixture of Balanced Information Bottlenecks for Long-Tailed Visual Recognition},
year = {2026},
howpublished = {\url{https://pith.science/paper/PA2UMU32}},
note = {Machine review of arXiv:2509.01804}
}
read the original abstract
Deep neural networks (DNNs) have achieved significant success in various applications with large-scale and balanced data. However, data in real-world visual recognition are usually long-tailed, bringing challenges to efficient training and deployment of DNNs. Information bottleneck (IB) is an elegant approach for representation learning. In this paper, we propose a balanced information bottleneck (BIB) approach, in which loss function re-balancing and self-distillation techniques are integrated into the original IB network. BIB is thus capable of learning a sufficient representation with essential label-related information fully preserved for long-tailed visual recognition. To further enhance the representation learning capability, we also propose a novel structure of mixture of multiple balanced information bottlenecks (MBIB), where different BIBs are responsible for combining knowledge from different network layers. MBIB facilitates an end-to-end learning strategy that trains representation and classification simultaneously from an information theory perspective. We conduct experiments on commonly used long-tailed datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018. Both BIB and MBIB reach state-of-the-art performance for long-tailed visual recognition.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Alexander A. Alemi, Ian S. Fischer, Joshua V. Dillon, and Kevin P. Murphy. Deep variational information bottleneck. ArXiv, abs/1612.00410,
-
[5]
Drop-Bottleneck: Learning Discrete Compressed Representation for Noise-Robust Exploration
Jaekyeom Kim, Minjung Kim, Dongyeon Woo, and Gunhee Kim. Drop-bottleneck: Learning discrete com- pressed representation for noise-robust exploration.ArXiv, abs/2103.12300,
-
[7]
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in an open world.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2532–2541,
work page 2019
-
[9]
Long-tail learning via logit adjustment.ArXiv, abs/2007.07314,
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment.ArXiv, abs/2007.07314,
arXiv 2007
-
[10]
Decoupled Training for Long-Tailed Classification With Stochastic Representations
Giung Nam, Sunguk Jang, and Juho Lee. Decoupled training for long-tailed classification with stochastic representations. arXiv preprint arXiv:2304.09426,
-
[12]
The information bottleneck method
N Tishby. The information bottleneck method. InProc. 37th Annual Allerton Conference on Communica- tions, Control and Computing, 1999, pp. 368–377,
work page 1999
-
[15]
Margin Calibration for Long-Tailed Visual Recognition
Yidong Wang, Bowen Zhang, Wenxin Hou, Zhen Wu, Jindong Wang, and Takahiro Shinozaki. Margin calibration for long-tailed visual recognition.ArXiv, abs/2112.07225, 2021d. Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning.ArXiv, abs/2006.07529,
work page Pith review arXiv 2006
-
[16]
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2361–2370,
work page 2021
Show all 21 references
-
[17]
Deep long-tailed learning: A survey
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023a. Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A ...
2021
-
[18]
Rebalanced siamese contrastive mining for long-tailed recognition.ArXiv, abs/2203.11506,
Zhisheng Zhong, Jiequan Cui, Eric Lo, Zeming Li, Jian Sun, and Jiaya Jia. Rebalanced siamese contrastive mining for long-tailed recognition.ArXiv, abs/2203.11506,
-
[19]
However, when the labels are long-tailed, we need to re-balance them
A Proof of the Re-Balance Technique According to VIB,−I(v,y ) and−I(z,y ) are bounded asEp(v,y)−logq(y|v) and Ep(z,y)−logq(y|z). However, when the labels are long-tailed, we need to re-balance them. Our purpose is to train an end-to-end model, that is, the output of the model ...
2019
-
[20]
To address these issues, logit adjustment methods have been proposed (Menon et al., 2021)
have highlighted their limitations, including the lack of Fisher consistency. To address these issues, logit adjustment methods have been proposed (Menon et al., 2021). Furthermore, these methods can be combined, as demonstrated in Zhang et al. (2023a). As a result, we incorpo...
2021
-
[21]
MBIB w/o logits adjustment means removing logits adjustment in MBIB
MBIB w/o class re-balancing meansm = 0 and the weight in Eq.7 is 1 for all classes. MBIB w/o logits adjustment means removing logits adjustment in MBIB. MBIB w/o re-balancing means not using any re-balancing techniques in MBIB. The experimental results reveal that removing eit...
2023
-
[1999]
Deep learning and the information bottleneck principle.2015 IEEE Information Theory Workshop (ITW), pp
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle.2015 IEEE Information Theory Workshop (ITW), pp. 1–5,
2015
-
[2006]
Farewell to mutual information: Variational distillation for cross-modal person re-identification.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp
Xudong Tian, Zhizhong Zhang, Shaohui Lin, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Farewell to mutual information: Variational distillation for cross-modal person re-identification.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1522–1531,
2021
-
[2008]
Label-aware distribution calibration for long-tailed classification.ArXiv, abs/2111.04901, 2021a
Chaozheng Wang, Shuzheng Gao, Cuiyun Gao, Pengyun Wang, Wenjie Pei, Lujia Pan, and Zenglin Xu. Label-aware distribution calibration for long-tailed classification.ArXiv, abs/2111.04901, 2021a. Jing Wang, Yuanjie Zheng, Jingqi Song, and Sujuan Hou. Cross-view representation lea...
2010 arXiv
-
[2019]
Significance-aware information bottleneck for domain adaptive semantic segmentation.2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp
Yawei Luo, Ping Liu, Tao Guan, Junqing Yu, and Yi Yang. Significance-aware information bottleneck for domain adaptive semantic segmentation.2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6777–6786,
2019
-
[2020]
HaoChen, Adrien Gaidon, and Tengyu Ma
Hong Liu, Jeff Z. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. ArXiv, abs/2110.05025,
-
[2021]
Disentangling label distribution for long-tailed visual recognition.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp
Youngkyu Hong, Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition.2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6622–6632,
2021
-
[2022]
Belongie
Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge J. Belongie. Class-balanced loss based on effective number of samples.2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9260–9269,
2019
-
[2023]
Botvinick, H
Anirudh Goyal, Riashat Islam, Daniel Strouse, Zafarali Ahmed, Matthew M. Botvinick, H. Larochelle, Sergey Levine, and Yoshua Bengio. Infobot: Transfer and exploration via the information bottleneck.ArXiv, abs/1901.10902,
1901 arXiv
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.