Pith. sign in

REVIEW 4 major objections 3 minor 63 references

Label Smoothing is a Pragmatic Information Bottleneck

T0 review · 4 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Label smoothing, as its smoothing strength varies, provably traces the information-bottleneck-optimal family of output distributions, making the technique a practical implementation of the information bottleneck.

desk verdict A clean, plausible abstract-level claim that label smoothing traces the IB frontier, but the supplied body is unreadable so the derivation stays unverified. read the letter →

arxiv 2508.14077 v1 pith:2N5LM7ZR submitted 2025-08-12 cs.LG cs.AI

Sota Kudo This is my paper · ORCID
classification cs.LGcs.AI MSC 68T0794A17
keywords labelsmoothinginformationbottleneckmutualcross-entropytrainingrepresentationlearningtrade-offcurvefeatureselectionmodeloutputdistribution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that label smoothing—the common trick of replacing one-hot training targets with a mixture of the true label and a uniform distribution—is not just a regularizer. Under the premises that the model is flexible enough to fit any output distribution and that no input carries conflicting labels, the paper argues that the distribution a label-smoothed model outputs is exactly the one a certain information bottleneck problem would pick. Varying the smoothing strength then sweeps along the trade-off between compressing the input and keeping label information. If true, label smoothing becomes a cheap, practical way to run information-bottleneck-style representation learning, and it should inherit IB's known insensitivity to irrelevant or redundant features; the paper adds experiments showing exactly that insensitivity.

What carries the argument

The central object is the smoothed conditional target distribution $$q_\$\alpha$(y\mid x)=(1-\$\alpha$)\,\delta_{y^*}(y)+\frac{\$\alpha$}{K},$$ used as the cross-entropy training target. Because a sufficiently flexible softmax model can interpolate any target, the trained output becomes $q_\alpha$, and the paper's derivation identifies this output family with the solution set of an information bottleneck optimization, where $\alpha$ acts as the Lagrange multiplier balancing compression of $x$ against prediction of $y$. This identification is what turns a training heuristic into a principled IB method.

What would settle it

Construct a dataset whose information-plane frontier is computable in closed form, such as a deterministic label plus an independent nuisance feature, train a high-capacity network under label smoothing at several strengths $\alpha$, and measure the mutual information of the learned output features with the input and with the label. If the measured points do not lie on the predicted IB frontier, or if the trained output departs measurably from $q_\alpha$ in a regime where capacity should suffice, the claimed equivalence is refuted.

Watch

Extended reading notes

Core claim

This paper's central claim is that label smoothing is a pragmatic information bottleneck. When a model is trained with cross-entropy against the smoothed target $$q_\$\alpha$(y\mid x)=(1-\$\alpha$)\,\delta_{y^*}(y)+\frac{\$\alpha$}{K},$$ sufficient flexibility lets the model's output match $q_\alpha$ for every input, and the paper derives that this output family is exactly the family a suitably formulated information bottleneck problem selects as $\alpha$ varies. The strength $\alpha$ therefore plays the role of the Lagrange multiplier that trades compression of the input representation against preservation of label information. Equivalently, each label-smoothed network is already doing IB-optimal compression on its output layer, with no auxiliary network or variational approximation needed, under the stated assumptions of sufficient model flexibility and no conflicting labels for the same input. The paper also demonstrates experimentally that label smoothing suppresses features that carry no information about the target, or no additional information once another feature is known, which is the behavioral signature of an information bottleneck solution.

Load-bearing premise

The entire equivalence depends on the trained model being flexible enough to output exactly the smoothed target distribution for every input and every smoothing strength, and on no input having two different labels; real networks only approximate the first condition, so the theoretical claim may not hold exactly for them.

Editorial extensions

If this is right

  • Label smoothing can serve as a drop-in information bottleneck objective, requiring no auxiliary networks, variational bounds, or separate information-theoretic losses.
  • The smoothing strength $\alpha$ is not merely a regularization knob; it selects a specific point on the IB trade-off curve between input compression and label preservation.
  • Models trained with label smoothing should ignore features that carry no target information, whether alone or conditioned on other features, matching the behavioral signature of IB-optimal representations.
  • The reported calibration and robustness effects of label smoothing can be read as consequences of IB-style compression rather than as unrelated side effects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the claimed equivalence holds, the trained network's output entropy becomes an observable proxy for the IB trade-off point selected by $\alpha$, so practitioners could choose $\alpha$ by targeting a desired compression level rather than by grid search on validation accuracy.
  • The 'no conflicting labels' premise carves out exactly the noisy-label regime where smoothing is often recommended; extending the argument there would require modeling conflicts as part of the label distribution instead of excluding them, which the paper does not attempt.
  • The same mechanism suggests a broader design rule: any training target formed as a convex mixture of a hard label and a fixed reference distribution induces an IB-style trade-off, so teacher-smoothed distillation objectives might trace the same frontier with the teacher playing the role of the reference distribution.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper claims that, under sufficient model flexibility and no conflicting labels for the same input, the output distribution of a model trained with label smoothing explores the optimal solution of the information bottleneck (IB) as the smoothing strength varies. On this basis, the paper interprets label smoothing as a simple, practical implementation of the IB principle and reports experiments showing that label smoothing is insensitive to input factors that carry no information about the target, or no additional information beyond other variables. The abstract is legible, but the full text is severely corrupted: the body consists of mojibake characters, and no theorem, derivation, experimental protocol, figure, or table can be inspected.

Significance. If the claimed equivalence were established rigorously, the paper would provide a clean conceptual bridge between label smoothing and information-theoretic regularization, making the IB objective accessible through a one-line modification of standard training. The reported insensitivity property is also a falsifiable prediction with practical value. These strengths are real but conditional: the manuscript as supplied contains no inspectable proof or experimental evidence, and the stated assumptions (sufficient model flexibility, no conflicting labels) substantially narrow the regime in which the central claim is supposed to hold. The contribution therefore cannot currently be assessed beyond its abstract-level promise.

major comments (4)
  1. [Full text (all sections after the Abstract)] The body of the manuscript is unreadable: it is a sequence of replacement characters and garbled text, with no legible definitions, theorem statements, derivations, or experimental results. The central claim is a mathematical equivalence between label-smoothed outputs and IB-optimal distributions, and that claim cannot be verified from the supplied text. This is a load-bearing issue, not a cosmetic one. Please provide a readable version with the full derivation, the formal statement of the theorem, and the experimental details.
  2. [Abstract and Theorem (unreadable body)] The abstract does not state the mapping between the label-smoothing coefficient ε and the IB trade-off parameter β. The IB objective is L = I(X;Z) - β I(Z;Y) (or similar), so some relationship between β and ε is required for the claimed equivalence. If β is defined implicitly through ε so that the two objectives coincide by construction, then the claim that label smoothing 'explores' the IB optimal family risks being circular. Please give the explicit mapping and show that it is not an identity imposed by definition.
  3. [Abstract, assumption of sufficient model flexibility] The assumption of sufficient model flexibility is doing the load-bearing work: the theoretical equivalence requires that the trained model can represent the IB-optimal output distribution for every smoothing strength. Real neural networks have finite capacity and finite training time, so the empirically obtained output generally will not equal the theoretical minimizer. The experimental section is the only possible evidence for the finite-capacity regime, but it cannot be inspected in the corrupted text. Please state the theorem's exact representational condition and provide the experiments that validate the finite-capacity case.
  4. [Abstract, assumption of no conflicting labels] The assumption of no conflicting labels for the same input excludes exactly the noisy-label setting in which label smoothing is commonly applied in practice. The paper's title and framing suggest broad applicability, but the stated assumption narrows the scope considerably. Please clarify whether the equivalence or the insensitivity property holds when the dataset contains label noise, or, if not, state the limitation prominently.
minor comments (3)
  1. [Header] The header contains 'arXiv:2508.14074v1 [cs.LG] 12 Aug 2025', which does not match the manuscript number 2508.14077 cited in this review; please correct the arXiv identifier.
  2. [Abstract] The word 'explores' is imprecise: please specify whether the label-smoothed output equals the IB-optimal distribution exactly, converges to it in the limit, or approximates it in a particular metric.
  3. [Abstract] The phrase 'no conflicting labels for the same input' needs a formal definition, including how it interacts with stochastic or soft labels in the label-smoothing construction.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity can be established from the supplied text; the abstract's conditional equivalence claim is not visibly definitional, and the full text is garbled, so no equation-level reduction can be quoted.

full rationale

To mark this paper circular, I would need to quote a specific reduction: for example, an equation showing the label-smoothing coefficient is substituted directly into the information-bottleneck trade-off parameter so that the two objectives coincide by construction, or a fitted quantity being renamed as a prediction. The supplied full text is mojibake; the lemmas, theorem statements, derivations, and experimental details are unreadable, and the abstract alone does not define the IB parameter in terms of the smoothing coefficient or vice versa. The abstract's assumptions (sufficient model flexibility, no conflicting labels) are scope conditions rather than circular reductions; they may limit the practical applicability of the claim, but they do not make the claimed equivalence tautological. No self-citation chain is visible in the readable portions, and no fitted-input-called-prediction pattern appears. Because the hard rules require an exhibited reduction before a circularity finding, and none can be identified from the available text, the appropriate result is no significant circularity (score 0).

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central equivalence rests on two assumptions the abstract itself announces: sufficient model flexibility and single deterministic labels per input. Under those, the paper claims the smoothed output family traces the IB frontier. The only hand-chosen numeric input visible from the abstract is the smoothing coefficient epsilon. No new physical or conceptual entities are introduced. The full derivation, and any additional implicit assumptions (for example about the exact form of the IB Lagrangian or about which mutual information terms are computed), could not be inspected because the body text is unreadable in the provided dump.

free parameters (1)
  • label smoothing coefficient epsilon (per-class uniform mixture weight) = not given in abstract; conventionally 0.1 in practice
    The smoothing coefficient is the knob that, per the abstract, moves the model output along the information bottleneck frontier. Its value in the experiments is a hand-chosen hyperparameter; the abstract does not derive an optimal epsilon from the data.
assumptions (3)
  • domain assumption The model class is flexible enough to represent the information-bottleneck-optimal output distribution, stated in the abstract as 'sufficient model flexibility.'
    This is the paper's own stated premise. If real networks cannot represent the smoothed-target output family, trained outputs need not explore the IB frontier; the abstract's experiments are the only evidence for the finite-capacity case.
  • domain assumption No input has conflicting labels, i.e., the training target for each input is a single deterministic label, stated in the abstract as 'no conflicting labels for the same input.'
    This keeps the target conditional p(y|x) degenerate, which is what makes the IB optimum tractable; noisy or multi-label data, a common real-world setting, is excluded by assumption.
  • standard math Standard information-theoretic toolkit: mutual information definitions, the data processing inequality, and the variational form of the IB objective with a Lagrange multiplier beta.
    These are unproved background results the derivation necessarily assumes; they are standard and not controversial.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Label Smoothing is a Pragmatic Information Bottleneck." pith.science (2026). https://pith.science/paper/2N5LM7ZR

@misc{pith2026250814077,
  author       = {Pith},
  title        = {Pith review of: Label Smoothing is a Pragmatic Information Bottleneck},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2N5LM7ZR}},
  note         = {Machine review of arXiv:2508.14077}
}
read the original abstract

This study revisits label smoothing via a form of information bottleneck. Under the assumption of sufficient model flexibility and no conflicting labels for the same input, we theoretically and experimentally demonstrate that the model output obtained through label smoothing explores the optimal solution of the information bottleneck. Based on this, label smoothing can be interpreted as a practical approach to the information bottleneck, enabling simple implementation. As an information bottleneck method, we experimentally show that label smoothing also exhibits the property of being insensitive to factors that do not contain information about the target, or to factors that provide no additional information about it when conditioned on another variable.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

63 extracted references · 56 canonical work pages

  1. [1]

    Emergence of invariance and disentanglement in deep representations

    A Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. International Conference on Internet Technology and Applications, 19 0 (50): 0 1--9, June 2017

    work page 2017

  2. [2]

    Information dropout: Learning optimal representations through noisy computation

    Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE Trans. Pattern Anal. Mach. Intell., 40 0 (12): 0 2897--2905, December 2018

    work page 2018

  3. [3]

    Invariance principle meets information bottleneck for out-of-distribution generalization

    Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. Adv. Neural Inf. Process. Syst., 34: 0 3438--3450, 2021

    work page 2021

  4. [4]

    Deep variational information bottleneck

    Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv [cs.LG], December 2016

    work page 2016

  5. [5]

    Uncertainty in the variational information bottleneck

    Alexander A Alemi, Ian Fischer, and Joshua V Dillon. Uncertainty in the variational information bottleneck. arXiv [cs.LG], July 2018

    work page 2018

  6. [6]

    Recognition in terra incognita

    Sara Meghan Beery, Grant Van Horn, and P Perona. Recognition in terra incognita. ECCV, abs/1807.04975: 0 456--473, July 2018

    arXiv 2018

  7. [7]

    An investigation of how label smoothing affects generalization

    Blair Chen, Liu Ziyin, Zihao Wang, and Paul Pu Liang. An investigation of how label smoothing affects generalization. arXiv [cs.LG], October 2020

    work page 2020

  8. [8]

    For better or for worse? learning minimum variance features with label augmentation

    Muthu Chidambaram and Rong Ge. For better or for worse? learning minimum variance features with label augmentation. In The Thirteenth International Conference on Learning Representations, October 2024

    work page 2024

Show all 63 references
  1. [9]

    Towards better decoding and language model integration in sequence to sequence models

    Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence to sequence models. arXiv [cs.NE], December 2016

    2016

  2. [10]

    ImageNet : A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet : A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255. IEEE, June 2009

    2009

  3. [11]

    The conditional entropy bottleneck

    Ian Fischer. The conditional entropy bottleneck. arXiv [cs.LG], February 2020

    2020

  4. [12]

    Learning better structured representations using low-rank adaptive label smoothing

    Asish Ghoshal, Xilun Chen, Sonal Gupta, Luke Zettlemoyer, and Yashar Mehdad. Learning better structured representations using low-rank adaptive label smoothing. In International Conference on Learning Representations, October 2020

    2020

  5. [13]

    Escaping the big data paradigm with compact transformers

    Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. 2021. URL https://arxiv.org/abs/2104.05704

    2021 arXiv

  6. [14]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv [cs.CV], December 2015

    2015

  7. [15]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv [stat.ML], March 2015

    2015

  8. [16]

    Multilayer feedforward networks are universal approximators

    Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Netw., 2 0 (5): 0 359--366, January 1989

    1989

  9. [17]

    GPipe : Efficient training of giant neural networks using pipeline parallelism

    Yanping Huang, Yonglong Cheng, Dehao Chen, Hyoukjoong Lee, Jiquan Ngiam, Quoc V Le, and Z Chen. GPipe : Efficient training of giant neural networks using pipeline parallelism. Neural Inf Process Syst, 32: 0 103--112, November 2018

    2018

  10. [18]

    How does information bottleneck help deep learning? arXiv [cs.LG], pp.\ 16049--16096, May 2023

    Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? arXiv [cs.LG], pp.\ 16049--16096, May 2023

    2023

  11. [19]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv [stat.ML], December 2013

    2013

  12. [20]

    Caveats for information bottleneck in deterministic scenarios

    Artemy Kolchinsky, Brendan D Tracey, and Steven Van Kuyk. Caveats for information bottleneck in deterministic scenarios. arXiv [stat.ML], August 2018

    2018

  13. [21]

    Nonlinear information bottleneck

    Artemy Kolchinsky, Brendan D Tracey, and David H Wolpert. Nonlinear information bottleneck. Entropy, 21 0 (12): 0 1181, November 2019

    2019

  14. [22]

    Why do better loss functions lead to less transferable features? arXiv [cs.CV], October 2020

    Simon Kornblith, Ting Chen, Honglak Lee, and Mohammad Norouzi. Why do better loss functions lead to less transferable features? arXiv [cs.CV], October 2020

    2020

  15. [23]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, and Others . Learning multiple layers of features from tiny images. 2009

    2009

  16. [24]

    Flexible variational information bottleneck: Achieving diverse compression with a single training

    Sota Kudo, N Ono, Shigehiko Kanaya, and Ming Huang. Flexible variational information bottleneck: Achieving diverse compression with a single training. ArXiv, abs/2402.01238, February 2024 a

    2024 arXiv

  17. [25]

    Exploring the trade-off in the variational information bottleneck for regression with a single training run

    Sota Kudo, Naoaki Ono, Shigehiko Kanaya, and Ming Huang. Exploring the trade-off in the variational information bottleneck for regression with a single training run. Entropy (Basel), 26 0 (12): 0 1043, November 2024 b

    2024

  18. [26]

    Lagrangian relaxation

    Claude Lemaréchal. Lagrangian relaxation. In Lecture Notes in Computer Science, Lecture notes in computer science, pp.\ 112--156. Springer Berlin Heidelberg, Berlin, Heidelberg, 2001

    2001

  19. [27]

    Invariant information bottleneck for domain generalization

    Bo Li, Yifei Shen, Yezhen Wang, Wenzhen Zhu, Colorado Reed, Dongsheng Li, Kurt Keutzer, and Han Zhao. Invariant information bottleneck for domain generalization. Proc. Conf. AAAI Artif. Intell., 36 0 (7): 0 7399--7407, June 2022

    2022

  20. [28]

    Regularization via structural label smoothing

    Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing. AISTATS, 108: 0 1453--1463, January 2020

    2020

  21. [29]

    Understanding instance-level label noise: Disparate impacts and treatments

    Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. ICML, 139: 0 6725--6735, February 2021

    2021

  22. [30]

    A ConvNet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s . arXiv [cs.CV], January 2022

    2022

  23. [31]

    Does label smoothing mitigate label noise? ICML, abs/2003.02819: 0 6448--6458, March 2020

    M Lukasik, Srinadh Bhojanapalli, A Menon, and Surinder Kumar. Does label smoothing mitigate label noise? ICML, abs/2003.02819: 0 6448--6458, March 2020

    2003 arXiv

  24. [32]

    Generalized entropy regularization or: There's nothing special about label smoothing

    Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's nothing special about label smoothing. arXiv [cs.CL], May 2020

    2020

  25. [33]

    Recurrent models of visual attention

    Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. arXiv [cs.LG], June 2014

    2014

  26. [34]

    When does label smoothing help? Neural Inf Process Syst, abs/1906.02629, June 2019

    Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Neural Inf Process Syst, abs/1906.02629, June 2019

    1906 arXiv

  27. [35]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp.\ 722--729. IEEE, December 2008

    2008

  28. [36]

    No language left behind: Scaling human-centered machine translation

    NLLB Team , Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prang...

    2022

  29. [37]

    Disentangled information bottleneck

    Ziqi Pan, Li Niu, Jianfu Zhang, and Liqing Zhang. Disentangled information bottleneck. arXiv [cs.LG], December 2020

    2020

  30. [38]

    Regularizing neural networks by penalizing confident output distributions

    Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv [cs.NE], January 2017

    2017

  31. [39]

    Human uncertainty makes classification more robust

    Joshua C Peterson, Ruairidh M Battleday, Thomas L Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust. arXiv [cs.CV], August 2019

    2019

  32. [40]

    The dual information bottleneck

    Zoe Piran, Ravid Shwartz-Ziv, and Naftali Tishby. The dual information bottleneck. arXiv [cs.IT], June 2020

    2020

  33. [41]

    On variational bounds of mutual information

    Ben Poole, Sherjil Ozair, Aäron van den Oord, Alexander A Alemi, and G Tucker. On variational bounds of mutual information. ICML, abs/1905.06922: 0 5171--5180, May 2019

    1905 arXiv

  34. [42]

    Regularized evolution for image classifier architecture search

    Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. Proc. Conf. AAAI Artif. Intell., 33 0 (01): 0 4780--4789, July 2019

    2019

  35. [43]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp.\ 618--626, 2017

    2017

  36. [44]

    Learning and generalization with the information bottleneck

    Ohad Shamir, Sivan Sabato, and Naftali Tishby. Learning and generalization with the information bottleneck. Theor. Comput. Sci., 411 0 (29): 0 2696--2711, June 2010

    2010

  37. [45]

    The deterministic information bottleneck

    D J Strouse and David J Schwab. The deterministic information bottleneck. arXiv [q-bio.NC], April 2016

    2016

  38. [46]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv [cs.CV], December 2015

    2015

  39. [47]

    Cover and Joy A Thomas

    Thomas M. Cover and Joy A Thomas. Elements of Information Theory. John Wiley & Sons, Nashville, TN, 2 edition, June 2006

    2006

  40. [48]

    The information bottleneck method

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv [physics.data-an], April 2000

    2000

  41. [49]

    Attention is all you need

    Ashish Vaswani, Noam M Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Neural Inf Process Syst, 30: 0 5998--6008, June 2017

    2017

  42. [50]

    The role of the information bottleneck in representation learning

    Matias Vera, Pablo Piantanida, and Leonardo Rey Vega. The role of the information bottleneck in representation learning. In 2018 IEEE International Symposium on Information Theory (ISIT), pp.\ 1580--1584. IEEE, June 2018

    2018

  43. [51]

    The role of mutual information in variational classifiers

    Matias Vera, Leonardo Rey Vega, and Pablo Piantanida. The role of mutual information in variational classifiers. arXiv [stat.ML], October 2020

    2020

  44. [52]

    Diversifying dialog generation via adaptive label smoothing

    Yida Wang, Yinhe Zheng, Yong Jiang, and Minlie Huang. Diversifying dialog generation via adaptive label smoothing. arXiv [cs.CL], May 2021 a

    2021

  45. [53]

    PAC -bayes information bottleneck

    Zifeng Wang, Shao-Lun Huang, Ercan E Kuruoglu, Jimeng Sun, Xi Chen, and Yefeng Zheng. PAC -bayes information bottleneck. arXiv [cs.LG], September 2021 b

    2021

  46. [54]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, S Gadre, R Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Y Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inferen...

    2022 arXiv

  47. [55]

    Towards understanding why label smoothing degrades selective classification and how to fix it

    Guoxuan Xia, Olivier Laurent, Gianni Franchi, and Christos-Savvas Bouganis. Towards understanding why label smoothing degrades selective classification and how to fix it. arXiv [cs.LG], March 2024

    2024

  48. [56]

    Towards understanding label smoothing

    Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, and Rong Jin. Towards understanding label smoothing. arXiv [cs.LG], June 2020

    2020

  49. [57]

    CoCa : Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. CoCa : Contrastive captioners are image-text foundation models. arXiv [cs.CV], May 2022

    2022

  50. [58]

    Deep deterministic information bottleneck with matrix-based entropy functional

    Xi Yu, Shujian Yu, and Jose C Principe. Deep deterministic information bottleneck with matrix-based entropy functional. arXiv [cs.LG], January 2021

    2021

  51. [59]

    Revisiting knowledge distillation via label smoothing regularization

    Li Yuan, Francis E H Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. arXiv [cs.CV], September 2019

    2019

  52. [60]

    Efficient defenses against adversarial attacks

    Valentina Zantedeschi, Maria-Irina Nicolae, and Ambrish Rawat. Efficient defenses against adversarial attacks. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, New York, NY, USA, November 2017. ACM

    2017

  53. [61]

    Delving deep into label smoothing

    Chang-Bin Zhang, Peng-Tao Jiang, Qibin Hou, Yunchao Wei, Qi Han, Zhen Li, and Ming-Ming Cheng. Delving deep into label smoothing. IEEE Trans. Image Process., 30: 0 5984--5996, June 2021

    2021

  54. [62]

    Learning transferable architectures for scalable image recognition

    Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 8697--8710. IEEE, June 2018

    2018

  55. [63]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.