Pith. sign in

REVIEW 3 major objections 5 minor 50 references

Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Larger models run with early exiting can beat smaller models at the same compute cost, and the new PCEE rule keeps accuracy above a user-chosen floor by thresholding on validation-estimated accuracy rather than raw confidence.

desk verdict Useful, simple method for accuracy-controllable early exiting, but the claimed 'lower bound' guarantee is undermined by selection bias in how the reliability diagrams are built; the empirical core holds up and the paper deserves a serious referee. read the letter →

arxiv 2412.19325 v1 pith:G7CTOBTM submitted 2024-12-26 cs.LG cs.AI

Mehrnaz Mofakhami , Reza Bayat , Ioannis Mitliagkas , Joao Monteiro , Valentina Zantedeschi This is my paper · ORCID
classification cs.LGcs.AI
keywords earlyexitingperformancecontrolreliabilitydiagramsmodelcalibrationinferenceefficiencyaccuracythresholdingoverconfidenceMSDNet
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that early exiting is not just a way to trade accuracy for speed but a way to run a larger model at roughly the same computational cost as a smaller one while ending up more accurate. The supporting claim is that larger models with exit classifiers often make correct predictions at early layers for easy inputs, so their average compute can be lower than that of a smaller model run in full. To make this usable, the authors introduce Performance Control Early Exiting (PCEE), which replaces confidence thresholding with accuracy thresholding: bins built from a held-out validation set map each confidence score to the average accuracy of samples with similar confidence, and the model exits when that bin accuracy clears a user-chosen floor. Experiments on MSDNets (multi-scale dense networks with exit classifiers) and vision transformers across CIFAR-10, CIFAR-100, and ImageNet show that PCEE keeps accuracy above the threshold more consistently than confidence-based baselines, with one meaningful accuracy threshold per model instead of per-layer confidence cutoffs.

What carries the argument

The central object is the reliability diagram, a per-layer histogram built from a held-out validation set that maps confidence bins to the average accuracy of the samples in each bin. At inference, PCEE reads the confidence of an exit classifier, looks up which bin that confidence falls into, and exits only if the bin's average accuracy is at least the user-chosen threshold δ. Because the exit decision is tied to measured accuracy rather than to the classifier's raw confidence, the method does not require a calibrated model or per-layer threshold search. PCEE-WS adds a smoothing step: before binning, each validation sample's 0/1 correctness is replaced by the average correctness of its H nearest neighbors in confidence space (H = 150), which stabilizes bins containing few examples.

What would settle it

Measure whether the accuracy floor holds under distribution shift: train a model on CIFAR-100, build PCEE's reliability diagrams on the standard validation split, then evaluate on a corrupted or shifted variant; if many samples land in bins whose deployed accuracy drops far below the validation bin accuracy, the controllability claim fails exactly where the method would be used.

Watch

Extended reading notes

Core claim

The paper's central claim is that scaling up model size and coupling it with early exiting can yield both higher accuracy and lower average inference cost than running a smaller model in full: for example, MSDNet Large with PCEE reaches roughly 93.88% on CIFAR-10 and 72.13% on ImageNet at compute equal to or less than MSDNet Small's full forward pass, compared with 93.04% and 70.7% for the small model. The mechanism behind this, PCEE, converts each exit layer's confidence into an accuracy estimate using reliability diagrams built from a held-out validation set, then exits when that estimated accuracy exceeds a threshold δ. This is what the authors call performance control: the threshold is literally a target accuracy, not an abstract confidence value, and one threshold serves all layers because each layer has its own confidence-to-accuracy mapping. They also report that PCEE can beat the Oracle exit rule on some settings because intermediate layers can be correct when the final layer is wrong, a phenomenon known as destructive overthinking.

Load-bearing premise

The load-bearing premise is that the reliability diagram built on a held-out validation set still describes the test data, so that a test sample falling into a confidence bin has about the same chance of being correct as the validation samples in that bin.

Editorial extensions

If this is right

  • Deploying a larger model with early exiting becomes an inference-efficiency strategy: at matched FLOPs the large model can be more accurate than the full small model, and at matched accuracy it can use less compute.
  • The user sets one accuracy floor δ for the whole network; no per-layer confidence threshold search is needed, so changing the target accuracy does not require re-running an expensive search.
  • Because PCEE thresholds on accuracy, it stays above the target accuracy even for overconfident exit layers, whereas confidence thresholding can exit prematurely and fall below the target.
  • Adding post-hoc calibration such as temperature scaling before PCEE or PCEE-WS further raises accuracy, and PCEE also works on underconfident models, where confidence thresholding loses low-accuracy controllability.
  • The compute savings translate into latency gains under dynamic batching, since requests that leave the batch early free capacity for the remaining ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: The accuracy-floor guarantee rests on validation-to-test transfer, so a natural extension is an online version of PCEE that continuously updates reliability diagrams from fresh deployment data and gives a formal bound on how far the floor can drift.
  • Editorial inference: The argument suggests a concrete scaling recipe for practitioners: for a fixed compute budget, choose the largest model whose early exit layers already reach the target accuracy, because its average cost will be lower than a small model run in full; this could be tested by scanning model sizes per dataset.
  • Editorial inference: The confidence-to-accuracy lookup is not tied to the maximum softmax score, so the same mechanism could be applied to entropy-based or distance-based confidence measures, and to non-classification tasks where a binned performance metric replaces accuracy.
  • Editorial inference: The result that PCEE sometimes beats the Oracle under destructive overthinking hints that early exits can act as a regularizer; an explicit test would be whether training with a PCEE-aware objective increases intermediate-layer accuracy beyond what joint cross-entropy training achieves.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Performance Control Early Exiting (PCEE), a post-hoc early-exiting method for classification networks that replaces per-layer confidence thresholds with a single user-chosen accuracy threshold. For each layer, a reliability diagram is built on a held-out validation set, mapping confidence bins to empirical accuracy; at inference the model exits at a layer when the bin accuracy for that sample's confidence is at least delta, and otherwise continues. A variant, PCEE-WS, smooths per-sample accuracy using H nearest neighbors in confidence space before binning. Experiments on CIFAR-10, CIFAR-100, and ImageNet with MSDNet and ViT compare PCEE against confidence thresholding, temperature scaling, a Laplace-approximation baseline, and an Oracle. The paper reports that larger models with EE can achieve higher accuracy at similar or lower FLOPs than smaller full models, and that PCEE keeps test accuracy above the chosen threshold more consistently than the confidence-based baselines.

Significance. The 'scale for inference efficiency' observation and the single-global-accuracy-threshold interface are practically appealing, and the method is simple, post-hoc, and applicable to any classifier with per-exit confidence estimates. The paper's strengths include experiments across multiple datasets and architectures, several baselines including an Oracle upper bound, an ablation showing the negligible cost of adding exit layers (Table 3), and an explicit discussion of the distribution-shift limitation (Section 7). The central claim that PCEE 'ensures a lower bound on accuracy' is, however, only as strong as the validation-derived reliability diagrams, and the paper does not address the sequential-selection issue that arises because samples reaching deeper layers under the policy are a filtered subset of the validation population. If that issue is resolved, the empirical contribution would be solid and useful.

major comments (3)
  1. [Section 4.2 and Algorithm 1] The reliability diagrams are described as constructed from the full validation dataset, but the deployment population at layer i under PCEE is the subset of samples that failed to exit at every earlier layer. The quantity needed for the exit decision is P(correct | confidence in bin, survived to layer i), whereas a diagram built on all validation samples estimates P(correct | confidence in bin). Because early-layer exit decisions are correlated with sample difficulty and with correctness at deeper layers, the unconditional estimate can be larger than the conditional one, so the claimed lower bound is not guaranteed even when the test distribution matches the validation distribution. The paper does not state that diagrams are constructed sequentially using only validation samples that reach each layer under the policy, nor does it provide an argument or experiment showing that the selection effect is negligible. Section 7 acknowledges only distribution shift, not this selection bias. This is a load-bearing issue for the central controllability claim and should be fixed by either constructing conditional diagrams or providing empirical evidence that the unconditional diagrams are adequate.
  2. [Abstract and Section 4.2] The statement that PCEE 'ensures a lower bound on accuracy' is stronger than what the method actually provides. The bin accuracies are finite-sample estimates from a validation set, and with 50 bins (as stated in the implementation details) some bins will contain very few samples; PCEE-WS's smoothing changes the estimand but does not add an uncertainty guarantee. The user-chosen threshold delta is therefore not a certified lower bound in any statistical sense unless additional guarantees are provided. I recommend replacing 'ensures' with a more precise phrasing such as 'empirically maintains' or 'targets', or supplementing the method with a calibration/confidence-interval analysis that supports a formal bound.
  3. [Table 8, Appendix D.2, and Table 9] The cross-dataset evidence is thinner than the text suggests. Table 8 reports ImageNet results without standard deviations and for only two threshold values plus a 'best' row; Table 9 reports no error bars for the ViT comparison. Furthermore, Appendix D.2 notes that on ImageNet the Confidence baseline also satisfies the control property and achieves higher accuracy at the cost of more compute, so the claimed advantage of PCEE is less clear in that setting. Adding more threshold values, seeds, and variance estimates would substantiate the claim that PCEE provides better control across datasets.
minor comments (5)
  1. [Contributions and typos] The text contains several typos and formatting inconsistencies, including 'Ealy Exiting', 'amout', 'MSDNET', and 'I MAGENET-1K'; these should be corrected.
  2. [Section 5, Baselines] The Laplace baseline is described as a Monte Carlo estimate with sample size equal to 1; this is a degenerate Monte Carlo estimate and needs clarification as to whether it truly matches the original method's protocol or effectively reduces to a point estimate.
  3. [Table 8 and Figure 6] The 'best ACC' row in Table 8 is undefined; please specify how this row is obtained, for example whether it is the maximum over all thresholds or a particular operating point.
  4. [Implementation details, PCEE-WS] The hyperparameters H (nearest-neighbor count) and the number of bins are fixed to H=150 and 50 bins without a sensitivity study; a short analysis of their effect would be useful since these are free parameters of the method.
  5. [Table 4] The caption states that accuracies below the threshold are shown in red, but no entries in the table appear to be below the threshold; either remove the note or explain why it is included.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PCEE thresholds are set on a held-out validation split and evaluated on test data, with no fitted parameter renamed as a prediction.

full rationale

The paper's central derivation is an empirical evaluation loop, not a circular one. PCEE builds reliability diagrams on a held-out validation set (Section 5: 'we use 10% of the training data for the CIFAR datasets and 4% for ImageNet respectively as held-out validation set to learn the confidence-to-accuracy mappings'), then applies the resulting confidence-to-accuracy mapping to test samples. The accuracy threshold delta is user-chosen, not fitted to test outcomes, and the reported test accuracies are not equal to the validation bin estimates by construction. Algorithm 1 exits when 'acci >= delta', where acci comes from the validation reliability diagram; the subsequent test accuracy is a genuinely independent measurement. The phrase 'which is by design' in Section 5 refers to the rule that the method only exits on validation bins whose average accuracy exceeds the threshold, not to the test result being forced. The paper explicitly acknowledges the main assumption of the method, namely that validation and test distributions match, in Section 7: 'like when running the original model without EE, our method does not handle out-of-distribution data well and suffers from discrepancies between the validation and test sets.' This is a robustness limitation, not circularity. There are no load-bearing self-citations: the cited prior work (e.g., Guo et al. 2017, Huang et al. 2017) is external to the authors, and no uniqueness theorem or ansatz is smuggled in via citation. The sequential-filtering concern raised by a skeptical reader—that reliability diagrams built on the full validation set may overestimate accuracy for samples that survive to deeper layers—is a statistical validity question about the method's guarantee, not a case of the prediction reducing to its input by construction. The paper's claims are therefore self-contained against external test data, and no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method's central load-bearing input is a validation set and the choice of binning/smoothing hyperparameters. No new physical or conceptual entities are introduced; PCEE is a post-hoc inference procedure built on standard confidence scores and validation statistics.

free parameters (2)
  • Number of reliability-diagram bins = 50
    Chosen for all experiments; no sensitivity analysis reported, and bin count affects the accuracy estimates used for exit decisions.
  • Nearest-neighbor count H in PCEE-WS = 150
    Chosen for all experiments; controls the smoothing of per-sample accuracy before binning and is not swept or justified by data.
assumptions (3)
  • domain assumption The held-out validation set is representative of the test distribution.
    Reliability diagrams from validation are used to estimate test-time bin accuracy; the paper's 'lower bound on accuracy' holds only under this transfer, acknowledged in Section 7.
  • domain assumption Maximum softmax output is a stable confidence measure for ordering samples within each layer.
    The method bins samples by max softmax at each exit in Section 4.2; if this quantity does not track accuracy within a layer, the reliability-diagram estimate is uninformative.
  • domain assumption Intermediate exit classifiers can be trained without materially degrading the final layer.
    Table 3 supports this empirically, but it is a premise for the 'large model with EE vs full small model' comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones." pith.science (2026). https://pith.science/paper/G7CTOBTM

@misc{pith2026241219325,
  author       = {Pith},
  title        = {Pith review of: Performance Control in Early Exiting to Deploy Large Models at the Same Cost of Smaller Ones},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G7CTOBTM}},
  note         = {Machine review of arXiv:2412.19325}
}
read the original abstract

Early Exiting (EE) is a promising technique for speeding up inference by adaptively allocating compute resources to data points based on their difficulty. The approach enables predictions to exit at earlier layers for simpler samples while reserving more computation for challenging ones. In this study, we first present a novel perspective on the EE approach, showing that larger models deployed with EE can achieve higher performance than smaller models while maintaining similar computational costs. As existing EE approaches rely on confidence estimation at each exit point, we further study the impact of overconfidence on the controllability of the compute-performance trade-off. We introduce Performance Control Early Exiting (PCEE), a method that enables accuracy thresholding by basing decisions not on a data point's confidence but on the average accuracy of samples with similar confidence levels from a held-out validation set. In our experiments, we show that PCEE offers a simple yet computationally efficient approach that provides better control over performance than standard confidence-based approaches, and allows us to scale up model sizes to yield performance gain while reducing the computational cost.

Figures

Figures reproduced from arXiv: 2412.19325 by the authors.

Figure 1
Figure 1. Larger models coupled with early exiting can achieve lower prediction errors for the same computa￾tional budget compared to smaller models. This plot shows prediction error (%) versus average flops used for different MSDNET sizes on CIFAR-10: small (4 layers) and large (8 layers). Various exiting strategies are com￾pared: ours (PCEE, PCEE-WS) and Oracle (exiting as soon as a layer’s prediction matches that of the fi… view at source ↗
Figure 2
Figure 2. Heatmap of the layers used by an Oracle EE [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Confidence levels across different layers of a [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Reliability Diagrams for Layers 1, 5, 8 of MSDN [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: PCEE: The structural overview of PCEE. In a multi-layer model with exit points at each layer, the input representation ri is processed through an exit layer block Ei . The exit layer calculates a confidence score ci and uses a reliability diagram (confidence-to-accurac…
Figure 6
Figure 6. Figure 6: Performance of three MSDNET models (Small, Medium, and Large) evaluated with different thresholds. Each model exits with one of the following methods: confidence (blue), PCEE (orange), and PCEE-WS (green). The threshold values correspond to confidence levels that trans…
Figure 7
Figure 7. Figure 7: The evolution of train and test errors for ViT on CIFAR-10. The vertical dashed line is where we plotted the [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Reliability diagrams for MSDNet-Large and ViT on CIFAR datasets [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Reliability diagrams for MSDNet Small and Large on CIFAR-100 [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: The plot shows the performance of MSDNet Small and Large evaluated with different threshold values on [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]
Figure 11
Figure 11. Figure 11: Benefits of scaling up model size coupled with EE on inference efficiency for MSDNet on three datasets: [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Benefits of scaling up model size coupled with EE on inference efficiency for MSDNet on three datasets: [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Reliability Diagrams for Layers 1, 5, 9, 12 of ViT with 12 layers on [PITH_FULL_IMAGE:figures/full_fig_p018_13.png]
Figure 14
Figure 14. Figure 14: Performance of pre-trained ViT on ImageNet as a function of the selected threshold. [PITH_FULL_IMAGE:figures/full_fig_p018_14.png]
Figure 15
Figure 15. Figure 15: Accuracy/compute trade-off for pre-trained ViT on ImageNet, averaged over 5 runs and with points on the Pareto [PITH_FULL_IMAGE:figures/full_fig_p019_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    M., Mallinar, N., Lucas, J., and Nakkiran, P

    Carrell, A. M., Mallinar, N., Lucas, J., and Nakkiran, P. The calibration generalization gap. ICML 2022 Workshop on Distribution-Free Uncertainty Quantification, 2022

    work page 2022

  3. [3]

    Accelerating large language model decoding with speculative sampling

    Chen, C., Borgeaud, S., Irving, G., Lespiau, J.-B., Sifre, L., and Jumper, J. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    arXiv 2023

  4. [4]

    and Ge, R

    Chidambaram, M. and Ge, R. On the limitations of temperature scaling for distributions with overlaps. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024 . OpenReview.net, 2024. URL https://openreview.net/forum?id=zavLQJ1XjB

    work page 2024

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  6. [6]

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 0 30318--30332, 2022

    2022

  7. [7]

    Qlora: Efficient finetuning of quantized llms

    Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020

    work page 2020

Show all 50 references
  1. [9]

    Depth-adaptive transformer

    Elbayad, M., Gu, J., Grave, E., and Auli, M. Depth-adaptive transformer. arXiv preprint arXiv:1910.10073, 2019 a

    1910 arXiv

  2. [10]

    Depth-adaptive transformer

    Elbayad, M., Gu, J., Grave, E., and Auli, M. Depth-adaptive transformer. International Conference on Learning Representations, 2019 b

    2019

  3. [11]

    O., Hajimirsadeghi, H., and Abdi, A

    Feng, L., Ahmed, M. O., Hajimirsadeghi, H., and Abdi, A. Towards better selective classification. arXiv preprint arXiv:2206.09034, 2022

    2022 arXiv

  4. [12]

    Compressing BERT : Studying the effects of weight pruning on transfer learning

    Gordon, M., Duh, K., and Andrews, N. Compressing BERT : Studying the effects of weight pruning on transfer learning. In Gella, S., Welbl, J., Rei, M., Petroni, F., Lewis, P., Strubell, E., Seo, M., and Hajishirzi, H. (eds.), Proceedings of the 5th Workshop on Representation Le...

    2020 doi

  5. [13]

    and Bagnell, D

    Grubb, A. and Bagnell, D. Speedboost: Anytime prediction with uniform near-optimality. In Artificial Intelligence and Statistics, pp.\ 458--466. PMLR, 2012

    2012

  6. [14]

    Minillm: Knowledge distillation of large language models

    Gu, Y., Dong, L., Wei, F., and Huang, M. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2023

    2023

  7. [15]

    Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. On calibration of modern neural networks. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp.\ 1321--1330. PMLR...

    2017

  8. [16]

    R., and Koyuncu, E

    Görmez, A., Dasari, V. R., and Koyuncu, E. E ^2 cm: Early exit via class means for efficient supervised and unsupervised learning. arXiv preprint arXiv: 2103.01148, 2021

    2021 arXiv

  9. [17]

    Learning to Weight Samples for Dynamic Early-Exiting Networks, pp.\ 362--378

    Han, Y., Pu, Y., Lai, Z., Wang, C., Song, S., Cao, J., Huang, W., Deng, C., and Huang, G. Learning to Weight Samples for Dynamic Early-Exiting Networks, pp.\ 362--378. Springer Nature Switzerland, 2022. doi:10.1007/978-3-031-20083-0_22. URL https://link.springer.com/content/pd...

    2022 doi

  10. [18]

    A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration

    Hebbalaguppe, R., Prakash, J., Madan, N., and Arora, C. A stitch in time saves nine: A train-time regularizing loss for improved neural network calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 16081--16090, June 2022

    2022

  11. [19]

    Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem

    Hein, M., Andriushchenko, M., and Bitterwolf, J. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. Computer Vision and Pattern Recognition, 2018. doi:10.1109/CVPR.2019.00013

    2018

  12. [20]

    Distilling the knowledge in a neural network

    Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  13. [21]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  14. [22]

    Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes

    Hsieh, C.-Y., Li, C.-L., Yeh, C.-K., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023

    2023 arXiv

  15. [23]

    Huang, G., Chen, D., Li, T., Wu, F., Maaten, L., and Weinberger, K. Q. Multi-scale dense networks for resource efficient image classification. International Conference on Learning Representations, 2017

    2017

  16. [24]

    Adaptive deep neural network inference optimization with eenet

    Ilhan, F., Chow, K.-H., Hu, S., Huang, T., Tekin, S., Wei, W., Wu, Y., Lee, M., Kompella, R., Latapie, H., Liu, G., and Liu, L. Adaptive deep neural network inference optimization with eenet. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (...

    2024

  17. [25]

    U., Zhang, D., and Nalisnick, E

    Jazbec, M., Allingham, J. U., Zhang, D., and Nalisnick, E. Towards anytime classification in early-exit architectures by enforcing conditional monotonicity. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Akslsk891N

    2023

  18. [26]

    To trust or not to trust a classifier

    Jiang, H., Kim, B., Guan, M., and Gupta, M. To trust or not to trust a classifier. Advances in neural information processing systems, 31, 2018

    2018

  19. [27]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  20. [28]

    Shallow-deep networks: Understanding and mitigating network overthinking

    Kaya, Y., Hong, S., and Dumitras, T. Shallow-deep networks: Understanding and mitigating network overthinking. International Conference on Machine Learning, 2018

    2018

  21. [29]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  22. [30]

    Laskaridis, S., Kouris, A., and Lane, N. D. Adaptive inference through early-exit networks: Design, challenges and directions. Proceedings of the 5th International Workshop on Embedded and Mobile Deep Learning, 2021. URL https://api.semanticscholar.org/CorpusID:235377371

    2021

  23. [31]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  24. [32]

    P., Salakhutdinov, R

    Liu, Z., Wang, Z., Liang, P. P., Salakhutdinov, R. R., Morency, L.-P., and Ueda, M. Deep gamblers: Learning to abstain with portfolio theory. Advances in Neural Information Processing Systems, 32, 2019

    2019

  25. [33]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  26. [34]

    The era of 1-bit llms: All large language models are in 1.58 bits

    Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. The era of 1-bit llms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764, 2024

    2024 arXiv

  27. [35]

    Fixing overconfidence in dynamic neural networks

    Meronen, L., Trapp, M., Pilzer, A., Yang, L., and Solin, A. Fixing overconfidence in dynamic neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp.\ 2680--2690, January 2024

    2024

  28. [36]

    P., Cooper, G

    Naeini, M. P., Cooper, G. F., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI'15, pp.\ 2901–2907. AAAI Press, 2015. ISBN 0262511290

    2015

  29. [37]

    On-the-fly operation batching in dynamic computation graphs

    Neubig, G., Goldberg, Y., and Dyer, C. On-the-fly operation batching in dynamic computation graphs. Advances in Neural Information Processing Systems, 30, 2017

    2017

  30. [38]

    W., Zhang, L., Jerfel, G., and Tran, D

    Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., and Tran, D. Measuring calibration in deep learning. In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019 , pp.\ 38--41. Computer Vision Foundati...

    2019

  31. [39]

    Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift

    Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, J. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019

    2019

  32. [40]

    and McDaniel, P

    Papernot, N. and McDaniel, P. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765, 2018

    2018 arXiv

  33. [41]

    Consistent accelerated inference via confident adaptive transformers

    Schuster, T., Fisch, A., Jaakkola, T., and Barzilay, R. Consistent accelerated inference via confident adaptive transformers. arXiv preprint arXiv:2104.08803, 2021

    2021 arXiv

  34. [42]

    Confident adaptive language modeling

    Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., and Metzler, D. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35: 0 17456--17472, 2022

    2022

  35. [43]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. International Conference on Learning Representations, 2023. doi:10.48550/arXiv.2306.11695

    2023 doi arXiv

  36. [44]

    and Naganuma, H

    Tada, K. and Naganuma, H. How image corruption and perturbation affect out-of-distribution generalization and calibration. In 2023 International Joint Conference on Neural Networks (IJCNN), pp.\ 1--6, 2023. doi:10.1109/IJCNN54540.2023.10191806

    2023

  37. [45]

    Branchynet: Fast inference via early exiting from deep neural networks

    Teerapittayanon, S., McDanel, B., and Kung, H.-T. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd international conference on pattern recognition (ICPR), pp.\ 2464--2469. IEEE, 2016

    2016

  38. [46]

    Open-set recognition: A good closed-set classifier is all you need? 2021

    Vaze, S., Han, K., Vedaldi, A., and Zisserman, A. Open-set recognition: A good closed-set classifier is all you need? 2021

    2021

  39. [47]

    Calibration in deep learning: A survey of the state-of-the-art

    Wang, C. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv: 2308.01222, 2023

    2023 arXiv

  40. [48]

    Rethinking calibration of deep neural networks: Do not be afraid of overconfidence

    Wang, D.-B., Feng, L., and Zhang, M.-L. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Neural Information Processing Systems, 34: 0 11809--11820, 2021

    2021

  41. [49]

    Emergent abilities of large language models

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022

    2022 arXiv

  42. [50]

    Pytorch image models

    Wightman, R. Pytorch image models. https://github.com/huggingface/pytorch-image-models, 2019

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.