REVIEW 4 major objections 8 minor 33 references
Enhancing Monte Carlo Dropout Performance for Uncertainty Quantification
T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Adding a predictive-entropy penalty and hyperparameter search makes Monte Carlo Dropout 2–3% more accurate and better calibrated.
desk verdict The optimizer sweep is real but the entropy-augmented loss cannot produce the claimed uncertainty separation, and the paper's own Table 4 contradicts the central mechanism. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the modified loss of Eq. (9): per sample, the binary cross-entropy computed on the mean prediction over $M$ dropout forward passes plus the sum of per-pass predictive entropies, $\sum_{m=1}^M \mathrm{PE}(x_b^{(m)})$. Predictive entropy is $\mathrm{PE}=-\sum_c \mu_{\mathrm{pred},c} \log \mu_{\mathrm{pred},c}$, ranging from 0 for a confident one-class prediction to 1 for a flat distribution. This loss doubles as the fitness function for GWO, BO, and PSO, which search over dropout rates $P_1, P_2$ and hidden-layer sizes $L_1, L_2$. The intended effect is that the entropy penalty pushes the model to be decisive while the search finds a configuration that balances accuracy and calibration.
What would settle it
Train the same architectures and optimizers with the predictive-entropy term removed from Eq. (9) on a fixed validation split; if UAcc and ECE are not materially worse, the entropy penalty is not carrying the claimed benefit.
Extended reading notes
Core claim
The paper's central claim is that conventional MCD's main weakness—poorly calibrated uncertainty tied to a hand-chosen dropout rate—can be mitigated by jointly searching hyperparameters with GWO, BO, or PSO while training with a loss that sums binary cross-entropy and the predictive entropies of the Monte Carlo passes. The authors report average gains of 2–3% over the MCD baseline in both accuracy and Uncertainty Accuracy across backbones and datasets, and lower ECE for most optimized variants. They also report larger separation between the predictive-entropy distributions of correct and incorrect predictions, which they read as the model being uncertain when wrong and confident when right.
Load-bearing premise
The load-bearing premise is that adding the summed per-pass predictive entropy to the binary cross-entropy loss will make the model uncertain on misclassified samples and confident on correct ones, even though the loss contains no term conditioned on label correctness.
Editorial extensions
If this is right
- Optimized MCD variants would give classifiers that are about 2–3% more accurate and better calibrated, which matters for medical imaging and autonomous systems where a confident wrong answer is costly.
- Higher Uncertainty Accuracy means a model's confidence is a more reliable indicator of correctness, so downstream systems can send uncertain predictions to human review.
- The recipe applies to any dropout-based architecture, since it changes only the loss and the hyperparameter search.
- The uncertainty-aware loss can serve as a generic fitness function for other hyperparameter optimizers beyond the three tested.
- Lower ECE for most optimized variants means the reported probability scores can be used as trustworthy confidence thresholds.
Reading between the lines
- Editorial inference: because the entropy term in Eq. (9) penalizes high entropy for every sample rather than conditioning on label correctness, the mechanism that produces high entropy specifically on misclassified samples is indirect; the paper's own Table 4 shows the entropy term alone sometimes worsens ECE (e.g., Myocardit ResNet50: 2.81 vs 1.97).
- Editorial inference: if the entropy penalty contributes little on its own, much of the reported gain may come from the hyperparameter search finding better dropout rates; a controlled ablation separating the two terms would settle that.
- Editorial inference: a natural testable extension is to weight the entropy penalty by prediction correctness, explicitly rewarding high uncertainty on errors and low uncertainty on correct predictions.
- Editorial inference: the same loss-plus-search recipe could transfer to regression by replacing predictive entropy with predictive variance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes to improve Monte Carlo Dropout (MCD) for classification by (i) adding a predictive-entropy penalty to the binary cross-entropy loss and (ii) optimizing dropout rates and hidden-layer sizes with Grey Wolf Optimizer (GWO), Bayesian Optimization (BO), or Particle Swarm Optimization (PSO). Experiments are run on a synthetic Circles dataset and three real datasets (Myocardit, Cats vs Dogs, Wisconsin) with DenseNet121, ResNet50, and VGG16 feature extractors, reporting accuracy, AUC, UAcc, and ECE. The authors claim a 2-3% average improvement over the MCD baseline in both conventional accuracy and uncertainty accuracy, as well as better calibration. The core novelty is the uncertainty-aware loss in Eq. (9) combined with hyperparameter search.
Significance. If the central claims were supported, the proposed approach could offer a practical way to obtain better-calibrated MCD uncertainties for safety-critical applications. The paper deserves credit for evaluating on multiple datasets and backbones, for reporting ECE alongside accuracy, and for systematically comparing three hyperparameter optimizers. However, the central mechanism is not implemented by the proposed loss, the evaluation metric is partly self-referential, the baseline comparison is incomplete, and several reported tables contradict the qualitative claims. As a result, the reported improvements do not support the stated mechanism.
major comments (4)
- [Section 4, Eq. (9)] The proposed loss adds an unconditional predictive-entropy penalty, summed over MC passes, to binary cross-entropy. This penalty is minimized when every sample's softmax output is close to a one-hot vector, irrespective of whether the sample is correctly classified. The paper's stated mechanism (Section 4, Fig. 1) is that incorrect predictions should exhibit high PE and correct predictions low PE, but the loss contains no term conditioned on label correctness. On misclassified samples, the penalty pushes the model toward overconfident wrong predictions. Table 4 shows the PE term alone often worsens ECE (e.g., Myocardit ResNet50: 2.81 vs 1.97 for MCD; Cats vs Dogs ResNet50: 1.80 vs 1.48), which is direct evidence against the assumed benefit.
- [Section 3.2, Eq. (3); Section 4, Eq. (9)] UAcc is computed by thresholding predictive entropy, and the proposed loss directly penalizes predictive entropy. Therefore UAcc is partly a function of the quantity being minimized; a model that is globally more confident will tend to have more correct-and-certain (CC) samples, inflating UAcc without necessarily providing better uncertainty for misclassified samples. The reported UAcc gains are thus not independent evidence of better uncertainty quantification. The paper should evaluate with a metric that is not directly optimized, or should report separation of error/success distributions, which Table 5 actually contradicts (see below).
- [Section 5, Tables 3 and 4] The MCD baseline hyperparameters are not reported. The paper states 'For consistency, we used the same architecture for all five models' (Section 5.1), but Table 3 lists optimized L1, L2, P1, P2 only for the proposed methods; no corresponding configuration is given for the MCD or MCD-plus-PE baselines. Without knowing the baseline's dropout rates and layer sizes, the claimed 2-3% improvement could be entirely due to hyperparameter search rather than to the proposed uncertainty-aware loss or the specific optimizers.
- [Section 5.2, Table 5] The text claims that the proposed methods 'help the two distributions being better differentiated (higher value for the distance of the centers),' but the table entries do not consistently support this. For Myocardit ResNet50, the distance for MCD is 0.256 while GWO and PSO give 0.214 and 0.206, respectively; for Cats vs Dogs ResNet50, GWO/BO/PSO distances (0.379, 0.378, 0.375) are essentially unchanged from MCD (0.375). These results undermine the qualitative claim of improved separation of correct and incorrect predictions.
minor comments (8)
- [Section 2.1] The statement '1000 samples from each dataset' is unclear, as Circles is a single synthetic dataset rather than multiple datasets.
- [Section 4, Eq. (9)] Equation (9) contains a typographical error: the second binary cross-entropy term shows 'hatµ' instead of the correct predicted probability.
- [Sections 3.3 and 4] The symbol M is used for both the number of MC passes (Eq. 7) and the number of ECE bins (Eq. 4); please disambiguate the notation.
- [Section 5.1 and Table 1] The text refers to 'MCD plus entropy' while the table and later sections use 'MCD plus PE'; please unify the terminology.
- [Figure 4 caption] The caption contains a typo: 'wight' should be 'weights'.
- [Section 3.1, Eq. (2)] The predictive entropy is said to vary between 0 and 1, but this depends on the logarithm base, which is not specified.
- [References] References [23] and [27] are the same paper (Lakshminarayanan et al.) and should be merged or cross-referenced.
- [Section 5, Fig. 6] The choice of UAcc threshold 0.5 is not justified, and since UAcc is threshold-dependent (as demonstrated in Fig. 3), a sensitivity analysis or a stated selection criterion is needed.
Circularity Check
No significant circularity: the loss's per-pass entropy term is not the same quantity as the thresholded averaged-entropy UAcc metric, and the calibration claim is tested by independent ECE; remaining concerns are unsupported mechanism and possible data leakage, not circular derivation.
full rationale
The paper's derivation chain does not reduce to its inputs. The proposed loss (Eq. 9) adds the sum of per-pass predictive entropies to binary cross-entropy, while the evaluation metric UAcc (Eq. 3) is defined by thresholding the predictive entropy of the MC-averaged softmax (Eqs. 1, 2, and 7). These are related but distinct quantities: minimizing per-pass entropy can increase disagreement among passes and thereby raise averaged entropy, so the loss does not directly optimize UAcc by construction. The 'significantly better calibration' claim is assessed by ECE (Eq. 4), which is independent of the loss's PE term and is often worsened by PE augmentation in Table 4 (e.g., Myocardit ResNet50: 2.81 vs 1.97), indicating that any ECE gain comes from the hyperparameter search rather than from a tautological loss/metric pairing. The MCD-plus-PE baseline is drawn from the authors' own prior work ([28, 29]), but it is used as a comparison point, not as a load-bearing premise that forces the paper's conclusions. The genuine weaknesses are (i) Section 4's claim that the loss 'ensures that incorrect predictions exhibit high PE' is not implemented by Eq. 9, since the PE term has no label-conditioned weighting; and (ii) the hyperparameter optimizers appear to be evaluated on the same data used for search, with no reported MCD baseline hyperparameters, which threatens the 2-3% attribution. These are correctness and reproducibility concerns, not circularity under the definitions used here.
Assumptions & free parameters
free parameters (5)
- L1 (first hidden layer size) =
210-256 across datasets/optimizers (Table 3)
- L2 (second hidden layer size) =
29-53 across datasets/optimizers (Table 3)
- P1 (dropout rate, layer 1) =
0.13-0.50 across datasets/optimizers (Table 3)
- P2 (dropout rate, layer 2) =
0.12-0.51 across datasets/optimizers (Table 3)
- UAcc threshold =
0.5 for Tables 4-5 and Fig. 6; varied in Figs. 3 and 5
assumptions (3)
- domain assumption Predictive entropy is a valid and sufficient measure of predictive uncertainty for binary classification.
- domain assumption The pretrained ImageNet features (VGG16, ResNet50, DenseNet121) plus PCA to 100 components form a suitable representation for the small medical and image datasets.
- domain assumption Hyperparameters selected by minimizing the training/validation loss Eq. (9) generalize to the test set.
Cite this review
Pith. "Pith review of Enhancing Monte Carlo Dropout Performance for Uncertainty Quantification." pith.science (2026). https://pith.science/paper/2LHU2UWR
@misc{pith2026250515671,
author = {Pith},
title = {Pith review of: Enhancing Monte Carlo Dropout Performance for Uncertainty Quantification},
year = {2026},
howpublished = {\url{https://pith.science/paper/2LHU2UWR}},
note = {Machine review of arXiv:2505.15671}
}
read the original abstract
Knowing the uncertainty associated with the output of a deep neural network is of paramount importance in making trustworthy decisions, particularly in high-stakes fields like medical diagnosis and autonomous systems. Monte Carlo Dropout (MCD) is a widely used method for uncertainty quantification, as it can be easily integrated into various deep architectures. However, conventional MCD often struggles with providing well-calibrated uncertainty estimates. To address this, we introduce innovative frameworks that enhances MCD by integrating different search solutions namely Grey Wolf Optimizer (GWO), Bayesian Optimization (BO), and Particle Swarm Optimization (PSO) as well as an uncertainty-aware loss function, thereby improving the reliability of uncertainty quantification. We conduct comprehensive experiments using different backbones, namely DenseNet121, ResNet50, and VGG16, on various datasets, including Cats vs. Dogs, Myocarditis, Wisconsin, and a synthetic dataset (Circles). Our proposed algorithm outperforms the MCD baseline by 2-3% on average in terms of both conventional accuracy and uncertainty accuracy while achieving significantly better calibration. These results highlight the potential of our approach to enhance the trustworthiness of deep learning models in safety-critical applications.
Reference graph
Works this paper leans on
-
[1]
nature 542(7639), 115–118 (2017)
Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. nature 542(7639), 115–118 (2017)
2017
-
[2]
In: 2023 24th International Conference on Digital Signal Processing (DSP), pp
Asgharnezhad, H., Shamsi, A., Bakhshayeshi, I., Alizadehsani, R., Chamaani, S., Alinejad-Rokny, H.: Improving ppg signal classification with machine learning: The power of a second opinion. In: 2023 24th International Conference on Digital Signal Processing (DSP), pp. 1–5 (2023). IEEE
work page 2023
-
[3]
Drug discovery today 23(6), 1241–1250 (2018)
Chen, H., Engkvist, O., Wang, Y., Olivecrona, M., Blaschke, T.: The rise of deep learning in drug discovery. Drug discovery today 23(6), 1241–1250 (2018)
work page 2018
-
[4]
Advances in neural information processing systems 25, 1097–1105 (2012)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25, 1097–1105 (2012)
work page 2012
-
[5]
arXiv preprint arXiv:2406.13875 (2024)
Osowiechi, D., Noori, M., Hakim, G.A.V., Yazdanpanah, M., Bahri, A., Cher- aghalikhani, M., Dastani, S., Beizaee, F., Ayed, I.B., Desrosiers, C.: Watt: Weight average test-time adaptation of clip. arXiv preprint arXiv:2406.13875 (2024)
arXiv 2024
-
[6]
arXiv preprint arXiv:2407.20891 (2024)
Doan, B.G., Shamsi, A., Guo, X.-Y., Mohammadi, A., Alinejad-Rokny, H., Sejdi- novic, D., Teney, D., Ranasinghe, D.C., Abbasnejad, E.: Bayesian low-rank learning (bella): A practical approach to bayesian neural networks. arXiv preprint arXiv:2407.20891 (2024)
arXiv 2024
-
[7]
In: Extreme Man-made and Natural Hazards in Dynamics of Structures, pp
Matthies, H.G.: Quantifying uncertainty: modern computational representation of probability and applications. In: Extreme Man-made and Natural Hazards in Dynamics of Structures, pp. 105–135. Springer, . (2007)
work page 2007
-
[8]
Neal, R.M.: Bayesian Learning for Neural Networks vol. 118. Springer, . (2012) 18
work page 2012
Show all 33 references
-
[9]
Advances in neural information processing systems 29, 4134–4142 (2016)
Springenberg, J.T., Klein, A., Falkner, S., Hutter, F.: Bayesian optimization with robust bayesian neural networks. Advances in neural information processing systems 29, 4134–4142 (2016)
2016
-
[10]
In: International Conference on Machine Learning, pp
Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural network. In: International Conference on Machine Learning, pp. 1613–1622 (2015). PMLR
2015
-
[11]
Advances in neural information processing systems 24 (2011)
Graves, A.: Practical variational inference for neural networks. Advances in neural information processing systems 24 (2011)
2011
-
[12]
In: Proceedings of the 22nd International Conference on Machine Learning, pp
Rasmussen, C.E., Quinonero-Candela, J.: Healing the relevance vector machine through augmentation. In: Proceedings of the 22nd International Conference on Machine Learning, pp. 689–696 (2005)
2005
-
[13]
In: International Conference on Machine Learning, pp
Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In: International Conference on Machine Learning, pp. 1050–1059 (2016). PMLR
2016
-
[14]
Photogrammetric Engineering & Remote Sensing 82(3), 189–197 (2016)
Coulston, J.W., Blinn, C.E., Thomas, V.A., Wynne, R.H.: Approximating prediction uncertainty for random forest regression models. Photogrammetric Engineering & Remote Sensing 82(3), 189–197 (2016)
2016
-
[15]
Advances in engi- neering software 69, 46–61 (2014)
Mirjalili, S., Mirjalili, S.M., Lewis, A.: Grey wolf optimizer. Advances in engi- neering software 69, 46–61 (2014)
2014
-
[16]
In: System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31–September 4, 1981, pp
Mockus, J.: The bayesian approach to global optimization. In: System Modeling and Optimization: Proceedings of the 10th IFIP Conference New York City, USA, August 31–September 4, 1981, pp. 473–481 (2005). Springer
2005
-
[17]
Evolutionary computation 25(1), 1–54 (2017)
Bonyadi, M.R., Michalewicz, Z.: Particle swarm optimization for single objec- tive continuous space problems: a review. Evolutionary computation 25(1), 1–54 (2017)
2017
-
[18]
Kaggle (2021)
Sharifrazi, D.: Myocarditis Dataset. Kaggle (2021). https://www.kaggle.com/ datasets/danialsharifrazi/myocarditis-dataset
2021
-
[19]
Mathematical Biosciences and Engineering 19(3), 2381– 2402 (2022)
Sharifrazi, D., Alizadehsani, R., Joloudari, J.H., Band, S.S., Hussain, S., Sani, Z.A., Hasanzadeh, F., Shoeibi, A., Dehzangi, A., Sookhak, M., et al.: Cnn-kcl: Automatic myocarditis diagnosis using convolutional neural network combined with k-means clustering. Mathematical Bi...
2022
-
[20]
https://www.microsoft
Microsoft Corporation: Kaggle Cats and Dogs Dataset. https://www.microsoft. com/en-us/download/details.aspx?id=54765. Version 1.0, Published July 15, 2024 (2024) 19
2024
-
[21]
In: Biomedical Image Processing and Biomedical Visualization, vol
Street, W.N., Wolberg, W.H., Mangasarian, O.L.: Nuclear feature extraction for breast tumor diagnosis. In: Biomedical Image Processing and Biomedical Visualization, vol. 1905, pp. 861–870 (1993). SPIE
1993
-
[22]
The journal of machine learning research 15(1), 1929–1958 (2014)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1), 1929–1958 (2014)
2014
-
[23]
arXiv preprint arXiv:1612.01474 (2016)
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv:1612.01474 (2016)
2016 arXiv
-
[24]
arXiv preprint arXiv:1705.07832 (2017)
Gal, Y., Hron, J., Kendall, A.: Concrete dropout. arXiv preprint arXiv:1705.07832 (2017)
2017 arXiv
-
[25]
Scientific Reports 12(1), 1–11 (2022)
Asgharnezhad, H., Shamsi, A., Alizadehsani, R., Khosravi, A., Nahavandi, S., Sani, Z.A., Srinivasan, D., Islam, S.M.S.: Objective evaluation of deep uncertainty predictions for covid-19 detection. Scientific Reports 12(1), 1–11 (2022)
2022
-
[26]
In: International Conference on Machine Learning, pp
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neu- ral networks. In: International Conference on Machine Learning, pp. 1321–1330 (2017). PMLR
2017
-
[27]
Advances in neural information processing systems 30 (2017)
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems 30 (2017)
2017
-
[28]
arXiv preprint arXiv:2110.03260 (2021)
Shamsi, A., Asgharnezhad, H., Abdar, M., Tajally, A., Khosravi, A., Nahavandi, S., Leung, H.: Improving mc-dropout uncertainty estimates with calibration error- based optimization. arXiv preprint arXiv:2110.03260 (2021)
2021 arXiv
-
[29]
Neural Computing and Applications 35(30), 22179–22188 (2023)
Shamsi, A., Asgharnezhad, H., Bouchani, Z., Jahanian, K., Saberi, M., Wang, X., Razzak, I., Alizadehsani, R., Mohammadi, A., Alinejad-Rokny, H.: A novel uncertainty-aware deep learning technique with an application on skin cancer diagnosis. Neural Computing and Applications 35...
2023
-
[30]
arXiv e-prints (2014)
Simonyan, K., Zisserman, A.: Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv e-prints (2014)
2014
-
[31]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recogni- tion. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
2016
-
[32]
In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017) 20
2017
-
[33]
Jolliffe, I.T., Cadima, J.: Principal component analysis: a review and recent developments. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 374(2065), 20150202 (2016) 21 T able 5: The centers of the two distributions, µ1 and µ...
2016
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.