{"id":"58fa42be-a228-479b-a0fe-a829d88af054","arxiv_id":"1908.08972","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A decoupled Bayesian neural network that maps a DNN's logits to calibrated probabilities consistently reduces expected calibration error, at the cost of some accuracy loss on complex datasets.","lead":"This paper trains a small Bayesian neural network on top of a pre-trained classifier's logits to recalibrate its probabilities. The method improves expected calibration error over temperature scaling on several image datasets, but it sometimes lowers accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: a deterministic MLP calibrator of the same architecture is never compared, so the reported ECE gains cannot be attributed to Bayesian averaging.","rationale":"The reader identified the logit-only input as the weakest assumption, which is a real limitation but not the most load-bearing issue for the central claim. The more fundamental gap is that the experimental protocol never isolates the Bayesian component. Since the paper argues that uncertainty handling is the reason complex models can improve calibration, a fair control is a point-estimate neural network calibrator with identical architecture and training data. The absence of this control weakens the causal interpretation, but it does not invalidate the empirical observation that the proposed pipeline achieves lower ECE than TS on the reported benchmarks. The disclosed validation-based K selection is an additional confounder: it is a post-hoc tuning mechanism that can improve calibration independently of Bayesian marginalization. Therefore the paper still warrants a conditional verdict pending the recommended ablation, matching the reader's overall assessment.","tokens_in":20506,"tokens_out":7381,"duration_ms":82270,"concrete_test":"Train a deterministic MLP calibrator with the same architecture as the BNN stage (same hidden layers, ReLU activations, softmax output) on the same logits, using MAP training with a Gaussian prior whose precision matches the BNN's prior and beta. Tune the number of stochastic forward passes (or an equivalent temperature) on the same validation split used for K selection, then compare test ECE and accuracy on CIFAR10, CIFAR100, and SVHN. If the deterministic MLP matches or beats the BNN's ECE, the reported improvement is not specifically due to Bayesian averaging.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a decoupled Bayesian stage, implemented as a BNN, consistently improves calibration over temperature scaling and other baselines. The experiments in Tables 6 and 7 compare the full MFVI/MFVILR BNN pipeline against TS, ensembles, MC dropout, and MMCE, but they never compare against a non-Bayesian neural network calibrator with the same architecture trained on the same logits. Table 2 does compare Bayesian versus point-estimate linear logistic regression, but the actual calibration stage used in the main results is a multilayer BNN; the linear comparison does not isolate the effect of Bayesian marginalization in the deeper model. Section 4.5 further discloses that the mean-field posterior under-estimates variance and that the number of Monte Carlo samples K is selected on a validation set to mitigate this effect. This validation-based K selection is an additional tuning step not present in the baselines, and it could be the actual source of the ECE improvement. Without a MAP-trained MLP calibrator of identical capacity and without an ablation that fixes K (or replaces averaging with a single point estimate), the observed gains cannot be assigned to Bayesian inference as opposed to the extra representational capacity of a learned logit-to-logit map plus validation-based tuning. The claim that 'incorporating uncertainty provides more reliable probabilistic models' is therefore under-supported by the experimental design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a decoupled calibration stage for pre-trained DNN classifiers: a Bayesian Neural Network (BNN) trained on the DNN's logits maps them to calibrated logits, with the posterior approximated by mean-field variational inference (MFVI or MFVILR) and predictions averaged over Monte Carlo samples. The claimed contribution is that this Bayesian stage consistently improves expected calibration error (ECE) over temperature scaling and other baselines across seven image classification datasets, while retaining the accuracy of the original DNN in most cases. The paper also contains a toy experiment contrasting Bayesian and point-estimate models, a linear logistic comparison, an analysis of variance under-estimation, and a discussion of the validation-based selection of the number of Monte Carlo samples K.","tokens_in":20750,"tokens_out":2251,"duration_ms":26215,"significance":"If the empirical claims are supported, the work is significant because it challenges the view that calibration spaces for DNNs are inherently simple and that temperature scaling should be preferred over more complex calibrators. The paper provides a broad experimental matrix, including several architectures and datasets, and the authors release code for reproducibility. It also honestly discloses known limitations of the mean-field approximation, including accuracy degradation on fine-grained tasks and the need for validation-based selection of K. However, the central attribution of the calibration gains to Bayesian averaging is weakened by the absence of a deterministic neural calibrator control, by the lack of variance estimates for the reported ECE improvements, and by asymmetric hyperparameter treatment (validation-based K) relative to the baselines.","major_comments":[{"comment":"The experimental design never compares the proposed BNN calibrator against a deterministic MLP calibrator of identical architecture trained on the same logits with the same optimizer and hyperparameter-selection procedure. Without this control, the reported ECE gains cannot be attributed to Bayesian marginalization as opposed to the increased representational capacity of a learned logit-to-logit map. I ask the authors to add a MAP-trained (or otherwise non-Bayesian) MLP calibrator with the same hidden-layer configuration, the same ELBO-derived loss without the KL term (or an equivalent regularized loss), and the same validation-based selection of predictive samples, and to report ECE and accuracy for that model in Tables 6 and 7.","section":"Section 5.6, Tables 6 and 7"},{"comment":"The number of Monte Carlo predictive samples K is selected on a validation set to mitigate variance under-estimation, whereas no comparable extra tuning step is applied to temperature scaling or the other baselines. This asymmetry is a plausible source of the ECE improvement and is not controlled for. The authors should either fix K across methods, or apply the same validation-based selection to the baselines (e.g., selecting the temperature on the same validation criterion), and report the sensitivity of ECE to K in the main tables.","section":"Section 4.5 and Section 5.4"},{"comment":"No error bars, confidence intervals, or repeated-run statistics are reported for any ECE or accuracy value, although the text states that accuracy degradation is 'slight' on CIFAR100, BIRDS, and CARS. Given that Table 3 shows accuracy drops of roughly 2.5, 2.0, and 3.5 points on those tasks, and that several ECE differences between methods in Tables 6 and 7 are small (e.g., SVHN rows), the claims of consistent calibration improvement and only slight accuracy loss require variance estimates over multiple training runs or at least over the selected BNN architectures.","section":"Table 3 and Section 5.5"},{"comment":"It is unclear which data split is used to calibrate each baseline. The text states that a validation set is used to select K for the proposed method but does not explicitly state whether temperature scaling, network ensembles, and the other baselines are fit on the same calibration split or on the training split. The authors should specify this precisely and, if the splits differ, rerun the comparisons on a common calibration set.","section":"Section 5.1 and Tables 6-7"}],"minor_comments":[{"comment":"The table header reads 'MVFI' and 'MVFILR' instead of 'MFVI' and 'MFVILR'; this should be corrected for consistency with the text.","section":"Table 4"},{"comment":"The phrase 'miss-classified' appears in the qualitative analysis and should be 'misclassified'.","section":"Section 5.7"},{"comment":"The acronym 'NNL' in the Introduction should be 'NLL' (negative log-likelihood); the paper otherwise uses 'NLL'.","section":"Introduction, Section 2"},{"comment":"Equation (6) is written as an argmax over q of the ELBO with a beta-weighted KL term, but the text later refers to beta as a hyperparameter selected from a set; the authors should clarify whether beta is optimized or grid-searched, and how it interacts with the validation-based selection of K.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and the central idea is promising, but the missing deterministic-MLP control is a load-bearing gap for the paper's main attribution claim. I would be willing to reconsider a revised version that adds this control, reports error bars, and equalizes the hyperparameter treatment across methods. The authors' honest disclosure of the variance under-estimation problem and the code release are positives."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Juan,\n\nQuick read for your desk: the paper is a real contribution, not a breakthrough. It trains a mean-field variational BNN on the logits of a pre-trained DNN and shows, across a wide set of architectures and datasets, that this consistently lowers ECE compared to temperature scaling and a few other baselines. The breadth of experiments and the fact that code is provided are genuine assets. The discussion of variance under-estimation and the validation-based selection of K is honest and unusually candid for this literature.\n\nWhere it is soft: the central attribution. The main tables never compare against a deterministic MLP calibrator of the same architecture trained on the same logits. Table 2 does a Bayesian vs point-estimate linear comparison, but the main method is a multilayer BNN, so that does not isolate the effect of marginalization in the model actually used. The disclosed K-on-validation trick could plausibly be doing much of the work, and the extra tuning step is not applied to the baselines. So the headline claim that Bayesian inference is what drives the calibration gains is under-supported, even though the empirical method itself works. That is a fixable gap, not a fatal flaw: the method is well motivated, the paper is clear about its own limitations, and the accuracy degradation on CIFAR-100/BIRDS/CARS is acknowledged and analyzed.\n\nAlso worth noting: no error bars anywhere, and the baseline calibration-data split is not fully explicit. Those are minor-to-moderate. The claim that incorporating uncertainty provides more reliable models is a reasonable empirical conclusion for the settings tested, but it should be softened to 'this particular approximate Bayesian stage plus validation-based K selection.'\n\nVerdict: it deserves a serious referee. The method is simple, clearly described, reproducible in principle, and the empirical comparison is broad. A good referee could ask for the missing control and for error bars, and the paper would be stronger for it. I'd cite it if I worked on post-hoc calibration.\n\nCheers.","headline":"Useful decoupled BNN calibrator with consistent ECE gains over TS, but the missing deterministic-MLP control makes the 'Bayesian averaging is the cause' claim under-supported.","tokens_in":21246,"tokens_out":1136,"would_cite":true,"duration_ms":11355,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Decoupled Bayesian neural networks recalibrate deep classifiers, beating temperature scaling on seven benchmarks.","keywords":["calibration","Bayesian neural networks","decoupled recalibration","expected calibration error","variational inference","temperature scaling","image classification","uncertainty quantification"],"falsifier":"Construct two groups of test images whose deep-network logit vectors are identically distributed but whose true-class rates differ, for example distinct subpopulations that collapse to the same logits. Since the BNN is trained only on logits, it must assign the same calibrated probability to both groups, so measuring a nonzero expected calibration error within either group after calibration would falsify the logit-sufficiency assumption.","tokens_in":20297,"feed_emoji":"🎯","tokens_out":5231,"duration_ms":52396,"temperature":0.7,"pith_summary":"This paper proposes attaching a Bayesian calibration stage, a small Bayesian neural network, after a pre-trained deep classifier, feeding it only the classifier's uncalibrated logits and training it to produce calibrated class probabilities. The authors claim this decoupled Bayesian stage consistently lowers expected calibration error relative to temperature scaling and other state-of-the-art calibrators across several image-classification benchmarks. Because the Bayesian stage averages over many weight samples rather than committing to a single point estimate, it treats calibration as an uncertainty problem rather than a simple one-parameter fix. The broader claim is that complex models can improve calibration without overfitting as long as parameter uncertainty is handled properly.","feed_headline":"Bayesian recalibration beats temperature scaling on seven image sets","feed_subtitle":"A small Bayesian net remaps a pretrained model's logits, cutting calibration error well below temperature scaling's.","key_machinery":"The load-bearing object is the decoupled Bayesian calibration stage: a fully-connected Bayesian neural network with ReLU hidden units and a softmax output, whose input is the class-dimensional logit vector produced by a pre-trained deep network and whose output is a calibrated probability vector of the same dimension. Inference uses variational Bayes with a factorized Gaussian posterior approximation, optimized through the evidence lower bound using the reparameterization trick and its local reparameterization variant, and predictions are formed by averaging many softmax outputs over samples from the variational posterior. That Monte Carlo average is the mechanism that expresses parameter uncertainty in the final probabilities, and choosing the number of samples on a validation set is the device that controls the variance under-estimation of mean-field inference.","core_discovery":"At the paper's core is the claim that a Bayesian neural network used as a post-hoc recalibrator, trained on the logit vectors of a frozen, high-accuracy deep network, maps those logits to probabilities whose confidence matches empirical accuracy more closely than the network's own softmax output. The predictive distribution is a Monte Carlo average of softmax outputs over weights drawn from a factorized Gaussian variational posterior, with the number of samples chosen on a validation set to counteract variance under-estimation. In the reported experiments, this BNN recalibrator achieves the lowest expected calibration error among the compared explicit and implicit calibration techniques, including temperature scaling, deep ensembles, Monte Carlo dropout, MMCE, and VWCI, while leaving the original deep network untouched and requiring only one forward pass through it. The authors interpret this as evidence that Bayesian treatment of uncertainty is what makes calibrated predictions reliable, and that the calibration space is not inherently simple.","pith_inferences":["A natural extension the paper does not test is to feed the Bayesian stage additional deep-network features, such as penultimate-layer activations, alongside the logits; if expected calibration error drops further, the logit-only input assumption would need revision.","The validation-selected number of Monte Carlo samples behaves like a regularization knob on epistemic uncertainty, so one could study whether its optimal value tracks dataset difficulty or network capacity.","The decoupled design suggests that a single calibrated BNN might be reused across different deep networks trained on the same label space, offering a cheap route to recalibrating a whole family of models."],"forward_implications":["On the seven image benchmarks reported, the decoupled BNN yields lower expected calibration error than temperature scaling, the strongest explicit baseline, and lower than Monte Carlo dropout and deep ensembles among implicit methods.","A single forward pass through the large deep network plus several light forward passes through the BNN replaces any retraining or modification of the original network, so calibration can be improved post-hoc at modest computational cost.","Because the Bayesian stage is decoupled, it can be upgraded with richer variational approximations or better inference methods without touching the pre-trained network's architecture or weights.","If uncertainty is truly the limiting factor, then better posterior approximations than mean-field should push calibration error lower still, a direct implication the paper draws for future work."],"supporting_citations":[{"why":"Supplies the formal definition of a well-calibrated probabilistic model used to frame the problem.","marker":"[7]"},{"why":"Supplies the temperature scaling baseline and the expected calibration error metric used throughout the experiments.","marker":"[20]"},{"why":"Supplies the deep-ensemble implicit calibration baseline, also evaluated in decoupled form.","marker":"[18]"},{"why":"Supplies the Monte Carlo dropout baseline, which the paper argues is not calibrated by default.","marker":"[29]"},{"why":"Supplies the MMCE trainable calibration-cost baseline compared in the implicit-technique experiments.","marker":"[22]"},{"why":"Supplies the VWCI stochastic-inference calibration baseline compared in the implicit-technique experiments.","marker":"[23]"},{"why":"Supplies the local reparameterization trick used to build the MFVILR training estimator.","marker":"[34]"}],"fun_headline_variants":["Decoupled Bayesian net recalibrates DNN probabilities with less error","BNN post-hoc recalibrator reduces calibration error on seven benchmarks","Bayesian uncertainty improves calibration beyond temperature scaling","Post-hoc Bayesian stage yields better calibrated deep models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The calibration stage's input is only the pre-trained network's logit vector, so the method assumes that all miscalibration is already contained in those logits and can be removed by a learned transformation of them alone.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled Bayesian net recalibrates DNN probabilities with less error","BNN post-hoc recalibrator reduces calibration error on seven benchmarks","Bayesian uncertainty improves calibration beyond temperature scaling","Post-hoc Bayesian stage yields better calibrated deep models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001325,"raw_usage":{"total_tokens":5354,"prompt_tokens":866,"completion_tokens":4488,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":4422}},"tokens_in":482,"tokens_out":4488,"duration_ms":29446,"temperature":1.0,"reasoning_tokens":4422,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:24:44.638913+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct two groups of test images whose deep-network logit vectors are identically distributed but whose true-class rates differ, for example distinct subpopulations that collapse to the same logits. Since the BNN is trained only on logits, it must assign the same calibrated probability to both groups, so measuring a nonzero expected calibration error within either group after calibration would falsify the logit-sufficiency assumption.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the formal definition of a well-calibrated probabilistic model used to frame the problem."},{"cited_title":"Guo, et al., On calibration of modern neural networks, in: D","cited_arxiv_id":null,"evidence_quote":"Supplies the temperature scaling baseline and the expected calibration error metric used throughout the experiments."},{"cited_title":"Lakshminarayanan, et al., Simple and scalable predictive uncertainty estimation using deep ensembles, in: I","cited_arxiv_id":null,"evidence_quote":"Supplies the deep-ensemble implicit calibration baseline, also evaluated in decoupled form."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Monte Carlo dropout baseline, which the paper argues is not calibrated by default."},{"cited_title":"Kumar, et al., Trainable calibration measures for neural networks from kernel mean embeddings, in: J","cited_arxiv_id":null,"evidence_quote":"Supplies the MMCE trainable calibration-cost baseline compared in the implicit-technique experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the VWCI stochastic-inference calibration baseline compared in the implicit-technique experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the local reparameterization trick used to build the MFVILR training estimator."}],"review_version":1}