{"id":"e1a566d9-da7d-4000-9bb4-9c97d751c846","arxiv_id":"1908.11528","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Bin-wise temperature scaling, applied to per-confidence bins and supplemented by augmented validation samples, reduces expected calibration error relative to temperature scaling across tested image classifiers.","lead":"This workshop paper proposes bin-wise temperature scaling (BTS) and an augmented variant (ABTS), which replace temperature scaling's single temperature with per-bin temperatures for neural network confidence calibration. The methods lower expected calibration error compared to temperature scaling on several image classification datasets and architectures.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The all-12 superiority claim for ABTS (#samples) depends on temperatures fitted to shifted validation images transferring to unshifted test images; the paper supplies no distributional check and admits the risk.","rationale":"Reader's weakest assumption identifies the same point: the augmented validation set is not shown to be representative of test confidence, especially in low-confidence bins. My read of the method confirms this is the most load-bearing insecure step. The method is not internally inconsistent; BTS itself is a simple extension of TS, and number-of-samples binning addresses sparse-bin variance in a principled way. The empirical tables are internally consistent, and the claimed all-12 superiority is exactly what Table 1 shows. But the extra step that makes ABTS outperform BTS and TS is the augmentation, and the paper's own caveat concedes the central risk. Because no distributional check or significance analysis is given, the correct verdict is conditional: the main claim should be accepted only after a check that the augmentation does not change the validation/test confidence relationship in the low-confidence bins, and that the ECE differences are not within noise. No reason to reject: the method is plausible and the experimental protocol is standard, but the condition is unmet.","tokens_in":7981,"tokens_out":8859,"duration_ms":85187,"concrete_test":"On CIFAR-100 with WideResNet-28-10, compute the ECE difference (ABTS #samples minus TS) on the original test set under three controls: (a) fit ABTS as in the paper; (b) fit the same per-bin temperatures using the same number of additional real validation images drawn from a held-out split instead of shifted images; (c) fit on shifted validation images but evaluate on test images shifted with the same transformation. If (b) matches (a) within bootstrap error, the gain is due to more low-confidence samples rather than shift-specific artifacts; if (c) is much lower than (a), the fitted temperatures are tied to the shifted distribution and the transfer to unshifted test is not established. Report 95% bootstrap confidence intervals for the ABTS-vs-TS ECE difference on the original test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4 is that ABTS based on the number of samples (last column of Table 1) outperforms TS in all 12 settings. The mechanism for the improvement is augmentation of low-confidence validation bins: shifted images are added under the same labels, bin counts are rebalanced, and per-bin temperatures are fit on the augmented set. The load-bearing premise is that these temperatures remain correct for original test samples with the same pre-scaling confidence. This premise is not checked. A shift of 4-8 pixels on 32x32 CIFAR images is a strong geometric perturbation; the augmented validation images are not draws from the test distribution. The paper's only acknowledgment is the sentence in Section 4 that 'strong data augmentation can change the distribution between the validation and test sets, thereby diminishing the effectiveness of ABTS.' No held-out comparison, no comparison of confidence histograms before and after augmentation, and no error bar or bootstrap on the ECE differences is reported. Because the claimed advantage is concentrated in exactly the low-confidence bins that have few test samples, a distribution mismatch or small-sample noise in those bins could overturn the all-12 result. Tables 2 and 3 show robustness across augmentation types and bin counts, but not across validation/test distribution shift.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two post-hoc confidence calibration methods that generalize temperature scaling (TS): bin-wise temperature scaling (BTS), which partitions validation samples into bins and fits one temperature per bin, and augmentation-based BTS (ABTS), which augments low-confidence validation bins before fitting the temperatures. Two binning schemes are considered: equal-width confidence intervals and equal sample counts with a fixed high-confidence threshold of 0.999. The methods are evaluated on CIFAR-10/100, Caltech-UCSD Birds, and Stanford Cars with several CNN architectures, using expected calibration error (ECE). The central empirical claim is that ABTS with sample-count binning outperforms TS in all 12 dataset/model configurations in Table 1.","tokens_in":8243,"tokens_out":6060,"duration_ms":57309,"significance":"If the results are reproducible, the contribution is practically useful: it shows that a simple bin-wise extension of TS, with a modest number of augmented low-confidence samples, can reduce ECE substantially on common benchmarks while preserving test error rates. The evaluation covers a reasonable spread of datasets and architectures, and Tables 2 and 3 provide useful sensitivity analyses for augmentation type and the number of bins. The paper is self-contained and follows the standard validation-set fitting / test-set evaluation protocol for post-hoc calibration. Its main limitations are the absence of uncertainty quantification and the unexamined distribution shift introduced by validation augmentation.","major_comments":[{"comment":"The number of bins N used to compute ECE is never stated. ECE values depend strongly on the evaluation binning; Table 3 demonstrates that similar bin-count choices affect the reported ECE of the proposed methods. Without specifying the evaluation bin count, the reported numbers cannot be reproduced or compared with published results, and the all-12 claim in Section 4 is not fully checkable. Please state the evaluation bin count and report ECE for at least two evaluation bin counts.","section":"3.3, Eq. (4)"},{"comment":"The headline claim that ABTS with number-of-samples binning outperforms TS in all 12 cases is based on single point estimates of ECE. No error bars, bootstrap confidence intervals, or significance tests are reported. The margin is small in some rows (e.g., Birds DenseNet: 1.44 vs 1.40), so the claim of consistent improvement is not statistically established. Please provide repeated runs or bootstrap confidence intervals over test samples, and consider a paired comparison across configurations.","section":"Table 1 and Section 4"},{"comment":"The ABTS mechanism assumes that temperatures fitted on augmented validation images transfer to unshifted test images. The paper acknowledges that strong augmentation can change the validation/test distribution, but it provides no check that this is not happening here. The x-axis shifts of 4–8 pixels on 32×32 images are a substantial geometric perturbation, and the benefit of ABTS is concentrated in low-confidence bins where test samples are few. An empirical check is needed, e.g., comparing confidence histograms of the augmented validation set and the test set, or fitting on augmented data and evaluating on a clean held-out validation split; without such a check, the improvement of ABTS over BTS could be an artifact of the distribution shift rather than a genuine calibration improvement.","section":"2.3 and Section 4"}],"minor_comments":[{"comment":"The paper discusses how low-confidence bins contain very few validation samples but does not report the actual bin counts or confidence ranges. Reporting these for at least one representative model would help assess the stability of the per-bin temperature estimates.","section":"Section 2.2"},{"comment":"The random split of CIFAR training data into training and validation sets is not seeded. Please provide the random seed or release the exact split so that the experiments can be reproduced.","section":"Section 3.1"},{"comment":"The effective number of bins under the 0.999 threshold can be smaller than the nominal 50, but the actual number of bins used in Tables 1–3 is not reported. This would matter for interpreting the claimed insensitivity to bin count.","section":"Section 2.2 and Table 3"},{"comment":"The sentence in Section 4 that ABTS outperforms TS in all cases is true only for the number-of-samples variant; ABTS with confidence-interval binning does not beat TS on Birds DenseNet (1.70 vs 1.44). The text is mostly clear on this point, but a brief clarification would avoid over-reading the table.","section":"Section 4"},{"comment":"The augmentation parameters (shift ranges, brightness range, contrast strength, blur sigma) are described in prose and the table caption. Releasing the exact code or configuration used with the imgaug library would remove ambiguity about the implementation.","section":"Section 4 and Table 1 caption"}],"recommendation":"major_revision","confidential_remarks":"This is a concise empirical study that is likely suitable for a workshop or a specialized calibration venue. For a journal, the main gaps are the missing uncertainty quantification and the unexamined distribution shift from validation augmentation; both are fixable with additional experiments. The narrow comparison to TS only is a scope consideration that the editor may wish to weigh, but it does not by itself invalidate the paper's stated claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate, low-cost extension of temperature scaling. The specific binning rule (equal-count bins with a 0.999 high-confidence cap) and the augmentation of low-confidence validation bins are new relative to Guo et al., and the empirical claim that ABTS with number-of-samples binning beats TS in all twelve settings is directly visible in Table 1. The authors also show robustness to the choice of augmentation (Table 2) and to bin count (Table 3), which is more than many calibration papers do.\n\nWhere it gets soft: the comparison is only against TS. No histogram binning, no isotonic regression, no Brier or NLL reported. ECE is reported without error bars or significance tests, and the evaluation bin count isn't specified, so we don't know how sensitive the numbers are to that choice. The abstract says the proposed methods 'consistently improve' calibration, but Table 1 shows BTS loses to TS in a couple of cases, so that wording overclaims.\n\nMore importantly, the stress-test concern is real. ABTS fits temperatures on shifted or brightened validation images, then applies them to untouched test images. That only works if the augmented validation confidence distribution is close to the test distribution. The authors acknowledge the risk in Section 4 but never check it. A shift of 4-8 pixels on 32x32 CIFAR is not a small perturbation, and the gains are concentrated in low-confidence bins with few test samples, so small-sample noise or a distribution mismatch could flip some of the twelve results. The paper needs at least a histogram comparison of confidence before and after augmentation and ideally bootstrap confidence intervals on the ECE differences.\n\nThat said, the core idea is reasonable and clearly described. The claimed advantage is modest, roughly 0.3 to 2.7 points ECE, and the experiments are easy to reproduce. I would send it to review: an editor should ask for the missing baselines and the distributional check, but the work is coherent and honest. I would cite it as a representative bin-wise scaling variant in a calibration paper. It is a workshop-level contribution, but a legitimate one.","headline":"Bin-wise temperature scaling with augmented validation bins is a plausible, clearly described extension of temperature scaling, but the all-12 superiority claim rests on an unverified distributional assumption.","tokens_in":8780,"tokens_out":2885,"would_cite":true,"duration_ms":25841,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing a single global temperature with per-bin temperatures, plus augmenting validation samples, lowers expected calibration error below temperature scaling across all tested models and datasets.","keywords":["confidence calibration","temperature scaling","bin-wise temperature scaling","expected calibration error","data augmentation","post-hoc calibration","deep neural networks","reliability diagrams"],"falsifier":"Train a classifier on one of the paper's datasets, hold out a separate split of the test set, fit ABTS temperatures on augmented validation data, and compute ECE on the held-out split; if the ECE is not lower than temperature scaling's, or drops sharply as augmentation strength increases, the transfer assumption fails. A more direct check is to compare the confidence histograms of the augmented validation set and the test set and look for a shift in the low-confidence bins.","tokens_in":1594,"feed_emoji":"🌡️","tokens_out":1797,"duration_ms":71732,"temperature":0.7,"pith_summary":"Modern image classifiers tend to be overconfident, and temperature scaling fixes this with a single global temperature fitted on a validation set. This paper argues that one temperature is too coarse: because most predictions have high confidence, the fitted temperature is biased toward high-confidence samples and leaves low-confidence samples miscalibrated. The authors propose bin-wise temperature scaling (BTS), which splits validation samples into confidence bins and fits a separate temperature for each bin, and augmentation-based BTS (ABTS), which adds transformed copies of validation images to low-confidence bins so those temperatures are stable. Across CIFAR-10/100, Caltech Birds, and Stanford Cars, with ResNet, DenseNet, VGG, and Wide ResNet models, ABTS with number-of-samples binning reports lower expected calibration error than temperature scaling in all twelve experimental settings. The calibration gain comes without retraining or changing test error, since all scaling happens after the model produces its logits.","feed_headline":"Per-bin temperatures beat temperature scaling on all twelve test runs","feed_subtitle":"Each of twelve model-dataset tests shows lower ECE with per-bin temperatures.","key_machinery":"The central object is the bin-wise temperature vector $\\{t_j\\}$, one scalar per confidence bin, applied as in Eq. (3) to soften the softmax of each test sample according to the bin its confidence falls in. Two details carry the argument: the number-of-samples binning rule, which avoids tiny low-confidence bins by collecting equal counts and uses a 0.999 threshold to keep extremely high-confidence samples in one bin, and the augmentation step, which doubles the validation samples in the eight bins below confidence 0.8 using simple transformations (shifting, brightness, contrast, blur). The bin-wise temperatures are what let low-confidence samples be treated differently from high-confidence ones; the augmentation is what keeps the low-confidence temperature estimates from being unstable.","core_discovery":"The central claim is that the calibration quality of a deep classifier can be improved by replacing temperature scaling's single scalar temperature with a set of bin-specific temperatures, and that fitting those temperatures on an augmented validation set makes them stable enough to beat the baseline consistently. Formally, the calibrated confidence for a test sample in bin $j$ is $\\hat{q}_i = \\max_c \\sigma_{SM}(z^j_i / t_j)_c$, where each $t_j$ is optimized on validation samples whose confidence falls in bin $j$. The paper evaluates two binning schemes: equal-width confidence intervals and equal-count bins with a 0.999 threshold that collects all near-certain samples into one bin. It then augments the eight bins below confidence 0.8 by image shifting, brightness, contrast, or blur, doubling the number of validation samples per augmentation. Using expected calibration error (ECE) as the metric, the experiments report that ABTS with number-of-samples binning beats temperature scaling in every case in Table 1, and BTS beats temperature scaling in 10 or 11 of 12 cases depending on the binning scheme.","pith_inferences":["One consequence the paper leaves implicit is that the improvement should transfer to any post-hoc calibration pipeline where the validation set can be augmented; on small validation sets the augmentation step may matter more.","A testable extension is to replace the fixed augmentation of the eight bins below 0.8 with a learned or per-dataset selection of which bins and which augmentation strengths, which might further improve low-confidence calibration.","Since the bin-wise temperatures are fitted independently, a natural next step is to smooth the temperature curve across bins, retaining the gain while removing discontinuities at bin boundaries.","The paper compares only against temperature scaling; comparing BTS and ABTS against histogram binning or isotonic regression would isolate whether the benefit comes from per-bin scaling or from binning itself."],"forward_implications":["Calibration of deployed classifiers can be improved by a post-processing step that requires no retraining and leaves test error unchanged.","ABTS with number-of-samples binning matches or beats TS in every configuration reported in Table 1, and BTS beats TS in most cases.","The choice among the four tested augmentation types makes only a marginal difference; all beat TS on CIFAR-100.","The method is not sensitive to the number of bins between 5 and 50, so it does not require fine per-model tuning.","Low-confidence samples, which TS tends to push further under-confident, become better calibrated under ABTS as shown in the reliability diagrams."],"supporting_citations":[{"why":"Supplies the temperature scaling baseline, the NLL objective used to fit temperatures, and the ECE metric.","marker":"[3]"},{"why":"Provides the bin-based expected calibration error measure used for evaluation.","marker":"[10]"},{"why":"Supplies the CIFAR-10 and CIFAR-100 datasets used in the main experiments.","marker":"[8]"},{"why":"Supplies the Stanford Cars dataset used as a fine-grained benchmark.","marker":"[7]"},{"why":"Supplies the Caltech-UCSD Birds dataset used as a fine-grained benchmark.","marker":"[17]"},{"why":"Provides the ResNet architectures used for the experiments.","marker":"[4]"},{"why":"Provides the DenseNet architectures used for the experiments.","marker":"[5]"},{"why":"Provides the VGG-16 architecture used for the experiments.","marker":"[15]"},{"why":"Provides the Wide ResNet architecture used for the experiments.","marker":"[18]"}],"fun_headline_variants":["Per-bin temps beat single temp in every calibration test","Augmented bin-wise temperatures beat single temperature in all 12 tests","Per-bin temperatures with augmentation improve calibration consistently","Simple bin-wise scaling outperforms single temperature on all 12 trials"],"cache_read_input_tokens":10880,"weakest_assumption_plain":"The augmented validation set, including the synthetically transformed low-confidence samples, has a confidence distribution close enough to the test set that temperatures fitted on it transfer to test samples.","fun_headline_variants_meta":{"raw":{"variants":["Per-bin temps beat single temp in every calibration test","Augmented bin-wise temperatures beat single temperature in all 12 tests","Per-bin temperatures with augmentation improve calibration consistently","Simple bin-wise scaling outperforms single temperature on all 12 trials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000891,"raw_usage":{"total_tokens":3811,"prompt_tokens":884,"completion_tokens":2927,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":2860}},"tokens_in":500,"tokens_out":2927,"duration_ms":24284,"temperature":1.0,"reasoning_tokens":2860,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:12:24.088905+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a classifier on one of the paper's datasets, hold out a separate split of the test set, fit ABTS temperatures on augmented validation data, and compute ECE on the held-out split; if the ECE is not lower than temperature scaling's, or drops sharply as augmentation strength increases, the transfer assumption fails. A more direct check is to compare the confidence histograms of the augmented validation set and the test set and look for a shift in the low-confidence bins.","supporting_citations":[{"cited_title":"Krause, M","cited_arxiv_id":null,"evidence_quote":"Supplies the Stanford Cars dataset used as a fine-grained benchmark."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the bin-based expected calibration error measure used for evaluation."},{"cited_title":"Krizhevsky, V","cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-10 and CIFAR-100 datasets used in the main experiments."}],"review_version":1}