{"id":"9bdd486d-2987-4e1b-a595-618f09bcad22","arxiv_id":"2608.00928","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Models are systematically overconfident on unseen subtypes of known coarse categories, and this calibration failure is not repaired by temperature scaling, vector scaling, or standard OOD detection scores.","lead":"Across four image-classification benchmarks, models become overconfident when tested on new fine-grained variants of categories they know: accuracy drops sharply while confidence barely moves. This study measures the gap, shows it is not explained by image corruption, and finds that standard recalibration and out-of-distribution scores only partially help.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2 matched-corruption control rests on too few bracketing corruptions on iNat-25 and on linear severity interpolation; the specificity claim needs a per-corruption, finer-grid robustness check.","rationale":"The paper is careful: RQ1 is backed by five architectures, four datasets, binning-free proper scores, and per-sample measures; the seen-tuned recalibration and weak detection results also support the headline. The weakest point is exactly the matched-corruption control, which the reader identified. My proposed check would settle whether the iNat contrast survives a broader, representative corruption set. I agree with the reader's CONDITIONAL verdict; no change is needed, but the condition should explicitly include this robustness check.","tokens_in":20295,"tokens_out":9345,"duration_ms":89672,"concrete_test":"On iNat-25, re-run RQ2 with a supertype split that produces a larger accuracy gap (e.g., family-level supertypes) so that at least 10 of the 15 ImageNet-C corruptions bracket the gap, using a finer severity grid (e.g., 10 levels) to avoid linear extrapolation. Report per-corruption matched confidence drops, their median and range, and compare with the subtype confidence drop. If the median or most individual corruptions no longer exceed the subtype drop, the specificity claim is an artifact of the 3 surviving corruptions on iNat-25.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's specificity half rests on RQ2: at matched accuracy loss, generic corruption lowers confidence far more than subtype novelty. That comparison admits a corruption only if its five severity levels bracket the subtype accuracy gap, then linearly interpolates between severities and averages the resulting confidence drops. On IN-25, 14/15 corruptions bracket, so the mean is stable; on iNat-25 only 3/15 do, because the ResNet-50 subtype gap (12.6 points) is below most severity-1 corruption costs. The three survivors are a nonrandom subset (likely mild photometric corruptions), so the reported iNat generic-corruption confidence drop (8.3 points vs 3.8 for subtype shift) may not represent generic corruption. The manuscript reports only the mean over survivors and no per-corruption distribution, so we cannot tell whether the median, or all individual corruptions, exceed the subtype drop. Linearity of the accuracy-confidence trade-off between severities is also assumed; if that curve is nonlinear, the interpolated confidence drop is biased. A further confound is that corruption curves are measured on seen images, while the subtype gap is measured on unseen images, so matching aggregate accuracy loss does not control for per-supertype difficulty variation. RQ1 is robust to all this, but the 'not a general consequence of losing accuracy' conclusion is not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies coarse-label prediction when test inputs come from fine-grained subtypes absent from training but still belonging to a known coarse category. It defines subtype robustness through matched accuracy, confidence, and calibration gaps, and measures these across ImageNet-25, BREEDS, iNaturalist-25, and CIFAR-100 using five architectures. The principal findings are: (1) coarse accuracy drops on unseen subtypes while confidence barely moves, producing systematic overconfidence; (2) at matched accuracy loss, generic ImageNet-C corruption lowers confidence far more than subtype novelty does; and (3) post-hoc temperature/vector scaling tuned on seen subtypes narrows but does not close the seen–unseen calibration gap, while standard OOD scores detect unseen subtypes only weakly. The paper includes protocol and initialization controls, binning-free proper scores, a per-supertype analysis of the collapse, and a k-sweep relating seen-subtype diversity to calibration.","tokens_in":20518,"tokens_out":6030,"duration_ms":56705,"significance":"If the findings hold, the paper makes a strong and useful case that subtype robustness should be evaluated with calibration metrics and not accuracy alone. Its strengths are substantial: this is the first systematic study of calibration under unseen subtype shift across four datasets and five architectures; the central RQ1 result is a direct measurement on held-out splits, corroborated by equal-mass binning and two binning-free proper scores (Appendix D); protocol and initialization controls (Appendix B) show the effect is not a training-protocol artifact; and the k-sweep and per-supertype analysis give a plausible mechanistic direction. The weakness is that the specificity claim—that the effect is not a general consequence of losing accuracy—rests entirely on RQ2, and the RQ2 evidence on iNat-25 is thin: only 3 of 15 corruptions bracket the subtype accuracy gap, and the manuscript reports only the mean confidence drop over those survivors. The paper's headline claim is therefore defensible in its RQ1 half but currently overstated in its RQ2 half.","major_comments":[{"comment":"RQ2 is the only evidence for the claim that the calibration failure is not a general consequence of losing accuracy. On iNat-25 only 3 of the 15 ImageNet-C corruptions bracket the ResNet-50 subtype accuracy gap of 12.6 points, and the manuscript reports only the mean confidence drop over those survivors (8.3 points versus 3.8 for subtype shift), with no per-corruption values, median, or characterization of which corruptions survive. The survivors are likely a nonrandom subset, so the iNat-25 \"generic corruption\" mean may not represent generic corruption at all. The linear interpolation between the two bracketing severities also assumes an approximately linear accuracy–confidence trade-off, which is not tested. I ask the authors to report the per-corruption distribution (e.g., a scatter or box plot over all 15 corruptions), to list the surviving corruptions on each dataset, to test the linearity assumption, and ideally to extend the severity grid (or use corruptions whose severity-1 accuracy cost is below the iNat gap) so that more than three corruptions can be matched on iNat-25.","section":"Experiments, RQ2 (Figure 3)"},{"comment":"The matched-accuracy control matches only aggregate accuracy. The corruption arm is applied to seen-subtype images, whereas the subtype arm is measured on images of unseen subtypes, so the two arms differ not only in shift type but also in the per-supertype composition and difficulty of the evaluated images. If the unseen subtypes concentrate in supertypes where the accuracy loss is 'cheap' in confidence terms, the reported difference could reflect dataset composition rather than shift type. A per-supertype version of RQ2, or at least a report of per-supertype confidence drops in both arms, would make the specificity conclusion considerably more solid.","section":"Experiments, RQ2"}],"minor_comments":[{"comment":"The table states that every reported number is a single run at seed42 and that a second-seed check appears in Appendix B, but Appendix B gives only one sentence saying the gaps are 'closely matched' with no numbers. Please provide the second-seed table or soften the claim.","section":"Table 5, Seeds row; Appendix B"},{"comment":"The row says '32 (CIFAR-100, except AlexNet/ViT-B/16 upsampled to 224)', yet CIFAR-100 uses only AlexNet and ResNet-18. The mention of ViT-B/16 is confusing and should be corrected.","section":"Table 5, Input resolution"},{"comment":"Reporting only the mean confidence drop over surviving corruptions hides the spread, especially on iNat-25 where only three corruptions survive. Error bars or a per-corruption plot would help readers judge whether the reported means are stable.","section":"Figure 3 and RQ2 text"},{"comment":"The ECE formula contains a typographical inconsistency: the second term is written as 'dconf(Bb)' rather than 'conf(Bb)'. Please define the bin notation and use consistent symbols throughout.","section":"Appendix C, Equation (1)"},{"comment":"The claim that 1/sqrt(nj) dominates seen accuracy as a predictor (t=4.2) should report the regression R^2 and coefficient standard errors, since the regression uses only 25 supertypes from a single seed.","section":"Analysis section, regression sentence"}],"recommendation":"major_revision","confidential_remarks":"The RQ1 core of this paper is careful and convincing, and the appendices (equal-mass binning, proper scores, protocol/initialization controls, within-correct detection check) are exactly the right kind of support. My main concern is that the abstract's specificity claim rests on RQ2, and the iNat-25 arm of RQ2 is too thin to carry it. This is fixable within the manuscript's scope by reporting the per-corruption distribution, adding a finer severity grid or a corruption set with smaller severity-1 costs, and ideally a per-supertype-matched comparison. I would be comfortable with acceptance after those additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is real and worth knowing: on unseen subtypes, accuracy drops 29–40 points while confidence drops only 2–16, leaving models overconfident exactly where they are wrong. That is the first clean demonstration I know of that subtype robustness must be evaluated with calibration, not accuracy alone.\n\nWhat the paper does well: the measurement is unusually careful. Four datasets including iNaturalist with a real taxonomic split, five architectures, a matched-corruption control, binning-free proper scores, a second-seed check, and protocol/initialization controls. The calibration gap is robust to binning scheme and shows up in NLL and Brier as well as ECE. The per-supertype analysis relating collapse to seen-subtype diversity (nj) and the controlled k-sweep are a nice bonus and give the phenomenon a mechanistic handle. The authors also honestly report that pretraining only partially mitigates the failure.\n\nSoft spots, in proportion. The biggest is the matched-corruption comparison (RQ2), which carries the \"not just accuracy\" specificity claim. On iNat-25 only 3 of 15 corruptions bracket the subtype accuracy gap, and the extrapolation is linear along severity. The paper reports only the mean over survivors, not the per-corruption distribution, so I can't tell whether all 3 behave the same or one is carrying the average. Also, the corruption arm is run on ResNet-50 only. RQ1 does not depend on this, but the abstract's \"not a general consequence of losing accuracy\" is somewhat stronger than what the matched arm can support. A per-corruption, finer-grid robustness check would settle it.\n\nThe single-seed main table is a real but minor gap; the second-seed appendix check is fine as a sanity check, but error bars on the main metrics would be better. These are addressable in revision and do not undermine the core finding.\n\nWho this is for: anyone evaluating robustness of classifiers under fine-grained novelty, and anyone working on calibration under shift. It deserves a serious referee. My recommendation: send it to review, and ask for the corruption-matching sensitivity analysis and multi-seed main metrics.","headline":"Solid, careful empirical paper showing calibration collapses under unseen subtype shift; the matched-corruption control is the one soft spot that needs a robustness check.","tokens_in":21045,"tokens_out":2286,"would_cite":true,"duration_ms":20365,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that classifiers lose far more accuracy than confidence on unseen fine-grained subtypes, becoming overconfident exactly where they fail, and that this miscalibration resists recalibration and detection.","keywords":["subtype robustness","calibration","expected calibration error","distribution shift","overconfidence","hierarchical classification","out-of-distribution detection","ImageNet-C"],"falsifier":"Train a coarse classifier on seen subtypes, then take a matched unseen-subtype test set and a corrupted version of the seen test set whose accuracy loss equals the subtype arm's. If the corrupted arm's confidence drop equals or falls below the subtype arm's, the paper's dissociation fails; a direct check would vary the severity grid and the choice of corruptions to see whether the iNat-25 result flips when more than 3 corruptions reach its small accuracy loss.","tokens_in":20080,"feed_emoji":"📉","tokens_out":10849,"duration_ms":73331,"temperature":0.7,"pith_summary":"This paper asks whether a classifier stays calibrated when it is tested on fine-grained subtypes it never saw, such as a new species under a known genus. It finds that coarse accuracy typically drops by 7–40 percentage points on unseen subtypes, while confidence drops by only 2–16 points, so the model becomes overconfident exactly where it is now wrong. The effect is specific to in-taxonomy novelty: when generic image corruption is tuned to cause the same accuracy loss, confidence falls far more. Post-hoc temperature scaling trained on seen subtypes narrows the seen–unseen calibration gap but never closes it, and standard out-of-distribution scores flag the unseen inputs only weakly. The paper concludes that subtype robustness should be measured with calibration, not accuracy alone.","feed_headline":"Models stay overconfident on the unseen subtypes they get wrong","feed_subtitle":"Accuracy drops sharply on novel fine-grained variants while confidence barely moves, breaking calibration where it matters.","key_machinery":"The argument is carried by three matched gaps computed on seen and unseen subtype splits of the same coarse classifier: $\\Delta_{\\mathrm{acc}}$, $\\Delta_{\\mathrm{conf}}$, and $\\Delta_{\\mathrm{cal}} \\approx \\Delta_{\\mathrm{acc}} - \\Delta_{\\mathrm{conf}}$ in the paper's accounting. When $\\Delta_{\\mathrm{conf}} \\ll \\Delta_{\\mathrm{acc}}$, the model exhibits what the authors call silent overconfidence. To show the effect is specific, the paper builds a matched-corruption arm in which each ImageNet-C corruption is interpolated along its severity curve to the exact accuracy loss of the subtype shift, so the only difference between arms is the kind of shift, not the amount of accuracy lost. A secondary mechanism is the per-supertype seen-subtype count $n_j$: overconfidence on unseen subtypes scales roughly as $1/\\sqrt{n_j}$ (Spearman $\\rho = -0.82$).","core_discovery":"The central discovery is silent overconfidence under unseen-subtype shift: across ImageNet-25, BREEDS (Living-17, Nonliving-26), iNaturalist-25, and CIFAR-100, and across five architectures, coarse accuracy falls on held-out fine subtypes while mean confidence stays high, so expected calibration error rises by 5–28 percentage points. The dissociation is quantified by three matched gaps, $\\Delta_{\\mathrm{acc}}$ (seen minus unseen accuracy), $\\Delta_{\\mathrm{conf}}$ (seen minus unseen mean confidence), and $\\Delta_{\\mathrm{cal}}$ (unseen minus seen ECE), and it holds per sample: failure-detection AUROC drops by 0.05–0.18 and risk–coverage AURC rises 2.3×–18×. A matched-corruption control shows that at equal accuracy loss, ImageNet-C corruption costs roughly twice the confidence that subtype novelty costs, so the effect is not a generic consequence of losing accuracy. Seen-tuned recalibration narrows the gap without closing it, and MSP, energy, and Mahalanobis scores separate seen from unseen subtypes only weakly, with Mahalanobis near chance. The authors conclude that subtype robustness must be evaluated as a joint requirement on prediction and confidence.","pith_inferences":["The authors do not claim this, but the matched-corruption result suggests a diagnostic: measuring the confidence drop per unit accuracy loss under a corruption sweep could serve as a cheap calibrated-trust test for deployed models facing novel subtypes.","Extension: because the paper studies images only, an immediate test is whether language or tabular models show the same dissociation when novel subtypes arrive as new vocabulary or as new sparse categorical values.","Extension: the observed $1/\\sqrt{n_j}$ scaling of overconfidence implies a testable law—doubling the number of seen subtypes per supertype should shrink unseen overconfidence by roughly $1/\\sqrt{2}$—which a controlled diversity sweep could confirm.","Extension: in dense prediction, where subtype novelty acts at pixel or region level, calibration collapse may be sharper still; measuring per-pixel ECE under unseen-object-part shift would be a concrete test."],"forward_implications":["Subtype robustness evaluations that report only coarse accuracy will miss the failure: a model can lose 30 or more points of accuracy and almost no confidence, so the claimed robustness is overstated.","Post-hoc recalibration tuned on seen subtypes cannot be relied on to restore calibration on unseen subtypes; it narrows but does not close the seen–unseen gap.","Standard OOD scores (MSP, energy, Mahalanobis) are not an adequate safety net: they separate seen from unseen subtypes at AUROC around 0.56–0.74, and the feature-based score is at chance.","The size of the collapse depends on how densely the coarse class is sampled in training: supertypes with few seen subtypes collapse more than well-sampled ones, so ranking by seen-subtype count works as a coarse triage signal."],"supporting_citations":[{"why":"Supplies the ImageNet-C corruption set for the matched generic-corruption arm and the earlier seen–unseen subtype accuracy observation.","marker":"(Hendrycks and Dietterich 2019)"},{"why":"Provides the BREEDS hierarchy-aware subtype splits used for Living-17 and Nonliving-26.","marker":"(Santurkar et al. 2021)"},{"why":"Supplies the iNaturalist taxonomy and images used to build iNat-25 with true unseen species.","marker":"(Van Horn et al. 2021)"},{"why":"Defines the ECE calibration measure and the temperature and vector scaling calibrators tested in RQ3a.","marker":"(Guo et al. 2017)"},{"why":"Documents the standard expectation that confidence falls under distribution shift, which the subtype results contrast against.","marker":"(Ovadia et al. 2019)"},{"why":"Provides the maximum-softmax-probability detection score evaluated in RQ3b.","marker":"(Hendrycks and Gimpel 2017)"},{"why":"Provides the energy-based detection score evaluated in RQ3b.","marker":"(Liu et al. 2020)"},{"why":"Provides the Mahalanobis feature-distance detection score evaluated in RQ3b.","marker":"(Lee et al. 2018)"},{"why":"Offers the inflated-logits mechanism the authors invoke to explain why under-represented groups are overconfident.","marker":"(Lyu, Zhou, and Zhong 2025)"},{"why":"Defines failure-detection AUROC and risk–coverage AURC used for per-sample discrimination.","marker":"(Jaeger et al. 2023)"}],"fun_headline_variants":["Unseen subtypes: accuracy drops, confidence doesn't","Silent overconfidence on novel subtypes breaks calibration","Models overconfident on unseen subtypes they get wrong","Calibration blind spot: overconfidence on novel subtypes","Novel subtypes: confidence stays high while accuracy falls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The matched-corruption comparison assumes that interpolating each ImageNet-C corruption along its severity curve to the subtype shift's accuracy loss, and then averaging over the corruptions whose severities bracket that loss, yields an unbiased estimate of the confidence drop generic corruption would cause; on iNat-25 only 3 of 15 corruptions survive this bracketing, so the 'generic corruption' mean there rests on a very small, possibly unrepresentative set.","fun_headline_variants_meta":{"raw":{"variants":["Unseen subtypes: accuracy drops, confidence doesn't","Silent overconfidence on novel subtypes breaks calibration","Models overconfident on unseen subtypes they get wrong","Calibration blind spot: overconfidence on novel subtypes","Novel subtypes: confidence stays high while accuracy falls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000323,"raw_usage":{"total_tokens":1827,"prompt_tokens":970,"completion_tokens":857,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":782}},"tokens_in":586,"tokens_out":857,"duration_ms":7314,"temperature":1.0,"reasoning_tokens":782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:13:40.545878+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a coarse classifier on seen subtypes, then take a matched unseen-subtype test set and a corrupted version of the seen test set whose accuracy loss equals the subtype arm's. If the corrupted arm's confidence drop equals or falls below the subtype arm's, the paper's dissociation fails; a direct check would vary the severity grid and the choice of corruptions to see whether the iNat-25 result flips when more than 3 corruptions reach its small accuracy loss.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BREEDS hierarchy-aware subtype splits used for Living-17 and Nonliving-26."},{"cited_title":"V.; Lakshminarayanan, B.; and Snoek, J","cited_arxiv_id":null,"evidence_quote":"Documents the standard expectation that confidence falls under distribution shift, which the subtype results contrast against."},{"cited_title":"D.; and Li, Y","cited_arxiv_id":null,"evidence_quote":"Provides the energy-based detection score evaluated in RQ3b."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Mahalanobis feature-distance detection score evaluated in RQ3b."},{"cited_title":"F.; L \\\"u th, C","cited_arxiv_id":null,"evidence_quote":"Defines failure-detection AUROC and risk–coverage AURC used for per-sample discrimination."}],"review_version":1}