{"id":"8533a246-9c66-4839-b8e3-43d3b171bcf6","arxiv_id":"2608.05945","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"ZAEC anchors calibration to each sample's zero-shot entropy and selectively softens over-sharpened TTA predictions, reaching the lowest macro-average calibration error among evaluated post-hoc methods on ViT-B/16.","lead":"The paper finds that test-time adaptation of vision-language models often makes predictions sharply more confident without making them more accurate, and proposes a label-free, post-hoc calibration method, ZAEC, that restores each sample's original zero-shot uncertainty via minimal temperature scaling.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The zero-shot entropy anchor is the load-bearing assumption; the paper's own Caltech101 result shows it can push calibration the wrong way whenever TTA sharpening already corrects zero-shot underconfidence.","rationale":"The reader identified the same load-bearing assumption: ZAEC's anchor is the zero-shot entropy of each sample, and if the zero-shot model is over- or underconfident, restoring that entropy can push calibration in the wrong direction. The paper's Caltech101 experiment is direct evidence that this failure mode is real and not merely hypothetical. I agree with the reader that this warrants a conditional acceptance rather than rejection: the method is simple, label-free, decision-preserving, and achieves strong average results across 15 datasets, and the paper explicitly discusses the boundary case. The concrete test I propose would determine whether the Caltech101 case is an isolated anomaly or a systematic regime in which the method's core assumption fails. If the latter, the central claim needs qualification; if the former, the conditional concerns about code, error bars, and anchor discussion suffice. I do not see a separate load-bearing flaw in the theoretical propositions or the empirical average, so the verdict should remain conditional.","tokens_in":14515,"tokens_out":5657,"duration_ms":63055,"concrete_test":"Run the full ZAEC pipeline on all 15 datasets, and for each dataset determine whether zero-shot CLIP is underconfident by computing the sign of the confidence--accuracy gap in the highest-confidence bins (or by decomposing ECE into over/under-confidence components). Then compute the correlation between zero-shot underconfidence and the ECE change induced by ZAEC relative to raw TTA. If ZAEC systematically raises ECE on datasets where zero-shot is underconfident — as it does on Caltech101 with DPE — the anchor failure is structural rather than a single boundary case, and the paper should restrict its claim to settings where zero-shot is not underconfident or replace the fixed anchor with an adaptive reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ZAEC's central mechanism (Eq. 11) assumes that the paired zero-shot entropy is a trustworthy per-sample uncertainty reference: any adapted distribution sharper than this reference is judged to have undergone unsupported predictive sharpening and is smoothed back up. The paper itself documents a structural counterexample in the 'Boundary case on Caltech101' section (Fig. 4): with DPE, zero-shot CLIP is underconfident on Caltech101, DPE improves ECE from 5.07% to 1.69% by raising confidence, and ZAEC reverses that improvement to 5.38% by restoring the zero-shot entropy. This is not a numerical accident; it follows directly from the one-sided design. Whenever TTA-induced sharpening corrects zero-shot underconfidence, the entropy floor is the wrong direction. The aggregate correlations in Figs. 2(b)-(c) support the claim that entropy reduction often accompanies overconfidence, but they do not identify when the zero-shot anchor is unreliable. The empirical headline — lowest post-hoc macro-average ECE — could therefore be driven by datasets where the anchor happens to be sensible, while the method is harmful on exactly the datasets where TTA already fixes calibration. The paper acknowledges this in the conclusion ('future work will explore adaptive anchors'), but the central claim as stated does not carry this qualification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies calibration degradation caused by test-time adaptation (TTA) of vision-language models. It introduces the notion of prediction-preserving sharpening, in which TTA increases confidence and reduces predictive entropy without changing the top-1 prediction, and shows empirically that larger zero-shot-relative entropy reductions are associated with larger increases in expected calibration error. On the basis of this diagnostic, the authors propose ZAEC, a parameter-free post-hoc method that applies minimal temperature scaling to restore the paired zero-shot entropy, but only for samples whose adapted entropy falls below the zero-shot reference. The method is evaluated on five TTA methods and 15 datasets with two CLIP backbones, and an ablation study tests the effect of correcting different subsets. The paper also documents a boundary case on Caltech101 in which the zero-shot anchor is itself underconfident and ZAEC reverses a beneficial TTA-induced improvement.","tokens_in":14748,"tokens_out":8136,"duration_ms":77627,"significance":"The main contribution is a simple, mechanism-agnostic calibration baseline with no learned parameters and decision-preserving guarantees. The empirical scope is broad (five TTA methods, 15 datasets, two backbones), and the paper is candid about a structural failure case. If the diagnostic association holds beyond the reported settings, zero-shot-relative entropy reduction is a practically useful label-free signal. The paper's own counterexample, however, shows that the anchor can be systematically wrong, and the macro-averaged headline can conceal per-dataset deterioration. The method is likely to be a useful reference baseline, but the central claim needs qualification before the paper can be accepted.","major_comments":[{"comment":"The central design assumption is that the paired zero-shot entropy is a trustworthy sample-specific uncertainty reference. Section 'Method', Eq. (11), implements this as a one-sided entropy floor: any adapted distribution with H(p^TTA_i) < H(p^ZS_i) is smoothed back up to the zero-shot level. The paper's own boundary case in Figure 4 is a direct counterexample to the generality of this assumption: zero-shot CLIP is underconfident on Caltech101, DPE raises confidence and reduces ECE from 5.07% to 1.69%, and ZAEC restores the zero-shot entropy and increases ECE to 5.38%. This is not a numerical accident of the example; it follows from the one-sided design whenever TTA sharpening corrects zero-shot underconfidence. The abstract and the contribution list state the method without this qualification. The authors should either restrict the central claim to settings where the zero-shot model is not systematically underconfident, add a dataset-level test for anchor reliability, or frame ZAEC explicitly as 'entropy-conservative' rather than 'calibration-improving' in general.","section":"Method (Eq. 11) and 'Boundary case on Caltech101'"},{"comment":"The headline 'lowest post-hoc macro-average ECE' is an average over datasets and can hide per-dataset degradation that is directly relevant to a user choosing whether to apply ZAEC. For example, on Caltech101 with DPE, raw TTA has ECE 1.69% and ZAEC increases it to 5.38% (Table 3); in Table 2, TDA on ImageNet-R has raw ECE 1.50% and TDA+ZAEC has 4.48%. The paper should report, for each method, the number of datasets on which ZAEC improves or worsens ECE relative to raw TTA, and ideally show the per-dataset reliability diagrams or a scatter plot of raw vs. ZAEC ECE. Without this, the reader cannot tell whether the favorable macro-average reflects consistent small gains or a few large wins offsetting many failures.","section":"Experiments, Tables 2 and 3, Figure 4"},{"comment":"The paper reports r = -0.94 for the dataset-method-level association between mean zero-shot-relative entropy change and ECE change. This single correlation pools samples from multiple TTA methods and datasets; it is not clear how many independent points it is based on, whether the relationship holds separately within each TTA method, or how sensitive it is to the four ImageNet variants versus the fine-grained datasets. Because this correlation is the main evidence for the diagnostic signal, the authors should provide per-method correlations, sample sizes, and confidence intervals (or an equivalent breakdown). If the association is driven by only one or two methods, the motivation for a method-agnostic post-hoc correction is weakened.","section":"Explaining Calibration Degradation via Zero-Shot-Relative Entropy (Eq. 8, Fig. 2b)"}],"minor_comments":[{"comment":"The Ablation Study section opens with a run-on, space-omitted sentence ('All predictions matcheseveryadaptedprediction...'). Rewrite for clarity and separate the definitions of 'All predictions', 'Entropy-increased', and 'Entropy-reduced'.","section":"Ablation Study"},{"comment":"The quantity Delta^m_R is an average absolute relative deviation; the text 'TPT+SaLS nearly eliminates the range deviation' should say 'on average' to avoid implying per-sample alignment.","section":"Eq. (6) and 'Why logit-range statistics are insufficient'"},{"comment":"The gray arrows connect SaLS and ZAEC for TPT, but the legend does not indicate the TTA method for the other points; add per-method markers and axis labels for all panels.","section":"Figure 2(a)"},{"comment":"The caption says 'across 11 datasets' while the paper's overall claim is over 15; clarify that Figure 1 covers the second benchmark group only.","section":"Figure 1 caption"},{"comment":"The row labels would be clearer as 'Raw TTA (none)', 'All samples', 'Entropy-increased only', and 'Entropy-reduced only (ZAEC)' to align with the ablation text.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The evaluation follows the TTABC benchmark (Huang et al. 2026), a benchmark developed by a group with substantial author overlap with the present paper. While adopting one's own benchmark is not inherently improper, the paper should either explicitly disclose this relationship in the main text or compare against an independent benchmark to rule out hidden favorable protocol choices. The two-proposition theory section is standard, but a version with the appendix included would allow the reviewer to verify the 'forward KL' claim in Proposition 2. I recommend major revision mainly because of the anchor-reliability issue, not because of the benchmark overlap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Candid take: ZAEC is a genuinely useful post-hoc calibration layer for TTA VLMs, and the paper earns its headline with a new observation and broad experiments. But its load-bearing assumption — that per-sample zero-shot entropy is the right uncertainty reference — is wrong exactly in the cases the paper's own Caltech101 numbers expose, so the claimed 'lowest ECE' needs a caveat.\n\nWhat's new: the paper identifies and names prediction-preserving sharpening: TTA often reduces entropy and raises confidence on samples where the top-1 prediction and correctness don't change. That's a clean diagnostic, and the dataset-level correlation (r=-0.94) between entropy reduction and ECE increase is striking. Using zero-shot entropy rather than logit range as a per-sample reference is a real departure from SaLS, and the one-sided design (only smooth samples that got sharper) is well motivated by the ablation. The theoretical properties are standard but correct: positive temperature scaling preserves ranking, and ZAEC provably preserves accuracy. No learned parameters, no labels, mechanism-agnostic: that's a good practical package.\n\nThe soft spots are real but not fatal. First, the anchor. If zero-shot CLIP is already underconfident on a dataset, DPE can actually fix calibration by raising confidence, and ZAEC reverses that improvement — the paper's own Figure 4 shows ECE going from 1.69% back to 5.38% on Caltech101. The paper labels this a 'boundary case' and promises adaptive anchors in future work, but the central claim as stated doesn't carry that qualification. A referee should push for a safeguard that detects when zero-shot entropy is a poor anchor, or at least a discussion of prevalence. Second, the main tables lack error bars; the text says standard deviations are in Appendix D, but I can't verify those numbers from the manuscript. Third, no code is released, which makes it hard to trust the exact bisection implementation across 15 datasets. Minor: the 'lowest macro-average ECE' could be driven by datasets where the anchor happens to be sensible; a per-dataset breakdown would help.\n\nOverall, this is a solid paper with a useful method and an honest limitation statement. It deserves a serious referee, but I'd ask for code, error bars in the main tables, and a clearer treatment of when the zero-shot anchor should be abandoned.","headline":"A simple, honest post-hoc calibration layer with a real diagnostic behind it, but the zero-shot entropy anchor has a documented failure mode that the paper underplays.","tokens_in":15292,"tokens_out":2699,"would_cite":true,"duration_ms":25679,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that test-time adaptation sharpens predictions even when nothing else changes, and that restoring each sample's zero-shot entropy by minimal temperature scaling — ZAEC — lowers calibration error without altering accuracy.","keywords":["test-time adaptation","vision-language models","calibration","expected calibration error","predictive entropy","temperature scaling","zero-shot reference","distribution shift"],"falsifier":"Compute, for any held-out dataset and TTA method, the paired per-sample entropy changes and ECE: a single dataset–method pair with strongly negative mean relative entropy change and a decrease in ECE would refute the core association on which ZAEC is built.","tokens_in":14327,"feed_emoji":"🎯","tokens_out":8631,"duration_ms":75194,"temperature":0.7,"pith_summary":"Test-time adaptation (TTA) makes vision-language models more accurate but often wrecks their calibration: confidence grows faster than correctness. The paper shows this happens even when the top-1 prediction is unchanged, a failure mode it calls prediction-preserving sharpening, and that the degree of sharpening relative to the paired zero-shot prediction tracks the rise in Expected Calibration Error. Building on that signal, it proposes ZAEC, a post-hoc method that, for each sample whose adapted entropy fell below its zero-shot entropy, raises the temperature just enough to restore that entropy, leaving all other predictions untouched. ZAEC needs no labels or learned parameters and provably preserves class rankings and accuracy. Across five TTA methods and 15 datasets, it reports the lowest macro-average post-hoc ECE on ViT-B/16, with consistent gains on RN50.","feed_headline":"Zero-shot entropy anchors a label-free TTA calibration fix","feed_subtitle":"ZAEC restores pre-adaptation uncertainty for sharpened predictions, cutting ECE across five TTA methods and 15 datasets.","key_machinery":"The load-bearing object is a sample-specific entropy floor defined by the paired zero-shot prediction. The paper measures sharpening as the normalized entropy change δH_i = (H(p_TTA_i) − H(p_ZS_i))/log K, and for samples in $I^{{−}}$ = {i : H(p_TTA_i) < H(p_ZS_i)} solves for T*_i = min{T ≥ 1 : H(softmax(z_TTA_i/T)) = H(p_ZS_i)} by bisection, leaving samples outside $I^{{−}}$ at T = 1. Proposition 1 ensures temperature scaling strictly increases entropy and preserves the complete class ranking; Proposition 2 shows the chosen temperature is the unique minimum feasible one, is the closest feasible point to the raw TTA output in forward KL divergence among temperature-scaled predictions, and restores entropy exactly to max(H(p_TTA_i), H(p_ZS_i)). The mechanism therefore acts only on the concentration of the predictive distribution, never on its decisions.","core_discovery":"The paper's central claim is that test-time adaptation injects confidence that the underlying correctness does not support, and that the drop in predictive entropy relative to the paired zero-shot prediction is a label-free, sample-specific measurement of that unsupported sharpening. It establishes that, for samples whose top-1 prediction is unchanged by adaptation, confidence rises by 8.78–29.74 points while ECE rises by 8.59–20.34 points on ImageNet-A, and that across dataset–method pairs the mean entropy change and the ECE change are strongly correlated (r = −0.94). On that basis it defines ZAEC: for each sample whose adapted entropy fell below its zero-shot entropy, apply the smallest temperature T ≥ 1 that restores the zero-shot entropy; leave every other prediction untouched. The paper claims this one-sided, label-free correction achieves the lowest macro-average post-hoc ECE among compared post-hoc methods on ViT-B/16 across five TTA methods and 15 datasets, while preserving rankings, top-1 predictions, and accuracy, with consistent gains on RN50.","pith_inferences":["If the entropy-reduction signal generalizes beyond CLIP, the same one-sided temperature restoration could apply to any paired frozen-reference/adaptation setup, such as a fine-tuned model compared with its base checkpoint.","The Caltech101 boundary case suggests a natural extension the paper leaves open: detect when zero-shot confidence sits systematically below empirical accuracy and make the anchor adaptive so the entropy floor is not restored in that regime.","The strong dataset–method correlation between mean relative entropy drop and ECE increase could be repurposed as a label-free monitoring statistic: flag any TTA run with a large mean entropy drop as high-risk for miscalibration before any labels arrive.","One-sided minimal temperature scaling can be viewed as projecting the adapted distribution onto the zero-shot entropy level set in forward KL divergence, which may connect ZAEC to risk-control or conformal post-processing theory."],"forward_implications":["On every entropy-reduced sample, ZAEC provably leaves the complete class ranking, top-1 prediction, and classification accuracy intact, so the calibration gain comes at no accuracy cost.","Because ZAEC requires only paired zero-shot and adapted logits and no training or labels, it can be plugged into prompt-, prototype-, cache-, and logit-based TTA methods alike.","The diagnostic association implies that calibration damage is concentrated among sharpened samples; the ablation confirms that correcting entropy-increased predictions gives little benefit, supporting the one-sided design.","Restoring the full zero-shot entropy floor, rather than a fraction of it, yields the lowest ECE in the evaluated settings, as shown by the monotone improvement in the ablation over restoration strength λ from 0 to 1.","Since ZAEC preserves the adapted ranking, it can serve as a drop-in uncertainty layer for downstream selective prediction or human-review triage without changing the decisions being triaged."],"supporting_citations":[{"why":"Supplies the SaLS baseline that ZAEC must beat; its logit-range reference is the alternative the paper argues is too coarse.","marker":"Murugesan et al. 2024"},{"why":"Provides the temperature scaling baseline and the ECE metric used for all comparisons.","marker":"Guo et al. 2017"},{"why":"TPT is one of the five TTA methods whose raw outputs ZAEC calibrates.","marker":"Shu et al. 2022"},{"why":"Systematic evidence that VLM TTA methods degrade calibration; motivates the problem.","marker":"Sheng et al. 2026"},{"why":"TPS prototype-shifting TTA method used in the evaluation.","marker":"Sui, Wang, and Yeung-Levy 2025"},{"why":"TDA cache-based TTA method used in the evaluation.","marker":"Karmanov et al. 2024"},{"why":"DPE prototype-evolving TTA method used in the evaluation.","marker":"Zhang et al. 2024a"},{"why":"DLAE logit-adjustment TTA method used in the evaluation.","marker":"Wu et al. 2026"}],"fun_headline_variants":["Restore zero-shot uncertainty to fix TTA calibration","ZAEC uses zero-shot entropy to calm TTA-overconfident predictions","Label-free calibration: anchor TTA to zero-shot entropy","Zero-shot entropy predicts and fixes TTA calibration loss","One-sided temperature scaling with zero-shot reference cuts ECE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes each sample's zero-shot entropy is the right uncertainty floor; if the zero-shot model is already over- or underconfident on a dataset, restoring that entropy moves calibration in the wrong direction, as the paper's Caltech101 result shows.","fun_headline_variants_meta":{"raw":{"variants":["Restore zero-shot uncertainty to fix TTA calibration","ZAEC uses zero-shot entropy to calm TTA-overconfident predictions","Label-free calibration: anchor TTA to zero-shot entropy","Zero-shot entropy predicts and fixes TTA calibration loss","One-sided temperature scaling with zero-shot reference cuts ECE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1763,"prompt_tokens":1012,"completion_tokens":751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":669}},"tokens_in":628,"tokens_out":751,"duration_ms":8265,"temperature":1.0,"reasoning_tokens":669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T20:32:44.478315+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, for any held-out dataset and TTA method, the paired per-sample entropy changes and ECE: a single dataset–method pair with strongly negative mean relative entropy change and a decrease in ECE would refute the core association on which ZAEC is built.","supporting_citations":[],"review_version":1}