{"id":"da29dbb8-d629-416d-9f13-d224f574924e","arxiv_id":"2412.14097","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"An unlabeled test-time adaptation method updates concept vectors, the linear predictor, and adds residual concepts so that concept bottleneck classifiers stay accurate under distribution shifts.","lead":"Concept-based classifiers make a foundation model's predictions easier to explain, but they can break when the data shifts after deployment. This paper introduces an adaptation method that updates the concept definitions and the prediction rule on unlabeled test data, recovering accuracy that would otherwise be lost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-label reliability is the load-bearing assumption; severity-5 evidence already shows CONDA can underperform no adaptation, so the headline gain is not established for standard TTA regimes.","rationale":"The reader's weakest-assumption analysis identifies pseudo-label accuracy as the load-bearing premise, and my reading agrees. The paper's three adaptation objectives (Eqs. 9, 11, 14) all consume pseudo-labels, so the entire adaptation signal is the quality of the ZS+LP ensemble. If that ensemble is accurate, CONDA can plausibly distill non-interpretable FM knowledge into an interpretable bottleneck; if it is inaccurate, adaptation should degrade. The Appendix F, Table 8 severity-5 Metashift results show exactly this degradation, and Appendix C.1 explains that the main CIFAR-C evaluation deliberately uses severity 2 because severity 5 destroys the pseudo-label oracle. This makes the empirical scope of the central claim narrower than the abstract suggests, but it does not invalidate the method: the paper provides useful ablations, an oracle-label upper bound in Table 5, negative results in the appendix, and a clear account of when the method should not be applied. A severity-5 rerun with per-corruption reporting would settle whether the failure is localized to Metashift or generalizes to standard corruption benchmarks. Since the reader's conditional verdict already captures this uncertainty, no verdict change is needed.","tokens_in":26526,"tokens_out":9072,"duration_ms":83240,"concrete_test":"Run the CONDA implementation described in Algorithm 1 on CIFAR10-C and CIFAR100-C at severity 5, plus the Appendix F Metashift severity-5 setting, using the Table 4 hyperparameters; report AVG/WG per corruption type and also report the pseudo-label accuracy of ZS alone, LP alone, and the confidence ensemble on the same target sets. If CONDA's WG falls below the unadapted CBM on more than a small number (e.g., 3 of 15) corruption types, the abstract's claim should be scoped to settings where the ZS/LP pseudo-labels are reliable; if CONDA remains robust across severity 5, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"All three adaptation stages are supervised by pseudo-labels: CSA uses them as class proxies in the Mahalanobis distances (Eqs. 6-9), while LPA and RCB minimize cross-entropy against them (Eqs. 11 and 14). The pseudo-labels come from a confidence-based ensemble of the zero-shot (ZS) and linear-probing (LP) predictors, which are the same feature-based models used as comparison baselines in Table 1. CONDA's post-adaptation accuracy is therefore heavily dependent on the quality of these pseudo-labels, and its claimed 'alignment with non-interpretable classification' is to a large degree an inheritance from those predictors rather than an independent property of concept-bottleneck adaptation. The paper honestly documents the failure mode in Appendix F and Table 8: at severity-5 noise on Metashift, a CONDA variant performs worse than the unadapted CBM, because the ZS/LP pseudo-labels are not reliable. The main text, however, evaluates CIFAR-C at severity 2, with Appendix C.1 explicitly stating that severity 5 'severely hurts' the backbone and makes it impossible to use as a decent pseudo-label oracle. Thus the central claim that CONDA 'boosts post-deployment accuracy by up to 28%' is not established for the standard severity-5 corruption regime, and the boundary of the claimed improvement is exactly where the pseudo-label assumption fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies test-time adaptation of concept bottleneck models (CBMs) built on frozen foundation models under distribution shifts. It formalizes three failure modes—non-robust concept bottleneck under low-level shift, non-robust classifier under concept-level shift, and incomplete concept set—and proposes CONDA, a three-stage adaptation method: concept-score alignment (CSA, Eq. 9), linear-probing adaptation (LPA, Eq. 11), and a residual concept bottleneck (RCB, Eq. 14). All stages are supervised by pseudo-labels from an ensemble of zero-shot and linear-probing predictors. Experiments on CIFAR10-C/CIFAR100-C, Waterbirds, Metashift, and Camelyon17, across three CBM construction methods, report accuracy gains in several settings, with the largest gains on concept-level shifts. The paper includes ablations, interpretability analysis, complexity analysis, and a candid limitations appendix.","tokens_in":26906,"tokens_out":7067,"duration_ms":61832,"significance":"CONDA is, to my knowledge, the first test-time adaptation framework specifically for concept bottleneck models with a foundation-model backbone. If the results held broadly, the paper would make a real contribution by showing that interpretable concept-based pipelines can be adapted online without labels. Strengths include the clean decomposition of failure modes, the residual concept bank idea, the use of multiple CBM construction methods, and the honest reporting of negative results (Appendix F, Table 8) and pseudo-label sensitivity (Table 5). However, the main claims are partly undercut by the method's dependence on the same non-interpretable predictors it aims to match, and by the non-standard severity-2 CIFAR-C evaluation; these issues are fixable but require re-framing or additional experiments.","major_comments":[{"comment":"The adaptation objectives in all three stages are supervised by pseudo-labels taken from the zero-shot (ZS) and linear-probing (LP) predictors, which are exactly the non-interpretable baselines used as comparison targets in Table 1. As a result, the statement that CONDA 'aligns the CBM performance with that of non-interpretable classification' (abstract) is to a nontrivial degree a propagation of ZS/LP accuracy rather than an independent property of the concept-bottleneck adaptation. Table 5 shows the strong dependence on pseudo-label quality (e.g., CIFAR100-C AVG increases from 53.88 with ZS/LP pseudo-labels to 97.31 with perfect pseudo-labels), and Table 8 documents a regime where poor pseudo-labels cause CONDA to underperform the unadapted CBM. The manuscript discloses this in Appendix F, but the main text and abstract should either be reworded to state the conditional nature of the claim or supplemented with an ablation that uses a pseudo-labeling scheme not derived from the comparison baselines (e.g., entropy minimization or rotation-based self-supervision).","section":"Section 3 (Pseudo-labeling), Eqs. (9), (11), (14); Tables 5 and 8"},{"comment":"The low-level shift benchmark is evaluated at corruption severity 2 rather than the standard severity 5, with the rationale that severity 5 degrades the backbone's pseudo-label oracle. This choice is consequential because low-level shift is one of the three failure modes motivating CONDA, and CIFAR-C is the only low-level-shift dataset. The negative result in Table 8 (Metashift with severity-5 Gaussian noise) shows exactly the boundary regime where the method fails. To establish the headline 'boosts post-deployment accuracy by up to 28%' for low-level shifts, please report CIFAR-C at multiple severities (at least severity 5) or, if that is infeasible due to pseudo-label failure, state in the abstract and Section 4.2 that the low-level-shift results are limited to mild corruptions.","section":"Appendix C.1; CIFAR-C evaluation; Table 8"},{"comment":"CONDA reduces accuracy for the Yeh et al. (2020) bottleneck on three of five datasets (CIFAR10-C AVG 89.76 to 85.14, CIFAR100-C AVG 72.33 to 70.82, Camelyon17 AVG 95.01 to 92.54). The caption's claim that CONDA 'significantly improves' target accuracy is therefore not true for a substantial subset of the reported configurations. Appendix F acknowledges this, but the main text should explicitly delimit the method's scope (e.g., to concept banks with interpretable/annotatable concepts) and, ideally, analyze why the optimization-based concepts of Yeh et al. behave differently. Without this qualification, the central claim of broad effectiveness is overgeneralized.","section":"Table 1; Appendix F"}],"minor_comments":[{"comment":"The definition of the Mahalanobis distance has unbalanced parentheses: Dmah(xt ; µy, Σy) should be (vC(xt) − µy)ᵀ Σ_y⁻¹ (vC(xt) − µy).","section":"Section 3.1"},{"comment":"The main text says Figure 3 corresponds to adapting the CBM of Yeh et al. (2020), while the figure caption says it is for the CBM method of Yuksekgonul et al. (2023); please reconcile this discrepancy.","section":"Section 4.3 and Figure 3"},{"comment":"The pseudo-labeling ensemble is described informally as taking 'the class predicted with higher confidence across both predictors'; a precise definition (e.g., maximum softmax probability or margin) would improve reproducibility.","section":"Section 3 and Algorithm 1"},{"comment":"The method assumes access to source-domain class-conditional Gaussian statistics {(µy, Σy)}; since these are derived from labeled source data, the claim of operating 'without access to the source dataset' should be qualified to say that precomputed source statistics are required.","section":"Section 3.1"},{"comment":"The hyper-parameters are dataset-specific; please state how they were selected (e.g., source-domain validation, grid search) rather than reporting only the final values.","section":"Table 4"},{"comment":"The phrase 'boosts post-deployment accuracy by up to 28%' should be tied to a specific table entry (e.g., Waterbirds AVG for Yuksekgonul et al., 32.03 to 60.69) rather than left as a global claim, since the method does not improve all reported configurations.","section":"Abstract and Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is unusually candid about its limitations, and the negative results in Table 8 and Appendix F are a credit to the authors. However, the abstract and Section 4.2 overstate the method's generality relative to Table 1 and the severity-2 choice for CIFAR-C. I believe the issues are addressable within the scope of the paper: re-framing the claims, adding severity-5 results or an explicit scope limitation, and clarifying the pseudo-label dependence. No ethical concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First take: this is a genuine first: TTA for CBMs with a frozen FM backbone, and the three-component recipe (CSA, LPA, RCB) maps cleanly onto the failure modes they define. The reported gains on Waterbirds and Metashift for the Yuksekgonul and Oikarinen bottlenecks are substantial — e.g., Waterbirds AVG from 32 to 60 for PCBM — and the paper does a lot of things right: honest ablations, hyperparameter sensitivity, pseudo-labeling variants, a perfect-label upper bound, and a stated limitation section that actually names where the method fails. I appreciate that they report negative results in Appendix F/Table 8 instead of hiding them.\n\nThe soft spot is exactly where the stress-test note lands. All three adaptation stages are supervised by pseudo-labels from a ZS/LP ensemble, and those are the same feature-based predictors used as baselines. So the \"on par with non-interpretable classification\" claim is partly inherited: if ZS/LP are good, CONDA can ride them; if they are bad, CONDA can be worse than doing nothing. The severity-5 Metashift results in Table 8 show CONDA underperforming the unadapted CBM, and the main text sidesteps this by evaluating CIFAR-C at severity 2 instead of the standard severity 5. That's a meaningful gap between the abstract's \"up to 28%\" claim and what is actually established. I also note that Table 1 shows CONDA hurting accuracy for the Yeh et al. bottleneck on CIFAR10, CIFAR100, and Camelyon17; the paper mentions this in Appendix F, but it should be in the main text.\n\nMinor but worth saying: no code is provided, which makes the many hyperparameters (five per dataset plus r, ngrad, k) hard to trust as anything other than tuned to the reported benchmarks. The source-domain class statistics assumption is stated clearly, so that's fine as a limitation, not a flaw.\n\nOverall: the central idea is sound, the engineering is careful, and the negative evidence is honestly reported. What's missing is a narrowed claim: CONDA helps when the backbone's ZS/LP pseudo-labels are reliable, and the evaluation should include severity-5 and the failure cases in the main text. I would send this to review — a serious referee could push the authors to add code, TTA baselines (e.g., TENT on the feature classifier), and a clearer statement of the pseudo-label condition. I'd cite it if working on interpretable robustness; it's a useful reference point even with the caveats.","headline":"A genuine first stab at test-time adaptation for concept bottlenecks, with real gains in some settings but a headline claim that softens once you look at pseudo-label dependence and the severity-2 evaluation.","tokens_in":27385,"tokens_out":1927,"would_cite":true,"duration_ms":17655,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes CONDA, the first test-time adaptation method for concept bottleneck models with foundation-model backbones, and reports that it restores post-deployment accuracy by up to 28% using only unlabeled target data.","keywords":["concept bottleneck models","test-time adaptation","distribution shift","foundation models","interpretability","pseudo-labeling","concept alignment","residual concept bottleneck"],"falsifier":"Evaluate CONDA on a target batch whose pseudo-labels are deliberately replaced with uniform random labels; if the adapted batch accuracy does not degrade relative to the unadapted CBM, the paper's stated dependence on pseudo-label quality would be contradicted, whereas the paper's own severity-5 Metashift result already exhibits the predicted degradation.","tokens_in":26336,"feed_emoji":"🎯","tokens_out":7101,"duration_ms":61369,"temperature":0.7,"pith_summary":"Concept bottleneck models make foundation-model predictions interpretable by routing features through a small set of high-level concept scores before classification, but the paper shows that these concept-level predictions are not automatically robust to distribution shifts at deployment. It proposes CONDA, a test-time adaptation method that updates the concept bank and the prediction layer using only unlabeled batches from the target domain, with no access to the source training data. Across CIFAR-C, Waterbirds, Metashift, and Camelyon17 benchmarks, the method reports post-deployment accuracy gains of up to 28%, bringing concept-based predictions to parity with non-interpretable zero-shot and linear-probing baselines. The overall claim is that interpretability and deployment robustness are not in tension if the concept bottleneck itself is adapted rather than frozen.","feed_headline":"Adapting concept banks lifts deployment accuracy by up to 28%","feed_subtitle":"CONDA refreshes concepts and classifier with unlabeled target data, matching black-box accuracy.","key_machinery":"The load-bearing object is the adaptable concept bottleneck itself, treated as a matrix of unit-norm concept vectors followed by a linear head. CSA re-solves the concept matrix so that, under pseudo-labels, Mahalanobis distances from target concept scores to source class-conditional Gaussians shrink within each class and grow across classes. LPA then minimizes the cross-entropy of the main branch against pseudo-labels with an elastic-net penalty that keeps the head interpretable. RCB appends a second branch of fresh concept vectors and a linear head, trained jointly with a cosine-similarity penalty for diversity and a coherency term that ties each new concept to the target patches that activate it most; because the branches share the backbone, the combined predictor is exactly the main branch plus the residual branch, equivalent to augmenting the concept matrix with new rows and the weight matrix with new columns. These three steps isolate the three failure modes so that each adaptation stage can be ablated and attributed.","core_discovery":"The central claim is that a deployed concept bottleneck need not be static: each failure mode caused by distribution shift corresponds to a component that can be repaired online. The paper formalizes two shift types, low-level shifts that change inputs but not concept semantics and concept-level shifts that change high-level semantics, and names three failure modes: a non-robust concept mapping, a classifier that no longer maps concepts to labels consistently, and a concept set that is incomplete for the target domain. CONDA addresses them in order: concept-score alignment (CSA) adapts the concept vectors so target concept scores match the source class-conditional distributions; linear-probing adaptation (LPA) re-tunes the label predictor using pseudo-labels; and a residual concept bottleneck (RCB) adds new concepts to cover what the original bank missed. With a frozen foundation-model backbone and only unlabeled test batches, the paper reports target-domain accuracy gains up to 28% and worst-group gains that often exceed the non-interpretable baselines, while the adapted concept weights visibly shift toward target-relevant semantics, such as land concepts contributing to waterbird predictions and shelf concepts to both Metashift classes.","pith_inferences":["Beyond the paper, the three-stage recipe of align, re-fit the head, and extend the concept set is a template for other interpretable architectures: any bottleneck that exposes a linear concept projection could be repaired by the same sequence, even if the concepts are built differently.","The residual branch's ability to discover target-specific concepts suggests a testable extension: feed the learned residual concept vectors back into the original concept-annotation pipeline, then measure whether adding them to a static concept bank recovers most of CONDA's gain without online adaptation.","Because CSA only needs class-conditional Gaussian statistics of source concept scores, a deployment team could publish those statistics alongside the model and let each test site run CONDA without transferring source data; the paper does not explore the privacy or bandwidth aspects of this protocol.","The dependence on pseudo-label quality implies an upper bound on the method: with perfect pseudo-labels, the paper's own ablation shows near-perfect accuracy on Metashift and large gains on CIFAR100-C, so future improvements in test-time pseudo-labeling should transfer almost directly into CONDA's accuracy."],"forward_implications":["If the paper's claim is correct, a deployed concept bottleneck can recover from distribution shifts without any labeled target data or access to the source dataset, matching or exceeding non-interpretable baselines in worst-group accuracy.","The failure-mode decomposition implies that practitioners can choose the component they need: CSA for low-level input corruptions, LPA and RCB for concept-level or semantic shifts, with little cost from the unused components.","The residual branch demonstrates that the original concept bank may be incomplete, and that automatically discovered residual concepts can restore both accuracy and interpretability, as seen with bird-related concepts on Waterbirds.","Because adaptation is online and batchwise, the method can be deployed in streaming settings where target data arrives continuously and the model must not be reset between batches.","The paper's component analyses support a broader design rule: adapting the bottleneck layer matters most for input-level shifts, while adapting the prediction layer matters most for output-level shifts."],"supporting_citations":[{"why":"Defines Concept Bottleneck Models, the architecture that CONDA adapts at test time.","marker":"Koh et al. (2020)"},{"why":"Supplies the post-hoc CBM baseline and the concept-activation-vector concept bank used in most experiments.","marker":"Yuksekgonul et al. (2023)"},{"why":"Supplies a label-free CBM baseline whose concept set is generated from language models; one of the three tested construction methods.","marker":"Oikarinen et al. (2023)"},{"why":"Supplies the unsupervised concept-discovery baseline and the completeness-style objective that RCB's coherency term modifies.","marker":"Yeh et al. (2020)"},{"why":"Provides the class-aware feature alignment idea behind CSA's intra-class and inter-class distance loss.","marker":"Jung et al. (2023)"},{"why":"Provides the refined pseudo-labeling variant that the paper tests as an alternative to its simple ensemble pseudo-labeling.","marker":"Chen et al. (2022)"},{"why":"Defines the online test-time adaptation protocol of adapting on fixed batches sequentially, which CONDA adopts.","marker":"Wang et al. (2021)"},{"why":"Provides the CIFAR-C corruption benchmark and severity levels used for low-level shift evaluation and for the negative-results experiment.","marker":"Hendrycks & Dietterich (2019)"},{"why":"Supplies evidence that fine-tuning only certain layers helps under different shift types, which the paper uses to interpret why CSA and LPA matter differently.","marker":"Lee et al. (2023)"},{"why":"Provides the CLIP backbone and zero-shot prediction protocol used as both a baseline and one source of pseudo-labels.","marker":"Radford et al. (2021)"}],"fun_headline_variants":["Adaptive concept banks lift accuracy by 28% under distribution shift","Concept adaptation with unlabeled data matches black-box accuracy","CONDA self-adapts concepts using only unlabeled target data","Interpretable concept models adapt to shifts, gaining up to 28%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the pseudo-labels produced by the zero-shot and linear-probing ensemble are accurate enough to supervise all three adaptation objectives; Appendix F's Table 8 shows that on severity-5 Gaussian noise over Metashift, where both baselines collapse, CONDA's adapted accuracy falls below the unadapted CBM, so the claim is conditional on a backbone that is at least partly reliable on the target domain.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive concept banks lift accuracy by 28% under distribution shift","Concept adaptation with unlabeled data matches black-box accuracy","CONDA self-adapts concepts using only unlabeled target data","Interpretable concept models adapt to shifts, gaining up to 28%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1794,"prompt_tokens":1021,"completion_tokens":773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":700}},"tokens_in":637,"tokens_out":773,"duration_ms":6895,"temperature":1.0,"reasoning_tokens":700,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:29:55.137807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate CONDA on a target batch whose pseudo-labels are deliberately replaced with uniform random labels; if the adapted batch accuracy does not degrade relative to the unadapted CBM, the paper's stated dependence on pseudo-label quality would be contradicted, whereas the paper's own severity-5 Metashift result already exhibits the predicted degradation.","supporting_citations":[{"cited_title":"Concept bottleneck models","cited_arxiv_id":null,"evidence_quote":"Defines Concept Bottleneck Models, the architecture that CONDA adapts at test time."},{"cited_title":"On completeness-aware concept-based explanations in deep neural networks","cited_arxiv_id":null,"evidence_quote":"Supplies the unsupervised concept-discovery baseline and the completeness-style objective that RCB's coherency term modifies."},{"cited_title":"CAFA : Class-aware feature alignment for test-time adaptation","cited_arxiv_id":null,"evidence_quote":"Provides the class-aware feature alignment idea behind CSA's intra-class and inter-class distance loss."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP backbone and zero-shot prediction protocol used as both a baseline and one source of pseudo-labels."}],"review_version":1}