{"id":"87533cc2-37ed-4ce4-a960-29016c163f24","arxiv_id":"2607.15556","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CARVE is a label-free, cardinality-aware test-time adaptation method that consistently improves multi-label CT diagnosis when the base model is already discriminative and input depth matches pretraining.","lead":"These authors test when adapting a zero-shot CT vision-language model on unlabeled scans actually helps. They find it works only when the model receives full 3D depth and already separates healthy from abnormal cases, and they introduce CARVE, a label-free method that preserves multiple co-occurring abnormalities during adaptation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cardinality estimate (Eq. 8) is unvalidated: AUROC discriminability does not imply p_j sums approximate true label count, so CARVE's top-k set may be wrong exactly in the claimed regime.","rationale":"The reader's weakest assumption — that Eq. 8 requires prompt-pair probabilities to be calibrated enough for their sum to approximate the true label count — is precisely the load-bearing soft spot. My stress test confirms it rather than identifying a different failure. The paper's own Fig. 6 demonstrates that the model's self-confidence signals are weak, which makes the additive-calibration assumption especially fragile under the external shift where the method is meant to apply. However, this is an addressable validation gap, not an observed contradiction of the empirical trends: CARVE consistently improves AUROC across architectures and cohorts, and the reported seed-level σ is very small. The reader's CONDITIONAL verdict already reflects this uncertainty, so no verdict change is warranted. If the proposed oracle-k/calibration experiment shows that Eq. 8 is the bottleneck, then the central claim would need to be weakened or the method revised; until then, the concern remains a conditional risk rather than a demonstrated failure.","tokens_in":18869,"tokens_out":6399,"duration_ms":80493,"concrete_test":"On CT-RATE validation (labels available), compute true positive count k_true per scan. Compare CARVE with (a) Eq. 8 k̂, (b) oracle k_true, (c) fixed k=median true count, and report mean |k̂−k_true| and AUROC for each on RAD-ChestCT/CT-RATE. If oracle-k CARVE clearly exceeds Eq. 8-k CARVE, the cardinality estimator is the bottleneck. Also fit a temperature to validation p_j and use recalibrated k̂ on external data; if gains persist, miscalibration is not the active failure mode.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Secs. 3, 5.2) is that CARVE helps because it matches prompt-pair multi-label structure via sample-specific cardinality. That claim rests on Eq. 8: k̂ = clip(round(Σ_j p̄_j), k_min, k_max). This is an additive-calibration assumption: the sum of Bernoulli probabilities must approximate the true positive count. AUROC, used to define 'base model already discriminative,' is order-based and invariant to monotone transforms, so a model can separate present from absent while all p_j sit near 0.45/0.55, yielding k̂≈L/2 instead of e.g. 2–3. External shift makes miscalibration likely; Fig. 6 shows the same probabilities carry little per-scan viability signal. If k̂ is wrong, Eq. 9's top-k index set can exclude true positives and the −λ_neg log(1−p) term actively suppresses them, so the reported gains over TENT/ML-TTA would not be evidence for cardinality-aware adaptation. The paper never validates k̂ against true counts or ablates Eq. 8, leaving the central mechanism unsecured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies when test-time adaptation (TTA) helps zero-shot 3D CT vision-language models (VLMs). Through a controlled diagnostic protocol with two backbones (CT-CLIP, fVLM) and four datasets (CT-RATE, RAD-ChestCT, CC-CCII, LUNA16), it identifies preconditions for TTA: the input pipeline must preserve the volumetric depth structure expected by the encoder, and the base representation must transfer to the target cohort. For the regime where the base model is already discriminative, the paper proposes CARVE, which estimates a sample-specific positive-label cardinality, optimizes a top-k objective on a set of retained weak 3D views, and updates only visual normalization parameters in an episodic manner. The main empirical claim is that CARVE provides consistent AUROC improvements of roughly 0.03–0.05 over no adaptation and over existing TTA baselines (TENT, RLCF, ML-TTA) when the base model already separates present from absent abnormalities, while the baselines show near-zero or inconsistent gains.","tokens_in":19215,"tokens_out":4887,"duration_ms":56060,"significance":"If the findings hold, the paper makes a useful contribution: it is, to my knowledge, the first TTA method designed for multi-label zero-shot 3D CT diagnosis, and it clearly delineates regimes where TTA helps or does not. The diagnostic study is careful in isolating depth structure and base representational transfer, and the authors are transparent about limitations (e.g., the dispersion gate in Fig. 6 is explicitly shown to be a weak per-scan viability signal). The code is released, and the protocol controls for label shift by fixing hyperparameters on the internal validation set. However, the central mechanism—the sample-specific cardinality estimate of Eq. (8)—is not directly validated, and the reported statistical evidence is based on seed-level variability rather than patient-level uncertainty. These issues are load-bearing for the central claim, so the paper cannot be accepted in its current form.","major_comments":[{"comment":"The cardinality estimate k_hat = clip(round(sum_j pbar_j), k_min, k_max) assumes that the sum of per-label Bernoulli probabilities approximates the true positive-label count (additive calibration). AUROC discriminability is order-based and does not imply such calibration; a model can have high AUROC while all probabilities sit near 0.5, yielding k_hat near L/2. Under external distribution shift, miscalibration is likely, and Fig. 6 shows that the same probabilities carry little per-scan viability signal. If k_hat is wrong, the top-k index set in Eq. (9) can exclude true positives, and the negative loss term actively suppresses them. The paper never validates k_hat against true counts nor ablates Eq. (8) (e.g., comparing fixed k, oracle k, or a recalibrated estimator). Since the central claim attributes CARVE's gains to cardinality-aware adaptation, this mechanism is load-bearing and curr","section":"Sec. 4.4, Eq. (8)"},{"comment":"The reported ±σ is over adaptation seeds, not patients. The authors explicitly state in Appendix B that the standard deviation reflects run-to-run stability under stochastic inference-time distortions and is \"not statistical uncertainty over patients.\" AUROC improvements are about 0.03–0.05, while seed-level σ values are often 0.0001–0.003. Without patient-level bootstrap confidence intervals or an appropriate significance test across patients, the claim that CARVE \"provides the most consistent improvements\" is not statistically grounded. This is a load-bearing issue because the central conclusion rests on the consistency of small-magnitude gains across cohorts.","section":"Sec. 5.1 / Appendix B"},{"comment":"The implementation of the baselines RLCF and ML-TTA is not specified. ML-TTA [19] in its original form relies on caption retrieval for pseudo-label filtering; zero-shot CT scans have no paired captions. If ML-TTA is run without its caption component, it is not the published method, and the comparison may be unfair; if captions are simulated or omitted, that should be stated. Similarly, RLCF [23] uses a CLIP reward whose exact computation for CT-CLIP/fVLM is not described. Without these details, the experimental comparison is not reproducible, undermining the conclusion that CARVE outperforms strong baselines.","section":"Sec. 5.1, Methods and protocol"}],"minor_comments":[{"comment":"The hyperparameter sensitivity sweep is performed post-hoc on the external RAD-ChestCT test set. The authors state it is not used for selection, but this is still an exploration of a test cohort and should be acknowledged more strongly, e.g., as a limitation on the strength of the robustness claim.","section":"Sec. 5.3, Fig. 2"},{"comment":"The fVLM experiments omit the official organ-ROI segmentation crop. The authors disclose this omission, which is good, but the resulting numbers should be presented as an approximation rather than a direct comparison to published fVLM results.","section":"Appendix B, fVLM evaluation"},{"comment":"The choice of k_min=1 and k_max=8 is reported in Appendix A but not justified in the main text. It would be helpful to explain whether the upper bound is derived from the label-space size, a prevalence prior, or an empirical heuristic, since this directly affects the behavior of Eq. (8).","section":"Sec. 4.4, Eq. (8)"},{"comment":"The phrase \"the first TTA method for this setting\" is strong; suggest softening to \"to our knowledge, the first\" to hedge against possible concurrent work.","section":"Abstract / Introduction"},{"comment":"Figure captions are dense, especially Fig. 4, which mixes two failure axes, gradient magnitude and direction. Expanding the caption or adding a short text explanation in Sec. 5.3 would improve readability.","section":"Throughout"},{"comment":"Minor typo: \"We minimize it over the adapted normalization parameters φ and then predict on the original unperturbed volume.\" The sentence is missing a period after \"volume.\" Also, the equation label \"L_TTA\" is referred to inconsistently in the text.","section":"Sec. 4.5, Eq. (10)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision/medical imaging journal and the diagnostic framing is genuinely useful. The main risk is the unvalidated cardinality estimator: if the authors can add a calibration study, an ablation of Eq. (8), and patient-level confidence intervals, the central claim would become solid. The baseline implementation details are also essential for reproducibility. I see no evidence of novelty disclosure concerns; the related work is appropriately cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, this is one of the few TTA papers that seriously asks when adaptation helps rather than assuming it always does, and the diagnostic part is genuinely useful. Second, the method's core mechanism—estimating the positive-label count from the model's own probabilities—is plausible but never directly validated, and the central claim leans on it more than the experiments justify.\n\nWhat's actually new: CARVE is the first TTA method designed for zero-shot multi-label 3D CT VLMs with prompt-pair Bernoulli probabilities. The memory-efficient retained-view adaptation is a practical contribution for volumetric inference. The empirical trends are consistent: CARVE improves AUROC by about 0.03–0.05 across CT-RATE, RAD-ChestCT, CC-CCII, and LUNA16 on both CT-CLIP and fVLM, while TENT, RLCF, and ML-TTA mostly produce near-zero or inconsistent gains. The paper is also honest about its limits: it explicitly says the dispersion gate is not a per-scan viability detector, and that no objective fixes a near-chance base model. That honesty is real and welcome.\n\nThe soft spots. Eq. 8 estimates k-hat as the rounded sum of per-label Bernoulli probabilities. That assumes those probabilities are additively calibrated—their sum approximates the true label count. AUROC, which the paper uses to define \"the base model is already discriminative,\" is order-based, so a model can rank present vs. absent perfectly while all probabilities sit near 0.5, making k-hat roughly L/2. The paper never validates k-hat against true counts and never ablates it. Figure 6 shows the same probabilities carry little per-scan viability signal. So the \"cardinality-aware\" part of CARVE is unsecured; the reported gains could in principle come from the retained-view selection or the norm-only update. That's addressable—an ablation with true k or a fixed k would settle it—but as written it's a real gap. Also, all reported variability is seed-level sigma, not patient-level confidence intervals, so the effect sizes look more certain than they are. The code link has no commit hash, a minor reproducibility annoyance.\n\nThis paper is for anyone working on TTA or medical VLMs. It deserves a serious referee: the question is important, the diagnostic is well-designed, and the weaknesses are fixable rather than structural. Send it to review; ask the authors to either validate the cardinality estimate or show that CARVE works without it.","headline":"A careful, honest empirical study of when TTA helps zero-shot CT VLMs; the main weakness is that CARVE's cardinality estimator (Eq. 8) is never directly validated.","tokens_in":19685,"tokens_out":3040,"would_cite":true,"duration_ms":30385,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Test-time adaptation helps zero-shot 3D CT vision-language models only when the scan keeps its native depth and the base model already distinguishes present from absent abnormalities; in that regime, the new CARVE method gives consistent mu","keywords":["test-time adaptation","zero-shot","3D CT","vision-language model","multi-label classification","entropy minimization","cardinality estimation","distribution shift"],"falsifier":"Compute the per-scan cardinality estimate k̂ against the true number of abnormalities on an external cohort; if the mean absolute error is large or if CARVE's AUROC gain disappears when k̂ is replaced by the true count, the central claim fails.","tokens_in":18776,"feed_emoji":"🩻","tokens_out":4280,"duration_ms":44880,"temperature":0.7,"pith_summary":"Test-time adaptation is not automatically useful for zero-shot 3D CT vision-language models. The paper shows two conditions must hold: the input volume must preserve the depth structure the encoder was pretrained on (cutting depth alone dropped internal AUROC by more than 0.12), and the base model's prompt-pair probabilities must already separate present from absent abnormalities. When those conditions hold, standard entropy objectives are structurally mismatched to the multi-label prompt-pair setup, so the paper introduces CARVE, which estimates how many abnormalities a scan actually has and sharpens exactly that many labels. In experiments, CARVE gives the most consistent AUROC improvements across backbones and label settings when the base is discriminative, while no objective can create discrimination when the base representation is near chance.","feed_headline":"Adapting CT models at test time pays off only under two conditions","feed_subtitle":"A new method, CARVE, then estimates per-scan label counts and preserves co-occurring abnormalities with consistent AUROC gains.","key_machinery":"The central object is the prompt-pair Bernoulli probability p_j = σ(s_j^+ − s_j^-), obtained by a two-class softmax over the positive and negative text-prompt scores for each abnormality. CARVE's mechanism: estimate the number of present abnormalities as k̂ = clip(round(Σ_j p̄_j), k_min, k_max) from the retained-view average; then optimize L_CARVE = −(1/|P|)Σ_{j∈P} log p_j − λ_neg (1/|N|)Σ_{j∈N} log(1−p_j), where P is the top-k̂ label set and N its complement. This replaces standard entropy minimization, which collapses co-occurring positives, with an objective that preserves exactly k̂ abnormalities. View selection scores stochastic weak 3D views without gradients and retains the K lowest-e","core_discovery":"The paper's central claim is that TTA helps zero-shot 3D CT VLMs only in a specific regime: when the volumetric input matches the encoder's pretraining depth and the base prompt-pair probabilities already separate present from absent abnormalities. Within that regime, the correct adaptation objective is cardinality-aware. CARVE estimates the scan-specific number of positive abnormalities k̂ from the sum of per-label Bernoulli probabilities, then optimizes a top-k̂ objective that sharpens the estimated positive set while suppressing the rest, using only the most confident low-entropy weak views for memory-efficient gradient updates. The paper demonstrates that this beats entropy-based baselin","pith_inferences":["The cardinality estimate in Eq. 8 assumes the zero-shot probabilities are calibrated enough that their sum approximates the true label count; if miscalibration is large under distribution shift, the top-k̂ objective will sharpen an incorrect positive set. A direct calibration check on external cohorts would test this.","The paper's framing implies a deployment rule: first measure base discriminability on a small labeled sample (or via dispersion) and only then decide whether to run TTA. This could be formalized into a decision criterion with a threshold on base separation.","Since the base model gap dominates the adapter gap under external shift, annotations spent on adapting the base representation (e.g., few-shot fine-tuning) may pay off more than any TTA objective; this is an implicit priority for practitioners."],"forward_implications":["If CARVE's conditions hold, zero-shot CT VLMs can be adapted per-scan without labels, making cross-institution deployment more reliable when the base is already discriminative.","Depth reduction (e.g., z=40 from z=240) is not benign: it can hurt AUROC by >0.12 before adaptation, so deployment pipelines must preserve native depth.","CARVE transfers to smaller label spaces (three-class and binary) without modification because the cardinality estimate reduces naturally.","In the paper's experiments, when the base is near chance, adaptation only shifts the operating point, so effort should go to improving base transfer rather than TTA.","The dispersion gate is not a reliable per-scan viability detector; it only abstains when predictions collapse, so it does not rescue adaptation in the non-discriminative regime."],"fun_headline_variants":["CT zero-shot TTA: only helps if depth and base transfer","CARVE conditions: TTA works when base is already discriminative","Test-time adaptation for CT: two conditions for success, then CARVE","When TTA fails for CT VLMs, and when CARVE fixes it","Zero-shot CT TTA: conditional, and CARVE targets the right regime"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes the sum of the zero-shot per-label probabilities is a calibrated estimate of the true number of abnormalities in the scan, so rounding that sum gives the right positive-label count to sharpen.","fun_headline_variants_meta":{"raw":{"variants":["CT zero-shot TTA: only helps if depth and base transfer","CARVE conditions: TTA works when base is already discriminative","Test-time adaptation for CT: two conditions for success, then CARVE","When TTA fails for CT VLMs, and when CARVE fixes it","Zero-shot CT TTA: conditional, and CARVE targets the right regime"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000415,"raw_usage":{"total_tokens":2036,"prompt_tokens":859,"completion_tokens":1177,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1081}},"tokens_in":603,"tokens_out":1177,"duration_ms":11994,"temperature":1.0,"reasoning_tokens":1081,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T22:57:18.940056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the per-scan cardinality estimate k̂ against the true number of abnormalities on an external cohort; if the mean absolute error is large or if CARVE's AUROC gain disappears when k̂ is replaced by the true count, the central claim fails.","supporting_citations":[],"review_version":1}