{"id":"2e3a3494-ea78-48d2-934e-1706f0581128","arxiv_id":"2506.06076","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Full conformal adaptation lets few-shot adapted medical vision-language models produce prediction sets with the same coverage guarantee as split conformal prediction, while reducing average set size by up to 27%.","lead":"This paper shows a way to give medical image AI models formal reliability guarantees even after they are quickly adapted to a new task with only a few labeled examples. The method combines classical full conformal prediction with a fast, training-free linear probe solver that is about 150 times faster than standard adaptation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The balanced few-shot support set used in the experiments violates exchangeability with the imbalanced test sets, so the central coverage-guarantee claim is never tested in the regime where it is supposed to hold.","rationale":"The reader's weakest_assumption identifies exchangeability, and my analysis converges on the same point but sharpens it: the experimental protocol in Section 5.1 uses a balanced support set, which systematically breaks exchangeability with the naturally imbalanced test sets. This is not a peripheral caveat; it means the empirical coverage reported in Table 1 and Figure 2 is not a test of the theoretical guarantee at all. The guarantee is conditional on exchangeability, and the paper's own setting explicitly violates that condition. A controlled experiment with an exchangeable adaptation set would either validate the central claim under its assumptions or reveal a gap in the FCA argument. The SS-Text approximation is not a threat to validity because it is a closed-form symmetric function of the data, so the permutation-invariance requirement of FCP is preserved. The missing error bars for the 27% efficiency claim are secondary, since the per-dataset results in Figure 2 provide some qualitative support, and the reader already flagged it. Overall, the paper is methodologically sound in theory but incomplete in empirical validation, which matches the reader's CONDITIONAL verdict; hence no adjustment is needed.","tokens_in":13836,"tokens_out":17649,"duration_ms":176334,"concrete_test":"Select 2-3 datasets (e.g., SICAPv2, MESSIDOR, CheXpert). For each, create a support set by sampling K shots per class from a training patient split that preserves the natural class proportions of the test set (or a separate held-out fold), run FCA with SS-Text at alpha=0.10 and 0.05 over 20 seeds, and compare empirical coverage to the target. If coverage is within sampling error of 1-alpha, the guarantee holds exchangeably and Table 1's undercoverage is attributable to balanced-support-set shift; if not, the FCA validity proof has a gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim (Section 4.2, Eq. 5) is that FCA inherits the FCP coverage guarantee. That guarantee requires an exchangeable adaptation set with respect to each test point. The experiments, however, deliberately use a class-balanced support set of K shots per class (Section 5.1) while the medical test sets are naturally imbalanced. This breaks exchangeability even for random patient splits, because the label marginal distribution differs between adaptation and test. The authors acknowledge this in Section 5.2 ('the label-marginal distribution of the testing data is unknown, and typical few-shot adaptation pipelines assume a balanced support set'). Consequently, the coverage numbers in Table 1 (e.g., FCA LAC at alpha=0.10 reaches 0.896, not 0.900) are obtained in a regime where the guarantee is not expected to hold, and the paper never demonstrates empirically that FCA attains nominal coverage when exchangeability is satisfied. The SS-Text solver is permutation-invariant, so it does not threaten validity, but the experimental protocol leaves the core validity claim untested in its intended regime.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes full conformal adaptation (FCA), a transductive conformal prediction framework for adapting medical vision-language models (VLMs) using a few-shot adaptation set. For each test point, FCA fits a linear probe on the adaptation set augmented with each candidate label, computes label-wise non-conformity scores, and forms a prediction set via the full conformal quantile. The authors also introduce SS-Text, a training-free solver that approximates a constrained linear probing objective by minimizing only the hard-label term. Experiments on 9 public datasets and 3 medical VLMs report that FCA maintains coverage near nominal while reducing set sizes by up to 27% relative to split conformal prediction, and that SS-Text is about 150× faster than gradient-based adapters.","tokens_in":14042,"tokens_out":7585,"duration_ms":72067,"significance":"If validated, FCA would be a practical way to combine few-shot VLM adaptation with distribution-free finite-sample coverage guarantees while using the same labeled data as SCP, and SS-Text addresses a real computational bottleneck of transductive conformal prediction. The paper provides code, evaluates across multiple modalities and datasets, and correctly inherits the FCP coverage argument. However, the current experimental protocol does not actually test the coverage guarantee under the exchangeability assumption on which it rests, and key tables lack uncertainty estimates; these issues need to be addressed before the central claim can be accepted.","major_comments":[{"comment":"The coverage guarantee in Eq. (5) requires the adaptation set D_adapt and each test point to be exchangeable, and the fitting procedure to be permutation-invariant. The experimental protocol in §5.1 uses a class-balanced few-shot support set (N = K·C, as stated in §4.1) while the medical test sets are naturally imbalanced; §5.2 concedes that 'the label-marginal distribution of the testing data is unknown, and typical few-shot adaptation pipelines assume a balanced support set.' This violates exchangeability even for random patient splits, so Table 1's coverage numbers (e.g., FCA LAC at α=0.10 reaches 0.896, not 0.900) are obtained in a regime where the guarantee is not supposed to hold. The paper therefore never empirically demonstrates the central claim that FCA maintains nominal coverage under exchangeable data. Please add an experiment with exchangeable test sampling (e.g., balanced test labels or sampling test points from the same class-balanced distribution), or apply a label-shift-robust conformal quantile, and report the resulting coverage.","section":"§4.2, Eq. (5); §5.1–5.2; Table 1"},{"comment":"Despite stating that all results are repeated over 20 random seeds, Table 1 reports only point estimates for coverage, set size, and CCV. The differences relative to nominal coverage are small (e.g., 0.896 vs 0.900 for FCA-LAC at α=0.10), and without standard deviations or confidence intervals it is impossible to tell whether the deficit is statistically significant or an artifact of finite sampling. Please report means with error bars (or confidence intervals) for all metrics in Tables 1 and 2 and for the per-dataset results in Fig. 2.","section":"Table 1"},{"comment":"The derivation of SS-Text minimizes only the g1 term of the loss in Eq. (6) and sets aside g2, the log-sum-exp term that couples classes through the softmax denominator. The text calls the resulting closed-form solution an approximation, but no bound on the approximation error is provided, and the experiments do not quantify how far the solution is from the true minimizer of Eq. (6). Since SS-Text is a central contribution and is used for all FCA experiments, please either justify the neglect of g2 (e.g., show that under λ = 1/(Nτ) the g2 gradient is small or the objective is dominated by g1) or explicitly present SS-Text as a heuristic and discuss the sensitivity of the downstream conformal results to this approximation.","section":"§4.3, Eqs. (6)–(10)"}],"minor_comments":[{"comment":"The phrase 'same coverage guarantees' is stronger than what the experiments show; consider saying 'target coverage' or adding the caveat that under exchangeability the guarantee holds, while in the evaluated label-shifted setting coverage is approximate.","section":"Abstract and §5.2"},{"comment":"The statement that FCA 'translates the exchangeability assumption to the feature space' is imprecise; features are deterministic transforms of images, so exchangeability is not translated but inherited, and the converse does not follow.","section":"Fig. 1 caption"},{"comment":"Please specify whether the resulting class vectors w*_c are ℓ2-normalized before being used in Eq. (1); the text normalizes v and t, but the sum in Eq. (10) is not automatically normalized.","section":"Eq. (10)"},{"comment":"References [22] and [23] are duplicate entries of the same Lei et al. paper; one should be removed.","section":"References"},{"comment":"The header 'SS-Text λc ≃ zero-shot perf.' is cryptic; it refers to CLAP's class-wise multiplier from [39], but the notation is unexplained in the text.","section":"Table 2"},{"comment":"The phrase 'tenths of tags' should be 'tens of tags'.","section":"Discussion"}],"recommendation":"major_revision","confidential_remarks":"The self-citation concentration in the few-shot VLM baselines (refs 37–40) is understandable given the authors' prior work in this specific area and does not affect my assessment. The main concern is the gap between the theoretical coverage claim and the experimental regime, which I believe is addressable with additional experiments and a more careful statement of the validity conditions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Julio's group has written a solid paper. The main idea is straightforward: instead of doing split conformal prediction on a model adapted with a few-shot set, they re-run the adaptation for each candidate label of each test point, transductively, which restores exchangeability with the adaptation set. That's a sensible and genuine application of full conformal prediction to few-shot VLM adaptation, and it works: coverage stays near the nominal level while set sizes shrink relative to SCP. The SS-Text solver is a closed-form linear probe that makes the transductive loop cheap; it's not deep, but it's honest and effective.\n\nWhat is actually new: applying transductive conformal to few-shot adaptation of medical VLMs, plus the efficiency analysis. The coverage theory is inherited, not new, and the authors know this.\n\nSoft spots. The biggest is that the empirical protocol never actually tests the guarantee in the regime where it holds. The support set is class-balanced while the test sets are naturally imbalanced, so exchangeability is violated from the start. The authors acknowledge this in Section 5.2 and the Discussion, which is good, but it means the coverage numbers in Table 1 are not evidence for the central claim. You would want to see at least one experiment with a balanced test partition, or a re-weighted support set, to show nominal coverage when the assumption is met. Second, 20 seeds with no error bars makes it hard to know how much of the coverage gap is noise. Third, SS-Text is derived by dropping the normalization term g2 of the loss; the paper's own tables show it matches or beats gradient-based probes, but there's no theoretical justification for why the approximation should hold. That's fine if presented as an empirical tool, and it is.\n\nThe citation pattern looks fine. The self-citations are to their own few-shot VLM adaptation work and FLAIR, which is appropriate. The code is available, which helps.\n\nWho is this for? People building conformalized reliability for medical VLMs, and researchers working on few-shot foundation model adaptation. They'll find it useful. It deserves a serious referee; the revisions should focus on the exchangeability mismatch in the experiments and on adding error bars. I'd take it.","headline":"A genuinely useful transductive conformal adaptation framework for few-shot medical VLMs, undercut mostly by experiments that never quite test the coverage guarantee in the regime where it is supposed to hold.","tokens_in":14588,"tokens_out":1980,"would_cite":true,"duration_ms":18493,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Full conformal adaptation gives medical vision-language models the same coverage guarantees as split conformal prediction while producing prediction sets up to 27% smaller, using exactly the same few-shot data.","keywords":["conformal prediction","vision-language models","few-shot adaptation","transductive learning","medical image analysis","uncertainty quantification","linear probing","coverage guarantee"],"falsifier":"Split a dataset by patient so no patient appears in both the support/adaptation set and the test set, then run FCA with K=16 and α=0.10; if observed coverage falls below roughly 0.90 by more than sampling error, the coverage claim is violated. The paper's own SICAPv2 and MMAC results already show such gaps, so the test would quantify how far the guarantee degrades under patient-level shift.","tokens_in":13666,"feed_emoji":"🩺","tokens_out":5949,"duration_ms":52029,"temperature":0.7,"pith_summary":"The paper's central claim is that a pre-trained medical vision-language model can be adapted to a new task and conformalized in a single transductive procedure—full conformal adaptation (FCA)—using exactly the same few-shot labeled support set that split conformal prediction would spend only on calibration. If this works as stated, reliability guarantees and few-shot transfer no longer compete for scarce labeled medical data: the model both learns the new task and produces prediction sets that cover the true label with the demanded frequency. The authors report that FCA matches the accuracy of standard adaptation, preserves marginal coverage at the nominal level, and produces sets up to 27% smaller than split conformal prediction across nine medical classification tasks spanning histology, retina, and chest X-ray. They also introduce SS-Text, a closed-form linear probe solver that sidesteps gradient training, making the per-test-point adaptation computationally feasible.","feed_headline":"Medical AI prediction sets shrink 27% with coverage intact","feed_subtitle":"A transductive few-shot scheme gives medical VLMs split-conformal coverage with much tighter prediction sets.","key_machinery":"The central object is full conformal adaptation (FCA), a transductive conformal procedure in which each test image, paired in turn with each candidate label, is added to a few-shot adaptation set while a linear probe is fit over frozen VLM features; the per-label quantile of non-conformity scores then defines the prediction set. Its validity is carried by the exchangeability of the adaptation set with each test point, together with permutation invariance of the fitting procedure, plus the standard full-conformal argument. The supporting machinery is SS-Text, a closed-form solution to a constrained linear probe objective whose minimizer is a weighted sum of the class text prototype and the mean of the per-class visual features, with the regularization weight set to $1/(N\\tau)$ so that the textual prior fades as support data grows.","core_discovery":"In classical full conformal prediction, the model is retrained for each test point and each candidate label, using that test point's hypothetical label alongside training data, so that scores for the true label are exchangeable with training scores. The paper transplants this logic into few-shot VLM adaptation: instead of retraining a large network, it fits a lightweight linear probe on top of frozen VLM features, once for each candidate label of each test image, with the adaptation set replacing the pre-training dataset as the exchangeable reference. The result is a prediction set per test point that, the paper argues, inherits FCP's finite-sample marginal coverage guarantee while using exactly the same data as SCP. Empirically, across three medical VLMs and nine tasks, FCA maintains coverage at the target 1-α while cutting average set size relative to SCP by up to 27%, and it improves per-class coverage gaps substantially. A training-free solver, SS-Text, solves the constrained linear-probe objective in closed form and is about 150× faster per fit than gradient-based adaptation, making the transductive loop practical.","pith_inferences":["The same transductive adaptation trick could apply to other foundation-model families with frozen features and a cheap probe, opening a general route to conformalized few-shot transfer beyond VLMs.","Because the validity argument lives in feature space, coverage might be maintained even when the raw image distribution shifts, as long as the adapted features of new patients remain interchangeable with the support features—an empirically testable condition.","The closed-form solver's $\\lambda = 1/(N\\tau)$ schedule could be replaced by a data-driven per-class weight, potentially improving low-shot accuracy and coverage on grading tasks where inter-observer variability is high.","For dense prediction, the per-label per-image fits remain a bottleneck; making the probe fitting incremental or amortized would be the natural next step to extend FCA beyond image-level classification."],"forward_implications":["FCA lets a single few-shot support set do double duty—adaptation and conformal calibration—so medical deployments no longer need separate calibration subsets to get coverage guarantees.","At the claimed 27% set-size reduction, clinicians using FCA would on average read fewer candidate labels per image while retaining the same 90% or 95% coverage promise.","The closed-form SS-Text solver removes the computational objection to transductive conformal prediction, since the per-label fits cost milliseconds rather than gradient-training runs.","Across 9 tasks and 3 modality-specialized VLMs, the framework gives better class-conditional coverage than Adapt+SCP, so underrepresented classes are less likely to be dropped from the sets.","The coverage guarantee holds at the marginal level and is not exact per patient or per class; patient-shifted datasets such as SICAPv2 and MMAC show coverage gaps even for the proposed method."],"supporting_citations":[{"why":"Introduces full conformal prediction, whose transductive validity FCA inherits.","marker":"[1]"},{"why":"Defines split conformal prediction, the SCP baseline FCA is compared against.","marker":"[30]"},{"why":"Provides the exchangeability conditions and coverage guarantees underlying both SCP and FCP.","marker":"[45]"},{"why":"Supplies the CLIP-style zero-shot VLM formulation and text-prototype construction used throughout.","marker":"[33]"},{"why":"Establishes the few-shot adaptation setting and gradient-based linear probing baselines.","marker":"[39]"},{"why":"Formalizes transduction with confidence, grounding the per-label model-fitting argument.","marker":"[36]"},{"why":"Extends transductive confidence machines to pattern recognition, justifying per-label score exchangeability.","marker":"[31]"},{"why":"Provides the LP++ linear probe and training-free baseline used for SS-Text comparison.","marker":"[14]"}],"fun_headline_variants":["Transductive conformal adaptation shrinks medical prediction sets by 27%","Full conformal adaptation: tighter sets for medical VLMs without extra data","Training-free conformal solver cuts set size 27% for medical VLMs","New conformal method keeps coverage but shrinks medical AI sets by 27%","Few-shot conformal adaptation trims medical set size while preserving coverage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee holds only if the few-shot adaptation set is exchangeable with each new test image and the probe fitting is permutation-invariant; when test patients differ systematically from the support set, the marginal coverage promise can fail.","fun_headline_variants_meta":{"raw":{"variants":["Transductive conformal adaptation shrinks medical prediction sets by 27%","Full conformal adaptation: tighter sets for medical VLMs without extra data","Training-free conformal solver cuts set size 27% for medical VLMs","New conformal method keeps coverage but shrinks medical AI sets by 27%","Few-shot conformal adaptation trims medical set size while preserving coverage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000679,"raw_usage":{"total_tokens":3101,"prompt_tokens":979,"completion_tokens":2122,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":2024}},"tokens_in":595,"tokens_out":2122,"duration_ms":14612,"temperature":1.0,"reasoning_tokens":2024,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T06:01:04.565839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split a dataset by patient so no patient appears in both the support/adaptation set and the test set, then run FCA with K=16 and α=0.10; if observed coverage falls below roughly 0.90 by more than sampling error, the coverage claim is violated. The paper's own SICAPv2 and MMAC results already show such gaps, so the test would quantify how far the guarantee degrades under patient-level shift.","supporting_citations":[{"cited_title":"In: Conference on Uncertainty in Artificial Intelligence","cited_arxiv_id":null,"evidence_quote":"Introduces full conformal prediction, whose transductive validity FCA inherits."},{"cited_title":"In: European Conference on Machine Learning (ECML)","cited_arxiv_id":null,"evidence_quote":"Defines split conformal prediction, the SCP baseline FCA is compared against."},{"cited_title":"Springer (01 2005) 2, 4, 5","cited_arxiv_id":null,"evidence_quote":"Provides the exchangeability conditions and coverage guarantees underlying both SCP and FCP."},{"cited_title":"In: International Conference on Machine Learning (ICML)","cited_arxiv_id":null,"evidence_quote":"Supplies the CLIP-style zero-shot VLM formulation and text-prototype construction used throughout."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"Establishes the few-shot adaptation setting and gradient-based linear probing baselines."},{"cited_title":"In: International Joint Conference on Artificial Intelligence (IJCAI)","cited_arxiv_id":null,"evidence_quote":"Formalizes transduction with confidence, grounding the per-label model-fitting argument."},{"cited_title":"In: European Conference on Machine Learning (ECML)","cited_arxiv_id":null,"evidence_quote":"Extends transductive confidence machines to pattern recognition, justifying per-label score exchangeability."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)","cited_arxiv_id":null,"evidence_quote":"Provides the LP++ linear probe and training-free baseline used for SS-Text comparison."}],"review_version":1}