{"id":"65aecedc-17ec-49d2-b48c-2973a38e3c51","arxiv_id":"2505.08798","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Few-shot in-context prompting improves frozen vision-language model F1 scores on three cancer image datasets, with GPT-4o reaching 0.81 binary and 0.60 multi-class.","lead":"This paper tests whether four vision-language models can classify cancer images using only a few labeled examples in the prompt, with no retraining. On three public cancer datasets, few-shot prompting improved scores, with GPT-4o reaching F1 of 0.81 on binary and 0.60 on seven-class tasks, below fine-tuned systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Significance claim is unsupported: CIs bootstrap only the test set, and the paper itself notes small support-set variations produce inconsistent outputs; one random support draw per setting cannot establish 'significant gains.'","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the evaluation ignores variance introduced by random support set selection and stochastic decoding, so the claimed 'significant gains' are not statistically supported. This is not a disagreement with the consensus direction of the results—the point estimates are consistent across models and datasets—but it is a correctness risk because the word 'significant' is used without any test that accounts for the sampling process that most affects few-shot ICL. The paper itself contains an explicit limitation statement in the Summary admitting that small variations in support examples lead to inconsistent outputs, which strengthens the concern and aligns with the reviewing rule to flag such passages. The proposed concrete test—resampling support sets and testing the gain distribution—would settle whether the reported improvements are robust or artifacts of one fortunate draw. Because the current verdict is already CONDITIONAL and this concern is the same one that justifies that conditionality, the verdict should remain unchanged.","tokens_in":10080,"tokens_out":3410,"duration_ms":37077,"concrete_test":"Repeat the full evaluation for at least 10 independent random support sets per (dataset, model, k) pair (e.g., k=5 and 10), holding the test set fixed, and report the mean and 95% CI of weighted F1 across support sets; also run a paired bootstrap or Wilcoxon signed-rank test comparing 10-shot vs zero-shot within each pair. If the across-support-set CI for the gain includes zero for any model or dataset, or if the distribution of gains overlaps zero, the 'significant gains' claim fails. For GPT-4o, additionally fix temperature and run the same 10 support sets to separate decoding variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 'all models showed significant gains with few-shot prompting' rests on point estimates from a single randomly drawn support set per shot count, with 95% CIs computed by bootstrapping only the 158/161-image test set (Methods, Performance Evaluation and Statistical Analysis). The bootstrap does not resample support examples or decoding stochasticity, so it cannot support the word 'significant.' The paper's own Summary concedes that 'even small variations can lead to inconsistent outputs which were observed in our experiments.' For ICL with k=3-10, support-set variance is known to be substantial; with one draw per condition, the reported gains (e.g., MHIST GPT-4o 0.60 to 0.81, HAM10000 GPT-4o 0.34 to 0.60) could shift or disappear under a different support set. Because every dataset/model/k combination is a single realization, the central claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript reports an empirical comparison of four vision-language models (Paligemma, CLIP, ALIGN, and GPT-4o) under zero-shot and few-shot conditions on three public oncology imaging datasets: MHIST, PatchCamelyon, and HAM10000. The authors report weighted F1 scores with bootstrap 95% CIs and claim that, without any parameter updates, all models show significant gains from few-shot prompting, with GPT-4o achieving F1 0.81 on binary tasks and 0.60 on the seven-class task. The paper positions in-context learning as a label-efficient alternative to fine-tuning for resource-constrained oncology settings.","tokens_in":10246,"tokens_out":5357,"duration_ms":52940,"significance":"If the reported gains are robust, the study would provide a useful practical benchmark for frozen generalist VLMs in oncology classification and would support the feasibility of in-context learning in low-data clinical settings. The manuscript has concrete strengths: it uses standard public datasets, evaluates multiple models under a common protocol, includes class-wise breakdowns, and explicitly checks option-order bias by randomizing label positions. It also acknowledges the performance ceiling relative to fine-tuned systems and the sensitivity of outputs to small prompt variations. However, the central 'significant gains' claim is not backed by appropriate statistical inference, and the CLIP/ALIGN setups are not true in-context learning; both issues need to be fixed before the conclusions can be relied upon.","major_comments":[{"comment":"The claim that 'all models showed significant gains with few-shot prompting' is not supported by the reported statistics. The 95% CIs in Table 2 are bootstrap resamples of the fixed 158/161-image test set only, as stated in Methods ('the test dataset was resampled with replacement'), so they do not capture variance from the random selection of 3, 5, or 10 support examples or from stochastic decoding. With only one randomly drawn support set per condition, the point estimates are single realizations. In fact, several zero-shot and 3-shot CIs overlap substantially, e.g., MHIST ALIGN 0.59 (0.52–0.65) vs. 0.62 (0.57–0.69) and PatchCamelyon CLIP 0.52 (0.46–0.55) vs. 0.59 (0.48–0.65). This concern is reinforced by the Summary's own admission that 'even small variations can lead to inconsistent outputs which were observed in our experiments.' The authors should either provide a formal significance test (e.g., paired bootstrap or permutation test across repeated support-set draws) or soften the 'significant' language and restrict the claim to the observed point estimates.","section":"Methods, Performance Evaluation and Statistical Analysis; Abstract; Results"},{"comment":"The CLIP and ALIGN evaluations are retrieval-based approximations rather than in-context learning in the generative sense. The Methods state that for 'image-text matching models (i.e. CLIP and ALIGN), we adapted the ICL structure to a retrieval-based formulation, aggregating similarity scores between the query image and class-conditioned prompts informed by the support set.' This is essentially a prototype/nearest-neighbor classifier over image-text similarities, not the same mechanism as autoregressive in-context learning in GPT-4o or Paligemma. Because the title and abstract generalize across all four models, the paper should explicitly separate the generative ICL results from the retrieval-based few-shot baselines, or rename the latter accordingly.","section":"Methods, In-Context Learning for Sample-Efficient Adaptation; Abstract"},{"comment":"The paper states that temperature and top_p values in the range 0.5–0.7 were 'explored' during experimentation but the final values used for each model are never reported. If these hyperparameters were chosen after inspecting test-set performance, the reported results would be optimistically biased even though no model parameters were updated. The authors should report the exact decoding settings per model, and ideally show that the main conclusions are stable across a small grid of these settings.","section":"Summary; Methods, Model Implementation and Inference Setup"}],"minor_comments":[{"comment":"Several confidence intervals in Table 2 contain typographical errors, e.g., PatchCamelyon CLIP 3-shot '(0.48-65)' and HAM10000 ALIGN 3-shot '(28-0.42)' are missing leading zeros; these should be corrected to '(0.48–0.65)' and '(0.28–0.42)'.","section":"Table 2"},{"comment":"The sample model response panel in Figure 1 appears to be empty or garbled in the manuscript text, making it impossible to verify the claimed prompting format; the figure should be regenerated with a visible example response.","section":"Figure 1"},{"comment":"The model referred to as 'ALIGN' is explicitly an open-source approximation (kakaobrain/align-base), not the original Google ALIGN model; this is acknowledged in Methods but should also be stated in the Abstract or at least in the first Results paragraph so readers do not infer results for the commercial model.","section":"Model Implementation and Inference Setup"},{"comment":"There are minor formatting inconsistencies in the reference list (e.g., 'ICCAI' for MICCAI, incomplete conference names) and several citations are given as URLs without access dates or version identifiers; the reference list should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward empirical comparison and I did not find evidence of fabrication or circular evaluation. However, the abstract's 'significant gains' language goes beyond what the experimental design can support, and the CLIP/ALIGN framing as in-context learning will likely draw criticism from reviewers who are strict about mechanism definitions. These are fixable with additional experiments and rewriting, so I see no need for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a straightforward extension of Ferber et al. to a four-model, three-dataset comparison, not a new mechanism. What it does well: clean public benchmarks, frozen models, option randomization, balanced evaluation, internally consistent tables, and an honest acknowledgement that fine-tuned models still set the ceiling. The trend is coherent across all settings: more shots help, GPT-4o leads, open models are competitive. That is useful confirmatory evidence for ICL in pathology and dermatology.\n\nThe soft spot is load-bearing. The claim of 'significant gains' rests on one random support draw per condition. The reported 95% CIs bootstrap only the 158- or 161-image test set, so they capture test-sample uncertainty, not support-set or decoding stochasticity. The authors themselves note in the Summary that even small variations in support examples can lead to inconsistent outputs. With k=3-10, support-set variance can be substantial; a different draw could shift or erase several reported gains, especially on HAM10000. As stated, the central claim is not established. This is fixable: repeat support draws, report the distribution or a bootstrap that resamples support sets, and drop 'significant' unless actually tested.\n\nOther soft spots, in proportion: CLIP and ALIGN are retrieval-based approximations, so 'in-context learning' is doing different work for those models; the abstract's 'first multi-VLM' claim is undercut by their own reference 25; no code, exact prompts, or fine-tuned baselines are provided; temperature and top_p were explored but not reported, which is a minor reproducibility gap. The 'bypass alignment filters' sentence is unfortunate phrasing but does not change the results. None of these are fatal. The comparison itself is honest and the numbers appear internally consistent.\n\nWho gets value: researchers wanting a quick map of zero-shot versus few-shot VLM performance across three standard oncology tasks. It deserves a serious referee because it is a legitimate empirical benchmark with addressable flaws, not a desk reject. My recommendation: send it to review with a request for repeated support-set sampling and proper uncertainty quantification.","headline":"A believable but statistically under-supported few-shot benchmark; worth serious peer review only if the authors fix the support-set variance problem.","tokens_in":10765,"tokens_out":1505,"would_cite":false,"duration_ms":17523,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen AI reaches 0.81 F1 on cancer images via few-shot prompts","keywords":["in-context learning","vision-language models","cancer image classification","few-shot prompting","histopathology","dermoscopy","label-efficient learning","GPT-4o"],"falsifier":"Repeatedly rerun the 3-, 5-, and 10-shot evaluations with many different randomly drawn support sets and several decoding temperatures, then compare the spread of F1 scores. If the gains shrink to zero or vary by more than the reported confidence intervals across support draws, the central claim of consistent few-shot improvement is not robust.","tokens_in":1468,"feed_emoji":"🔬","tokens_out":3218,"duration_ms":70956,"temperature":0.7,"pith_summary":"This paper claims that vision-language models can be adapted to cancer image classification at inference time, with no parameter updates, by embedding a few labeled examples in the prompt. Across three oncology datasets, all four models improved with 3, 5, and 10-shot prompting over their zero-shot baselines. The headline result is GPT-4o reaching a weighted F1 of 0.81 on binary tasks and 0.60 on seven-class skin lesion classification. The authors argue this makes generalist foundation models a practical route for label-scarce settings, such as rare cancers or resource-constrained clinics, while noting results stay below fine-tuned systems.","feed_headline":"Frozen AI reaches 0.81 F1 on cancer images via few-shot prompts","feed_subtitle":"No retraining: 10 examples per class lift four vision-language models on three cancer datasets.","key_machinery":"The central mechanism is in-context learning by few-shot prompting: a prompt is built from n labeled image-text support examples followed by an unlabeled query image, and the model produces a diagnosis from randomized multiple-choice options. Generative models like GPT-4o and Paligemma consume the support examples as direct image-text sequences; contrastive models like CLIP and ALIGN are adapted to a retrieval formulation that aggregates similarity scores between the query and class-conditioned prompts informed by the support set. Prompt engineering—stepwise chain-of-thought instructions and option order randomization—carries part of the effect, but the claimed gain is the few-shot conditioning itself.","core_discovery":"On its own terms, the paper establishes that in-context learning can serve as a label-efficient alternative to fine-tuning for oncology imaging. Four frozen vision-language models—GPT-4o, Paligemma, CLIP, and ALIGN—were prompted with multiple-choice diagnostics on colorectal polyp, breast metastasis, and skin lesion datasets. Every model improved consistently as support examples increased from 0 to 3, 5, and 10 per class, with GPT-4o reaching F1 0.81 on MHIST, 0.79 on PatchCamelyon, and 0.60 on HAM10000 at 10-shot. The authors interpret the mechanism as localized task conditioning or amortized inference: the support examples re-weight the model's attention toward class-relevant visual features without gradient updates, mirroring how clinicians reason from prior cases.","pith_inferences":["Editorial inference: The reported confidence intervals resample the test set only; drawing new support examples or changing decoding temperature would likely widen them, so multi-seed evaluation should be the next test.","Editorial inference: Part of the few-shot gain may come from the model exploiting label frequencies or option-position priors rather than learning new visual features; permuting support labels or comparing with scrambled examples would separate these.","Editorial inference: If the mechanism is general, similar few-shot gains should appear on other histopathology and dermoscopy benchmarks; that is a direct and cheap extension of this protocol.","Editorial inference: The retrieval-based adaptation used for CLIP and ALIGN suggests that even contrastive models without generative output heads can be pulled into the in-context learning paradigm, which broadens the set of deployable models."],"forward_implications":["Label-hungry oncology AI can be deployed without retraining: a handful of labeled images per class is enough to move generalist models from near-chance to substantially better diagnostic performance.","Open-source models such as Paligemma and CLIP improve nearly in parallel with GPT-4o, so compute-constrained settings can benefit without relying on proprietary APIs.","The gap between binary (F1 around 0.8) and seven-class (F1 around 0.6) results shows in-context learning is strongest when the label space is simple, so multi-class deployment would still need additional support.","Because the method changes only the prompt, the same frozen model can be repurposed across different diagnostic tasks in rapid succession, which matters where turnaround time is short.","The consistent improvement across three tissue types and four model families suggests the effect is not specific to one dataset or one model."],"supporting_citations":[{"why":"Provides the precedent that in-context learning lets multimodal LLMs classify cancer pathology images, which this study extends across more models and datasets.","marker":"[25]"},{"why":"Defines few-shot in-context learning in language models, the paradigm the paper transfers to vision-language oncology tasks.","marker":"[10]"},{"why":"Supplies the MHIST colorectal polyp dataset used for the binary MHIST task.","marker":"[26]"},{"why":"Supplies the PatchCamelyon breast metastasis benchmark derived from Camelyon16.","marker":"[27-28]"},{"why":"Supplies the HAM10000 dermatoscopic image dataset for seven-class skin lesion classification.","marker":"[29]"},{"why":"Provides the pretrained Paligemma checkpoint used as one of the four evaluated VLMs.","marker":"[30]"},{"why":"Provides the CLIP ViT-B/32 checkpoint used for the CLIP evaluations.","marker":"[31]"},{"why":"Provides the ALIGN architecture implementation used to approximate ALIGN's behavior.","marker":"[33]"},{"why":"Documents prior use of the same ALIGN implementation, referenced as the basis for running ALIGN in this setup.","marker":"[36]"}],"fun_headline_variants":["10 examples per class: frozen VLMs hit F1 0.81 on cancer images","Few-shot prompts beat fine-tuning? Not quite, but no retraining needed","Cancer diagnostics: in-context learning from a handful of cases","Label-efficient oncology: ICL lifts frozen models without gradient updates","Zero training, few shots: GPT-4o scores 0.81 F1 on cancer datasets"],"cache_read_input_tokens":13056,"weakest_assumption_plain":"The results rest on the assumption that one randomly chosen set of a few support examples represents how these models behave, since the confidence intervals only resample the test images and ignore the variance from which support examples were picked and from stochastic text generation.","fun_headline_variants_meta":{"raw":{"variants":["10 examples per class: frozen VLMs hit F1 0.81 on cancer images","Few-shot prompts beat fine-tuning? Not quite, but no retraining needed","Cancer diagnostics: in-context learning from a handful of cases","Label-efficient oncology: ICL lifts frozen models without gradient updates","Zero training, few shots: GPT-4o scores 0.81 F1 on cancer datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000614,"raw_usage":{"total_tokens":2876,"prompt_tokens":993,"completion_tokens":1883,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1780}},"tokens_in":609,"tokens_out":1883,"duration_ms":16361,"temperature":1.0,"reasoning_tokens":1780,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:59:55.712591+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeatedly rerun the 3-, 5-, and 10-shot evaluations with many different randomly drawn support sets and several decoding temperatures, then compare the spread of F1 scores. If the gains shrink to zero or vary by more than the reported confidence intervals across support draws, the central claim of consistent few-shot improvement is not robust.","supporting_citations":[{"cited_title":"C., Ligero, M., Sainath, S., Ghaffari Laleh, N.,","cited_arxiv_id":null,"evidence_quote":"Provides the precedent that in-context learning lets multimodal LLMs classify cancer pathology images, which this study extends across more models and datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines few-shot in-context learning in language models, the paradigm the paper transfers to vision-language oncology tasks."},{"cited_title":"& Hassanpour, S","cited_arxiv_id":null,"evidence_quote":"Supplies the MHIST colorectal polyp dataset used for the binary MHIST task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the HAM10000 dermatoscopic image dataset for seven-class skin lesion classification."},{"cited_title":"Last accessed on 4/15/2025","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained Paligemma checkpoint used as one of the four evaluated VLMs."},{"cited_title":"Last accessed on 4/11/2025","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP ViT-B/32 checkpoint used for the CLIP evaluations."},{"cited_title":"Last accessed on 4/08/2025","cited_arxiv_id":null,"evidence_quote":"Provides the ALIGN architecture implementation used to approximate ALIGN's behavior."},{"cited_title":"(2024, May)","cited_arxiv_id":null,"evidence_quote":"Documents prior use of the same ALIGN implementation, referenced as the basis for running ALIGN in this setup."}],"review_version":1}