{"id":"9c95fb1e-907f-439a-b261-174590693bea","arxiv_id":"2506.11166","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Sampling multiple VLM-generated visual descriptions and letting a text-only LLM vote on the diagnosis improves zero-shot medical image classification on three MedMNIST datasets.","lead":"The paper proposes a zero-shot framework for medical image diagnosis: a vision-language model first writes several neutral descriptions of the image, then a text-only LLM chooses the diagnosis from those descriptions, and repeated sampling with majority voting improves accuracy. The authors test it on chest X-ray, colon pathology, and retina images and report higher AUC than direct question-answering and chain-of-thought prompting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains are single-run point estimates: Table 1 has no variance, significance test, or seed/temperature sensitivity for N=16 stochastic sampling, so the central AUC improvement may be sampling noise.","rationale":"After reading the paper in good faith, the strongest claim is not the mathematical framework but the empirical conclusion that the proposed two-stage TTS recipe beats zero-shot QA and CoT on PneumoniaMNIST, PathMNIST, and RetinaMNIST. The equations are not the weak point: Eq. (1) is a legitimate Monte Carlo approximation if the Stage-1 samples are reasonably diverse, and the authors candidly note that the method performs poorly at N=1. The load-bearing condition is therefore that the observed AUC/AP gaps reflect a real improvement rather than the noise of a stochastic decoder run once. Table 1 has no error bars, Figure 2 has no error bars, and Figure 3 has no error bars; every conclusion is drawn from point estimates. This is why the reader's CONDITIONAL verdict is appropriate. I considered other candidate concerns, including the claim that unconstrained prompting reduces overdiagnosis while only rank-based AUC is reported, and the absence of prompt templates and parsing details that would hinder reproduction, but neither is as load-bearing as the missing statistical support: even a perfect reproduction of the exact pipeline could produce a different table if the sampling seed changes. The proposed repeated-seed check directly settles whether the reported improvements are stable, so I keep the CONDITIONAL verdict and agree with the reader's weakest-assumption analysis.","tokens_in":7901,"tokens_out":6293,"duration_ms":81547,"concrete_test":"Run R=20 independent repetitions of the full pipeline on PneumoniaMNIST (and PathMNIST) with different random seeds at fixed temperature, holding all prompts and parsing fixed, and record per-image Monte Carlo scores for (a) direct zero-shot QA-TTS and (b) describe-then-diagnose-TTS. Compute the paired per-seed AUC difference between (b) and (a) and its 95% bootstrap CI. If the interval includes 0, or if the reported 0.821 and 0.653 values fall outside the repeated-run ranges, the central improvement claim is not supported; if the interval excludes 0 across seeds, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that describe-then-diagnose with test-time scaling (TTS) improves zero-shot diagnostic AUC over direct zero-shot QA and one-stage CoT (Table 1: PneumoniaMNIST 0.821 vs 0.737 TTS baseline; PathMNIST 0.653 vs 0.562). This claim rests on Eq. (1), a Monte Carlo estimate obtained by drawing N=16 temperature-sampled visual descriptions and aggregating Stage-2 LLM labels. The paper reports exactly one run of this stochastic procedure per cell, with no standard errors, confidence intervals, paired significance tests, or sensitivity analysis over sampling temperature and seed. Because the proposed advantage is produced by the very same random sampling that is unquantified, the headline gaps could be within run-to-run variation; the 'consistent enhancement' and 'power law' claims in Section 3 extrapolate from this unquantified evidence. The omission of prompts, parsing failure rates, and code compounds the issue but is secondary. This is an evidential gap rather than an internal inconsistency: the derivation of Eq. (1) is straightforward, but Eq. (1) defines a stochastic estimator whose variance is never reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a zero-shot, two-stage framework for medical image diagnosis. In Stage 1, a vision-language model (Llama-3.2-11B-Vision-Instruct) generates N temperature-sampled textual descriptions of the input image under a neutral prompt. In Stage 2, a text-only LLM converts each description into a diagnosis, and the final prediction is obtained by majority voting over the N outputs. The method is evaluated on PneumoniaMNIST, PathMNIST, and RetinaMNIST, with the central empirical claim that test-time scaling (N=16) improves AUC over single-sample zero-shot QA and one-stage CoT, e.g., PneumoniaMNIST AUC 0.821 versus 0.737, and PathMNIST 0.653 versus 0.562. The paper also reports that unconstrained Stage-1 prompting mitigates apparent bias compared with disease-dictated prompting, and that smaller Stage-2 LLMs (3B) perform comparably to larger ones.","tokens_in":8101,"tokens_out":1761,"duration_ms":23195,"significance":"If the reported gains are real and reproducible, the proposed describe-then-diagnose pipeline with test-time scaling would be a useful, annotation-free recipe for improving zero-shot medical VQA, with the practical merit of decoupling visual description from diagnostic reasoning and allowing cheaper Stage-2 models. The method is simple, the Monte Carlo estimator in Eq. (1) is conceptually sound and not circular, and the evaluation uses external MedMNIST benchmarks rather than fitted constants. The main significance, however, is presently conditional: the headline improvement is supported only by single-run point estimates without variance or significance analysis of a stochastic procedure, and the paper itself omits prompts and code. The bias-mitigation and compute-savings analyses are suggestive but similarly lack statistical grounding.","major_comments":[{"comment":"The central claim that TTS with N=16 improves diagnostic AUC rests on point estimates from a stochastic sampling procedure, yet the paper reports no standard errors, confidence intervals, paired tests, or repetition over seeds. On datasets with 226 to 1,233 test cases, the headline gaps (e.g., PneumoniaMNIST 0.821 vs. 0.737) could plausibly be within run-to-run variation of temperature-sampled descriptions. Please report multiple seeds, bootstrap intervals, or paired significance tests for the main comparisons, and state the sampling temperature and parsing failure rate for Eq. (1).","section":"Table 1 and Eq. (1)"},{"comment":"The claim that increasing N improves performance 'following a power law' is asserted from a small number of empirical points (N=1, 2, 4, maybe 8, 16) with no fitted curve, no error bars, and no quantitative exponent or goodness-of-fit. Please either provide a fitted power-law model with confidence bounds or soften the claim to 'monotone improvement in the tested range,' and ideally show repeated runs at each N to establish that the trend is not sampling noise.","section":"Figure 2"},{"comment":"The bias-mitigation conclusion compares 'unconstrained' versus 'dictated' Stage-1 prompts, but the comparison is shown only as point AUC values with no variance or significance testing, and the exact prompts are not given. Since this analysis is one of the paper's stated contributions and is used to argue that neutral prompting reduces hallucination and overdiagnosis, please report effect sizes with uncertainty, and disclose the full prompt templates so the comparison is reproducible.","section":"Section 3, 'Generic Prompt Enhances Reliability' and Figure 3a"},{"comment":"The manuscript states that 'detailed prompts and codes used in the baseline and proposed frameworks are omitted here but will be shared at publication time.' For a zero-shot prompting paper, prompts are part of the method, not merely implementation detail; without them, the method cannot be reproduced or compared fairly across baselines. Please include the complete prompts for zero-shot QA, one-stage CoT, Stage 1 description, Stage 2 diagnosis, and the parsing instruction in the main text or an appendix.","section":"Section 3, 'Datasets and Models'"}],"minor_comments":[{"comment":"The notation p(ŷ=1|x) is written as an integral over v, but the empirical approximation is a simple Monte Carlo average; please clarify that the indicator function I[a(i)=\"\\boxed{1}\"] is the Stage-2 LLM's parsed answer, and note that the approximation ignores any dependence of the Stage-2 LLM on the sampling temperature.","section":"Eq. (1)"},{"comment":"The temperature scaling reference [4] is a calibration paper; please cite the original temperature sampling heuristic (e.g., Ackley et al. or the language-model sampling literature) instead of, or in addition to, a calibration reference.","section":"Section 2.2, 'Enhancing Reliability using Test-time Scaling'"},{"comment":"The phrase 'the best and the second best are highlighted' is not visible in the text table in this version; please ensure the formatting is rendered or describe the highlighting in the caption.","section":"Section 3, 'Comparison with Baselines'"},{"comment":"The sentence 'the theoretical foundation and empirical evidence for Experts Don't Cheat concept' is grammatically incomplete; please revise to a full sentence and clarify how Refs. [8,23] support the claim about VLM hallucination.","section":"Section 3, 'Generic Prompt Enhances Reliability'"},{"comment":"The comparison with Med42-v2-8B in Figure 3b is described as 'similar' to the 3B Llama model, but no numerical values are reported in the text; please add the exact AUC/AP values or a table so the claim is verifiable.","section":"Section 3, 'Ablation on LLM size in the second stage'"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a computer vision or medical imaging venue, but the empirical core currently lacks the statistical support that a stochastic test-time-scaling claim requires. The omission of prompts and code is a reproducibility barrier, and the power-law and bias-mitigation claims rest on unquantified single-run comparisons. These are fixable with additional experiments and fuller disclosure, so I recommend major revision rather than rejection. I would also ask the editor to encourage the authors to report parsing failure rates, since the second-stage instruction to output only '\\boxed{}' may produce unparseable outputs that are silently dropped."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2506.11166. The paper does something simple and reasonable: split diagnosis into \"describe the image\" (VLM) then \"classify from the description\" (LLM), and at test time sample N=16 descriptions and majority-vote. The combination on MedMNIST is new, and the gains in Table 1 are non-trivially large (PneumoniaMNIST AUC 0.821 vs 0.737 for zero-shot TTS; PathMNIST 0.653 vs 0.562). The paper also shows that neutral Stage-1 prompts beat disease-directed prompts (Figure 3a) and that a 3B Stage-2 LLM is enough (Figure 3b). Those are useful empirical data points.\n\nWhat's not to like: the central claim rests on exactly one stochastic run per cell. No error bars, no confidence intervals, no significance test, no seed or temperature sensitivity for the N=16 sampling. The gap between 0.821 and 0.737 could be within run-to-run variation; we just don't know. The \"power law\" in Figure 2 is asserted from a handful of points without a fit or error bars. The paper also omits prompts and parsing failure rates, which matters for reproducibility in LLM pipelines. These are evidential gaps, not internal contradictions. Eq. (1) is just a Monte Carlo identity for P(y|x); no circularity there.\n\nThe reader's CONDITIONAL verdict gets it right. The method is plausible and the direction is worth publishing, but as submitted the empirical support is thinner than the prose claims. The authors should add repeated runs with different seeds and temperatures, report mean ± std over seeds, include prompt templates, and dial back the power-law language unless they fit it properly.\n\nWho's this for? Anyone working on zero-shot medical VQA or test-time scaling for LLMs. I'd send it to review; the recipe is clean enough and the effect, if real, is useful. I wouldn't cite it in my own work until the variance question is answered.\n\nRecommendation: send to peer review, but expect heavy revision on the experimental reporting.","headline":"A clean zero-shot medical VQA recipe with a plausible test-time-sampling gain, but the headline numbers are single-run estimates with no variance, so treat the effect sizes as provisional.","tokens_in":8653,"tokens_out":1377,"would_cite":false,"duration_ms":15557,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Sampling 16 image descriptions at test time improves zero-shot medical diagnosis.","keywords":["zero-shot medical diagnosis","test-time scaling","visual-language reasoning","describe-then-diagnose","medical image classification","hallucination mitigation","pneumonia detection","histopathology"],"falsifier":"Run the same protocol on the chest X-ray benchmark with $N=16$ across a large number of independent seeds (random seeds, sampling temperatures, and model checkpoints) and compute a confidence interval for the AUC gap; if the interval for the reported 0.737-to-0.821 improvement includes zero, the central claim is not distinguishable from sampling noise.","tokens_in":7679,"feed_emoji":"🩻","tokens_out":14942,"duration_ms":144809,"temperature":0.7,"pith_summary":"This paper asks whether a vision-language model can become a reliable zero-shot medical diagnostician without any fine-tuning. The proposed answer is to split the job in two: first sample 16 neutral descriptions of the image from a vision-language model, then pass each description to a text-only LLM that outputs a class label, and average the labels. The paper interprets this average as a Monte Carlo estimate of the probability that the image shows the disease. Reported results show that this test-time scaling recipe beats direct zero-shot question answering and one-stage chain-of-thought on chest X-ray, colorectal histopathology, and retinal fundus benchmarks, with the chest X-ray AUC rising from 0.737 to 0.821. The same sampling strategy also improves the baseline methods and allows the second-stage LLM to be as small as 1B–3B parameters without losing much accuracy.","feed_headline":"16 sampled descriptions beat direct zero-shot diagnosis","feed_subtitle":"A two-stage describe-then-diagnose pipeline lifts accuracy on X-ray, pathology, and retinal images.","key_machinery":"The load-bearing mechanism is the two-stage decomposition plus a Monte Carlo estimate of the posterior. Stage 1 samples $N$ visual descriptions $v^{(1)},\\dots,v^{(N)}$ from a vision-language model under temperature scaling, using a neutral prompt that does not name a disease. Stage 2 feeds each description to a text-only LLM that must return only a boxed class label, and the final prediction is the majority label, i.e. Equation (1), which the paper reads as a Monte Carlo approximation of $\\int p(\\hat y=1\\mid v)\\,p(v\\mid x)\\,dv$. The key move is separating visual feature extraction from clinical reasoning, so the unconstrained Stage-1 prompt is not contaminated by diagnosis hints, and the sampling turns one potentially hallucinated description into an aggregate over a distribution. This same TTS recipe also boosts the baseline methods, which is why the paper treats sampling, not just the two-stage prompt, as the active ingredient.","core_discovery":"The central claim is that a single greedy forward pass underuses a zero-shot vision-language model, and that test-time scaling closes the gap. Specifically, the paper proposes a two-stage 'describe-then-diagnose' pipeline in which Stage 1 draws $N=16$ temperature-sampled descriptions from the VLM using a neutral, non-diagnostic prompt, and Stage 2 classifies each description with a text-only LLM forced to emit a boxed label. The final diagnosis is the average of those labels, written as Equation (1): $\\hat{p}(\\hat{y}=1\\mid x) \\approx \\frac{1}{N}\\sum_{i=1}^N \\mathbb{I}(a^{(i)}=1)$, where $a^{(i)}$ is the LLM's parsed answer for description $i$. The paper reports that this aggregation improves AUC and average precision over single-sample decoding for the proposed method and for both baselines across all three modalities, that the gains follow a power law in $N$, that a 3B-parameter second-stage model matches the 11B model, and that unconstrained Stage-1 prompts reduce hallucinated findings compared with disease-dictated prompts.","pith_inferences":["If the gain comes mainly from description diversity, the same two-stage recipe should transfer to other vision-language and text-only model pairings, including much smaller models, as long as Stage-1 descriptions remain clinically faithful.","A controlled experiment fixing the compute budget—comparing one-stage CoT with the same $N=16$ samples and the same aggregator against the two-stage pipeline—would isolate whether the separation of description and diagnosis, rather than sampling alone, drives the improvement.","The power-law trend suggests a testable scaling law for zero-shot medical diagnosis, but it may saturate once the sampled description distribution stops adding new clinical features; measuring where that plateau occurs would set expectations for larger $N$.","The bias-mitigation result could be turned into a training objective for small medical VLMs: supervise description-only outputs first and diagnoses second, rather than asking a single model to do both at once."],"forward_implications":["With $N=16$ samples, test-time scaling improves AUC and average precision over single-sample decoding for zero-shot QA, one-stage CoT, and the proposed method on all three modalities.","Even $N=4$ samples yields large gains over a single sample, and performance follows a power law as $N$ grows, implying more samples continue to help.","A second-stage LLM with around 3B parameters performs comparably to the 11B model, so the framework reduces inference cost without sacrificing accuracy.","Neutral, unconstrained prompts in Stage 1 reduce hallucinated and overdiagnosed descriptions relative to disease-dictated prompts, making prompt design part of reliability.","Because TTS also improves the baseline methods, the benefit is an inference-time intervention that generalizes beyond the proposed two-stage prompt."],"supporting_citations":[{"why":"Introduces test-time scaling as sampling and aggregating multiple candidate outputs, the central inference strategy.","marker":"[14,25]"},{"why":"Supplies the three benchmark datasets used in all evaluations.","marker":"[24]"},{"why":"Defines the zero-shot chain-of-thought baseline that the method is compared against.","marker":"[10]"},{"why":"Provides evidence that transformers benefit from task decomposition, motivating the two-stage architecture.","marker":"[1]"},{"why":"Temperature scaling used to draw multiple visual descriptions in Stage 1.","marker":"[4]"},{"why":"The model family used in both stages and in the size-ablation experiments.","marker":"[18]"},{"why":"A medical-domain fine-tuned model used as the comparison point in the Stage-2 size ablation.","marker":"[2]"},{"why":"Hallucination and overconfidence findings used to explain why neutral prompts reduce biased descriptions.","marker":"[8,23]"}],"fun_headline_variants":["Sampled descriptions beat single-shot diagnosis","Test-time scaling improves zero-shot medical diagnosis","16 descriptions outperform one-shot diagnosis","Power-law gains from multi-sample diagnosis","Describe-then-diagnose lifts zero-shot accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only if averaging sixteen temperature-sampled image descriptions produces a stable, unbiased estimate of the true disease probability; if those sixteen draws are too noisy or share a common bias, the majority diagnosis is not a reliable estimate, and the reported gains could be sampling noise.","fun_headline_variants_meta":{"raw":{"variants":["Sampled descriptions beat single-shot diagnosis","Test-time scaling improves zero-shot medical diagnosis","16 descriptions outperform one-shot diagnosis","Power-law gains from multi-sample diagnosis","Describe-then-diagnose lifts zero-shot accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000639,"raw_usage":{"total_tokens":2970,"prompt_tokens":998,"completion_tokens":1972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":1908}},"tokens_in":614,"tokens_out":1972,"duration_ms":18177,"temperature":1.0,"reasoning_tokens":1908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:31:11.023897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same protocol on the chest X-ray benchmark with $N=16$ across a large number of independent seeds (random seeds, sampling temperatures, and model checkpoints) and compute a confidence interval for the AUC gap; if the interval for the reported 0.737-to-0.821 improvement includes zero, the central claim is not distinguishable from sampling noise.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems37, 27850–27895 (2025)","cited_arxiv_id":null,"evidence_quote":"Provides evidence that transformers benefit from task decomposition, motivating the two-stage architecture."}],"review_version":1}