Test-Time-Scaling for Zero-Shot Diagnosis with Visual-Language Reasoning

Ji Young Byun; Navid Azizan; Rama Chellappa; Young-Jin Park

REVIEW 4 major objections 5 minor 25 references

Test-Time-Scaling for Zero-Shot Diagnosis with Visual-Language Reasoning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Sampling 16 image descriptions at test time improves zero-shot medical diagnosis.

desk verdict A clean zero-shot medical VQA recipe with a plausible test-time-sampling gain, but the headline numbers are single-run estimates with no variance, so treat the effect sizes as provisional. read the letter →

arxiv 2506.11166 v1 pith:IUKL3MGT submitted 2025-06-11 cs.CV cs.AI

Ji Young Byun , Young-Jin Park , Navid Azizan , Rama Chellappa This is my paper · ORCID

classification cs.CVcs.AI

keywords zero-shotmedicaldiagnosistest-timescalingvisual-languagereasoningdescribe-then-diagnoseimageclassificationhallucinationmitigationpneumoniadetectionhistopathology

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a vision-language model can become a reliable zero-shot medical diagnostician without any fine-tuning. The proposed answer is to split the job in two: first sample 16 neutral descriptions of the image from a vision-language model, then pass each description to a text-only LLM that outputs a class label, and average the labels. The paper interprets this average as a Monte Carlo estimate of the probability that the image shows the disease. Reported results show that this test-time scaling recipe beats direct zero-shot question answering and one-stage chain-of-thought on chest X-ray, colorectal histopathology, and retinal fundus benchmarks, with the chest X-ray AUC rising from 0.737 to 0.821. The same sampling strategy also improves the baseline methods and allows the second-stage LLM to be as small as 1B–3B parameters without losing much accuracy.

What carries the argument

The load-bearing mechanism is the two-stage decomposition plus a Monte Carlo estimate of the posterior. Stage 1 samples $N$ visual descriptions $v^{(1)},\dots,v^{(N)}$ from a vision-language model under temperature scaling, using a neutral prompt that does not name a disease. Stage 2 feeds each description to a text-only LLM that must return only a boxed class label, and the final prediction is the majority label, i.e. Equation (1), which the paper reads as a Monte Carlo approximation of $\int p(\hat y=1\mid v)\,p(v\mid x)\,dv$. The key move is separating visual feature extraction from clinical reasoning, so the unconstrained Stage-1 prompt is not contaminated by diagnosis hints, and the sampling turns one potentially hallucinated description into an aggregate over a distribution. This same TTS recipe also boosts the baseline methods, which is why the paper treats sampling, not just the two-stage prompt, as the active ingredient.

What would settle it

Run the same protocol on the chest X-ray benchmark with $N=16$ across a large number of independent seeds (random seeds, sampling temperatures, and model checkpoints) and compute a confidence interval for the AUC gap; if the interval for the reported 0.737-to-0.821 improvement includes zero, the central claim is not distinguishable from sampling noise.

Watch

Extended reading notes

Core claim

The central claim is that a single greedy forward pass underuses a zero-shot vision-language model, and that test-time scaling closes the gap. Specifically, the paper proposes a two-stage 'describe-then-diagnose' pipeline in which Stage 1 draws $N=16$ temperature-sampled descriptions from the VLM using a neutral, non-diagnostic prompt, and Stage 2 classifies each description with a text-only LLM forced to emit a boxed label. The final diagnosis is the average of those labels, written as Equation (1): $\hat{p}(\hat{y}=1\mid x) \approx \frac{1}{N}\sum_{i=1}^N \mathbb{I}(a^{(i)}=1)$, where $a^{(i)}$ is the LLM's parsed answer for description $i$. The paper reports that this aggregation improves AUC and average precision over single-sample decoding for the proposed method and for both baselines across all three modalities, that the gains follow a power law in $N$, that a 3B-parameter second-stage model matches the 11B model, and that unconstrained Stage-1 prompts reduce hallucinated findings compared with disease-dictated prompts.

Load-bearing premise

The method works only if averaging sixteen temperature-sampled image descriptions produces a stable, unbiased estimate of the true disease probability; if those sixteen draws are too noisy or share a common bias, the majority diagnosis is not a reliable estimate, and the reported gains could be sampling noise.

Editorial extensions

If this is right

With $N=16$ samples, test-time scaling improves AUC and average precision over single-sample decoding for zero-shot QA, one-stage CoT, and the proposed method on all three modalities.
Even $N=4$ samples yields large gains over a single sample, and performance follows a power law as $N$ grows, implying more samples continue to help.
A second-stage LLM with around 3B parameters performs comparably to the 11B model, so the framework reduces inference cost without sacrificing accuracy.
Neutral, unconstrained prompts in Stage 1 reduce hallucinated and overdiagnosed descriptions relative to disease-dictated prompts, making prompt design part of reliability.
Because TTS also improves the baseline methods, the benefit is an inference-time intervention that generalizes beyond the proposed two-stage prompt.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the gain comes mainly from description diversity, the same two-stage recipe should transfer to other vision-language and text-only model pairings, including much smaller models, as long as Stage-1 descriptions remain clinically faithful.
A controlled experiment fixing the compute budget—comparing one-stage CoT with the same $N=16$ samples and the same aggregator against the two-stage pipeline—would isolate whether the separation of description and diagnosis, rather than sampling alone, drives the improvement.
The power-law trend suggests a testable scaling law for zero-shot medical diagnosis, but it may saturate once the sampled description distribution stops adding new clinical features; measuring where that plateau occurs would set expectations for larger $N$.
The bias-mitigation result could be turned into a training objective for small medical VLMs: supervise description-only outputs first and diagnoses second, rather than asking a single model to do both at once.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick take on arXiv:2506.11166. The paper does something simple and reasonable: split diagnosis into "describe the image" (VLM) then "classify from the description" (LLM), and at test time sample N=16 descriptions and majority-vote. The combination on MedMNIST is new, and the gains in Table 1 are non-trivially large (PneumoniaMNIST AUC 0.821 vs 0.737 for zero-shot TTS; PathMNIST 0.653 vs 0.562). The paper also shows that neutral Stage-1 prompts beat disease-directed prompts (Figure 3a) and that a 3B Stage-2 LLM is enough (Figure 3b). Those are useful empirical data points.

What's not to like: the central claim rests on exactly one stochastic run per cell. No error bars, no confidence intervals, no significance test, no seed or temperature sensitivity for the N=16 sampling. The gap between 0.821 and 0.737 could be within run-to-run variation; we just don't know. The "power law" in Figure 2 is asserted from a handful of points without a fit or error bars. The paper also omits prompts and parsing failure rates, which matters for reproducibility in LLM pipelines. These are evidential gaps, not internal contradictions. Eq. (1) is just a Monte Carlo identity for P(y|x); no circularity there.

The reader's CONDITIONAL verdict gets it right. The method is plausible and the direction is worth publishing, but as submitted the empirical support is thinner than the prose claims. The authors should add repeated runs with different seeds and temperatures, report mean ± std over seeds, include prompt templates, and dial back the power-law language unless they fit it properly.

Who's this for? Anyone working on zero-shot medical VQA or test-time scaling for LLMs. I'd send it to review; the recipe is clean enough and the effect, if real, is useful. I wouldn't cite it in my own work until the variance question is answered.

Recommendation: send to peer review, but expect heavy revision on the experimental reporting.

Referee Report

4 major / 5 minor

Summary. The paper proposes a zero-shot, two-stage framework for medical image diagnosis. In Stage 1, a vision-language model (Llama-3.2-11B-Vision-Instruct) generates N temperature-sampled textual descriptions of the input image under a neutral prompt. In Stage 2, a text-only LLM converts each description into a diagnosis, and the final prediction is obtained by majority voting over the N outputs. The method is evaluated on PneumoniaMNIST, PathMNIST, and RetinaMNIST, with the central empirical claim that test-time scaling (N=16) improves AUC over single-sample zero-shot QA and one-stage CoT, e.g., PneumoniaMNIST AUC 0.821 versus 0.737, and PathMNIST 0.653 versus 0.562. The paper also reports that unconstrained Stage-1 prompting mitigates apparent bias compared with disease-dictated prompting, and that smaller Stage-2 LLMs (3B) perform comparably to larger ones.

Significance. If the reported gains are real and reproducible, the proposed describe-then-diagnose pipeline with test-time scaling would be a useful, annotation-free recipe for improving zero-shot medical VQA, with the practical merit of decoupling visual description from diagnostic reasoning and allowing cheaper Stage-2 models. The method is simple, the Monte Carlo estimator in Eq. (1) is conceptually sound and not circular, and the evaluation uses external MedMNIST benchmarks rather than fitted constants. The main significance, however, is presently conditional: the headline improvement is supported only by single-run point estimates without variance or significance analysis of a stochastic procedure, and the paper itself omits prompts and code. The bias-mitigation and compute-savings analyses are suggestive but similarly lack statistical grounding.

major comments (4)

[Table 1 and Eq. (1)] The central claim that TTS with N=16 improves diagnostic AUC rests on point estimates from a stochastic sampling procedure, yet the paper reports no standard errors, confidence intervals, paired tests, or repetition over seeds. On datasets with 226 to 1,233 test cases, the headline gaps (e.g., PneumoniaMNIST 0.821 vs. 0.737) could plausibly be within run-to-run variation of temperature-sampled descriptions. Please report multiple seeds, bootstrap intervals, or paired significance tests for the main comparisons, and state the sampling temperature and parsing failure rate for Eq. (1).
[Figure 2] The claim that increasing N improves performance 'following a power law' is asserted from a small number of empirical points (N=1, 2, 4, maybe 8, 16) with no fitted curve, no error bars, and no quantitative exponent or goodness-of-fit. Please either provide a fitted power-law model with confidence bounds or soften the claim to 'monotone improvement in the tested range,' and ideally show repeated runs at each N to establish that the trend is not sampling noise.
[Section 3, 'Generic Prompt Enhances Reliability' and Figure 3a] The bias-mitigation conclusion compares 'unconstrained' versus 'dictated' Stage-1 prompts, but the comparison is shown only as point AUC values with no variance or significance testing, and the exact prompts are not given. Since this analysis is one of the paper's stated contributions and is used to argue that neutral prompting reduces hallucination and overdiagnosis, please report effect sizes with uncertainty, and disclose the full prompt templates so the comparison is reproducible.
[Section 3, 'Datasets and Models'] The manuscript states that 'detailed prompts and codes used in the baseline and proposed frameworks are omitted here but will be shared at publication time.' For a zero-shot prompting paper, prompts are part of the method, not merely implementation detail; without them, the method cannot be reproduced or compared fairly across baselines. Please include the complete prompts for zero-shot QA, one-stage CoT, Stage 1 description, Stage 2 diagnosis, and the parsing instruction in the main text or an appendix.

minor comments (5)

[Eq. (1)] The notation p(ŷ=1|x) is written as an integral over v, but the empirical approximation is a simple Monte Carlo average; please clarify that the indicator function I[a(i)="\boxed{1}"] is the Stage-2 LLM's parsed answer, and note that the approximation ignores any dependence of the Stage-2 LLM on the sampling temperature.
[Section 2.2, 'Enhancing Reliability using Test-time Scaling'] The temperature scaling reference [4] is a calibration paper; please cite the original temperature sampling heuristic (e.g., Ackley et al. or the language-model sampling literature) instead of, or in addition to, a calibration reference.
[Section 3, 'Comparison with Baselines'] The phrase 'the best and the second best are highlighted' is not visible in the text table in this version; please ensure the formatting is rendered or describe the highlighting in the caption.
[Section 3, 'Generic Prompt Enhances Reliability'] The sentence 'the theoretical foundation and empirical evidence for Experts Don't Cheat concept' is grammatically incomplete; please revise to a full sentence and clarify how Refs. [8,23] support the claim about VLM hallucination.
[Section 3, 'Ablation on LLM size in the second stage'] The comparison with Med42-v2-8B in Figure 3b is described as 'similar' to the 3B Llama model, but no numerical values are reported in the text; please add the exact AUC/AP values or a table so the claim is verifiable.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Eq. (1) is a Monte Carlo estimator over sampled descriptions, and the core comparisons use external MedMNIST benchmarks with no fitted parameters or self-citation chain.

full rationale

The paper's central derivation is Eq. (1), which approximates p(y=1|x) by averaging indicator outputs from N temperature-sampled visual descriptions. This is a standard Monte Carlo identity expressing an intended estimator, not a result derived from data that the paper then claims to predict. Nothing in the method is fitted to the test labels, and the AUC/AP improvements are measured on the external MedMNIST v2 benchmark. The claimed gains therefore have independent empirical content: the describe-then-diagnose pipeline under TTS is compared against zero-shot QA and one-stage CoT baselines on the same images and labels. The references used for motivation (e.g., Abbe et al. on decomposition, Snell et al. on test-time scaling, Yao et al. on tree-of-thoughts) are external and are not invoked to define the method's validity. There is no self-citation load-bearing premise, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via the authors' own citations. The only notable weakness is that all TTS results are reported as single-run point estimates without variance, significance tests, or seed/temperature sensitivity analysis, given that N=16 sampling is stochastic. That is an evidential gap concerning reliability of the empirical claim, not a circularity: Eq. (1) does not reduce to a fitted input or to the conclusion by construction. Under the review rules, the absence of error bars should be weighed as a reproducibility/correctness risk, not as circularity, and it does not raise the circularity score.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters into the diagnosis itself and no invented entities; Eq. (1) is a Monte Carlo identity rather than a fitted model. The empirical reliability of the method, however, depends on unstated sampling details (temperature), a hand-chosen N=16, and assumptions about VLM description quality and unbiasedness.

free parameters (2)

N (number of sampled descriptions) = 16 for main results (ablated over N=1-16)
The paper selects N=16; performance grows with N, and no saturation or validation-based selection analysis is shown.
Temperature for Stage 1 sampling = not reported
Stage 1 uses temperature scaling to sample N descriptions, but the temperature value is not given; sample diversity depends on it.

assumptions (3)

domain assumption The N sampled visual descriptions are independent draws from p(v|x) and contain enough information for diagnosis.
Used in Eq. (1) to justify the Monte Carlo estimate; the paper does not test sensitivity to temperature, seed, or N beyond 16.
domain assumption Neutral, unconstrained Stage 1 prompts elicit less biased visual features than disease-directed prompts.
Figure 3a gives point estimates only, without error bars or ablations across models; this assumption underpins the bias-mitigation claim.
standard math MedMNIST v2 ground-truth labels are correct.
Standard benchmark assumption; the paper relies on the dataset's resident labels.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Test-Time-Scaling for Zero-Shot Diagnosis with Visual-Language Reasoning." pith.science (2026). https://pith.science/paper/IUKL3MGT

@misc{pith2026250611166,
  author       = {Pith},
  title        = {Pith review of: Test-Time-Scaling for Zero-Shot Diagnosis with Visual-Language Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IUKL3MGT}},
  note         = {Machine review of arXiv:2506.11166}
}

read the original abstract

As a cornerstone of patient care, clinical decision-making significantly influences patient outcomes and can be enhanced by large language models (LLMs). Although LLMs have demonstrated remarkable performance, their application to visual question answering in medical imaging, particularly for reasoning-based diagnosis, remains largely unexplored. Furthermore, supervised fine-tuning for reasoning tasks is largely impractical due to limited data availability and high annotation costs. In this work, we introduce a zero-shot framework for reliable medical image diagnosis that enhances the reasoning capabilities of LLMs in clinical settings through test-time scaling. Given a medical image and a textual prompt, a vision-language model processes a medical image along with a corresponding textual prompt to generate multiple descriptions or interpretations of visual features. These interpretations are then fed to an LLM, where a test-time scaling strategy consolidates multiple candidate outputs into a reliable final diagnosis. We evaluate our approach across various medical imaging modalities -- including radiology, ophthalmology, and histopathology -- and demonstrate that the proposed test-time scaling strategy enhances diagnostic accuracy for both our and baseline methods. Additionally, we provide an empirical analysis showing that the proposed approach, which allows unbiased prompting in the first stage, improves the reliability of LLM-generated diagnoses and enhances classification accuracy.

Figures

Figures reproduced from arXiv: 2506.11166 by the authors.

**Figure 2.** An ablation study examining the effect of sample size (N) in TTS set [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Panel (a) illustrates how biased prompting affects diagnostic performance. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 7 canonical work pages

[1]

Advances in Neural Information Processing Systems37, 27850–27895 (2025)

Abbe, E., Bengio, S., Lotfi, A., Sandon, C., Saremi, O.: How far can transform- ers reason? the globality barrier and inductive scratchpad. Advances in Neural Information Processing Systems37, 27850–27895 (2025)

work page 2025
[2]

arXiv preprint arXiv:2408.06142 (2024)

Christophe, C., Kanithi, P.K., Raha, T., Khan, S., Pimentel, M.A.: Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142 (2024)

arXiv 2024
[3]

arXiv preprint arXiv:2110.14168 (2021)

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

arXiv 2021
[4]

In: International conference on machine learning

Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International conference on machine learning. pp. 1321–1330. PMLR (2017)

2017
[5]

Hastie, T., Tibshirani, R., Friedman, J.H., Friedman, J.H.: The elements of statis- tical learning: data mining, inference, and prediction, vol. 2. Springer (2009)

2009
[6]

Applied Sciences11(14), 6421 (2021)

Jin, D., Pan, E., Oufattole, N., Weng, W.H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences11(14), 6421 (2021)

2021
[7]

arXiv preprint arXiv:1909.06146 (2019)

Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., Lu, X.: Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019)

arXiv 2019
[8]

Johnson, D.D., Tarlow, D., Duvenaud, D., Maddison, C.J.: Experts don’t cheat: Learningwhatyoudon’tknowbypredictingpairs.arXivpreprintarXiv:2402.08733 (2024)

arXiv 2024

Show all 25 references

[9]

Research square pp

Johnson, D., Goodman, R., Patrinely, J., Stone, C., Zimmerman, E., Donald, R., Chang, S., Berkowitz, S., Finn, A., Jahangir, E., et al.: Assessing the accuracy and reliability of ai-generated medical responses: an evaluation of the chat-gpt model. Research square pp. rs–3 (2023)

2023
[10]

Advances in neural information processing systems35, 22199–22213 (2022)

Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Advances in neural information processing systems35, 22199–22213 (2022)

2022
[11]

Advances in Neural Information Processing Systems36, 28541–28564 (2023) 10 Ji Young Byun, Young-Jin Park, Navid Azizan, and Rama Chellappa

Li, C., Wong, C., Zhang, S., Usuyama, N., Liu, H., Yang, J., Naumann, T., Poon, H., Gao, J.: Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems36, 28541–28564 (2023) 10 Ji Young Byun, Young-Ji...

2023
[12]

In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=v8L0pN6EOi

Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., Cobbe, K.: Let’s verify step by step. In: The Twelfth International Conference on Learning Representations (2024), https://openreview.net/forum?id=v8L0pN6EOi

2024
[13]

In: Machine Learning for Health (ML4H)

Moor, M., Huang, Q., Wu, S., Yasunaga, M., Dalmia, Y., Leskovec, J., Zakka, C., Reis, E.P., Rajpurkar, P.: Med-flamingo: a multimodal medical few-shot learner. In: Machine Learning for Health (ML4H). pp. 353–367. PMLR (2023)

2023
[14]

arXiv preprint arXiv:2408.03314 (2024)

Snell, C., Lee, J., Xu, K., Kumar, A.: Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314 (2024)

2024 arXiv
[15]

chatgpt in healthcare: a new frontier in medical ai reasoning

Temsah, M.H., Jamal, A., Alhasan, K., Temsah, A.A., Malki, K.H.: Openai o1- preview vs. chatgpt in healthcare: a new frontier in medical ai reasoning. Cureus 16(10) (2024)

2024
[16]

In: The Thirteenth International Conference on Learning Representa- tions (2025), https://openreview.net/forum?id=mTCbq2QssD

Toshniwal, S., Du, W., Moshkov, I., Kisacanin, B., Ayrapetyan, A., Gitman, I.: Openmathinstruct-2: Accelerating AI for math with massive open-source instruc- tion data. In: The Thirteenth International Conference on Learning Representa- tions (2025), https://openreview.net/for...

2025
[17]

Advances in Neural Information Processing Systems37, 34737–34774 (2024)

Toshniwal, S., Moshkov, I., Narenthiran, S., Gitman, D., Jia, F., Gitman, I.: Openmathinstruct-1: A 1.8 million math instruction tuning dataset. Advances in Neural Information Processing Systems37, 34737–34774 (2024)

2024
[18]

arXiv preprint arXiv:2302.13971 (2023)

Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

2023 arXiv
[19]

Nejm Ai 1(3), AIoa2300138 (2024)

Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.C., Carroll, A., Lau, C., Tanno, R., Ktena, I., et al.: Towards generalist biomedical ai. Nejm Ai 1(3), AIoa2300138 (2024)

2024
[20]

arXiv preprint arXiv:2211.14275 (2022)

Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., Higgins, I.: Solving math word problems with process-and outcome- based feedback. arXiv preprint arXiv:2211.14275 (2022)

2022 arXiv
[21]

CoRR abs/2312.08935 (2023), https://doi.org/10.48550/arXiv.2312.08935

Wang, P., Li, L., Shao, Z., Xu, R.X., Dai, D., Li, Y., Chen, D., Wu, Y., Sui, Z.: Math-shepherd: Verify and reinforce llms step-by-step without human annotations. CoRR abs/2312.08935 (2023), https://doi.org/10.48550/arXiv.2312.08935

2023 doi arXiv
[22]

Advances in neural information processing systems35, 24824–24837 (2022)

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022)

2022
[23]

arXiv preprint arXiv:2406.02543 (2024)

Yadkori, Y.A., Kuzborskij, I., György, A., Szepesvári, C.: To believe or not to believe your llm. arXiv preprint arXiv:2406.02543 (2024)

2024 arXiv
[24]

Scientific Data 10(1), 41 (2023)

Yang,J.,Shi,R.,Wei,D.,Liu,Z.,Zhao,L.,Ke,B.,Pfister,H.,Ni,B.:Medmnistv2- a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data 10(1), 41 (2023)

2023
[25]

Advances in neural information processing systems36, 11809–11822 (2023)

Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., Narasimhan, K.: Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems36, 11809–11822 (2023)

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Advances in Neural Information Processing Systems37, 27850–27895 (2025)

Abbe, E., Bengio, S., Lotfi, A., Sandon, C., Saremi, O.: How far can transform- ers reason? the globality barrier and inductive scratchpad. Advances in Neural Information Processing Systems37, 27850–27895 (2025)

work page 2025

[2] [2]

arXiv preprint arXiv:2408.06142 (2024)

Christophe, C., Kanithi, P.K., Raha, T., Khan, S., Pimentel, M.A.: Med42-v2: A suite of clinical llms. arXiv preprint arXiv:2408.06142 (2024)

arXiv 2024

[3] [3]

arXiv preprint arXiv:2110.14168 (2021)

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021)

arXiv 2021

[4] [4]

In: International conference on machine learning

Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International conference on machine learning. pp. 1321–1330. PMLR (2017)

2017

[5] [5]

Hastie, T., Tibshirani, R., Friedman, J.H., Friedman, J.H.: The elements of statis- tical learning: data mining, inference, and prediction, vol. 2. Springer (2009)

2009

[6] [6]

Applied Sciences11(14), 6421 (2021)

Jin, D., Pan, E., Oufattole, N., Weng, W.H., Fang, H., Szolovits, P.: What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences11(14), 6421 (2021)

2021

[7] [7]

arXiv preprint arXiv:1909.06146 (2019)

Jin, Q., Dhingra, B., Liu, Z., Cohen, W.W., Lu, X.: Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146 (2019)

arXiv 2019

[8] [8]

Johnson, D.D., Tarlow, D., Duvenaud, D., Maddison, C.J.: Experts don’t cheat: Learningwhatyoudon’tknowbypredictingpairs.arXivpreprintarXiv:2402.08733 (2024)

arXiv 2024