{"id":"7129b7bf-7734-4df5-a710-75e1609ec9ac","arxiv_id":"2506.13102","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Test-time scaling helps only certain medical AI models and only on hard questions, with parallel sampling best for short-reasoning models and sequential revision best for deep-reasoning models.","lead":"This paper tested several ways to give medical AI models extra time to think during exams, comparing longer answers, step-by-step revisions, and multiple answer attempts. The authors found that the best strategy depends on the model and the difficulty of the question, and they offer practical guidance for clinicians and developers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sequential-vs-parallel recommendations rest on only two 7B medical-reasoning LLMs; Tables III/IV generalize beyond the measured model categories.","rationale":"The reader's weakest_assumption is about representativeness of models and benchmarks. The concern identified here is a sharper instance of that: the main strategy-comparison experiment covers only two 7B medical-reasoning LLMs, yet the recommendations in Tables III and IV are stated for all model-type categories, including non-reasoning, general-domain, and VLM classes. This is an internal evidence gap rather than merely an external validity caveat, because the paper's own figure and table structure present these as measured findings. The paper is honest about other limitations (open-source models only, no verifiers, limited VLM evaluation), and the token-budget experiments do provide useful descriptive evidence. However, the central claim about optimal strategy depending on model type and task difficulty needs direct sequential/parallel measurements across the categories before the guidelines can be treated as definitive. This does not change the reader's CONDITIONAL verdict; it sharpens the condition that should be imposed: either restrict the guidelines to the measured model class or supply the missing category-level experiments.","tokens_in":16203,"tokens_out":7989,"duration_ms":82228,"concrete_test":"Run the sequential and parallel scaling protocol from Section III-C3 on at least one general non-reasoning LLM (e.g., Qwen2.5-7B-Instruct), one general reasoning LLM (e.g., DeepSeek-R1-Distill-7B), and one medical non-reasoning LLM (e.g., UltraMedical-8B), on MedQA and MedXpertQA, using the same token budgets and shortest-majority-vote selection with at least three random seeds. If 'prefer parallel for non-reasoning' and 'sequential for difficult QA' replicate across these categories, the central model/task-aware claim is supported; if not, the recommendations in Tables III and IV should be restricted to medical-reasoning LLMs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's practical output is the model- and task-dependent recommendations in Tables III and IV. For those to be valid, the comparison among scaling strategies has to be measured on the categories the tables cover. It is not. Section IV-B (Fig. 5) compares sequential, parallel, and hybrid scaling on exactly two models, HuatuoGPT-o1-7B and m1-7B-1k, both medical-reasoning LLMs. No general non-reasoning LLM, no general reasoning LLM, and no VLM is run through the sequential/parallel protocol. The table entries for those categories are inferred from Section IV-A's token-budget experiments, but observing that a model does not consume a larger budget is different from observing how it behaves under majority voting or iterative 'Wait.' revision; there is no measurement that parallel scaling is preferable for 'General, Non-Reasoning' models or that sequential scaling helps 'General, Reasoning' models on difficult QA. Section IV-C then uses the hybrid configuration selected from these same two models as 'the optimal configuration' when testing robustness, which compounds the extrapolation. The central claim is therefore only directly supported for a narrow slice of the stated design space.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper empirically studies test-time scaling strategies for medical AI across LLMs and VLMs. The authors vary token budgets for a broad set of open-source models on five text and two multimodal medical benchmarks; compare sequential revision ('Wait.'-style), parallel sampling with shortest-majority-vote, and hybrid configurations on two medical reasoning LLMs (HuatuoGPT-o1-7B and m1-7B); and then test robustness to misleading user-provided physician opinions. The central claim is that longer reasoning is not universally beneficial in medicine: effectiveness depends on model type and task difficulty, with reasoning models benefiting more on difficult tasks, VLMs benefiting only marginally, and parallel scaling being preferable for models that naturally produce short reasoning traces. The paper distills the findings into recommended strategies in Tables III and IV.","tokens_in":16572,"tokens_out":4025,"duration_ms":40566,"significance":"The question addressed is timely and practically important, and the paper has several genuine strengths: it covers a wide range of open-source LLMs and VLMs, separates token-budget scaling from sequential/parallel scaling, distinguishes easy and difficult medical tasks, and explicitly studies robustness to user-driven factors, which is rarely examined in test-time-scaling work. The main qualitative finding that longer reasoning is not universally beneficial is plausible and consistent with recent literature. The paper is also commendably explicit about its limitations, including the exclusion of proprietary models and the limited VLM analysis. If the recommendations in Tables III and IV were fully supported, the paper would provide useful guidance for practitioners deploying medical QA systems. However, as detailed below, the central model-aware recommendation rests on a much narrower experimental base than the tables claim, and the absence of uncertainty quantification weakens several quantitative comparisons.","major_comments":[{"comment":"The sequential-versus-parallel comparison is run on only two medical-reasoning 7B LLMs (HuatuoGPT-7B and m1-7B), yet Tables III and IV issue strategy recommendations for all four model-type categories, including General Non-Reasoning, General Reasoning, Medical Non-Reasoning, and VLMs. No non-reasoning model, general reasoning model, or VLM is subjected to the 'Wait.' revision or shortest-majority-vote protocol, so the entries 'Prefer parallel scaling' and 'Apply iterative sequential scaling' are inferred from token-budget saturation in Section IV-A rather than measured. Observing that a model does not consume a larger token budget does not establish how it behaves under parallel majority voting or iterative self-revision. Either the sequential/parallel protocol should be run on representative models from each category, or the tables should be explicitly restricted to the measured model types.","section":"Section IV-B, Fig. 5, Tables III-IV"},{"comment":"The robustness experiments use 'the optimal configuration identified in Section IV-B' as the Seq-Para condition, but that configuration was selected on the same two models and on the same benchmarks (MedQA, MedBullets, MedXpertQA) using the results of Fig. 5. The subsequent robustness evaluation therefore reuses the selection data and does not provide an independent test of whether the recommended hybrid strategy generalizes. The authors should report the exact pair counts and token budgets used, justify the absence of any held-out model or dataset split, and ideally compare against a small predefined set of configurations rather than a data-selected optimum.","section":"Section IV-C"},{"comment":"No error bars, repeated runs, or statistical significance tests are reported anywhere. Several load-bearing conclusions rest on small or visually noisy differences; for example, the claim that QVQ shows a 'consistent upward trend' in accuracy with increased token budgets on OmniMedVQA in Fig. 4(b), and the differences between sequential, parallel, and hybrid arms in Fig. 5. Given that the paper offers practical guidance for choosing among scaling strategies, the authors should provide variance estimates over at least a few independent runs, or confidence intervals, for the headline comparisons.","section":"Section IV-A, Figs. 4-5"},{"comment":"Models with 70B or more parameters are run in 4-bit quantization (indicated by '*' in the legends), and this choice is not validated against full-precision baselines. Since quantization can affect both token usage and reasoning behavior, the comparisons between large and small models, such as the DeepSeek-R1 size trends, may be confounded. A small validation set comparing quantized and unquantized outputs for at least one large model would substantiate the assumption that 4-bit quantization does not change relative behavior.","section":"Section III-B and Section IV-A"}],"minor_comments":[{"comment":"There are several typos: 'Followng' in Section II-A, 'experiments wit VLMs' in Section IV-A2, 'OmniMedQV A' in Table I, 'Whle' in Section IV-A1, and 'gain' in Section IV-A2.","section":"General"},{"comment":"The caption of Fig. 4(b) says 'Accuracy of LLMs' but the panel reports VLM accuracy; the caption should be corrected.","section":"Figure 4"},{"comment":"The notation for HuatuoGPT-o1 variants (7B, 8B, 70B, 72B) and m1-7B-1k is not fully defined; Table II lists HuatuoGPT-o1 with base models Llama or Qwen but does not specify which variant corresponds to which base model.","section":"Table II and Section III-B"},{"comment":"The rows for General Reasoning and Medical Reasoning list both 'High performance on reasoning-intensive tasks' and 'Relatively low performance on reasoning-intensive medical question answering tasks'; the apparent contradiction should be resolved, since the latter appears to be intended for calculation tasks.","section":"Table IV"},{"comment":"The difficulty ordering of datasets is asserted from prior work and the number of answer choices, but since Table III depends directly on the easy/intermediate/difficult categorization, providing baseline accuracies or a validation of the ordering would make the recommendations easier to interpret.","section":"Section III-A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical study of a timely topic, and the central qualitative message is reasonable. My main concern is the gap between the narrow sequential/parallel experiments (two 7B medical-reasoning LLMs) and the broad recommendations in Tables III and IV, which cover all model categories and task difficulties. This is fixable either by expanding the experiments or by carefully narrowing the claims. The lack of error bars is also a general weakness for a paper whose output is practical guidance. Overall, the manuscript is within scope for the journal and the findings are worth publishing after revision, but the generalization claims need to be brought in line with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful empirical study, but the headline strategy tables generalize further than the experiments support. The core finding—that test-time scaling helps only some models on some tasks, and mostly not VLMs—holds up across the broad token-budget sweeps. The specific sequential-vs-parallel recommendations in Tables III and IV, however, rest on only two 7B medical-reasoning LLMs, and that overreach should be fixed before the guidelines are trusted.\n\nWhat's new: they actually run VLMs through token-budget scaling on medical benchmarks, which prior work mostly skipped, and they include a user-driven robustness angle (misleading physician comments). That's a real contribution. The observation that m1 and HuatuoGPT-o1 respond differently to budgets, and that calculation tasks behave differently from QA, is useful and consistent with the O1-replication and s1 literature. The paper is honest about its limitations, which I appreciate.\n\nSoft spots, in proportion. First, the stress-test concern is real: the sequential/parallel comparison is only on HuatuoGPT-o1-7B and m1-7B-1k. The tables then recommend 'prefer parallel scaling' for general non-reasoning models and general reasoning models that were never run through that protocol. That's an extrapolation from token-usage patterns, not from measurement. Second, no error bars or significance tests; with these model sizes and benchmarks, some of the differences they discuss could be noise. Third, the 'optimal' hybrid config is selected from the same data it's then evaluated on in Section IV-C, so that robustness result is optimistic. Fourth, 4-bit quantization for ≥70B models is an assumption that isn't validated. None of these are fatal to the main qualitative claims, but they do mean the guidelines should be read as hypotheses, not settled.\n\nWho it's for: anyone working on medical QA systems or inference optimization for LLMs/VLMs. It's a solid applied contribution. I'd cite it for the VLM scaling results and the model-dependency point. It deserves a serious referee—the design is reasonable, the analysis is careful, and the issues are fixable. I'd send it to review with a request to either expand the sequential/parallel experiments to at least one general model and one VLM, or soften Table IV accordingly, and to report variance across seeds.","headline":"Useful applied study with a solid core result—longer reasoning isn't universally better in medicine—but its strategy tables outrun the measurements.","tokens_in":16940,"tokens_out":2341,"would_cite":true,"duration_ms":21836,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that test-time scaling in medical AI only helps reasoning models on hard tasks, and that the best strategy depends on the model and the task.","keywords":["test-time scaling","medical question answering","large language models","vision-language models","chain-of-thought reasoning","sequential and parallel scaling","model robustness","token budgets"],"falsifier":"Run a medical reasoning model on the hard subset of MedXpertQA with token budgets 512, 2048, and 8192: if accuracy is flat or falls with budget, the core difficulty-based claim fails. Conversely, if a non-reasoning general model such as Llama 3-8B shows a large accuracy jump on PubMedQA when the budget grows, the claim that easy tasks gain nothing from more tokens would be contradicted.","tokens_in":16037,"feed_emoji":"🩺","tokens_out":3833,"duration_ms":33852,"temperature":0.7,"pith_summary":"The paper sets out to show that giving medical AI models more time to think during inference is not always helpful. Across five text-based and two vision-based medical benchmarks, it finds that longer chain-of-thought reasoning improves accuracy mainly for models that naturally produce long reasoning traces, and mainly on difficult tasks; on easy tasks extra tokens either do nothing or hurt. It then compares sequential self-revision with parallel sampling, arguing that the best choice depends on the model's token usage and on task difficulty. If this holds, choosing a scaling strategy per model and task could deliver most of the benefit of test-time scaling without wasting compute, and could guide how to avoid misleading prompts in high-stakes medical question answering.","feed_headline":"Longer medical AI reasoning only pays off on hard tasks","feed_subtitle":"Study maps which test-time scaling strategy suits which model and task, and warns that extra tokens can hurt easy cases.","key_machinery":"The central object is the test-time scaling strategy itself, operationalized along two axes: how many tokens the model is allowed to emit, controlled by the maximum sequence length with forced boxed-answer completion, and how reasoning is extended, either by iterative 'Wait.' self-revision in sequential scaling or by generating multiple independent responses aggregated with a shortest majority vote in parallel scaling. The argument is carried by comparing accuracy and coverage across token budgets, and by classifying models as reasoning versus non-reasoning and medical versus general through their measured token usage.","core_discovery":"The paper's central claim is that longer reasoning is not universally beneficial across medical tasks. On easy benchmarks such as PubMedQA and OmniMedVQA, increasing the token budget fails to improve accuracy, and sequential revision can drive models away from originally correct answers. On difficult benchmarks such as MedXpertQA and MedCalc-Bench, models that use extended reasoning traces gain from more tokens, with the paper reporting clear scalability for m1 among LLMs and QVQ among VLMs. The paper further claims that medical fine-tuned models underperform general models on medical calculation because their training data lacks numerical reasoning, and that a hybrid sequential-parallel strategy suits intermediate difficulty, while fully parallel scaling suits models that produce short traces and easy tasks. It concludes with strategy tables: parallel scaling for easy tasks and short-reasoning models, sequential scaling for difficult QA, token-budget expansion for calculation tasks, and avoidance of all user-driven misleading cues on hard tasks.","pith_inferences":["Editorial: If the difficulty-based pattern transfers to clinical deployment, easy triage questions could run with short parallel samples while long sequential reasoning is reserved for complex cases, a workflow the paper does not itself test.","Editorial: The observed gap between coverage and final accuracy suggests that adding a trained verifier to select among parallel candidates, which the paper lists as unexplored, could convert correct intermediate reasoning into correct final answers.","Editorial: For VLMs, the evidence implies that current medical vision benchmarks may not separate visual reasoning from textual clue-following, so a benchmark that removes text-side hints could change the measured benefits of scaling."],"forward_implications":["Easy medical QA tasks should be handled with parallel sampling rather than longer reasoning, because sequential revision can reduce accuracy.","For difficult QA, iterative sequential scaling should be used for reasoning models, while calculation-heavy tasks favor a single extended reasoning path with a larger token budget.","Models that naturally produce short answers, such as HuatuoGPT-o1, gain little from token-budget scaling, whereas models like m1 that naturally reason long benefit strongly.","Users should avoid injecting physician-opinion hints, especially definitive or expert-toned ones, because perturbed prompts hurt more on hard tasks and scaling does not fully restore accuracy.","Medical fine-tuned models may need calculation-oriented training data to match general models on numeric tasks."],"supporting_citations":[{"why":"Supplies evidence that longer chain-of-thought responses do not always improve accuracy and provides the shortest majority vote strategy used for parallel scaling.","marker":"[20]"},{"why":"Establishes that choosing the optimal test-time scaling strategy based on task difficulty can matter more than scaling model parameters.","marker":"[18]"},{"why":"Introduces budget forcing with 'Wait' tokens, the mechanism behind the token-budget scaling experiments.","marker":"[17]"},{"why":"Provides m1, a key medical reasoning model whose token usage and scalability anchor the central claims.","marker":"[34]"},{"why":"Provides HuatuoGPT-o1, a medical reasoning baseline whose short reasoning traces illustrate when scaling fails.","marker":"[27]"},{"why":"Motivates the user-driven misleading-hint experimental setup used to test robustness under test-time scaling.","marker":"[50]"},{"why":"Supplies MedXpertQA, the difficult benchmark that anchors the claim that hard tasks benefit from longer reasoning.","marker":"[48]"},{"why":"Supplies MedCalc-Bench, the medical calculation benchmark used to show that calculation tasks need extended single-path reasoning.","marker":"[49]"},{"why":"Supplies OmniMedVQA, the easy multimodal benchmark used to show that simple tasks gain little from test-time scaling.","marker":"[51]"}],"fun_headline_variants":["Medical AI: longer reasoning helps only on hard tasks","Test-time scaling for medical AI: easy tasks don't need it","Medical LLMs: more tokens can hurt easy cases","Hard medical tasks gain from longer reasoning, easy ones don't"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that a handful of open-source models and the authors' ordering of datasets by difficulty are representative enough that the observed patterns generalize to other medical LLMs and VLMs.","fun_headline_variants_meta":{"raw":{"variants":["Medical AI: longer reasoning helps only on hard tasks","Test-time scaling for medical AI: easy tasks don't need it","Medical LLMs: more tokens can hurt easy cases","Hard medical tasks gain from longer reasoning, easy ones don't"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1863,"prompt_tokens":898,"completion_tokens":965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":897}},"tokens_in":514,"tokens_out":965,"duration_ms":8332,"temperature":1.0,"reasoning_tokens":897,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:05:03.130811+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a medical reasoning model on the hard subset of MedXpertQA with token budgets 512, 2048, and 8192: if accuracy is flat or falls with budget, the core difficulty-based claim fails. Conversely, if a non-reasoning general model such as Llama 3-8B shows a large accuracy jump on PubMedQA when the budget grows, the claim that easy tasks gain nothing from more tokens would be contradicted.","supporting_citations":[{"cited_title":"Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning,","cited_arxiv_id":null,"evidence_quote":"Establishes that choosing the optimal test-time scaling strategy based on task difficulty can matter more than scaling model parameters."},{"cited_title":"MedCalc- Bench: Evaluating large language models for medical calculations,","cited_arxiv_id":null,"evidence_quote":"Supplies MedCalc-Bench, the medical calculation benchmark used to show that calculation tasks need extended single-path reasoning."},{"cited_title":"Om- niMedVQA: A new large-scale comprehensive evaluation benchmark for medical LVLM,","cited_arxiv_id":null,"evidence_quote":"Supplies OmniMedVQA, the easy multimodal benchmark used to show that simple tasks gain little from test-time scaling."}],"review_version":1}