{"id":"81da0067-380f-4067-8ed9-34cbbdb746e0","arxiv_id":"2507.02822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A learned router for a dual-mode LLM raises medical QA accuracy from 0.827 to 0.839 while cutting inference time by 36.8% and tokens by 39.7% versus always using thinking mode.","lead":"This paper builds a router that decides, for each medical question, whether to use a large language model's expensive 'thinking' mode or its fast 'non-thinking' mode. On four medical QA datasets, the routed system slightly beat always-thinking accuracy while cutting inference time by 36.8% and token use by 39.7%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy gain may be an artifact of test-set threshold selection; no validation-based threshold or confidence interval is reported.","rationale":"The reader's verdict of CONDITIONAL is appropriate. The main concern is sharper than the reader's stated weakest assumption (model-derived labels): the threshold selection procedure directly affects the validity of the reported accuracy improvement on the evaluated model itself. If the threshold was chosen on the test set, the accuracy comparison is optimistically biased. The model-derived labels are a real generalization limitation, but they do not invalidate the within-model comparison; the threshold issue potentially does. The cost reductions (36.8% time, 39.66% tokens) are arithmetic consequences of routing many questions to non-thinking mode and are unlikely to be overturned. Thus the paper remains a plausible engineering contribution, but the central accuracy claim needs a re-analysis with a validation-based threshold and uncertainty quantification. This does not change the reader's CONDITIONAL verdict; it reinforces it with a more concrete methodological requirement.","tokens_in":15694,"tokens_out":7596,"duration_ms":87673,"concrete_test":"Re-tune the router threshold using only the 4,710 training items, e.g., via 5-fold cross-validation to pick the F1-maximizing threshold, then freeze it and evaluate on the 1,177-question test set. Report SynapseRoute accuracy, the difference vs thinking mode, and a bootstrap 95% confidence interval for that difference. If the difference is no longer positive or the CI straddles zero, the accuracy-improvement claim should be removed or downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported accuracy advantage of SynapseRoute over thinking mode (0.8390 vs 0.8272, Table 1) is the only non-mechanical part of the central claim. In the Routing Algorithm section, the authors state that for the Logistic Regression router they 'aimed to maximize the F1 score, selecting an optimal threshold for final class prediction,' but they do not specify that this threshold is chosen on a held-out validation set. The same section then reports test-set AUC and accuracy, suggesting the threshold may have been tuned on the test set itself. With only 1,177 binary test questions, the 0.0118 accuracy gap corresponds to roughly 14 questions; the binomial standard error is about 0.011, so the difference is within one standard error. No confidence intervals or significance tests accompany the accuracy comparison. If the threshold were selected on a validation split, the accuracy advantage could shrink or disappear, undermining the 'improves overall accuracy' claim. The cost savings are robust and mechanical, but the accuracy claim is not yet established under a properly held-out threshold selection procedure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SynapseRoute, a routing framework that decides, per medical multiple-choice question, whether to invoke the thinking or non-thinking mode of a dual-mode LLM (Qwen3-30B-A3B). Questions are automatically labeled as 'thinking' or 'non-thinking' from the model's own correctness, inference time, and token usage in the two modes, with 'fail' questions excluded. A logistic regression classifier over bge-large embeddings is trained on these labels, and a decision threshold is chosen to maximize F1. On four medical QA datasets, the authors report that the dynamic mode improves accuracy over thinking-only mode (0.8390 vs 0.8272) while reducing inference time by 36.8% and token consumption by 39.66%. The paper also introduces an Accuracy-Inference-Token (AIT) index for jointly evaluating accuracy and cost.","tokens_in":15882,"tokens_out":5567,"duration_ms":64444,"significance":"If the reported accuracy gain is real, the work is practically useful: it shows that a lightweight router can cut inference cost substantially within a single dual-mode model while preserving or slightly improving accuracy. The automated labeling pipeline is cheap and scales to other datasets, and the AIT index is a reasonable way to make cost-accuracy trade-offs explicit. The cost savings are large and robust because the router sends most questions to the cheap mode. However, the central accuracy claim is currently fragile: the gain is small, is not accompanied by confidence intervals or significance tests, and may be affected by test-set threshold selection and by the exclusion of 'fail' questions from the evaluation. The paper's broader claim that the 'thinking'/'non-thinking' distinction is a property of questions rather than of one particular model is not supported by the experiments.","major_comments":[{"comment":"The paper states that for the logistic regression router, 'we further aimed to maximize the F1 score, selecting an optimal threshold for final class prediction,' but it does not say that this threshold was chosen on a held-out validation set. The same section reports test-set AUC and accuracy, which suggests the threshold may have been tuned on the test set. Since the end-to-end accuracy advantage of SynapseRoute over thinking mode (0.8390 vs 0.8272) is the only non-mechanical part of the central claim, this is load-bearing. On 1,177 binary test questions, the difference corresponds to roughly 14 questions, and the binomial standard error is about 0.011, so the observed gap is within one standard error. Please specify the threshold selection protocol, and report confidence intervals or a significance test for the accuracy comparison.","section":"Routing Algorithm; Table 1"},{"comment":"The end-to-end accuracy comparison appears to be computed only on the 1,177 non-fail test questions, not on the full 1,273-question test set. Table 7 shows that the full test set has 1,273 questions and the binary (non-fail) test set has 1,177, meaning 96 'fail' questions are excluded. The abstract claims that SynapseRoute 'improves overall accuracy,' but on the full test set, where both modes are incorrect on fail questions, the comparison is not reported. The accuracy of thinking mode on the full set may differ from the listed 0.8272. Please clarify the exact evaluation set for Tables 1 and 2, and report the accuracy comparison on the full test set as well.","section":"Table 7; Results"},{"comment":"The abstract and introduction state that 'approximately 58% of medical questions can be accurately answered by the non-thinking mode alone,' but the 57.8% figure is the proportion of questions labeled as 'non-thinking' after the labeling rule that breaks ties between modes by efficiency. Under that rule, a question where both modes are correct is labeled 'thinking' if the thinking mode has lower token usage and shorter latency, even though the non-thinking mode also answers it correctly. Therefore the non-thinking label count is a lower bound on the set of questions answerable correctly by non-thinking mode, and the actual percentage could be larger. The paper conflates the router's target label distribution with the model's capability. Please report the actual proportion of questions for which the non-thinking mode produces the correct answer, independent of the efficiency tie-break.","section":"Data Labelling; Figure 2b; Abstract"},{"comment":"The automated labeling defines 'thinking' and 'non-thinking' questions by the behavior of a single model, Qwen3-30B-A3B. The 57.8% non-thinking figure and the router's training targets are therefore properties of this model, not intrinsic properties of the questions. The paper claims the framework can be applied broadly to other dual-mode LLMs, but no cross-model validation is provided, and the Discussion's limitations section does not mention this model dependence. At minimum, the manuscript should state this limitation explicitly; ideally, it should include a small experiment with a second dual-mode model to assess label transfer.","section":"Data Labelling; Discussion"}],"minor_comments":[{"comment":"The subsection heading 'Cross-Dimensional Joint Evaluation Metrics Anlysis' contains a typo; it should be 'Analysis'.","section":"Results heading"},{"comment":"The caption states '1.5% Accuracy Gain,' but the reported improvement is 0.8390 - 0.8272 = 0.0118, i.e., about 1.18 percentage points, not 1.5%. Please use percentage points or reconcile the numbers.","section":"Figure 1 caption"},{"comment":"In the Discussion, 'Zhang et al. (Yang et al., 2023)' is inconsistent: the citation label says Zhang et al., but the reference is Yang et al. (2023). Please correct the in-text citation.","section":"Discussion / References"},{"comment":"Table 2 reports an average thinking-mode token size of 789.4933, while Table 6 reports a mean of 799.31 tokens for thinking mode. The difference presumably arises from the different evaluation subsets (test set vs full dataset), but this is not stated; please clarify the populations used in each table.","section":"Tables 2 and 6"},{"comment":"The paper lists USMLE as a data source, but the Discussion refers to 'MedQA' instead. MedQA and USMLE are sometimes used interchangeably, but the names should be consistent throughout.","section":"Data source description"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical problem, and the cost savings are substantial and likely real. The main blockers are the test-set threshold selection and the exclusion of fail questions from the accuracy comparison; both are addressable with additional experiments. I would not reject the paper, but the central accuracy claim needs to survive a properly held-out evaluation and a full-set comparison before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is a clean engineering story: route each medical question to either the thinking or non-thinking mode of a single dual-mode LLM (Qwen3-30B-A3B), with labels derived automatically from that same model's correctness in each mode. The cost savings are real and mechanical—about 57.8% of questions get the cheap path, and the reported 36.8% time and 39.66% token reductions are consistent with that split. The AIT index is a reasonable composite, and the bootstrapped CIs on AIT are a nice touch.\n\nThe accuracy claim is the fragile part. The gain over thinking mode is 0.8390 vs 0.8272, about 14 questions on a 1,177-question test set, and no confidence interval or significance test is given. The routing threshold for logistic regression is described as 'aimed to maximize F1' without stating that it was chosen on a held-out validation set, so the stress-test concern about test-set threshold selection is legitimate. The automated labeling is also model-specific: 'non-thinking' means Qwen3 answers correctly without thinking, which is a property of this model, not of the questions. The 58% figure should be read as '58% of these questions are easy for Qwen3's fast mode,' not as a universal fact.\n\nI'd send it to a serious referee because the routing framework is practical and the cost savings are solid; the accuracy claim needs a validation-split threshold and ideally a second dual-mode model. It's a useful contribution to efficient inference, not a breakthrough. I'd cite it if I worked on routing, but I wouldn't hinge anything on the accuracy gain.","headline":"Practical routing for dual-mode LLMs with solid cost savings, but the accuracy gain over thinking mode is not statistically established.","tokens_in":16438,"tokens_out":2396,"would_cite":true,"duration_ms":26236,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A trained router can beat thinking-only accuracy while using 40% fewer tokens.","keywords":["dynamic routing","dual-mode LLM","thinking mode","non-thinking mode","medical question answering","inference cost reduction","over-reasoning","AIT index"],"falsifier":"Run the same automated labeling and the same logistic-regression router on a different dual-mode LLM over the same 6,365 questions; if the non-thinking-answerable fraction falls well below 58% or the router's routing accuracy approaches the label-noise floor, the central claim that question complexity rather than model-specific behavior drives the savings would be wrong.","tokens_in":15512,"feed_emoji":"🩺","tokens_out":5231,"duration_ms":52843,"temperature":0.7,"pith_summary":"The paper argues that a dual-mode LLM's expensive 'thinking' mode is wasted on a large share of medical questions, and that a cheap classifier can decide per question which mode to run. Its central result is a routing framework, SynapseRoute, which on 6,365 medical multiple-choice questions attains accuracy 0.8390 versus 0.8272 for thinking-only mode, while cutting inference time by 36.8% and token use by 39.66%. The reason this works is that roughly 58% of those questions are answered correctly by the non-thinking mode, and the thinking mode can actually hurt on short factual questions by over-reasoning. If correct, the finding means dual-mode LLMs can be made cheaper and more accurate at the same time without deploying separate models.","feed_headline":"Router lifts accuracy over thinking mode and cuts tokens 40%","feed_subtitle":"SynapseRoute sends easy questions to the fast mode, cutting time by 37% and tokens by 40% while gaining accuracy.","key_machinery":"The load-bearing component is a Logistic Regression classifier trained on vector embeddings of the question text. Its training labels come from an automated annotation rule: run the dual-mode model in both modes; if only one mode answers correctly, label the question accordingly; if both answer correctly, label it by the faster and cheaper mode; if both fail, discard the question. At inference time this classifier decides whether to activate the thinking mode or the non-thinking mode. The AIT index is the auxiliary evaluation mechanism that turns accuracy, inference time, and token count into one normalized score.","core_discovery":"The paper claims that in a sample of 6,365 medical multiple-choice questions, about 58% can be answered correctly by the non-thinking mode of a dual-mode LLM, and that a supervised router can learn to identify such questions from text embeddings alone. On the test set, SynapseRoute beats thinking-only accuracy (0.8390 vs. 0.8272) while reducing inference time by 36.8% and token consumption by 39.66%. The accuracy gain is attributed to avoiding over-reasoning on direct, information-poor questions, where the thinking mode tends to bring in irrelevant associations and self-defined terminology that lead to wrong answers. The paper also introduces the Accuracy-Inference-Token (AIT) index, a weighted combination of accuracy, inference time, and token count, as a single scalar for comparing modes under different cost-performance preferences.","pith_inferences":["The same routing idea likely transfers to non-medical dual-mode LLM use, such as customer support or code generation, but the paper only tests medicine; labeling and routing questions from law or finance would show whether the non-thinking share and savings persist.","The automatic labeler is model-specific: a different dual-mode LLM may have a different non-thinking-answerable fraction, so a router trained for one base model may need retraining when the base model changes.","The paper's failure analysis suggests that a simple text-length or information-density heuristic might capture much of the router's behavior; testing whether a length threshold alone reproduces the accuracy and cost results would isolate what the embedding-based router adds."],"forward_implications":["Routing roughly 58% of medical questions to the non-thinking mode preserves or improves accuracy while cutting inference time and token cost.","Avoiding the thinking mode on short, direct questions removes a source of over-reasoning hallucinations, which is why overall accuracy can rise above thinking-only accuracy.","A lightweight logistic regression router suffices for the routing task, outperforming a LoRA-fine-tuned 4-billion-parameter model on routing accuracy while being cheaper to deploy.","The AIT index gives deployers a single score for comparing modes under different accuracy-versus-cost weighting schemes."],"supporting_citations":[{"why":"Supplies the Qwen3 dual-mode LLM with the thinking/non-thinking switch that SynapseRoute controls.","marker":"Yang et al., 2025"},{"why":"Provides the USMLE dataset, the largest share of the evaluation questions.","marker":"Jin et al., 2021"},{"why":"Provides the MedMCQA dataset used in training and evaluation.","marker":"Pal et al., 2022"},{"why":"Provides the PubMedQA dataset used in training and evaluation.","marker":"Jin et al., 2019"},{"why":"Provides the CareQA dataset used in training and evaluation.","marker":"Arias-Duart et al., 2025"},{"why":"Supports the over-reasoning failure mode by showing thinking-mode accuracy can decline as reasoning length grows.","marker":"Su et al., 2025"},{"why":"Supplies the LoRA method used in the comparison fine-tuning path.","marker":"Hu et al., 2022"}],"fun_headline_variants":["Router sends easy queries to fast mode, cuts tokens 40%","Adaptive router beats thinking-only mode on accuracy and cost","58% of queries need no heavy reasoning, router learns to skip it","Router cuts tokens 40% and time 37%, still improves accuracy","Dual-mode LLM router boosts accuracy while slashing token use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The labels that tell the router which questions need thinking are generated by one particular model's behavior, so if that model's mode behavior is not representative of other dual-mode LLMs, the reported gains may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Router sends easy queries to fast mode, cuts tokens 40%","Adaptive router beats thinking-only mode on accuracy and cost","58% of queries need no heavy reasoning, router learns to skip it","Router cuts tokens 40% and time 37%, still improves accuracy","Dual-mode LLM router boosts accuracy while slashing token use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000998,"raw_usage":{"total_tokens":4243,"prompt_tokens":978,"completion_tokens":3265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":3174}},"tokens_in":594,"tokens_out":3265,"duration_ms":25532,"temperature":1.0,"reasoning_tokens":3174,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:19:46.658404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same automated labeling and the same logistic-regression router on a different dual-mode LLM over the same 6,365 questions; if the non-thinking-answerable fraction falls well below 58% or the router's routing accuracy approaches the label-noise floor, the central claim that question complexity rather than model-specific behavior drives the savings would be wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the USMLE dataset, the largest share of the evaluation questions."},{"cited_title":"K., and Sankarasubbu, M","cited_arxiv_id":null,"evidence_quote":"Provides the MedMCQA dataset used in training and evaluation."}],"review_version":1}