{"id":"02b1d48d-4aa3-4f5b-af39-660ec833ce85","arxiv_id":"2502.00409","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey that classifies LLM routing strategies into pre-generation and post-generation approaches and four implementation families, framed as a performance-cost optimization problem.","lead":"This paper reviews the many ways developers can send each user question to the cheapest or most suitable AI model instead of always calling one large general-purpose model. It sorts these routing methods into clear categories, which helps engineers and researchers compare options that save money and computing power.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cross-paper performance comparisons in Sections 4.1-4.4 are not normalized; the survey's qualitative rankings of routing families may be artifacts of primary-study setups.","rationale":"The reader's weakest_assumption is that primary-study numbers are accurate and comparable. My review confirms this is the load-bearing concern. The paper's central claim—that routing can be usefully classified and that families differ in effectiveness—depends on cross-paper comparisons. The paper repeatedly contrasts methods from different papers, datasets, baselines, and cost models (e.g., Section 4.4.1 compares token probability to FrugalGPT and HybridLLM on different datasets; Section 4.2.3 compares MixLLM to RouterBench, Zooter, MetaLLM, etc. without a shared setup). Section 5.2.2 explicitly concedes the lack of standardisation. The internal evidence also shows contradictions: AutoMix is reported as strong on QASPER/COQA but no better than random on RouterBench (Section 4.4.5); RouteLLM is reported as barely better than random in one study (Section 4.1.3) and as a strong baseline in another (Section 4.2.3). These inconsistencies are not necessarily flaws in the survey—it often flags them—but they undermine the qualitative rankings that are part of its contribution. A survey that describes methods accurately can still be valuable; the taxonomy and the formalisation in Eq. (1) are useful. However, the comparative claims about which family 'works better' would need a standardized evaluation to be asserted. The practical fix is to label comparative statements as primary-source dependent and to add a table of evaluation setups (datasets, model pools, baselines, cost models). My concrete_test would settle whether the comparative claims survive a shared-benchmark re-audit. The verdict stays CONDITIONAL: the survey should add an explicit methodology section on inclusion criteria and a standardized comparison table or clear caveats; if the re-audit shows that comparative claims evaporate under shared benchmarks, the verdict would move toward REJECT or UNVERDICTED for the comparative dimension, though the taxonomy could still stand.","tokens_in":28391,"tokens_out":2117,"duration_ms":16995,"concrete_test":"Re-audit the five strongest comparative claims in Sections 4.1-4.4 (e.g., 'token probability outperforms FrugalGPT and HybridLLM'; 'RL state-based methods match text-davinci-002'; 'MixLLM outperforms RouterBench, Zooter, MetaLLM'; 'similarity-based methods perform worse than random baselines'; 'supervised routing surpasses larger standalone LLMs'). For each, check whether the cited primary study reports (a) the same evaluation dataset, (b) the same candidate model pool, (c) the same cost metric, and (d) a shared baseline (random/oracle/best standalone). If any two compared results differ in dataset or candidate pool, the comparative claim is unsupported; recompute the relative ranking only on entries that share a benchmark (e.g., MMLU or RouterBench) and identical model pools. Report how many of the survey's comparative statements survive.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The survey's central contribution is a taxonomy plus comparative summaries that repeatedly rank or contrast routing families (e.g., 'RL routing strategies can adapt...', 'similarity-based approaches often fail on complex tasks', 'token probability... outperforms FrugalGPT and HybridLLM'). These conclusions assume that cost and performance numbers from different primary papers are compatible. The paper itself concedes in Section 5.2.2 that 'the field lacks a standardised framework for evaluating routing strategies.' In practice, the primary sources use different model pools, datasets, baselines, budgets, and cost models. For instance, Section 4.4.5 notes that AutoMix [1] outperforms HybridLLM and FrugalGPT on QASPER/COQA, but immediately reports that Wang et al. [98] found AutoMix did not outperform random routing on RouterBench. Section 4.2.3 discusses MixLLM [98] 'outperformed several methods proposed in the survey, including RouterBench [37]', without noting that RouterBench is a benchmark, not a method, and without reporting RouterBench's own comparison conditions. Section 4.1.1 cites Jang et al. [42] as showing query-similarity routing has 'inferior results on generative tasks,' and Section 4.4.3 cites Ning et al. [74] as showing GPT-4 did not outperform fine-tuned RoBERTa, while Appendix A lists Ning et al. as 'No routing' baseline—so the comparison is against a routing-free baseline, not against other routing methods. These heterogeneous comparisons make the qualitative rankings of routing families in Section 4 potentially artifacts of primary setups.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This survey reviews routing strategies for LLM-based systems. It formalizes routing in Eq. (1) as a constrained performance-cost optimization problem and organizes the literature along two axes: pre-generation versus post-generation routing (Section 3) and four implementation families—similarity-based, supervised, reinforcement learning-based, and generative routing (Section 4). It also discusses industrial practice and open challenges, most notably the lack of standardized evaluation. The paper's central contribution is the taxonomy and the comparative synthesis, not new empirical results.","tokens_in":28638,"tokens_out":6075,"duration_ms":63089,"significance":"If the taxonomy is accepted, the survey provides a useful structured map of a fast-growing area and a concrete checklist of open problems. It is careful in several places to flag missing baselines and undocumented choices in primary studies (for example, Section 4.2.3 on Shnitzer et al. and Section 4.4.5 on AutoMix). The paper does not ship code or machine-checked artifacts, which is normal for a survey; its value lies in the synthesis. The main risk is that comparative statements inherit the heterogeneous setups of the primary papers; the paper itself acknowledges this in Section 5.2.2. The scope exclusion of answer-selection and ensemble methods is clearly stated in Section 1 and is reasonable.","major_comments":[{"comment":"The sentence 'It outperformed several methods proposed in the survey, including RouterBench [37], Zooter multi-perceptron [65], Sakota et al. [81]’s supervised strategy...' treats RouterBench as a routing method, but RouterBench [37] is a benchmark, not a method. The comparison should name the actual baseline methods used inside RouterBench (e.g., its MLP or k-NN baselines) or be rephrased to say that MixLLM outperformed methods evaluated on RouterBench. As written, this is an inaccurate comparative statement in a section whose purpose is to rank methods.","section":"Section 4.2.3"},{"comment":"Section 4.4.5 reports that AutoMix outperforms FrugalGPT and HybridLLM on QASPER and COQA, and immediately notes that Wang et al. [98] found it did not outperform random routing on RouterBench; Section 4.2.3 also lists AutoMix among the methods MixLLM outperformed. These reports are not necessarily contradictory because they come from different benchmarks, model pools, and cost budgets, but the survey does not state this explicitly and continues to use such cross-paper comparisons to characterize families throughout Sections 4.1-4.4. I recommend adding an explicit comparability caveat before each cross-paper ranking or downgrading these conclusions to 'as reported in the primary study.' This is important because Section 5.2.2 itself concedes that the field lacks a standardized evaluation framework.","section":"Section 4.4.5 and Section 4.2.3"},{"comment":"Ning et al. [74] is described in the main text as a prompt-based routing method (Section 4.4.3) and appears in Table 1 under both query complexity inference and prompt-based routing, but Appendix A labels both rows for [74] as 'No routing.' This inconsistency makes the taxonomy hard to apply to this entry and should be resolved, either by correcting the appendix or by clarifying that the routing variant is a re-analysis of the original work.","section":"Section 4.4.3 and Appendix A"}],"minor_comments":[{"comment":"Equation (1) is under-specified: the scoring function s(q, M) and the cost C_M(q) are not tied to the generated response, and post-generation/cascade routing does not fit the argmax-over-M form because the decision depends on the output of a previously selected model. A short clarifying remark would strengthen the claimed formalization.","section":"Section 1, Eq. (1)"},{"comment":"The statement that query-similarity methods 'often fail to capture complex query-response relationships and often perform worse than random baselines' is too strong for the cited evidence; it should be attributed to specific studies and their evaluation conditions, since the comparison setups differ.","section":"Section 4.1.1"},{"comment":"There are several typos and infelicities: 'several possible scoring function' (Section 2.1), 'Leveraging multiple nearest neighbours address this limitation' (Section 4.1.1), and a missing period before 'Sikeridis et al. [87]' in Section 4.3.1. A copy-editing pass would improve readability.","section":"Throughout"},{"comment":"The discussion of Shnitzer et al. [85] correctly notes that the labeling procedure for the training set is not described, but this caveat should also be reflected in the comparative summary for that entry rather than only in the descriptive paragraph.","section":"Section 4.2.3"}],"recommendation":"minor_revision","confidential_remarks":"The manuscript fits the scope of a survey journal and I see no citation or novelty concerns. The main editorial risk is that readers will cite the qualitative rankings without checking the primary setups; I recommend that the final version carry a prominent comparability disclaimer wherever cross-paper performance comparisons are made."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The survey earns its place. Its timing-based taxonomy (pre- vs post-generation routing) and the four-family grouping (similarity, supervised, RL, generative) are a genuinely helpful map of a fast-growing applied area. It collects a lot of recent work in one place, cites earlier surveys, and is careful enough to note missing baselines (Ning et al.), unreported labeling (Shnitzer et al.), and unspecified cluster assignment (Pichlmeier et al.). That kind of transparency is welcome. The formalization in Eq. (1) is simple, but it usefully frames routing as performance-cost optimization. The main weakness is exactly what the stress-test note hits: the survey repeatedly compares results across papers that were not designed to be comparable. Different model pools, datasets, baselines, cost models, and evaluation protocols are mixed together in Sections 4.1-4.4, and the qualitative rankings of routing families inherit that heterogeneity. To the paper's credit, it explicitly acknowledges the lack of a standardized framework in Section 5.2.2, and it flags one specific contradiction (AutoMix outperforming HybridLLM/FrugalGPT in its own experiments but not beating random routing on RouterBench). Still, some phrasings are sloppy: calling MixLLM's result \"outperforming RouterBench\" treats a benchmark as a method, and the Ning et al. comparison is against a no-routing baseline, not other routers. Those are fixable in revision. The bigger issue is that the survey does not provide a search protocol or inclusion criteria, so it is not reproducible as a survey. That is a moderate limitation, not a fatal one. The taxonomy boundaries are also a bit fuzzy - some methods fit into more than one family - but that is typical for this kind of synthesis. Who gets value from this? Practitioners choosing a routing strategy and researchers new to the area who need a structured overview. It does not advance the state of the art, but it is a solid consolidation. I would send it to peer review and ask for revisions that (a) normalize or clearly hedge cross-paper comparisons, (b) correct the RouterBench phrasing, and (c) add a brief methods note on literature coverage. Desk rejection would be a mistake.","headline":"A useful, well-organized survey of LLM routing that will help practitioners navigate the field; cross-paper performance comparisons are the main soft spot, but the paper itself flags the comparability problem.","tokens_in":627,"tokens_out":794,"would_cite":true,"duration_ms":30617,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey argues that LLM routing is a performance–cost optimisation problem, and that all existing strategies can be organised by timing (pre- vs post-generation) and by implementation family (similarity, supervised, reinforcement…","keywords":["LLM routing","performance-cost optimization","pre-generation routing","post-generation routing","similarity-based routing","supervised routing","reinforcement learning routing","generative routing"],"falsifier":"Run representatives of each routing family (similarity, supervised, reinforcement learning, and generative) on one shared benchmark with a fixed model pool, a single cost model, and random routing plus the best stand-alone model as baselines; if no lightweight strategy reliably beats those baselines on that common setup, the survey's comparative conclusions would not hold.","tokens_in":28127,"feed_emoji":"🔀","tokens_out":6748,"duration_ms":66783,"temperature":0.7,"pith_summary":"Routing in LLM-based systems means sending each user query to the most suitable component, often a smaller or specialised model, instead of letting one large generalist model handle everything. The paper tries to establish that this is a practical, low-resource way to cut financial, computational, and environmental costs while keeping quality high. It formalises the goal as an optimisation problem and then organises the literature along two axes: when the routing decision happens (before or after generation) and how the router is implemented (similarity-based, supervised, reinforcement learning, or generative). A sympathetic reading of the survey is that routing does not require expensive machinery: lightweight methods can approach the quality of much larger models while substantially reducing calls to them. The payoff is a structured map that lets designers choose a routing strategy by budget, latency, and task.","feed_headline":"Routing queries to small LLMs cuts cost without losing quality","feed_subtitle":"Survey formalizes routing as a cost-performance trade-off and sorts every strategy into a two-axis map for builders.","key_machinery":"The load-bearing object is the router function of Equation (1), which selects the candidate that maximises a scoring function $s(q, M)$ under a cost or budget constraint $C_M(q) \\le B$. Around it, the survey builds a two-axis taxonomy: pre-generation routing (predict candidate performance before any output is generated, based on domain or complexity) versus post-generation routing (cascade routing, where each generated answer is evaluated and the query escalates to a larger model if the answer seems inadequate); and four implementation families, namely similarity-based routing (nearest neighbours, clustering, preference similarity), supervised routing (recommendation, domain classification, query complexity inference, answer confidence inference, knowledge graphs), reinforcement-learning-based routing (stateless, state-based, reward inference), and generative routing (prompts, sequence or token probabilities, repeated calls, LLM fine-tuning, code execution). This machinery organises the literature and lets the authors compare strategies on cost, generalisation, and resource requirements.","core_discovery":"The central claim is that routing should be treated as a performance–cost optimisation problem, written as $R_M(q) = \\arg\\max_{M \\in \\mathcal{M}} s(q, M)$ subject to $C_M(q) \\le B$, and that the literature can be understood through two axes: timing (before or after generation) and implementation family. The survey finds that routing does not require expensive machinery: similarity and supervised methods can match or approach large-model quality while cutting calls to the expensive model substantially, and even the most resource-intensive family (generative routing via LLM fine-tuning) can beat much larger generalists on specific tasks. It also reports that post-generation cascade routing is generally more resource-intensive than pre-generation routing, that routers generalise poorly to new model candidates unless they project options into a shared semantic space, and that the field's evaluations are not standardised, making direct comparisons between strategies unreliable. The authors conclude that the router can be effective and lightweight, and that its gains depend on the complementarity of the candidate models.","pith_inferences":["A direct consequence the authors leave implicit is that routing benchmarks should start treating energy and environmental cost as first-class constraints; otherwise a router that appears cheap in API dollars may simply shift the burden onto compute and carbon.","The taxonomy suggests a concrete modular design: a router could be updated for a new model by computing one small performance vector or cluster assignment, making the router a plug-and-play component of evolving LLM systems.","If the field adopted the baselines the survey recommends, a routing method that only beats random routing when the candidate pool is skewed would be exposed; the survey's own comparisons already hint that some reported gains vanish under such controls.","The optimisation framing points toward end-to-end co-adaptation: the router and the downstream pipeline could be optimised together, so routing decisions adapt to changing user behaviour and model availability rather than being fixed at training time."],"forward_implications":["Pre-generation routing is the cheaper default when candidate performance can be predicted from the query; post-generation (cascade) routing buys reliability at the price of extra generations per query.","Lightweight similarity and supervised routers can approach large-model quality while cutting expensive-model calls by 50 to 80 percent in the studies reviewed, making them viable for industrial deployment.","Routing applies beyond model choice to retrieval strategies, guardrails, prompts, and embedding selection, so whole conversational pipelines can be made adaptive.","Routers that represent candidates in a shared semantic space, such as per-cluster performance vectors or graph embeddings, can accept new models without full retraining.","Because the field lacks standardised evaluation, the relative rankings of strategies are provisional; adopting random, oracle, and best-standalone-model baselines is needed before the comparisons become reliable."],"supporting_citations":[{"why":"FrugalGPT supplies the canonical post-generation cascade baseline and is used throughout the survey as a cost-performance comparison point.","marker":"[11]"},{"why":"RouteLLM anchors the preference-based and supervised routing families and is repeatedly used as a comparison baseline for other strategies.","marker":"[75]"},{"why":"RouterBench provides the standardised benchmark the discussion of evaluation challenges points to.","marker":"[37]"},{"why":"GraphRouter is the key example of inductive graph-based routing that generalises to new LLM candidates.","marker":"[26]"},{"why":"Zooter demonstrates reward-guided routing and is used as a baseline across several comparisons.","marker":"[65]"},{"why":"Jitkrittum et al. supply the per-cluster performance vector approach that shows routing can generalise to unseen model pools.","marker":"[47]"},{"why":"HybridLLM defines query complexity inference via sequence-level scores and is a recurring comparison baseline.","marker":"[22]"},{"why":"AutoMix exemplifies the repeated-calls and confidence-based routing family and is compared against cascade and standalone baselines.","marker":"[1]"},{"why":"RouterDC shows contrastive-learning-based similarity routing with strong out-of-distribution performance.","marker":"[12]"}],"fun_headline_variants":["Routing LLM queries to small models saves cost without losing quality","Survey maps LLM routing to cut costs and keep performance","Do more with less: routing strategies for efficient LLM systems","LLM routing: a cost-performance trade-off, decoded","Lightweight routers can match big models at lower cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's rankings and contrasts across strategies assume the performance and cost numbers reported in different primary studies are accurate and comparable with one another, even though the paper itself concedes in Section 5.2.2 that the field lacks a standardised evaluation framework.","fun_headline_variants_meta":{"raw":{"variants":["Routing LLM queries to small models saves cost without losing quality","Survey maps LLM routing to cut costs and keep performance","Do more with less: routing strategies for efficient LLM systems","LLM routing: a cost-performance trade-off, decoded","Lightweight routers can match big models at lower cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1803,"prompt_tokens":1061,"completion_tokens":742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":677,"completion_tokens_details":{"reasoning_tokens":660}},"tokens_in":677,"tokens_out":742,"duration_ms":8412,"temperature":1.0,"reasoning_tokens":660,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:07:32.239326+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run representatives of each routing family (similarity, supervised, reinforcement learning, and generative) on one shared benchmark with a fixed model pool, a single cost model, and random routing plus the best stand-alone model as baselines; if no lightweight strategy reliably beats those baselines on that common setup, the survey's comparative conclusions would not hold.","supporting_citations":[],"review_version":1}