{"id":"9f75bdfd-dd28-482e-8532-13e691c147e8","arxiv_id":"2506.22716","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.","lead":"BEST-Route is a system that selects both which LLM answers a query and how many candidate answers to generate, so small models can match large ones through best-of-n sampling. It reports up to 60% cost reduction with less than 1% quality loss on its benchmark, which matters for cutting LLM serving costs at scale.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The <1% quality-drop claim is measured only with armoRM, the same reward used to train both the proxy selector and the router; BLEU/ROUGE results show drops above 18%, so the headline is not yet established for general response quality.","rationale":"The reader's weakest_assumption focuses on whether R_proxy preserves rankings under R_GT, which is a real concern. My analysis goes one step further: even if R_proxy perfectly preserved armoRM rankings, the central claim still would not be established because the ground-truth reward is itself armoRM, the same metric used for final evaluation. The paper's own BLEU/ROUGE numbers (Table 4) provide direct evidence that the '% quality drop' is highly metric-dependent, and the claimed '<1% drop' is not a general property of the routed outputs. This does not make the method unsound; it does mean the headline is over-stated relative to the evidence. The reader's CONDITIONAL verdict remains appropriate: the method is promising, but the core quality claim needs independent validation with human judgments or task-level metrics, and the artifacts should be released for reproducibility. My concern is complementary rather than contradictory to the reader's, hence 'partial' agreement and no change to the verdict.","tokens_in":16519,"tokens_out":4568,"duration_ms":48863,"concrete_test":"On a held-out sample of at least 500 queries per operating point from the same test distribution, collect human pairwise preferences between the BEST-Route-selected response and the GPT-4o single response at the 10%, 20%, 40%, and 60% cost-reduction thresholds, and also compute task-specific accuracy on the coding and safety subsets (e.g., code correctness / pass@1 and safety violation rate). Additionally, compare the best-of-n response chosen by R_proxy against a randomly selected response from the same n samples using both human ratings and armoRM. If the human or task-level quality drop at 60% cost reduction is not below 1%, or if R_proxy does not beat random selection, the headline claim fails for general response quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, 'up to 60% cost reduction with less than 1% performance drop,' depends entirely on the choice of response-quality metric, and that metric is also the training reward for every learned component. Section 3.3 adopts armoRM as the evaluation metric; Section 4.1 uses armoRM scores as R_GT to construct pairwise training pairs for the proxy reward model R_proxy; Section 4.2 trains the multi-head router to predict match probabilities using labels derived from armoRM comparisons. At test time, the router selects responses by maximizing R_proxy, and quality is then re-measured by armoRM. This creates a closed optimization loop: the system is explicitly trained to maximize armoRM and then evaluated on armoRM. The only 'independent' evidence offered for armoRM as a quality measure is Appendix C, a single anecdotal example, not a correlation or calibration study. The paper's own Table 4 shows that at 60% cost reduction, BLEU drops by 18.07% and ROUGE by 21.97%, more than an order of magnitude larger than the 0.8% armoRM drop. This does not prove the method is wrong, but it shows the headline is metric-specific and unvalidated against human judgment or task-level accuracy. If armoRM is not strongly aligned with human-perceived quality on the test distribution, the claimed cost savings may come at a much larger real quality cost than reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BEST-Route, a routing framework that combines a multi-head router with best-of-n sampling. For each query, the router predicts, for each small model and sample count n, the probability that best-of-n sampling from that model matches the quality of a single GPT-4o response; it then selects the cheapest (model, n) combination whose predicted match probability exceeds a threshold t, falling back to GPT-4o when no combination qualifies. The selected responses are ranked by a fine-tuned proxy reward model, which is itself trained on pairwise preferences derived from armoRM scores. Experiments on a new 10K-example dataset and on MT-Bench report up to 60% cost reduction with less than 1% armoRM score drop, and additional BLEU/ROUGE results. The paper also analyzes latency overhead, model usage before/after adding a specialized coding model, and cost-estimation error.","tokens_in":16846,"tokens_out":3589,"duration_ms":41393,"significance":"If the reported results hold, BEST-Route would be a practically useful contribution: it demonstrates that adaptive best-of-n sampling can extend the Pareto frontier of cost versus quality in LLM serving, and the multi-head router design is a sensible way to avoid training K×N independent routers. The paper also ships a concrete system design, detailed cost model, and a new evaluation dataset, which are valuable assets. However, the headline claim is currently supported mainly by armoRM, which is also the training reward for every learned component; the independent BLEU/ROUGE evidence shows much larger quality drops. The missing reporting of the threshold tuning procedure and the absence of error bars make the quantitative claims difficult to interpret. The central idea is defensible, but the evidence needs to be strengthened before the stated conclusions are established.","major_comments":[{"comment":"The evaluation metric armoRM is also the training reward for both the proxy reward model and the router. armoRM is used as R_GT to construct the pairwise training set for R_proxy (Section 4.1), as R_GT in the router label y_n(q) (Section 4.2, Eq. 3), and as the response-quality metric in Tables 1–3. The headline claim of '0.8% quality drop' at 60% cost reduction therefore measures closeness to the training objective, not quality as judged by an independent criterion. The paper's own Table 4 shows that under BLEU and ROUGE the same operating point has 18.07% and 21.97% drops, respectively. To support the claimed '<1% performance drop,' the authors should provide an independent evaluation (for example, human judgments, task accuracy, or an out-of-training reward model) and should present the BLEU/ROUGE results as primary evidence alongside armoRM rather than as a secondary robustness check.","section":"Section 3.3, Section 4.1, Section 4.2, Table 1"},{"comment":"The label y_n(q) is defined as a probability, but the paper never specifies how this probability is estimated during training. It is not stated whether a single reference response is sampled per query, whether the event R_GT(s*_small) >= R_GT(s_ref) is treated as a binary label, whether multiple responses are averaged, or how ties are handled. Without a concrete estimator, the router training procedure is not reproducible. Please specify the exact construction of the training labels, including the number of reference samples and any smoothing or averaging used.","section":"Section 4.2, Eq. (3)"},{"comment":"The match probability threshold t is a free parameter, and its values are never reported. The cost-reduction operating points of 10%, 20%, 40%, and 60% in Tables 1–4 appear to be produced by adjusting t, but the paper does not state how t is chosen, whether it is tuned on the validation set, or what values correspond to each row. This is load-bearing because the claimed cost-quality trade-off is a function of a fitted operating point; without reporting the selection procedure, the comparison to baselines that do not have an equivalent tunable parameter is not apples-to-apples. Please report the threshold values and the validation-based selection protocol.","section":"Algorithm 1, Section 5.2, Tables 1–4"},{"comment":"All reported cost-reduction and quality-drop numbers are single point estimates with no error bars, confidence intervals, or significance tests. The test set has only 1K examples, and differences such as 0.19% vs. 0.63% at 10% cost reduction in Table 1 may be within noise. The authors should report means and variances over multiple seeds or bootstrap resamples, at least for the main tables.","section":"Tables 1–4"},{"comment":"The claim that BEST-Route shows 'robustness under distribution shifts and generalizability to alternative quality metrics' is overstated. Table 3 shows that on MT-Bench the 60% cost-reduction point has a 1.59% armoRM drop, which is already larger than the '<1%' headline; Table 4 shows much larger BLEU/ROUGE drops. The MT-Bench result is still an in-training-metric evaluation, since armoRM is the training reward, and BLEU/ROUGE are lexical overlap metrics with known weak correlation with human judgment, as the paper itself notes in Section 3.3. Please temper the generalization claim and provide an evaluation that does not rely on the training reward.","section":"Section 5.5, Table 3, Table 4"}],"minor_comments":[{"comment":"The word 'specialized' is misspelled as 'specilized' in the description of Codestral-22b.","section":"Appendix A.1"},{"comment":"The y-axis label of Figure 2 reads 'Avg. armoRM score ( )' with an empty parenthetical; please complete the label or remove the empty parentheses.","section":"Section 4.1, Figure 2"},{"comment":"The symbol n is used both as a specific sample count (e.g., in Eq. 2) and as the maximum sample count in Algorithm 1. Please use distinct notation, such as N_max, to avoid confusion.","section":"Section 4.2"},{"comment":"The notation y_n(q) for the training label and p_n(q) for the predicted probability is confusingly similar; consider using a different symbol for the label, such as l_n(q).","section":"Section 4.2, Eq. (3)"},{"comment":"The armoRM case study is a single anecdotal example. It is fine as an illustration, but it should not be presented as validation; a sentence clarifying its illustrative status would help.","section":"Appendix C"},{"comment":"The text says 'Codes will be released upon acceptance of this work' while the code is also listed with a GitHub URL; please clarify the current availability status.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea—adaptive best-of-n sampling combined with a multi-head router—is timely and potentially useful for LLM serving systems. In my view the main barrier is not the method but the evidence: the evaluation metric is the same as the training reward, the threshold t is an unreported fitted parameter, and there are no error bars. These are fixable within the scope of a revision. I would also encourage the authors to consider whether the BLEU/ROUGE drops in Table 4 should be moved into the main text, since they materially qualify the headline claim. There are no concerns about novelty or attribution that I would raise to the editor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the core idea is a real step forward. Prior routing picks one model and samples once; prior test-time compute scales n for one model. BEST-Route is the first system I've seen that jointly predicts the match probability for every (model, n) pair and then picks the cheapest valid combination. That is a new and useful contribution, and the multi-head router is a reasonable way to avoid training K×N separate classifiers. The paper also reports OOD results on MT-Bench and BLEU/ROUGE, which is more than many routing papers bother to do.\n\nBut the headline—up to 60% cost reduction with less than 1% quality drop—is only true under armoRM, and armoRM is the same reward used to train the proxy selector and the router. That is a closed loop. The paper's own Table 4 shows BLEU dropping 18% and ROUGE dropping 22% at 60% cost reduction, more than an order of magnitude worse than the 0.8% armoRM drop. That does not prove the method is worthless, but it does mean the central claim has not been established for general response quality. The only independent evidence for armoRM is a single anecdotal example in Appendix C, not a correlation study.\n\nThere are smaller soft spots too. The threshold t is fitted to hit the target cost reductions, but its value is never reported. The label y_n(q) in Equation (3) is underspecified: the paper does not say how it estimates a probability for each n when only 20 responses are generated per query. There are no error bars anywhere. And the code availability statements conflict: Section 5.1 says \"Codes will be released upon acceptance,\" while the end of the same section gives a GitHub link as if the code already exists. That needs to be fixed.\n\nTo be fair, the paper does list dependency on the proxy reward model as a limitation, and the method itself is coherent. The routing behavior is sensible: it sends easy queries to small models with small n, and harder ones to bigger models or more samples. The cost model is simple and the overhead analysis is honest. I just don't think the paper currently proves its advertised claim.\n\nWho is this for? People building LLM serving systems who want a cost-quality knob that controls both model choice and sampling effort. They will find the framework useful even if the glitzy numbers need pinning down. The paper deserves a serious referee: the idea is timely and the combination is new, but the evaluation needs substantial strengthening before acceptance—independent metrics, human judgments, and a clear statement of how thresholds and labels are set.","headline":"Genuinely new combination of routing and best-of-n, but the headline cost-quality claim is only measured under the same reward model used for training; needs independent validation.","tokens_in":17407,"tokens_out":1853,"would_cite":true,"duration_ms":56763,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Routing queries to cheap models with best-of-n sampling can cut LLM serving cost by up to 60% while keeping response quality within 0.8% of GPT-4o.","keywords":["LLM routing","best-of-n sampling","test-time compute","proxy reward model","cost-quality trade-off","multi-head router","match probability"],"falsifier":"Take held-out queries, generate n=1,5,20 responses from a cheap model, score them with the proxy model and with a more authoritative judge such as human raters or LLM-as-judge, and compare the proxy-best response's judge score against a random sample's judge score; if the gap does not rise with n or turns negative on a nontrivial fraction of queries, the proxy-ranking premise fails. Separately, count how often combinations predicted above threshold t actually beat a single GPT-4o response on a set with known outcomes; systematic overprediction would break the reported cost-quality trade-off on other query distributions.","tokens_in":16343,"feed_emoji":"🤖","tokens_out":11621,"duration_ms":107703,"temperature":0.7,"pith_summary":"BEST-Route tries to settle a practical question: when must a serving system pay for a powerful model like GPT-4o on every query, and when is a cheaper model enough? Its answer is that a cheap model can often be made as good as the expensive one by sampling several responses and keeping the best, and that a router can predict ahead of time which (model, sample count) combination will match the reference quality. The paper trains a multi-head router to output, for each cheaper model and each value of n, the probability that best-of-n beats a single reference response; at inference time it picks the cheapest combination whose probability clears a user-set threshold, and falls back to the reference model when none qualifies. On a 10K-example mix of question answering, coding, and safety tasks, the paper reports up to 60% cost reduction with less than 1% response-quality drop, with the trade-off tunable by threshold. If this is right, LLM services could cut GPU spend without swapping models.","feed_headline":"Cheap models plus best-of-n sampling cut LLM costs 60%","feed_subtitle":"A test-time router picks the cheapest model and sample count that still matches GPT-4o, with under 1% quality drop.","key_machinery":"The load-bearing machinery is the match-probability multi-head router plus the proxy reward model. The router uses a shared DeBERTa-v3-small backbone that encodes the query once, and $K$ by $N$ lightweight heads that each output $p_{k,n}(q)$, the probability that best-of-n responses from model $k$ match or beat a single GPT-4o response; it is trained on labels $y_n(q)$ that are ground-truth indicators of that event. The proxy reward model is a DeBERTa-v3-large fine-tuned on worst/median/best pairs from 20 responses per training query with the logistic pairwise ranking loss, and it selects the best of the sampled responses at inference. Algorithm 1 combines these pieces: predict match probabilities, filter by threshold t, estimate costs from input and output token prices and average output lengths, and return the highest-proxy-score response from the cheapest valid combination.","core_discovery":"The central claim is that the cost-quality ordering of LLMs is not fixed: a small model combined with best-of-n sampling sits on a new frontier that a router can exploit. Concretely, BEST-Route presents an algorithm that, for each query, predicts a match probability for every (cheap model, sample count) pair against a powerful reference model, filters to pairs whose predicted probability clears a threshold t, and executes the cheapest of those pairs; if no pair clears t, it uses the reference model once. Cost is estimated from token prices and average output length, and the best response is selected by a lightweight proxy reward model trained with a pairwise ranking loss. The paper reports experiments where this achieves up to 60% cost reduction with only a 0.8% quality drop measured by armoRM on the in-distribution test set, and a 1.59% drop on the out-of-distribution MT-Bench set, with negative quality drops on coding queries when a specialized model is added to the pool.","pith_inferences":["Because the router separates a shared query encoder from per-(model, n) heads, adding a new cheap model may only require training new lightweight heads; the paper does not demonstrate this transfer, but the architecture invites it.","The threshold rule assumes predicted match probabilities are calibrated well enough that crossing t means the cheap combination is genuinely as good as the reference; a deployment should track realized match rates to detect silent miscalibration.","The same proxy-reward best-of-n mechanism could be applied as a test-time compute policy for a single large model, deciding per query how many samples to draw; the paper studies it only in the routing setting."],"forward_implications":["A serving system can keep its model portfolio unchanged and still reduce cost by letting the router decide how many responses to sample from each cheap model.","The threshold t gives operators a tunable cost-quality dial: lower thresholds favor cheap combinations, higher thresholds protect quality by falling back to the reference model.","Router overhead is small enough for real-time use: at n=20, match-probability prediction takes 0.04s and best-of-n scoring adds 0.58s, roughly 18.7x faster than the fastest local model evaluated.","Specialized cheaper models can be mixed into the pool without changing the routing algorithm; adding Codestral-22b on coding queries produced better-than-GPT-4o quality at 20% lower cost.","The approach carries over to out-of-distribution data and other quality metrics, with MT-Bench quality drop of 1.59% at 60% cost reduction."],"supporting_citations":[{"why":"introduces best-of-n sampling with a learned reward model, the core quality-boosting mechanism BEST-Route applies to small models.","marker":"Stiennon et al. [2020]"},{"why":"extends best-of-n to long-form web tasks, grounding the idea that multiple samples plus selection improve quality.","marker":"Nakano et al. [2021]"},{"why":"supplies the pairwise ranking loss used to train the proxy reward model.","marker":"Ouyang et al. [2022]"},{"why":"provides armoRM, the ground-truth quality scorer and evaluation metric used throughout the experiments.","marker":"Wang et al. [2024a]"},{"why":"prior routing work showing single small-model responses tend not to beat the large model, motivating best-of-n.","marker":"Ding et al. [2024]"},{"why":"RouteLLM, the preference-data routing approach that BEST-Route extends and compares against.","marker":"Ong et al. [2024]"},{"why":"frames test-time compute as a scaling axis, motivating per-query choice of sample count n.","marker":"Snell et al. [2024]"},{"why":"defines the N-label and clustering baselines and the multi-model routing setting.","marker":"Srivatsa et al. [2024]"},{"why":"provides the DeBERTa architecture used as the router backbone.","marker":"He et al. [2020]"}],"fun_headline_variants":["Best-of-n sampling on cheap models beats single GPT-4o calls","Router picks model and sample count: up to 60% cheaper LLM serving","Small models with best-of-n rival large ones for 60% less cost","Adaptive LLM routing minimizes cost while keeping quality within 1%","Test-time compute: cheap models sampled multiple times lose only 0.8% quality"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the proxy reward model ranks the sampled responses in the same order as the true quality score on the queries the router serves; if that ordering is wrong, best-of-n sampling no longer improves quality and the router's cost savings vanish.","fun_headline_variants_meta":{"raw":{"variants":["Best-of-n sampling on cheap models beats single GPT-4o calls","Router picks model and sample count: up to 60% cheaper LLM serving","Small models with best-of-n rival large ones for 60% less cost","Adaptive LLM routing minimizes cost while keeping quality within 1%","Test-time compute: cheap models sampled multiple times lose only 0.8% quality"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3439,"prompt_tokens":924,"completion_tokens":2515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2413}},"tokens_in":540,"tokens_out":2515,"duration_ms":18651,"temperature":1.0,"reasoning_tokens":2413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:59:43.422766+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take held-out queries, generate n=1,5,20 responses from a cheap model, score them with the proxy model and with a more authoritative judge such as human raters or LLM-as-judge, and compare the proxy-best response's judge score against a random sample's judge score; if the gap does not rise with n or turns negative on a nontrivial fraction of queries, the proxy-ranking premise fails. Separately, count how often combinations predicted above threshold t actually beat a single GPT-4o response on a set with known outcomes; systematic overprediction would break the reported cost-quality trade-off on other query distributions.","supporting_citations":[],"review_version":1}