REVIEW 16 cited by
A Unified Approach to Routing and Cascading for LLMs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The availability of a wide range of large language models (LLMs) embedded in various agentic systems has significantly increased the potential of model selection strategies to improve the cost-performance tradeoff. Existing strategies involve either routing, where a single model is chosen per query, or cascading, which sequentially runs increasingly larger models until a satisfactory answer is found. However, current approaches face three key limitations: they (1) lack formal proofs of optimality, (2) fail to identify the conditions under which these strategies are most effective to improve the cost-performance tradeoff, and (3) are unable to combine both paradigms for further improvements. To address these issues, we first derive a novel optimal strategy for cascading and prove the optimality of an existing routing strategy. Further, we propose cascade routing, a unified framework that integrates routing and cascading into a theoretically optimal strategy. Through our analysis, we identify good quality estimators as the critical factor for the success of model selection paradigms. Finally, in our experiments, we show that cascade routing consistently outperforms the individual approaches by a large margin and we analyze quality estimators to determine when routing and/or cascading are useful paradigms for model selection.
Forward citations
Cited by 16 Pith papers
-
Cache-Aware Prompt Compression:A Two-Tier Cost Model for LLM API Caching
CAPC combines query-agnostic compression with prompt caching and is reported cheapest in 16/16 LongBench configurations, but the empirical support has unresolved specification and consistency gaps.
-
How Much of the Routing Gap Is Real? Decomposing the Router-to-Oracle Gap into Reproducible Specialist Advantage and Single-Draw Label Noise
Under stochastic decoding, 12–36% of the LLM router-to-oracle gap is single-draw noise unreachable by any single-commit router, while the rest is recoverable specialist advantage; best-of-K sampling closes the floor.
-
When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models
Any single-output LLM ensemble is accuracy-capped at 1-beta where beta is the all-models-wrong rate, a quantity not captured by pairwise correlations and frequently underestimated by copula models.
-
Flexible Routing via Uncertainty Decomposition
A router that decomposes uncertainty to flexibly route queries between cheap models and oracles while providing regret bounds and supporting abstention in classification tasks with multiple annotations.
-
A Regime Theory of Controller Class Selection for LLM Action Decisions
A regime theory selects the optimal controller class for LLM action decisions from a nested lattice of four classes using three data-estimable bottlenecks, with a Bernstein-tight threshold and empirical matches on mul...
-
Multi-Head Latent Control: A Unified Interface for LLM Agent Decision Making
Hidden-state traces of frozen LLMs/VLMs can be read by lightweight trained heads to predict when to defer, clarify, call tools, or abstain, cutting routed inference cost 27–90%.
-
FairTutor: Equity-Aware Pedagogical LLM Routing for Budget-Constrained AI Tutoring
FairTutor achieves 97.1% of premium pedagogical quality at 71.6% lower cost via equity-aware LLM routing and introduces the TutorAccessEval benchmark to measure access-tier gaps.
-
LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?
LatentRouter routes image-question queries to the best MLLM by predicting counterfactual performance via latent communication between learned query capsules and model capability tokens.
-
A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliability
LLM reliability techniques are unified as communication channel operators, with a new cost-aware router achieving superior quality-cost tradeoffs on hard tasks.
-
Orchestration for Domain-specific Edge-Cloud Language Models
ECO-LLM jointly selects query processing, retrieval, and model components per query, cutting cost by 60% and latency up to 6x versus model routing in edge-cloud tests.
-
Influence of Prompt Engineering on Small Language Models for Guarded Query Routing
Few-shot prompt optimization lifts small language models' guarded-routing scores on GQR-Bench, bringing Qwen3.5 9B to 95.74, within 0.3 points of Gemma 3 27B without weight updates.
-
Training-Free versus Training-Based Intent Classification in LLMs: Accuracy, Robustness, and Failure Modes
Statistical classifiers built on LLM activation norms and coordinates match or beat trained MLP heads on coarse intent routing and resist camouflage better, while MLPs win on fine-grained subfield distinctions.
-
AutoRelAnnotator: Calibrated Model Cascades for Cost-Efficient Relevance Evaluation in Sponsored Search
AutoRelAnnotator routes queries through fine-tuned classifier cascades with isotonic calibration to deliver high-accuracy relevance labels at roughly half the compute cost while adding a small accuracy gain.
-
Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges
A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.
-
T-TAMER: Provably Taming Trade-offs in ML Serving
T-TAMER claims recall is necessary and sufficient for provably optimal early-exit and cascade serving policies, but the main extensions are under-derived and partly reduce to known Gittins-index results.
-
Harnessing Multiple Large Language Models: A Survey on LLM Ensemble
A systematic survey of LLM ensemble methods organized into a taxonomy of ensemble-before-inference, ensemble-during-inference, and ensemble-after-inference stages, with review of benchmarks, applications, and future d...
Discussion (0). Sign in to comment.