REVIEW 30 cited by
Large Language Model Routing with Benchmark Datasets
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Large Language Model Routing with Benchmark Datasets
read the original abstract
There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets, where we consistently improve performance upon using any single model for all tasks.
Forward citations
Cited by 30 Pith papers
-
Switchcraft: AI Model Router for Agentic Tool Calling
Switchcraft routes agentic tool-calling queries to the lowest-cost model that preserves correctness, reaching 82.9% accuracy and 84% cost reduction on five benchmarks.
-
Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers
A well-tuned kNN router matches or exceeds state-of-the-art learned routers on new standardized benchmarks spanning instruction, QA, reasoning, and the first multi-modal visual routing dataset, due to locality of mode...
-
TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning
A heterogeneous GNN that jointly routes each time-series query to the best modality–model pair under a user-chosen accuracy–cost trade-off, yielding large gains on four reasoning tasks.
-
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
A conformal-prediction set-size deferral rule gives LLM cascades a distribution-free marginal coverage bound of 1−Kα, with empirical gains over tuned heuristics reported at test-set-oracle operating points.
-
Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference
Using conformal prediction-set size as the cascade deferral rule yields distribution-free cascade accuracy bounds and usually beats confidence-threshold heuristics on multiple-choice LLM benchmarks.
-
TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning
A heterogeneous-graph router jointly selects the optimal modality (text, vision, or both) and model per time series query, beating prior routing baselines and generalizing to unseen models and tasks.
-
Selective Ensemble Based on Preference-Directed Multi-Objective Bandits
Introduces Pareto C-optimality and PrefUCB algorithm for PDMOB with instance-dependent logarithmic regret bounds, validated on selective ensemble and asset allocation tasks.
-
Beyond Query Memorization: Large Language Model Routing with Query Decomposition and Historical Matching
DecoR routes LLM queries by decomposing them into capability dimensions and matching to historical examples, yielding higher accuracy and lower inference costs than direct-mapping routers on both in-distribution and OOD data.
-
When Efficiency Backfires: Cascading LLMs Trigger Cascade Failure under Adversarial Attack
LLM cascade systems are vulnerable to a new adversarial attack that simultaneously degrades accuracy and destroys the intended cost savings by targeting both the lightweight models and the escalation decision mechanism.
-
SOMA: Efficient Multi-turn LLM Serving via Small Language Model
SOMA estimates a local response manifold from early turns and adapts a small surrogate model via divergence-maximizing prompts and localized LoRA fine-tuning for efficient multi-turn serving.
-
Learning Agent Routing From Early Experience
BoundaryRouter routes queries to LLM or agent using early experience memory from a seed set, cutting inference time 60.6% versus always using agents and raising performance 28.6% versus always using direct LLM inference.
-
Select to Think: Unlocking SLM Potential with Local Sufficiency
Small language models can achieve near large-model reasoning performance by learning to re-rank their own top-K token predictions after distilling selection from the large model.
-
Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
Cross-model semantic disagreement adds an epistemic uncertainty term that improves total uncertainty estimation over self-consistency alone, helping flag confident errors in LLMs.
-
Privacy-Preserving LLMs Routing
PPRoute achieves plaintext-level LLM routing quality with MPC-based privacy and a 20x speedup over naive encrypted implementations via MPC-friendly encoders, multi-step training, and O(1) communication Top-k search.
-
Select-then-Solve: Paradigm Routing as Inference-Time Optimization for LLM Agents
A learned embedding-based router selecting among six reasoning paradigms improves LLM agent accuracy from 47.6% to 53.1% on average, beating the best fixed paradigm by 2.8pp.
-
Vanishing orders, suspensions and zero degree Tur\'an densities
Vanishing 2-degree Turán density forces a 2-vanishing vertex order; suspensions transfer zero-density between consecutive degree parameters, so non-classical degree densities accumulate at zero.
-
A Greedy PDE Router for Blending Neural Operators and Classical Methods
An approximate greedy router for hybrid PDE solvers that mimics optimal selection without true error access and shows faster, more stable error reduction on test equations.
-
Agentic Routing: The Harness-Native Data Flywheel
Step-level, harness-state model routing in OpenSquilla preserves agent quality at far lower cost and turns each decision into environment-labeled data for better routers and specialists.
-
Synthetic Consumer Insight Generation with Large Language Models
Across models, prompts, and temperatures, LLM projective responses match human city-tourism themes at a coarse level while remaining stylistically more polished and compound-adjective-heavy.
-
TriAgent: Divergence-Aware Multi-Agent Committees for Cost-Efficient Financial Sentiment Analysis
A divergence-routed VADER+FinBERT+LLM committee reaches ~0.87 F1 with a 1.5B critic, matching 7B with far less cost, while same-size persona voting regresses to 0.66.
-
RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation
On MovieLens-1M, request-level hard selection of recommender agents underperforms BM25, while item-level learned shortlist aggregation matches or beats it and improves further with gated LLM use.
-
Codifying the Judge: Scalable Evaluation via Program Distillation
LLM judge logic can be distilled into a committee of Python scoring programs that match mid-size LLM judge accuracy on filtered preference datasets, at orders-of-magnitude higher throughput.
-
Select to Think: Unlocking SLM Potential with Local Sufficiency
Select to Think reframes LLM help as ranking among SLM top-K candidates and distills the ranking ability back into the SLM for improved single-pass reasoning.
-
A pragmatic approach to regulating AI agents
AI agents require distinct regulation as AI systems under the EU AI Act with orchestration-layer oversight and a risk-based traffic light authorization system in contract law to preserve human accountability.
-
When Models Know When They Do Not Know: Calibration, Cascading, and Cleaning
A confidence-based, training-free routing and cleaning framework improves accuracy-efficiency trade-offs and identifies mislabeled data in image and language benchmarks.
-
Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Process
LLM-PeerReview ensembles LLMs by scoring responses with LLM-as-Judge and selecting the best via averaging or truth inference, beating Smoothie-Global by 6.9-7.3 points on four datasets.
-
A Scalable Multi-LLM Collaboration System with Retrieval-based Selection and Exploration-Exploitation-Driven Enhancement
SMCS coordinates 15 open-source LLMs via retrieval-based prior selection and exploration-exploitation posterior enhancement, outperforming GPT-4.1 by 5.36% and GPT-o3-mini by 5.28% on eight benchmarks.
-
OrcaRouter: A Production-Oriented LLM Router with Hybrid Offline-Online Learning
OrcaRouter applies LinUCB with hybrid offline-online learning to LLM routing and reports second place on RouterArena at 75.54% accuracy for $1 per 1,000 queries.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
A survey of large-model inference optimization, organized as a four-layer 'token-operations' taxonomy: multi-model fusion, model optimization, compute-model fusion, and compute-network-model fusion.
-
Token-Operations-Oriented Inference Optimization Techniques for Large Models
The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.