Pith. sign in

REVIEW 41 cited by

Large Language Model Routing with Benchmark Datasets

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2309.15789 v1 pith:GOBKAV42 submitted 2023-09-27 cs.CL cs.LG

Tal Shnitzer , Anthony Ou , Mírian Silva , Kate Soule , Yuekai Sun , Justin Solomon , Neil Thompson , Mikhail Yurochkin This is my paper · ORCID
classification cs.CLcs.LG
keywords modelbenchmarkdatasetstasksmodelsbestcollectionlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets, where we consistently improve performance upon using any single model for all tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 41 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TSRouter: Dynamic Modality-Model Selection for Time Series Reasoning

    cs.LG 2026-07 conditional novelty 6.5 of 10

    A heterogeneous-graph router jointly selects the optimal modality (text, vision, or both) and model per time series query, beating prior routing baselines and generalizing to unseen models and tasks.

  2. Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

    cs.AI 2026-08 conditional novelty 6.0 of 10

    Model rankings in LLM evaluation are budget-dependent: the best-performing model changes with the token generation budget on all three benchmarks tested.

  3. LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A new open-source library and benchmark, xRouteBench, evaluates LLM routers on a shared cost-aware protocol across text, memory, vision, time-series, and personalized tasks.

  4. Scrouting: Cost-Aware Routing of Coding Agents by Scouting the Repository First

    cs.SE 2026-08 conditional novelty 6.0 of 10

    A 7B searcher's sandbox-verified handoff lets a cheap coding model match a frontier model's solve rate on SWE-bench Pro at roughly one fifth the cost, while the router that picks the fixer contributes nothing on this ...

  5. Conformal Cascade: Distribution-Free Accuracy Guarantees for Multi-Tier LLM Inference

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Using conformal prediction-set size as the cascade deferral rule yields distribution-free cascade accuracy bounds and usually beats confidence-threshold heuristics on multiple-choice LLM benchmarks.

  6. Vanishing orders, suspensions and zero degree Tur\'an densities

    math.CO 2026-03 unverdicted novelty 6.0 of 10

    Vanishing 2-degree Turán density forces a 2-vanishing vertex order; suspensions transfer zero-density between consecutive degree parameters, so non-classical degree densities accumulate at zero.

  7. Delta Activations: A Representation for Finetuned Large Language Models

    cs.LG 2025-09 conditional novelty 6.0 of 10

    Delta Activations embed finetuned LLMs as the average difference in hidden states between the finetuned model and its base model on a small set of generic prompts, yielding domain clusters and approximate additive com...

  8. Dynamic Collaboration of Multi-Language Models based on Minimal Complete Semantic Units

    cs.AI 2025-08 conditional novelty 6.0 of 10

    MCSU-based vocabulary alignment plus distance-based dynamic selection (DDS) lets several LLMs vote token-by-token, beating single models and prior ensemble baselines on multiple reasoning benchmarks without training.

  9. Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Experts

    cs.NI 2025-08 conditional novelty 6.0 of 10

    A DRL router using graph attention state abstraction and QoS-aware rewards improves average QoS by up to 35.78% over four baselines in simulated edge LLM routing.

  10. BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute

    cs.LG 2025-06 conditional novelty 6.0 of 10

    A routing system that chooses both the model and the number of samples per query to meet a quality threshold, yielding up to 60% cost savings.

  11. IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theory

    cs.AI 2025-06 conditional novelty 6.0 of 10

    An IRT-based router that models each LLM's latent ability and each query's difficulty outperforms RouterBench on cost-performance reward across ID and OOD benchmarks.

  12. COSMOS: Predictable and Cost-Effective Adaptation of LLMs

    cs.LG 2025-04 conditional novelty 6.0 of 10

    COSMOS predicts the performance and cost of QLoRA fine-tuning and retrieval-augmented in-context learning across LLMs, achieving 1.09% mean absolute error and 92.72% average cost reduction on eight benchmarks.

  13. Synergistic Weak-Strong Collaboration by Aligning Preferences

    cs.AI 2025-04 conditional novelty 6.0 of 10

    Preference-tuning a weak model on whether its drafts improve a strong model's outputs makes the weak-strong pair outperform both models alone.

  14. When One LLM Drools, Multi-LLM Collaboration Rules

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A position paper that introduces a four-level taxonomy of multi-LLM collaboration (API, text, logit, weight) and argues it is essential for reliability, pluralism, and democratization.

  15. CARROT: A Cost Aware Rate Optimal Router

    stat.ML 2025-02 conditional novelty 6.0 of 10

    A plug-in router that estimates each model's per-query cost and accuracy achieves the minimax optimal excess risk for LLM routing, and CARROT plus the SPROUT dataset demonstrate the approach in practice.

  16. Rerouting LLM Routers

    cs.CR 2025-01 conditional novelty 6.0 of 10

    Adversarially optimized, query-independent token prefixes can reroute nearly all queries to the expensive strong model in both open-source and commercial LLM routers.

  17. MMFactory: A Universal Solution Search Engine for Vision-Language Tasks

    cs.CV 2024-12 conditional novelty 6.0 of 10

    MMFactory automatically generates and benchmarks a pool of reusable programmatic vision-language solutions from a few examples, letting users pick one that fits their accuracy and speed constraints.

  18. Channel Merging: Preserving Specialization for Merged Experts

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Channel Merging clusters similar channel parameters from fine-tuned LLMs and reconstructs the selected expert at inference, matching unmerged accuracy with about 53% of the ensemble's parameters.

  19. Adaptive Routing of Text-to-Image Generation Requests Between Large Cloud Model and Light-Weight Edge Model

    cs.CV 2024-11 conditional novelty 6.0 of 10

    RouteT2I is a learned router that predicts the quality gap between edge and cloud text-to-image models from the prompt alone and sends only hard prompts to the cloud.

  20. Agentic Routing: The Harness-Native Data Flywheel

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Step-level, harness-state model routing in OpenSquilla preserves agent quality at far lower cost and turns each decision into environment-labeled data for better routers and specialists.

  21. Synthetic Consumer Insight Generation with Large Language Models

    cs.AI 2026-07 conditional novelty 5.5 of 10

    Across models, prompts, and temperatures, LLM projective responses match human city-tourism themes at a coarse level while remaining stylistically more polished and compound-adjective-heavy.

  22. Task-to-Model Optimization for Enterprise LLM Coding Assistants: A Data-Driven Framework for Cost-Optimal Routing

    cs.LG 2026-08 conditional novelty 5.0 of 10

    T2MO routes coding-assistant traffic by task category and difficulty, using an expected-cost-per-completed-task criterion whose routing boundary, Eq. (2), is the minimum pass rate a cheaper model needs to justify disp...

  23. TriAgent: Divergence-Aware Multi-Agent Committees for Cost-Efficient Financial Sentiment Analysis

    cs.CL 2026-07 conditional novelty 5.0 of 10

    A divergence-routed VADER+FinBERT+LLM committee reaches ~0.87 F1 with a 1.5B critic, matching 7B with far less cost, while same-size persona voting regresses to 0.66.

  24. RouteRec: Strict Evaluation of Recommender-Agent Selection and Aggregation

    cs.CL 2026-07 conditional novelty 5.0 of 10

    On MovieLens-1M, request-level hard selection of recommender agents underperforms BM25, while item-level learned shortlist aggregation matches or beats it and improves further with gated LLM use.

  25. Codifying the Judge: Scalable Evaluation via Program Distillation

    cs.AI 2026-05 conditional novelty 5.0 of 10

    LLM judge logic can be distilled into a committee of Python scoring programs that match mid-size LLM judge accuracy on filtered preference datasets, at orders-of-magnitude higher throughput.

  26. When Models Know When They Do Not Know: Calibration, Cascading, and Cleaning

    cs.AI 2026-01 conditional novelty 5.0 of 10

    A confidence-based, training-free routing and cleaning framework improves accuracy-efficiency trade-offs and identifies mislabeled data in image and language benchmarks.

  27. LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead

    cs.AI 2025-05 conditional novelty 5.0 of 10

    LightRouter uses short preview outputs to filter a pool of LLMs down to two, then aggregates their full responses, beating ensemble baselines and matching costlier models.

  28. Universal Model Routing for Efficient LLM Inference

    cs.CL 2025-02 conditional novelty 5.0 of 10

    UniRoute represents each language model by its error rates on a few prompt clusters, letting a router choose among models it has never seen during training.

  29. KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems

    cs.AI 2025-02 reject novelty 5.0 of 10

    A knowledge-aware Bayesian bandit that selects specialist LLM experts per task is reported to beat Mixture-of-Agents on AlpacaEval 2.0 and MT-Bench while using fewer models.

  30. MixLLM: Dynamic Routing in Mixed Large Language Models

    cs.CL 2025-02 conditional novelty 5.0 of 10

    MixLLM routes each query to a cost-effective LLM by combining tag-enhanced embeddings, per-model quality and cost predictors, a latency penalty, and online bandit feedback.

  31. LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing

    cs.LG 2025-02 conditional novelty 5.0 of 10

    A preference-conditioned PPO routing policy with IRT-based model identity vectors selects cost-effective LLMs per query and generalizes to unseen models from a handful of evaluation prompts.

  32. Unsupervised Query Routing for Retrieval Augmented Generation

    cs.IR 2025-01 conditional novelty 5.0 of 10

    An unsupervised method labels queries by comparing each search engine's answer to a multi-engine 'upper-bound' answer, then trains a router on those labels.

  33. Real-time Adapting Routing (RAR): Improving Efficiency Through Continuous Learning in Software Powered by Layered Foundation Models

    cs.LG 2024-11 reject novelty 5.0 of 10

    RAR uses shadow inference plus a stored library of large-model reasoning hints to continuously shift requests from an expensive LLM to a cheaper one.

  34. CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering

    cs.CL 2025-07 reject novelty 4.0 of 10

    CoE-Ops routes DevOps questions to specialized LLM experts using an LLM classifier plus retrieval, reporting gains on DevOps-Eval that are compromised by possible test-set leakage.

  35. Economic Evaluation of LLMs

    cs.AI 2025-07 conditional novelty 4.0 of 10

    Reasoning models and single large LLMs beat cheaper alternatives on a dollar-adjusted score once the assumed price per error exceeds roughly $0.01 to $0.20, depending on latency assumptions.

  36. Position: Enough of Scaling LLMs! Lets Focus on Downscaling

    cs.CL 2025-05 conditional novelty 4.0 of 10

    The paper advocates for downscaling LLMs, derives that training CO2 scales linearly with parameters and data, and proposes a condition under which ensembles of pruned small models beat the original at equal compute.

  37. Doing More with Less: A Survey on Routing Strategies for Resource Optimisation in Large Language Model-Based Systems

    cs.AI 2025-02 conditional novelty 4.0 of 10

    A survey that classifies LLM routing strategies into pre-generation and post-generation approaches and four implementation families, framed as a performance-cost optimization problem.

  38. PickLLM: Context-Aware RL-Assisted Large Language Model Routing

    cs.LG 2024-12 conditional novelty 4.0 of 10

    A reinforcement-learning router that converges to one LLM per query session, reducing cost and latency while keeping answer quality competitive.

  39. Token-Operations-Oriented Inference Optimization Techniques for Large Models

    cs.SE 2026-06 unverdicted novelty 3.0 of 10

    The paper introduces a four-layer technical architecture for token-operations-oriented inference optimization in large models and reviews key technologies and industry status at each layer.

  40. ORI: O Routing Intelligence

    cs.CL 2025-02 reject novelty 3.0 of 10

    ORI routes queries by embedding cluster to the best model for the cluster's dominant benchmark, reporting modest gains that are not supported by its own routing rule or evaluation protocol.

  41. The Internet of Large Language Models: An Orchestration Framework for LLM Training and Knowledge Exchange Toward Artificial General Intelligence

    cs.AI 2025-01 reject novelty 2.0 of 10

    The paper proposes the Internet of LLM framework for model sharing, unified environments, agent-path optimization, and compute-sharing incentives, but presents no implementation or empirical evidence that it works.

Pith tools