{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:R2R7U6PQDVZ5DOTL4W5K6EMKN7","short_pith_number":"pith:R2R7U6PQ","schema_version":"1.0","canonical_sha256":"8ea3fa79f01d73d1ba6be5baaf118a6ff00aa633fb76013072f0dec967846623","source":{"kind":"arxiv","id":"2403.12031","version":2},"attestation_state":"computed","paper":{"title":"RouterBench: A Benchmark for Multi-LLM Routing System","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"RouterBench supplies a benchmark and over 405k inference results to evaluate systems that route queries across multiple LLMs.","cross_cats":["cs.AI"],"primary_cat":"cs.LG","authors_text":"Benjamin Keigwin, Gaurav Ranganath, Jacob Bieker, Kurt Keutzer, Nan Jiang, Qitian Jason Hu, Shriyash Kaustubh Upadhyay, Xiuyu Li","submitted_at":"2024-03-18T17:59:04Z","abstract_excerpt":"As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. Yet, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To bridge th"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2403.12031","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.LG","submitted_at":"2024-03-18T17:59:04Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"c7fc028c8c3dbff34ae1bd9f8a30b5c54291e07cb78f6951b382744d222528bd","abstract_canon_sha256":"bda8039c94f4424c5e347f0074762cab29d9678cf0818500f158041527c5c654"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:48.141231Z","signature_b64":"qTHO8m9dgiOp+qB6UkQIS9HWSBfc5D0qD02e2PjzLXRVRGZkv4r3TUZ+g3YERIPZ5mqWyWeBujewBosajU5BBw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"8ea3fa79f01d73d1ba6be5baaf118a6ff00aa633fb76013072f0dec967846623","last_reissued_at":"2026-05-17T23:38:48.140645Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:48.140645Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"RouterBench: A Benchmark for Multi-LLM Routing System","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"RouterBench supplies a benchmark and over 405k inference results to evaluate systems that route queries across multiple LLMs.","cross_cats":["cs.AI"],"primary_cat":"cs.LG","authors_text":"Benjamin Keigwin, Gaurav Ranganath, Jacob Bieker, Kurt Keutzer, Nan Jiang, Qitian Jason Hu, Shriyash Kaustubh Upadhyay, Xiuyu Li","submitted_at":"2024-03-18T17:59:04Z","abstract_excerpt":"As the range of applications for Large Language Models (LLMs) continues to grow, the demand for effective serving solutions becomes increasingly critical. Despite the versatility of LLMs, no single model can optimally address all tasks and applications, particularly when balancing performance with cost. This limitation has led to the development of LLM routing systems, which combine the strengths of various models to overcome the constraints of individual LLMs. Yet, the absence of a standardized benchmark for evaluating the performance of LLM routers hinders progress in this area. To bridge th"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The chosen tasks, models, and inference outcomes in the 405k dataset are sufficiently representative of real-world usage patterns and future models to make the benchmark generalizable.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"RouterBench supplies a standardized benchmark, 405k+ inference dataset, theoretical framework, and comparative analysis for multi-LLM routing systems.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"RouterBench supplies a benchmark and over 405k inference results to evaluate systems that route queries across multiple LLMs.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"65c51edcf555b3810acd52be277621715afc3535c789b593fffe6a3073315abe"},"source":{"id":"2403.12031","kind":"arxiv","version":2},"verdict":{"id":"c0a1f036-fac3-4bc0-85a6-ca4041050dcb","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T10:43:52.460809Z","strongest_claim":"We present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies.","one_line_summary":"RouterBench supplies a standardized benchmark, 405k+ inference dataset, theoretical framework, and comparative analysis for multi-LLM routing systems.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The chosen tasks, models, and inference outcomes in the 405k dataset are sufficiently representative of real-world usage patterns and future models to make the benchmark generalizable.","pith_extraction_headline":"RouterBench supplies a benchmark and over 405k inference results to evaluate systems that route queries across multiple LLMs."},"references":{"count":110,"sample":[{"doi":"","year":2024,"title":"2024 , journal =","work_id":"1cef5e88-d6f5-4fef-b8a9-15e89f3ebe83","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"The Twelfth International Conference on Learning Representations , year=","work_id":"8299efba-a1bc-46f2-939b-57c0bc277e6c","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","work_id":"ed240a10-5b19-406c-baa5-30803f465785","ref_index":4,"cited_arxiv_id":"1810.04805","is_internal_anchor":true},{"doi":"","year":2000,"title":"Langley , title =","work_id":"6cd283dc-0548-45e9-af07-6bc1005593ad","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":1980,"title":"T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980","work_id":"6b09bca6-ef5d-4a83-8c8e-219f23cbd761","ref_index":6,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":110,"snapshot_sha256":"d58bb57c5da0012c18249ad284b083c1ce58a7c131778c95df4c5321b7bc21ed","internal_anchors":16},"formal_canon":{"evidence_count":2,"snapshot_sha256":"9e85b8316895179fdbed5bb389d11b9ac1d93fa7e95c70761bf3fc7d95be2d2a"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2403.12031","created_at":"2026-05-17T23:38:48.140741+00:00"},{"alias_kind":"arxiv_version","alias_value":"2403.12031v2","created_at":"2026-05-17T23:38:48.140741+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2403.12031","created_at":"2026-05-17T23:38:48.140741+00:00"},{"alias_kind":"pith_short_12","alias_value":"R2R7U6PQDVZ5","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"R2R7U6PQDVZ5DOTL","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"R2R7U6PQ","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":28,"internal_anchor_count":28,"sample":[{"citing_arxiv_id":"2605.18859","citing_title":"TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2502.18036","citing_title":"Harnessing Multiple Large Language Models: A Survey on LLM Ensemble","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2505.12601","citing_title":"Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18796","citing_title":"UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16604","citing_title":"R2V Agent: Teaching SLMs When to Ask for Help","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18859","citing_title":"TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17110","citing_title":"Capturing LLM Capabilities via Evidence-Calibrated Query Clustering","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.19099","citing_title":"DecisionBench: A Benchmark for Emergent Delegation in Long-Horizon Agentic Workflows","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2506.09749","citing_title":"Large Language Models for Combinatorial Optimization of Design Structure Matrix","ref_index":56,"is_internal_anchor":true},{"citing_arxiv_id":"2509.24814","citing_title":"A Greedy PDE Router for Blending Neural Operators and Classical Methods","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2602.10139","citing_title":"Anonymization-Enhanced Privacy Protection for Mobile GUI Agents: Available but Invisible","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2605.14241","citing_title":"Latency-Quality Routing for Functionally Equivalent Tools in LLM Agents","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11920","citing_title":"Domain Restriction via Multi SAE Layer Transitions","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12001","citing_title":"CR^2: Cost-Aware Risk-Controlled Routing for Wireless Device-Edge LLM Inference","ref_index":48,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11301","citing_title":"LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11603","citing_title":"GAR: Carbon-Aware Routing for LLM Inference via Constrained Optimization","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2605.10805","citing_title":"Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09588","citing_title":"Efficient Ensemble Selection from Binary and Pairwise Feedback","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2406.18665","citing_title":"RouteLLM: Learning to Route LLMs with Preference Data","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.01214","citing_title":"Agentic AI Systems Should Be Designed as Marginal Token Allocators","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2605.00180","citing_title":"RouteProfile: Elucidating the Design Space of LLM Profiles for Routing","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2605.07075","citing_title":"ModelLens: Finding the Best for Your Task from Myriads of Models","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2604.06296","citing_title":"AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2604.10907","citing_title":"RouterWise: Joint Resource Allocation and Routing for Latency-Aware Multi-Model LLM Serving","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2604.13757","citing_title":"Rethinking AI Hardware: A Three-Layer Cognitive Architecture for Autonomous Agents","ref_index":6,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7","json":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7.json","graph_json":"https://pith.science/api/pith-number/R2R7U6PQDVZ5DOTL4W5K6EMKN7/graph.json","events_json":"https://pith.science/api/pith-number/R2R7U6PQDVZ5DOTL4W5K6EMKN7/events.json","paper":"https://pith.science/paper/R2R7U6PQ"},"agent_actions":{"view_html":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7","download_json":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7.json","view_paper":"https://pith.science/paper/R2R7U6PQ","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2403.12031&json=true","fetch_graph":"https://pith.science/api/pith-number/R2R7U6PQDVZ5DOTL4W5K6EMKN7/graph.json","fetch_events":"https://pith.science/api/pith-number/R2R7U6PQDVZ5DOTL4W5K6EMKN7/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7/action/timestamp_anchor","attest_storage":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7/action/storage_attestation","attest_author":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7/action/author_attestation","sign_citation":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7/action/citation_signature","submit_replication":"https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7/action/replication_record"}},"created_at":"2026-05-17T23:38:48.140741+00:00","updated_at":"2026-05-17T23:38:48.140741+00:00"}