Pith Number

pith:R2R7U6PQ

pith:2024:R2R7U6PQDVZ5DOTL4W5K6EMKN7

not attested not anchored not stored refs resolved

RouterBench: A Benchmark for Multi-LLM Routing System

Benjamin Keigwin, Gaurav Ranganath, Jacob Bieker, Kurt Keutzer, Nan Jiang, Qitian Jason Hu, Shriyash Kaustubh Upadhyay, Xiuyu Li

RouterBench supplies a benchmark and over 405k inference results to evaluate systems that route queries across multiple LLMs.

arxiv:2403.12031 v2 · 2024-03-18 · cs.LG · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{R2R7U6PQDVZ5DOTL4W5K6EMKN7}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

We present RouterBench, a novel evaluation framework designed to systematically assess the efficacy of LLM routing systems, along with a comprehensive dataset comprising over 405k inference outcomes from representative LLMs to support the development of routing strategies.

C2weakest assumption

The chosen tasks, models, and inference outcomes in the 405k dataset are sufficiently representative of real-world usage patterns and future models to make the benchmark generalizable.

C3one line summary

RouterBench supplies a standardized benchmark, 405k+ inference dataset, theoretical framework, and comparative analysis for multi-LLM routing systems.

References

110 extracted · 110 resolved · 16 Pith anchors

[1] 2024 , journal = 2024

[2] The Twelfth International Conference on Learning Representations , year=

[4] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · arXiv:1810.04805

[5] Langley , title = 2000

[6] T. M. Mitchell. The Need for Biases in Learning Generalizations. 1980 1980

Formal links

2 machine-checked theorem links

Cited by

28 papers in Pith

TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing

Harnessing Multiple Large Language Models: A Survey on LLM Ensemble

Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers

UCCI: Calibrated Uncertainty for Cost-Optimal LLM Cascade Routing

R2V Agent: Teaching SLMs When to Ask for Help

Receipt and verification

First computed	2026-05-17T23:38:48.140645Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

8ea3fa79f01d73d1ba6be5baaf118a6ff00aa633fb76013072f0dec967846623

Aliases

arxiv: 2403.12031 · arxiv_version: 2403.12031v2 · doi: 10.48550/arxiv.2403.12031 · pith_short_12: R2R7U6PQDVZ5 · pith_short_16: R2R7U6PQDVZ5DOTL · pith_short_8: R2R7U6PQ

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/R2R7U6PQDVZ5DOTL4W5K6EMKN7 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 8ea3fa79f01d73d1ba6be5baaf118a6ff00aa633fb76013072f0dec967846623

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "bda8039c94f4424c5e347f0074762cab29d9678cf0818500f158041527c5c654",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2024-03-18T17:59:04Z",
    "title_canon_sha256": "c7fc028c8c3dbff34ae1bd9f8a30b5c54291e07cb78f6951b382744d222528bd"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2403.12031",
    "kind": "arxiv",
    "version": 2
  }
}