Agentic Routing: The Harness-Native Data Flywheel

Hailin Hu; Hang Zhou; Kai Han; Liuyang Song; Mengyu Zheng; Runke Liu; Shuo Zhang; Siyang Cheng; Wei He; Xiang Kuang

Step-level model routing inside an agent harness can cut cost by roughly 90% while keeping quality, and every decision becomes training data for better routers and specialist models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 05:50 UTC pith:AKWKOBNY

load-bearing objection Solid cost–quality systems result on agent routing; the flywheel is a clean design story that the experiments never actually close. the 3 major comments →

arxiv 2607.11399 v1 pith:AKWKOBNY submitted 2026-07-13 cs.CL cs.AI

Agentic Routing: The Harness-Native Data Flywheel

Xinchen Liu , Hang Zhou , Yingjie Zong , Yuchuan Tian , Liuyang Song , Shuo Zhang , Yulong Li , Wei He

show 7 more authors

Mengyu Zheng Runke Liu Siyang Cheng Xiang Kuang Hailin Hu Kai Han Yunhe Wang

This is my paper

classification cs.CL cs.AI

keywords agentic routingharness-native routingmodel routingdata flywheelmulti-model ensemblecost-quality frontierarena recordsagent harness

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

AI agents no longer run as one model call; a harness manages tools, context, recovery, and verification while models specialize on different axes. This paper argues that model choice must therefore be a step-level decision conditioned on the full harness state—not a one-shot query router. In cost-saving mode the router fields one best-fit model; in high-accuracy mode it fields complementary models and aggregates them. The central claim is that this harness-native routing preserves end-to-end task quality at much lower realized cost (about 90% lower on PinchBench and about 43% lower on DRACO versus strong fixed baselines) and can also push quality past strong single models. Crucially, every routing step yields an environment-labeled arena record—state, choice, trace, outcome, cost—so routing becomes a data engine: more efficient execution yields more traces under the same budget, which train better routers and harness-native specialists that re-enter the pool.

Core claim

Harness-native agentic routing—selecting one model or a complementary model set at each execution step from the full harness state—preserves agent quality while substantially lowering realized cost, and multi-model ensembles can move the quality–cost frontier beyond strong singletons. Because outcomes and costs are labeled by the environment rather than by the router, the same decisions form a reusable arena corpus that improves future routers and specialist models.

What carries the argument

The harness-native arena record and data flywheel: each step logs query, harness state, model (set), execution trace, verification outcome, and cost; environment labels (not router self-labels) train iterated routing policies and harness-native specialists that rejoin the model pool.

Load-bearing premise

That records from earlier, imperfect routers—plus controlled exploration and off-policy correction—give enough honest coverage that later policies improve the real cost–quality frontier instead of copying old mistakes or thin labels.

What would settle it

Train later routers only on the logged arena corpus with the paper’s exploration and propensity corrections; if they fail to lower cost at equal quality (or raise quality at fixed budget) versus the open cold-start ranker on held-out agent runs with a shifted model pool, the flywheel claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Agents can run mostly on cheaper specialists and escalate only when harness state (failures, context pressure, verification risk) demands it, without rewriting the harness.
Routing logs become a primary training substrate for agent-native models—specialists for code repair, recovery, tool formatting, and verification—not only preference chat data.
Multi-model routing can substitute ensembles of cheaper complementary proposers for a single expensive frontier model and still improve quality and cost.
Open cold-start rankers can seed a proprietary policy path: code may be public while arena corpora and later high-accuracy routers remain the durable asset.
The model pool ends as three layers—general foundation models, harness-native specialists, and ultra-cheap fast paths—composed by the router.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If harness state is the right conditioning signal, the same flywheel should transfer across agent frameworks that expose comparable tool, verifier, and recovery events—even when base models change.
Verifier bias or sparse exploration would show up as routers that look cheap on logged metrics but fail when forced to use under-sampled models, a natural stress test for production deployments.
Cache-aware and token-level routing mentioned as future work would turn step-level capability matching into a tighter coupling with inference systems (KV locality, speculative decoding).
As model pools densify with many mid-tier specialists, the value of routing rises faster than raw model quality alone, shifting competition toward who owns the execution-labeled corpus.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid cost–quality systems result on agent routing; the flywheel is a clean design story that the experiments never actually close.

read the letter

The thing worth knowing is that the measured singleton and multi-model operating points look real, while the paper’s distinctive pitch—the harness-native data flywheel that trains progressively better routers—is mostly asserted, not shown.

What is new is the packaging, not the atoms. Query-level routing, cascades, MoA-style aggregation, and agent harnesses are all prior art they cite. Their contribution is step-level selection conditioned on full harness state (tools, recovery, verification, context pressure), dual singleton/ensemble regimes under one Pareto objective, a typed arena-record schema, and an open LightGBM cold-start stack in OpenSquilla. That framing is clear and useful. The formal objective (Eq. 1 and the singleton/ensemble specializations) is readable systems math, not theater. Aggregate results are the paper’s strength: on PinchBench, routed OpenSquilla holds ~93 score at ~$0.02 vs Opus at ~$0.22; on DRACO, quality is essentially matched at ~43% lower cost, and selected multi-model configs can beat Fable 5 / Opus on score at lower spend. Appendix cases are illustrative, not load-bearing.

Soft spots, in proportion. The stress-test is right: §3.4–3.5 and §5 define a staged ladder g^(0)→g^(1)→… with IPS/DR off-policy learning and anti-poisoning arguments, but no table shows a later generation beating the LightGBM seed on the true frontier under a fixed pool. So claim (i)—cost–quality allocation—is supported; claim (ii)—closed flywheel—is a plausible systems design. Other real but secondary issues: hand-selected ensembles in places, no uncertainty on aggregate scores, commercial model pools that will age, and a deliberate proprietary wall around arena data and later policies. Free parameters (λ, α, β, threshold 0.95) are deployment knobs, not hidden fitting of the main claim. Circularity burden is low; labels come from the environment.

This is for people building multi-model agent harnesses and production routers, not for pure theory. I would bring it to a systems reading group as a working design plus measured frontiers, not as a finished learning result. It deserves a serious referee: important enough in agent systems, formally grounded enough, and evidentially sharp enough on the cost–quality half. I would cite the operating points and the arena-record idea if I were writing on agent routing; I would not cite the flywheel as demonstrated multi-iteration training until someone ships g^(r) vs g^(0).

Referee Report

3 major / 0 minor

Summary. The paper proposes harness-native agentic routing: step-level selection of a single model or complementary model set conditioned on full agent harness state (observation, context, tools, verification, recovery), rather than query-level serving. It formalizes a Pareto quality–cost objective (Eq. 1), instantiates singleton capability matching and multi-model ensemble routing with a complementarity regularizer, and defines typed environment-labeled arena records that are argued to form a data flywheel for successive router and specialist training. OpenSquilla implements a four-layer stack with an open LightGBM cold-start ranker. On PinchBench and DRACO, singleton routing retains ~99.8–99.9% of strong fixed-model scores at ~90% and ~43% lower billed cost respectively; selected multi-model configurations can exceed strong singletons (e.g., Fable 5) on DRACO at lower cost. The report positions routing as both cost control and a data engine for agent-native training.

Significance. If the empirical routing results generalize, the work is a useful systems contribution: it reframes model selection as a stateful harness decision, reports large realized cost savings at near-parity quality on agentic benchmarks, and ships a concrete open cold-start router plus a typed arena-record schema. The dual-regime formulation (cost-effective singleton vs high-accuracy ensemble) and the explicit separation of action from environment-supplied labels are well motivated relative to single-turn cascades and preference-based routers. The open LightGBM seed and staged policy path are practical strengths. The distinctive flywheel claim—that logged records train progressively stronger routers and harness-native specialists—would be highly significant if closed experimentally, but that loop is not yet demonstrated; significance currently rests mainly on the single-shot cost–quality operating points and the systems framing.

major comments (3)

The title, abstract, and §5–6 present the harness-native data flywheel as a central contribution: arena records train better routers g^(r) and specialists that improve the frontier and generate more traces. §3.4 defines a staged ladder g^(0)→g^(1)→… with inverse-propensity / doubly-robust off-policy training on D^(r). The experiments (§4.2–4.3) evaluate only the LightGBM cold-start g^(0) and hand-selected or single-pass multi-model configurations. No table or figure reports a later generation that improves realized cost or quality relative to g^(0) under fixed harness and model pool. Without that measurement, claim (ii) remains a design argument; either add multi-iteration frontier results or substantially reframe the flywheel as prospective rather than demonstrated.
The paper’s distinguishing claim versus prior routing is conditioning on full harness state (tool failures, context pressure, verifier/recovery signals), not only the user query (§1, §3.1–3.2, Fig. 2). Aggregate gains vs fixed Opus and OpenRouter Auto (Tables 1–2) do not isolate which harness-state features drive the savings. An ablation that disables the context-aware refiner / risk-pricing signals (or replaces them with query-only features) under the same model pool and harness is needed to support “harness-native” over ordinary multi-model cost routing. Without it, the large PinchBench/DRACO savings could be explained by pool diversity and cheap-model availability alone.
The strongest multi-model DRACO points (Tables 3 and 5: score 60.82 / 64.09) use proposer sets fixed in advance; only Table 7 couples the agentic router to dynamic proposer assembly, and those scores (≤60.31) fall short of the selected configuration. The abstract and §4.3 present multi-model routing as an agentic routing regime. Please either (a) make the primary multi-model claims rest on fully routed ensembles with clear comparison to selected oracles, or (b) clearly separate “selected ensemble baselines” from “routed ensembles” in the abstract and conclusions so the contribution of the router is not overstated.

Circularity Check

0 steps flagged

No significant circularity: empirical cost–quality claims rest on external benchmarks and billed costs, not on self-definitional or fitted-as-prediction reductions.

full rationale

This is systems/empirical work, not a first-principles derivation. Singleton and multi-model claims are scored by external harness benchmarks (PinchBench, DRACO) and realized monetary cost against fixed strong-model baselines (Opus 4.8, Fable 5, etc.); those labels are not constructed from the router’s own parameters. Equation 1 and the singleton/multi-model decision rules (Eqs. 5, 8) are standard Pareto / scalarized objectives, not tautologies that force the reported operating points. The arena-record schema explicitly separates the routing action from environment-supplied outcome, verification, and cost labels, so later training is framed as off-policy logged-feedback learning rather than imitation of prior choices. Self-citations (OpenSquilla, related harness/bench work) describe the instantiation and related systems; they do not supply a uniqueness theorem or ansatz that makes the measured frontiers true by construction. The multi-generation flywheel (g^(0)→g^(r)) is asserted as a design path and is under-measured in the experiments—that is an evidence gap, not circularity. No step reduces a claimed prediction to a fitted input or to a self-cited uniqueness result by construction.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 4 invented entities

The central empirical claims rest on standard agent/LLM assumptions plus several free trade-off weights and a fixed-harness isolation choice. The flywheel claim adds stronger assumptions about off-policy coverage and verifier quality. Conceptual entities (arena records, four-layer stack, flywheel) are engineering constructs with partial experimental support, not new physical objects.

free parameters (6)

cost–quality weight λ
Scalarizes the Pareto objective in Eqs. 1, 5, 8; different λ define cost-saving vs accuracy-seeking regimes. Chosen by deployment, not derived.
complementarity weight α
Regularizer in multi-model objective (Eq. 8); annealed toward zero. Directly affects proposer-set selection (diversity-heavy policy in Table 7).
per-turn reward weight β
Trades immediate step reward ρ_t against trajectory loss/cost in Eqs. 5 and 8; densifies credit assignment but is a free design choice.
routing threshold 0.95
Noted for DRACO singleton runs (Table 2); gates when the ranker escalates. Hand-set operating point.
max ensemble size K
Caps |S_t| in Eq. 1; combinatorial search limit for multi-model routing.
LightGBM ranker features and hyperparameters
Cold-start matching engine (§3.2); feature engineering and boosting params are fitted/chosen but not fully specified for independent refit.

axioms (5)

domain assumption No single model dominates all operational axes (coding, long-context, tool use, latency, cost); the model pool remains heterogeneous enough that state-conditioned selection has value.
Stated throughout §1 and §5.5; if models converge, routing value shrinks though traces may still help training.
domain assumption Task loss and realized cost of full trajectories, plus environment verification, are valid labels for routing actions (not imitation of the logging policy).
Core of Eqs. 1–2 and §5.3 off-policy framing; standard contextual-bandit assumption applied to agents.
ad hoc to paper Fixing surrounding harness policies isolates gains to model/model-set allocation alone.
Explicit isolation choice in §3.1; if harness co-adaptation is the real driver, attributed gains are misallocated.
domain assumption Controlled exploration (uncertainty escalation, alternative-model replay) plus inverse-propensity/doubly-robust estimation suffice for coverage and selection-bias control.
§3.4–3.5, §5.3; load-bearing for iterative router improvement claims.
standard math Pareto minimization of expected task loss and cost is the right multi-objective for agent routing (latency optional third axis).
Eq. 1; standard multi-objective formulation, not proved unique for agents.

invented entities (4)

Harness-native agentic routing operator g(M|h_t) independent evidence
purpose: Formal step-level model or model-set selection conditioned on full harness state under a quality–cost frontier.
Central construct of §3; engineering operator, not a physical entity. Independent evidence via reported benchmarks is partial.
Typed arena record R_t (model-arena corpus) no independent evidence
purpose: Structured environment-labeled log of state, action, trace, verification, cost, and provenance for off-policy learning.
Schema in Eq. 11 / §3.5; the proprietary corpus is the claimed durable asset. Schema is specified; corpus not released.
Four-layer OpenSquilla routing stack + staged router-model path g^(0), g^(1), … independent evidence
purpose: Practical cold-start (filters + LightGBM) and iterative learned policies that start the flywheel.
§3.2–3.4 implementation path; g^(0) is open in principle, later generations are product path.
Harness-native data flywheel no independent evidence
purpose: Closed loop: base models → harness execution → routing data → specialists/routers → richer model pool.
§5 conceptual engine; long-run compounding is argued from mechanism more than multi-generation measured frontier movement.

pith-pipeline@v1.1.0-grok45 · 31261 in / 4290 out tokens · 51193 ms · 2026-07-14T05:50:50.709137+00:00 · methodology

0 comments

read the original abstract

Large language model agents are increasingly executed not by a single model call, but by an execution harness that manages observation, context, control, action, state, and verification. At the same time, frontier and open models are becoming structurally specialized: a model that is strong at code editing, long-context recovery, tool use, mathematical reasoning, or low-latency response may not dominate on the other axes. This makes model selection inside an agent a core systems problem rather than a per-query serving trick. Existing routing methods mostly optimize single-turn cost-quality trade-offs and therefore miss the execution state, intermediate failures, and feedback loops that make agents different from chat completion. We propose Harness-Native agentic routing, a step-level routing paradigm that selects either a single best-fit model for cost-effective execution or multiple complementary models for ensemble-style accuracy improvement, conditioned on the full harness state. The key insight is that every routing decision naturally produces a structured data record -- consisting of the query, harness state, model choice or model set, execution trace, outcome, and cost -- whose labels are supplied by the environment rather than by the router itself. These records form a harness-native data flywheel: execution traces train better routers and harness-native models, which improve cost-quality trade-offs and generate more traces under the same budget. We instantiate this idea in OpenSquilla with a four-layer routing stack, an open LightGBM cold-start ranker, and a staged router-model path that turns logged arena records into progressively stronger routing policies. The report studies singleton and multi-model routing on agentic benchmarks including DRACO and PinchBench, and argues that agentic routing is not merely cost control, but a data engine for agent-native training.

discussion (0)

Reference graph

Works this paper leans on

42 extracted references · 6 canonical work pages

[1]

Agarwal, P

S. Agarwal, P . Namyar, A. Wolman, R. Ambavat, A. Gupta, and Q. Zhang. Switchcraft: AI model router for agentic tool calling.arXiv preprint arXiv:2605.07112, 2026. URL https://arxiv.org/abs/2605.07112

Pith/arXiv arXiv 2026
[2]

System card: Claude opus 4.7

Anthropic. System card: Claude opus 4.7. Anthropic Technical Report, April 2026. URL https://www-cdn.anthropic.com/037f06850df7fbe871e206dad004c3db5fd50 340/Claude%20Opus%204.7%20System%20Card.pdf. Accessed: 2026-06-24

2026
[3]

L. Chen, M. Zaharia, and J. Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

Pith/arXiv arXiv 2023
[4]

Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao. Agent-FLAN: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9354–9366, Bangkok, 29 Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.fin ...

work page doi:10.18653/v1/2024.fin 2024
[5]

Z. Chen, Z. Wei, Y. Bai, X. Xiong, and J. Wu. TagRouter: Learning route to LLMs through tags for open-domain text generation tasks. InFindings of the Association for Computational Linguistics: ACL 2025, pages 21539–21564, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-acl.1110. URL https: //aclantholo...

work page doi:10.18653/v1/2025.findings-acl.1110 2025
[6]

Contributors

O. Contributors. Opensquilla — token-efficient ai agent. https://github.com/opens quilla/opensquilla, 2026. GitHub repository

2026
[7]

D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V . Ruhle, L. V . S. Lakshmanan, and A. H. Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. InInternational Conference on Learning Representations, 2024. URL https://openreview.net/forum?i d=02f3mUtqnM

2024
[8]

Dudik, J

M. Dudik, J. Langford, and L. Li. Doubly robust policy evaluation and learning.arXiv preprint arXiv:1103.4601, 2011

Pith/arXiv arXiv 2011
[9]

A. Feng, B. Srinivasan, Y. Zhou, Z. Xu, K. Zhou, S. Guan, Y. Chen, X. Wu, N. Kulkarni, Y. Zhang, Z. Shen, D. Bespalov, S. S. Mishra, Y. Teng, D. Y.-B. Wang, H. Ding, and L. L. Cheong. IPR: Intelligent prompt routing with user-controlled quality-cost trade-offs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Indu...

2025
[10]

J. Guo, Z. Hao, C. Wang, C. Fan, T. Luo, H. Li, Y. Gao, H. Mei, J. Peng, R. Xu, et al. From question answering to task completion: A survey on agent system and harness design. arXiv preprint arXiv:2606.20683, 2026

Pith/arXiv arXiv 2026
[12]

URLhttps://arxiv.org/abs/2403.12031

Pith/arXiv arXiv
[13]

Jiang, X

D. Jiang, X. Ren, and B. Y. Lin. LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-l ...

work page doi:10.18653/v1/2023.acl-l 2023
[14]

G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: A highly efficient gradient boosting decision tree. InAdvances in Neural Information Processing Systems, 2017

2017
[15]

H. Li, Y. Zhang, Z. Guo, C. Wang, S. Tang, Q. Zhang, Y. Chen, B. Qi, P . Ye, L. Bai, Z. Wang, and S. Hu. LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. InFindings of the Association for Computational Linguistics: ACL 2026, pages 37733– 37754, San Diego, California, United States, July 2026. Association for Computational Linguis...

work page doi:10.18653/v1/2026.findings-acl.1881 2026
[16]

Mohammadshahi, A

A. Mohammadshahi, A. R. Shaikh, and M. Yazdani. Routoo: Learning to route to large language models effectively.arXiv preprint arXiv:2401.13979, 2024. URL https://arxiv. org/abs/2401.13979

Pith/arXiv arXiv 2024
[17]

Moslem and J

Y. Moslem and J. D. Kelleher. Dynamic model routing and cascading for efficient llm inference: A survey.arXiv preprint arXiv:2603.04445, 2026

Pith/arXiv arXiv 2026
[18]

Hermes-agent

Nous Research. Hermes-agent. https://github.com/NousResearch/hermes-agent,
[19]

Hermes agent: Mixture of agents

Nous Research. Hermes agent: Mixture of agents. https://hermes-agent.nousresea rch.com/docs/user-guide/features/mixture-of-agents, 2026. Documentation

2026
[20]

Okamoto, A

M. Okamoto, A. K. Erol, and M. Riedl. Explainable model routing for agentic workflows. arXiv preprint arXiv:2604.03527, 2026. URLhttps://arxiv.org/abs/2604.03527

Pith/arXiv arXiv 2026
[21]

I. Ong, A. Almahairi, V . Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica. Routellm: Learning to route llms with preference data.arXiv preprint arXiv:2406.18665, 2024

Pith/arXiv arXiv 2024
[22]

Panda, R

P . Panda, R. Magazine, C. Devaguptapu, S. Takemori, and V . Sharma. Adaptive LLM routing under budget constraints. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 23934–23949, Suzhou, China, Nov. 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-emnlp.1301. URL https://aclanthology .org/2025.fi...

work page doi:10.18653/v1/2025.findings-emnlp.1301 2025
[23]

Šakota, M

M. Šakota, M. Peyrard, and R. West. Fly-swat or cannon? cost-effective language model choice via meta-modeling. InProceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM ’24, pages 606–615, 2024. doi: 10.1145/3616855.3635825. URLhttps://doi.org/10.1145/3616855.3635825

doi:10.1145/3616855.3635825 2024
[24]

Schick, J

T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Yacmpz84TH

2023
[25]

Shinn, F

N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, volume 36, 2023. URL https://papers.nips.cc/paper_files/paper/2023/hash /1b44b878bb782e6954cd888628510e90-Abstract-Conference.html

2023
[27]

URLhttps://arxiv.org/abs/2309.15789

Pith/arXiv arXiv
[28]

Singh, A

A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

Pith/arXiv arXiv 2025
[29]

Steinberger and OpenClaw Contributors

P . Steinberger and OpenClaw Contributors. Openclaw — personal ai assistant. https: //github.com/openclaw/openclaw, 2026. GitHub repository

2026
[30]

C. Tran, S. Paracha, A. Hafeez, and S. Chen. Arch-router: Aligning LLM routing with human preferences.arXiv preprint arXiv:2506.16655, 2025. URL https://arxiv.org/ab s/2506.16655. 31

Pith/arXiv arXiv 2025
[31]

V . Trivedy. The anatomy of an agent harness.https://www.langchain.com/blog/the -anatomy-of-an-agent-harness, 2026. LangChain blog post

2026
[32]

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

Pith/arXiv arXiv 2023
[33]

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=ehfRiF0R 3a

2024
[34]

J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Y. Zou. Mixture-of-agents enhances large language model capabilities. InInternational Conference on Learning Representations,
[35]

URLhttps://openreview.net/forum?id=h0ZfDIrj7T
[36]

J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE- agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, volume 37, pages 50528–50652, 2024. URL https: //papers.nips.cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc523 0172e1e7c-Abstract-Conference.html

2024
[37]

P . Yang, W. Chen, T. Yang, P . Feng, J. Xing, W. Guo, Y. Yao, Y. Han, H. Li, and X. Wang. Twinrouterbench: Fast static and live dynamic evaluation for realistic agentic llm routing. arXiv preprint arXiv:2605.18859, 2026

Pith/arXiv arXiv 2026
[38]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

Pith/arXiv arXiv 2022
[39]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=WE_vluYUL-X

2023
[40]

Y. Yue, G. Zhang, B. Liu, G. Wan, K. Wang, D. Cheng, and Y. Qi. MasRouter: Learning to route LLMs for multi-agent systems. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15549–15572, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-l o...

work page doi:10.18653/v1/2025.acl-l 2025
[41]

A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang. AgentTuning: En- abling generalized agent abilities for LLMs. InFindings of the Association for Computa- tional Linguistics: ACL 2024, pages 3053–3077, Bangkok, Thailand, Aug. 2024. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.181. URL https://aclanthology.or...

doi:10.18653/v1/2024.findings-acl.181 2024
[42]

Zheng, K

M. Zheng, K. Han, B. Li, H. Xu, Y. Tian, W. He, H. Zhou, J. Guo, H. Hu, L. Ma, C. Xu, G. Dai, L. Xia, Y. Wei, Y. Wang, and W. Yu. Claw-swe-bench: A benchmark for evaluating openclaw-style agent harnesses on coding tasks.arXiv preprint arXiv:2606.12344, 2026

Pith/arXiv arXiv 2026
[43]

Zhong, H

J. Zhong, H. Zhang, C. Southern, J. Yang, T. Wang, K. Jung, S. Zhang, D. Yarats, J. Ho, and J. Ma. Draco: A cross-domain benchmark for deep research accuracy, completeness, and objectivity.arXiv preprint arXiv:2602.11685, 2026. 32

arXiv 2026
[44]

P . Zhou, Z. Tang, Y. Ma, J. Tang, Y. Han, Z. Wan, F. Meng, W. Wang, B. Zhuang, W. Zhao, and Y. Yang. Agent-as-a-router: Agentic model routing for coding tasks.arXiv preprint arXiv:2606.22902, 2026. 33

Pith/arXiv arXiv 2026

[1] [1]

Agarwal, P

S. Agarwal, P . Namyar, A. Wolman, R. Ambavat, A. Gupta, and Q. Zhang. Switchcraft: AI model router for agentic tool calling.arXiv preprint arXiv:2605.07112, 2026. URL https://arxiv.org/abs/2605.07112

Pith/arXiv arXiv 2026

[2] [2]

System card: Claude opus 4.7

Anthropic. System card: Claude opus 4.7. Anthropic Technical Report, April 2026. URL https://www-cdn.anthropic.com/037f06850df7fbe871e206dad004c3db5fd50 340/Claude%20Opus%204.7%20System%20Card.pdf. Accessed: 2026-06-24

2026

[3] [3]

L. Chen, M. Zaharia, and J. Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176, 2023

Pith/arXiv arXiv 2023

[4] [4]

Z. Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, K. Chen, and F. Zhao. Agent-FLAN: Designing data and methods of effective agent tuning for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9354–9366, Bangkok, 29 Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.fin ...

work page doi:10.18653/v1/2024.fin 2024

[5] [5]

Z. Chen, Z. Wei, Y. Bai, X. Xiong, and J. Wu. TagRouter: Learning route to LLMs through tags for open-domain text generation tasks. InFindings of the Association for Computational Linguistics: ACL 2025, pages 21539–21564, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-acl.1110. URL https: //aclantholo...

work page doi:10.18653/v1/2025.findings-acl.1110 2025

[6] [6]

Contributors

O. Contributors. Opensquilla — token-efficient ai agent. https://github.com/opens quilla/opensquilla, 2026. GitHub repository

2026

[7] [7]

D. Ding, A. Mallick, C. Wang, R. Sim, S. Mukherjee, V . Ruhle, L. V . S. Lakshmanan, and A. H. Awadallah. Hybrid LLM: Cost-efficient and quality-aware query routing. InInternational Conference on Learning Representations, 2024. URL https://openreview.net/forum?i d=02f3mUtqnM

2024

[8] [8]

Dudik, J

M. Dudik, J. Langford, and L. Li. Doubly robust policy evaluation and learning.arXiv preprint arXiv:1103.4601, 2011

Pith/arXiv arXiv 2011

[9] [9]

A. Feng, B. Srinivasan, Y. Zhou, Z. Xu, K. Zhou, S. Guan, Y. Chen, X. Wu, N. Kulkarni, Y. Zhang, Z. Shen, D. Bespalov, S. S. Mishra, Y. Teng, D. Y.-B. Wang, H. Ding, and L. L. Cheong. IPR: Intelligent prompt routing with user-controlled quality-cost trade-offs. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Indu...

2025

[10] [10]

J. Guo, Z. Hao, C. Wang, C. Fan, T. Luo, H. Li, Y. Gao, H. Mei, J. Peng, R. Xu, et al. From question answering to task completion: A survey on agent system and harness design. arXiv preprint arXiv:2606.20683, 2026

Pith/arXiv arXiv 2026

[11] [12]

URLhttps://arxiv.org/abs/2403.12031

Pith/arXiv arXiv

[12] [13]

Jiang, X

D. Jiang, X. Ren, and B. Y. Lin. LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-l ...

work page doi:10.18653/v1/2023.acl-l 2023

[13] [14]

G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightgbm: A highly efficient gradient boosting decision tree. InAdvances in Neural Information Processing Systems, 2017

2017

[14] [15]

H. Li, Y. Zhang, Z. Guo, C. Wang, S. Tang, Q. Zhang, Y. Chen, B. Qi, P . Ye, L. Bai, Z. Wang, and S. Hu. LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing. InFindings of the Association for Computational Linguistics: ACL 2026, pages 37733– 37754, San Diego, California, United States, July 2026. Association for Computational Linguis...

work page doi:10.18653/v1/2026.findings-acl.1881 2026

[15] [16]

Mohammadshahi, A

A. Mohammadshahi, A. R. Shaikh, and M. Yazdani. Routoo: Learning to route to large language models effectively.arXiv preprint arXiv:2401.13979, 2024. URL https://arxiv. org/abs/2401.13979

Pith/arXiv arXiv 2024

[16] [17]

Moslem and J

Y. Moslem and J. D. Kelleher. Dynamic model routing and cascading for efficient llm inference: A survey.arXiv preprint arXiv:2603.04445, 2026

Pith/arXiv arXiv 2026

[17] [18]

Hermes-agent

Nous Research. Hermes-agent. https://github.com/NousResearch/hermes-agent,

[18] [19]

Hermes agent: Mixture of agents

Nous Research. Hermes agent: Mixture of agents. https://hermes-agent.nousresea rch.com/docs/user-guide/features/mixture-of-agents, 2026. Documentation

2026

[19] [20]

Okamoto, A

M. Okamoto, A. K. Erol, and M. Riedl. Explainable model routing for agentic workflows. arXiv preprint arXiv:2604.03527, 2026. URLhttps://arxiv.org/abs/2604.03527

Pith/arXiv arXiv 2026

[20] [21]

I. Ong, A. Almahairi, V . Wu, W.-L. Chiang, T. Wu, J. E. Gonzalez, M. W. Kadous, and I. Stoica. Routellm: Learning to route llms with preference data.arXiv preprint arXiv:2406.18665, 2024

Pith/arXiv arXiv 2024

[21] [22]

Panda, R

P . Panda, R. Magazine, C. Devaguptapu, S. Takemori, and V . Sharma. Adaptive LLM routing under budget constraints. InFindings of the Association for Computational Linguistics: EMNLP 2025, pages 23934–23949, Suzhou, China, Nov. 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.findings-emnlp.1301. URL https://aclanthology .org/2025.fi...

work page doi:10.18653/v1/2025.findings-emnlp.1301 2025

[22] [23]

Šakota, M

M. Šakota, M. Peyrard, and R. West. Fly-swat or cannon? cost-effective language model choice via meta-modeling. InProceedings of the 17th ACM International Conference on Web Search and Data Mining, WSDM ’24, pages 606–615, 2024. doi: 10.1145/3616855.3635825. URLhttps://doi.org/10.1145/3616855.3635825

doi:10.1145/3616855.3635825 2024

[23] [24]

Schick, J

T. Schick, J. Dwivedi-Yu, R. Dessi, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. InThirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Yacmpz84TH

2023

[24] [25]

Shinn, F

N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, volume 36, 2023. URL https://papers.nips.cc/paper_files/paper/2023/hash /1b44b878bb782e6954cd888628510e90-Abstract-Conference.html

2023

[25] [27]

URLhttps://arxiv.org/abs/2309.15789

Pith/arXiv arXiv

[26] [28]

Singh, A

A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. Openai gpt-5 system card.arXiv preprint arXiv:2601.03267, 2025

Pith/arXiv arXiv 2025

[27] [29]

Steinberger and OpenClaw Contributors

P . Steinberger and OpenClaw Contributors. Openclaw — personal ai assistant. https: //github.com/openclaw/openclaw, 2026. GitHub repository

2026

[28] [30]

C. Tran, S. Paracha, A. Hafeez, and S. Chen. Arch-router: Aligning LLM routing with human preferences.arXiv preprint arXiv:2506.16655, 2025. URL https://arxiv.org/ab s/2506.16655. 31

Pith/arXiv arXiv 2025

[29] [31]

V . Trivedy. The anatomy of an agent harness.https://www.langchain.com/blog/the -anatomy-of-an-agent-harness, 2026. LangChain blog post

2026

[30] [32]

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

Pith/arXiv arXiv 2023

[31] [33]

G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models.Transactions on Machine Learning Research, 2024. URL https://openreview.net/forum?id=ehfRiF0R 3a

2024

[32] [34]

J. Wang, J. Wang, B. Athiwaratkun, C. Zhang, and J. Y. Zou. Mixture-of-agents enhances large language model capabilities. InInternational Conference on Learning Representations,

[33] [35]

URLhttps://openreview.net/forum?id=h0ZfDIrj7T

[34] [36]

J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. SWE- agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, volume 37, pages 50528–50652, 2024. URL https: //papers.nips.cc/paper_files/paper/2024/hash/5a7c947568c1b1328ccc523 0172e1e7c-Abstract-Conference.html

2024

[35] [37]

P . Yang, W. Chen, T. Yang, P . Feng, J. Xing, W. Guo, Y. Yao, Y. Han, H. Li, and X. Wang. Twinrouterbench: Fast static and live dynamic evaluation for realistic agentic llm routing. arXiv preprint arXiv:2605.18859, 2026

Pith/arXiv arXiv 2026

[36] [38]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022

Pith/arXiv arXiv 2022

[37] [39]

S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations, 2023. URLhttps://openreview.net/forum?id=WE_vluYUL-X

2023

[38] [40]

Y. Yue, G. Zhang, B. Liu, G. Wan, K. Wang, D. Cheng, and Y. Qi. MasRouter: Learning to route LLMs for multi-agent systems. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15549–15572, Vienna, Austria, July 2025. Association for Computational Linguistics. doi: 10.18653/v1/2025.acl-l o...

work page doi:10.18653/v1/2025.acl-l 2025

[39] [41]

A. Zeng, M. Liu, R. Lu, B. Wang, X. Liu, Y. Dong, and J. Tang. AgentTuning: En- abling generalized agent abilities for LLMs. InFindings of the Association for Computa- tional Linguistics: ACL 2024, pages 3053–3077, Bangkok, Thailand, Aug. 2024. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.181. URL https://aclanthology.or...

doi:10.18653/v1/2024.findings-acl.181 2024

[40] [42]

Zheng, K

M. Zheng, K. Han, B. Li, H. Xu, Y. Tian, W. He, H. Zhou, J. Guo, H. Hu, L. Ma, C. Xu, G. Dai, L. Xia, Y. Wei, Y. Wang, and W. Yu. Claw-swe-bench: A benchmark for evaluating openclaw-style agent harnesses on coding tasks.arXiv preprint arXiv:2606.12344, 2026

Pith/arXiv arXiv 2026

[41] [43]

Zhong, H

J. Zhong, H. Zhang, C. Southern, J. Yang, T. Wang, K. Jung, S. Zhang, D. Yarats, J. Ho, and J. Ma. Draco: A cross-domain benchmark for deep research accuracy, completeness, and objectivity.arXiv preprint arXiv:2602.11685, 2026. 32

arXiv 2026

[42] [44]

P . Zhou, Z. Tang, Y. Ma, J. Tang, Y. Han, Z. Wan, F. Meng, W. Wang, B. Zhuang, W. Zhao, and Y. Yang. Agent-as-a-router: Agentic model routing for coding tasks.arXiv preprint arXiv:2606.22902, 2026. 33

Pith/arXiv arXiv 2026