{"id":"dab5f3db-c485-4ff0-bb19-16600aecb6d1","arxiv_id":"2605.18859","paper_version":2,"verdict":"ACCEPT","confidence":"LOW","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"TwinRouterBench supplies 970 execution-verified router prefixes across five datasets plus a live harness for 100 held-out SWE-bench cases, scoring routers on tier accuracy, trajectory success, and realized token cost without LLM judges.","lead":"The paper introduces TwinRouterBench, a two-track benchmark for testing LLM routers on multi-step agent tasks using prefixes from real trajectories and live execution on SWE-bench. It enables offline iteration with deterministic scoring followed by end-to-end cost and success validation.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Downgrade-and-cascade tier labels may lack stability outside the 520-instance construction set","rationale":"The reader's weakest assumption directly identifies the same point. Because the benchmark's value rests on the reliability of those tiers for both tracks, confirming stability on held-out data is the minimal check that would either clear or qualify the ACCEPT verdict.","tokens_in":1763,"tokens_out":287,"duration_ms":16274,"concrete_test":"Re-apply the exact downgrade-and-cascade procedure (as released in the repo) to a fresh 100-prefix sample drawn from the unused portion of SWE-bench Verified; compute Cohen's kappa between the new labels and the original labels on the overlapping prefixes; if kappa < 0.85 the tier assignment is not stable enough to support the headline evaluation claim.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that each prefix has an execution-verified target tier usable for deterministic router scoring. The protocol constructs these labels by attempting cheaper models and cascading on failure, but the manuscript provides no held-out validation set, no inter-run reproducibility statistics, and no sensitivity analysis to the exact cascade rules or model pool ordering. If label noise exceeds a few percent or if the protocol overfits the observed trajectories, routers tuned on the static track will report inflated performance that fails to transfer to the dynamic track or new tasks.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces TwinRouterBench, a step-level benchmark for LLM routing in agentic workflows. The static track supplies 970 router-visible prefixes drawn from 520 instances across SWE-bench, BFCL, mtRAG, QMSum, and PinchBench; each prefix is paired with an execution-verified target tier produced by a released downgrade-and-cascade protocol. Scoring is performed by deterministic arithmetic over tier labels, trajectory membership, and token costs with no online LLM judge. The dynamic track supplies a harness that executes routers on the full 500-case SWE-bench Verified suite (reporting results on a 100-case held-out set disjoint from the static SWE supervision split), measuring official task resolution and realized API spend under live execution with a locked model pool.","tokens_in":1860,"tokens_out":527,"duration_ms":34196,"significance":"If the tier labels are stable, the benchmark supplies a useful advance by enabling fast offline router iteration on execution-verified step-level targets followed by end-to-end live validation. Strengths include the released downgrade protocol, deterministic arithmetic scoring, absence of evaluator-side LLM judges, and open code/data release. These features directly address documented shortcomings of one-shot routing benchmarks and could accelerate work on cost-efficient routing for long-horizon agents.","major_comments":[{"comment":"Abstract: The central claim that each prefix carries an 'execution-verified target tier' rests on the downgrade-and-cascade protocol applied to the 520-instance construction set. No held-out validation set, inter-run reproducibility statistics, or sensitivity analysis to cascade ordering or model pool is reported. This is load-bearing for both tracks, because label noise or overfitting would inflate static-track router scores and undermine transfer to the dynamic track or new tasks.","section":"Abstract"},{"comment":"Abstract (dynamic track paragraph): The 100-case held-out evaluation is described as disjoint from the static SWE supervision split, yet the manuscript does not state whether the tier labels constructed on the static set are used to score dynamic runs or whether success is measured solely by official task resolution. Clarification is required to confirm that the two tracks provide independent validation of the routing protocol.","section":"Abstract"}],"minor_comments":[{"comment":"Abstract: The size and identity of the locked model pool used for both tracks and for the downgrade protocol are not stated; adding these details would improve immediate usability of the benchmark description.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments on TwinRouterBench. We address each major comment below with clarifications from the manuscript and indicate revisions where the presentation can be strengthened.","responses":[{"response":"The downgrade-and-cascade protocol produces execution-verified labels by construction: each candidate tier is tested by actually executing the prefix with the downgraded model and confirming whether the downstream trajectory succeeds or requires a cascade. The full protocol, model pool, and ordering are released with the benchmark to enable external verification. We agree that the manuscript would be strengthened by reporting inter-run reproducibility (e.g., label stability across repeated protocol runs) and sensitivity to cascade ordering and model pool. We will add these analyses to the revised version, using the released code to compute them on the construction set.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The central claim that each prefix carries an 'execution-verified target tier' rests on the downgrade-and-cascade protocol applied to the 520-instance construction set. No held-out validation set, inter-run reproducibility statistics, or sensitivity analysis to cascade ordering or model pool is reported. This is load-bearing for both tracks, because label noise or overfitting would inflate static-track router scores and undermine transfer to the dynamic track or new tasks."},{"response":"The dynamic track evaluates routers under live execution on the 100-case held-out SWE-bench Verified subset. Success is measured exclusively by official task resolution rate and realized API spend; the static tier labels are not used to score or supervise the dynamic runs. The held-out set is disjoint from the static SWE supervision split by design, ensuring the dynamic track supplies independent end-to-end validation. We will revise the abstract and dynamic-track section to state this explicitly.","revision_made":"yes","referee_comment":"[Abstract] Abstract (dynamic track paragraph): The 100-case held-out evaluation is described as disjoint from the static SWE supervision split, yet the manuscript does not state whether the tier labels constructed on the static set are used to score dynamic runs or whether success is measured solely by official task resolution. Clarification is required to confirm that the two tracks provide independent validation of the routing protocol."}],"tokens_in":1525,"tokens_out":476,"duration_ms":24093,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main contribution is a benchmark that moves router evaluation past one-shot prompts. It supplies 970 router-visible prefixes drawn from actual agent trajectories across SWE-bench, BFCL and a few other sets, each labeled with an execution-verified tier produced by a released downgrade-and-cascade procedure. Scoring uses simple arithmetic on tier, trajectory membership and token cost, with no LLM judge at evaluation time. A second track runs routers live on a 100-case held-out slice of SWE-bench Verified and records both task success and real API spend. Code and data are public. That combination is new relative to prior router benchmarks I have seen. The static track lets people iterate quickly; the dynamic track gives an end-to-end check under live execution. Both are useful for anyone working on tiered routing inside long-horizon agents. The soft spot is the tier labels themselves. The downgrade protocol constructs the target tiers on the same 520 instances used for the static track, and the manuscript does not appear to include a separate validation set, inter-run reproducibility numbers, or sensitivity tests on cascade order or model pool. If those labels shift with small changes in the protocol, routers tuned on the static track could overstate performance when moved to new tasks or the dynamic track. The held-out dynamic set helps, but it does not directly test label stability. This work is aimed at researchers and practitioners building routers for coding agents and similar systems. It is concrete enough and the artifacts are public enough that a serious editor should send it to referees rather than desk-reject it; the label-construction details will need scrutiny in review, but the overall framing addresses a real gap.","headline":"TwinRouterBench adds step-level prefixes from agent trajectories plus a live dynamic SWE-bench track with deterministic scoring, but the downgrade protocol for target tiers lacks reported stability checks.","tokens_in":2384,"tokens_out":411,"would_cite":false,"duration_ms":20512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"TwinRouterBench is an empirical LLM routing benchmark with no structural overlap to RS cost or distinction-forcing machinery","alignment":"orthogonal","rationale":"The paper's central machinery (downgrade-and-cascade tier labeling, static/dynamic tracks, deterministic arithmetic scoring over ROWPASS/TRAJPASS/COSTSAVE, prefix-based conditional routing π: x_i → t_i) is a practical ML evaluation protocol for agentic LLM cost optimization. It has zero intersection with RS theorems such as reality_from_one_distinction, Jcost uniqueness (Cost.FunctionalEquation.washburn_uniqueness_aczel), phi-ladder constants, 8-tick periodicity, or Alexander-duality D=3 forcing. No J-cost, ratio symmetry, or parameter-free derivation appears.","tokens_in":56294,"confidence":"high","tokens_out":176,"duration_ms":6343,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"TwinRouterBench supplies step-level prefixes with execution-verified target tiers and a live dynamic harness for evaluating agentic LLM routers.","keywords":["LLM routing","agentic evaluation","benchmark","model tiering","dynamic harness","cost measurement","SWE-bench","static prefixes"],"falsifier":"If re-running the downgrade-and-cascade protocol on the same prefixes with a different model pool or additional instances produces inconsistent tier assignments, the benchmark labels would not reliably indicate the cheapest sufficient model.","tokens_in":2679,"feed_emoji":"📊","tokens_out":504,"duration_ms":34895,"temperature":0.7,"pith_summary":"The paper contends that one-shot prompt benchmarks are insufficient for routing in long-horizon agentic applications like coding agents where multiple model calls occur. It presents TwinRouterBench with a static track offering 970 router-visible prefixes from 520 instances across several datasets, each labeled with target tiers via a downgrade-and-cascade protocol, scored deterministically without LLM judges. The dynamic track provides a harness for running routers on full SWE-bench Verified trajectories, measuring both task resolution and actual API spend. This dual approach enables rapid offline development of routers followed by realistic live validation. Readers would care because effective routing can lower costs in complex agent systems without losing performance on downstream tasks.","feed_headline":"Benchmark evaluates LLM routers on agent steps with verified tiers","feed_subtitle":"Static track gives 970 prefixes with target tiers; dynamic track measures live task success and API spend on SWE-bench without judges.","key_machinery":"The downgrade-and-cascade protocol for assigning target tiers to prefixes by sequentially testing cheaper models to find the minimal sufficient tier that preserves execution success.","core_discovery":"TwinRouterBench establishes a step-level routing benchmark consisting of two tracks: the static track pairs 970 prefixes with execution-verified target tiers derived from a downgrade-and-cascade protocol across multiple benchmarks, allowing deterministic arithmetic scoring based on tier labels, trajectory membership, and token costs; the dynamic track runs routers in a live harness on the SWE-bench Verified suite, selecting models at each call and evaluating official task success alongside realized spend, with a 100-case held-out set reported.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["TwinRouterBench: step-level static and dynamic LLM router benchmark","970 prefixes with verified tiers for static router evaluation","Dynamic track: live SWE-bench routing with success and spend metrics","No LLM judges in TwinRouterBench evaluation protocol","Fast offline iteration followed by end-to-end agent validation"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The downgrade-and-cascade protocol assigns target tiers that remain stable and generalize beyond the specific 520 instances and model pool used to create the labels.","fun_headline_variants_meta":{"raw":{"variants":["TwinRouterBench: step-level static and dynamic LLM router benchmark","970 prefixes with verified tiers for static router evaluation","Dynamic track: live SWE-bench routing with success and spend metrics","No LLM judges in TwinRouterBench evaluation protocol","Fast offline iteration followed by end-to-end agent validation"]},"model":"grok-4.3","cost_usd":0.005797,"raw_usage":{"total_tokens":2797,"prompt_tokens":741,"num_sources_used":0,"completion_tokens":77,"cost_in_usd_ticks":57974500,"prompt_tokens_details":{"text_tokens":741,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1979,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":741,"tokens_out":77,"duration_ms":20290,"temperature":1.0,"reasoning_tokens":1979,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-25T05:56:18.802748+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If re-running the downgrade-and-cascade protocol on the same prefixes with a different model pool or additional instances produces inconsistent tier assignments, the benchmark labels would not reliably indicate the cheapest sufficient model.","supporting_citations":[],"review_version":2}