{"id":"00a910ec-bfcc-4c4b-bc24-139f475c99aa","arxiv_id":"2607.22465","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Task-consistent sticky routing with a contextual bandit improves accuracy-latency trade-offs on agentic benchmarks by learning from delayed task-level reward.","lead":"TRACE-Router picks one AI model for an entire multi-step agent task instead of routing every call separately, and learns which model to use from the task's final outcome. The paper reports better accuracy-latency trade-offs on agentic benchmarks than random model mixing or heuristic routers, though the evidence is thin and no code is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing per-call router baseline leaves the central task-consistency claim untested.","rationale":"The reader's explicit weakest assumption was the requirement of persistent task IDs and graded terminal rewards. That is a stated scope condition, not an internal flaw: the paper claims to work under those assumptions, and the benchmarks satisfy them. A more central threat is the absence of any per-call router baseline. The paper's abstract and introduction argue that existing per-call routers are fundamentally misaligned, and the proposed solution is task-consistent routing. Yet the evaluation only compares against single models and task-level heuristic routers, so the specific contribution—switching the routing granularity—is never directly tested. Even if the reported margins are real, they could be driven by simple task-level adaptation rather than by the sticky per-trace commitment. The reader's rationale did flag this missing baseline, but their weakest_assumption field focused elsewhere, so agreement is partial. I recommend no change to the reader's CONDITIONAL verdict: the missing baseline is a serious gap, but it does not by itself invalidate the results; it means the central claim is under-supported and should be conditional on adding this comparison.","tokens_in":12306,"tokens_out":4650,"duration_ms":56059,"concrete_test":"On the same three benchmarks and backend pairs, add a per-request learned router baseline—e.g., a RouteLLM-style classifier or per-call contextual bandit that selects a backend for each LLM call and is updated with the same terminal reward (credited to all calls in the trace). Report its accuracy–latency frontier with bootstrap CIs. If it matches or dominates TRACE-Router's frontier, the central task-consistency claim is not supported; if it is dominated, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"TRACE-Router's central claim is that per-call routers are misaligned with agentic feedback and that task-consistent routing fixes this (§1). The evaluation, however, never includes a per-call router. §4.1 lists baselines: two single models, a semantic router, and a complexity router; all make one decision per task, not per request. Figure 3 and Table 1 confirm no per-request learned baseline, and the offline replay ablation (§4.5) compares bandit policies only. Without a per-call router, the observed frontier gains over latency-matched random mixing could be achieved by any adaptive task-level selector; they do not demonstrate that pinning a single backend per trace is the operative mechanism, or that per-call routers actually fail. Related work mentions SWE-Router and other trajectory-level routers but no comparison is reported. This is load-bearing because the paper's motivation and novelty rest on the granularity mismatch, yet the experiments never manipulate granularity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TRACE-Router, a task-level router for agentic LLM workloads. Its core idea is to make one routing decision per task trace rather than per LLM call: a coarse context classifier assigns each task to a bandit, the bandit selects a single backend, a persistent task id pins all subsequent requests of that trace to the same backend, and the policy is updated only from the delayed terminal reward, scalarized as an accuracy–latency trade-off. The evaluation uses three benchmark suites (τ2-Bench retail/telecom, LiveCodeBench, Terminal-Bench) with two-backend pools and an additional four-model ablation. The paper reports that TRACE-Router occupies interior Pareto-frontier points, beats latency-matched random model mixing by 7–8 accuracy points on τ2-Bench, and on Terminal-Bench exceeds the larger backend by 7.1 accuracy points at 36% lower latency.","tokens_in":12465,"tokens_out":4900,"duration_ms":58945,"significance":"If the empirical claims held with adequate statistical support, the task-consistency principle would be a useful design correction for agentic serving, and the online contextual-bandit formulation is a clean way to operationalize delayed task-level feedback. The paper is also commendably transparent: it explicitly notes that the Terminal-Bench margin is only three to four tasks, that turn-exhaustion is not instrumented, that the context classifier is a deliberate minimal regex, and that the offline replay uses 80 seeds for the bandit-policy comparison. The four-model ablation is a valuable scaling check. However, the central comparison needed to establish the paper's motivating claim—that per-call routing is misaligned and task-consistent routing fixes it—is missing, and the headline live frontier results are reported without variance. The contribution is therefore plausible but not yet convincingly demonstrated.","major_comments":[{"comment":"The paper's motivating claim is that per-call routers are misaligned with agentic feedback and that task-consistent routing resolves this (§1, §2). Yet the evaluation contains no per-call router baseline. The baselines listed in §4.1—two single models, a semantic router, and a complexity router—all make one routing decision per task, not per request. Figure 3 and Table 1 confirm that no request-level learned or adaptive router is compared. Gains over latency-matched random mixing demonstrate that an adaptive task-level selector beats random mixing, but they do not show that pinning one backend per trace is the operative mechanism, nor that per-call routers actually fail. This is load-bearing because the novelty and the stated motivation rest on the granularity mismatch. Add at least one per-request router baseline (e.g., per-call UCB/ε-greedy with the same delayed terminal reward, or an","section":"§4.1, Fig. 3, Table 1"},{"comment":"The headline frontier results appear to come from single live runs with no error bars, confidence intervals, or seed counts. The paper itself notes in §4.2 that Terminal-Bench has only 48 matched tasks and that the 7.1-point margin is 'between three and four tasks,' and in §4.4 that a 3.8-point warm-start difference is 'a margin worth under two tasks.' These disclosures are welcome, but they also show that the 'consistently improves' and 'non-dominated Pareto frontier' claims are not supported by the reported uncertainty. The offline replay in §4.5 supplies 80 seeds, but only for the policy-family ablation, not for the live accuracy–latency frontiers. Please provide repeated live runs (or bootstrap intervals from recorded task outcomes) for at least the central τ2 and Terminal-Bench claims, or present the results more cautiously as indicative single-run measurements.","section":"§4.2, Fig. 3"},{"comment":"The claim 'TRACE-Router holds the interior of the frontier' is weakened by the fact that the context-free variant is dominated at two operating points but the difference is described as 'within noise elsewhere' (§4.3). If the aggregate context-conditioning advantage is within noise on some benchmarks, the paper should state which benchmark-level advantages are not within noise, and the frontier plots should display this uncertainty. Otherwise the reader cannot tell whether the interior-frontier positions are stable or artifacts of a single draw.","section":"§4.2, §4.3"}],"minor_comments":[{"comment":"The contributions list says 'Across four agentic benchmarks,' while the abstract and evaluation describe three benchmark suites (τ2-Bench, LiveCodeBench, Terminal-Bench) plus a small τ2-airline panel. Please reconcile the count.","section":"§1, contributions"},{"comment":"There is a typo: 'the learns to favor Qwen3.5-9B' should be 'the router learns to favor...' Also, the numbers in the four-model section ('31.24%' vs. earlier single-pair telecom numbers) need a clear explanation of the different model pool and experimental conditions.","section":"§4.6.1"},{"comment":"The LiveCodeBench accuracy metric uses difficulty weights w_d = 1 - p̄_d derived from the same two single-model baselines. Since the router selects among those models, the weighting is not independent of the models being compared. Please discuss whether this biases the metric and report sensitivity to alternative weights, or state why the effect is negligible.","section":"§4.1, LiveCodeBench"},{"comment":"The warm-start variant's pseudo-count prior is described clearly, but the claim that 'the prior is not overturned but self-reinforcing' deserves a more formal explanation: under the UCB rule, a high prior mean for the preferred arm also suppresses exploration of alternatives for some time, but the mechanism by which it becomes self-reinforcing rather than merely slow to correct is not fully argued.","section":"§4.4 / Appendix A"},{"comment":"The 'latency position' normalization is useful but not defined in the caption. Please state explicitly that 0 corresponds to the small backend's latency and 1 to the large backend's, as implied by the text.","section":"Figure 4"},{"comment":"The related-work discussion mentions SWE-Router and other trajectory-level routers, but no empirical comparison is reported. Even a short discussion of why a direct comparison is not feasible (different harnesses/backends) would help the reader calibrate the novelty claim.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable systems/ML submission: the task-consistency idea is coherent and the authors are unusually transparent about limitations. The main gap is evaluation methodology: missing per-call router baseline and no uncertainty on the headline claims. These are fixable within the scope of a revision—adding a baseline and error bars, and softening claims as needed. I do not see an internal inconsistency that would warrant rejection, but the current evidence is insufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper's core idea is simple and right-sized: route at the granularity where feedback is actually generated. Instead of making a separate model choice for every LLM call in an agentic trace, it pins one backend for the whole task and updates a contextual bandit from the terminal outcome. That is a genuine change of granularity relative to request-level routers like RouteLLM, and doing it online without offline training is a real difference from SWE-Router's offline value model. The writeup is also refreshingly honest: it tells you the Terminal-Bench 7.1-point margin is between three and four tasks, that turn-exhaustion isn't instrumented, and that the warm-start prior didn't help. That transparency deserves credit.\n\nThe soft spot is load-bearing. The paper motivates itself with 'the unit of routing should match the unit of feedback' and argues per-call routers misattribute delayed task reward. But no per-call router appears in the evaluation. Every baseline—single models, semantic, complexity—makes one decision per task. The offline replay ablation compares bandit policies only. So the observed gains over latency-matched random mixing could be produced by any adaptive task-level selector, and the paper never tests whether per-call routing actually fails. You don't get to the claimed mechanism because granularity is never manipulated. That's the first thing a reviewer should ask for.\n\nSecond, the headline results come from single live runs with no error bars; task totals are small—48 tasks on Terminal-Bench—and the difficulty weights for LiveCodeBench are derived from the same single-model baselines. These are minor if the missing-baseline issue is fixed, but as reported the quantitative claims are provisional.\n\nThird, the framework assumes a persistent task ID and a graded terminal reward. The paper is upfront about these as requirements, but it only evaluates on benchmarks where they exist. The enterprise setting that motivates the work may not provide them cleanly. That's a scope limitation, not a fatal flaw.\n\nOverall, the paper deserves a serious referee. I'd send it out, but with an explicit request to add at least one per-call learned router as a baseline and to rerun the main figures with multiple seeds or state clearly why they can't. If the mechanism survives that comparison, the paper is solid. As it stands, it's a well-presented conditional result.","headline":"A clear task-level routing idea with an honest writeup, but the evaluation omits the one baseline—a per-call router—that the entire motivation hinges on.","tokens_in":13023,"tokens_out":2770,"would_cite":true,"duration_ms":30141,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TRACE-Router shows that routing agentic tasks as whole traces—not individual LLM calls—improves accuracy–latency trade-offs by learning from delayed terminal feedback.","keywords":["LLM routing","agentic AI","contextual bandit","task-level routing","delayed feedback","accuracy-latency trade-off","online model selection","credit assignment"],"falsifier":"Run TRACE-Router on a workload where tasks lack reliable identifiers or terminal rewards are noisy/unattributable, and show it performs no better than per-call routing—or a benchmark where mid-task model switches are beneficial, so task-consistent binding actually hurts.","tokens_in":12132,"feed_emoji":"🧭","tokens_out":3066,"duration_ms":35381,"temperature":0.7,"pith_summary":"The paper argues that the unit of routing should match the unit of feedback: in agentic workloads, feedback is a delayed, task-level outcome, so routing decisions should be made once per task and pinned to a single backend. TRACE-Router implements this with a contextual bandit that assigns a model at task admission and updates its policy from the task's terminal reward. Across three agentic benchmarks, the method achieves non-dominated accuracy–latency Pareto frontier points, beating latency-matched random model interpolation by 7–8 points on τ2-Bench and outperforming the stronger single model by 7.1 points at 36% lower latency on Terminal-Bench. If correct, this means online, feedback-driven task-level routing can replace both static per-call routers and offline-trained classifiers in agentic deployments.","feed_headline":"Task-level routing beats per-call LLM routing","feed_subtitle":"Agentic AI rewards arrive at task end; TRACE-Router pins one backend per trace and learns from terminal outcomes.","key_machinery":"The core mechanism is a contextual UCB bandit combined with a sticky task-to-model binding: each task carries a persistent identifier, the router selects a backend once at admission using a coarse context (e.g., regex-based difficulty tier), and all subsequent requests in the trace reuse that backend. The reward is r = (1−α)·accuracy − α·clipped latency, where α trades accuracy against latency; terminal outcomes update only the context–backend statistics that made the original decision. A small round-robin cold-start phase gives every arm an initial estimate, and the confidence parameter δ controls exploration strength.","core_discovery":"The paper's central claim is that task-consistent routing—binding one model to an entire execution trace and learning from the task's terminal outcome—enables principled delayed credit assignment and yields better accuracy–latency trade-offs than request-level routing or static model assignment. TRACE-Router maintains per-context contextual UCB bandits, initializes with a small number of forced pulls, and updates only the bandit responsible for each completed task using a scalarized reward of accuracy and normalized latency. The experiments show that this simple mechanism occupies interior frontier positions on three benchmarks, that context partitioning determines which operating points are","pith_inferences":["We infer the principle extends beyond model selection: any per-request decision whose payoff is only observable at task end should be made once per task to preserve credit assignment.","We infer that noisy, sparse, or delayed terminal rewards would degrade the method; testing on workloads with partial feedback could reveal its tolerance to attribution errors.","We infer there is a crossover point where warm-starting becomes favorable—longer streams or reliable priors might overturn the paper's cold-start default.","We infer the same mechanism could route not just models but tools, agents, or configurations, provided a task identifier and terminal score exist."],"forward_implications":["Agentic LLM deployments can select models online from task outcomes without offline training data or explicit task-complexity estimation.","Task-consistent binding gives a single credit-assignment target, so delayed terminal rewards can be correctly attributed to the routing decision.","Adaptive task-level routing can dominate single-model baselines on the accuracy–latency frontier, not merely interpolate between them.","Context partitioning is necessary for reaching certain frontier points, not just for improving average performance.","Cold-start exploration is sufficient in finite task streams; prior warm-starting can lock in slower models and suppress useful exploration."],"fun_headline_variants":["Task-level routing beats per-call LLM calls","One model per task: better accuracy and speed","LLM routing that learns from task outcomes","Delayed task rewards improve LLM routing","Pick one LLM per task for better trade-offs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Every task must expose a persistent identifier and a graded terminal reward; without those, the delayed-credit-assignment mechanism collapses.","fun_headline_variants_meta":{"raw":{"variants":["Task-level routing beats per-call LLM calls","One model per task: better accuracy and speed","LLM routing that learns from task outcomes","Delayed task rewards improve LLM routing","Pick one LLM per task for better trade-offs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1196,"prompt_tokens":746,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":379}},"tokens_in":490,"tokens_out":450,"duration_ms":5701,"temperature":1.0,"reasoning_tokens":379,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T04:38:05.494534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TRACE-Router on a workload where tasks lack reliable identifiers or terminal rewards are noisy/unattributable, and show it performs no better than per-call routing—or a benchmark where mid-task model switches are beneficial, so task-consistent binding actually hurts.","supporting_citations":[],"review_version":1}