{"id":"9c184213-e3c9-4107-a1c9-c98cb2be50c0","arxiv_id":"2608.08528","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"T2MO routes coding-assistant traffic by task category and difficulty, using an expected-cost-per-completed-task criterion whose routing boundary, Eq. (2), is the minimum pass rate a cheaper model needs to justify displacement.","lead":"Enterprise coding assistants can waste millions when every request hits top-tier models; this paper lays out a nine-stage system for routing each task to the cheapest model that can finish it, measured by cost per completed task. The paper is a detailed design memo from PayPal's LLM operations, with a correct two-line dominance theorem and illustrative, not yet validated, savings figures.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Routing boundary depends on Eq. (1)'s one-attempt/single-escalation assumption; the paper's own Section 17 concedes no live A/B or ablation, so the conditional verdict is appropriate.","rationale":"The reader's weakest_assumption is exactly the load-bearing concern: Eq. (1) is a one-attempt, single-escalation model with an uncalibrated additive W, and the paper's own Section 17 concedes that there is no live A/B result set and no full ablation. The formal result in Section 4 is internally sound; I checked the algebra of Proposition 1 and the dominance argument, and they hold within the model. But the framework's practical claim of cost-optimal routing depends on Eq. (1) pricing failures correctly. The paper itself flags the missing empirical support, and page-facing completeness requires that I weigh that concession in the verdict. I agree with the reader's conditional verdict: the formal contribution supports an eventual ACCEPT, but the validation conditions are not met. I do not see a stronger internal flaw; the strength of the paper is that it is unusually explicit about what is measured, what is illustrative, and what is missing.","tokens_in":8478,"tokens_out":1543,"duration_ms":14170,"concrete_test":"Implement Eq. (1) on a logged production session corpus, comparing the predicted expected cost per cell against the realized end-to-end cost per cell (actual retries, actual escalations, actual developer wait time), and then re-derive the routing boundary of Eq. (2) from the realized cost curve. If the boundary from realized costs shifts by more than, say, 10 percentage points of pass rate on the cells that matter for the waterfall, the single-attempt/single-escalation model is the weak point; report the shift on the two benchmarked subcategories of Table 2. A second check, independent but cheap: re-run the dominance proof with two attempts and two escalations, and give the new boundary; if it differs materially from Eq. (2), note that the paper's boundary is not robust to the natural generalization.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central formal claim, Proposition 1 in Section 4, is correct only relative to Eq. (1): E[cost|m,c] = Cm(c) + (1-Pm(c))[CM(c)+W]. This model assumes a single candidate attempt, exactly one escalation to the incumbent, no further retries or cascades, and a fixed additive wait cost W that is known and independent of the cell, the model, and the session. In production sessions, failures can cascade through multiple candidate attempts, escalation may consume more than one incumbent run, and W is not measured or calibrated anywhere in the paper; Section 17 explicitly acknowledges that no live A/B result set or full ablation is included. The routing boundary of Eq. (2), and the savings waterfall of Eq. (4) that gates every entry on Proposition 1, therefore inherit whatever mispricing Eq. (1) introduces. The paper is honest about this: the worked Table 2 savings figures are explicitly illustrative, the within-subcategory difficulty split rho is a placeholder, and Section 17 concedes the lack of a live A/B result set. So the concern is not internal inconsistency, but that the paper's headline claim to be a data-driven framework for cost-optimal routing is not yet empirically validated at the level the title and abstract imply. The dominance theorem is real and independently checkable, but it proves dominance only under the model; it does not prove that the model prices reality correctly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Task-to-Model Optimization (T2MO), a nine-stage framework for selecting LLM routing policies in enterprise coding assistants. The central formal contribution is an expected-completion-cost objective, E[cost|m,c] = Cm(c) + (1-Pm(c))[CM(c)+W], and a proof (Proposition 1, Section 4) that this objective is weakly more conservative than token-cost minimization, with Eq. (2) giving the minimum pass rate a cheaper candidate must reach before displacement is cost-justified. The manuscript then details a pipeline for telemetry, taxonomy discovery, difficulty grading, benchmark construction, candidate evaluation, mix derivation, the savings waterfall of Eq. (4), forecasting, staged routing deployment, and governance. Empirical content appears in Figure 1 and Table 2, where the authors report measured pass rates for two subcategories and present illustrative savings figures, while explicitly labeling the difficulty splits and displacement fraction as placeholders. Section 17 acknowledges that the paper does not yet include a live A/B result set or a full ablation study.","tokens_in":8721,"tokens_out":4826,"duration_ms":56322,"significance":"If the framework performs as claimed, it addresses a real and costly operational problem: concentrated enterprise LLM inference spend and the failure of naive token-cost routing once retries and developer waiting are priced in. The formal dominance argument in Section 4 is correct as a piece of algebra and is genuinely useful as a decision rule, and the paper is commendably explicit about the model assumptions behind it. The strengths are the clean per-cell displacement criterion, the cost-weighted taxonomy perspective, the staged deployment path, and the honest statement of limitations. However, the significance is currently conditional: the paper's empirical contribution is an illustrative waterfall with placeholder inputs, and the cost model in Eq. (1) is not validated against telemetry. As a methodology proposal the paper is plausible; as a data-driven demonstration of cost-optimal routing it is not yet supported.","major_comments":[{"comment":"The paper explicitly concedes in Section 17 that it 'does not yet include a full ablation study or a live A/B result set', yet the empirical backbone elsewhere is presented as measured: Table 2 reports GLM 5.2 pass rates of 100/50/25% and 100/100/47% per difficulty tier as 'measured in-harness', but gives no sample size, no binomial confidence interval, no evaluation date, and no evidence that the Section 8 rule of at least 30 tasks per confirmed cell was met. Without these quantities, the reader cannot verify the central empirical claim that the framework identifies opposite displacement decisions in two near-identical-traffic subcategories, and the title's 'data-driven' claim is stronger than what the manuscript demonstrates.","section":"Section 17; Sections 8, 9, 11; Table 2"},{"comment":"The routing boundary of Eq. (2) and the dominance result of Proposition 1 are derived under a specific cost model: one candidate attempt followed by exactly one escalation to the incumbent, with an additive and known developer wait cost W. The manuscript does not estimate W, does not report realized retry or escalation rates from telemetry, and does not validate Eq. (1) as a predictor of observed end-to-end costs; Section 15 lists escalation-related risk as a qualitative mitigation, but that does not constitute validation. If production sessions cascade through multiple candidate attempts or consume more than one incumbent run per failure, the boundary in Eq. (2) does not guarantee the claimed end-to-end savings. A concrete remedy would be a telemetry-based check comparing predicted E[cost] from Eq. (1) with realized per-cell costs and reporting the distribution of retry counts per failed candidate attempt.","section":"Section 4, Eqs. (1)-(2)"},{"comment":"The headline quantitative illustration, the savings waterfall, is a placeholder computation: the within-subcategory difficulty shares rho_d|k are labeled illustrative, the displacement fraction delta_tier=0.35 is hand-set, contracted prices are withheld, and only two subcategories are shown, so Eq. (4) is instantiated with partly unmeasured inputs. While the manuscript is transparent about this, the abstract and title present a 'data-driven framework for cost-optimal routing', and the only numerical demonstration is not a measured result. The authors should either add a real end-to-end measurement on at least one cell with confidence intervals, or reposition the paper as a purely methodological framework and remove the implication that the specific dollar figures represent demonstrated savings.","section":"Section 11; Figure 1; Table 2; Eq. (4)"}],"minor_comments":[{"comment":"Eq. (3) defines cells as (category, difficulty), a two-level hierarchy, but Figure 1 and Table 2 introduce a third subcategory level; please clarify whether the cell is defined at subcategory × difficulty or root-category × difficulty, and align the notation.","section":"Section 11, Eq. (3); Figure 1"},{"comment":"The ex ante difficulty features in Section 7 and the five-dimension difficulty check invoked in Section 11 (scope, next-step clarity, evidence clarity, action risk, judgment required) are listed but not operationalized; providing scoring definitions or indicating where they are calibrated would make the methodology reproducible.","section":"Section 7; Section 11"},{"comment":"The working rule of at least 30 tasks per cell is stated without a power or precision justification, and for displacement decisions the relevant quantity is the confidence interval on the pass-rate delta between candidate and incumbent, not the per-cell pass-rate CI alone.","section":"Section 8"},{"comment":"The table's footnote says savings are computed via Eq. (4) with a displacement fraction delta_tier=0.35, but Eq. (4) as written contains no delta_tier term; please reconcile the formula and the footnote so the illustrative calculation is unambiguous.","section":"Table 2, footnote; Eq. (4)"}],"recommendation":"major_revision","confidential_remarks":"The paper is honest about its own limitations, and the central deficiency is empirical validation rather than mathematical incorrectness. If the venue is open to methodology-only papers, the authors should resubmit with a substantially toned-down title and abstract; if the venue expects empirical evidence, the paper needs at least one fully measured cell with confidence intervals and a telemetry-based check of Eq. (1). The formal result in Section 4 is sound and should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this is a systems/framework paper, not an empirical validation, and the authors are unusually honest about that. The formal core, Proposition 1, is correct and worth knowing: if you define expected completion cost as C_m(c) + (1-P_m(c))[C_M(c)+W], then routing to displace the incumbent only when pass rate clears the threshold in Eq. (2) weakly dominates token-cost minimization. The algebra checks out, and the result is self-contained — no circularity. That is a genuinely useful way to frame escalation in LLM routing.\n\nWhat it does well: it connects existing ideas (Clio taxonomy, FrugalGPT cascades, RouteLLM, benchmark-based selection) into a nine-stage production pipeline with explicit governance, confidence tiers, and a savings waterfall. The waterfall is a nice practical artifact: it ranks displacement opportunities by dollar impact. The authors separate measured data (cost shares, pass rates) from illustrative placeholders (difficulty split, delta_tier), which is rare. The citation pattern is appropriate: they engage the routing and cascade literature directly and don't pad with self-citations.\n\nWhere it is soft — in predictable places. The empirical backbone is not there. Pass rates appear without confidence intervals or sample sizes. The within-subcategory difficulty split is explicitly illustrative. The savings figures in Table 2 are hand-calculated with delta_tier=0.35. Section 17 concedes no live A/B or full ablation. So the headline claim of 'data-driven cost-optimal routing' is not actually demonstrated on data. The formal dominance result is conditional on Eq. (1) being a faithful production-cost model: one candidate attempt, exactly one escalation, additive wait cost W. Real sessions can cascade, and W is unmeasured. That said, the authors flag all of this themselves; the gap is disclosed, not hidden.\n\nWho this is for: applied LLMOps engineers and researchers working on cost-aware routing or cascades. They'll get a clear framework and a citable formal result. It is not a paper that proves a deployed system works.\n\nRecommendation: yes, send it to peer review, but treat it as a framework+theory submission, not a validated empirical study. The conditions in the reader's report are exactly right: require sample sizes and CIs, a retrospective or live A/B comparison, and sensitivity analysis on W. If those get added, it becomes a solid applied paper.\n\nBest,","headline":"A correct formal dominance result wrapped in a production framework that is honest about being empirically unvalidated; worth a referee but not a strong empirical paper.","tokens_in":9339,"tokens_out":3469,"would_cite":true,"duration_ms":31814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that routing coding-assistant traffic by expected cost per completed task, including retries and developer waiting, beats minimizing token price, and derives the exact pass-rate bar a cheaper model must clear.","keywords":["LLM routing","cost per completed task","model cascades","task taxonomy","difficulty grading","enterprise coding assistants","savings waterfall","expected completion cost"],"falsifier":"Run a production A/B on a set of task cells: route one arm with the expected-completion-cost policy of Eq. (1) and the other with token-cost minimization, and measure realized cost per completed task including every retry, escalation, and measured developer wait. If the token-cost arm achieves lower realized cost on any cell, or if the failure logs show multiple candidate attempts or multiple incumbent runs per failed attempt, then the single-escalation model has underpriced failure and the dominance claim would be falsified.","tokens_in":8248,"feed_emoji":"💸","tokens_out":11312,"duration_ms":99417,"temperature":0.7,"pith_summary":"Enterprise AI coding assistants spend heavily on a few frontier models, and switching to cheaper models based on token price can backfire once retries, escalations, and developer wait time are counted. This paper proposes Task-to-Model Optimization (T2MO), a nine-stage pipeline that discovers a cost-weighted task taxonomy from production traffic, grades each cell's difficulty, benchmarks candidate models in a production-like harness, and routes each task category and difficulty cell to the cheapest model able to complete it. The decision rule minimizes expected cost per completed task, explicitly pricing a failed attempt as one candidate run plus one incumbent run and an additive developer wait cost $W$. Under that rule, a cheaper model displaces the incumbent on a cell only when its pass rate clears the routing boundary, and the paper proves this policy weakly dominates naive token-cost minimization under escalation. A traffic-weighted savings waterfall ranks displacement opportunities by dollar impact, and the framework matures from static policy to shadow-mode classifiers, verified cascades, and a full router.","feed_headline":"Cheapest LLM tokens can cost more per finished task","feed_subtitle":"It prices retries, escalations, and wait time, then routes each coding task to the cheapest model that can finish.","key_machinery":"The load-bearing object is the expected-completion-cost rule of Eq. (1), paired with the routing boundary of Proposition 1. Eq. (1) models a failure as one candidate attempt followed by one escalation to the incumbent plus an additive developer wait cost $W$; Eq. (2) rearranges the displacement condition into a pass-rate threshold: a cheaper model is worth deploying on a cell only if $P_m(c) > 1 - \\frac{C_M(c)-C_m(c)}{C_M(c)+W}$. The second piece of machinery is the task-cell decomposition: production traffic is split into a cost-weighted taxonomy of root categories and within-category difficulty tiers, so routing decisions are made per leaf cell rather than per prompt or token. The third piece is the traffic-weighted savings waterfall, which sums per-cell savings over only the cells that clear the routing boundary and ranks them by dollar impact, turning the optimization objective into an auditable deployment backlog.","core_discovery":"The paper's central claim is that task-to-model routing should optimize expected cost per completed task rather than cost per token or per prompt. For a fixed task cell $c$ and candidate model $m$, the expected cost is $E[\\mathrm{cost}\\mid m,c] = C_m(c) + (1-P_m(c))[C_M(c)+W]$, where $C_m(c)$ and $C_M(c)$ are in-harness costs for the candidate and incumbent, $P_m(c)$ is the verifier pass rate, and $W$ is the developer wait and context-switch cost. Because a failed candidate attempt triggers a full run on the incumbent plus a wait penalty, a lower-priced model can be more expensive end-to-end. Proposition 1 states that $m$ should displace incumbent $M$ on cell $c$ only if $P_m(c) > 1 - \\frac{C_M(c)-C_m(c)}{C_M(c)+W}$, which is the routing boundary: the minimum pass rate for cost-effective displacement. Every cell displaced under this rule is also displaced under token-cost minimization, but not conversely, so the expected-cost policy weakly dominates token-cost minimization, meaning its realized expected cost is never higher on any cell. The paper then organizes decisions as a two-level hierarchy of task category and difficulty tier, and aggregates per-cell displacements into a traffic-weighted savings waterfall, claiming each entry weakly reduces realized end-to-end cost.","pith_inferences":["Beyond the paper, the same expected-completion-cost criterion should transfer to non-coding LLM workloads with an escalation hierarchy, such as enterprise support or document processing; calibrating $W$ per domain would be the main adaptation step.","Beyond the paper, the routing boundary implies a comparative static the paper does not test: as $W$ rises, the per-cell pass-rate threshold rises, so organizations with expensive developer interruptions should concentrate more traffic on frontier models; this is testable with a controlled A/B that varies wait-cost assumptions.","Beyond the paper, the confidence tiers imply a benchmark-investment policy the paper does not spell out: evaluation effort should concentrate on high-traffic cells whose pass rates sit near the boundary, because those cells carry the largest expected value of information.","Beyond the paper, if in-harness pass rates transfer to production, Eq. (2) can be used as a per-cell classifier threshold for an online router, using ex ante difficulty features; verifying that pass-rate transfer is the key empirical risk."],"forward_implications":["Under the expected-cost rule, no task cell is displaced to a cheaper model unless its pass rate clears the routing boundary, so realized end-to-end cost per completed task is never higher than under token-cost minimization.","The routing boundary gives operations teams a concrete go/no-go number: the threshold rises as developer wait cost or escalation penalty rises, so high-interruption environments will keep more traffic on frontier models.","The savings waterfall makes displacement opportunities auditable: cells are ranked by traffic-weighted dollar savings, so benchmark and rollout effort goes to the highest-value cells first.","Staged routing lets an enterprise start with a static policy plus developer guidance, then move to shadow-mode classifiers and verified cascades before trusting a full router, and verified cascades remove the need for ex ante difficulty prediction where cheap automatic verifiers exist.","Rolling twelve-month forecasts decompose spend into adoption, activity, mix, and price effects, so model releases and discount expiries are handled as named scenario events rather than surprises."],"supporting_citations":[{"why":"Introduces the cascade idea and the cost-quality trade-off that this paper generalizes from query-level to task-cell level, and supplies the token-cost baseline behavior.","marker":"[2]"},{"why":"Provides the learned strong-versus-weak binary routing baseline against which the expected-cost rule is positioned.","marker":"[3]"},{"why":"Supplies the quality-aware difficulty-prediction routing approach that motivates the paper's per-cell difficulty tiers.","marker":"[4]"},{"why":"Establishes verification-based escalation cascades, which Eq. (1) prices explicitly as failure escalation.","marker":"[5]"},{"why":"Supplies the bottom-up clustering methodology used to discover the cost-weighted task taxonomy from production traffic.","marker":"[1]"},{"why":"Shows per-task model selection from benchmark data, the per-task lineage the paper extends to per-cell displacement decisions.","marker":"[6]"},{"why":"Provides repository-scale issue-resolution benchmarks used in constructing production-like evaluation sets.","marker":"[7]"},{"why":"Provides the self-contained code-generation benchmark used alongside repository-scale tasks in the evaluation harness.","marker":"[8]"},{"why":"Validates the LLM-as-judge calibration protocol that motivates the paper's verifier-first grading and quarterly recalibration.","marker":"[9]"}],"fun_headline_variants":["Optimize LLM routing by cost per completed task, not token price","Cost per finished task, not per token, should guide LLM routing","To cut coding costs, route by expected cost per completed task","LLM routing: price retries and wait time, not just tokens","Cheaper tokens? Only if they pass the routing boundary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Eq. (1) faithfully models production cost: a failed candidate attempt costs exactly one candidate run plus exactly one incumbent run plus a known, additive developer wait cost $W$. If real failures cascade through multiple attempts, if a failed cheap-model attempt triggers more than one incumbent run, or if $W$ is not actually measured, the routing boundary of Eq. (2) no longer guarantees the claimed end-to-end savings.","fun_headline_variants_meta":{"raw":{"variants":["Optimize LLM routing by cost per completed task, not token price","Cost per finished task, not per token, should guide LLM routing","To cut coding costs, route by expected cost per completed task","LLM routing: price retries and wait time, not just tokens","Cheaper tokens? Only if they pass the routing boundary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3924,"prompt_tokens":1122,"completion_tokens":2802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":738,"completion_tokens_details":{"reasoning_tokens":2721}},"tokens_in":738,"tokens_out":2802,"duration_ms":19154,"temperature":1.0,"reasoning_tokens":2721,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:32:25.895564+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a production A/B on a set of task cells: route one arm with the expected-completion-cost policy of Eq. (1) and the other with token-cost minimization, and measure realized cost per completed task including every retry, escalation, and measured developer wait. If the token-cost arm achieves lower realized cost on any cell, or if the failure logs show multiple candidate attempts or multiple incumbent runs per failed attempt, then the single-escalation model has underpriced failure and the dominance claim would be falsified.","supporting_citations":[],"review_version":1}