Pith. sign in

REVIEW 4 major objections 5 minor 16 references

LLM routing should gate on business value as well as difficulty, or expensive calls get wasted on hard-but-cheap cases.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 02:07 UTC pith:CJRUMFIZ

load-bearing objection Honest synthetic design notes with one real monitoring insight and one load-adaptive budget idea; the Tier-1 precision win is partly baked into a near-noiseless value gate. the 4 major comments →

arxiv 2607.25068 v1 pith:CJRUMFIZ submitted 2026-07-27 cs.AI

How Often Should a Recommender Call an LLM? Value-Weighted Routing, Monitoring, and Seasonal Robustness

Bhavtosh Rath This is my paper
classification cs.AI
keywords LLM routingcost-aware inferencecascade systemsvalue-weighted routingdecision monitoringconfidence calibrationseasonal robustnesse-commerce merchandising
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most systems that decide when to call an expensive language model treat the problem as one of difficulty: send the uncertain cases to the slow path. This paper argues that is incomplete, because a wrong decision on a cheap item and a wrong decision on a costly item are not the same mistake. In a fully synthetic retail merchandising simulation, a router that requires both high estimated difficulty and high estimated value matches a difficulty-only baseline’s recall of truly high-value items while wasting far less slow-path budget on hard-but-cheap ones. A second stage shows that aggregate calibration can look healthy while the difficulty estimator has almost no power inside each category—the overall correlation was mostly a between-category effect. A third stage shows that a slow-path budget capped as a fraction of that day’s own value estimates absorbs a simulated demand surge automatically, whereas a fixed daily budget collapses high-value recall. The work offers design principles for cost-aware routing, not validated production numbers.

Core claim

Difficulty and business value are distinct axes for routing between a cheap heuristic and an expensive LLM call. On a synthetic catalog with inverse volume–value structure, gating the slow path on both estimated difficulty and estimated value matches difficulty-only recall of true high-value items (~60%) while raising precision from 94.3% to 98.3%. Aggregate estimator–truth correlation can be almost entirely between-category; within each category the difficulty scorer collapses to near zero. An elastic budget scaled to the day’s own value estimates tracks a 2.5× surge without season logic, while a fixed budget drops high-value recall from 70.1% to 16.2%.

What carries the argument

Value-weighted threshold router: send an item to the expensive path only if estimated difficulty and estimated value both clear fixed thresholds, using only observables (category, price), never ground truth. Supporting machinery is within-segment calibration monitoring and an elastic slow-path cap equal to a fraction of that day’s total estimated value.

Load-bearing premise

The hand-built synthetic catalog, noisy estimators, and chosen Black-Friday mix shift are faithful enough stand-ins for real merchandising uncertainty and traffic that the qualitative ranking of these policies would still hold on real data.

What would settle it

On a real catalog with logged LLM error costs, check whether adding a value gate still improves precision at matched high-value recall, whether difficulty calibration collapses inside real segments, and whether an elastic value-fraction budget preserves high-stakes recall under a real traffic surge better than a fixed daily cap.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Cascade and confidence routers should add a cheap stakes or value estimate beside uncertainty, not escalate solely on difficulty.
  • Any confidence-based router should re-check calibration inside the segments it must discriminate within, not only in aggregate.
  • LLM spend caps should scale with already-computed batch value or demand estimates rather than a fixed daily or monthly dollar figure.
  • Under load that scales cost and value together, a value-fraction budget can absorb surges without calendar-specific logic.
  • Spend-concentration and starvation checks by segment become first-class health metrics for routing systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same two-axis gate applies beyond retail: support tickets by churn risk, code generation by production blast radius, or agent tool calls by downstream cost of a wrong action.
  • If within-segment calibration failure is common, many published cascade gains may partly reflect type separation rather than true instance-level difficulty ranking.
  • Elastic budgets invite a next experiment: whether the binding ratio stays stable when cost and value do not scale together (e.g., value mix shifts without volume surge).
  • A learned router trained on logged outcomes could replace hand thresholds while keeping the value axis and segment monitors as hard constraints.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript presents value-router, a fully synthetic simulation study of routing items between a cheap heuristic and an expensive (LLM-backed) path in a retail merchandising pipeline. Its central argument is that routing should consider two axes — estimated difficulty and estimated business value — rather than difficulty alone. Three staged contributions are reported: (i) a dual-threshold router (ValueWeighted) that matches a difficulty-only baseline's recall of true high-value items (~60%) while improving precision (98.3% vs 94.3%, Table 2); (ii) a monitoring tier showing that a difficulty estimator with aggregate estimate–truth correlation 0.792 has near-zero within-category correlation (Table 4), i.e., the aggregate is a between-category artifact; (iii) a seasonal-robustness tier in which an elastic slow-path budget (cap = α × the day's own total estimated value) absorbs a hand-constructed 2.5× demand surge without season-specific logic, while a fixed daily budget collapses high-value recall from 70.1% to 16.2% (Table 6). The paper is unusually explicit about scope: all numbers are from a single-seed synthetic environment with experimenter-defined ground truth, and the intended contribution is design principles, not validated real-world claims. Code with per-table reproduction commands is provided.

Significance. If the results hold up under variance and sensitivity analysis, the paper makes a useful, clearly written contribution to the cost-aware LLM-serving literature: it articulates the value-vs-difficulty distinction crisply (the cascades/routing work cited in §2 is indeed organized around a single uncertainty axis), elevates within-segment calibration from a training-time audit to a continuous routing diagnostic, and proposes a simple, implementable load-adaptive budget rule. The work ships reproducible code with CLI commands keyed to each table, which is a genuine strength and rare for a study of this size. The honest scoping — the abstract and §6 both state the results are illustrative design principles from a synthetic environment — is creditable. The ceiling on significance is set by that same scope: each individual mechanism (a second threshold, a groupby, a ratio cap) is simple, and the quantitative findings are currently single-seed properties of one hand-tuned configuration. The paper's value is as a well-executed design study and cautionary methodology note, not as evidence about production systems.

major comments (4)
  1. [§3.6, Table 2 (with Eqs. 1, 3 and §3.5)] The headline Tier-1 result — equal recall, +4 points precision from the value gate — is close to a structural consequence of the construction rather than an empirical finding, and the paper should say so and bound it. The evaluation label H = {p_i·m_i ≥ τ_v=20} and the gate v̂_i = p_i·m̄(c_i)+ε_v ≥ 20 share the same underlying quantity and threshold; the gate is a strongly label-correlated filter (Table 4: within-category value correlations 0.744–0.938), so AND-ing it onto DifficultyOnly can mechanically remove false positives while rarely touching true positives — exactly what Table 2 shows (recall 60.6%→60.4%). The size of the precision gap is governed by per-item margin dispersion and boundary mass near τ_v (mid_tier mean value 20.48 sits essentially at the threshold), both hand-chosen. The realistic regime — value estimated with substantial error, where the gate would also excise tru
  2. [All results tables (Tables 2–7)] Every number is a single seed (seed 42, n=2,000). The headline differences are small in item counts: the Tier-1 precision gap (98.3% vs 94.3%) corresponds to roughly 9–10 items on a slow set of ~236, and the peak-day recall gap between elastic and calendar-aware (70.1% vs 66.7%) is of similar magnitude. Without multi-seed variance or confidence intervals, neither headline comparison is established even within the synthetic environment. §6 acknowledges this as future work, but these comparisons anchor the paper's three claims, so acknowledgment is not sufficient. Multi-seed runs with reported means and intervals are cheap in this setting and should be included before the quantitative statements (including the abstract's 98.3%/94.3% and 70.1%→16.2% figures) are retained as findings rather than anecdotes.
  3. [§5.3, Table 6 (with Listing 2 and Eq. 8)] The elastic-budget result is weaker than its presentation. At α=0.08 the cap never binds on peak days, so the elastic row is identical to the unconstrained static router by construction (the paper states this); 'absorbed a 2.5× surge' thus reduces to 'the cap was loose enough,' and the comparison says nothing about elastic behavior in any regime where the budget actually constrains. Meanwhile the fixed-budget baseline is calibrated exactly to normal-day spend with no safety margin, which stacks the comparison. §5.4's ratio argument (cost-need and estimated value scale together under volume surge, so a fractional cap is correctly sized on both day types) is the real mechanism and is well explained — but it also reveals the result depends on the surge being value-proportional. Needed: (a) a sweep over α showing where the elastic cap begins to bind and how recall degrades relative to static
  4. [§4.2, Table 4 (with Eq. 2 and §3.1)] The within-category calibration collapse — called the paper's 'most important methodological finding' — appears guaranteed by the generative process rather than discovered by the monitor. If true difficulty d_i is sampled within a category range independently of price (the text states independence from value; the ~0 within-category correlations in Table 4, despite the β(c)·p_i/500 term in Eq. 2, indicate effective independence from all observables), then no estimator of category and price could have within-category discriminative power, and the monitor is rediscovering the simulator's design. The lesson — aggregate correlation can be a between-category artifact — is valid, but as constructed it cannot fail to appear. Two fixes: (a) state precisely how d_i is generated, including any price dependence; (b) include a control variant where difficulty has genuine within-category signal tied t
minor comments (5)
  1. [Throughout] Rendering: the estimator notation appears as 'b𝑑𝑖'/'b𝑣𝑖' (Eqs. 2–4 and §5.2), evidently a failed hat glyph; should read d̂_i, v̂_i. Also missing spaces at 'presentvalue-router' (abstract/§1) and 'measuredwithineach' (§4.2), and 'Both crush Random' (§3.6) is informal for the venue.
  2. [Availability / References [11]] The Availability section points to a GitHub repository, but reference [11] is titled and typed as a blog post with a different URL; please reconcile the citation with the actual artifact (ideally an archived release or DOI for the version used).
  3. [Table 3] The 'under'/'over' row annotations are not defined in the caption (they appear to denote spend share below/above volume share); the 'Slow %' column for commodity/accessory is 0.0 while 'Spend %' is 12.1/10.0, which needs a one-line explanation (fast-path spend) to avoid misreading.
  4. [§3.3, §3.5, Listing 2] Default thresholds τ_d=0.5 and τ_v=20 and the 20:1 cost ratio are asserted without motivation; since τ_v doubles as the evaluation label boundary, a brief note on how results shift with τ_v (or a pointer to the requested sensitivity analysis) is warranted. Listing 2 truncates the admission loop with '...' — the greedy highest-estimated-value-first rule should be stated in full, since ordering affects which high-value items are dropped when the cap binds.
  5. [Front matter / Figure 1] The manuscript uses a 'Conference'17, July 2017' ACM template header, and the keyword list (13 entries, several near-duplicates) is excessive; both should be cleaned up. Figure 1's 'Tiers 1–2' label is never defined as a tiering scheme before §5 introduces 'Tier 3.'

Circularity Check

2 steps flagged

Tier-1 precision gain is largely structural: the value gate is a near-oracle of the same quantity and threshold that define the evaluation label H.

specific steps
  1. self definitional [§3.1–3.5 Eqs. 1,3,4; Table 2; Table 4]
    "value_i = p_i · m_i. (1) ... v̂_i = p_i · (m̄(c_i) + ε_v), ε_v ~ N(0, 0.02) (3) ... path(i) = slow if d̂_i ≥ τ_d and v̂_i ≥ τ_v (4) ... Let H be the set of items with true value above τ_v ... precision = |H ∩ S| / |S| ... ValueWeighted ... Precision 98.3% ... DifficultyOnly ... 94.3% ... The value estimator's within-category correlation stays high (0.744–0.938)"

    H is defined by true value ≥ τ_v; the ValueWeighted gate is estimated value ≥ the same τ_v, and v̂ is a near-copy of v (σ=0.02 on margins; within-cat corr up to 0.938). Filtering DifficultyOnly's slow set by a near-oracle of the label definition forces precision up and leaves recall almost unchanged by construction. The reported 4-point precision win at matched recall is therefore largely the algebra of AND-ing the evaluation criterion onto the baseline, not an independent empirical discovery about routing under realistic value uncertainty.

  2. other [§2 contribution bullet 1; §3.6]
    "Section 3 shows this combination is not redundant: on a catalog with realistic volume/value structure, adding a value gate to an uncertainty-only router improves precision by 4 points at equal recall, i.e., value is not merely a proxy for difficulty even when the two happen to be correlated. ... ValueWeighted and DifficultyOnly achieve nearly identical recall (≈60%) – unsurprising on this catalog, since difficulty and value are positively correlated across categories by construction"

    The paper interprets the structural precision lift as evidence that value and difficulty are distinct non-redundant axes. But difficulty–value correlation is admitted to be by construction across categories, and the value gate is nearly the label itself—so the experiment cannot separate 'value is a distinct useful axis' from 'we AND-ed a near-perfect label proxy onto a category-level difficulty gate.' The non-redundancy claim overreaches what the coupled construction can show.

full rationale

This is a forward synthetic simulation graded against hidden ground truth, not a first-principles derivation, and most of the paper is non-circular: the within-category calibration collapse is an honest diagnostic of the authors' own weak difficulty scorer, and the elastic-budget result is a transparent mechanism demo (cap as fraction of that day's Σv̂) rather than a forced prediction. The load-bearing circularity is confined to the headline Tier-1 claim. True high-value set H is defined as true value v_i=p_i·m_i ≥ τ_v (Eq. 1, §3.5), while ValueWeighted admits to the slow path only if v̂_i≥τ_v with v̂_i=p_i·(m̄(c_i)+ε_v), ε_v~N(0,0.02) (Eqs. 3–4)—noise negligible against values spanning ~0.77–552 (Table 1), and the paper's own Table 4 reports within-category value correlations 0.744–0.938. AND-ing a near-label filter onto DifficultyOnly therefore mechanically strips hard-but-cheap false positives while almost never removing true positives, so equal recall and higher precision (98.3% vs 94.3%) is a structural consequence of coupling the gate to the label definition under tiny noise, not independent evidence that 'value is not merely a proxy for difficulty.' The realistic high-error value-estimate regime is untested. No self-citation chain or uniqueness import. Score 5 reflects partial by-construction reduction of the central precision claim, with the rest of the paper independently contentful.

Axiom & Free-Parameter Ledger

8 free parameters · 6 axioms · 4 invented entities

The central claims rest on a hand-built synthetic world and hand-chosen policy knobs, not on external empirical laws. Free parameters set thresholds, costs, noise, surge shape, and budget fractions. Domain assumptions encode that value and difficulty are separable axes and that category-level priors plus price are what a router can see. Invented entities are the simulator artifacts (catalog generative model, mock agents, Value Router policies), not new physical objects; they have no independent evidence outside this codebase.

free parameters (8)
  • difficulty threshold τ_d = 0.5
    Hand-set gate for slow path; default 0.5. Directly controls who enters the expensive path.
  • value threshold τ_v = 20
    Hand-set value gate; default 20. Primary lever for value-weighting vs difficulty-only.
  • slow/fast unit cost ratio = 1 : 20
    Arbitrary fixed costs c_fast=1, c_slow=20 used for budget and efficiency; only ratio matters but absolute choice affects reported budgets.
  • elastic budget fraction α = 0.08
    Cap = α × sum of day's value estimates; chosen 0.08 so elastic often unbound on peak days in the reported run.
  • fixed daily budget k = 540
    Constant cap calibrated off normal-day spend; set to 540 in reported peak comparison.
  • peak volume multiplier and category mix shifts = 2.5×; premium×2.0, luxury×2.5, commodity×0.7
    Hand-picked 2.5× volume and premium/luxury/commodity reweights; not fit to real seasonal data (§5.1, §6).
  • estimator noise and category baselines = ε_d~N(0,0.05), ε_v~N(0,0.02); category tables
    Gaussian noise σ_d=0.05, σ_v=0.02 and per-category difficulty baselines, β(c), mean margins define how informative estimates are.
  • calendar-aware value multiplier = 2.5× on peak
    Manual seasonal scaling of τ_v (reported 2.5× on peak) representing ops tuning.
axioms (6)
  • domain assumption Difficulty and business value are distinct routing axes; cost of error is not reducible to uncertainty alone.
    Motivates the whole paper (Abstract, §1); standard decision-theoretic idea applied to inference routing.
  • ad hoc to paper Ground-truth value equals price × margin; difficulty is sampled in category-specific ranges independent of value within category.
    §3.1 Eq. 1 and difficulty sampling define the hidden labels the eval harness uses.
  • ad hoc to paper Router may observe only category and price; estimators are category baselines plus price terms with noise (Eqs. 2–3).
    §3.2 encodes information asymmetry; drives the between- vs within-category calibration story.
  • domain assumption Catalog has inverse correlation between category volume and mean value (commodity high volume/low value, luxury opposite).
    §3.1 Table 1; presented as common e-commerce shape and the stress condition for naive routers.
  • domain assumption Aggregate calibration can mask subgroup miscalibration (invoked via Guo et al. and demonstrated on the difficulty scorer).
    Related work and §4.2; standard calibration fact used as monitoring principle.
  • ad hoc to paper Peak-day estimators are not recalibrated; robustness must come from routing/budget policy alone.
    §5.1 experimental rule that makes elastic vs calendar comparison meaningful.
invented entities (4)
  • Value Router (value-weighted dual-threshold routing policy) no independent evidence
    purpose: Send items to slow path only if estimated difficulty and estimated value both clear thresholds.
    Core proposed mechanism (§3.3 Eq. 4); a policy design, not a physical entity.
  • Synthetic five-category merchandising catalog simulator no independent evidence
    purpose: Provide observable fields, hidden value/difficulty, and controllable volume–value structure for grading routers.
    §3.1; entire evidence base is this generative model.
  • Elastic slow-path budget (cap = α × day's total estimated value) no independent evidence
    purpose: Adapt expensive-path spend to load without explicit season detection.
    §5.2 Listing 2 / Eq. 8; compared to fixed and calendar-aware policies.
  • Decision logger + within-segment calibration monitor no independent evidence
    purpose: Expose spend concentration and within-category estimate–truth correlation failures hidden by aggregates.
    §4; methodological artifact of the study.

pith-pipeline@v1.2.0-grok45-kimik3 · 15868 in / 4488 out tokens · 89837 ms · 2026-07-31T02:07:59.267166+00:00 · methodology

0 comments
read the original abstract

Routing decisions between a cheap heuristic and an expensive large language model (LLM) are typically framed as a difficulty problem: send the hard cases to the expensive path. We argue this framing is incomplete because difficulty and business value are distinct axes - a difficult cheap item and a difficult costly item do not have the same cost of error. We present Value Router, a fully synthetic simulation of a retail merchandising pipeline that routes items using only estimated difficulty and estimated value, never ground truth. The study has three stages. First, a value-weighted threshold router is compared with a difficulty-only and a random baseline on a synthetic catalog with an inverse correlation between category volume and value. Value-weighting matches the difficulty-only baseline's recall of true high-value items (60%) while achieving substantially higher precision (98.3% vs. 94.3%). Second, a decision logger and monitor expose a failure mode hidden by aggregate metrics showing that the aggregate result is driven almost entirely by between-category differences rather than per-item discrimination. Third, a simulated Black Friday demand surge (2.5 volume with a shift toward higher-value categories) compares a static router, a seasonally tuned router, and two slow-path budget policies. All results are from a controlled synthetic simulation with experimenter-defined ground truth and illustrate design principles for cost-aware routing systems rather than validated real-world claims.

Figures

Figures reproduced from arXiv: 2607.25068 by Bhavtosh Rath.

Figure 1
Figure 1. Figure 1: The core pipeline (Tiers 1–2). Only category and price are observable to the router; value and difficulty are hidden ground truth consumed only by the Eval Harness and Monitor, never by the router itself [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Tier 3 evaluation loop. Each day is scored and routed [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 6 linked inside Pith

  1. [1]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2020. FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 33. 10685–10696

    2020

  2. [2]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.arXiv preprint arXiv:2305.05176(2023)

    Pith/arXiv arXiv 2023

  3. [3]

    Lakshmanan, and Ahmed Hassan Awadallah

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V.S. Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hy- brid LLM: Cost-Efficient and Quality-Aware Query Routing. InInternational Conference on Learning Representations (ICLR)

    2024

  4. [4]

    Charles Elkan. 2001. The Foundations of Cost-Sensitive Learning. InProceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI), Vol. 2. 973–978

    2001

  5. [5]

    William Fedus, Barret Zoph, and Noam Shazeer. 2022. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.Journal of Machine Learning Research23, 120 (2022), 1–39

    2022

  6. [6]

    Weinberger

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibration of Modern Neural Networks. InProceedings of the 34th International Conference on Machine Learning (ICML). 1321–1330

    2017

  7. [7]

    Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran- Johnson, et al. 2022. Language Models (Mostly) Know What They Know.arXiv preprint arXiv:2207.05221(2022)

    Pith/arXiv arXiv 2022

  8. [8]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. InProceedings of the 40th International Conference on Machine Learning (ICML). 19274–19286

    2023

  9. [9]

    Aman Madaan, Pranjal Aggarwal, Ankit Anand, Srividya Pranavi Potharaju, Swa- roop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Krishna Kappaganthu, Yiming Yang, et al . 2023. AutoMix: Automatically Mixing Language Models. arXiv preprint arXiv:2310.12963(2023)

    Pith/arXiv arXiv 2023

  10. [10]

    Gonzalez, M

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2024. RouteLLM: Learning to Route LLMs with Preference Data.arXiv preprint arXiv:2406.18665(2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Bhavtosh Rath. 2026. value-router: Simulating Value-Weighted Routing Decisions over a Stream of Items. https://bhavtoshrath.github.io/posts/value-router-value- weighted-routing. Blog post

    2026

  12. [12]

    Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine- Tuned with Human Feedback. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing...

    2023

  13. [13]

    Neeraj Varshney and Chitta Baral. 2022. Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systems. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP). 11007– 11021

    2022

  14. [14]

    Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. 2023. Tabi: An Efficient Multi-Level Inference System for Large Language Models. InProceedings of the Eighteenth European Conference on Computer Systems (EuroSys). 233–248

    2023

  15. [15]

    Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. 2023. Large Language Model Cascades with Mixture of Thought Representations for Cost-Efficient Reasoning.arXiv preprint arXiv:2310.03094(2023)

    Pith/arXiv arXiv 2023

  16. [16]

    Awadallah, and Chi Wang

    Jieyu Zhang, Ranjay Krishna, Ahmed H. Awadallah, and Chi Wang. 2023. EcoAs- sistant: Using LLM Assistant More Affordably and Accurately.arXiv preprint arXiv:2310.03046(2023)

    Pith/arXiv arXiv 2023