Pith. sign in

REVIEW 4 major objections 5 minor 9 cited by

Adaptive LLM Routing under Budget Constraints

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that LLM routing can be learned online from binary user feedback using a preference-prior-informed linear bandit, and that this reaches GPT-4-level performance at a fraction of its cost while adapting to new queries.

desk verdict A useful bandit formulation for LLM routing with a solid empirical setup, but the linear reward assumption is untested, error bars are missing, and the theory has a fixable algebraic slip. read the letter →

arxiv 2508.21141 v2 pith:FVQIWQ7S submitted 2025-08-28 cs.LG

Pranoy Panda , Raghav Magazine , Chaitanya Devaguptapu , Sho Takemori , Vishal Sharma This is my paper
classification cs.LG
keywords LLMroutingcontextualbanditsLinUCBhumanpreferencedatabudgetconstraintsonlinemulti-choiceknapsackadaptiveRouterbench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that LLM routing can be treated as a budget-constrained contextual bandit problem, where the only supervision is binary feedback such as thumbs up/down on the selected model's answer. To learn from this sparse feedback, the authors build a shared embedding space for queries and LLMs, pretrain it on human preference data, and refine it online with a preference-prior-informed LinUCB algorithm called PILOT. They report that PILOT reaches 93% of GPT-4's performance at 25% of its cost on the multi-task Routerbench setting, with higher performance than the bandit baselines, and that their online multi-choice knapsack cost policy handles budget constraints better than fixed per-query allocation and generally matches a hindsight-tuned offline policy. If the claim is right, deployable routers no longer need exhaustive query-LLM labels and can adapt to changing query distributions through ordinary user feedback.

What carries the argument

The mechanism is PILOT, a LinUCB variant whose ridge-regression posterior is initialized with preference-prior embeddings (A0_a = λ_a I, b0_a = λ_a θ_pref_a). The expected reward is defined as the cosine similarity between a normalized query projection and an LLM embedding (Eq. 1), which is linear in the arm parameter and therefore fits the upper-confidence-bound and ridge-regression machinery of linear bandits. The companion online cost policy frames each query as a choice set in an online multi-choice knapsack problem and uses the ZCL algorithm to select the highest-reward LLM whose cost fits a budget-utilization threshold, with binned budgets and forward spillover.

What would settle it

On a held-out set of Routerbench queries, compute PILOT's predicted cosine affinity cos(ψ(q), θ_l) for every LLM and compare it with the recorded reward s(q, y_l). If the rank correlation is weak or the relation is strongly nonlinear, Eq. (1) is false and the UCB rule loses its justification; similarly, estimating θ* by ridge regression and checking whether ||θ_pref − θ*|| ≤ ||θ*|| holds would settle the Proposition C.1 comparison.

Watch

Extended reading notes

Core claim

The central claim is that a router initialized from human-preference embeddings and run as a linear contextual bandit can learn good routing from bandit feedback alone. The paper models the expected reward of routing query q to LLM a as the cosine similarity between normalized embeddings, E[r_t | a, q_t] = cos(ψ(q_t), θ_a), which makes the problem linear in the arm parameters. PILOT then runs ridge-regression UCB with the preference embeddings as the prior (A0_a = λ_a I, b0_a = λ_a θ_pref_a), so the arm estimate starts at the human-preference cosine affinity and moves toward observed rewards. In the multi-LLM Routerbench setting the paper reports performance equal to 93% of GPT-4 at 25% of i

Load-bearing premise

The load-bearing premise is that the true reward for routing a query to an LLM is a linear function of cosine similarity in the learned embedding space, and that the preference prior is at least as close to the true reward vector as the origin; if either fails, the UCB exploration bonuses and the regret comparison no longer transfer.

Editorial extensions

If this is right

  • Deployment no longer requires running every LLM on every training query; the router learns only from the selected model's reward.
  • Because the router updates online, it can track shifts in the query distribution instead of freezing at training time.
  • Budget control is separated from reward learning: the cost policy can be adjusted or replaced without retraining the bandit.
  • A good preference prior reduces the regret bound relative to standard OFUL, implying fewer online samples are needed to reach the same routing quality.
  • Routing overhead stays small relative to LLM inference time, so online adaptation is practical in a serving loop.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own setting takes the linear reward model in Eq. (1) as given and gives no diagnostic for it; a direct test would compare PILOT's cosine predictions with recorded scores on held-out Routerbench queries. If rewards are nonlinear in the embedding, a kernel or neural extension of the UCB would be needed.
  • The preference-prior bandit recipe transfers to other black-box model-selection problems—retrieval rerankers, code executors, image models—wherever a pool of services can be embedded and feedback is binary.
  • Because the paper explicitly separates bandit learning from the budget policy, an immediate extension is to fold costs into the bandit objective itself and learn under budget constraints, not just deploy under them.
  • The reported cost-performance frontier depends on the Routerbench score function and the embedding model; rerunning PILOT with human ratings or task-specific metrics as the reward would show whether the 93%-at-25%-cost result is a property of the algorithm or of that scoring choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes to reformulate LLM routing as a budget-constrained contextual bandit problem, where for each query the router selects one LLM and observes binary-quality feedback only for the selected arm. The proposed method, PILOT, (1) pretrains a shared query/LLM embedding space using offline human preference data from ChatArena with a triplet/BCE objective, (2) initializes per-arm ridge-regression parameters in LinUCB with these embeddings, and (3) adds an online multi-choice knapsack cost policy (ZCL-style) for deployment under a token-cost budget. Experiments on Routerbench compare PILOT with LinUCB, Epoch-Greedy, explore-only, random, and all-to-one baselines; the paper reports 93% of GPT-4's performance at 25% of its cost in the multi-task setting. Appendix C.1 claims a regret-bound improvement for a preference-prior-initialized OFUL algorithm under the condition ||theta_pref - theta*|| <= ||theta*||.

Significance. The formulation is a useful step: it removes the need for exhaustive query-LLM labeling and permits online adaptation to query drift. The held-out deployment protocol is a reasonable way to simulate a two-bucket system, and using ChatArena as an external preference prior avoids circular evaluation. If the empirical results are robust, the method could be practically valuable. However, the central linear reward model is not validated, the theoretical proposition contains an algebraic error, the stated budget-constrained problem is only solved at deployment, and the empirical claims rest on single runs without uncertainty estimates. These issues must be addressed before the contribution can be fully assessed.

major comments (4)
  1. [Section 2.2.2, Eq. (1)] The paper assumes E[rt | a, qt] = cos(psi_hat(qt), theta_hat_a), and this exact linearity is used for UCB selection, ridge-regression posterior, and cost-policy value estimates. No diagnostic is provided that Routerbench rewards (GPT-4 eval or exact match) are linear in the preference-pretrained embedding space, or that the cosine scores are even monotonically related to rewards. Please add a calibration/correlation analysis on the tuning set (e.g., Spearman correlation, binned calibration plots, or a comparison against a non-linear reward model). Without this, the exploration bonus is not a valid confidence bound and the headline 93%/25% result is not theoretically supported.
  2. [Appendix C.1, proof of Proposition C.1] The displayed expression for tilde_theta_t is dimensionally incorrect: the term lambda(X^T X + lambda I)(theta* - theta_pref) is missing the inverse (X^T X + lambda I)^{-1}. The correct form is theta* + (X^T X + lambda I)^{-1} X^T epsilon + lambda (X^T X + lambda I)^{-1}(theta_pref - theta*). This error propagates to the confidence-bound derivation. In addition, the condition ||theta_pref - theta*|| <= ||theta*|| is not verified for the actual pretrained embeddings; without it, Proposition C.1 does not apply. Please fix the proof and report whether the learned theta_pref is closer to theta* than the zero vector is, or discuss how this condition can be checked.
  3. [Section 2.1 vs Algorithm 1 / Limitations] The problem statement defines an objective with a budget constraint over Q queries, but Algorithm 1 runs unconstrained and the cost policy is applied only after learning, as the Limitations section acknowledges. The stated problem is therefore not solved as posed. Either revise the problem formulation to explicitly define a two-phase protocol (unconstrained bandit learning followed by constrained deployment) or modify the algorithm to respect budgets during learning. As written, contribution (i) overclaims the scope of the proposed solution.
  4. [Section 4 and Tables 5-6] All empirical results appear to be single-run; no confidence intervals, standard errors, or repeated-seed averages are reported. For a stochastic bandit algorithm, the reported differences between PILOT and LinUCB (e.g., Table 5: $1 budget 0.63 vs 0.60; $3 budget 0.73 vs 0.68) may be within noise. Please run multiple random seeds (at least 5) and report mean +/- standard deviation, or apply paired significance tests. This is necessary to support the central performance claim.
minor comments (5)
  1. [Section 2.2.3] The upper/lower bounds UB and LB on the reward-to-cost ratio are never specified, although the ZCL-style guarantee relies on them. Please state the values used and report sensitivity to them.
  2. [Algorithm 3] The 'Insufficient budget' termination path is not analyzed. What happens to the remaining queries in the bin, and how often does this branch occur in the experiments?
  3. [Section 2.2.2] The setting 'lambda_a as the inverse of arm a's accuracy during the pretraining phase' is not precisely defined. Please specify how the accuracy is measured and how the inverse is computed.
  4. [General] Typos and formatting issues: 'birds eye', 'PILOTadds', 'maybe be interested', 'worth noting', and the threshold formula in Algorithm 2 is rendered without necessary parentheses. Figure 3 is dense and the sub-column labels should be more legible.
  5. [References] The paper refers to 'ChatArena' but the cited reference is Chatbot Arena (Chiang et al., 2024). Please align the terminology with the citation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical routing result is measured on held-out Routerbench data after pretraining on an external preference dataset, and the theoretical prior-benefit claim is an explicitly conditional statement rather than a fitted prediction.

full rationale

The central empirical claim (93% of GPT-4 performance at 25% cost) is not circular. The preference prior is learned from ChatArena, a dataset distinct from Routerbench, and the evaluation is performed on a held-out deployment bucket after online bandit learning. The cost policy is imported from the external ZCL/online-knapsack result of Zhou et al. (2008) and is applied uniformly to all baselines, so it does not encode PILOT's own fitted values as a prediction. The linear reward model in Eq. (1), E[r_t | a, q_t] = cos(ψ(q_t), θ_a), is an explicit modeling assumption; it is not derived from, nor fitted to, the deployment rewards being predicted, so any misspecification is a correctness/validity concern rather than a circularity. Proposition C.1 is conditional: it states that IF ||θ_pref − θ*|| ≤ ||θ*||, THEN the preference-prior-initialized OFUL has a smaller regret bound. This is a monotonicity statement about the regret bound and does not assume the conclusion it purports to establish; the unverified condition is a limitation of the theory, not an instance of the paper predicting its own inputs. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no renaming of a known result as a new organizational principle. The paper's stated limitations, such as decoupling bandit learning from the budget-constrained deployment phase and focusing on single-turn queries, are acknowledged design choices rather than circular steps.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central method rests on a small set of unvalidated premises: cosine similarity is a faithful linear reward model, preference data transfers to the deployment task, and the online knapsack's UB/LB bounds are known. The learned projection and LLM embeddings are not counted as free parameters here because they are trained on separate data, but the hyperparameters controlling the prior, exploration, and budget policy are genuinely free and underreported.

free parameters (5)
  • exploration parameter alpha = selected from {1, 1.5, 2, 5, 10} on tuning data
    Balances UCB exploration and exploitation for PILOT and LinUCB; Section 3.2 reports tuning but not the selected values or sensitivity across datasets.
  • per-arm regularization lambda_a = inverse of arm a's pretraining accuracy, value not reported
    Controls the strength of the preference prior in the ridge regression initialization (Section 2.2.2); no formula or measured accuracies are reported.
  • reward-to-cost bounds UB, LB
    Assumed known in the online multi-choice knapsack policy (Section 2.2.3); values never reported, yet they shape the eligibility threshold and the ZCL guarantee.
  • bin size S
    Algorithm 3 partitions Q queries into bins of size S to handle the finite horizon; S is not specified or swept.
  • mean output token count per LLM = mean from tuning data
    Used to estimate total query cost at deployment (Section 3.2); assumes tuning responses predict deployment output lengths.
assumptions (6)
  • domain assumption E[r_t | a, q_t] = cos(psi(q_t), theta_a), a linear reward model in the projected embedding space (Eq. 1).
    PILOT and its UCB rule rely on this linearity; the paper does not validate that Routerbench rewards are well approximated by cosine affinity.
  • domain assumption Human preference data (ChatArena) transfers to the target routing task and provides a prior closer to the true reward vector than zero (Prop. C.1 condition ||theta_pref - theta*|| <= ||theta*||).
    This transfer is the justification for pretraining; the closeness condition is stated, not measured.
  • domain assumption Known upper/lower bounds UB, LB on the reward-to-cost ratio and query costs small relative to budget B.
    Required for the online knapsack guarantee from Zhou et al. (Section 2.2.3); values not provided.
  • ad hoc to paper Hard negatives are defined by model size: queries where the preferred LLM lost to a smaller LLM are hard negatives.
    Section 2.2.1 defines the negative pool using size(l_w) < size(l_win); this is a heuristic choice not justified by data.
  • standard math OFUL regret bound of Abbasi-Yadkori et al. applies to the per-arm ridge regression setting.
    The regret comparison in Prop. C.1 inherits Theorems 8 and 13 of Abbasi-Yadkori et al. 2011b, assumed valid.
  • domain assumption Binary feedback r_t = s(q_t, y_l_t) is available immediately after each selected response and is an unbiased reward signal.
    The bandit formulation requires evaluative feedback for the selected model; delayed or noisy feedback is not modeled.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive LLM Routing under Budget Constraints." pith.science (2026). https://pith.science/paper/FVQIWQ7S

@misc{pith2026250821141,
  author       = {Pith},
  title        = {Pith review of: Adaptive LLM Routing under Budget Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FVQIWQ7S}},
  note         = {Machine review of arXiv:2508.21141}
}
read the original abstract

Large Language Models (LLMs) have revolutionized natural language processing, but their varying capabilities and costs pose challenges in practical applications. LLM routing addresses this by dynamically selecting the most suitable LLM for each query/task. Previous approaches treat this as a supervised learning problem, assuming complete knowledge of optimal query-LLM pairings. However, real-world scenarios lack such comprehensive mappings and face evolving user queries. We thus propose to study LLM routing as a contextual bandit problem, enabling adaptive decision-making using bandit feedback without requiring exhaustive inference across all LLMs for all queries (in contrast to supervised routing). To address this problem, we develop a shared embedding space for queries and LLMs, where query and LLM embeddings are aligned to reflect their affinity. This space is initially learned from offline human preference data and refined through online bandit feedback. We instantiate this idea through Preference-prior Informed Linucb fOr adaptive rouTing (PILOT), a novel extension of LinUCB. To handle diverse user budgets for model routing, we introduce an online cost policy modeled as a multi-choice knapsack problem, ensuring resource-efficient routing.

Figures

Figures reproduced from arXiv: 2508.21141 by the authors.

Figure 1
Figure 1. Pretraining with Human Preference Data ⃝1 We leverage human preference dataset to learn query embeddings which are aligned w.r.t. human pref￾erences on query-LLM mapping. Then, in ⃝2 we learn LLM embeddings aligned with projected queries 2.2.1 Pretraining with Human Preferences Human preference data provides rich insights into query-LLM fit. We use it to establish a meaningful shared embedding space before incorpora… view at source ↗
Figure 2
Figure 2. Bandit Router Framework: Our router takes three inputs: (i) User query (ii) cost constraints and, (iii) a model pool. It learns and adapts automatically based on user feedback, optimizing LLM selection over time. 2.2.2 Evolving with Online Bandit Feedback Having learned query and LLM embeddings, we will now discuss how to incorporate the online ban￾dit feedback for the end task of routing queries to appropriate LLMs… view at source ↗
Figure 3
Figure 3. Bandit Feedback based LLM Routing Evaluation: In column (a) we report results for single task data source setting (MMLU), and in column (b) we report results for multi-task data source setting (Routerbench). The sub-column (i) in each column represents performance vs cost curves on the held-out deployment set; sub-column (ii) represents performance across different learning bucket sizes; sub-column (iii) represents … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Cost Policy Comparison: (Left) Mean Re￾ciprocal Rank of the chosen arms for various budgets (Right) Performance of cost policies with diff budget Cost ($) P − λC PILOT (Ours) Difference 0.25 0.6079 0.6557 +0.0478 0.50 0.6602 0.6840 +0.0238 1.00 0.7265 0.7240 -0.0025 1.…
Figure 6
Figure 6. Figure 6: Performance vs Cost comparison with Su￾pervised HybridLLM (Ding et al. 2024) The left figure shows binary LLM routing comparisons for GPT￾4 and Mistral-7b in the LLM pool. The right figure presents similar comparisons, this time for GPT-4 and Mixtral-8x7b. This study u…
Figure 7
Figure 7. Figure 7: Adaptability to Shift in Query Distribution The left figure shows average reward comparison across different time instants - “Before", “During" and “After" shift in query distribution from MMLU to GSM8k. The right figure shows that during the drift the exploration incr…
Figure 8
Figure 8. Figure 8: Bandit Feedback based Binary LLM Rout￾ing Evaluation: Figure reports results for multi task data source setting (Routerbench). In the top row we have Claude-v1 and Mixtral-7x8b LLMs in the LLM pool, and in the botton row we have GPT-4 and Llama2- 70b LLMs in the LLM po…

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    MCPP is a Monte Carlo simulation-based online planner that improves the probability of agentic workflows completing successfully under explicit budget and deadline constraints compared to baselines on CodeFlow and Pro...

  2. ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Serving

    cs.LG 2026-03 unverdicted novelty 7.0 of 10

    ParetoBandit uses contextual bandits with an online primal-dual budget pacer and geometric forgetting to enforce cost ceilings and adapt to non-stationary pricing and quality shifts in LLM serving, achieving 0.4% budg...

  3. RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Serving

    cs.DC 2026-06 unverdicted novelty 6.0 of 10

    RouteBalance fuses routing and load balancing for heterogeneous LLM serving and traces the upper quality-cost-throughput frontier on a 13-instance 28-GPU cluster.

  4. Learning Agent Routing From Early Experience

    cs.CL 2026-05 unverdicted novelty 6.0 of 10

    BoundaryRouter routes queries to LLM or agent using early experience memory from a seed set, cutting inference time 60.6% versus always using agents and raising performance 28.6% versus always using direct LLM inference.

  5. On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    MCPP uses Monte Carlo simulations of workflow executions to dynamically allocate resources and replan, raising constrained completion probability over baselines on CodeFlow and ProofFlow.

  6. Zero-Shot Confidence Estimation for Small LLMs: When Supervised Baselines Aren't Worth Training

    cs.AI 2026-05 conditional novelty 6.0 of 10

    Average token log-probability provides a zero-shot confidence signal for small LLMs that matches supervised baselines in-distribution and outperforms them out-of-distribution, with a new retrieval-conditional variant ...

  7. Adaptive Test-Time Compute Allocation for Reasoning LLMs via Constrained Policy Optimization

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    A Lagrangian-relaxation plus imitation-learning pipeline adaptively allocates test-time compute to LLMs, outperforming uniform baselines by up to 12.8% relative accuracy on MATH while staying within a fixed average budget.

  8. Learning to Assign Prediction Tasks to Agents with Capacity Constraints

    cs.HC 2026-05 unverdicted novelty 5.0 of 10

    Sequential explore-exploit algorithms for assigning tasks to capacity-constrained agents demonstrate performance gains over non-contextual baselines on tabular, image, and text tasks with both LLMs and humans.

  9. Retrieval-Conditioned Topology Selection with Provable Budget Conservation for Multi-Agent Code Generation

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    RGAO combines retrieval-based complexity assessment with a formal budget algebra to enable dynamic topology selection in multi-agent code generation with provable conservation.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages · cited by 8 Pith papers

  1. [1]

    Yasin Abbasi-Yadkori, D \'a vid P \'a l, and Csaba Szepesv \'a ri. 2011 a . Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24

    work page 2011

  2. [2]

    Yasin Abbasi-Yadkori, D \'a vid P \'a l, and Csaba Szepesv \'a ri. 2011 b . Online least squares estimation with self-normalized processes: An application to bandit problems . arXiv preprint arXiv:1102.2670

    arXiv 2011

  3. [3]

    Appcues. 2024. https://www.appcues.com/blog/rating-system-ux-star-thumbs Rating systems in ux: Star vs. thumbs up . Appcues Blog

    work page 2024

  4. [4]

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. 2024. Simple linear attention language models balance the recall-throughput tradeoff. In Forty-first International Conference on Machine Learning

    work page 2024

  5. [5]

    Djallel Bouneffouf and Irina Rish. 2019. A survey on practical applications of multi-armed and contextual bandits. arXiv preprint arXiv:1904.10040

    arXiv 2019

  6. [6]

    Deeparnab Chakrabarty, Yunhong Zhou, and Rajan Lukose. 2008. Online knapsack problems. In Workshop on internet and network economics (WINE), pages 1--9

    work page 2008

  7. [7]

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and 1 others. 2024. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations

    work page 2024

  8. [8]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2022. Efficient online ml api selection for multi-label classification tasks. In International conference on machine learning, pages 3716--3746. PMLR

    work page 2022

Show all 34 references
  1. [9]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176

    2023 arXiv

  2. [10]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. https://arxiv.org/abs/2403.04132 Chatbot arena: An open platform for evaluating llms by human pr...

    2024 arXiv

  3. [11]

    Yu-Neng Chuang, Helen Zhou, Prathusha Sarma, Parikshit Gopalan, John Boccio, Sara Bolouki, and Xia Hu. 2025. Learning to route llms with confidence tokens. arXiv preprint arXiv:2410.13284, 3

    2025 arXiv

  4. [12]

    Delighted. 2024. https://delighted.com/blog/7-customer-experience-survey-types 7 types of customer experience surveys you should know about . Delighted Blog

    2024

  5. [13]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks VS Lakshmanan, and Ahmed Hassan Awadallah. 2024. Hybrid llm: Cost-efficient and quality-aware query routing. arXiv preprint arXiv:2404.14618

    2024 arXiv

  6. [14]

    Tao Feng, Yanzhen Shen, and Jiaxuan You. 2024. Graphrouter: A graph-based router for llm selections. arXiv preprint arXiv:2410.03834

    2024 arXiv

  7. [15]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  8. [16]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv e-prints, pages arXiv--2106

    2021

  9. [17]

    Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. 2024. Routerbench: A benchmark for multi-llm routing system. arXiv preprint arXiv:2403.12031

    2024 arXiv

  10. [18]

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vis...

    2018

  11. [19]

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165--14178

    2023

  12. [20]

    John Langford and Tong Zhang. 2007. The epoch-greedy algorithm for contextual multi-armed bandits. Advances in neural information processing systems, 20(1):96--1

    2007

  13. [21]

    Tor Lattimore and Csaba Szepesv \'a ri. 2020. Bandit algorithms. Cambridge University Press

    2020

  14. [22]

    Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in neural information processing systems, 2

    1989

  15. [23]

    Baolin Li, Yankai Jiang, Vijay Gadepally, and Devesh Tiwari. 2024. Llm inference serving: Survey of recent advances and opportunities. arXiv preprint arXiv:2407.12391

    2024 arXiv

  16. [24]

    Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on World wide web, pages 661--670

    2010

  17. [25]

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations

    2024

  18. [26]

    Xiaoding Lu, Adian Liusie, Vyas Raina, Yuwen Zhang, and William Beauchamp. 2024. Blending is all you need: Cheaper, better alternative to trillion-parameters llm. arXiv preprint arXiv:2401.02994

    2024 arXiv

  19. [27]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. https://arxiv.org/abs/2406.18665 Routellm: Learning to route llms with preference data . Preprint, arXiv:2406.18665

    2024 arXiv

  20. [28]

    Marija S akota, Maxime Peyrard, and Robert West. 2024. Fly-swat or cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 606--615

    2024

  21. [29]

    Dimitris Stripelis, Zhaozhuo Xu, Zijian Hu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Jipeng Zhang, Tong Zhang, Salman Avestimehr, and Chaoyang He. 2024. Tensoropera router: A multi-model router for efficient llm inference. In EMNLP (Industry Track)

    2024

  22. [30]

    Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text embeddings. In Findings of the Association for Computational Linguistics: ACL 2023, pa...

    2023

  23. [31]

    Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, and 1 others. 2024. A careful examination of large language model performance on grade school arithmetic. Advances in Neural Information Proces...

    2024

  24. [32]

    Yunhong Zhou, Deeparnab Chakrabarty, and Rajan Lukose. 2008. Budget constrained bidding in keyword auctions and online knapsack problems. In Proceedings of the 17th international conference on world wide web, pages 1243--1244

    2008

  25. [33]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  26. [34]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.