Pith. sign in

REVIEW 4 major objections 5 minor 61 references

Economic Evaluation of LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that model choice should be decided by expected dollar reward, and that on hard MATH questions the most capable models win once a mistake costs a few cents.

desk verdict Useful economic lens with correct but standard math; the headline thresholds are unreliable because of an abstract/body inconsistency and a single unvalidated judge. read the letter →

arxiv 2507.03834 v1 pith:PVQFLF7H submitted 2025-07-04 cs.AI

Michael J. Zellinger , Matt Thomson This is my paper
classification cs.AI
keywords economicevaluationlargelanguagemodelspriceoferrorLLMcascadesreasoningParetofrontierMATHbenchmarkself-verification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes replacing Pareto-frontier accuracy-cost plots with a single expected reward, expressed in dollars, that penalizes errors, latency, and abstention at user-chosen prices. Applying this framework to six LLMs on difficult MATH questions, it reports that reasoning models beat non-reasoning models once a mistake costs more than roughly one to twenty cents, and that a single large model beats a cascade once a mistake costs more than about ten cents. The paper's broader conclusion is that when AI is automating human work, the most powerful available model is usually the right choice, because inference costs are small compared with the economic cost of errors. A sympathetic reader would take the contribution to be a method for turning multi-objective model selection into a single dollar-denominated decision, with the empirical thresholds as evidence that accuracy, not API price, dominates that decision.

What carries the argument

The machinery is the per-query reward $r = -(C + \lambda_L L + \lambda_E \mathbf{1}_E + \lambda_A \mathbf{1}_A)$, whose expectation over queries is maximized over model identity or cascade threshold; the $\lambda$'s are shadow prices expressing how much the user would pay to avoid one error, one second of latency, or one abstention. Theorems 1 and 2 connect this scalarization to Pareto optimality, showing that sweeping $\lambda$ recovers the Pareto surface and that reward dominance implies Pareto-surface dominance. For cascades, the paper proves a decomposition of cascade error into base error rates plus the difference of two covariances, and defines the cascade error reduction $\mathrm{CER} = \mathrm{Cov}(\mathbf{1}_D, \mathbf{1}^{\text{Msmall}}_{\text{error}})$, which measures how well the small model's deferral flag tracks its own mistakes; this covariance, not cost or latency, explains why Llama3.1 405B makes a better cascade partner than its standalone accuracy would suggest.

What would settle it

Have human graders evaluate the same difficult MATH answers and recompute the expected-reward crossover between reasoning and non-reasoning models; if the crossover moves above $1 per error, or if a cascade beats the single big model at error prices below $0.10 under human labels, the paper's headline thresholds fail. A complementary check is to rerun on the MATH test split, which the models are less likely to have memorized.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a small number of economic prices—price of error $\lambda_E$, price of latency $\lambda_L$, price of abstention $\lambda_A$—are enough to turn LLM selection into a well-posed optimization problem, and that under those prices the accuracy differences between today's models outweigh their cost differences. On difficult MATH questions, averaging over three reasoning and three non-reasoning models, the reasoning category crosses over at a critical price of error of $\$0.01$ in the latency-free setup stated in the abstract, while the body-text plot shows the crossover at about $\$0.20$; with latency priced at $\$10/$minute, the crossover rises to $\$10$ in the introduction and to $\$100$ in the sensitivity map. For cascades, sending every query to Qwen3 235B-A22B beats the Llama3.3 70B and GPT-4.1 cascades once the price of error exceeds $\$0.10$, with the crossover rising as latency is priced; the exception is Llama3.1 405B as the small model, whose self-verification signal makes its cascade win across most of the tested economic grid even though it is the weakest standalone model.

Load-bearing premise

The load-bearing premise is that Llama3.1 405B's correctness judgments and the MATH training-split labels are reliable; if that judge is wrong or the models memorized the questions, every error rate and every crossover price is off.

Editorial extensions

If this is right

  • If the empirical thresholds hold, then for any use case where an error costs more than a few cents per query, the rational deployment is a top reasoning model, and minimizing API spend is a false economy.
  • Cascades are only worth building when the small model has a genuinely informative uncertainty signal; otherwise the extra deferral machinery does not pay for itself at realistic error prices.
  • For medical-diagnosis-style use cases, where the paper estimates a price of error around $333, the framework selects the most powerful model over any cascade, and this conclusion is not sensitive to the exact estimate.
  • The framework converts the vague advice 'consider accuracy and cost' into a sensitivity table over $(\lambda_E, \lambda_L)$, so a practitioner with a known wage and error cost can read off the optimal model without a Pareto plot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same framework, applied with human-judged labels on a held-out MATH test split, could shift the critical thresholds by an order of magnitude, so the headline dollar figures should be read as point estimates tied to the Llama3.1 405B judge.
  • Inference: because the framework only needs per-query cost, latency, and correctness, it transfers to code generation, medical note-taking, or legal drafting; the thresholds would be benchmark-specific, but the qualitative conclusion that error cost dominates inference cost should become stronger as API prices fall.
  • Inference: the CER view suggests that model selection for cascades should be based on self-verification quality rather than standalone accuracy, which could be tested by training small models specifically to have calibrated uncertainty and measuring whether cascade wins extend to higher error prices.
  • Inference: if future models reduce error rates further, the empirical thresholds would drop, reinforcing the paper's conclusion; conversely, if a benchmark is found where cheap models are not much worse than expensive ones, the thresholds could rise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces an economic framework for evaluating LLMs and LLM systems by scalarizing accuracy, cost, latency, and abstention into a per-query dollar-denominated reward. It applies this framework to compare reasoning versus non-reasoning models on difficult MATH problems, and to compare single large models against cascades. The central empirical claims are that reasoning models become preferable once the price of error exceeds a small threshold (given as $0.01 in the abstract and $0.20 in Section 4.3), and that a single large model usually beats a cascade once the price of error exceeds roughly $0.10. The paper also provides theoretical results connecting the economic scalarization to Pareto optimality and giving an algebraic identity for cascade error rates.

Significance. If the central claims were robust, the paper would provide practically useful guidance: rather than minimizing inference cost, practitioners should often deploy the most capable model because error costs dominate. The framework itself is a straightforward weighted-sum scalarization, but the paper's contribution lies in translating it into explicit dollar thresholds and applying it to current models. The theoretical results are simple and mostly correct, and the empirical measurements use real API pricing and latencies. However, the headline economic thresholds rest on a small, single-judge, possibly contaminated evaluation, and the reported threshold values are internally inconsistent. The paper would benefit from releasing code and data to support reproducibility.

major comments (4)
  1. [Abstract, Section 4.3, Section 6] The headline threshold for when reasoning models beat non-reasoning models is inconsistent: the abstract and Section 6 state $0.01, while Section 4.3 and Figure 3 state $0.20 (with the figure caption reporting $0.20 for level 3 and $0.14 for level 5). This is not a cosmetic discrepancy because the abstract's $0.01 is the quantity a practitioner would carry away, and the body's own figure contradicts it. Please reconcile the two values and state which quantity is supported by the data.
  2. [Section 4.1, Section 7] All correctness labels are produced by Llama3.1 405B, and the paper does not validate this judge against human labels or against a second judge. Since the judge is also one of the evaluated models, this creates a direct risk of non-uniform bias across models, especially for the self-verification results in Section 4.4. The contamination risk from using the MATH training split is acknowledged in Section 4.1 and Section 7, but the paper still reports precise dollar thresholds. Please provide a human or alternative-judge validation subset, or at minimum quantify judge disagreement and show how the critical thresholds shift under alternative labels.
  3. [Section 4.1, Section 4.4] The cascade comparison is based on a test set of n=250 queries with no confidence intervals or error bars reported (Figure 5). The claim that a single large model beats a cascade for prices of error as low as $0.10 is therefore presented without any statistical uncertainty. Please report bootstrap or other confidence intervals for the critical price-of-error values and for the reward curves, and clarify whether the $0.10 threshold is within the noise of a 250-sample estimate.
  4. [Section 4.4, Section 6] The paper's broad conclusion that practitioners should 'typically use the most powerful available model' is undercut by its own cascade results: Section 4.4 reports that Llama3.1 405B -> Qwen3 235B-A22B outperforms the standalone big model for prices of error up to $10,000 and latencies up to $10/minute, which is the vast majority of the considered scenario grid. This is not a minor caveat; it directly contradicts the 'typically' in the conclusion. Please either revise the conclusion to reflect the significant exception or provide a clearer explanation of why this exception is not practically decisive.
minor comments (5)
  1. [Section 1] There is a grammatical error: 'we model a concrete use cases' should be 'we model a concrete use case'.
  2. [Section 3.3] The text refers to 'sections 4.3 and 4.3'; the second reference should presumably be to Section 4.4.
  3. [Figure 5] The caption for panel (c) says 'Llama3.3 405B', but the model is Llama3.1 405B; please fix this typo.
  4. [Section 6] The conclusion contains 'Mbig → Mbig', which should read 'Msmall → Mbig'.
  5. [Section 4.4, Figure 6] The text says CER = Cov(1D, 1Msmall_error), but Figure 6 uses 'dCov'; please make the notation consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the critical price-of-error thresholds are solved from measured error rates, costs, and latencies, and the cascade-error identity is proved rather than used to fit the headline results.

full rationale

The paper's central empirical claims are derived from directly measured quantities: error rates, per-query dollar costs, and latencies are recorded from API calls, and the critical prices of error are obtained by solving for the crossing points of expected-reward curves. These thresholds are not fitted parameters that are then reported as predictions; they are consequences of the measured means. The economic framework itself is an explicit linear scalarization, and the paper acknowledges that this corresponds to the standard weighted-sum method for Pareto frontiers, so it is not presenting a known method as a new derivation. The cascade error formula in Theorem 3 is an algebraic identity proved in the text from the definitions of the deferral indicator and the error indicators; the CER metric is definitionally related to cascade error, which makes some explanatory commentary near-tautological, but the cascade comparisons in Section 4.4 are measured directly on a held-out test split (n = 250) rather than derived from CER. The self-citations to Zellinger and Thomson (2024, 2025) for the cascade formula and for self-verification are not load-bearing: the formula is re-proved in this paper, and self-verification is an established external method (Kadavath et al., 2022). The unvalidated Llama3.1 405B correctness judge and the use of the MATH training split are legitimate correctness and robustness risks, but they do not make the derivation circular. The discrepancy between the abstract's $0.01 threshold and Section 4.3's $0.20 is an internal inconsistency, not circularity. Overall, the derivation chain is self-contained with respect to the framework's mathematics and the empirical measurements.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central framework has no fitted constants; it takes the user-supplied economic prices (λE, λL, λA) as inputs and computes thresholds from measured error rates, costs, and latencies. The results inherit the reliability of the correctness judge and the representativeness of the benchmark, as listed in the axioms.

assumptions (6)
  • domain assumption The expected per-query reward R(λ; θ) = -E[C + λL L + λE 1E + λA 1A] is the correct objective for selecting an LLM.
    Section 3.2 defines the reward; this presumes all relevant costs are linear and additive in error, latency, and abstention.
  • domain assumption The binary error-calling mechanism s(y), implemented by Llama3.1 405B with ground-truth reference answers, is reliable.
    Section 4.1; biased judge labels would change every error rate and threshold.
  • domain assumption The MATH level-5 questions and the six models tested are representative of meaningful human tasks for the general recommendation.
    Section 6 extrapolates from one benchmark to all meaningful human tasks.
  • standard math Regularity conditions (effectively convexity of the attainable performance set) hold so that scalarized solutions cover the Pareto surface.
    Theorems 1 and 2 (Section 3.5, Appendix B) assume this without verification for LLM systems.
  • domain assumption Self-verification P(True) provides a valid confidence signal for cascade deferral decisions.
    Appendix D builds on Kadavath et al. (2022) and Zellinger and Thomson (2025); if the confidence signal is miscalibrated, cascade comparisons change.
  • domain assumption Data contamination in the training split of MATH does not materially change the relative ranking of the models.
    Section 7 acknowledges the contamination risk and assumes it does not change the conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Economic Evaluation of LLMs." pith.science (2026). https://pith.science/paper/PVQFLF7H

@misc{pith2026250703834,
  author       = {Pith},
  title        = {Pith review of: Economic Evaluation of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PVQFLF7H}},
  note         = {Machine review of arXiv:2507.03834}
}
abstract

Practitioners often navigate LLM performance trade-offs by plotting Pareto frontiers of optimal accuracy-cost trade-offs. However, this approach offers no way to compare between LLMs with distinct strengths and weaknesses: for example, a cheap, error-prone model vs a pricey but accurate one. To address this gap, we propose economic evaluation of LLMs. Our framework quantifies the performance trade-off of an LLM as a single number based on the economic constraints of a concrete use case, all expressed in dollars: the cost of making a mistake, the cost of incremental latency, and the cost of abstaining from a query. We apply our economic evaluation framework to compare the performance of reasoning and non-reasoning models on difficult questions from the MATH benchmark, discovering that reasoning models offer better accuracy-cost tradeoffs as soon as the economic cost of a mistake exceeds \$0.01. In addition, we find that single large LLMs often outperform cascades when the cost of making a mistake is as low as \$0.1. Overall, our findings suggest that when automating meaningful human tasks with AI models, practitioners should typically use the most powerful available model, rather than attempt to minimize AI deployment costs, since deployment costs are likely dwarfed by the economic impact of AI errors.

Figures

Figures reproduced from arXiv: 2507.03834 by the authors.

Figure 1
Figure 1. Pareto frontiers of LLM performance do not reveal which model is best-suited for [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. On the most difficult questions of the MATH benchmark, reasoning models have [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Reasoning models offer superior accuracy-cost trade-offs as soon as the price of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Optimal models for different combinations of [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Directly sending queries to Qwen3 235B-A22B ( [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Using Llama3.1 405B as Msmall yields superior cascading performance, de￾spite its subpar performance as a standalone model, because it performs better at self￾verification—indicated by a higher cascade error reduction (CER) (a). In contrast to error rate, cost and late…
Figure 7
Figure 7. Figure 7: Non-reasoning models prompted with chain-of-thought exhibit similar scaling [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 46 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., Upadhyay, S., Faruqui, M., and Mausam (2024)

    Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., Upadhyay, S., Faruqui, M., and Mausam (2024). Automix: Automatically mixing language models

    work page 2024

  3. [3]

    Qwen3 technical report

    Alibaba AI (2025). Qwen3 technical report

    work page 2025

  4. [4]

    and Volkwein, S

    Banholzer, S. and Volkwein, S. (2019). Hierarchical convex multiobjective optimization by the euclidean reference point method. Preprint SPP1962-117

    work page 2019

  5. [5]

    Bertsekas, D. P. (1999). Nonlinear Programming . Athena Scientific, Belmont, MA, 2 edition

    work page 1999

  6. [6]

    Multiobjective Optimization , volume 5252 of Lecture Notes in Computer Science

    Branke, J., Deb, K., Miettinen, K., and Słowiński, R., editors (2008). Multiobjective Optimization , volume 5252 of Lecture Notes in Computer Science . Springer Berlin Heidelberg, Berlin; Heidelberg. XX\,+\,470 pages

    work page 2008

  7. [7]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    2020

  8. [8]

    Brynjolfsson, E., Li, D., and Raymond, L. R. (2023). Generative ai at work. Working Paper 31161, National Bureau of Economic Research

    work page 2023

Show all 61 references
  1. [9]

    Chen, L., Davis, J., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J. (2024). Are more llm calls all you need? towards the scaling properties of compound ai systems. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C., editors...

    2024

  2. [10]

    Chen, L., Zaharia, M., and Zou, J. (2023). Frugalgpt: How to use large language models while reducing cost and improving performance

    2023

  3. [11]

    N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J

    Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. E., and Stoica, I. (2024). Chatbot arena: An open platform for evaluating LLM s by human preference. In Forty-first International Conference on Machine Learning

    2024

  4. [12]

    Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope,...

    2022

  5. [13]

    W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S

    Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Hu...

    2022

  6. [14]

    Coello, C. A. C., Aguirre, A. H., and Zitzler, E., editors (2005). Evolutionary Multi - Criterion Optimization\: Third International Conference, EMO 2005, Guanajuato, Mexico, March 9--11, 2005, Proceedings , volume 3410 of Lecture Notes in Computer Science , Berlin Heidelberg....

    2005

  7. [15]

    and Dennis, J.\, E

    Das, I. and Dennis, J.\, E. (1997). A closer look at drawbacks of minimizing weighted sums of objectives for pareto set generation in multicriteria optimization problems. Structural Optimization , 14(1):63--69

    1997

  8. [16]

    DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning

    DeepSeek AI (2025). DeepSeek-R1 : Incentivizing reasoning capability in LLMs via reinforcement learning

    2025

  9. [17]

    Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., R \"u hle, V., Lakshmanan, L. V. S., and Awadallah, A. H. (2024). Hybrid LLM : Cost-efficient and quality-aware query routing. In The Twelfth International Conference on Learning Representations

    2024

  10. [18]

    and Wiener, Y

    El-Yaniv, R. and Wiener, Y. (2010). On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11:1605--1641

    2010

  11. [19]

    Eloundou, T., Manning, S., Mishkin, P., and Rock, D. (2024). Gpts are gpts: Labor market impact potential of llms. Science , 384(6702):1306--1308

    2024

  12. [20]

    H., El, B., Suzgun, M., Yuksekgonul, M., and Zou, J

    Erol, M. H., El, B., Suzgun, M., Yuksekgonul, M., and Zou, J. (2025). Cost-of-pass: An economic framework for evaluating language models

    2025

  13. [21]

    Hammond, G. (2024). Speed of ai development stretches risk assessments to breaking point. Artificial intelligence’s complexity exposes flaws in traditional methods used to evaluate safety and accuracy

    2024

  14. [22]

    Hari, S. N. and Thomson, M. (2023). Tryage: Real-time, intelligent routing of user prompts to large language models. arXiv preprint arXiv:2308.11601

    2023 arXiv

  15. [23]

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. (2021). Measuring mathematical problem solving with the math dataset

    2021

  16. [24]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. (2018). A baseline for detecting misclassified and out-of-distribution examples in neural networks

    2018

  17. [25]

    J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., and Upadhyay, S

    Hu, Q. J., Bieker, J., Li, X., Jiang, N., Keigwin, B., Ranganath, G., Keutzer, K., and Upadhyay, S. K. (2024). Routerbench: A benchmark for multi- LLM routing system. In Agentic Markets Workshop at ICML 2024

    2024

  18. [26]

    Jahn, J., Klose, J., and Merkel, A. (1991). On the application of a method of reference point approximation to bicriterial optimization problems in chemical engineering. In Oettli, W. and Pallaschke, D., editors, Advances in Optimization , pages 478--491. Springer, Berlin; Hei...

    1991

  19. [27]

    B., Seabury, S., Lakdawalla, D., and Chandra, A

    Jena, A. B., Seabury, S., Lakdawalla, D., and Chandra, A. (2011). Malpractice risk according to physician specialty. New England Journal of Medicine , 365(7):629--636

    2011

  20. [28]

    Multi-Objective Machine Learning , volume 16 of Studies in Computational Intelligence

    Jin, Y., editor (2006). Multi-Objective Machine Learning , volume 16 of Studies in Computational Intelligence . Springer, Berlin & Heidelberg

    2006

  21. [29]

    K., Narasimhan, H., Rawat, A

    Jitkrittum, W., Gupta, N., Menon, A. K., Narasimhan, H., Rawat, A. S., and Kumar, S. (2024). When does confidence-based cascade deferral suffice?

    2024

  22. [30]

    S., Juneja, J., Wang, Z., Lee, C.-Y., Shenoy, P., Panigrahy, R., Menon, A

    Jitkrittum, W., Narasimhan, H., Rawat, A. S., Juneja, J., Wang, Z., Lee, C.-Y., Shenoy, P., Panigrahy, R., Menon, A. K., and Kumar, S. (2025). Universal LLM routing with correctness-based representation. In First Workshop on Scalable Optimization for Efficient and Adaptive Fou...

    2025

  23. [31]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobs...

    2022

  24. [32]

    B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D

    Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models

    2020

  25. [33]

    and Longbotham, R

    Kohavi, R. and Longbotham, R. (2007). Online experiments: Lessons learned. Computer , 40(9):103--105

    2007

  26. [34]

    S., Reid, M., Matsuo, Y., and Iwasawa, Y

    Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. (2022). Large language models are zero-shot reasoners. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems

    2022

  27. [35]

    Koski, J. (1988). Multicriteria truss optimization. In Stadler, W., editor, Multicriteria Optimization in Engineering and in the Sciences , pages 263--307. Plenum Press, New York

    1988

  28. [36]

    and Richardson, J

    Kudo, T. and Richardson, J. (2018). S entence P iece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Blanco, E. and Lu, W., editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: Syst...

    2018

  29. [37]

    Laskar, M. T. R., Alqahtani, S., Bari, M. S., Rahman, M., Khan, M. A. M., Khan, H., Jahan, I., Bhuiyan, A., Tan, C. W., Parvez, M. R., Hoque, E., Joty, S., and Huang, J. (2024). A systematic survey and critical review on evaluating large language models: Challenges, limitation...

    2024

  30. [38]

    D., Re, C., Acosta-Navas, D., Hudson, D

    Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., Re, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F....

    2023

  31. [39]

    Lin, S., Hilton, J., and Evans, O. (2022). T ruthful QA : Measuring how models mimic human falsehoods. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , p...

    2022

  32. [40]

    Mankiw, N. G. (2020). Principles of Economics . Cengage Learning, 9th edition

    2020

  33. [41]

    The Llama 3 herd of models

    Meta AI (2024). The Llama 3 herd of models

    2024

  34. [42]

    S., Kim, S., Gupta, N., Menon, A

    Narasimhan, H., Jitkrittum, W., Rawat, A. S., Kim, S., Gupta, N., Menon, A. K., and Kumar, S. (2024). Faster cascades via speculative decoding

    2024

  35. [43]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. (2018). Don ' t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J., editors, Proceedings of the 2018 Conference on Empi...

    2018

  36. [44]

    E., Kadous, M

    Ong, I., Almahairi, A., Wu, V., Chiang, W.-L., Wu, T., Gonzalez, J. E., Kadous, M. W., and Stoica, I. (2025). Route LLM : Learning to route LLM s from preference data. In The Thirteenth International Conference on Learning Representations

    2025

  37. [45]

    GPT-4 technical report

    OpenAI (2024). GPT-4 technical report

    2024

  38. [46]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022). Training language mod...

    2022

  39. [47]

    Ravaut, M., Ding, B., Jiao, F., Chen, H., Li, X., Zhao, R., Qin, C., Xiong, C., and Joty, S. (2025). A comprehensive survey of contamination detection methods in large language models

    2025

  40. [48]

    Shnitzer, T., Ou, A., Silva, M., Soule, K., Sun, Y., Solomon, J., Thompson, N., and Yurochkin, M. (2023). Large language model routing with benchmark datasets. arXiv preprint arXiv:2309.15789

    2023 arXiv

  41. [49]

    Singh, H., Meyer, A. N. D., and Thomas, E. J. (2014). The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving us adult populations. BMJ Quality & Safety , 23(9):727--731

    2014

  42. [50]

    Srivastava, A. et al. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research . Featured Certification

    2023

  43. [51]

    M., Mello, M

    Studdert, D. M., Mello, M. M., Gawande, A. A., Gandhi, T. K., Kachalia, A., Yoon, C., Puopolo, A. L., and Brennan, T. A. (2006). Claims, errors, and compensation payments in medical malpractice litigation. New England Journal of Medicine , 354(19):2024--2033

    2006

  44. [52]

    N., Kaiser, L

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural ...

    2017

  45. [53]

    S., Menon, A

    Wang, C., Augenstein, S., Rush, K., Jitkrittum, W., Narasimhan, H., Rawat, A. S., Menon, A. K., and Go, A. (2024). Cascade-aware training of language models

    2024

  46. [54]

    W., Lester, B., Du, N., Dai, A

    Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. (2022). Finetuned language models are zero-shot learners. In International Conference on Learning Representations

    2022

  47. [55]

    J., Liu, R., and Thomson, M

    Zellinger, M. J., Liu, R., and Thomson, M. (2025). Cost-saving llm cascades with early abstention

    2025

  48. [56]

    Zellinger, M. J. and Thomson, M. (2024). Efficiently deploying llms with controlled risk

    2024

  49. [57]

    Zellinger, M. J. and Thomson, M. (2025). Rational tuning of llm cascades via probabilistic modeling

    2025

  50. [58]

    Zhang, K., Wang, C., Peng, L., Go, A., and Liu, X. (2024). Privacy-preserved llm cascade via cot-enhanced policy learning. arXiv preprint arXiv:2410.08014

    2024 arXiv

  51. [59]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  52. [60]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  53. [61]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.