Pith. sign in

REVIEW 2 major objections 4 minor 56 references

Rational Tuning of LLM Cascades via Probabilistic Modeling

T0 review · 2 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a Markov-copula model of LLM confidences enables rational threshold tuning, improving error-cost trade-offs by 4.3% on average for cascades with at least three models.

desk verdict Solid empirical paper on tuning LLM cascade thresholds with a copula-Markov model; the Markov assumption is imperfect but the held-out gains hold up. read the letter →

arxiv 2501.09345 v4 pith:JANXLI64 submitted 2025-01-16 cs.LG cs.AIstat.ML

Michael J. Zellinger , Matt Thomson This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords LLMcascadesconfidencecalibrationMarkov-copulamodelGumbelcopulathresholdtuningerror-costtrade-offBayesianoptimizationsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the confidence scores of models in an LLM cascade can be modeled jointly by a simple parametric structure, making it possible to set confidence thresholds rationally instead of by trial and error. The authors claim that their Markov-copula model, which combines per-model marginal distributions, pairwise Gumbel copulas, and a Markov factorization, yields analytical expressions for a cascade's error rate and expected cost, enabling continuous optimization of the thresholds. If correct, this would give practitioners a data-efficient way to deploy sequences of LLMs with better error-cost trade-offs, particularly for long cascades and in low-sample settings where Bayesian optimization and grid search struggle. The paper reports a 4.3% average improvement in area under the error-cost curve relative to Bayesian optimization for cascades with at least three models, and a 10.2% improvement when only 30 training examples are available.

What carries the argument

The central object is a Markov-copula factorization for the joint distribution of calibrated confidences: $P(\Phi_1 \le \phi_1, \ldots, \Phi_k \le \phi_k) \approx P(\Phi_1 \le \phi_1) \prod_{j=2}^k P(\Phi_j \le \phi_j \mid \Phi_{j-1} \le \phi_{j-1})$. Combined with Gumbel copulas for pairwise transition probabilities and mixture-of-$\beta$ marginals with point masses at the confidence bounds, this factorization yields closed-form expressions for correctness probability and expected cost, enabling continuous optimization of the confidence thresholds.

What would settle it

An experiment that estimates the conditional dependence between $\Phi_i$ and $\Phi_{i-2}$ given $\Phi_{i-1}$ on a large held-out dataset and checks whether the Markov approximation incurs a material error; if the error is large, the analytical formulas for correctness probability and expected cost will produce threshold choices whose predicted error-cost curves deviate systematically from the empirical curves.

Watch

Extended reading notes

Core claim

The central discovery is that the joint distribution of calibrated confidences across a cascade of LLMs is well approximated by a product of pairwise conditional distributions, each modeled with a Gumbel copula and mixed discrete-continuous beta marginals. Under this Markov-copula assumption, the probability that the cascade answers correctly and its expected inference cost can be written as closed-form sums that are differentiable in the confidence thresholds. This turns threshold selection from a high-dimensional search problem into a continuous optimization problem solvable with L-BFGS-B, and reduces the computational scaling in cascade length from exponential to low-order polynomial. The authors demonstrate on six benchmarks that the resulting Rational Tuning framework yields better error-cost curves than Bayesian optimization and high-resolution grid search, with the gains growing as cascade length increases and in the low-sample regime.

Load-bearing premise

The load-bearing premise is the Markov assumption that the confidence of each model depends only on the immediately preceding model's confidence, which the paper shows holds only approximately and is strongest for cascades from the same model family.

Editorial extensions

If this is right

  • Cascade threshold tuning becomes a continuous optimization problem with $O(k)$ evaluation cost, making longer cascades with $k \ge 5$ models practical to tune.
  • The improvement relative to Bayesian optimization grows with cascade length: a 4.3% reduction in area under the error-cost curve for $k \ge 3$ and 7.2% for $k = 5$ with about 300 training examples.
  • In the low-sample regime with $n \le 30$ training examples, the improvement over Bayesian optimization widens to 10.2% on average for $k \ge 3$, indicating that the model's inductive assumptions improve sample efficiency.
  • The computational scaling changes from exponential in cascade length for grid search to low-order polynomial, and the dependence on the resolution of the error-cost curve becomes linear.
  • Hyperparameter-free feature transforms for logistic-regression calibration reduce expected calibration error by 28.2% on average across ten LLMs and six benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Markov assumption is strongest for same-family cascades, the method's gains may be smaller for mixed-family cascades, a prediction that could be tested by comparing performance on same-family versus mixed cascades.
  • The analytical error-cost expressions could extend to LLM routing, where a router sends a query to a single model based on noisy confidence estimates, replacing sequential threshold events with a routing decision.
  • The single-parameter Gumbel copula per adjacent pair may be too restrictive for datasets where confidence correlations differ between correct and incorrect answers; the appendix's conditional rank correlation results suggest this is worth investigating.
  • The same Markov-copula approach could be applied to other confidence signals, such as self-consistency or semantic entropy, to see whether the tuning gains persist when the base confidence measure changes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a parametric Markov-copula model for the joint distribution of calibrated confidences of LLMs in a cascade, and uses it to derive analytical expressions for cascade correctness probability and expected cost (Proposition 2). These expressions enable continuous optimization of confidence thresholds via L-BFGS-B, which the authors call Rational Tuning. The method is evaluated on six benchmarks with Llama, Qwen, and GPT models, comparing against Bayesian optimization and high-resolution grid search. The paper reports improved area under the error-cost curve (AUC) for cascades of length k>=3, with average gains of 4.3% over Bayesian optimization and 2.0% over grid search, and larger gains in the low-sample regime (n<=30), plus favorable runtime scaling. The authors also report extensive goodness-of-fit tests for the copula and marginal components.

Significance. If the claims hold, this is a useful contribution to LLM cascade tuning: it replaces black-box optimization with a interpretable probabilistic model, provides O(k) computation of the objective, and shows strong empirical gains in low-data settings. The paper is unusually thorough on the empirical side: it includes statistical significance tests (Wilcoxon rank-sum), goodness-of-fit diagnostics with Cramer-von Mises statistics, a sensitivity analysis, code for reproduction, and a clear statement of assumptions and limitations. The main empirical finding—that the model-based tuning beats general-purpose baselines on held-out test sets—is credible and not circular, since model parameters are fit on training data and thresholds are evaluated on test data.

major comments (2)
  1. [§3.1, Eq. (5); §3.3, Prop. 2, Eqs. (14)-(15)] The Markov assumption in Eq. (5) is the key structural assumption that makes the analytical error/cost formulas in Proposition 2 valid. The proof of Proposition 2 applies Eq. (5) twice for each i: once to factor the joint event and once to replace the conditional measure in the integral. However, the validation in §4.3.1 is indirect: Kendall-tau heatmaps and the logistic-regression significance tests in Table 3 do not directly estimate the conditional-CDF equality P(Phi_i <= phi_i | chain of ancestor events) = P(Phi_i <= phi_i | Phi_{i-1} <= phi_{i-1}) over the threshold grid used in optimization. Table 3 actually shows that earlier ancestors often retain statistical significance even when the immediate predecessor is included (e.g., MMLU 70B row: the 3B ancestor has log10 p = -3.48; TruthfulQA 405B row: the 3B and 8B ancestors have log10 p below -0.5), so Eq. (5) is violated to some degree. This means the error and cost surfaces minimized in (11) can be biased. I recommend adding a direct goodness-of-fit test of Eq. (5) against empirical conditional CDFs, or an analysis quantifying how deviations from Eq. (5) affect the optimized thresholds and the resulting AUC. Without this, the paper's claim that the thresholds are 'rational' (i.e., optimal under the true joint distribution) is stronger than the evidence supports.
  2. [§4.3.1 and §4.4.2] The paper explicitly concludes in §4.3.1 that the Markov property 'holds approximately, though not exactly,' and the sensitivity analysis in §4.4.2 shows that larger Cramer-von Mises divergences are associated with smaller performance gains. This is an honest and useful analysis, but it does not establish how much of the approximation error propagates into the optimized thresholds. The reported AUC improvements are empirical and may survive even if Eq. (5) is not exact, but the central conceptual claim—that the derived formulas provide a rational basis for tuning—requires either a direct test of the conditional-CDF equality or a formal or empirical analysis of the bias-variance trade-off of using a misspecified model. I would like to see the authors address this gap directly, for example by comparing thresholds chosen with the fitted model against oracle thresholds computed on the test set, or by a Monte Carlo study where the data are generated from a known non-Markovian process and the degradation in AUC is quantified.
minor comments (4)
  1. [Eq. (5)] The right-hand side of the conditioning event uses 'Φ1≤t1' instead of 'Φ1≤ϕ1'; this is clearly a typo and should be corrected.
  2. [§4.4] The sentence 'the mean reduction in AUC compared to Bayesian optimization is 4.3% for k≥3; 5.8% for k≥3; and 7.2% for k=5' appears to contain a duplicate 'k≥3'; the middle value presumably refers to k=4 or another subset and should be reworded.
  3. [Table 5 and §4.3.3] Although the average √CvM of 4% is small, 74% of the null hypotheses are rejected at the 5% level; this high rejection rate deserves a more prominent discussion in the main text rather than only in the appendix-level interpretation.
  4. [Figure 5 and §4.4] The caption refers to 'Cost ($/Query)' but the AUC is later described as being normalized to [0,1]; please clarify in the caption and in the text how the normalization interacts with the cost values shown on the x-axis.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central threshold-tuning claim is an out-of-sample test-set evaluation of a model fitted on training data, and the paper's self-citations are independently re-validated.

full rationale

The paper's derivation chain is self-contained against external benchmarks. The Markov-copula model is fitted on training data (n≈300 or n≤30 examples), and the confidence thresholds chosen by minimizing the analytic objective (11) are subsequently evaluated by computing the area under the error-cost curve on held-out test sets (Section 4.4, Tables 7 and 8). This is a genuine predictive claim, not a fitted input renamed as a prediction. Proposition 2's formulas (14) and (15) are obtained by explicitly stated assumptions: the Markov property in Eq. (5) and the independence of token counts from calibrated confidences. These are inductive assumptions about the data-generating process, and the paper tests them empirically in Section 4.3, even conceding that the Markov property holds only approximately. An assumption that is load-bearing but imperfectly validated is a robustness concern, not a circularity. The self-citations to Zellinger and Thomson (2024) concern the hyperparameter-free feature transforms used in calibration preprocessing and the adverse-selection observation; these are not the paper's central rational-tuning claim, and Table 2 independently demonstrates on held-out test sets that the transforms reduce expected calibration error relative to no transform. No parameter is fitted to a subset and then called a prediction of that same subset, no uniqueness theorem is imported from the authors' prior work, and no known empirical pattern is merely renamed. Therefore the central claim does not reduce by construction to its inputs.

Assumptions & free parameters 6 free parameters · 7 assumptions · 0 invented entities

The model relies on standard probability tools (Sklar's theorem, probability integral transform) and domain assumptions about the structure of LLM confidence scores (Markov property, Gumbel copula, beta mixture marginals, cost independence). The fitted parameters are estimated from training data; the modeling choices (mixture components, Gumbel family, scaling rate r, interpolation gap q) are made by hand and are reported. No invented entities are introduced.

free parameters (6)
  • Logistic regression calibrator weights
    Per model and benchmark, transforms raw confidence to calibrated confidence; fitted on the training set (n~300).
  • Beta mixture parameters (pi, alpha1, beta1, alpha2, beta2, w_min, w_max, phi_min, phi_max)
    Parameters of the mixed discrete-continuous marginal distribution in Eq. (7), fitted via EM and counting on the training set.
  • Gumbel copula dependence parameter theta = theta = 1/(1-tau)
    Computed from Kendall's tau on training data for each adjacent pair of models.
  • Lambda scaling rate r = 0.25 to 1
    Chosen by the authors; controls the geometric progression of cost sensitivity in Eq. (12).
  • Interpolation gap q = < 0.2
    Chosen by the authors; controls threshold infilling in Eq. (13).
  • Number of beta mixture components = 2
    Modeling choice selected by hand; not fitted to data.
assumptions (7)
  • standard math Sklar's theorem
    Invoked in Theorem 1 to construct joint distributions from marginals and copulas; standard result from Nelsen (2006).
  • standard math Probability integral transform
    Underlies copula modeling; standard result from Casella and Berger (2002).
  • domain assumption Markov property (Eq. 5)
    Explicitly assumed and tested; holds approximately for same-family cascades.
  • domain assumption Gumbel copula form
    Chosen to model pairwise dependence; goodness-of-fit shows adequate but not exact fit for mixed family cascades.
  • domain assumption Mixture of two beta distributions with point masses
    Chosen to model marginals; CvM tests reject at higher than expected rates, so it is an approximation.
  • domain assumption Token counts independent of confidences
    Needed for Proposition 2 cost formula; not explicitly tested.
  • domain assumption Logistic regression yields calibrated confidence
    Standard calibration method; validated via ECE.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rational Tuning of LLM Cascades via Probabilistic Modeling." pith.science (2026). https://pith.science/paper/JANXLI64

@misc{pith2026250109345,
  author       = {Pith},
  title        = {Pith review of: Rational Tuning of LLM Cascades via Probabilistic Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JANXLI64}},
  note         = {Machine review of arXiv:2501.09345}
}
abstract

Understanding the reliability of large language models (LLMs) has recently garnered significant attention. Given LLMs' propensity to hallucinate, as well as their high sensitivity to prompt design, it is already challenging to predict the performance of an individual LLM. However, the problem becomes more complex for compound LLM systems such as cascades, where in addition to each model's standalone performance, we must understand how the error rates of different models interact. In this paper, we present a probabilistic model for the joint performance distribution of a sequence of LLMs, which enables a framework for rationally tuning the confidence thresholds of a LLM cascade using continuous optimization. Compared to selecting confidence thresholds using Bayesian optimization, our parametric Markov-copula model yields more favorable error-cost trade-offs, improving the area under the error-cost curve by 4.3% on average for cascades with $k\geq 3$ models. In the low-sample regime with $n \leq 30$ training examples, the performance improvement widens to 10.2%, suggesting that our framework's inductive assumptions about the interactions between the error rates of different LLMs enhance sample efficiency. Overall, our Markov-copula model provides a rational basis for tuning LLM cascade performance and points to the potential of probabilistic methods in analyzing systems of LLMs.

Figures

Figures reproduced from arXiv: 2501.09345 by the authors.

Figure 1
Figure 1. Evaluates the Markov property by showing the Kendall’s [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Correlations between the calibrated confidences of selected pairs of LLMs on different benchmarks, [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Correlations between the calibrated confidences of smaller Llama models (1B, 8B, 70B) ( [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (19 more)
Figure 4
Figure 4. Figure 4: Selection of trained marginal distributions (fitted on [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Performance evaluation via the area under the error-cost curve (AUC). (a) Error-cost curves [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Reduction in the area under the error-cost curve (AUC) on the test set when using our Rational [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Reduction in the area under the error-cost curve (AUC) as cascade length grows, in the low-sample [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Sensitivity of Rational Tuning’s performance gains to the Cramér-von Mises (CvM) test statistics [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Shows runtime scaling for computing the full error-cost curve, comparing our continuous [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Verifies that confidence thresholding works by showing that for most benchmarks and models, [PITH_FULL_IMAGE:figures/full_fig_p030_10.png]
Figure 11
Figure 11. Figure 11: MMLU: Kendall’s τ rank correlations of Llama3 models ordered by size. llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o 0.01 0.03 0.04 0.08 0.02 0.02 …
Figure 12
Figure 12. Figure 12: MMLU: Kendall’s τ rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size. 6 Appendix F: Recomputing Rank Correlations on Correct and Incorrect Answers llama3.2-1b llama3.2-3b llama3.1-8b llama3.1-70b llama3.1-405b llama3.2-1b llama3.2-3b llama3.1-8b l…
Figure 13
Figure 13. Figure 13: MedMCQA: Kendall’s τ rank correlations of Llama3 models ordered by size. In this section, we verify the rank correlations between the confidence scores of different LLMs by recomputing them conditioned on both models answering correctly or incorrectly. Figures 11-22 e…
Figure 14
Figure 14. Figure 14: MedMCQA: Kendall’s τ rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size. llama3.2-1b llama3.2-3b llama3.1-8b llama3.1-70b llama3.1-405b llama3.2-1b llama3.2-3b llama3.1-8b llama3.1-70b llama3.1-405b -0.03 0.02 0.02 -0.03 0.16 0.15 0.02 0.16 0.19 0…
Figure 15
Figure 15. Figure 15: TriviaQA: Kendall’s τ rank correlations of Llama3 models ordered by size. llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o -0.04 -0.07 0.02 0.02 -0.0…
Figure 16
Figure 16. Figure 16: TriviaQA: Kendall’s τ rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size. correlations with sample size less than 50; we use the n = 50 cut-off since it reduces the standard error for Kendall’s τ to around στ ≤ 0.1, based on a normal approximation…
Figure 17
Figure 17. Figure 17: XSum: Kendall’s τ rank correlations of Llama3 models ordered by size. llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o 0.03 0.15 0.05 0.03 0.24 0.03 …
Figure 18
Figure 18. Figure 18: XSum: Kendall’s τ rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size. llama3.2-1b llama3.2-3b llama3.1-8b llama3.1-70b llama3.1-405b llama3.2-1b llama3.2-3b llama3.1-8b llama3.1-70b llama3.1-405b -0.05 -0.04 -0.05 0.18 -0.04 0.18 ? ? ? ? ? ? ? ? ?…
Figure 19
Figure 19. Figure 19: GSM8K: Kendall’s τ rank correlations of Llama3 models ordered by size. llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o -0.04 -0.04 ? ? ? ? ? ? ? ? ?…
Figure 20
Figure 20. Figure 20: GSM8K: Kendall’s τ rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_20.png]
Figure 21
Figure 21. Figure 21: TruthfulQA: Kendall’s τ rank correlations of Llama3 models ordered by size. llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o llama3.2-1b gpt-4o-mini qwen2.5-32b-c llama3.1-70b qwen2.5-72b llama3.1-405b gpt-4o 0.15 0.09 0.09 0.13 0.12…
Figure 22
Figure 22. Figure 22: TruthfulQA: Kendall’s τ rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size [PITH_FULL_IMAGE:figures/full_fig_p034_22.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

56 extracted references · 35 canonical work pages

  1. [1]

    P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., Upadhyay, S., Faruqui, M., and Mausam (2024)

    Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., Upadhyay, S., Faruqui, M., and Mausam (2024). A uto M ix: Automatically mixing language models

    work page 2024

  2. [2]

    and Mitchell, T

    Azaria, A. and Mitchell, T. (2023). The internal state of an LLM knows when it's lying

    work page 2023

  3. [3]

    B \"o rzs \"o nyi, S., Kossmann, D., and Stocker, K. (2001). The skyline operator. Proceedings 17th International Conference on Data Engineering , pages 421--430

    work page 2001

  4. [4]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    2020

  5. [5]

    Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2024). Discovering latent knowledge in language models without supervision

    2024

  6. [6]

    and Berger, R

    Casella, G. and Berger, R. (2002). Statistical Inference . Duxbury Press, Pacific Grove, 2 edition

    work page 2002

  7. [7]

    Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. (2024a). INSIDE : LLM s' internal states retain the power of hallucination detection

    work page 2024

  8. [8]

    Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J

    Chen, L., Davis, J. Q., Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J. (2024b). Are more LLM calls all you need? towards scaling laws of compound inference systems

    work page 2024

Show all 56 references
  1. [9]

    Chen, L., Zaharia, M., and Zou, J. (2023). Frugalgpt: How to use large language models while reducing cost and improving performance

    2023

  2. [10]

    Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  3. [11]

    Cowen-Rivers, A., Lyu, W., Tutunov, R., Wang, Z., Grosnit, A., Griffiths, R.-R., Maravel, A., Hao, J., Wang, J., Peters, J., and Bou Ammar, H. (2022). Hebo: Pushing the limits of sample-efficient hyperparameter optimisation. Journal of Artificial Intelligence Research , 74

    2022

  4. [12]

    P., Laird, N

    Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B (Methodological) , 39(1):1--22

    1977

  5. [13]

    Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. (2024). LLM .int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference on Neural Information Processing Systems , NIPS '22, Red Hook, NY, USA. Curran Associates Inc

    2024

  6. [14]

    Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V., Lakshmanan, L. V. S., and Awadallah, A. H. (2024). Hybrid LLM : Cost-efficient and quality-aware query routing

    2024

  7. [15]

    Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature , 630(8017):625--630

    2024

  8. [16]

    Genest, C., Rémillard, B., and Beaudoin, D. (2009). Goodness-of-fit tests for copulas: A review and a power study. Insurance: Mathematics and Economics , 44(2):199--213

    2009

  9. [17]

    Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks

    2017

  10. [18]

    S., Menon, A

    Gupta, N., Narasimhan, H., Jitkrittum, W., Rawat, A. S., Menon, A. K., and Kumar, S. (2024). Language model cascades: Token-level uncertainty and beyond

    2024

  11. [19]

    Hari, S. N. and Thomson, M. (2023). Tryage: Real-time, intelligent routing of user prompts to large language models

    2023

  12. [20]

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR)

    2021

  13. [21]

    and Gimpel, K

    Hendrycks, D. and Gimpel, K. (2018). A baseline for detecting misclassified and out-of-distribution examples in neural networks

    2018

  14. [22]

    Jiang, D., Ren, X., and Lin, B. Y. (2023). LLM -blender: Ensembling large language models with pairwise ranking and generative fusion. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguist...

    2023

  15. [23]

    Jiang, Z., Araki, J., Ding, H., and Neubig, G. (2021). How can we know when language models know? on the calibration of language models for question answering

    2021

  16. [24]

    K., Narasimhan, H., Rawat, A

    Jitkrittum, W., Gupta, N., Menon, A. K., Narasimhan, H., Rawat, A. S., and Kumar, S. (2024). When does confidence-based cascade deferral suffice?

    2024

  17. [25]

    Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (2017). T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension. In Barzilay, R. and Kan, M.-Y., editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguist...

    2017

  18. [26]

    Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield-Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D., Hernandez, D., Jacobs...

    2022

  19. [27]

    Kag, A., Fedorov, I., Gangrade, A., Whatmough, P., and Saligrama, V. (2023). Efficient edge inference by selective query. In The Eleventh International Conference on Learning Representations

    2023

  20. [28]

    Kossen, J., Han, J., Razzak, M., Schut, L., Malik, S., and Gal, Y. (2024). Semantic entropy probes: Robust and cheap hallucination detection in LLM s

    2024

  21. [29]

    S., McCann, B., Xiong, C., and Socher, R

    Kryściński, W., Keskar, N. S., McCann, B., Xiong, C., and Socher, R. (2019). Neural text summarization: A critical evaluation

    2019

  22. [30]

    Lin, S., Hilton, J., and Evans, O. (2022a). Teaching models to express their uncertainty in words

    2022

  23. [31]

    Lin, S., Hilton, J., and Evans, O. (2022b). T ruthful QA : Measuring how models mimic human falsehoods. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , ...

    2022

  24. [32]

    Lin, Z., Trivedi, S., and Sun, J. (2024). Generating with confidence: Uncertainty quantification for black-box large language models

    2024

  25. [33]

    Liu, D. C. and Nocedal, J. (1989). On the limited memory BFGS method for large scale optimization. Mathematical Programming , 45(1):503--528

    1989

  26. [34]

    Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G - E val: NLG evaluation using GPT-4 with better human alignment

    2023

  27. [35]

    Manakul, P., Liusie, A., and Gales, M. J. F. (2023). Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models

    2023

  28. [36]

    P., Cooper, G

    Naeini, M. P., Cooper, G. F., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence , AAAI'15, page 2901–2907. AAAI Press

    2015

  29. [37]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. (2018). Don't give me the details, just the summary! T opic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , Brussels, Belgium

    2018

  30. [38]

    Nelsen, R. B. (2006). An Introduction to Copulas . Springer Series in Statistics. Springer, 2 edition

    2006

  31. [39]

    GPT-4 T echnical R eport

    OpenAI (2024). GPT-4 T echnical R eport

    2024

  32. [40]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022). Training language mod...

    2022

  33. [41]

    K., and Sankarasubbu, M

    Pal, A., Umapathi, L. K., and Sankarasubbu, M. (2022). M ed MCQA : A large-scale multi-subject multi-choice dataset for medical domain question answering. In Flores, G., Chen, G. H., Pollard, T., Ho, J. C., and Naumann, T., editors, Proceedings of the Conference on Health, Inf...

    2022

  34. [42]

    Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers , pages 61--74. MIT Press

    1999

  35. [43]

    Plaut, B., Nguyen, K., and Trinh, T. (2024). Softmax probabilities (mostly) predict large language model correctness on multiple-choice q&a

    2024

  36. [44]

    Proskurina, I., Brun, L., Metzler, G., and Velcin, J. (2024). When quantization affects confidence of large language models? In Duh, K., Gomez, H., and Bethard, S., editors, Findings of the Association for Computational Linguistics: NAACL 2024 , pages 1918--1928, Mexico City, ...

    2024

  37. [45]

    Ren, J., Luo, J., Zhao, Y., Krishna, K., Saleh, M., Lakshminarayanan, B., and Liu, P. J. (2023). Out-of-distribution detection and selective generation for conditional language models

    2023

  38. [46]

    Rudin, W. (1976). Principles of Mathematical Analysis . McGraw-Hill, New York, 3 edition

    1976

  39. [47]

    Sakota, M., Peyrard, M., and West, R. (2024). Fly-swat or cannon? cost-effective language model choice via meta-modeling. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining , volume 35 of WSDM ’24 , page 606–615. ACM

    2024

  40. [48]

    P., and de Freitas, N

    Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and de Freitas, N. (2016). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE , 104(1):148--175

    2016

  41. [49]

    S., Menon, A

    Wang, C., Augenstein, S., Rush, K., Jitkrittum, W., Narasimhan, H., Rawat, A. S., Menon, A. K., and Go, A. (2024). Cascade-aware training of language models

    2024

  42. [50]

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models

    2023

  43. [51]

    Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2024). Can LLM s express their uncertainty? an empirical evaluation of confidence elicitation in LLM s

    2024

  44. [52]

    Yue, M., Zhao, J., Zhang, M., Du, L., and Yao, Z. (2024). Large language model cascades with mixture of thoughts representations for cost-efficient reasoning

    2024

  45. [53]

    and Elkan, C

    Zadrozny, B. and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '02, page 694–699, New York, NY, USA. Association for...

    2002

  46. [54]

    Q., Miller, H., Potts, C., Zou, J., Carbin, M., Frankle, J., Rao, N., and Ghodsi, A

    Zaharia, M., Khattab, O., Chen, L., Davis, J. Q., Miller, H., Potts, C., Zou, J., Carbin, M., Frankle, J., Rao, N., and Ghodsi, A. (2024). The shift from models to compound AI systems. https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/. Accessed: January 10, 2025

    2024

  47. [55]

    Zellinger, M. J. and Thomson, M. (2024). Efficiently deploying LLMs with controlled risk

    2024

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.