Pith. sign in

REVIEW 2 major objections 6 minor 48 references

Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single score-gap threshold, calibrated by conformal risk control, turns any black-box pair of language models into a continuum of operating points whose guardrail loss is bounded by a user-set budget alpha.

desk verdict Solid, useful extension of conformal risk control to score-gap routing, but the PKU safety experiment overclaims what the guarantee covers; fixable. read the letter →

arxiv 2506.00911 v1 pith:LTLRO5OA submitted 2025-06-01 cs.AI

William Overman , Mohsen Bayati This is my paper
classification cs.AI MSC 62G1568T5068T07
keywords conformalriskcontrollanguagemodelroutingguardrailobjectivesscore-gapthresholdfinite-sampleguaranteesefficientfrontierhelpfulness-harmlessnesstradeoffcost-accuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conformal Arbitrage is a post-hoc router that mediates between a Primary model optimized for one objective and a Guardian model or human aligned with a guardrail objective. The paper's central claim is that a single scalar threshold on the Primary model's score gap, calibrated by conformal risk control, keeps the expected guardrail loss no larger than a user-specified budget $\alpha$ at finite sample size, with no distributional assumptions and no access to model weights or logits. Among routers that decide solely from the Primary's score gap, the paper proves that this calibrated threshold is utility-optimal up to an $O(n^{-1})$ term. Empirically, the method traces an efficient frontier between cost and accuracy and between helpfulness and harmlessness, beating cost-matched random routing. The practical payoff is that a fixed pair of black-box models becomes a continuum of dialable operating points that can be adjusted after deployment.

What carries the argument

The central object is the $\lambda$-relaxed candidate set $C_\lambda(x) = \{ a \in A(x) : p(x,a) \ge \max_{a'} p(x,a') - \lambda \}$, together with the guardrail loss $L_i(\lambda)$ defined as the drop in the Guardian's best score when restricted to this set. The conformal risk control threshold from Eq. (3) picks the smallest $\lambda$ whose finite-sample risk bound fits the budget $\alpha$. This machinery converts the Primary's score margin into a guardrail-risk-controlling action set, and the monotonicity and Lipschitz properties of the risk and utility curves yield the $O(n^{-1})$ utility-optimality result.

What would settle it

On any exchangeable held-out set, evaluate the empirical mean of $L_i(\hat\lambda)$ as defined in Eq. (2); if it consistently exceeds $\alpha$ across replications when calibration and test draws are genuinely exchangeable, the central guarantee fails. A sharper test for the PKU claim is to replace $g$ with real-valued severity, recalibrate, and check whether mean severity-excess of the selected answers respects $\alpha$; with the paper's binary $g$, mean severity can exceed $\alpha$ even when the binary candidate-set risk respects it.

Watch

Extended reading notes

Core claim

The paper claims that any black-box Primary-Guardian pair can be converted into a risk-controlled policy without retraining: define $C_\lambda(x)$ as all actions whose Primary score is within $\lambda$ of the top score, let $L_i(\lambda) = \max_a g(x_i,a) - \max_{a \in C_\lambda(x_i)} g(x_i,a)$, and choose the smallest $\lambda$ such that $\frac{n}{n+1} \hat{R}_n(\lambda) + \frac{1}{n+1} \le \alpha$. Conformal risk control then gives $\mathbb{E}[L_{n+1}(\hat\lambda)] \le \alpha$ for an exchangeable test point. Theorem 1 says that among all score-gap routers constrained to the same $\alpha$, no router can achieve expected primary utility more than $O(n^{-1})$ higher. If the candidate set is a singleton, the Primary's top action is selected; otherwise the Guardian selects from the set.

Load-bearing premise

The load-bearing premise is that the Guardian's score is an exact scalarization of the guardrail metric, so the loss $L_i(\lambda)$ in Eq. (2) is the quantity the user wants to bound; in the PKU experiment $g$ is binary severity membership, so the controlled quantity is the probability that the candidate set contains no safest reply, not the reported average severity loss.

Editorial extensions

If this is right

  • Deployers can set a quota on one guardrail metric, such as factual-error rate, safety violations, or cost, and let the router maximize a second objective without retraining either model.
  • Any API that returns per-action scores, even without logits, can use the method; each query costs one call to the Primary and at most one call to the Guardian.
  • A few hundred exchangeable logged examples suffice for calibration; the reported ablations show the frontier is stable from $n=300$ to $n=500$.
  • The calibrated threshold is near-optimal among score-gap routers: the expected utility loss relative to the oracle-constrained threshold is $O(n^{-1})$.
  • The router dominates cost-matched and risk-matched random routing on TruthfulQA, MMLU, and PKU-SafeRLHF, recovering most of a stronger model's accuracy at a fraction of its cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $\alpha$ bound attaches to the specific loss $L_i(\lambda)$, so a deployer must choose the Guardian's score so that $L_i(\lambda)$ equals the operational metric; with binary safety grades, the controlled quantity is membership of at least one safest reply in the candidate set, not average severity.
  • A testable extension is to measure the empirical utility gap at $n=100$, $400$, and $1000$; the theorem predicts the gap shrinks roughly in inverse proportion to $n$.
  • If the Primary's score gap is weakly informative about guardrail risk, the router will either escalate too often or miss risky contexts; comparing CA with a router using an additional learned risk signal under the same conformal budget would quantify that loss.
  • Adaptive or conditional conformal risk control, which the paper lists as future work, could convert the marginal $\alpha$ budget into per-group quotas on hard subpopulations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes Conformal Arbitrage, a post-hoc router that mediates between a Primary LLM and a more conservative Guardian by calibrating a single threshold on the Primary's score gap with conformal risk control (CRC). The candidate set C_lambda(x) contains all actions whose Primary score is within lambda of the best score; if this set is a singleton the Primary acts, otherwise the Guardian chooses from the set. The paper proves (Theorem 1) that the CRC-selected threshold is utility-optimal among score-gap routers up to O(n^{-1}), and reports experiments on TruthfulQA/MMLU (cost-accuracy) and PKU-SafeRLHF (helpfulness-harmlessness), claiming a finite-sample, distribution-free bound on the guardrail loss.

Significance. The core regulatory idea is attractive and the CRC mechanism is sound: under exchangeability and monotone bounded losses, the calibration rule in Eq. (3) indeed yields E[L_{n+1}(lambda-hat)] <= alpha. The paper's strengths are its simplicity, API-level applicability, the cost-matched random routing baseline, and the calibration-size ablations. However, as it stands, the central empirical demonstration of risk-controlled safety in the PKU experiment controls a binary event, not the reported severity magnitude, and Theorem 1 requires an additional regularity assumption that is not stated. With these corrected, the paper would be a solid and practical contribution.

major comments (2)
  1. [Section 5.2, Eq. (2), Table 8] With the Guardian defined as g(x,a) = 1{sev(x,a) = min_a' sev(x,a')} and |A(x)|=2, the loss in Eq. (2) reduces to L_i(lambda) = 1{the safest reply is not in C_lambda(x_i)}. Conformal risk control therefore certifies E[1{the candidate set misses all safest replies}] <= alpha, not E[severity-loss] <= alpha. The reported Safety-violation loss in Table 8 and the x-axis of Figure 2 is the average excess severity of the selected answer above the safer one, which can be 1, 2, or 3 on each violating example. Thus the binary bound only implies E[severity-loss] <= 3 alpha, and the statement in Appendix C.2 that 'every CA configuration respects the finite-sample bound Severity-loss <= alpha' is unsupported. The fix is to either report the binary miss rate as the controlled quantity, define the Guardian score so that Eq. (2) directly measures excess severity (e.g., g = -severity), or explicitly state the worst-case 3 alpha bound.
  2. [Theorem 1 and Appendix A] The proof of Theorem 1 introduces the assumption m = inf_{lambda in Lambda} |R'(lambda)| > 0 and uses it to claim that the inverse of R is Lipschitz, giving |lambda-hat - lambda-star| = O(1/n). However, the theorem statement only assumes that R is continuous and strictly decreasing. A continuous strictly decreasing function need not have a Lipschitz inverse (e.g., R(lambda) = alpha + (lambda-star - lambda)^{1/q} with q<1 has inverse derivative unbounded near alpha). As stated, the theorem is not proved. The theorem should either add the derivative lower bound or the assumption that R^{-1} is Lipschitz, or the proof must establish a slower convergence rate under the weaker assumptions.
minor comments (6)
  1. [Section 4.2, Eq. (3); Theorem 1] Equation (3) uses +1/(n+1) in the CRC inequality, while Theorem 1 and Appendix A use +B/(n+1). If the loss bound is B and not necessarily 1, the calibration equation should be written consistently with B.
  2. [Appendix A] In the proof, the inequality U(lambda-star) - U(lambda-hat) <= Umax |lambda-star - lambda-hat| should use the Lipschitz constant K, not Umax; the subsequent bound 2KB/(m(n+1)) shows that K is the intended constant.
  3. [Algorithm 1 and Definition 2] Algorithm 1 selects the Primary's answer when |C_lambda-hat(x)| = 1, which corresponds to the score gap being strictly greater than lambda-hat, whereas Definition 2's score-gap router acts when the gap is >= lambda. This boundary discrepancy is measure-zero under continuous scores but should be reconciled for formal consistency.
  4. [Section 1] There are duplicated words in the text: 'both the the cost versus accuracy tradeoff using the the TruthfulQA' should read 'both the cost versus accuracy tradeoff using the TruthfulQA'.
  5. [Table 4 text] The sentence 'For tighter risk budgets (alpha <= 0.10). accuracy rises' contains an errant period after the parenthetical; it should be 'For tighter risk budgets (alpha <= 0.10), accuracy rises'.
  6. [Section 5.1] The statement that the binarized Guardian makes 'the calibrated risk level alpha now translate directly to an alpha x 100% drop in accuracy relative to the Guardian' is stronger than what the loss bounds. The loss is P(Guardian correct and C_lambda misses the correct answer), while the accuracy drop equals this minus P(Guardian wrong and CA correct); the guarantee is therefore an upper bound on the drop, not an exact translation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; PKU severity-loss bound is an overclaim, not a circular reduction.

full rationale

The derivation is self-contained and does not reduce to its inputs. The calibrated quantity is the threshold λ̂ chosen by the standard conformal risk control rule (Eq. 3), while the claimed guarantee is the expected guardrail loss E[L_{n+1}(λ̂)] ≤ α on a disjoint test point; this guarantee comes from the external theorem of Angelopoulos et al. (2024) and is not obtained by reusing the fitted threshold as evidence. The utility-optimality theorem (Theorem 1) is derived from monotonicity of R and U, Lipschitzness, and the CRC lower bound, not assumed. The only self-citation, Overman et al. (2024), appears in Section 2 as a related-work lineage remark and is not load-bearing. Two non-circular correctness concerns should be flagged: (i) with the binary Guardian g(x,a)=1{sev(x,a)=min_{a'} sev(x,a')} in Section 5.2 and |A(x)|=2, Eq. (2) reduces to Li(λ)=1{the safest reply is not in Cλ(x)}, so the CRC guarantee controls the frequency of missing all safest replies, not the average excess severity reported in Table 8; the statement 'As guaranteed by theory, every CA configuration respects the finite-sample bound Severity-loss ≤ α' (Appendix C.2) is therefore an overclaim rather than a consequence of the stated theorem; and (ii) the proof of Theorem 1 invokes m=inf_{λ∈Λ}|R'(λ)|>0 without stating differentiability or a lower derivative bound as an assumption. Neither issue is a circular reduction, so no circular step is identified.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The central claim rests on exchangeability, monotonicity of the loss, and the assumption that Guardian scores are an exact guardrail metric. The PKU experiment additionally assumes that the binary worst-case loss bounds excess severity, which is not implied by Eq. (2).

free parameters (1)
  • relaxation threshold lambda = varies by risk budget; e.g., 0.277 +/- 0.067 for alpha=0.25 on TruthfulQA (Table 1)
    The threshold is fitted to calibration data via conformal risk control, Eq. (3). It is the method's designed operating parameter, not an ad hoc constant, but it determines the empirical frontier.
assumptions (5)
  • domain assumption Exchangeability of calibration and test contexts
    Section 4.2 and the CRC guarantee require the calibration and deployment contexts to be exchangeable; distribution shift would void the bound.
  • domain assumption Loss Li(lambda) is monotone non-increasing in lambda and bounded in [0, B]
    Eq. (2) satisfies monotonicity because C_lambda(x) grows with lambda; boundedness requires Guardian scores in [0, B], and the paper uses 0/1 or normalized scores.
  • domain assumption The risk curve R is continuous and strictly decreasing, and utility U is non-increasing and K-Lipschitz
    Theorem 1 requires these conditions on the population curves to convert the CRC risk bound into a bound on utility loss.
  • ad hoc to paper The derivative of R is bounded away from zero, m = inf |R'| > 0
    This is introduced in the Appendix A proof but is not stated in Theorem 1; strict monotonicity and continuity alone do not imply a Lipschitz inverse or a positive lower bound on |R'|.
  • domain assumption Guardian scores g(x,a) are a valid scalarization of the guardrail objective
    Section 4.1 and 5.2 assume the Guardian's scores measure the user's guardrail risk. In PKU-SafeRLHF the binary Guardian loss does not control excess severity, so this assumption is violated for the reported severity-loss claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models." pith.science (2026). https://pith.science/paper/LTLRO5OA

@misc{pith2026250600911,
  author       = {Pith},
  title        = {Pith review of: Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTLRO5OA}},
  note         = {Machine review of arXiv:2506.00911}
}
read the original abstract

Modern language model deployments must often balance competing objectives, for example, helpfulness versus harmlessness, cost versus accuracy, and reward versus safety. We introduce Conformal Arbitrage, a post hoc framework that learns a data driven threshold to mediate between a Primary model optimized for a primary objective and a more conservative Guardian which could be another model or a human domain expert aligned with a guardrail objective. The threshold is calibrated with conformal risk control, yielding finite sample, distribution free guarantees that the long run frequency of undesirable events, such as factual errors or safety violations, does not exceed a user specified quota. Because Conformal Arbitrage operates wholly at the API level, without requiring access to model logits or updating model weights, it complements weight based alignment techniques and integrates seamlessly with existing cost aware cascades. Empirically, Conformal Arbitrage traces an efficient frontier, allowing users to define an acceptable performance level for one objective while maximizing utility in another. We observe that our method outperforms, in terms of accuracy, cost matched random routing between models. These properties make Conformal Arbitrage a practical, theoretically grounded tool for trustworthy and economical deployment of large language models across a broad range of potentially competing objectives.

Figures

Figures reproduced from arXiv: 2506.00911 by the authors.

Figure 1
Figure 1. Accuracy vs. cost (TruthfulQA), mean ± 1 std over 30 trials; small points show individual CA runs [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Harmfulness vs. helpfulness (PKU-SafeRLHF), mean [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Accuracy vs. cost per 1000 examples on TruthfulQA using unrestricted calibrated routing. Each [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Cost–accuracy frontier on TruthfulQA when the Primary is [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: Cost–accuracy frontier on MMLU. Mean ± std over 30 trials. Faint dots show individual CA runs. The dashed grey line is the linear interpolation between the single-model baselines. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

48 extracted references · 18 canonical work pages

  1. [1]

    Mitigating llm hallucinations via conformal abstention

    Yasin Abbasi-Yadkori, Ilja Kuzborskij, David Stutz, András György, Adam Fisch, Arnaud Doucet, Iuliya Beloshapka, Wei-Hung Weng, Yao-Yuan Yang, Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating llm hallucinations via conformal abstention. arXiv preprint arXiv:2405.01563, 2024. URL https://arxiv.org/abs/2405.01563

    arXiv 2024

  2. [2]

    Automix: Automatically mixing language models

    Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Rajagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Manaal Faruqui, and Mausam. Automix: Automatically mixing language models. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), 2025. arXiv...

    arXiv 2025

  3. [3]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification, 2022. URL https://arxiv.org/abs/2107.07511

    arXiv 2022

  4. [4]

    Conformal risk control

    Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In The Twelfth International Conference on Learning Representations, 2024

    2024

  5. [5]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 a

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, ...

    arXiv 2022

  6. [6]

    Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

    arXiv 2022

  7. [7]

    Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets, 2021

    work page 2021

  8. [8]

    Automatically adaptive conformal risk control, 2025

    Vincent Blot, Anastasios N Angelopoulos, Michael I Jordan, and Nicolas J-B Brunel. Automatically adaptive conformal risk control, 2025. URL https://arxiv.org/abs/2406.17819

    arXiv 2025

Show all 48 references
  1. [9]

    Weak-to-strong generalization: Eliciting strong capabilities with weak supervision

    Collin Burns and et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023

    2023 arXiv

  2. [10]

    Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Bedi, and Mengdi Wang. Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences. In ICML Workshop on Models of Human Feedback for AI Alignment, 2024

    2024

  3. [11]

    Conformal tail risk control for large language model alignment, 2025

    Catherine Yu-Chi Chen, Jingyan Shen, Zhun Deng, and Lihua Lei. Conformal tail risk control for large language model alignment, 2025. URL https://arxiv.org/abs/2502.20285

    2025 arXiv

  4. [12]

    Frugalgpt: How to use large language models while reducing cost and improving performance

    Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023

    2023 arXiv

  5. [13]

    Cherian, Isaac Gibbs, and Emmanuel J

    John J. Cherian, Isaac Gibbs, and Emmanuel J. Candès. Large language model validity via enhanced conformal prediction methods. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/d02ff1aeaa5c268d...

    2024

  6. [14]

    C. K. Chow. On optimum recognition error and reject trade-off. IEEE Transactions on Information Theory, 16 0 (1): 0 41--46, 1970

    1970

  7. [15]

    Supervising strong learners by amplifying weak experts

    Paul Christiano, Evan Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts. In arXiv preprint arXiv:1810.08575, 2018

    2018 arXiv

  8. [16]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  9. [17]

    Safe rlhf: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023

    2023 arXiv

  10. [18]

    Mogu: A framework for enhancing safety of llms while preserving their usability

    Yanrui Du, Sendong Zhao, Danyang Zhao, Ming Ma, Yuhan Chen, Liangyu Huo, Qing Yang, Dongliang Xu, and Bing Qin. Mogu: A framework for enhancing safety of llms while preserving their usability. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang...

    2024

  11. [19]

    S elective N et: A deep neural network with an integrated reject option

    Yonatan Geifman and Ran El-Yaniv. S elective N et: A deep neural network with an integrated reject option. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learnin...

    2019

  12. [20]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  13. [21]

    Ai safety via debate

    Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate. arXiv preprint arXiv:1805.00899, 2018

    2018 arXiv

  14. [22]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. In Thirty-seventh Conference on Neural Information Processing Syst...

    2023

  15. [23]

    Trust or escalate: Llm judges with provable guarantees for human agreement

    Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: Llm judges with provable guarantees for human agreement. arXiv preprint arXiv:2407.18370, 2025

    2025 arXiv

  16. [24]

    Conformal prediction with large language models for multi-choice question answering

    Bhawesh Kumar, Charles Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering. In Proceedings of the ICML 2023 Workshop on Neural Conversational AI: Teaching Machines to C...

    2023 arXiv

  17. [25]

    Sam Lightman, Nikita Nangia, and Samuel R. Bowman. Process supervision improves mathematical reasoning in chain-of-thought models. arXiv preprint arXiv:2305.20050, 2023

    2023 arXiv

  18. [26]

    Truthfulqa: Measuring how models mimic human falsehoods, 2022

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. URL https://arxiv.org/abs/2109.07958

    2022 arXiv

  19. [27]

    Self‑refine: Iterative refinement with self‑feedback

    Aman Madaan, Guangtao Tu, Yiming Chen, Yulia Tsvetkov, and Graham Neubig. Self‑refine: Iterative refinement with self‑feedback. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023

    2023

  20. [28]

    Language models with conformal factuality guarantees

    Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. arXiv preprint arXiv:2402.10978, 2024. URL https://arxiv.org/abs/2402.10978

    2024 arXiv

  21. [29]

    Liu, Yanda Chen, Percy Liang, and Tatsunori Hashimoto

    Isaac Ong, Pranav Patil, Shivang Agarwal, Harsh Gupta, Nelson F. Liu, Yanda Chen, Percy Liang, and Tatsunori Hashimoto. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665, 2024

    2024 arXiv

  22. [30]

    Gpt-4 system card, 2023

    OpenAI. Gpt-4 system card, 2023. https://openai.com/blog/gpt-4

    2023

  23. [31]

    Introducing gpt-4.1 in the api, April 2025

    OpenAI . Introducing gpt-4.1 in the api, April 2025. URL https://openai.com/index/gpt-4-1/. Accessed: 2025-05-15

    2025

  24. [32]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  25. [33]

    Aligning model properties via conformal risk control

    William Overman, Jacqueline Jil Vallon, and Mohsen Bayati. Aligning model properties via conformal risk control. In Advances in Neural Information Processing Systems, volume 37, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2024/hash/c79625091a4f8b5d3abe29f3b14fa4...

    2024

  26. [34]

    Jaakkola, and Regina Barzilay

    Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling. In Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2306.10193

    2024 arXiv

  27. [35]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

    2023 arXiv

  28. [36]

    Rewardedsoups: Towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards

    Alexandre Ram\'e, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewardedsoups: Towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In NeurIPS, 2023

    2023

  29. [37]

    Allen Z. Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, Zhenjia Xu, Dorsa Sadigh, Andy Zeng, and Anirudha Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. I...

    2023

  30. [38]

    Api is enough: Conformal prediction for large language models without logit-access, 2024

    Jiayuan Su, Jing Luo, Hongwei Wang, and Lu Cheng. Api is enough: Conformal prediction for large language models without logit-access, 2024. URL https://arxiv.org/abs/2403.01216

    2024 arXiv

  31. [39]

    Scrit: Self‑evolving critic for scalable oversight

    Yunhao Tang, Rohan Anil, Hyung Won Chung, Zhang Chen, Zhifeng Dai, and Barret Zoph. Scrit: Self‑evolving critic for scalable oversight. arXiv preprint arXiv:2403.09613, 2024

    2024 arXiv

  32. [40]

    Doing more with less -- implementing routing strategies in large language model-based systems: An extended survey

    Clovis Varangot-Reille, Olivier Caelen, Emelyne Goffinet, Alison Baumann, Alexandre Chauvet, and Patrick von Platen. Doing more with less -- implementing routing strategies in large language model-based systems: An extended survey. arXiv preprint arXiv:2502.00409, 2025

    2025 arXiv

  33. [41]

    Algorithmic Learning in a Random World, Second Edition

    Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World, Second Edition. January 2005. doi:10.1007/978-3-031-06649-8. Springer-Verlag New York, Inc. 2005

    2005 doi

  34. [42]

    Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards

    Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024

    2024

  35. [43]

    Deepcritic: Large language model critics for scalable oversight

    Hanjiang Yang, Tianyu Fu, Xu Wang, Yao Yao, Sean Welleck, Etienne Levin, Anqi Nie, Kyunghyun Cho, and Jason Weston. Deepcritic: Large language model critics for scalable oversight. arXiv preprint arXiv:2402.05497, 2024

    2024 arXiv

  36. [44]

    Large language model cascades with mixture of thought representations for cost-efficient reasoning

    Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thought representations for cost-efficient reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=6okaSfANzh

    2024

  37. [45]

    Zellinger, Rex Liu, and Matt Thomson

    Michael J. Zellinger, Rex Liu, and Matt Thomson. Cost-saving llm cascades with early abstention. arXiv preprint arXiv:2502.09054, 2025

    2025 arXiv

  38. [46]

    Bi-factorial preference optimization: Balancing safety-helpfulness in language models

    Wenxuan Zhang, Philip Torr, Mohamed Elhoseiny, and Adel Bibi. Bi-factorial preference optimization: Balancing safety-helpfulness in language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=GjM61KRiTG

    2025

  39. [47]

    Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization

    Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023

    2023 arXiv

  40. [48]

    Improving alignment and robustness with circuit breakers

    Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. arXiv preprint arXiv:2406.04313, 2024

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.