Pith. sign in

REVIEW 2 major objections 5 minor 113 references

Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints

T0 review · 2 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pointwise constraints cut tail violations in language-model fine-tuning while keeping average performance.

desk verdict Clean dual recipe for pointwise LM constraints that actually shrinks violation tails on three small-model tasks; ordinary dual-method caveats, not a broken argument. read the letter →

arxiv 2607.08968 v1 pith:CAIULV7A submitted 2026-07-09 eess.SP

Ignacio Hounie , Ignacio Boero , Alejandro Ribeiro This is my paper
classification eess.SP
keywords pointwiseconstraintsaugmentedLagrangianlanguage-modelfine-tuningsafetyalignmentpreferenceoptimizationre-rankingsample-dependentrelaxation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Language-model fine-tuning often needs hard requirements (safety, correct tool use, relevance) to hold for every input, not just on average. Average constraints or fixed penalties can still leave large failures on hard or rare samples. This paper formulates fine-tuning as an average-loss objective subject to per-sample inequality constraints, then solves it with an augmented Lagrangian that carries one dual multiplier per training example. When constraints are too strict or data contain outliers, a learned sample-dependent relaxation trades a user-chosen cost against the training objective. Across three small-model tasks—safe instruction following, function-calling preferences, and length-aware re-ranking—the method shrinks the right tail of the violation distribution and yields more even downstream trade-offs than average constraints or common preference losses.

What carries the argument

The empirical augmented Lagrangian (and its quadratically regularized dual for the relaxed problem) that assigns a nonnegative multiplier to each training sample and is maximized by dual ascent interleaved with a few primal steps on the language-model parameters.

What would settle it

On a held-out test split of any of the three tasks, the empirical CDF of constraint values for the pointwise method fails to dominate the average-constraint and fixed-penalty baselines in the upper tail (high-violation region), or the reported CVaR of violations is no smaller while the average objective is worse.

Watch

Extended reading notes

Core claim

Enforcing constraints almost everywhere, rather than only in expectation, and optimizing the resulting dual with sample-wise multipliers and a quadratic augmentation term, reduces both the frequency and severity of large per-sample violations while still attaining competitive average objective values and task metrics.

Load-bearing premise

A handful of inexact gradient steps on the model parameters, together with a hand-chosen fixed augmentation strength, is enough for the dual iterates to recover near-feasible solutions that generalize beyond the training samples.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a pointwise constrained supervised fine-tuning framework for language models: minimize an average objective subject to almost-everywhere (per-sample) inequality constraints, rather than average constraints. To handle overly tight constraints and outliers it introduces a learned sample-dependent relaxation u with a user-chosen convex cost (instantiated as quadratic β‖u‖²), and solves the resulting problem via an empirical augmented Lagrangian dual with sample-wise multipliers and a fixed augmentation parameter α (Algorithm 1). Strong duality for the empirical relaxed problem is proved under mild assumptions (Prop. 3, App. B). The framework is instantiated on three small-LM tasks—safety refusal in instruction following, preference constraints in tool calling, and length-aware re-ranking—and compared to average constraints, fixed penalties, and task-specific baselines. Empirically, pointwise (and relaxed) methods reduce the tails of the constraint-violation distribution (CDFs, CVaR) while preserving or improving average objective and downstream metrics, often expanding the Pareto frontier (Figs. 1–4, Tables 2, 10–12).

Significance. If the claims hold, the work supplies a clean, reusable dual formulation that directly targets sample-level disparities that average-constrained alignment can leave unaddressed. The combination of pointwise constraints, resilient quadratic relaxation (Prop. 1), and an AL dual tailored to non-convex LM fine-tuning is a natural and useful extension of constrained learning theory to SFT. Strengths include: (i) clean duality derivations and a strong-duality result for the empirical relaxed problem; (ii) three distinct, carefully specified task instantiations that demonstrate flexibility; (iii) transparent empirical reporting (seed means ± std, full CDFs, CVaR, Pareto fronts, ablations on α/β, and explicit negative cases in E.4). The contribution is methods-oriented rather than a new theoretical guarantee for non-convex LMs, but it is a solid and timely addition to constrained alignment.

major comments (2)
  1. §3.3 and Algorithm 1 (and App. C): The central practical claim—that modest inexact primal steps plus a hand-chosen fixed α recover near-feasible, generalizing solutions—rests on the cited AL-COLE theory without finite-sample rates for the non-convex LM setting. The paper already scopes this as a limitation and shows empirical stability to α (App. F.2) and out-of-sample CDFs/CVaR, so it does not overturn the results; still, a short discussion of when the dual may fail to generalize (linking to the E.4 negative cases) would strengthen the load-bearing optimization claim.
  2. §4 / Tables 2, 10–12 and Figs. 1–4: The evidence that tail reduction improves downstream metrics is convincing on the reported small models and datasets, but the paper does not quantify how much of the gain is due to the sample-wise multipliers versus the particular choice of ε and β. A more systematic sensitivity analysis (or a simple ablation that freezes the dual variables after a few epochs) would make H2 more robust; without it the Pareto improvements remain empirical rather than fully attributed.
minor comments (5)
  1. Notation: the same symbol ε is used for different constraint levels across tasks (ε_U, ε_H, ε_win, ε_lose, margin ε); a short table of task-specific tolerances would help.
  2. Fig. 1 and related CDFs: axis labels and the meaning of positive vs. negative constraint values could be stated more explicitly in the caption for readers who skip §4.1.
  3. App. A / related work: the connection to soft-margin SVMs and scenario optimization is useful; a sentence clarifying that the present work targets infinite-dimensional (functional) duals rather than finite-sample scenario bounds would avoid confusion.
  4. Typos / polish: occasional missing spaces after periods and a few long sentences in §3.2–3.3; a light copy-edit would improve readability.
  5. Reproducibility: runtime/memory overhead is discussed (App. F) and is negligible; releasing the dual-ascent implementation or a minimal reference script would further help adoption.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant circularity: dual construction and empirical claims are independent of fitted targets; only minor self-citation of authors' AL theory for non-convex guarantees.

  1. self citation load bearing [§3.1 (after Eq. 6) and §3.3 / App. C (Alg. 1)]
    "under the conditions studied in [27], an approximate solution for a constrained learning problem can be obtained via the primal minimizer of the empirical augmented Lagrangian at the optimal augmented dual pair [27, Theorem 2.3]. ... convergence can still be guaranteed as long as the primal minimization error decreases sufficiently across iterations [27, Theorem 3.1]."

    The paper invokes its own concurrent AL-COLE preprint for the key non-convex duality-gap and inexact-ascent guarantees that justify using fixed-α dual ascent with few primal steps. This is a self-citation of overlapping authors, but it is not load-bearing for the central empirical claims (tail reduction, Pareto improvements), which are validated independently on held-out CDFs/CVaR/downstream metrics; the citation merely supplies theoretical scaffolding already partially re-derived for the relaxed case in App. B.

full rationale

The derivation chain is standard constrained optimization applied to LM fine-tuning: primal (P) with pointwise inequalities, Lagrangian dual, then augmented Lagrangian (6) to address non-convexity, plus quadratic relaxation whose dual is derived in Prop. 1 (rescaling + regularization of gA). These steps are algebraic identities or classical AL results (Rockafellar, Wierzbicki–Kurcyusz), not self-definitional. Hyperparameters ε, β, α are free knobs whose effects are ablated (Figs. 1, 4; App. F), not fitted then re-labeled as predictions. Downstream metrics (CVaR tails, F1, LC-WR, MRR/LenRank, Pareto) are held-out and distinct from the dual objective, so empirical claims C1–C4/H1–H2 stand independently. The sole self-citation of note is [27] (same authors) for AL strong duality / inexact primal-dual convergence (Thm. 2.3, 3.1); this supplies theoretical motivation but is not load-bearing for the reported reductions in violation tails, which rest on the experiments themselves. No uniqueness theorem forbids alternatives, no ansatz is smuggled, and soft-margin SVM is correctly acknowledged as a classical special case rather than renamed. Score 1 reflects only that minor self-citation; the paper is otherwise self-contained against external benchmarks.

Assumptions & free parameters 4 free parameters · 3 assumptions · 1 invented entities

The central empirical claims rest on standard non-convex duality theory, the modeling choice that pointwise almost-everywhere constraints are the right target, and a handful of free dual/relaxation hyper-parameters that are grid-searched rather than derived.

free parameters (4)
  • relaxation cost β
    Controls the quadratic penalty on dual multipliers / primal slacks; swept over orders of magnitude and shown to trade objective vs. constraint tails.
  • augmentation parameter α
    Fixed by hand (often 1 or 1000); ablation shows relative stability but remains a free choice.
  • constraint tolerances ε (ε_U, ε_H, ε_win, ε_lose, margin ε)
    Task-specific thresholds set by the user or grid search; directly define the feasible set.
  • dual learning rate η
    Chosen by grid search; affects stability of the dual ascent.
assumptions (3)
  • domain assumption The empirical augmented dual with inexact primal steps converges to a near-optimal dual pair when α is large enough (shifted-penalty theory).
    Invoked in §3.3 and Appendix C; relies on [27,40] without new finite-sample rates for transformers.
  • domain assumption Pointwise almost-everywhere constraints are the appropriate modeling target for safety/preference/length requirements.
    Stated as (C1) and used to define problem (P); alternative high-probability formulations are discussed but not used.
  • standard math Standard continuity, compactness, and bounded-loss assumptions that guarantee strong duality of the empirical relaxed problem.
    Assumptions 1–3 in Appendix B.4; mild and standard for the setting.
invented entities (1)
  • sample-dependent relaxation function u(x,y) with quadratic cost c(u)=β‖u‖²
    purpose: Softens overly restrictive pointwise constraints while remaining dual-regularizable.
    Defined in (PR); classical soft-margin idea lifted to functional dual form; no independent physical existence claimed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints." pith.science (2026). https://pith.science/paper/CAIULV7A

@misc{pith2026260708968,
  author       = {Pith},
  title        = {Pith review of: Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CAIULV7A}},
  note         = {Machine review of arXiv:2607.08968}
}
read the original abstract

Fine-tuning language models often requires enforcing constraints on individual inputs without compromising downstream performance. Existing constrained alignment methods impose constraints on average, which can induce undesirable disparities across inputs or users. We propose a novel alignment framework that addresses this gap by enforcing per-sample constraints while still minimizing an average loss. To mitigate the impact of overly restrictive constraints and outliers, we introduce a learned, sample-dependent relaxation that minimally adjusts the constraints, trading off a user-defined relaxation cost with the training objective. To address practical duality and optimization challenges, we develop an augmented Lagrangian approach tailored to this formulation. We demonstrate the flexibility of the framework by instantiating it under distinct small language-model fine-tuning tasks and constraints: safety in instruction following, preferences in function calling and length in re-ranking. Across these settings, our approach reduces tail constraint violations while largely preserving the model's performance.

Figures

Figures reproduced from arXiv: 2607.08968 by the authors.

Figure 1
Figure 1. Empirical Cumulative Distribution Function (CDF) of sample-wise constraint values [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Length normalised accuracy, Hallucination Rates and F1 scores for Llama-3.2-1B [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Safety vs Helpfulness trade-off. (Left) length controlled win rates on AlpacaEval [ [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: MRR@10 for ranking the positive passage versus LenRank@10 metric. The base model is [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p034_5.png]
Figure 6
Figure 6. Figure 6: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p034_6.png]
Figure 7
Figure 7. Figure 7: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]
Figure 8
Figure 8. Figure 8: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]
Figure 9
Figure 9. Figure 9: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]
Figure 10
Figure 10. Figure 10: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]
Figure 11
Figure 11. Figure 11: Empirical CDF of the tokenwise average log likelihood, i.e. [PITH_FULL_IMAGE:figures/full_fig_p036_11.png]
Figure 12
Figure 12. Figure 12: Empirical Cumulative Density Function (CDF) of length normalized likelihood for unsafe [PITH_FULL_IMAGE:figures/full_fig_p037_12.png]
Figure 14
Figure 14. Figure 14: The ablation also shows that the augmented Lagrangian is substantially more robust to the choice of dual step size. For the standard Lagrangian, decreasing the dual step size leads to a large degradation in MRR@10, from 0.711 at step size 1.0 to 0.488 at step size 10−…
Figure 13
Figure 13. Figure 13: Empirical Cumulative Density Function (CDF) of likelihood ratios for preference data. In [PITH_FULL_IMAGE:figures/full_fig_p039_13.png]
Figure 14
Figure 14. Figure 14: Comparison between the standard dual formulation and the augmented dual formulation [PITH_FULL_IMAGE:figures/full_fig_p039_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

113 extracted references · 1 canonical work pages

  1. [1]

    A comprehensive survey on evaluating large language model applications in the medical industry.arXiv preprint arXiv:2404.15777, 2024

    Yining Huang, Keke Tang, Meilian Chen, and Boyuan Wang. A comprehensive survey on evaluating large language model applications in the medical industry.arXiv preprint arXiv:2404.15777, 2024

    arXiv 2024

  2. [2]

    On large language models in mission-critical it governance: Are we ready yet?, 2025

    Matteo Esposito, Francesco Palagiano, Valentina Lenarduzzi, and Davide Taibi. On large language models in mission-critical it governance: Are we ready yet?, 2025. URL https: //arxiv.org/abs/2412.11698

    arXiv 2025

  3. [3]

    Safe RLHF: Safe reinforcement learning from human feedback

    Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  4. [4]

    One-shot safety alignment for large language models via optimal dualization.Advances in Neural Information Processing Systems, 37:84350–84383, 2024

    Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-shot safety alignment for large language models via optimal dualization.Advances in Neural Information Processing Systems, 37:84350–84383, 2024

    2024

  5. [5]

    Stepwise alignment for constrained language model policy optimization.Advances in Neural Information Processing Systems, 37:104471–104520, 2024

    Akifumi Wachi, Thien Q Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. Stepwise alignment for constrained language model policy optimization.Advances in Neural Information Processing Systems, 37:104471–104520, 2024

    2024

  6. [6]

    Enhancing LLM safety via constrained direct preference optimization.arXiv preprint arXiv:2403.02475, 2024

    Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing LLM safety via constrained direct preference optimization.arXiv preprint arXiv:2403.02475, 2024

    arXiv 2024

  7. [7]

    Enhancing safety in reinforcement learning with human feedback via rectified policy optimization.arXiv preprint arXiv:2410.19933, 2024

    Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, and Xin Liu. Enhancing safety in reinforcement learning with human feedback via rectified policy optimization.arXiv preprint arXiv:2410.19933, 2024

    arXiv 2024

  8. [8]

    L3ms–lagrange large language models.arXiv preprint arXiv:2410.21533, 2024

    Guneet S Dhillon, Xingjian Shi, Yee Whye Teh, and Alex Smola. L3ms–lagrange large language models.arXiv preprint arXiv:2410.21533, 2024

    arXiv 2024

Show all 113 references
  1. [9]

    Adversarial training for high-stakes reliability.Advances in neural information processing systems, 35: 9274–9286, 2022

    Daniel Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Benjamin Weinstein-Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability.Advances in neural information processing systems, 35: 9274–...

    2022

  2. [10]

    Confronting reward model overoptimization with constrained RLHF

    Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://ope...

    2024

  3. [11]

    Large lan- guage models struggle to learn long-tail knowledge

    Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large lan- guage models struggle to learn long-tail knowledge. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors,Proceed- ings of the 40...

    2023

  4. [12]

    The devil is in the tails: How long-tailed code distributions impact large language models

    Xin Zhou, Kisub Kim, Bowen Xu, Jiakun Liu, DongGyun Han, and David Lo. The devil is in the tails: How long-tailed code distributions impact large language models. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 40–52,

  5. [13]

    doi: 10.1109/ASE56229.2023.00157

    2023 doi

  6. [14]

    The neglected tails in vision-language models

    Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12988–12997, June 2024. 10

    2024

  7. [15]

    The law of knowledge overshadowing: Towards understanding, predicting, and preventing llm hallucination.arXiv preprint arXiv:2502.16143, 2025

    Yuji Zhang, Sha Li, Cheng Qian, Jiateng Liu, Pengfei Yu, Chi Han, Yi R Fung, Kathleen McKeown, Chengxiang Zhai, Manling Li, et al. The law of knowledge overshadowing: Towards understanding, predicting, and preventing llm hallucination.arXiv preprint arXiv:2502.16143, 2025

    2025 arXiv

  8. [16]

    Is gpt-oss good? a comprehensive evaluation of openai’s latest open source models.arXiv preprint arXiv:2508.12461, 2025

    Ziqian Bi, Keyu Chen, Chiung-Yi Tseng, Danyang Zhang, Tianyang Wang, Hongying Luo, Lu Chen, Junming Huang, Jibin Guan, Junfeng Hao, et al. Is gpt-oss good? a comprehensive evaluation of openai’s latest open source models.arXiv preprint arXiv:2508.12461, 2025

    2025

  9. [17]

    Probably approximately correct constrained learning

    Luiz Chamon and Alejandro Ribeiro. Probably approximately correct constrained learning. Advances in Neural Information Processing Systems, 33:16722–16735, 2020

    2020

  10. [18]

    Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach, 2022

    Qinbo Bai, Amrit Singh Bedi, Mridul Agarwal, Alec Koppel, and Vaneet Aggarwal. Achieving zero constraint violation for constrained reinforcement learning via primal-dual approach, 2022. URLhttps://arxiv.org/abs/2109.06332

    2022 arXiv

  11. [19]

    Strong duality relations in nonconvex risk- constrained learning

    Dionysis Kalogerias and Spyridon Pougkakiotis. Strong duality relations in nonconvex risk- constrained learning. In2024 58th Annual Conference on Information Sciences and Systems (CISS), pages 1–6, 2024. doi: 10.1109/CISS59072.2024.10480186

    2024 doi

  12. [20]

    Andrei Manolache, Luiz F. O. Chamon, and Mathias Niepert. Learning (approximately) equivariant networks via constrained optimization, 2025. URL https://arxiv.org/abs/ 2505.13631

    2025

  13. [21]

    Constrained sliced wasserstein embedding, 06 2025

    Navid Naderializadeh, Darian Salehi, Xinran Liu, and Soheil Kolouri. Constrained sliced wasserstein embedding, 06 2025

    2025

  14. [22]

    Near-optimal solutions of constrained learning problems.arXiv preprint arXiv:2403.11844, 2024

    Juan Elenter, Luiz FO Chamon, and Alejandro Ribeiro. Near-optimal solutions of constrained learning problems.arXiv preprint arXiv:2403.11844, 2024

    2024 arXiv

  15. [23]

    Simultaneous multi-robot motion planning with projected diffusion models.arXiv preprint arXiv:2502.03607, 2025

    Jinhao Liang, Jacob K Christopher, Sven Koenig, and Ferdinando Fioretto. Simultaneous multi-robot motion planning with projected diffusion models.arXiv preprint arXiv:2502.03607, 2025

    2025 arXiv

  16. [24]

    Qifeng Hu, Shamsulhaq Basir, and Inanc Senocak. Conditionally adaptive augmented lagrangian method for physics-informed learning of forward and inverse problems using artificial neural networks.arXiv preprint arXiv:2508.15695, 2025

    2025

  17. [25]

    Safe: Finding sparse and flat minima to improve pruning.arXiv preprint arXiv:2506.06866, 2025

    Dongyeop Lee, Kwanhee Lee, Jinseok Chung, and Namhoon Lee. Safe: Finding sparse and flat minima to improve pruning.arXiv preprint arXiv:2506.06866, 2025

    2025 arXiv

  18. [26]

    Constrained discrete diffusion.arXiv preprint arXiv:2503.09790, 2025

    Michael Cardei, Jacob K Christopher, Thomas Hartvigsen, Bhavya Kailkhura, and Ferdinando Fioretto. Constrained discrete diffusion.arXiv preprint arXiv:2503.09790, 2025

    2025

  19. [27]

    Dual optimistic ascent (pi control) is the augmented lagrangian method in disguise.arXiv preprint arXiv:2509.22500, 2025

    Juan Ramirez and Simon Lacoste-Julien. Dual optimistic ascent (pi control) is the augmented lagrangian method in disguise.arXiv preprint arXiv:2509.22500, 2025

    2025

  20. [28]

    Al-cole: Augmented lagrangian for constrained learning.arXiv preprint arXiv:2510.20995, 2025

    Ignacio Boero, Ignacio Hounie, and Alejandro Ribeiro. Al-cole: Augmented lagrangian for constrained learning.arXiv preprint arXiv:2510.20995, 2025

    2025

  21. [29]

    Xstest: A test suite for identifying exaggerated safety behaviours in large language models

    Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for...

    2024

  22. [30]

    Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    Geon-Hyeong Kim, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, Youngsoo Jang, and Moontae Lee. Safedpo: A simple approach to direct preference optimization with enhanced safety.arXiv preprint arXiv:2505.20065, 2025

    2025

  23. [31]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. InThe Twelfth International Conference on Learning Repr...

  24. [32]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. InProceedings of the Advances in Neural Information Processing Systems, volume 36, 2024

    2024

  25. [33]

    Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward.Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  26. [34]

    Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.arXiv preprint arXiv:2401.08417, 2024

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Ken- ton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.arXiv preprint arXiv:2401.08417, 2024

    2024 arXiv

  27. [35]

    Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic- hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023

    2023 arXiv

  28. [36]

    Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks.Advances in neural information processing systems,...

    2020

  29. [37]

    Tyrrell Rockafellar

    R. Tyrrell Rockafellar. Augmented lagrange multiplier functions and duality in nonconvex programming.SIAM Journal on Control, 12(2):268–285, 1974. doi: 10.1137/0312021

    1974 doi

  30. [38]

    The augmented lagrangian methods: Overview and recent advances.arXiv preprint arXiv:2510.16827, 2025

    Kangkang Deng, Rui Wang, Zhenyuan Zhu, Junyu Zhang, and Zaiwen Wen. The augmented lagrangian methods: Overview and recent advances.arXiv preprint arXiv:2510.16827, 2025

    2025

  31. [39]

    Resilient constrained learning

    Ignacio Hounie, Alejandro Ribeiro, and Luiz FO Chamon. Resilient constrained learning. Advances in Neural Information Processing Systems, 36, 2024

    2024

  32. [40]

    Rockafellar.Convex Analysis

    R.T. Rockafellar.Convex Analysis. Princeton Landmarks in Mathematics and Physics. Princeton University Press, 1997. ISBN 9780691015866. URL https://books.google.com/books? id=GV6YDwAAQBAJ

    1997

  33. [41]

    Wierzbicki and Stanislaw Kurcyusz

    Andrzej P. Wierzbicki and Stanislaw Kurcyusz. Projection on a cone, penalty functionals and duality theory for problems with inequaltity constraints in hilbert space.SIAM Journal on Control and Optimization, 15(1):25–56, 1977. doi: 10.1137/0315003. URL https://doi. org/10.1137/0315003

    1977 doi

  34. [42]

    Reproducibility in machine-learning-based research: Overview, barriers, and drivers.AI Magazine, 46(2):e70002, 2025

    Harald Semmelrock, Tony Ross-Hellauer, Simone Kopeinik, Dieter Theiler, Armin Haberl, Stefan Thalmann, and Dominik Kowald. Reproducibility in machine-learning-based research: Overview, barriers, and drivers.AI Magazine, 46(2):e70002, 2025

    2025

  35. [43]

    Small language models are the future of agentic ai,

    Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. Small language models are the future of agentic ai,

  36. [44]

    URLhttps://arxiv.org/abs/2506.02153

    arXiv

  37. [45]

    Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, et al. A comprehensive survey of small language models in the era of large language models: Techniques, enhancements, applications, collaboration with llms,...

    2025

  38. [46]

    When2call: When (not) to call tools

    Hayley Ross, Ameya Sunil Mahabaleshwarkar, and Yoshi Suhara. When2call: When (not) to call tools. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), ...

    2025

  39. [47]

    The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. InForty-second International Conference on Machine ...

    2025

  40. [48]

    Apigen: Generative api method recommendation

    Yujia Chen, Cuiyun Gao, Muyijie Zhu, Qing Liao, Yong Wang, and Guoai Xu. Apigen: Generative api method recommendation. In2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 171–182. IEEE, 2024

    2024

  41. [49]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  42. [50]

    Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay.arXiv preprint arXiv:2504.03601, 2025

    Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al. Apigen-mt: Agentic pipeline for multi-turn data generation via simulated agent-human interplay.arXiv preprint arXiv:2504.03601, 2025

    2025 arXiv

  43. [51]

    Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning

    Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. InInternational Conference on Machine Learning, pages 60674–60703. PMLR, 2024

    2024

  44. [52]

    Stanford alpaca: An instruction-following llama model, 2023

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023

    2023

  45. [53]

    Beavertails: Towards improved safety alignment of LLM via a human-preference dataset

    Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via a human-preference dataset. InProceedings of the Advances in Neural Information Processing Syst...

    2024

  46. [54]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023

  47. [55]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models, 2023

    2023

  48. [56]

    Ms marco: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset.arXiv preprint arXiv:1611.09268, 2016

    2016 arXiv

  49. [57]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirectional encoder f...

    2025

  50. [58]

    Multi-stage document ranking with bert.arXiv preprint arXiv:1910.14424, 2019

    Rodrigo Nogueira, Wei Yang, Kyunghyun Cho, and Jimmy Lin. Multi-stage document ranking with bert.arXiv preprint arXiv:1910.14424, 2019

    1910 arXiv

  51. [59]

    C- pack: Packed resources for general chinese embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C- pack: Packed resources for general chinese embeddings. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649, 2024

    2024

  52. [60]

    cross-encoder/ms-marco-minilm-l12-v2

    Sentence Transformers. cross-encoder/ms-marco-minilm-l12-v2. https://huggingface. co/cross-encoder/ms-marco-MiniLM-L12-v2 , 2026. Hugging Face model card, accessed 2026-04-21

    2026

  53. [61]

    Pattern recognition using generalized portrait method.Automa- tion and Remote Control, 24:774–780, 1963

    Vladimir Naumovich Vapnik. Pattern recognition using generalized portrait method.Automa- tion and Remote Control, 24:774–780, 1963. URL https://api.semanticscholar.org/ CorpusID:115205884. 13

    1963

  54. [62]

    Boser, Isabelle M

    Bernhard E. Boser, Isabelle M. Guyon, and Vladimir N. Vapnik. A training algorithm for optimal margin classifiers. InProceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT ’92, page 144–152, New York, NY , USA, 1992. Association for Computing Machinery...

    1992 doi

  55. [63]

    Support-vector networks.Machine learning, 20(3): 273–297, 1995

    Corinna Cortes and Vladimir Vapnik. Support-vector networks.Machine learning, 20(3): 273–297, 1995

    1995

  56. [64]

    A theory of the risk for optimization with relaxation and its application to support vector machines.Journal of Machine Learning Research, 22(288): 1–38, 2021

    Marco C Campi and Simone Garatti. A theory of the risk for optimization with relaxation and its application to support vector machines.Journal of Machine Learning Research, 22(288): 1–38, 2021

    2021

  57. [65]

    Non-convex scenario optimization.Mathematical Programming, 209(1):557–608, 2025

    Simone Garatti and Marco C Campi. Non-convex scenario optimization.Mathematical Programming, 209(1):557–608, 2025

    2025

  58. [66]

    Feasible learning, 2025

    Juan Ramirez, Ignacio Hounie, Juan Elenter, Jose Gallego-Posada, Meraj Hashemizadeh, Alejandro Ribeiro, and Simon Lacoste-Julien. Feasible learning, 2025. URL https://arxiv. org/abs/2501.14912

    2025 arXiv

  59. [67]

    Reinforce- ment learning with almost sure constraints

    Agustin Castellano, Hancheng Min, Enrique Mallada, and Juan Andrés Bazerque. Reinforce- ment learning with almost sure constraints. In Roya Firoozi, Negar Mehr, Esen Yel, Rika Antonova, Jeannette Bohg, Mac Schwager, and Mykel Kochenderfer, editors,Proceedings of The 4th Annual...

    2022

  60. [68]

    Anytime-constrained reinforcement learning

    Jeremy McMahan and Xiaojin Zhu. Anytime-constrained reinforcement learning. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li, editors,Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 ofProceedings of Machine Learning Rese...

    2024

  61. [69]

    Sauté rl: Almost surely safe reinforcement learning using state augmentation

    Aivar Sootla, Alexander I Cowen-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar. Sauté rl: Almost surely safe reinforcement learning using state augmentation. InInternational Conference on Machine Learning, pages 20423–20443. PMLR, 2022

    2022

  62. [70]

    On tilted losses in machine learning: Theory and applications.Journal of Machine Learning Research, 24(142):1–79, 2023

    Tian Li, Ahmad Beirami, Maziar Sanjabi, and Virginia Smith. On tilted losses in machine learning: Theory and applications.Journal of Machine Learning Research, 24(142):1–79, 2023. URLhttp://jmlr.org/papers/v24/21-1095.html

    2023

  63. [71]

    Large-scale methods for distributionally robust optimization.Advances in neural information processing systems, 33: 8847–8860, 2020

    Daniel Levy, Yair Carmon, John C Duchi, and Aaron Sidford. Large-scale methods for distributionally robust optimization.Advances in neural information processing systems, 33: 8847–8860, 2020

    2020

  64. [72]

    Probabilistically robust learning: Balancing average and worst-case performance

    Alexander Robey, Luiz Chamon, George J Pappas, and Hamed Hassani. Probabilistically robust learning: Balancing average and worst-case performance. InInternational Conference on Machine Learning, pages 18667–18686. PMLR, 2022

    2022

  65. [73]

    Robust llm alignment via distributionally robust direct preference optimization

    Zaiyan Xu, Sushil Vemuri, Kishan Panaganti, Dileep Kalathil, Rahul Jain, and Deepak Ra- machandran. Robust llm alignment via distributionally robust direct preference optimization. arXiv preprint arXiv:2502.01930, 2025

    2025

  66. [74]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. InProceedings of the Advances in Neural Information Processing Systems, volume 33, pages 3008–3021, 2020

    2020

  67. [75]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  68. [76]

    Rrhf: Rank responses to align language models with human feedback without tears.arXiv preprint arXiv:2304.05302, 2023

    Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears.arXiv preprint arXiv:2304.05302, 2023

    2023 arXiv

  69. [77]

    Repo: Understanding preference learning through relu-based optimization

    Junkang Wu, Kexin Huang, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, and Xiang Wang. Repo: Understanding preference learning through relu-based optimization. arXiv preprint arXiv:2503.07426, 2025

    2025

  70. [78]

    A general theoretical paradigm to understand learn- ing from human preferences

    Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learn- ing from human preferences. InProceedings of the International Conference on Artificial Intelligence ...

    2024

  71. [79]

    Cal-dpo: Calibrated direct preference optimization for language model alignment.Advances in Neural Information Processing Systems, 37:114289–114320, 2024

    Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G Honavar. Cal-dpo: Calibrated direct preference optimization for language model alignment.Advances in Neural Information Processing Systems, 37:114289–114320, 2024

    2024

  72. [80]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    2022 arXiv

  73. [81]

    Pku-saferlhf: Towards multi-level safety alignment for llms with human preference

    Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Alex Qiu, Jiayi Zhou, Kaile Wang, Boxun Li, et al. Pku-saferlhf: Towards multi-level safety alignment for llms with human preference. InProceedings of the 63rd Annual Meeting of the Association...

    2025

  74. [82]

    Alignment of large language models with constrained learning.arXiv preprint arXiv:2505.19387, 2025

    Botong Zhang, Shuo Li, Ignacio Hounie, Osbert Bastani, Dongsheng Ding, and Alejandro Ribeiro. Alignment of large language models with constrained learning.arXiv preprint arXiv:2505.19387, 2025

    2025

  75. [83]

    Cat-dpo: Category- adaptive safety alignment.arXiv preprint arXiv:2604.17299, 2026

    Tiankai Yang, Yi Nian, Xinyuan Li, Ruiyao Xu, Kaize Ding, and Yue Zhao. Cat-dpo: Category- adaptive safety alignment.arXiv preprint arXiv:2604.17299, 2026

    2026 arXiv

  76. [84]

    Bi-factorial preference op- timization: Balancing safety-helpfulness in language models.arXiv preprint arXiv:2408.15313, 2024

    Wenxuan Zhang, Philip HS Torr, Mohamed Elhoseiny, and Adel Bibi. Bi-factorial preference op- timization: Balancing safety-helpfulness in language models.arXiv preprint arXiv:2408.15313, 2024

    2024 arXiv

  77. [85]

    Midpo: Dual preference optimization for safety and helpfulness in large language models via a mixture of experts framework.arXiv preprint arXiv:2506.02460, 2025

    Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, and Lixin Cui. Midpo: Dual preference optimization for safety and helpfulness in large language models via a mixture of experts framework.arXiv preprint arXiv:2506.02460, 2025

    2025 arXiv

  78. [86]

    Mitigating the safety alignment tax with null-space constrained policy optimization.arXiv preprint arXiv:2512.11391, 2025

    Yifan Niu, Han Xiao, Dongyi Liu, Nuo Chen, and Jia Li. Mitigating the safety alignment tax with null-space constrained policy optimization.arXiv preprint arXiv:2512.11391, 2025

    2025

  79. [87]

    Rule based rewards for fine-grained llm safety

    Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian D Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for fine-grained llm safety. InICML 2024 Next Generation of AI Safety Workshop, 2024

    2024

  80. [88]

    Controllable preference optimization: Toward controllable multi-objective alignment

    Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, et al. Controllable preference optimization: Toward controllable multi-objective alignment. InProceedings of the 2024 Conference on Empirical Methods in Natural Lan...

    2024

  81. [89]

    Metaaligner: Conditional weak-to-strong correction for generalizable multi-objective alignment of language models.arXiv preprint arXiv:2403.17141, 2024

    Kailai Yang, Zhiwei Liu, Qianqian Xie, Tianlin Zhang, Nirui Song, Jimin Huang, Ziyan Kuang, and Sophia Ananiadou. Metaaligner: Conditional weak-to-strong correction for generalizable multi-objective alignment of language models.arXiv preprint arXiv:2403.17141, 2024

    2024 arXiv

  82. [90]

    Nonlinear programming.Journal of the Operational Research Society, 48 (3):334–334, 1997

    D P Bertsekas. Nonlinear programming.Journal of the Operational Research Society, 48 (3):334–334, 1997. doi: 10.1057/palgrave.jors.2600425. URL https://doi.org/10.1057/ palgrave.jors.2600425. 15

    1997 doi

  83. [91]

    Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    2023 arXiv

  84. [92]

    Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models

    Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, et al. Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models. InProceedings of the 2024 Conference on Empir...

    2024

  85. [93]

    ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Luis Chiruzzo, A...

    2025 doi

  86. [94]

    Lora: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    2022

  87. [95]

    Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  88. [96]

    Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837, 2025

    Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming.arXiv preprint arXiv:2501.18837, 2025

    2025 arXiv

  89. [97]

    Shieldgemma 2: Robust and tractable image content moderation.arXiv preprint arXiv:2504.01081, 2025

    Wenjun Zeng, Dana Kurniawan, Ryan Mullins, Yuchi Liu, Tamoghna Saha, Dirichi Ike-Njoku, Jindong Gu, Yiwen Song, Cai Xu, Jingjing Zhou, et al. Shieldgemma 2: Robust and tractable image content moderation.arXiv preprint arXiv:2504.01081, 2025

    2025 arXiv

  90. [98]

    Safety through reasoning: An empirical study of reasoning guardrail models.Findings of the Association for Computational Linguistics: EMNLP, 2025:21862–21880, 2025

    Makesh Narsimhan Sreedhar, Traian Rebedea, and Christopher Parisien. Safety through reasoning: An empirical study of reasoning guardrail models.Findings of the Association for Computational Linguistics: EMNLP, 2025:21862–21880, 2025

    2025

  91. [99]

    OR-bench: An over-refusal benchmark for large language models

    Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. OR-bench: An over-refusal benchmark for large language models. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the ...

    2025

  92. [100]

    kernel trick

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  93. [101]

    We use the When2call dataset [44] that aims to evaluate the decision-making capabilities of language models (LMs) when interacting with external tools

    and the Berkeley Function Calling Leaderboard (BFCL) [ 45], which primarily focus on the accuracy of tool selection and parameter filling, recent research highlights common failure modes when the correct tool is not provided or the user does not provide enough information to s...

  94. [102]

    2.Tool call:A correctly formatted call to an available tool

    Direct answer:A text response without using tools (which, in the context of this benchmark, constitutes a hallucination). 2.Tool call:A correctly formatted call to an available tool. 3.Follow-up question:Requesting missing information required by the tool parameters. 25 Table ...

  95. [103]

    Unable to answer:Correctly stating the request cannot be fullfilled with the available tools

  96. [104]

    unable to answer

    construct a preference dataset including the correct option as the preferred response and one disprefered response randomly sampled from the incorrect options. We utilize this preference dataset to train the model, minimizing the expected KL divergence between the policy πθ an...

  97. [105]

    Student-athletes to represent SEC at NCAA Convention

  98. [106]

    Division of Corporation Finance

    The SEC has four major divisions. . . Division of Corporation Finance

  99. [107]

    five commissioners

    Securities and Exchange Commission (SEC). . . five commissioners

  100. [108]

    assertion made in sec

    Historical Examples. . . assertion made in sec

  101. [109]

    filing the papers for BH Global Aviation with the sec

  102. [110]

    $100 million under management

    The sec disclosure filed last month. . . $100 million under management

  103. [111]

    sec= hypotenuse adjacent

    Trig Cheat Sheet. . .sec= hypotenuse adjacent

  104. [112]

    SEC Network’s Jimmy Dykes

  105. [113]

    i’m sorry

    The FEC has held. . . organized under sec.. . . Each instance contains one correct passage (the positive) and n related but incorrect passages (negatives). When the positive is placed first, shorter negatives are preferable to longer ones: in RAG, this reduces token usage. Bec...

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.