Pith. sign in

REVIEW 4 major objections 5 minor 67 references

A single logit-bias vector, learned from black-box rollouts, lets a frozen language model improve at math and reasoning without any weight update.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A fixed logit-bias vector estimated from black-box rollouts gives small accuracy gains and modest length compression, far below LoRA fine-tuning.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A clean IPS estimator and an honest empirical study of a single fixed logit-bias vector, but the reward-gap theorem is conditional on an unmeasured factorization assumption that likely makes the bound vacuous at the scales they run. the 4 major comments →

arxiv 2607.22837 v1 pith:DI4MY7UR submitted 2026-07-24 cs.LG cs.AIcs.CL

Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias

Ofek I. Cohen , Lior Shani , Aviv Rosenberg , Ankur Samanta , Tal Wagner , Yonathan Efroni This is my paper
classification cs.LG cs.AIcs.CL
keywords logit biasblack-box adaptationKL-regularized reinforcement learninginverse propensity scoringdecoding-time controlmath reasoning benchmarksreward-tilted policy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that one context-independent logit-bias vector — a fixed set of per-token offsets added to a frozen model's logits at every decoding step — is a useful minimal adaptation primitive under black-box constraints. Working from a KL-regularized reinforcement-learning objective, the authors show the ideal per-step correction is the soft value log Z(x, y_{1:t-1}, ·), and that a closed-form inverse-propensity estimator recovers a fixed approximation δ from rollouts, rewards, and token probabilities. The theory says that when the ideal correction approximately factorizes into a fixed token effect plus a prefix-dependent baseline, the fixed-bias policy sits within a bounded reward gap of the optimal tilted policy and provably beats the base model when reward variance is large enough. Empirically the learned vector shifts formatting, digit, and stop patterns, raising exact-match accuracy on MATH, GSM8K, and GPQA over the base model with about 128k trained parameters and no gradient access, while a compression variant shortens completions along an accuracy–length frontier. If correct, the result gives API-only deployers a lightweight, auditable steering mechanism that sits between prompt engineering and fine-tuning.

Core claim

Central claim: a single fixed logit-bias vector can stand in for the optimal prefix-dependent policy correction. From a KL-regularized objective whose optimum is the reward-tilted distribution p⋆_τ ∝ p0 e^{r/τ}, the per-step optimal next-token law is softmax(ℓ + log Z(x, y_{1:t-1}, ·)); if this correction approximately factorizes as a fixed token effect plus a prefix-dependent baseline (ε-approximate factorization), the fixed-bias policy stays within a reward gap R ε √(T/8) of the optimum and beats the base model whenever reward variance dominates. The vector is estimated by an unbiased inverse-propensity estimator from on-policy rollouts, and empirically lifts exact-match accuracy on MATH,

What carries the argument

Key machinery: the factorization condition (Definition 3) — the ideal correction log Z(x, y_{1:t-1}, y) must equal δ(y) + c_t(x, y_{1:t-1}) up to small error ε over all prompts, steps, and reachable prefixes — under which a fixed vector reproduces the optimal tilted policy. Theorem 4 converts the residual ε into a cost R ε √(T/8) subtracted from a gain growing with reward variance. The estimator is an unbiased inverse-propensity score bZ(y) = (1/NP) Σ 1{y_t = y} e^{r/τ}/π0(y | x, y_{<t}); the base probability in the denominator cancels the sampling propensity, leaving an interventional do(yt = y) expectation, and bZ = v^{1/τ} M makes τ-sweeps one matrix-vector product. Additive smoothing α g

Load-bearing premise

The whole guarantee rests on ε-approximate factorization: the ideal per-prefix logit correction must be nearly a fixed token vector plus a prefix-dependent constant everywhere — if the correction genuinely depends on which problem is being solved, the fixed vector cannot represent it, and the paper never measures ε for the vector it actually deploys.

What would settle it

On a benchmark where correct answers carry problem-specific surface tokens (proper names, theorem names, values) and no shared formatting conventions, estimate the soft-value corrections log Z(x, y_{1:t-1}, ·) by rollouts, fit the best fixed vector δ, and measure the mean residual ε; if the residual exceeds R√(T/8) relative to the variance term, the Theorem 4 bound predicts no improvement — and empirically the learned bias should stop beating the base model.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Deployers with only inference-API access gain a per-task adaptation knob: one vocabulary-sized vector (~128k floats) that improves exact-match accuracy on MATH, GSM8K, and GPQA over the base model with no weight updates or gradients.
  • The learned bias is a reusable surface-and-termination pattern — promoted stop tokens, math delimiters, digits, and answer headers; suppressed drift and page-furniture tokens — so the method is a lightweight primitive between prompt engineering and fine-tuning, not a replacement for it.
  • Under a compression reward, the vector traces an accuracy–completion-length frontier, so a single learned δ can trade verbosity against accuracy during serving without retraining.
  • The reward-gap bound gives a formal condition — base reward variance large enough to dominate R ε √(T/8) — under which fixed-bias steering provably improves on the base model.
  • Because the estimator factorizes into rollout statistics and reward/temperature weights, adapting to a new reward or τ is a single matrix-vector product over cached statistics.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper never measures ε for the vector it deploys, so the theory is conditional: a direct test would compute pointwise soft-value corrections on a real task, project onto the best fixed vector, and report the residual; one would expect the residual to be large on tasks where decisive tokens are problem-specific (names, values, theorem labels), which predicts the method's gains there would vanis
  • The qualitative token analysis suggests the edge is concentrated in shared surface conventions: a testable extension is a benchmark stripped of uniform formatting (no delimiters, no stop strings), where the factorization residual should grow and the accuracy gain should shrink toward zero.
  • The promoted 'artifact' tokens (three-digit numbers, page-furniture strings) point to a failure mode the paper acknowledges in its limitations: with finite rollouts, rare-token IPS noise can masquerade as steering signal; a cheap validation-side check is to mask promoted coordinates with negligible base probability and re-measure accuracy.
  • A natural generalization gradient from this result: context-dependent biases (a few vectors selected by prefix pattern) would interpolate between this primitive and fine-tuning, and the factorization condition suggests measuring per-prefix residuals to decide when one vector stops being enough.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes adapting a frozen language model by learning a single context-independent logit-bias vector δ that is added to the model's logits at every decoding step, using only black-box rollouts, rewards, and token probabilities. The authors formalize the problem as KL-regularized RL, show that the optimal per-prefix correction is log Z(x, y_{1:t-1}, ·), and introduce ε-approximate factorization (Definition 3) under which a fixed vector can approximate this correction. They derive a closed-form inverse-propensity estimator for the position-averaged soft value Z_avg (Eq. 6, Proposition 6) and prove a reward-gap bound (Theorem 4). Empirically, they evaluate on MATH, GSM8K, and GPQA for an accuracy reward, and on MATH for a compression reward, reporting small gains over the base model and much weaker performance than LoRA-GRPO.

Significance. If the central claim were established, the paper would demonstrate a genuinely lightweight black-box adaptation primitive: a fixed vocabulary-level logit bias learned from rollouts, with no weight access or gradients. The clean pieces deserve credit: the closed-form IPS estimator and its unbiasedness (Proposition 6), the matrix-factorization view (Proposition 5), and the exact-factorization corollary are useful and cleanly presented. However, the main theoretical guarantee (Theorem 4) is conditional on an ε-approximate factorization that is never verified, and its error term scales unfavorably — apparently to the point of vacuity at the experimental scale. Moreover, the experimental rewards do not satisfy the theorem's assumptions. The empirical gains are small and, for most benchmarks, not statistically distinguishable from the base model. The significance of the paper therefore rests on a theoretical bridge that is not actually crossed, and on evidence that is weaker than the abstract's 'consistent gains' claim.

major comments (4)
  1. [§6.1 vs. Theorem 4 / Eq. (7)] Theorem 4 assumes r ∈ [0, R], but the main accuracy experiments use r_log-accuracy ∈ {0, −∞} (Section 6.1) and the compression experiments use r_length = ln(T/length) (Eq. 8), which ranges up to ln(1024) ≈ 6.9 and is not normalized despite the claim in Section 5.2 that 'the length normalization used in the experiments' is motivated by the bound. Thus Theorem 4 cannot be applied to Tables 1–2. With the log-indicator reward, log Z(x, y_{1:t-1}, y) = −∞ for any token that cannot lead to a correct completion, so Definition 3's L2 norm is undefined or infinite for a finite bδ. In addition, p*_τ assigns zero mass to incorrect trajectories, so the per-step conditionals of p*_τ are not defined on prefixes that pbδ may reach; the KL chain rule in the proof of Theorem 4 relies on these conditionals.
  2. [Theorem 4 / Definition 3, experimental scale] Even under bounded rewards, the bound in Eq. (7) appears vacuous at the paper's own scale (T ≈ 1024, |V| ≈ 128k). Definition 3 measures ε as the L2 norm over the full vocabulary. If per-coordinate deviations are of magnitude d, then ε ≈ √|V|·d ≈ 358d, making the error term R·ε·√(T/8) ≈ 4050·R·d. For binary accuracy, the improvement term is at most V0/R·(1−e^{−R/τ}) ≤ 1/4, so the bound is positive only if d ≲ 6×10^{-5}. The paper never measures ε, never checks Definition 3 for the deployed bδ, and provides no argument that such a small per-coordinate error holds. Consequently, the theorem supplies no quantitative support for the claim that the fixed-bias policy approximates the KL-regularized optimum.
  3. [§4.1 / Definition 3] There is a mismatch between what is estimated and what the theory requires. The estimator targets Z_avg(y), an average of the soft value over positions, prompts, and prefixes (Eq. 5), and bδ is obtained by smoothing and centering log Ẑ. But Definition 3 requires that the per-prefix log-correction log Z(x, y_{1:t-1}, ·) be well approximated by a fixed vector plus a prefix-dependent constant at every reachable prefix. Averaging over positions can conceal exactly the prefix dependence that determines whether a fixed vector is adequate. Proposition 6 establishes unbiasedness of Ẑ for Z_avg, but no step connects the smoothed estimate of Z_avg to the ε-approximate factorization condition for the deployed δ. This is a load-bearing gap: the paper's central theoretical claim is that the learned δ approximates the optimal prefix-dependent correction, and this is never verified.
  4. [Table 1 / Table 2] The abstract's 'consistent gains' claim is not supported by the confidence intervals. In Table 1, GSM8K (33.89 ± 2.50 vs. 34.80 ± 2.50) overlaps substantially, and GPQA (22.20 ± 13.33 vs. 28.90 ± 13.33, n=45) overlaps massively. Only MATH shows a non-overlapping interval, and that by a narrow margin (30.30 ± 1.26 vs. 33.14 ± 1.30). In Table 2, Qwen3-4B shows no compression effect (1018.12 ± 1.57 vs. 1017.61 ± 1.59), and Gemma-9B's reduction is marginal relative to its interval. The empirical case for the method is therefore much weaker than advertised, and the current text overstates the evidence.
minor comments (5)
  1. [§6.1 / Eq. (8)] The notation r_length(y) uses 'length(y)' where the argument is a trajectory y_{1:T}, not a single token; rename for clarity.
  2. [Table 3] 'Sampled fraction' is not defined in the main text; clarify that it is the fraction of vocabulary coordinates visited in the rollouts, and note that the selected-coordinate fraction depends on the arbitrary gate |bδ(a)| ≥ 5·median|bδ|.
  3. [Appendix D.7] The reproducibility caveat about batch-chunking and fp16 logit flips affecting greedy decoding should be moved or at least summarized in the main experimental section, since it bears on the reliability of the reported exact-match numbers.
  4. [§4.2] The claim that α is chosen by validation is reasonable, but the paper should state that the same validation set is used for both α selection and checkpoint selection in the baselines, and report the sensitivity of results to α.
  5. [Appendices B/C] The qualitative token-family analysis is extensive but largely interpretive. It would be more convincing if the 'intervention score' distributions were compared against a null distribution from random bias vectors of the same norm, to control for the sheer number of tokens analyzed.

Circularity Check

0 steps flagged

No circularity: the logit-bias estimator targets a well-defined quantity and the theory is conditional on an explicit, unverified approximation assumption.

full rationale

The derivation chain is self-contained and does not reduce to its inputs by construction. Equation (4) is derived from the KL-regularized optimum in Equation (2) by marginalization, giving the per-step correction log Z(x, y_{1:t-1}, ·). The population target Z_avg in Equation (5) is a well-defined expectation over positions, prompts, and base-model interventions, and the IPS estimator in Equation (6) is shown unbiased for it in Proposition 6 (Appendix E). The conversion bδ(y) = log(α + bZ(y)) − c(α) is a smoothed, centered transform of that estimator, not a fitted value of the evaluation metric. The reward-gap guarantee in Theorem 4 is explicitly conditional: it applies to any bδ satisfying ε-approximate factorization (Definition 3), and the bound degrades with ε; the paper never claims to measure ε or to verify Definition 3 for the deployed vector. That is an unverified assumption and a potential vacuity of the guarantee, but it is not circularity. The empirical claims are evaluated on held-out test sets (MATH, GSM8K, GPQA) and the hyperparameters α and τ are selected on validation, which is standard model selection rather than a fitted-input-called-prediction construction. There is no load-bearing self-citation chain, no uniqueness theorem imported from the authors' own prior work, and no ansatz smuggled in via citation: the tilted distribution and Gibbs variational step are standard results cited to external literature. The use of r_log-accuracy = 0/−∞ and the unmeasured ε are correctness/robustness concerns, not definitional equivalences. Accordingly, no circular step is identified.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

Central claim rests on softmax full support (standard), the RLVR terminal-reward setup, an unverified ε-approximate factorization assumption, and IPS unbiasedness. Learned quantities are the logit-bias vector δ and per-dataset hyperparameters α and τ tuned on validation.

free parameters (3)
  • logit-bias vector δ = Learned per task (vocab-size vector, 128k–256k parameters)
    Central learned object; estimated via Eq. 6 and log(α + Ẑ) with centering.
  • additive smoothing α = MATH 0.052; GSM8K 0.110; GPQA 0.007; compression 0.738/0.052/0.465
    Tuned on validation; controls shrinkage of rare-token IPS estimates.
  • tilting temperature τ = Inert for accuracy; compression: 1.1 (Gemma), 0.7 (Llama), 0.8 (Qwen)
    Tuned on validation in the compression experiment; drops out under the log-indicator accuracy reward.
axioms (5)
  • standard math Softmax full support: π0(y|x, y<t) > 0 for all reachable prefixes
    Required for IPS division in Eq. 6; holds automatically for softmax models.
  • ad hoc to paper ε-approximate factorization of log Z(x, y1:t-1, ·) as δ + c_t(x, y1:t-1)1
    Definition 3; the theoretical bridge used in Theorem 4. Never verified for the learned δ.
  • domain assumption Terminal reward and fixed horizon T with EOS as absorbing state
    Simplifies the trajectory-level theory; variable-length generation is deferred.
  • standard math IPS termwise unbiasedness via the do(yt=y) intervention
    Proposition 6: prefix and suffix drawn from p0 cancel the propensity factor; relies on base-policy rollouts.
  • standard math Autoregressive KL chain rule and Gibbs variational optimum
    Used to derive Equation 4 and Corollary 2 from Ziegler et al. and related RLHF literature.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias." pith.science (2026). https://pith.science/paper/DI4MY7UR

@misc{pith2026260722837,
  author       = {Pith},
  title        = {Pith review of: Frustratingly Simple Black-Box Adaptation of Language Models via Logit Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DI4MY7UR}},
  note         = {Machine review of arXiv:2607.22837}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Many organizations aim to adapt language models for internal use, both to improve performance on domain-specific tasks and to address privacy concerns around sensitive data. However, such adaptation remains non-trivial: it often requires operationally challenging fine-tuning of open-source models or ad hoc prompt optimization. We study a minimal alternative based on a simple API-level control: allowing users to bias the model's logits with a user-defined vector. We develop a black-box method for learning a single context-independent logit-bias vector, added at every decoding step, without modifying model weights or requiring gradients. Starting from a KL-regularized reinforcement learning (RL) objective, we characterize when such a fixed logit-bias vector can approximate the optimal prefix-dependent correction and derive a closed-form inverse-propensity estimator from rollouts, rewards, and token probabilities. Empirically, this simple decoding-time intervention improves over base models on mathematical and reasoning benchmarks while using far fewer trainable parameters than conventional fine-tuning. Our results suggest that learned logit bias is a lightweight mechanism for adapting language models under minimal access requirements.

Figures

Figures reproduced from arXiv: 2607.22837 by Ankur Samanta, Aviv Rosenberg, Lior Shani, Ofek I. Cohen, Tal Wagner, Yonathan Efroni.

Figure 1
Figure 1. Figure 1: Overview of logit-bias steering. The fixed logit-bias vector [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Validation accuracy–completion-length Pareto frontier across [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 1 canonical work pages

  1. [1]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models

    Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022

    2022

  2. [2]

    Guiding llms the right way: Fast, non-invasive constrained generation.arXiv preprint arXiv:2403.06988, 2024

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding llms the right way: Fast, non-invasive constrained generation.arXiv preprint arXiv:2403.06988, 2024

    Pith/arXiv arXiv 2024

  3. [3]

    Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A

    Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick,...

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.14782 2024

  4. [4]

    Chen and Joshua Goodman

    Stanley F. Chen and Joshua Goodman. An empirical study of smoothing techniques for language modeling.Computer Speech & Language, 13(4):359–394, 1999. doi: 10.1006/csla.1999.0128

    arXiv 1999

  5. [5]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168,

    Pith/arXiv arXiv

  6. [6]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas.Elements of Information Theory. Wiley, 2nd edition, 2006

    2006

  7. [7]

    Plug and play language models: A simple approach to controlled text generation

    Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. InInternational Conference on Learning Representations, 2020

    2020

  8. [8]

    RLPrompt: Optimizing discrete text prompts with rein- forcement learning

    Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing discrete text prompts with rein- forcement learning. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3369–3391, Abu Dhabi, United Arab Emirates, December 2022. Association...

    doi:10.18653/v1/2022.emnlp-main.222 2022

  9. [9]

    Eleutherai/hendrycks_math

    EleutherAI. Eleutherai/hendrycks_math. https://huggingface.co/datasets/EleutherAI/ hendrycks_math, 2026. Accessed: 2026-05-04

    2026

  10. [10]

    Eleutherai/lm-evaluation-harness

    EleutherAI. Eleutherai/lm-evaluation-harness. https://github.com/EleutherAI/ lm-evaluation-harness, 2026. Accessed: 2026-05-07. 12

    2026

  11. [11]

    Fireworks ai chat completions api (logit_bias).https://docs.fireworks.ai/ api-reference/post-chatcompletions, 2026

    Fireworks AI. Fireworks ai chat completions api (logit_bias).https://docs.fireworks.ai/ api-reference/post-chatcompletions, 2026. Accessed: 2026-05-07

    2026

  12. [12]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, et al. Gemma 2: Improving open language models ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.00118 2024

  13. [13]

    Grammar-constrained decoding for structured NLP tasks without finetuning

    Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar-constrained decoding for structured NLP tasks without finetuning. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  14. [14]

    The Llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 1, 2021. URLhttps://arxiv.org/abs/2103.03874

    Pith/arXiv arXiv 2021

  16. [16]

    Logits are all we need to adapt closed models

    Gaurush Hiranandani, Haolun Wu, Subhojyoti Mukherjee, and Sanmi Koyejo. Logits are all we need to adapt closed models. InProceedings of the 42nd International Conference on Machine Learning, volume 267 ofProceedings of Machine Learning Research, pages 23261–23289. PMLR, 13–19 Jul 2025. URLhttps://proceedings.mlr.press/v267/hiranandani25a.html

    2025

  17. [17]

    Horvitz and Donovan J

    Daniel G. Horvitz and Donovan J. Thompson. A generalization of sampling without replacement from a finite universe.Journal of the American Statistical Association, 47(260):663–685, 1952. doi: 10.2307/2280784

    doi:10.2307/2280784 1952

  18. [18]

    Parameter-efficient transfer learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Maziar Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. InProceedings of the 36th International Conference on Machine Learning, 2019

    2019

  19. [19]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  20. [20]

    Idavidrein/gpqa.https://huggingface.co/datasets/Idavidrein/gpqa, 2026

    Hugging Face. Idavidrein/gpqa.https://huggingface.co/datasets/Idavidrein/gpqa, 2026. Accessed: 2026-05-04

    2026

  21. [21]

    Automata-based constraints for language model decoding

    Terry Koo, Frederick Liu, and Luheng He. Automata-based constraints for language model decoding. COLM 2024, 2024. URLhttps://openreview.net/forum?id=BDBdblmyzY. Camera- ready version; arXiv:2407.08103

    Pith/arXiv arXiv 2024

  22. [22]

    Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. RL with KL penalties is better viewed as Bayesian inference.Findings of EMNLP, 2022

    2022

  23. [23]

    GeDi: Generative discriminator guided sequence generation

    Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. InFindings of the Association for Computational Linguistics: EMNLP, 2021. 13

    2021

  24. [24]

    The power of scale for parameter-efficient prompt tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

    2021

  25. [25]

    Prefix-tuning: Optimizing continuous prompts for generation

    Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021

    2021

  26. [26]

    Note on the general case of the bayes-laplace formula for inductive or a posteriori probabilities.Transactions of the Faculty of Actuaries, 8(1):182–192, 1920

    George James Lidstone. Note on the general case of the bayes-laplace formula for inductive or a posteriori probabilities.Transactions of the Faculty of Actuaries, 8(1):182–192, 1920. doi: 10.1017/S0071368600003490

    work page doi:10.1017/s0071368600003490 1920

  27. [27]

    Smith, and Yejin Choi

    Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021

    2021

  28. [28]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning

    Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. InAdvances in Neural Information Processing Systems, volume 35, 2022. URL https://openreview.net/forum?id=rBCvMG-JsPd

    2022

  29. [29]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  30. [30]

    llama.cpp server documentation (logit_bias).https://github.com/ ggml-org/llama.cpp/blob/master/tools/server/README.md, 2026

    llama.cpp Contributors. llama.cpp server documentation (logit_bias).https://github.com/ ggml-org/llama.cpp/blob/master/tools/server/README.md, 2026. Accessed: 2026-05-07

    2026

  31. [31]

    LM Studio openai-compatible chat completions documentation (logit_bias)

    LM Studio Team. LM Studio openai-compatible chat completions documentation (logit_bias). https://lmstudio.ai/docs/developer/openai-compat/chat-completions, 2026. Accessed: 2026-05-07

    2026

  32. [32]

    Neurologic decoding: (un)supervised neural text generation with predicate logic constraints

    Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Neurologic decoding: (un)supervised neural text generation with predicate logic constraints. In Proceedings of NAACL-HLT, 2021

    2021

  33. [33]

    Azure openai chat completions reference (logit_bias)

    Microsoft Azure. Azure openai chat completions reference (logit_bias). https://learn. microsoft.com/azure/ai-services/openai/reference, 2025. Accessed: 2026-05-03

    2025

  34. [34]

    Prompt engineering techniques for azure openai.https://learn.microsoft

    Microsoft Azure. Prompt engineering techniques for azure openai.https://learn.microsoft. com/en-us/azure/foundry/openai/concepts/prompt-engineering, 2025. Accessed: 2026- 05-03

    2025

  35. [35]

    Kluwer Academic Publishers, 2004

    Yurii Nesterov.Introductory Lectures on Convex Optimization: A Basic Course. Kluwer Academic Publishers, 2004

    2004

  36. [36]

    Openai api pricing.https://openai.com/api/pricing, 2025

    OpenAI. Openai api pricing.https://openai.com/api/pricing, 2025. Accessed: 2026-05-03

    2025

  37. [37]

    Openai api reference (logit_bias)

    OpenAI. Openai api reference (logit_bias). https://platform.openai.com/docs/ api-reference/chat/create, 2025. Accessed: 2026-05-03. 14

    2025

  38. [38]

    Prompt engineering

    OpenAI. Prompt engineering. https://platform.openai.com/docs/guides/ prompt-engineering, 2025. Accessed: 2026-05-03

    2025

  39. [39]

    openai/gsm8k

    OpenAI and Hugging Face. openai/gsm8k. https://huggingface.co/datasets/openai/ gsm8k, 2026. Accessed: 2026-05-04

    2026

  40. [40]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback....

    2022

  41. [41]

    Cambridge University Press, 2nd edition, 2009

    Judea Pearl.Causality: Models, Reasoning, and Inference. Cambridge University Press, 2nd edition, 2009

    2009

  42. [42]

    The FineWeb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben Allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The FineWeb datasets: Decanting the web for the finest text data at scale. InAdvances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2024. URLhttps://arxiv.org/abs/2406.17557

    Pith/arXiv arXiv 2024

  43. [43]

    Pinsker.Information and Information Stability of Random Variables and Processes

    Mark S. Pinsker.Information and Information Stability of Random Variables and Processes. Holden-Day, 1964

    1964

  44. [44]

    Manning, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2023

    2023

  45. [45]

    Sentence-bert: Sentence embeddings using siamese bert- networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019

    2019

  46. [46]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level google-proof q&a benchmark.arXiv preprint arXiv:2311.12022, 2023. doi: 10.48550/arXiv.2311.12022. URL https://arxiv.org/abs/2311.12022

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.12022 2023

  47. [47]

    Comparing retrieval-augmentation and parameter-efficient fine-tuning for privacy-preserving personalization of large language models.arXiv preprint arXiv:2409.09510, 2024

    Alireza Salemi and Hamed Zamani. Comparing retrieval-augmentation and parameter-efficient fine-tuning for privacy-preserving personalization of large language models.arXiv preprint arXiv:2409.09510, 2024. doi: 10.48550/arXiv.2409.09510. URLhttps://arxiv.org/abs/2409. 09510

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.09510 2024

  48. [48]

    LaMP: When large language models meet personalization

    Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7370–7392, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024. ac...

    doi:10.18653/v1/2024 2024

  49. [49]

    sentence-transformers/all-mpnet-base-v2.https://huggingface.co/ sentence-transformers/all-mpnet-base-v2, 2026

    Sentence Transformers. sentence-transformers/all-mpnet-base-v2.https://huggingface.co/ sentence-transformers/all-mpnet-base-v2, 2026. Accessed: 2026-05-07

    2026

  50. [50]

    Sglang openai-compatible api documentation (logit_bias)

    SGLang Team. Sglang openai-compatible api documentation (logit_bias). https://docs. sglang.ai/backend/openai_api_completions.html, 2025. Accessed: 2026-05-03. 15

    2025

  51. [51]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024. doi: 10.48550/arXiv.2402.03300. URLhttps://arxiv.org/abs/2402.03300

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300 2024

  52. [52]

    MPNet: Masked and permuted pre-training for language understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MPNet: Masked and permuted pre-training for language understanding. InAdvances in Neural Information Processing Systems, 2020

    2020

  53. [53]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. Advances in Neural Information Processing Systems, 33, 2020

    2020

  54. [54]

    BBTv2: Towards a gradient-free future with large language models

    Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. BBTv2: Towards a gradient-free future with large language models. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3916–3930, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 1...

    2022

  55. [55]

    Black-box tuning for language-model-as-a-service

    Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service. InProceedings of the 39th International Conference on Machine Learning, volume 162 ofProceedings of Machine Learning Research, pages 20841–20855. PMLR, 17–23 Jul 2022. URLhttps://proceedings.mlr.press/v162/sun22e.html

    2022

  56. [56]

    Tsybakov.Introduction to Nonparametric Estimation

    Alexandre B. Tsybakov.Introduction to Nonparametric Estimation. Springer, 2009

    2009

  57. [57]

    vLLM sampling parameters documentation (logit_bias).https://docs.vllm

    vLLM Team. vLLM sampling parameters documentation (logit_bias).https://docs.vllm. ai/en/stable/api/vllm/sampling_params/, 2026. Accessed: 2026-05-07

    2026

  58. [58]

    Willard and Rémi Louf

    Brandon T. Willard and Rémi Louf. Efficient guided generation for large language models. arXiv preprint arXiv:2307.09702, 2023

    Pith/arXiv arXiv 2023

  59. [59]

    Transformers: State-of- the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of- the-art n...

    2020

  60. [60]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. doi: 10.48550/arXiv.2505.09388. URLhttps: //arxiv.org/ab...

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.09388 2025

  61. [61]

    FUDGE: Controlled text generation with future discriminators

    Kevin Yang and Dan Klein. FUDGE: Controlled text generation with future discriminators. In Proceedings of NAACL-HLT, 2021

    2021

  62. [62]

    Weinberger, Kianté Brantley, and Wen Sun.Q♯: Provably Optimal Distributional RL for LLM 16 Post-Training

    Jin Peng Zhou, Kaiwen Wang, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kilian Q. Weinberger, Kianté Brantley, and Wen Sun.Q♯: Provably Optimal Distributional RL for LLM 16 Post-Training. InAdvances in Neural Information Processing Systems, 2025. doi: 10.48550/ arXiv.2502.20548. URLhttps://arxiv.org/abs/2502.20548. NeurIPS 2025

    doi:10.48550/arxiv.2502.20548 2025

  63. [63]

    Prob- lem Keywords

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. Appendix Contents A Intervention score, token selection, and grouping 18 A.1 Intervention score as realized log-probability shift . . . . ....

    Pith/arXiv arXiv 1909

  64. [66]

    With L = 1 2 this gives Dψ(z′, z) ≤ 1 4 ∥z′ −z∥ 2

  65. [67]

    Letp= softmax(u+z) =∇ψ(z)

    It remains to identifyD ψ with the KL divergence. Letp= softmax(u+z) =∇ψ(z). Then Dψ(z′, z) =ψ(z′)−ψ(z)− ⟨∇ψ(z), z′ −z⟩ =ψ(z ′)−ψ(z)− X a pa(z′ a −z a) = X a pa logp a −(u a +z ′ a −ψ(z ′)) = X a pa log pa softmax(u+z ′)a = KL softmax(u+z)∥softmax(u+z ′) . G.2 KL lower bound via exponential tilting Lemma 9.Fix a prompt x. Let r : X ×VT → [0, R], σ2 0 := V...

  66. [2020]

    doi: 10.18653/v1/2020.emnlp-demos.6

    Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6/

    doi:10.18653/v1/2020.emnlp-demos.6 2020

  67. [2021]

    URLhttps://arxiv.org/abs/2110.14168

    doi: 10.48550/arXiv.2110.14168. URLhttps://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv doi:10.48550/arxiv.2110.14168

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.