Pith. sign in

REVIEW 2 major objections 5 minor 8 references

The standard LLM repetition penalty branches on an arbitrary logit zero-point, so the same setting means different things on different models and wrecks structured JSON.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 15:39 UTC pith:LI6FPVIX

load-bearing objection A real, widely-shipped defect: the CTRL sign-branch is gauge-dependent, and the paper measures both the flip rates and the structured-output collapse cleanly. the 2 major comments →

arxiv 2607.09791 v1 pith:LI6FPVIX submitted 2026-07-09 cs.LG cs.CL

Gauge dependence and structured-output corruption in sign-branched repetition penalties: measurements across models, inference stacks, and alternative repetition controls

Peter Hollows This is my paper
classification cs.LG cs.CL
keywords repetition penaltygauge dependencesoftmax shift-invariancestructured outputlogit normalizationLLM inferenceCTRL penalty
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The multiplicative repetition penalty used across major LLM inference engines decides whether to divide or multiply a seen token's logit by looking at its sign. Because adding any constant to every logit leaves the softmax unchanged, that zero-point is unconstrained by training and differs widely across real models. The paper shows this makes a fixed repetition_penalty setting a different operation on every checkpoint: re-centring the same model by a harmless constant flips 58-96% of greedy tokens at the routine value 1.3, while subtractive and normalized alternatives flip none. The same sign-branch also suppresses grammar-required delimiters in structured output, dropping schema-valid JSON from 97% to 23% on 200 real-world schemas. Applying the penalty to normalized log-probabilities instead of raw logits removes both effects; the operator already ships in one major stack but is off by default and runs after the penalty.

Core claim

The multiplicative CTRL-style repetition penalty is not well-defined under the gauge freedom of the softmax: its sign-branch reads an arbitrary logit zero-point that training does not constrain, so identical settings produce different operations on different models and different gauges of the same model, and at routine strength the operator systematically corrupts grammar-required tokens in structured output.

What carries the argument

The sign-branched map of Eq. (1) that divides positive raw logits by theta and multiplies negative ones, whose kink sits at the unconstrained zero-point; the paper's gauge-flip probe (identical distributions under c = +5 versus c = -5) and the closed-form delimiter-flip condition make the consequences measurable.

Load-bearing premise

That the large flip rates and JSON-validity drops measured under greedy decoding on models up to 7B remain representative of how the same operator behaves under the sampling regimes and larger models actually used in production.

What would settle it

Re-run the c = +5 versus c = -5 gauge probe and the 200-schema JSON validity test at theta = 1.3 on a production-scale model with nucleus or temperature sampling; if flip rates collapse near zero and validity stays near the unpenalized baseline, the claimed consequences do not hold under realistic decoding.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper shows that the multiplicative CTRL-style repetition penalty used across major LLM inference stacks (HuggingFace, vLLM, llama.cpp, and others) is gauge-dependent: it branches on the sign of raw logits (Eq. 1), but softmax is invariant under uniform logit shifts, so the zero-point is unconstrained by training. Two consequences are measured. (A1) Re-centring logits by a constant that is a no-op at θ=1 flips 58–96% of greedy tokens at θ=1.3 across five models ≤7B (WikiText-103 prefixes), while subtractive and normalized controls flip none; real checkpoints also sit at widely different zero-points (Table 1). (A2) On 200 JSONSchemaBench schemas the same θ=1.3 drops schema-valid output from 97% to 23%; a closed-form flip condition (Eq. 2) predicts delimiter flips with balanced accuracy 0.999 on HumanEval. Both effects replicate inside vLLM and llama.cpp. Applying the penalty to log-softmax instead removes both effects (Table 4); HuggingFace already ships the needed operator but applies it after the penalty by default.

Significance. If the results hold, the paper documents a load-bearing defect in a default operator shipped across essentially the entire open inference ecosystem, with direct consequences for structured generation. Strengths that raise confidence include: (i) the mechanism follows from the elementary identity softmax(z+c)=softmax(z) with no fitted parameters; (ii) the gauge-flip experiment is cleanly controlled (identical distributions at θ=1, zero flips for subtractive and normalized penalties, 40k positions); (iii) Eq. (2) is a falsifiable closed-form prediction validated at high accuracy; (iv) both effects are replicated through three independent samplers on the same inputs; (v) a repository with pre-registered decision rules and raw outputs is provided. The normalized fix is constructive and already present (mis-ordered) in HuggingFace. These are the kinds of measurements the community can act on immediately.

major comments (2)
  1. Section 6 and Tables 2–4: all quantitative rates (gauge flip 58–96%, JSON validity 97%→23%) are measured under greedy decoding on models ≤7B. The mechanism itself (sign-branch on an unconstrained coordinate) is sampling- and scale-independent, so the qualitative claim that the operator is ill-defined stands. However, the practical claim that a fixed repetition_penalty is a different operation on every model in production use rests on rates that may change under temperature sampling. Either a short temperature-sampling check on one model, or an explicit statement that the reported percentages are greedy-specific and that only the qualitative gauge dependence is claimed for production regimes, is needed so readers do not over-read the numbers.
  2. Section 3, Eq. (2) and the HumanEval audit: the closed-form flip condition and the 0.999 balanced accuracy apply only to positions where the top token is on the divide branch and the runner-up is unpenalized (48,919 of the generated positions). The end-to-end JSON drop (97%→23%) aggregates all positions. The manuscript should state what fraction of grammar-obligatory delimiter positions fall outside that regime (both tokens penalized, or top on the multiply branch) and whether those residual positions contribute materially to the validity collapse, so the link from Eq. (2) to Table 4 is fully transparent.
minor comments (5)
  1. Figure 1 caption and surrounding text: the phrase “adding c slides the same token across the kink” is clear once the figure is studied, but a one-sentence reminder that c is added to every logit (not only the seen token) would prevent a common misreading.
  2. Table 1 note on gpt2’s bimodal logits is important; consider moving a short version into the main text of Section 2 so readers who skip the table note still understand why a large negative median coexists with a non-zero divide fraction.
  3. Section 5: the statement that a given θ is “gentler” on log-probabilities is correct but qualitative. A single sentence giving the typical scale of raw logits vs. log-probs (or a recommended re-tuning range) would help practitioners who want to adopt the normalized form.
  4. Section 4 / Table 3: the survey of twelve further engines is classified “by form, not probed.” That is fine, but a footnote listing the exact source files or commit hashes used for the classification would make the survey reproducible without digging through the repository.
  5. References [7] and [8] document prior awareness of the kink and the original negative-logit bug. A single sentence in the conclusion noting that neither prior report measured gauge dependence or structured-output corruption would more cleanly locate the contribution.

Circularity Check

0 steps flagged

No significant circularity; claims rest on elementary softmax invariance plus direct measurements on external benchmarks with gauge-invariant controls.

full rationale

The derivation chain begins from the elementary, externally known identity softmax(z+c)=softmax(z) (Section 1) and the shipped CTRL operator (Eq. 1) that branches on the unconstrained sign of raw logits. Both consequences are obtained by direct measurement: A1 flip rates (Table 2) on WikiText-103 prefixes under controlled re-centring, and A2 validity collapse (Table 4) on JSONSchemaBench, each compared against subtractive and normalized controls that remain identically zero under the same gauge shifts. The closed-form delimiter-flip condition (Eq. 2) is an algebraic rearrangement of the operator definition and is verified post-hoc (balanced accuracy 0.999); it is not a fitted parameter re-labeled as a prediction. The normalized variant follows immediately by applying the same multiplicative map to log-softmax, which is shift-invariant by construction. No parameters are fitted to recover the reported numbers, no uniqueness theorem or ansatz is imported via self-citation, and no known empirical pattern is merely renamed. The paper is therefore self-contained against external benchmarks and exhibits no circular reduction.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The central claims rest almost entirely on the standard mathematical property of softmax and on direct empirical measurement. No free parameters are fitted to produce the headline numbers; the only modeling choices are the conventional values θ=1.3 and α=1 used for illustration. No new physical or mathematical entities are postulated.

free parameters (2)
  • θ (repetition_penalty) = 1.3 (illustrative)
    Routine illustrative value 1.3 used throughout; not fitted to data. Existing practice already treats it as a free knob.
  • α (subtractive presence penalty) = 1
    Mid-range control value α=1; again illustrative, not fitted.
axioms (2)
  • standard math softmax(z + c) = softmax(z) for any scalar c (gauge freedom of logits)
    Invoked in Section 1 and used to define the controlled flip experiment; elementary property of the softmax.
  • domain assumption The multiplicative CTRL operator of Eq. (1) is the form shipped by the surveyed engines
    Established by source survey (Section 4) and by direct re-implementation inside HF, vLLM and llama.cpp.

pith-pipeline@v1.1.0-grok45 · 12772 in / 2159 out tokens · 19832 ms · 2026-07-14T15:39:01.858343+00:00 · methodology

0 comments
read the original abstract

The multiplicative repetition penalty shipped across the LLM inference ecosystem (HuggingFace, vLLM, llama$.$cpp, and a dozen further engines) branches on the sign of each raw logit (divide positives by theta, multiply negatives). But the softmax is unchanged by adding a constant to every logit, so a model's logit zero-point is arbitrary, and the sign-branch reads that arbitrary point. The sign-branch is itself the accepted fix for an earlier bug, so the accepted fix branches on a quantity the training objective leaves unconstrained. Two measurable consequences follow. (1) The penalty is not well-defined: re-centring a model's logits by a constant is a provable no-op at theta=1, yet at a routine theta=1.3 it changes 58-96% of greedy tokens, where subtractive and normalized penalties change none; real checkpoints sit at widely different zero-points, so a fixed repetition_penalty is a different operation on every model. (2) It corrupts structured output: on 200 real-world JSON schemas, theta=1.3 drops the rate of valid, schema-conformant output from 97% to 23%. In our measurements, applying the penalty to normalized log-probabilities instead of raw logits removes both effects. HuggingFace already ships that operator (LogitNormalization); today it is off by default and applied after the penalty. This note gives the mechanism, the measurements (five models up to 7B, base and RLHF, on WikiText-103 prefixes; two code models on HumanEval and JSONSchemaBench; both effects replicated inside vLLM and llama$.$cpp through their own samplers on the same inputs), and the normalized variant.

Figures

Figures reproduced from arXiv: 2607.09791 by Peter Hollows.

Figure 1
Figure 1. Figure 1: The penalty maps a seen token’s logit z to z ′ : positive logits are divided by θ (shallow slope), negative logits multiplied (steep slope), meeting at a kink at z = 0. Open marks show each token’s unpenalized value on the identity line; the arrows show the penalty’s decrease, smaller on the divide branch and larger on the multiply branch. Because the softmax does not change when a constant is added to eve… view at source ↗
Figure 2
Figure 2. Figure 2: The cost of each intervention in model log-likelihood: traces scored token-by-token under [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

8 extracted references · 5 linked inside Pith

  1. [1]

    Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374, 2021. HumanEval

    Pith/arXiv arXiv 2021

  2. [2]

    Jsonschemabench: A rigorous benchmark of structured outputs for language models.arXiv preprint arXiv:2501.10868, 2025.https:// github.com/guidance-ai/jsonschemabench

    Saibo Geng, Hudson Cooper, Michał Moskal, Samuel Jenkins, Julian Berman, Nathan Ranchin, Robert West, Eric Horvitz, and Harsha Nori. Jsonschemabench: A rigorous benchmark of structured outputs for language models.arXiv preprint arXiv:2501.10868, 2025.https:// github.com/guidance-ai/jsonschemabench

    Pith/arXiv arXiv 2025

  3. [3]

    Ginart, Naveen Kodali, Jason Lee, Caiming Xiong, Silvio Savarese, and John R

    Antonio A. Ginart, Naveen Kodali, Jason Lee, Caiming Xiong, Silvio Savarese, and John R. Emmons. LZ penalty: An information-theoretic repetition penalty for autoregressive language models.arXiv preprint arXiv:2504.20131, 2025. 7

    Pith/arXiv arXiv 2025

  4. [4]

    Varshney, Caiming Xiong, and Richard Socher

    Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A conditional transformer language model for controllable generation.arXiv preprint arXiv:1909.05858, 2019

    Pith/arXiv arXiv 1909

  5. [5]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. InInternational Conference on Learning Representations (ICLR), 2017. WikiText-103. arXiv:1609.07843

    Pith/arXiv arXiv 2017

  6. [6]

    A con- trastive framework for neural text generation

    Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A con- trastive framework for neural text generation. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  7. [7]

    Suggested fixes for mathematical inaccuracy inllama_sample_repetition_- penaltyfunction, 2023

    tysam-code. Suggested fixes for mathematical inaccuracy inllama_sample_repetition_- penaltyfunction, 2023. llama.cpp issue #2970, September 2023; closed as stale.https: //github.com/ggml-org/llama.cpp/issues/2970

    2023

  8. [8]

    Repetition penalty work falsely in case the logit of the token is negativ

    Patrick von Platen. Repetition penalty work falsely in case the logit of the token is negativ. https://github.com/huggingface/transformers/issues/2302, 2019. HuggingFace Trans- formers issue #2302 (December 2019), reporting that dividing a negative logit by a penalty >1raises its probability; fixed by the author in PR #2303 with the sign-branched divide/m...

    2019