REVIEW 3 major objections 6 minor 12 references
Estimating Rare Events in Language Models with Proper Evaluation
T0 review · 3 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read Rare language-model failures as unlikely as one in a billion can be counted by searching the model's activation space with a gradient-guided sampler, and which estimator is best depends on the cost of underestimation versus overestimation.
desk verdict GA-AMLS is a promising activation-space rare-event estimator and SPB is a genuinely useful loss, but the pre-image gap keeps the central empirical claim from being fully established. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the score function s(u) = z_t(a(u)) − max_{j≠t} z_j(a(u)) evaluated at whitened pre-LayerNorm pre-unembedding activations u, together with the AMLS identity Pr(s ≥ τ) = ∏_k Pr(s ≥ L_k | s ≥ L_{k−1}). GA-AMLS estimates each factor with a population of N=2000 particles, rejecting proposals below adaptive thresholds and rejuvenating survivors with a MALA kernel whose drift is ∇_u log π(u) for a diagonal Student-t prior π with ν=5 fitted on 65,536 calibration activations. The companion object is the SPB loss, the Bregman divergence generated by weight ω_ε(t) = (t+ε)^{-α}, whose dataset-level weight q^{α−2−γ} cancels the leading q^{2−α} scale factor, making the aggregat
What would settle it
Run GA-AMLS on a modern (7B+) model over natural prompts, compare against exact ground-truth probabilities for tokens in the 1e-6 to 1e-4 range; if log-space squared error degrades relative to QLD as model scale increases, or if most high-score activations cannot be inverted to any discrete prefix, the Student-t/whitening approximation is the load-bearing point of failure.
Extended reading notes
Core claim
The central claim is that activation space is a tractable and preferable domain for rare-event probability estimation in language models. Concretely, GA-AMLS defines the event 'model outputs target token t' as the level set {s(u) ≥ 0} of the logit-margin score s over whitened pre-LayerNorm pre-unembedding activations u, and applies AMLS to estimate Pr(s(u) ≥ 0) as a product of conditional survival probabilities. The MALA kernel, whose drift is computed from a fitted diagonal Student-t prior with ν=5, rejuvenates the particle population within each level while a hard Metropolis filter rejects proposals below the current threshold. Under log-space squared error and SPB with α=1.5, GA-AMLS has
Load-bearing premise
The fitted diagonal Student-t prior in whitened coordinates closely matches the true distribution of model activations, and every high-score activation sampled by the Markov chain corresponds to at least one real input sequence; if either fails, the estimate is biased despite the estimator's formal unbiasedness.
Editorial extensions
If this is right
- GA-AMLS removes zero-estimate collapse: continuous gradient-guided search finds triggering regions even when discrete token search finds none, so the input-space estimators' systematic zero estimates are replaced by finite, lower-bias estimates.
- Under symmetric costs, deployment diagnostics should prefer GA-AMLS: it reduces average log-space squared error relative to QLD, ITGIS, and MHIS across model sizes.
- Under asymmetric costs that punish underestimation heavily (IS loss, SPB α=2), QLD's overestimation bias gives it lower loss; the paper's rank inversion means no single estimator is universally best.
- SPB Loss makes evaluation of zero-estimating methods well-posed without ground-truth-based affine calibration, and its α knob lets practitioners calibrate the FN-vs-FP tradeoff; Theorem 2 sets α=1.5 as the symmetric point.
Reading between the lines
- The Student-t prior is the fragile link; at modern scale, outlier features in activations may have heavier or more structured tails than a diagonal t can represent, so the MALA drift could overshoot into high-score regions with no valid token-level pre-image, silently biasing q_t upward—a hypothesis testable by comparing GA-AMLS against exhaustive sampling on a large model for tokens in the 1e-6–1
- The paper's own two-stage MH rejection makes acceptance sensitive to score smoothness; a natural extension is to replace the isotropic MALA proposal with a preconditioned kernel whose metric approximates the score Hessian, an option the authors tried (MAPLA) but did not adopt due to compute, not accuracy.
- SPB's ε bound is conservative; in practice, choosing ε heuristically as 0.01·q_min/m_max may still cause rank flips among estimators for the very rarest tokens, so public comparisons should report ε and the bound used.
- The adversarial-search literature finds one triggering input; GA-AMLS estimates its probability mass, so these two lines of work can compose: AMLS-level conditional distributions could also be used to generate diverse adversarial examples from the rare-event region, not just count them.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper addresses low-probability estimation for transformer argmax token predictions. It proposes GA-AMLS, which runs adaptive multilevel splitting in the whitened pre-unembedding activation space, using a MALA kernel targeting a diagonal Student-t prior, to estimate the probability that a target token has the largest logit. It also proposes SPB loss, a shifted-power Bregman divergence that is finite at zero estimates, approximately scale-invariant after weighting, and offers tunable asymmetry between underestimation and overestimation. Experiments on the Wu-Hilton benchmark (8 input distributions, 1/2/4-layer d=512 transformers) report that GA-AMLS has the lowest average log-space squared error and SPB (α=1.5) loss, while QLD is better under FN-averse metrics. The paper includes proofs of SPB distortion bounds and asymmetry (Theorems 1–2), synthetic validation of the loss geometry, and detailed ablations.
Significance. The SPB loss is a genuinely useful contribution: it is derived from a proper scoring rule, remains finite at zero estimates, has explicit sufficient conditions for approximate scale invariance and FN/FP asymmetry, and recovers squared error and IS loss as special cases. The synthetic experiments verify the theory. GA-AMLS is a plausible adaptation of AMLS to activation space, and the paper is transparent about the prior-mismatch and pre-image limitations. However, the central empirical claim that GA-AMLS estimates q_t is not yet established: the estimator is unbiased with respect to a fitted prior, and no experiment quantifies the gap between that prior's rare-event region and the pushforward of the input distribution D. The empirical ranking also lacks appropriate uncertainty quantification. If the target gap can be addressed with diagnostics and the statistical claims tightened, this would be a solid contribution to LPE methodology.
major comments (3)
- [§3 'Determine the prior distribution'; §7] The quantity estimated by GA-AMLS is P_{u~π}(s(u) ≥ 0) for a fitted diagonal Student-t prior π, whereas the target is q_t = P_{x~D}(s(f(x)) ≥ 0). AMLS unbiasedness (Section 2.2) holds for the sampling distribution G, i.e. π. The prior is fit on marginal mean/covariance and per-dimension tails, with no constraint that high-score activations lie on the manifold of activations induced by D. Section 7 explicitly concedes the 'activation pre-image issue' but offers no quantitative diagnostic. Appendix D.2 only demonstrates flattening of marginal variances; it says nothing about whether the extreme-score region is supported by real activations. This gap is load-bearing: if off-manifold mass dominates the rare-event region, the reported low log-space squared error does not establish that GA-AMLS estimates q_t. Please add diagnostics (e.g., density-ratio or nearest-neighbor distances between GA-
- [§3 'Determine the prior distribution'; §7; Appendix D.1] The paper contrasts GA-AMLS with QLD's 'independence-based recombination,' but the adopted prior is a product of independent Student-t components in whitened coordinates. Whitening guarantees only uncorrelatedness, not independence, so GA-AMLS also imposes an independent-factor model. The ablation in Appendix D.1 compares only Gaussian vs Student-t marginals and MALA vs RWMH; it does not quantify tail fit or dependence in the rare-event region. If the true activation distribution has tail dependence that a diagonal Student-t cannot capture, the conditional probabilities estimated by AMLS will be biased for q_t even when all sampled activations are realizable. Please measure the prior's tail adequacy (e.g., compare empirical vs prior tail probabilities on held-out activations, or include a nonparametric/copula prior in the tuning sweep).
- [Table 3; Figure 3] The claim that GA-AMLS outperforms all baselines under symmetric evaluation is not supported with appropriate uncertainty quantification. All GA-AMLS estimates come from a single run per (token, distribution, model) (Appendix D.1). Table 3b shows QLD beats GA-AMLS on hex (66.22 vs 272.59) and english (17.51 vs 21.46); Table 3c shows QLD beats GA-AMLS on hex (45.92 vs 133.83) and english (19.07 vs 22.30). Thus the text's 'consistent across 7 of 8 distributions' cannot be reproduced from the tables at the per-model level. Report per-token variance, confidence intervals or paired tests, and separate results by model size before making the aggregate claim.
minor comments (6)
- [Code Availability] The statement 'code is available upon request' should be replaced by a public repository. Reproducibility depends on the exact AMLS/MALA configuration, baseline bug fix, and calibration details.
- [Appendix D.1] The heading 'MAPLA tuning candidate' is confusing; MAPLA is discussed but the main text uses MALA. Ensure notation is consistent throughout, including Algorithm 1's step size adaptation.
- [Figure 16] Specify whether the '7 of 8 distributions' claim is averaged over model sizes or holds per model. The caption should state the aggregation to avoid apparent contradiction with Table 3.
- [Table 1] For each property, list the concrete SPB settings (α, γ, ε) used; otherwise the table reads as schematic. A footnote with default values would help.
- [Algorithm 1] Step 3 fixes ν=5 for the Student-t prior. Add a sensitivity analysis for ν or a citation justifying the fixed value.
- [Figure 3] The log-transformed y-axis is hard to interpret for methods that return zero estimates. Consider displaying the fraction of zero estimates or using a two-part axis.
Circularity Check
No significant circularity: the derivations rely on external AMLS/MCMC/proper-scoring-rule theory, and the acknowledged prior-mismatch limitation is an approximation, not a definitional reduction.
full rationale
Walking the derivation chain, no load-bearing step reduces to its own inputs by construction. GA-AMLS's unbiasedness is inherited from external AMLS and MCMC theory (Guyader et al.; Cérou et al.; Roberts & Tweedie; Srinivasan et al.), and the paper explicitly qualifies this guarantee: “because our MCMC kernel targets a fitted activation prior rather than the true activation distribution, GA-AMLS inherits this guarantee only up to prior mismatch (Section 7).” Section 7 then lists distributional mismatch, finite MCMC mixing, and the activation pre-image issue as acknowledged limitations, not as hidden equivalences. The SPB loss is derived from the Bregman-divergence representation of proper scoring rules attributed to Buja et al. and Gneiting & Raftery, with scale-invariance and asymmetry theorems proven in the appendix from stated definitions; no fitted parameter is renamed as a prediction. Hyperparameters were tuned on the first four distributions and finalized before testing on the last four, so the reported gains are not statistically forced by construction. The whitening diagnostic only supports the preconditioning choice; it is not used to claim that the fitted prior equals the true activation distribution. Overall, the residual concerns about prior fidelity and pre-image support are validity/approximation issues, not circularity.
Assumptions & free parameters
free parameters (4)
- Student-t prior degrees of freedom nu =
5
- MALA step size and adaptation settings =
h0=1e-3, target acceptance 0.57, burn-in 126 steps
- AMLS population size and threshold quantile =
N=2000, rho=0.6
- SPB evaluation parameters (alpha, gamma, epsilon) =
user-selected; epsilon default 0.01*q_min/m_max or Corollary 1 bound
assumptions (6)
- standard math AMLS product decomposition of tail probability and unbiasedness under exact sampling (Cérou et al. 2019)
- standard math Binary proper scoring rules correspond to Bregman divergences with nonnegative weight (Buja et al. 2005)
- ad hoc to paper Whitened activation distribution is adequately modeled by a diagonal Student-t prior
- domain assumption Whitened coordinates are sufficiently isotropic and unimodal for Euclidean MALA proposals
- domain assumption Every high-scoring sampled activation has a plausible input pre-image under D
- domain assumption Ground-truth q_t values from 2^32 forward passes are accurate at the 1e-9 end of the tail
Cite this review
Pith. "Pith review of Estimating Rare Events in Language Models with Proper Evaluation." pith.science (2026). https://pith.science/paper/VEWIA4IV
@misc{pith2026260718454,
author = {Pith},
title = {Pith review of: Estimating Rare Events in Language Models with Proper Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/VEWIA4IV}},
note = {Machine review of arXiv:2607.18454}
}
read the original abstract
Quantifying the risk of rare failures in language models, such as those triggered by adversarial distribution shifts or very large-scale deployments, requires estimating probabilities far too small for random sampling. While recent work has formalized Low Probability Estimation, existing pipelines remain fragile in the rarest regimes: estimators can suffer zero-estimate collapse or systematic bias, and standard evaluation losses can become unstable or poorly matched to asymmetric safety costs. In this work, we introduce Gradient Activation Adaptive Multi-Level Splitting (GA-AMLS), which adapts rare-event Monte Carlo methods to the continuous activation space of language models. Specifically, GA-AMLS uses a gradient-based MCMC kernel to navigate activation space, eliminating the zero-estimate collapse of input-space search and replacing the independence assumptions of prior activation-space estimators with conditional sampling under an explicit, heavier-tailed activation prior. We also propose the Shifted-Power Bregman (SPB) Loss, a proper scoring rule that remains finite for zero-estimates and offers tunable asymmetry between underestimation and overestimation penalties. Experiments on small transformer models reveal a bias-variance tradeoff: GA-AMLS achieves the lowest loss under symmetric evaluation, reducing average log-space squared error relative to the strongest baseline across model sizes, while methods with overestimation bias prevail under asymmetric penalties. Our findings highlight that estimator choice should be matched to deployment context. More broadly, our work establishes activation space as a tractable domain for rare-event estimation in language models, circumventing the brittleness of discrete input-space search.
Figures
Figures from the paper (14 more)
Reference graph
Works this paper leans on
-
[1]
approximate scale invariance within relative toleranceη: 0≤ IFP (α,m)−w(q)B ε(q|mq) IFP (α,m) ≤η, and 0≤ IFN (α,m)−w(q)B ε(q|q/m) IFN (α,m) ≤η
-
[2]
Both guarantees hold uniformly forq∈[q min,q max]andm∈[m 0,m max]
finite-error asymmetric sensitivity: Bε(q|q/m)>B ε(q|mq). Both guarantees hold uniformly forq∈[q min,q max]andm∈[m 0,m max]. Proof.The first term in the minimum in (24) is exactly the sufficient condition of Theorem 1. The second term is exactly the sufficient condition of Theorem 2. Therefore both conclusions hold simultaneously. 22 B.5 Interpretation of...
- [3]
-
[4]
Whileα hi−αlo >τ,setα mid = αlo+αhi 2 .IfR αmid(m0)<r ⋆,setα lo←α mid.Otherwisesetα hi←α mid
-
[5]
This gives a conservative numerical choice satisfyingRαhi(m0)≥r ⋆
Returnα hi. This gives a conservative numerical choice satisfyingRαhi(m0)≥r ⋆. Effect of the shift.For the shifted loss, the exact unshifted ratio no longer holds. However, the proof of Theorem 2 gives the conservative lower bound Bε(q|q/m) Bε(q|mq) ≥(1 +mε/q) −αRα(m). Therefore, uniformly overq∈[q min,q max]andm∈[m 0,m max], Bε(q|q/m) Bε(q|mq) ≥ ( 1 +mma...
arXiv 2025
-
[8]
Choose the minimum relevant multiplicative errorm0 >1, the desired worst-case ratior⋆ >1, and a numerical toleranceτ >0
-
[9]
Setα lo = 3 2.ThenR αlo(m0) = 1<r ⋆
-
[10]
Increase it, for example by repeatedly doubling,αhi← 2αhi,untilR αhi(m0)≥r ⋆
Choose an initial upper bracketαhi >3/2. Increase it, for example by repeatedly doubling,αhi← 2αhi,untilR αhi(m0)≥r ⋆
Show all 12 references
-
[1998]
doi: 10.1111/1467-9868.00123
ISSN 1369-7412, 1467-9868. doi: 10.1111/1467-9868.00123. URLhttps://academic.oup.com/ jrsssb/article/60/1/255/7083121. Gareth O. Roberts and Richard L. Tweedie. Exponential convergence of langevin distributions and their discrete approximations.Bernoulli, 2(4):341–363, 1996. I...
1996
-
[2015]
ArnaudGuyader, NicolasHengartner, andE.Matzner-Løber
URLhttps://arxiv.org/abs/1412.6572. ArnaudGuyader, NicolasHengartner, andE.Matzner-Løber. Simulationandestimationofextremequantiles and extreme probabilities.Applied Mathematics & Optimization, 64:171–196, 10 2011. doi: 10.1007/ s00245-011-9135-z. Dan Hendrycks and Kevin Gimpe...
2011 arXiv
-
[2023]
Andreas Buja, Werner Stuetzle, and Yi Shen
URLhttps://transformer-circuits.pub/2023/monosemantic-features. Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structureandapplications. Workingdraft, UniversityofPennsylvania, TheWhartonSchool, 2005. Step...
2023 arXiv
-
[2024]
Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L
URLhttps://arxiv.org/abs/2402.17762. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Cal- lum McDougall, Monte MacDiarmid, Alex Tamkin, Esin D...
2024 arXiv
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.