Pith.

Machine Learning · arXiv:2611.16803 v2 · submitted 12 Nov 2026

Detection Is Not Early Warning: Evaluating Token-Level Alarms on Language Model Failures

Marco De Negri, Luca Sartori, Lorenzo Baggi, Moritz Reihs, Eduard Frank Ďurech, Mennatallah El-Assady, Valentina Pyatkin, Anna Hedström

show 4 more authors

Anna Hedström (2, 3) ((1) ETH Zurich, (2) ETH AI Center, (3) Solaris Research)

The paper says

“Many failures of language models begin at a specific token, the onset.”

more of the abstract

A monitor that reads the model during generation can classify the finished response, detect the failure once it is under way, or warn before it begins, and only a warning leaves time to stop or steer the generation. Monitors are usually evaluated with response-level metrics such as AUROC, which ignore when the monitor fires. We introduce PREWARN, an evaluation protocol for any failure with an annotated onset. It calibrates each monitor's threshold on validation data to an error rate chosen for the application, runs the monitor, and scores its alarms relative to the onset on all three tasks. As a case study we apply it to repetitive degeneration in three open language models, with early warning as the main task, using linear probes on activations and six untrained baselines. Response-level metrics saturate on this failure, and a rule that reads only the response length scores almost perfectly, while probes that AUROC ranks as equal warn very differently under PREWARN. Probes warn before the onset on most degenerate responses of two of the three models, low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset, and no untrained baseline warns much more often than random alarms. We release the probes with calibrated thresholds at https://huggingface.co/collections/marcodene/prewarn-degeneration-probes

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions.

4 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 4.4, limitation b187

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.92):

“Fourth, on APERTUSand LLAMAthe onset usually comes after a healthy response has ended, so part of what the probes flag may reflect how long the response has run (Section 4.4).”

Pith says: Objection 1 of 4 · major

The paper acknowledges (b187) that on Apertus and Llama 'the onset usually comes after a healthy response has ended, so part of what the probes flag may reflect how long the response has run.' This is a load-bearing confound for the central claim that probes warn before the onset on most loops.

more of objection 1

Because every degenerate response runs to the token cap (4096 tokens) while healthy responses often terminate earlier, a probe that simply flags tokens based on position or length would appear to warn before the onset on degenerate responses, without actually detecting any signal of the failure. The authors acknowledge this limitation but do not control for it. Without a control—such as evaluating probes on healthy responses truncated to the same length as degenerate ones, or conditioning on response length—the headline claim that probes 'warn before the onset on most loops' does not demonstrate early knowledge of the failure. This confound is load-bearing because if the probes are largely reading length/position, then the early-warning interpretation collapses, and the paper's main empirical contribution would be an artifact of the data distribution rather than evidence of pre-onset signal detection. I consider this a major issue that needs to be addressed, either by providing such a control or by substantially tempering the claim.

This objection may describe the copy Pith gave the reviewer, not the paper.

The paper says, at Section 4.2, Figure 2 and Table 4

The review did not quote this passage.

Pith says: Objection 2 of 4 · major

The paper argues that response-level metrics are misleading because they rate scorers as equal that differ in early-warning performance. However, the comparison in Figure 2 and Table 4 uses the token-cap rule as a baseline, which is a straw man: it flags only the last token of a capped response, so it cannot warn.

more of objection 2

The more relevant comparison is between the frozen and adapted probes, which do differ in AUROC (0.9994 vs 0.9991 on Apertus, Table 1). While these differences are small, they are not zero, and the paper's claim that 'response-level metrics saturate' is supported by the fact that all probes have AUROC > 0.99. However, the claim that 'AUROC ranks the probes consistently, but not by how much they warn' (b111) is based on a Kendall's tau of 0.33 between validation AUROC and test warning coverage. This is a weak correlation, but the paper does not report whether the correlation is statistically significant or provide confidence intervals. Given that the number of probes is 27, a tau of 0.33 may be non-zero but still weak. The paper should report a p-value or confidence interval for this correlation to support the claim that AUROC is a poor predictor of early-warning performance.

The paper says, at Section 4.3, Figure 3 and Table 1

The review did not quote this passage.

Pith says: Objection 3 of 4 · major

The paper reports that on Mistral, probes fire mostly after the onset and work better as detectors. This is a clear and honest result. However, the paper also claims that 'probes warn before the onset on most degenerate responses of two of the three models' (b7).

more of objection 3

On Apertus, the early-warning rate is 0.66-0.72 (Table 1), meaning about 30% of alarms come after the onset. On Llama, the early-warning rate is 0.58-0.66. So on Apertus and Llama, the probes warn before the onset on most loops, but a substantial minority (28-42%) of alarms are still after the onset. The paper should be more precise about what 'most' means and discuss the implications for deployment: a monitor that warns before the onset on only 60-70% of failures may not be reliable enough for high-stakes applications.

The paper says, at Section 4.4, Figure 4 and Table 1

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.85):

“Fourth, on APERTUSand LLAMAthe onset usually comes after a healthy response has ended, so part of what the probes flag may reflect how long the response has run (Section 4.4).”

Pith says: Objection 4 of 4 · major

The paper finds that low-rank adaptation more than doubles warning coverage but barely raises the early-warning rate. The interpretation is that adaptation adds flags long before the onset, which increases coverage but does not change the first alarm. However, the paper also notes that on Llama, the added flags may partly track response length (b147).

more of objection 4

This is consistent with the length confound mentioned above. The paper should explicitly test whether the added flags on Llama are correlated with response length, and if so, whether the early-warning signal is actually present or is an artifact of length.

8 minor objections

The paper says, at Section 3, Step 1

Pith says: minor objection 1 of 8

Equation (1) defines the threshold as the minimum value such that the false-alarm rate on validation is at most beta. However, the paper does not specify how ties are broken or what happens if no threshold achieves exactly the budget.

more of objection 1

The text says 'the lowest value at which the monitor's error rate on the validation split stays within the budget,' which is clear, but it would be helpful to note that the threshold is chosen from the set of observed scores, and that the false-alarm rate is computed as the share of healthy responses with any alarm.

The paper says, at Section 4.1, Baselines

Pith says: minor objection 2 of 8

The baselines are tuned on validation using the same selection rule as the probes (highest Warn256 at beta=1% among those with post-onset coverage >= 0.3). However, for windowed entropy and negative log-probability on Llama, no setting reaches the floor, so the one with the highest Warn256 is selected.

more of objection 2

This is a reasonable approach, but the paper should report which settings were selected for each baseline and model, as is done in Tables 6-9. The main text could briefly summarize the selected window sizes (e.g., '1,024 to 4,096 tokens' for repetition rules, '256 to 512 tokens' for confidence baselines).

show 6 more

The paper says, at Section 4.5, Table 1

Pith says: minor objection 3 of 8

Table 1 reports that the token-cap rule has AUROC 0.9990 on Apertus, 0.9988 on Llama, and 0.9995 on Mistral. This is indeed near-perfect, but the paper should note that this is because the token-cap rule flags only the last token of a capped response, and since all degenerate responses are capped and most healthy responses are terminated, the rule perfectly separates the two classes except for the few healthy responses that are also capped.

more of objection 3

This is a trivial baseline that cannot warn, and its high AUROC is a feature of the data distribution, not of the rule's intelligence. The paper makes this point, but it could be stated more explicitly.

The paper says, at Section 5, Conclusion

Pith says: minor objection 4 of 8

The conclusion states that 'probes on Apertus and Llama fire before the onset on most of the loops they flag.' This is accurate, but it would be helpful to also state the early-warning rates explicitly (66-72% on Apertus, 58-66% on Llama) to give readers a quantitative sense of 'most.'

The paper says, at Section 6, Limitations

Pith says: minor objection 5 of 8

The paper lists five limitations, which is commendable. However, the fourth limitation (b187) is the most critical, as discussed in the major comments. The paper should consider adding a control experiment or at least a more detailed discussion of how the length confound could affect the results.

The paper says, at Appendix A.2, Human verification

Pith says: minor objection 6 of 8

The paper reports that on Apertus, 13 of 139 onsets are off by 256 tokens or more, and on 10 of these the judge places the onset 380-1250 tokens after the reviewer does. This is a significant discrepancy that could affect the early-warning metrics. The paper should discuss whether these cases are concentrated in particular sources or response types, and whether the human-verified labels are considered the ground truth for all analyses.

The paper says, at Appendix A.5, Table 4

Pith says: minor objection 7 of 8

Table 4 shows that shuffling scores within a response produces a scorer with the same AUROC but very different early-warning performance. This is a useful sanity check, but the paper should note that shuffling is not a realistic monitor because it uses future information. The point is to illustrate that AUROC cannot distinguish between a monitor that fires early and one that fires late, which is well-taken.

The paper says, at General

Pith says: minor objection 8 of 8

The paper is well-written and the figures are clear. The use of multiple seeds, human verification, and held-out sources strengthens the empirical contribution. The code and model releases are valuable.

What would settle it

Evaluate the frozen and adapted probes on healthy responses that are artificially truncated to the same length distribution as degenerate responses (e.g., by cropping each healthy response to the length at which the onset occurs in degenerate responses). If the probes' warning coverage and early-warning rate on these length-matched healthy responses are comparable to those on degenerate responses, then the probes are reading length/position rather than a signal of the coming loop. Alternatively, train a probe that uses only the token index (position) as input and compare its performance to the activation-based probes; if the position-only probe achieves similar warning coverage, the confound is confirmed.

from the skeptic's reading

How the review went

3 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic agrees 8 s · 37,801 tokens read, 698 written
  3. Referee major revision: 4 major objections, 8 minor 16 s · 38,039 tokens read, 2,360 written

Not run on this paper: the editor, the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The strongest objection

The paper's central claim that linear probes warn before the onset on most degenerate responses of Apertus and Llama rests on the assumption that pre-onset flags reflect a genuine signal of the coming loop rather than response length or position. The authors explicitly state this limitation (b187): 'on APERTUSand LLAMAthe onset usually comes after a healthy response has ended, so part of what the probes flag may reflect how long the response has run (Section 4.4).' Since every degenerate response runs to the token cap (4096 tokens) while healthy responses often terminate earlier, a probe that simply flags tokens based on position or length would appear to warn before the onset on degenerate responses, without actually detecting any signal of the failure. The paper acknowledges this confound but does not control for it. Without a control—such as evaluating probes on healthy responses truncated to the same length as degenerate ones, or conditioning on response length—the headline claim that probes 'warn before the onset on most loops' does not demonstrate early knowledge of the failure. The confound is load-bearing because if the probes are largely reading length/position, then the early-warning interpretation collapses, and the paper's main empirical contribution (that probes warn before the onset) would be an artifact of the data distribution rather than evidence of pre-onset signal detection.

The reader's weakest_assumption correctly identifies the length/position confound as the load-bearing concern. The paper acknowledges this limitation (b187) but does not resolve it. The headline claim that probes warn before the onset on most loops is the central empirical result, and if the confound is not controlled for, the interpretation that probes detect a signal of the coming failure is unsupported. The PREWARN protocol itself is solid, and the critique of AUROC is well-supported, but the early-warning interpretation for Apertus and Llama requires a control for length/position. The concrete test I propose would settle whether the confound actually explains the results. I agree with the reader that the verdict should remain CONDITIONAL until this confound is addressed. I do not recommend changing the verdict because the paper is honest about the limitation, and the protocol and negative results (Mistral, baselines) are still valuable. However, the headline claim should be softened or the confound controlled before the paper can be taken as establishing pre-onset signal detection.

The referee's report

The paper makes a clear methodological contribution by formalizing the distinction between detection and early warning for token-level monitors, and by providing a protocol (PREWARN) that is principled, practical, and applicable to any failure with an onset. The case study on repetitive degeneration is thorough, with careful human verification of labels, calibration on validation, held-out sources, and multiple seeds. The finding that response-level metrics are misleading for early-warning evaluation is important and likely to influence future work on monitoring LLMs during generation. The release of calibrated probes is a concrete asset for the community. If the results hold, the paper demonstrates that linear probes can provide early warning on some models, which has practical implications for safe deployment.

Pith's reading of the paper (a machine's summary, not the authors')

Language model failures such as harmful content or repetition loops start at a specific token, the onset. A monitor that reads the model during generation can flag the finished response, detect the failure once it has begun, or warn before it begins, and only the last leaves time to stop the generation. Most monitors are scored with response-level metrics like AUROC, which reward flagging the failure but not flagging it early, so a probe that fires only after the loop is visible scores as well as one that fires before it.

This paper proposes PREWARN, a protocol that calibrates each monitor's threshold on validation data to a fixed false-alarm budget, runs the monitor, and scores its alarms relative to the annotated onset on three tasks: classification, detection, and early warning. As a case study they use repetitive degeneration in Apertus, Llama, and Mistral, with labels from a judge and then human review, and compare linear probes on activations against six untrained baselines and a random-alarm reference.

The headline findings are that AUROC ranks the scorers as near-equal while the onset-aligned metrics show the adapted probes warn before most loops on Apertus and Llama but after the onset on Mistral, and that no untrained baseline warns much more than random alarms. The authors acknowledge a confound: the onset usually comes after a healthy response has ended, so the probes may partly be reading response length, and most of the adapted probe's warning-coverage gain comes from flags long before the onset, not from a sharpening signal.

Core claim. The central claim is that response-level metrics such as AUROC are misleading for token-level monitors of failures with an onset: 'Every unit after the onset counts as a positive, these metrics reward flagging the failure, not flagging it early,' so a probe that fires after the loop is visible scores as well as one that fires before it. Under PREWARN, with thresholds calibrated to a fixed false-alarm budget and alarms scored against the onset, probes that AUROC ranks as equal differ several-fold in warning coverage, and linear probes warn before the onset on most degenerate responses of Apertus and Llama but not Mistral.

The premise it rests on. The load-bearing premise for the early-warning interpretation is that the pre-onset flags reflect a genuine signal of the coming loop rather than response length, since the authors state that 'on APERTUSand LLAMAthe onset usually comes after a healthy response has ended, so part of what the probes flag may reflect how long the response has run' (Section 4.4; limitation b187). If the probes are largely reading position or length, which is confounded with degeneration because every degenerate response runs to the token cap, the headline 'warns before the onset on most loops' would not demonstrate early knowledge of the failure. This confound is acknowledged but not controlled for.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Many failures of language models begin at a specific token, the onset. A monitor that reads the model during generation can classify the finished response, detect the failure once it is under way, or warn before it begins, and only a warning leaves time to stop or steer the generation. Monitors are usually evaluated with response-level metrics such as AUROC, which ignore when the monitor fires. We introduce PREWARN, an evaluation protocol for any failure with an annotated onset. It calibrates each monitor's threshold on validation data to an error rate chosen for the application, runs the monitor, and scores its alarms relative to the onset on all three tasks. As a case study we apply it to repetitive degeneration in three open language models, with early warning as the main task, using linear probes on activations and six untrained baselines. Response-level metrics saturate on this failure, and a rule that reads only the response length scores almost perfectly, while probes that AUROC ranks as equal warn very differently under PREWARN. Probes warn before the onset on most degenerate responses of two of the three models, low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset, and no untrained baseline warns much more often than random alarms. We release the probes with calibrated thresholds at https://huggingface.co/collections/marcodene/prewarn-degeneration-probes

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 0.48

“Probes warn before the onset on most degenerate responses of two of the three models, low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset, and no untrained baseline warns much more often than random alarms.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.09 · states limitation 0.38 · human participants 0.03 · controlled comparison 0.92 · randomized assignment 0.02 · states quantitative result 0.72 · compares with prior results 0.95 · mentions code or data availability 0.96

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Many failures of language models begin at a specific token, the onset. background 1.00 0.15 0.03 asserted 0.98
A monitor that reads the model during generation can classify the finished response, detect the failure once it is under way, or warn before it begins, and only a warning leaves time to stop or steer the generation. background 0.90 0.07 0.22 asserted 1.00
Monitors are usually evaluated with response-level metrics such as AUROC, which ignore when the monitor fires. gap 0.71 0.09 0.05 asserted 1.00
We introduce PREWARN, an evaluation protocol for any failure with an annotated onset. contribution 0.99 0.92 0.06 asserted 0.99
It calibrates each monitor's threshold on validation data to an error rate chosen for the application, runs the monitor, and scores its alarms relative to the onset on all three tasks. method 0.99 0.89 0.06 asserted 0.98
As a case study we apply it to repetitive degeneration in three open language models, with early warning as the main task, using linear probes on activations and six untrained baselines. method 0.74 0.95 0.05 asserted 0.92
Response-level metrics saturate on this failure, and a rule that reads only the response length scores almost perfectly, while probes that AUROC ranks as equal warn very differently under PREWARN. result 0.99 0.92 0.38 asserted 1.00
Probes warn before the onset on most degenerate responses of two of the three models, low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset, and no untrained baseline warns much more often than random alarms. result 1.00 0.95 0.58 asserted 1.00
We release the probes with calibrated thresholds at https://huggingface.co/collections/marcodene/prewarn-degeneration-probes contribution 0.96 0.91 0.04 asserted 0.94

What kind of paper the classifier reports this is: method or tool 0.99 · evidence from observation or experiment 0.97

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 7,013 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (6)
Figure 1 of the paper
Figure 1. Overview of PREWARN. Left: a monitor runs a scorer over the generated tokens and alarms where the score crosses τ (dots mark flagged tokens, the flag the first alarm); AUROC rates three such monitors as equal although their first alarms fall well before, just before and after the onset o(r). Right: what PREWARN needs and its three steps: calibrate the threshold on validation to the budget β, run the monitor on the test split, and score the alarms against the onset on the three tasks. To score early warning, an evaluation needs two things: an annotated onset, so that alarms before it…
Figure 2 of the paper
Figure 2. Response-level metrics rate four scorers as equal, while only the onset-aligned metrics
Figure 3 of the paper
Figure 3. Probes warn before most loops on APERTUS and LLAMA but not on MISTRAL. Frozen (solid) and adapted (hatched) probe of each model at its selected layer (
Figure 4 of the paper
Figure 4. On APERTUS and LLAMA, probes flag more tokens as the onset approaches, and adaptation adds flags long before it; on MISTRAL, flags rise only after the onset. Top: share of flagged tokens on degenerate responses by distance from the onset (bins of 32 tokens; positions more than 1,024 tokens before the onset pooled on the left). Bottom: the same on healthy responses by position (bins of 128 tokens), where flags also rise with length; the strip below gives the number of responses still running, on a log scale. Panel titles give the layers of the three probes. β = 5%, human-verified…
Figure 5 of the paper
Figure 5. Signed distance between the judge’s onset and the reviewer’s onset on the capped test
Figure 6 of the paper
Figure 6. In the frozen model, the tokens before the warning window, in it and after the on

Figures are the authors', reproduced from the paper.

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment