{"paper":{"title":"Detection Is Not Early Warning: Evaluating Token-Level Alarms on Language Model Failures","arxiv_id":"2611.16803","primary_cat":"cs.LG","submitted_at":"2026-11-12T13:43:19+00:00","version_compiled":"v1"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v1","sentences":[{"i":0,"p":1.0,"id":"s0","own":false,"role":"background","text":"Many failures of language models begin at a specific token, the onset.","hedge":{"p":0.98,"label":"asserted"},"own_p":0.15,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.05,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":1.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":0.9,"id":"s1","own":false,"role":"background","text":"A monitor that reads the model during generation can classify the finished response, detect the failure once it is under way, or warn before it begins, and only a warning leaves time to stop or steer the generation.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.07,"main_claim_p":0.22,"quantitative":true,"quantitative_p":0.59,"role_probabilities":{"gap":0.09,"method":0.0,"result":0.0,"background":0.9,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.01}},{"i":2,"p":0.71,"id":"s2","own":false,"role":"gap","text":"Monitors are usually evaluated with response-level metrics such as AUROC, which ignore when the monitor fires.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.09,"main_claim_p":0.05,"quantitative":false,"quantitative_p":0.12,"role_probabilities":{"gap":0.71,"method":0.0,"result":0.0,"background":0.29,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":3,"p":0.99,"id":"s3","own":true,"role":"contribution","text":"We introduce PREWARN, an evaluation protocol for any failure with an annotated onset.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.92,"main_claim_p":0.06,"quantitative":false,"quantitative_p":0.04,"role_probabilities":{"gap":0.0,"method":0.01,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.99,"interpretation":0.0}},{"i":4,"p":0.99,"id":"s4","own":true,"role":"method","text":"It calibrates each monitor's threshold on validation data to an error rate chosen for the application, runs the monitor, and scores its alarms relative to the onset on all three tasks.","hedge":{"p":0.98,"label":"asserted"},"own_p":0.89,"main_claim_p":0.06,"quantitative":false,"quantitative_p":0.1,"role_probabilities":{"gap":0.0,"method":0.99,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.0}},{"i":5,"p":0.74,"id":"s5","own":true,"role":"method","text":"As a case study we apply it to repetitive degeneration in three open language models, with early warning as the main task, using linear probes on activations and six untrained baselines.","hedge":{"p":0.92,"label":"asserted"},"own_p":0.95,"main_claim_p":0.05,"quantitative":true,"quantitative_p":0.71,"role_probabilities":{"gap":0.0,"method":0.74,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.26,"interpretation":0.0}},{"i":6,"p":0.99,"id":"s6","own":true,"role":"result","text":"Response-level metrics saturate on this failure, and a rule that reads only the response length scores almost perfectly, while probes that AUROC ranks as equal warn very differently under PREWARN.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.92,"main_claim_p":0.38,"quantitative":true,"quantitative_p":0.91,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.99,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.01}},{"i":7,"p":1.0,"id":"s7","own":true,"role":"result","text":"Probes warn before the onset on most degenerate responses of two of the three models, low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset, and no untrained baseline warns much more often than random alarms.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.95,"main_claim_p":0.58,"quantitative":true,"quantitative_p":0.98,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":8,"p":0.96,"id":"s8","own":true,"role":"contribution","text":"We release the probes with calibrated thresholds at https://huggingface.co/collections/marcodene/prewarn-degeneration-probes","hedge":{"p":0.94,"label":"asserted"},"own_p":0.91,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.06,"role_probabilities":{"gap":0.0,"method":0.01,"result":0.01,"background":0.0,"limitation":0.0,"future_work":0.02,"contribution":0.96,"interpretation":0.0}}]},"compiled":{"main_claim":{"p":0.48,"text":"Probes warn before the onset on most degenerate responses of two of the three models, low-rank adapters below the probe more than double the share of tokens just before the onset that it flags, largely by flagging degenerate responses long before the onset, and no untrained baseline warns much more often than random alarms.","location":"abstract s7","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":7013,"cost_usd_reported":0.0},"compiled_at":"2026-11-13","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.09,"states_limitation":0.38,"human_participants":0.03,"controlled_comparison":0.92,"randomized_assignment":0.02,"states_quantitative_result":0.72,"compares_with_prior_results":0.95,"mentions_code_or_data_availability":0.96},"evidential_basis":{"p":0.97,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":0.97,"simulation_or_computation":0.03}},"contribution_type":{"p":0.99,"value":"method or tool","probabilities":{"review":0.0,"method_or_tool":0.99,"empirical_study":0.01,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":0.0,"position_or_commentary":0.0}},"contribution_statement":{"p":1.0,"text":"We introduce PREWARN, an evaluation protocol for any failure with an annotated onset.","location":"abstract s3"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v1"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":9,"sentences_in_abstract":9},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}