{"id":"c62daec3-c478-4208-9154-314633cca413","arxiv_id":"2608.10621","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ProbGuard uses early output distributions and Monte Carlo sampling to estimate calibrated unsafe-generation probability, beating 13 baselines on Brier/ECE and limiting jailbreak success to at most 1%.","lead":"ProbGuard predicts the probability that a chatbot's in-progress answer will become unsafe, using only the model's early word-choice probabilities. It reports large calibration gains and near-zero jailbreak success, but the risk labels come from a judge model the authors trained.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Calibration claims rest entirely on CalibEval-generated targets; CalibEval is validated only as a binary classifier, so the reported Brier/ECE reductions may measure agreement with a biased judge rather than true safety.","rationale":"The reader identified CalibEval as the weakest assumption, and I agree. The stress test sharpens the point: this is not merely a missing ablation; the training and evaluation targets are generated by the same model, so Brier/ECE measure internal consistency with CalibEval. Table 5 validates CalibEval only as a binary classifier, not as a calibrated probability over continuations. Eq. (9) averages binary CalibEval labels, and Eqs. (15)-(16) compare predictions to that average. If CalibEval's errors are systematic, a model can appear excellently calibrated while being wrong about true safety. The attack experiments use GPT-5 as judge, which partially mitigates the circularity for the ASR claim, but the headline calibration numbers stand or fall with CalibEval. The proposed human-label test would settle the concern. No code or data is released, so independent re-evaluation is not currently possible. I therefore maintain the reader's CONDITIONAL verdict: accept only if the human-label calibration check passes; otherwise the central claim is unsupported.","tokens_in":14889,"tokens_out":7591,"duration_ms":71006,"concrete_test":"Take a random subset of 200 evaluation prefixes per dataset (600 total). For each prefix, regenerate the same 16 continuations with the paper's sampling procedure and have human annotators label each continuation safe/unsafe. Compute per-prefix human-based unsafe proportions eC^human and recompute Brier/ECE for ProbGuard and the strongest baselines against eC^human. If ProbGuard's Brier/ECE degrade by more than 0.05 in absolute terms, or its relative improvement over baselines shrinks materially, the reported calibration gains are not robust to judge bias. As a secondary check, build a reliability diagram for CalibEval's individual-continuation scores against the human labels; large calibration error there would confirm the targets are biased.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that CalibEval provides a valid ground-truth safety probability. Eq. (9) defines the calibration target eC_{m,k} as the average of CalibEval labels over 16 sampled continuations, and Eqs. (15)-(16) evaluate Brier/ECE against exactly this eC. CalibEval is validated in Table 5 only as a binary classifier (F1=0.943, TPR=0.944, FPR=0.058 on PKU) using dataset-level annotations; there is no reliability diagram, no calibration check, and no validation on the continuation distribution. If CalibEval is systematically biased on certain continuation styles (e.g., refusals with unsafe follow-ups, verbose evasive replies), the targets in both training and evaluation inherit that bias. ProbGuard can then achieve low Brier/ECE by predicting the biased judge's labels rather than true safety, so the headline 79.6% Brier and 71.9% ECE reductions are not established as reductions relative to actual unsafe-continuation probability. The attack ASRs in Table 2 are judged by GPT-5 and are less directly affected, but the calibration claim, the core of the paper, remains unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ProbGuard, a guardrail that estimates the probability that an LLM's ongoing generation will become unsafe, using only the next-token output distributions of the protected model. The method formulates safety risk as an expectation over possible continuations (Eqs. (1)-(4)), approximates it with Monte Carlo sampling (Eqs. (5)-(6)), labels sampled continuations with a trained judge called CalibEval (Eq. (9)), encodes the prefix distribution as probability-weighted embeddings in the guardrail's own token space (Eqs. (10)-(13)), and trains a small LLM with a negative log-likelihood loss (Eq. (14)). Experiments compare Brier score and ECE against 13 baselines across three target LLMs and three datasets, and evaluate early-intervention attack success rates against six jailbreak attacks. The authors report average Brier and ECE reductions of 79.6% and 71.9% over the best baseline, and attack success rates at or below 1% for ProbGuard-8B.","tokens_in":15173,"tokens_out":5846,"duration_ms":63520,"significance":"If the calibration results are taken at face value, ProbGuard is a practically interesting contribution: it makes early safety-risk estimates from publicly observable output probabilities, avoids dependence on internal hidden states, and the experimental scope is broad (three LLM families, three datasets, six attacks, thirteen baselines). The formalization in Eqs. (1)-(6) is clean, and the probability-weighted representation in Eqs. (11)-(13) is a sensible way to handle tokenizer mismatch. However, the main empirical claim is currently measured against an internal target derived from the authors' own judge, CalibEval, which is validated only as a binary classifier. The claimed calibration gains are therefore not yet established with respect to true safety outcomes, which is the central load-bearing issue for the paper.","major_comments":[{"comment":"Equations (9), (15), and (16) define both the training target and the evaluation target as the same CalibEval-based Monte Carlo average eC_{m,k}. ProbGuard is trained to predict eC_{m,k}, and Brier/ECE are then computed against eC_i. This makes the evaluation circular with respect to the judge: a model that perfectly imitates CalibEval's labels will achieve near-zero Brier and ECE, regardless of whether those labels reflect actual unsafe-continuation probability. The manuscript needs an independent evaluation target, for example human labels or a separately validated judge on a sample of continuations, or at minimum a reliability analysis of CalibEval itself on the continuation distribution. Without this, the headline 79.6% Brier and 71.9% ECE reductions are reductions relative to an internal proxy, not demonstrated reductions in calibration error against real safety outcomes.","section":"Calibration Target Construction; Metrics"},{"comment":"Table 5 validates CalibEval only as a binary classifier on dataset-level annotations, reporting accuracy, F1, TPR, and FPR on the PKU evaluation set. There is no reliability diagram, no calibration curve, and no evaluation of whether the judge's labels are well-calibrated on the distribution of sampled continuations used in Eq. (9). Even a high-F1 binary judge can be systematically biased on certain continuation styles (for example, refusals followed by unsafe elaboration), and that bias would be inherited by every training and evaluation target. The paper should validate CalibEval as a probabilistic judge on continuation samples, or provide evidence that its binary errors are unbiased across the distribution of eC values.","section":"Evaluation of CalibEval (Table 5)"},{"comment":"The Brier score in Eq. (15) and ECE in Eq. (16) are not standard Brier/ECE metrics, because the 'ground truth' is the continuous Monte Carlo estimate eC_i rather than a binary outcome. Standard Brier score and ECE compare a probabilistic forecast to realized binary outcomes. Using eC_i as the target measures agreement between two estimates of the same quantity, not calibration of a probabilistic forecast. This is related to the circularity concern but is a distinct methodological point: the reported numbers are not directly comparable to Brier/ECE numbers in prior calibration literature unless the target is a binary label. The authors should either report standard Brier/ECE against binary outcomes or explicitly justify why the soft target defines a proper scoring rule.","section":"Metrics, Eqs. (15)-(16)"}],"minor_comments":[{"comment":"The figure contains stray non-English text '一列四行' in the lower right, which appears to be an editing artifact and should be removed.","section":"Figure 2"},{"comment":"The table note says 'bold indicates the default sampling budget', but the default N=16 row is marked with an asterisk rather than bold; the formatting should be made consistent.","section":"Table 3"},{"comment":"The dataset name is spelled both 'SEval' and 'S-Eval' in different places; please standardize the spelling.","section":"Experimental Settings (Datasets)"},{"comment":"The introduction cites a BGP security paper (Zhao et al. 2025a) as a motivation for LLM safety concerns; this citation seems tangential and may confuse readers.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the engineering is substantial, but the calibration evidence is internally circular because CalibEval supplies both the training target and the evaluation target. I would ask the authors to provide an external validation of CalibEval on the continuation distribution, or to evaluate ProbGuard against independent binary outcomes. If they can do that, the paper could become a strong contribution. I also note that the manuscript does not mention code or data release, which would be valuable for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ProbGuard is a genuine step forward within LLM guardrails. The core idea--predicting the probability that a continuation from an early prefix is unsafe, using only output distributions rather than hidden states--is new and well-motivated. The Monte Carlo target construction, the retokenized probability-weighted embeddings, and the post-training setup are all clean and clearly described. The evaluation is also thorough: three LLM families, three datasets, thirteen baselines, six jailbreak attacks, plus ablations on sampling budget, prefix length, and input representation. The ASR results, judged by GPT-5, are independent of the authors' own judge and show real early-intervention value. Credit where it is due: this is a solid, reproducible-looking method paper.\n\nThe soft spot is the one the stress test flags, and it is load-bearing. Every calibration target is the average of CalibEval labels over 16 sampled continuations (Eq. 9), and the Brier/ECE evaluation compares against exactly those same targets (Eqs. 15-16). CalibEval is validated only as a binary classifier on dataset-level annotations (Table 5), with no reliability diagram, no calibration check, and no validation on the continuation distribution. So the headline 79.6% Brier and 71.9% ECE reductions are reductions relative to CalibEval's judgment, not to ground-truth unsafe-continuation probability. If CalibEval is systematically biased on certain continuation styles, every reported calibration number inherits that bias. This does not sink the method--ProbGuard can still be a useful predictor of CalibEval's risk scores--but it means the paper should either validate CalibEval's calibration on continuation distributions or reframe the claims as agreement with a judge rather than calibrated safety.\n\nOther concerns are minor: ASRs come from 100 prompts per attack with no error bars, defense thresholds are tuned on a validation split, and no code or data is released. The internal math is straightforward and the ablations are honest, including the flattening of gains past N=16. The paper is coherent on its own terms; the central limitation is external validity of the judge, not internal inconsistency.\n\nBottom line: this deserves a serious referee. The method is novel enough and the evaluation broad enough to warrant the community's time, but the revision needs to address the CalibEval grounding head-on--either with a calibration study of the judge itself or with an independent judge for at least a subset of the targets. I would not cite the 79.6%/71.9% numbers as established, but I would cite the representation and the architecture-agnostic framing.","headline":"A well-built distribution-based guardrail whose headline calibration numbers all run through the authors' own judge; the method is worth engaging, the 'calibrated' framing is not yet earned.","tokens_in":765,"tokens_out":836,"would_cite":true,"duration_ms":25750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProbGuard predicts unsafe LLM continuations from 10 decoding steps","keywords":["probabilistic guardrail","LLM safety","calibration","safety risk estimation","Monte Carlo sampling","jailbreak defense","early stopping","output distribution signals"],"falsifier":"Take the same prefix states, sample continuations, and have independent human annotators or a separately calibrated judge label them; if ProbGuard's Brier score and ECE against this independently labeled target are close to those of the best baseline rather than roughly 80% better, the calibration claim collapses.","tokens_in":14719,"feed_emoji":"🛡️","tokens_out":5164,"duration_ms":43763,"temperature":0.7,"pith_summary":"The paper tries to establish that an LLM's own output probabilities, taken from only the first few decoding steps, contain enough signal to estimate the probability that the rest of the generation will be unsafe. It proposes ProbGuard, a guardrail that treats safety assessment as a calibrated probability-estimation problem rather than a deterministic safe/unsafe classification. The authors argue this matters because early-generation risk is inherently uncertain and discrete classifiers both discard probabilistic information and cannot express that uncertainty. Their experiments claim that ProbGuard beats 13 baselines on calibration across nine model-dataset settings and keeps jailbreak attack success at or below 1% after ten decoding steps, using only public probability outputs, not hidden states.","feed_headline":"ProbGuard predicts unsafe LLM continuations from 10 decoding steps","feed_subtitle":"A probabilistic guardrail cuts calibration error by 79.6% and holds jailbreak success to at most 1%.","key_machinery":"The load-bearing object is the probabilistic reformulation of guardrailing: given a prefix distribution, safety risk is the expectation over continuations of a binary safety judge. This is estimated by sampling $N=16$ continuations and averaging CalibEval's judgments. For input, each decoding step's top-$K$ probabilities are normalized, each candidate token is retokenized into ProbGuard's vocabulary, and embeddings are averaged and probability-weighted, producing a representation the guardrail can consume regardless of the target LLM's tokenizer or architecture. Training uses a negative log-likelihood loss against the Monte-Carlo risk targets.","core_discovery":"The central claim is that the safety risk of an ongoing LLM generation can be defined as $C_k = \\mathbb{E}_{R \\sim \\Omega_k}[J(R)]$, the expected safety judgment over the distribution of possible full responses reachable from the current prefix, and estimated by Monte-Carlo sampling of continuations. The paper asserts that this probability is learnable from a probability-weighted embedding of the prefix's top-$K$ token distributions, without any access to the target model's hidden states, and that post-training a small model on these distributional signals yields calibrated risk estimates that transfer across LLM families. Concretely, ProbGuard reports the best Brier score and ECE on all nine model-dataset combinations, with average reductions of 79.6% and 71.9% over the strongest baseline, and limits attack success rate to at most 1% across six jailbreak attacks after only ten decoding steps.","pith_inferences":["If output distributions alone carry this signal, then guardrailing could be offered as a service that reads only the public token probabilities of black-box models, avoiding any need for model-internal access.","The same probability-weighted representation could be applied to other early-decision tasks, such as detecting hallucinations or refusal before completion, not just safety.","A direct test: deliberately break CalibEval's calibration on the continuation distribution and check whether ProbGuard's reported Brier/ECE degrade proportionally; this would separate the guardrail's contribution from the judge's noise.","Because thresholds are tuned on a held-out PKU validation split, deployment would likely need per-dataset threshold recalibration; the paper does not claim a universal threshold."],"forward_implications":["Guardrails can operate in probability space rather than text space, enabling intervention before an unsafe sentence is written.","The same trained guardrail transfers across LLM families because it never touches hidden states, only output distributions.","Calibrated risk scores allow downstream systems to choose their own safety thresholds instead of relying on a fixed safe/unsafe label.","Early intervention after ten decoding steps can cut jailbreak success rates from over 50% to at most 1% on AdvBench and HarmBench.","Risk estimates improve as more prefix distributions are observed, from $k=5$ to $k=20$, including beyond the training range."],"supporting_citations":[{"why":"Supplies the Llama 3 target model and the Llama-Guard3 baseline guardrail.","marker":"Dubey et al. 2024"},{"why":"Provides the ShieldGemma baseline, a deterministic guardrail ProbGuard is compared against.","marker":"Zeng et al. 2025"},{"why":"Introduces SCM, the closest streaming-monitor baseline that assesses only observed token sequences.","marker":"Li et al. 2025"},{"why":"Provides the PKU-SafeRLHF dataset, one of the three training and evaluation sources.","marker":"Ji et al. 2025"},{"why":"Provides the WildGuard dataset, the second training and evaluation source.","marker":"Han et al. 2024"},{"why":"Provides the S-Eval dataset, the third training and evaluation source.","marker":"Yuan et al. 2025"},{"why":"Supplies AdvBench prompts and the GCG attack used for jailbreak evaluation.","marker":"Zou et al. 2023"},{"why":"Supplies the HarmBench benchmark for attack success rate evaluation.","marker":"Mazeika et al. 2024"},{"why":"Defines the P(True) confidence-based baseline that ProbGuard must beat to show calibration advantage.","marker":"Kadavath et al. 2022"}],"fun_headline_variants":["ProbGuard cuts jailbreak success to 1% in 10 steps","First probabilistic guardrail: 79.6% better calibration","Predict unsafe continuations before they finish","Ten decoding steps: ProbGuard flags unsafe LLM outputs","Safety risk calibrated from LLM output distributions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire calibration pipeline treats CalibEval, a judge model trained by the authors, as ground truth for safety, and CalibEval is validated only as a binary classifier, so every reported Brier and ECE number is relative to that assumed ground truth.","fun_headline_variants_meta":{"raw":{"variants":["ProbGuard cuts jailbreak success to 1% in 10 steps","First probabilistic guardrail: 79.6% better calibration","Predict unsafe continuations before they finish","Ten decoding steps: ProbGuard flags unsafe LLM outputs","Safety risk calibrated from LLM output distributions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":3010,"prompt_tokens":991,"completion_tokens":2019,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":1941}},"tokens_in":607,"tokens_out":2019,"duration_ms":14871,"temperature":1.0,"reasoning_tokens":1941,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:44:08.889299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same prefix states, sample continuations, and have independent human annotators or a separately calibrated judge label them; if ProbGuard's Brier score and ECE against this independently labeled target are close to those of the best baseline rather than roughly 80% better, the calibration claim collapses.","supporting_citations":[],"review_version":1}