Pith.

Cryptography and Security · arXiv:2610.20907 v1 · submitted 14 Oct 2026 · 9 references

From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

Qianlong Lan, Anuj Kaul, Vinothini Pandurangan, Indranil Sanyal

The paper says

“Prompt-injection defenses are commonly evaluated as binary detectors, although deployed LLM applications must make operational decisions about whether to allow, block, or escalate potentially malicious inputs.”

more of the abstract

Recent probabilistic decision models aim to support such decisions through calibrated confidence estimates, but their reliability under distribution shift remains uncertain. We evaluate a native probabilistic decision model (Jev), two Llama Prompt Guard 2 classifiers (22M and 86M), and an LLM-based judge on 47,611 held-out examples spanning direct and indirect prompt injections, hard-benign inputs, obfuscation, and distribution shift. Using separate development and calibration splits, we assess detection performance, probability calibration, and allow-block-defer policies under a frozen evaluation protocol. The LLM judge achieves the highest overall discrimination (AUROC 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714). However, Jev's native probabilities do not consistently outperform post-hoc calibrated classifier scores, and temperature scaling improves calibration without improving discrimination. At an empirical false-negative-rate target of 1%, all evaluated gate-reviewer systems require more than 91.8% escalation and incur substantial false-positive rates. These findings demonstrate that probabilistic outputs alone do not establish reliable security decisions under distribution shift. Effective LLM security systems must jointly evaluate detection, calibration, reviewer behavior, and operational decision policies.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 16 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. Well-executed empirical paper with a clear protocol and honest limitations, but the headline conclusion depends heavily on one extreme reviewer setting and the imbalanced benchmark mixture, making its generality weaker than stated. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 9 references say.

The paper says, at §3.2, sentence b62; §5; §7, sentence b148

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The main conclusion—'training for calibrated outputs alone is not sufficient for reliable deployment decisions' (sentence b148)—is primarily supported by the deferral curves (Table 4, Figure 3) obtained under a reviewer with an FNR-first operating point that yields 98.6% FPR (sentence b62).

more of objection 1

The paper acknowledges that 'Reviewer quality constrains the benefit of deferral' (Section 5), but the headline conclusion is stated without explicit conditioning on this extreme reviewer threshold. Because the deferral curves would likely shift substantially under a more balanced reviewer threshold (e.g., one that also penalizes high FPR), the generality of the central claim is unquantified. The authors should explicitly bound the claim to the tested reviewer operating point and discuss, or ideally present an ablation for, how the escalation requirement changes under alternative reviewer thresholds.

The paper says, at §4.2, Table 3; §6, sentences b138, b140

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.93):

“A post-hoc calibration ablation for Jev would help distinguish the transfer of its native calibration from its ability to be calibrated after adaptation.”

Pith says: Objection 2 of 2 · major

The comparison of Jev-native probabilities to temperature-scaled PG2 scores is asymmetric: PG2 benefits from post-hoc calibration on target calibration data, while Jev is used without any such adaptation. The authors acknowledge this asymmetry (sentence b140: 'A post-hoc calibration ablation for Jev would help distinguish the transfer of its native calibration from its ability to be calibrated after adaptation') but do not provide the ablation.

more of objection 2

As a result, the claim that 'native decision probabilities do not consistently outperform target-calibrated classifier probabilities' (sentence b30) conflates the model's native calibration with the effect of target-data adaptation. Adding a post-hoc calibration ablation for Jev (e.g., temperature scaling on the same calibration split) would make the comparison fairer and strengthen the interpretation. Without it, the main calibration comparison is not an apples-to-apples assessment of the models' inherent calibration quality.

6 minor objections

The paper says, at §3.2, sentence b62

Pith says: minor objection 1 of 6

The reviewer threshold of 0.0527 and the resulting 98.6% FPR are critical for interpreting RQ3. Consider explicitly labeling this operating point as 'FNR-first' in the title of the subsection or in a table caption for clarity.

The paper says, at Table 4

Pith says: minor objection 2 of 6

The normalized cost column is listed but not defined in the main text (only mentioned in §3.3 as 'illustrative expected cost'). Add a brief definition or a footnote explaining its components (C_FN=100, C_FP=1, C_review=0.5).

show 4 more

The paper says, at §3.3

Pith says: minor objection 3 of 6

The description of the deferral curve crossing as a 'descriptive confirmatory statistic' is clear, but consider adding a sentence explaining why this statistic is not a deployable threshold (i.e., it uses test labels). This is already implied but worth making explicit.

The paper says, at Abstract, sentence b9

Pith says: minor objection 4 of 6

The phrase 'confirmatory statistics' appears without definition; consider adding a brief definition in §3 or the abstract for readers unfamiliar with the term.

The paper says, at General

Pith says: minor objection 5 of 6

There are a few minor typos: 'System One' (Section 2) should be 'System 1' to match the original terminology; 'scalar temperature' is redundant with 'temperature scaling' in several places. Consider consistent phrasing.

The paper says, at §6, sentence b142

Pith says: minor objection 6 of 6

The sentence about benchmark transformations introducing dependence among examples is vague. Please clarify what transformations are meant (e.g., obfuscation perturbations, prompt augmentations).

What would settle it

Re-run the RQ3 deferral analysis with the same frozen gate models but using a reviewer threshold selected to minimize FPR subject to FNR ≤ 1% on the reviewer-calibration split (instead of the current FNR-first threshold). If the system still requires >91.8% escalation for FNR ≤ 1% on the confirmatory cohort, the claim is robust; if escalation drops substantially (e.g., below 50%), the original conclusion is specific to the reviewer choice and does not support the general statement that calibrated outputs alone are insufficient.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 16 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 31 s · 9,846 tokens read, 3,229 written
  3. Editor wrote the verdict above 14 s · 10,640 tokens read, 1,342 written
  4. Referee minor revision: 2 major objections, 6 minor 25 s · 10,201 tokens read, 2,561 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper does what it says: it runs a clean, leakage-controlled comparison of prompt-injection detectors on detection, calibration, and deferral. The experimental design is careful—separate splits, frozen parameters before confirmatory runs, transparent metric definitions. The authors also deserve credit for openly listing their limitations (e.g., single Jev prompt, single LLM judge, imbalanced mixture, no confidence intervals). That level of honesty is rare and welcome.

The new empirical joint evaluation of discrimination, calibration, and deferral under distribution shift is the real contribution. Prior work looked at these pieces separately.

Now the soft spots. The central claim—that calibrated outputs alone are insufficient—rests primarily on the deferral curves showing >91.8% escalation needed for 1% system FNR. Those curves are a descriptive property of the frozen ranking, but the ranking is driven by the LLM reviewer operating at an FNR-first threshold that produces 98.6% FPR. The paper acknowledges reviewer quality matters (Section 5), but it never tests whether a more balanced reviewer threshold (say, minimizing FPR subject to FNR ≤ 1% on calibration data) would change the conclusion. The stress-test concern is real: if a reviewer were calibrated to a less extreme trade-off, the required escalation might drop substantially, and the claim might weaken. The paper does not bound this.

Second, the benchmark mixture is 88.2% attacks, dominated by BIPIA. The authors say pooled results should not be transferred to deployment, but the conclusion about calibration sufficiency is drawn from this mixture. A more natural prevalence or different attack types could shift results.

Neither of these is a fatal flaw. The paper’s core finding—that calibration alone doesn’t guarantee good deferral behavior under this specific setup—holds up. But the generality of the claim is narrower than the prose suggests. I’d want the discussion to explicitly bound the conclusion to the reviewer threshold tested and call for a sensitivity analysis.

Who benefits: LLM security practitioners who need to know that putting a calibrated detector in front of a trigger-happy reviewer won’t magically fix deployment decisions. Researchers building deferral pipelines will get a solid baseline.

I’d recommend the editor send this to review, but with a note that the scope of the conclusion needs tightening. For my own work, I wouldn’t cite it because my interests are elsewhere, but it’s a serious, honest paper.

The strongest objection

The paper's headline conclusion—"training for calibrated outputs alone is not sufficient for reliable deployment decisions"—is primarily supported by the confirmatory deferral curves (Table 4, Figure 3) showing that all gates require >91.8% escalation to achieve FNR≤1%. These curves depend entirely on the frozen LLM reviewer, whose operating point was selected on the calibration split to minimize FNR, yielding a 98.6% FPR (sentence b62). The paper itself acknowledges that "Reviewer quality constrains the benefit of deferral" (Section 5). However, the conclusion that calibrated outputs are insufficient is stated as a general finding, not explicitly bounded to the reviewer's extreme threshold. If the reviewer were set to a more balanced trade-off (e.g., minimizing FPR subject to FNR≤1% on calibration data), the system might achieve low FNR at substantially lower escalation, potentially weakening the claim. The paper does not provide this ablation. The result is thus conditional on an extreme reviewer, and the generality of the conclusion remains unquantified.

The reader's weakest assumption correctly identified that the benchmark mixture and frozen reviewer are specific choices that may not generalize. My stress test sharpens this: the reviewer's extreme FPR (98.6%) is the key variable that drives the escalation requirement. The paper does not test alternative reviewer thresholds, leaving the generality of the central claim ambiguous. Nevertheless, the paper is transparent about this limitation (Section 6, b133) and does not overclaim beyond its specific setup. The results are internally consistent and the code is reproducible. The concern does not invalidate the paper; it suggests that the conclusion should be explicitly scoped to the reviewer policy used. Therefore the ACCEPT verdict remains appropriate.

The referee's report

If the central finding holds, it challenges the practical sufficiency of calibrated probabilistic outputs for security decision pipelines under distribution shift. The paper makes three valuable contributions: (1) a leakage-controlled comparison of a native probabilistic model, specialized classifiers, and an LLM judge on a large held-out cohort; (2) joint evaluation of discrimination, calibration, confident false negatives, latency, and operational deferral; (3) concrete evidence that, under the tested reviewer operating point, even well-calibrated models lead to very high escalation rates. The result is important for the security, calibration, and selective prediction communities. Strengths include a rigorous frozen-protocol design with separate splits for development, calibration, and confirmatory evaluation, and explicit acknowledgment of key limitations (mixture prevalence, single-Jev-prompt, lack of confidence intervals). The significance is somewhat tempered by the artificial mixture (88.2% attack prevalence, dominated by BIPIA) and the extreme FNR-first reviewer threshold (98.6% FPR), but these are transparently documented.

Pith's reading of the paper (a machine's summary, not the authors')

The paper asks whether a detector that outputs calibrated probabilities—specifically Jev, a model trained explicitly for calibrated decisions—actually supports safe operational decisions on prompt-injection attacks. It compares Jev with two sizes of Llama Prompt Guard 2 and an LLM judge on 47,611 held-out examples spanning direct attacks, indirect injections, hard-benign inputs (harmless prompts with attack-like wording), obfuscation, and distribution shift, with prompts, temperatures, and policies fixed before evaluation. The paper's central claim is that training for calibrated outputs alone is not sufficient for reliable deployment decisions: the LLM judge ranks attacks best, Jev's native probabilities do not consistently beat temperature-scaled classifier scores, and every tested gate–reviewer system reaches a system $\mathrm{FNR}\le 1\%$ only after more than 91.8% of inputs are escalated, with false-positive rates above 40%. If this is right, deployers should evaluate detection, calibration, reviewer behavior, and decision policy as one pipeline rather than trusting a single calibrated probability.

What carries the argument. The central mechanism is a three-stage decision pipeline: a security gate assigns an attack probability, a fixed policy maps that probability into ALLOW, DEFER, or BLOCK intervals fitted only on calibration data, and deferred inputs go to a frozen LLM reviewer whose threshold was also set in advance. The load-bearing quantity is the 'empirical confirmatory escalation requirement'—the first point on the frozen-ranking deferral curve at which the whole gate–reviewer system satisfies a system $\mathrm{FNR}\le 1\%$. Temperature scaling, fitting $T$ in $p_{\mathrm{cal}} = \sigma(z/T)$ on a calibration split, separates calibration quality from discrimination, and the deferral curves convert probability scores into policy outcomes. This machinery lets the paper show that calibration adjustments improve probability metrics while leaving score rankings and decisions unchanged, and that reviewer behavior rather than gate calibration dominates what the deferral curves deliver.

Core claim. On its own terms, the paper establishes that native calibrated probabilities from a purpose-built decision model do not consistently outperform ordinary classifier scores that have been temperature-scaled on target data, and that calibration quality is not the operational bottleneck. Across the confirmatory mixture, the LLM judge achieved the highest discrimination (area under the ROC curve, AUROC, 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714). Jev's native probabilities recorded an expected calibration error (ECE) of 0.541, a Brier score of 0.440, and a negative log-likelihood (NLL) of 1.432, compared with 0.571, 0.439, and 1.104 for temperature-scaled PG2-86M, so the model described as decision-calibrated was not consistently better calibrated on this shifted mixture. The policy-level finding is that every gate, paired with the frozen LLM reviewer, required more than 91.8% escalation of inputs to reach a system $\mathrm{FNR}\le 1\%$, with system false-positive rates above 40%. The paper reads these crossings as descriptive properties of rankings fixed before evaluation, not as deployment thresholds, and concludes that calibrated outputs alone are insufficient for reliable security decisions under distribution shift.

The premise it rests on. The load-bearing premise is that the confirmatory mixture—88.2% attacks dominated by BIPIA, reviewed by a frozen LLM judge whose miss-minimizing threshold blocks 98.6% of benign calibration inputs—represents the conditions under which deployment decisions are actually made; if production shift or reviewer quality differs, the more-than-91.8% escalation result can change.

If the paper is right (Pith's inferences, not the authors')

A deployer who trusts a model's stated attack probability, even from a model advertised as decision-calibrated, should not assume that probability supports safe routing once the input distribution shifts.

Post-hoc temperature scaling can make confidence scores look better calibrated without recovering missed attacks, so calibration and discrimination have to be reported separately.

When deferral is the safety mechanism, the reviewer's operating point—here a threshold with 98.6% false-positive rate on its calibration split—can dominate system outcomes more than gate calibration does.

On this mixture, none of the tested methods supports autonomous blocking at a low false-positive rate: true-positive rate at a 1% false-positive rate ranged from 0.9% to 12.5%.

Security evaluations of LLM guards should report decision-policy outcomes such as escalation rate, system false-negative and false-positive rates, normalized cost, and latency alongside AUROC and calibration metrics.

Because the pooled mixture is 88.2% attacks and dominated by BIPIA, the paper's headline escalation figure is a property of that constructed mixture; a production distribution with rarer attacks could produce different calibration and deferral curves.

The same frozen protocol run with a reviewer threshold chosen for lower false positives, or with a second reviewer, would separate gate weakness from reviewer weakness as the cause of the more-than-91.8% escalation requirement.

Adding a post-hoc calibrated Jev variant, which the paper identifies as missing, would show whether Jev's advantage is transferable native calibration or just calibratability, and could change the RQ2 comparison.

Plugging application-specific review costs into the illustrative (1,100,0.5) cost model could identify operating points where deferring is cheaper than blocking, changing the recommended policy for a given gate.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Prompt-injection defenses are commonly evaluated as binary detectors, although deployed LLM applications must make operational decisions about whether to allow, block, or escalate potentially malicious inputs. Recent probabilistic decision models aim to support such decisions through calibrated confidence estimates, but their reliability under distribution shift remains uncertain. We evaluate a native probabilistic decision model (Jev), two Llama Prompt Guard 2 classifiers (22M and 86M), and an LLM-based judge on 47,611 held-out examples spanning direct and indirect prompt injections, hard-benign inputs, obfuscation, and distribution shift. Using separate development and calibration splits, we assess detection performance, probability calibration, and allow-block-defer policies under a frozen evaluation protocol. The LLM judge achieves the highest overall discrimination (AUROC 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714). However, Jev's native probabilities do not consistently outperform post-hoc calibrated classifier scores, and temperature scaling improves calibration without improving discrimination. At an empirical false-negative-rate target of 1%, all evaluated gate-reviewer systems require more than 91.8% escalation and incur substantial false-positive rates. These findings demonstrate that probabilistic outputs alone do not establish reliable security decisions under distribution shift. Effective LLM security systems must jointly evaluate detection, calibration, reviewer behavior, and operational decision policies.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-16

The authors' main claim, as the classifier picked it out 0.81

“These findings demonstrate that probabilistic outputs alone do not establish reliable security decisions under distribution shift.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.78 · human participants 0.04 · controlled comparison 0.44 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.91 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Prompt-injection defenses are commonly evaluated as binary detectors, although deployed LLM applications must make operational decisions about whether to allow, block, or escalate potentially malicious inputs. background 0.95 0.04 0.03 asserted 1.00
Recent probabilistic decision models aim to support such decisions through calibrated confidence estimates, but their reliability under distribution shift remains uncertain. gap 0.88 0.04 0.04 asserted 0.62
We evaluate a native probabilistic decision model (Jev), two Llama Prompt Guard 2 classifiers (22M and 86M), and an LLM-based judge on 47,611 held-out examples spanning direct and indirect prompt injections, hard-benign inputs, obfuscation, and distribution shift. method 0.88 0.95 0.02 asserted 0.97
Using separate development and calibration splits, we assess detection performance, probability calibration, and allow-block-defer policies under a frozen evaluation protocol. method 0.99 0.95 0.03 asserted 0.94
The LLM judge achieves the highest overall discrimination (AUROC 0.897), followed by Jev (0.871), Prompt Guard 2 86M (0.757), and Prompt Guard 2 22M (0.714). result 1.00 0.95 0.07 asserted 1.00
However, Jev's native probabilities do not consistently outperform post-hoc calibrated classifier scores, and temperature scaling improves calibration without improving discrimination. result 1.00 0.94 0.11 asserted 0.98
At an empirical false-negative-rate target of 1%, all evaluated gate-reviewer systems require more than 91.8% escalation and incur substantial false-positive rates. result 1.00 0.95 0.12 asserted 1.00
These findings demonstrate that probabilistic outputs alone do not establish reliable security decisions under distribution shift. interpretation 0.99 0.88 0.73 asserted 1.00
Effective LLM security systems must jointly evaluate detection, calibration, reviewer behavior, and operational decision policies. interpretation 0.81 0.25 0.71 asserted 0.99

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 0.98

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-16: 1 request, 7,051 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. Evaluation design and decision pipeline. (A) Prompts, models, calibration parameters, and the gate policy
Figure 2 of the paper
Figure 2. Overall reliability diagrams for the frozen confirmatory mixture, which contains 88.2% attacks. The diagonal
Figure 3 of the paper
Figure 3. Overall security-deferral curves on the confirmatory cohort, with deferred inputs sent to the frozen LLM

Figures are the authors', reproduced from the paper.

References (9)
  1. [1] Diogo Almeida. Introducing system one models & Jev. https://typesafe.ai/blog/ introducing-system-one-models-and-jev, 2026. TypeSafe AI, accessed 2026-10-01 on Pith
  2. [2] Md Anas Biswas. Confidently wrong: Severity-aware calibration of prompt-injection detectors under attack shift. arXiv preprint arXiv:2606.22659, 2026. 7 CALIBRATION AND DEFERRAL on PithPith's reviewarXiv
  3. [3] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks.Advances in neural information processing systems, 30, 2017 on Pith
  4. [4] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. PMLR, 2017 on Pith
  5. [5] Hao Li and Xiaogeng Liu. Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models.arXiv preprint arXiv:2410.22770, 2024 on PithPith's reviewarXiv
  6. [6] Meta AI. Llama Prompt Guard 2 86M model card. https://huggingface.co/meta-llama/ Llama-Prompt-Guard-2-86M, 2025. Accessed 2026-10-01 on Pith
  7. [7] Yusuf Khalid Shire and Sang-Chul Kim. Pids-bench: Evaluating prompt-injection detectors under over-defense, obfuscation, and distribution shift.IEEE Access, 2026 on Pith
  8. [8] Sam Toyer, Olivia Watkins, Ethan Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Alan Ritter, et al. Tensor trust: Interpretable prompt injection attacks from an online game. InInternational Conference on Learning Representation... on Pith
  9. [9] Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Benchmarking and defending against indirect prompt injection attacks on large language models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 1,... on Pith
Cited by 5 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment