Cryptography and Security · arXiv:2610.17820 v1 · submitted 13 Oct 2026 · 40 references
Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers
The paper says
“TypeSafe AI's Jev is a new kind of safety judge. It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer.”
more of the abstract
The vendor admits that adversarial input can change these answers but gives no numbers. In this paper, we present the first controlled adversarial evaluation of Jev. We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. Jev is also the fastest judge, but it flags the most benign requests (27%). The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. First controlled comparison of Jev vs LLM judges; finds task-dependent robustness – solid methodology with honest limitations. This line is wrong
1 major objection · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 40 references say.
The paper says, at Conclusion and §9, Limitations
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.88):
“The conclusion about judgment-based tasks rests on one task, one corpus, English only, and single-turn calls.”
Pith says: Objection 1 of 1 · major
The paper's central claim that 'Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment' (Abstract, §11) rests on a single judgment-based task (phishing detection) and a single corpus.
more of objection 1
The authors acknowledge this limitation explicitly in §9: 'the conclusion about judgment-based tasks rests on one task, one corpus, English only, and single-turn calls.' While the admission is commendable, the conclusion paragraph in §11 drops this qualifier and states the task-dependence claim as a general result. To avoid overclaiming, the paper should either soften the concluding sentence to reflect the limited evidence (e.g., 'on the judgment-based task we tested') or add a forward-looking sentence that additional tasks are needed. This is a proportionate concern because the entire second half of the paper's main narrative (Jev exposed on phishing, robust on moderation) depends on this one contrast.
6 minor objections
The paper says, at Abstract and §1
Pith says: minor objection 1 of 6
The abstract contains the fragment 'On phishing detection. However' with a period and capital H, which is grammatically broken. The same phrase appears in §1. Please correct to 'On phishing detection, however,' or restructure the sentence.
The paper says, at Figure 3 caption and heatmap
Pith says: minor objection 2 of 6
The heatmap in Figure 3 uses variations of blue and orange that may be difficult for color-blind readers to distinguish. Consider adding pattern fills or text annotations for the cells with the highest ASR values.
show 4 more
The paper says, at Table 1
Pith says: minor objection 3 of 6
Row for A7: 'Language/encoding' should be 'Language/encoding' (missing slash consistency? Actually it's fine. But 'A7a Chinese translation' in the table has 'Chinese translation' but the full text says 'Chinese translation' – unify spelling.
The paper says, at §4.3 Attack Matrix
Pith says: minor objection 4 of 6
The paper describes 13 families and 50 variants, but Table 1 lists 13 families and a probe (A11). Counting the variants, A3 has 7, A13 has 8, etc. It would be helpful to add a row in Table 1 for the total number of variants per family to make the 50 count verifiable.
The paper says, at Appendix I, I.1
Pith says: minor objection 5 of 6
The template for A2c on T4 is described in the text but not given a template sentence in Appendix I. For completeness, include the exact wording of the training-example sentence used on phishing emails.
The paper says, at §6.3
Pith says: minor objection 6 of 6
The sentence 'This null is weaker than it looks' could be rephrased to 'This null result is weaker than it appears' for clarity.
What would settle it
Re-run the A5a 2x2 experiment on T1 using the OpenAI API directly (not via relay) to obtain token-logprobs for the LLM judges' severity scores, and recompute ASR using the exact logprob-derived probabilities. If the ASR gap between Jev and the GPT judges remains large (e.g., > 30 percentage points), the self-report calibration concern is empirically resolved.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 14 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper is the first controlled comparison of Jev, LLM judges, and purpose-built classifiers under identical payloads and policy. The headline result is that Jev's robustness depends on the task: on input moderation (rule-based) it's the hardest to fool, but on phishing detection (judgment-based) it's both less accurate and easier to attack. The work is methodical, honestly reported, and fills a gap that concurrent studies (which test Jev alone) leave open.
What's good: the harness is clean, they measure per-task flip rates as baselines, use bootstrap CIs, and they don't oversell. The 2x2 decomposition of A5a (appendix E) is particularly nice—it shows that the explanation request alone explains the LLM judges' vulnerability, while Jev requires both parts. The Llama Guard 3 result (62% bypass from a training-example statement) is a useful data point. They also flag their own limitations clearly: single judgment-based task, self-reported probabilities for LLM judges, phrase-pool bias toward Jev in the adaptive search.
Soft spots worth noting. The biggest one the paper itself acknowledges: LLM judges' probabilities are written in text, not logprobs. This could understate their robustness, but the binary pass/block comparison is robust to calibration issues, so the core ranking holds. The A4 dilution experiment is inconclusive (authors say so). The adaptive search phrase pool comes from LLM-effective attacks, so the comparison favors Jev—though the cost difference (8x faster against Jev) is still informative. More importantly, the conclusion about judgment-based tasks rests on one corpus (phishing emails). A second task would strengthen the claim, but the authors call this out.
Bottom line: solid empirical work, worth a serious referee. I'd recommend sending to review. No fatal flaws; minor concerns about generalizability that the authors handle honestly.
The strongest objection
The paper provides a controlled comparison, acknowledges its limitations (single judgment-based task, self-reported probabilities, phrase-pool favoritism), and supports its central claim with mechanistic analysis (Appendix E shows the explanation request alone reproduces the effect on LLM judges but not on Jev). The acknowledged weaknesses do not undermine the core result that Jev's robustness is task-dependent. The comparison of binary decisions (pass/block) is robust to calibration artifacts because the 2x2 experiment demonstrates a qualitative behavioral difference independent of probability magnitudes.
The reader identified self-reported probabilities as the weakest assumption. The paper acknowledges this limitation and its impact is bounded: the main ASR comparisons use binary thresholding, and the mechanistic 2x2 experiment in Appendix E shows that even the raw probabilities move differently between Jev and LLM judges. The central claim that Jev is more robust on rule-based tasks versus judgment-based tasks is supported by both clean-performance and attack-success evidence across multiple tasks. The paper's own abundant limitation statements (single judgment-based task, one corpus, phrase-pool bias, etc.) are transparent and do not invalidate the conclusion. Therefore no significant objection arises beyond what the paper already addresses.
The referee's report
If the results hold, the paper makes three significant contributions. First, it provides the first controlled comparison across three different judge designs (typed, LLM, purpose-built) under identical payloads and policy, thereby establishing a methodology that future work can reuse. Second, it demonstrates empirically that a judge's robustness is not a single property but is task-dependent: Jev is highly robust on rule-based tasks but exposed on judgment-based ones, a finding that challenges the notion of a universally better design. Third, it reveals an extreme vulnerability in Llama Guard 3 (62% ASR under a training-example statement) that is larger than any attack on the generative judges, showing that purpose-built classifiers are not inherently robust. The study's strengths include: a detailed threat model with four attacker goals, bootstrap confidence intervals on every proportion, a repeated-query stability baseline that sets the bar for attack-driven effects, and open disclosure of attack templates. Limitations (single judgment-based task, phrase-pool bias favoring Jev, self-reported LLM probabilities) are candidly stated.
Pith's reading of the paper (a machine's summary, not the authors')
This paper presents the first controlled adversarial comparison of Jev, a novel non-generative safety judge that returns probabilities for predefined questions, against five LLM judges and two purpose-built classifiers. The central finding is that Jev's robustness depends strongly on the task: on input moderation (a rule-based task) it blocks all eight structured forgeries and passes only 9% of a teaching-slide attack, compared to 56% for GPT-6 Astra. On phishing detection (a judgment-based task), the same forgeries bypass 19–29% of Jev's flagged emails, and Jev's clean detection rate (62%) lags behind the GLM judges (77–81%). The paper also finds that a purpose-built classifier can be the weakest under attack: a training-example sentence makes Llama Guard 3 miss 62% of flagged phishing. The results imply that no single judge design is universally robust, and that deployment decisions must be matched to the kind of judgment required.
What carries the argument. The central object is Jev, a typed, non-generative safety judge that returns probabilities for developer-defined questions (Noul, Choice, Score) without generating text. Its strict interpretation of question text and structured JSON state input provides robustness on rule-based tasks (input moderation, injection detection) but leaves it vulnerable to structured forgeries on judgment-based tasks (phishing detection). The paper's argument is carried by a controlled test harness that holds the same payload, policy, and metrics fixed across eight systems, so that any difference in attack success rate comes from the judge design, not from the input.
Core claim. On input moderation, Jev resists all eight forgeries of its structured input and passes only 9% of A5a teaching-slide attacks (95% CI 3–15%), while GPT-6 Astra passes 56% (44–67%) and GPT-5.6 Sol passes 54% (43–64%). On phishing detection, the same structured forgeries bypass 19–29% of Jev's flagged emails, and Jev's clean phishing detection rate is 62% (52–72%), below the 77–81% of the three GLM judges. A training-example sentence (A2) does nothing to Jev but makes Llama Guard 3 miss 62% (48–75%) of flagged phishing, the highest attack success rate in the study. Jev is the fastest judge (median 0.7 s vs. 4–24 s for LLM judges) but has the highest false-positive rate (27% vs. 9–17%). The paper concludes that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.
The premise it rests on. The comparison assumes that the LLM judges' self-reported probabilities (written as text in their answers) are a faithful measure of their confidence, which the authors acknowledge may understate their robustness because a model's stated number is weaker evidence than its generation log-probabilities.
If the paper is right (Pith's inferences, not the authors')
Deployers should treat Jev as suitable for input moderation and injection detection, where the judgment is rule-based, but not for phishing or other contextual judgments that require weighing multiple cues.
A confidence threshold on Jev's severity Score catches the few attacks that get through, because those attacks lower confidence rather than producing high-confidence wrong answers.
Jev's 27% false-positive rate on benign requests means its speed advantage comes with the operational cost of reviewing many harmless flagged cases.
Purpose-built classifiers are not inherently robust against adversarial inputs: the training-example attack disabled Llama Guard 3 on 62% of phishing cases, the largest effect in the study.
An adaptive search against Jev runs about eight times faster than against GPT-6 Astra, making Jev a more efficient target for attackers who can iterate.
The task-dependent robustness suggests that building a single universal safety judge is unlikely: the structure of the judgment determines which design is robust, not any intrinsic property of the model.
Jev's different reaction to the explanation request (both the purpose and the explanation needed together to move it, whereas LLMs moved with explanation alone) indicates that Jev's decision process is qualitatively different from an LLM's, not just a restricted generation.
The adaptive search running eight times faster against Jev means an attacker can probe more efficiently, making Jev a more attractive target for brute-force evasion attempts even if per-query success rates are similar.
The phishing result is based on one English-only corpus; multilingual phishing attacks or other judgment-based tasks (e.g., fake news, harmful intent) could shift the relative rankings.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
TypeSafe AI's Jev is a new kind of safety judge. It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer. The vendor admits that adversarial input can change these answers but gives no numbers. In this paper, we present the first controlled adversarial evaluation of Jev. We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. Jev is also the fastest judge, but it flags the most benign requests (27%). The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-14
The authors' main claim, as the classifier picked it out 1.00
“These results suggest that the classifier resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.”
How the main claim is stated: hedged possibility 1.00
What the abstract itself states, as probabilities: claims first 0.96 · states limitation 0.38 · human participants 0.04 · controlled comparison 0.77 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| TypeSafe AI's Jev is a new kind of safety judge. | background 0.97 | 0.05 | 0.01 | asserted 1.00 |
| It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer. | background 0.92 | 0.07 | 0.02 | asserted 1.00 |
| The vendor admits that adversarial input can change these answers but gives no numbers. | gap 0.71 | 0.06 | 0.02 | asserted 0.91 |
| In this paper, we present the first controlled adversarial evaluation of Jev. | contribution 1.00 | 0.91 | 0.04 | asserted 1.00 |
| We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. | method 0.96 | 0.96 | 0.03 | asserted 0.99 |
| The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). | method 0.96 | 0.48 | 0.02 | asserted 0.96 |
| We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. | result 1.00 | 0.96 | 0.12 | asserted 1.00 |
| However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. | result 1.00 | 0.95 | 0.10 | asserted 1.00 |
| Jev is also the fastest judge, but it flags the most benign requests (27%). | result 1.00 | 0.93 | 0.04 | asserted 1.00 |
| The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. | result 1.00 | 0.90 | 0.03 | asserted 1.00 |
| These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment. | interpretation 1.00 | 0.77 | 0.88 | hedged possibility 1.00 |
What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 8,323 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (10)
Figures are the authors', reproduced from the paper.
References (40)
- [1] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Se- hwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jail- breaking large... on Pith
- [2] Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Struq: Defending against prompt injection with structured queries. InProceedings of the 34th USENIX Conference on Security Symposium, 2025 on Pith
- [3] Sizhe Chen, Arman Zharmagambetov, Saeed Mahlou- jifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. Secalign: Defending against prompt injection with preference optimization. InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communi- cations Security, 2025 on Pith
- [4] Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. InPro- ceedings of the 38th International Conference on Neural Informatio... on Pith
- [5] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Li- dong Bing. Multilingual jailbreak challenges in large language models. InProceedings of the 2024 Interna- tional Conference on Learning Representations, 2024 on Pith
- [6] Bradley Efron and Robert J Tibshirani.An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993 on Pith
- [7] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injec- tion. InProceedings of the 16th ACM Workshop on Artificial Intellige... on Pith
- [8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, 2017 on Pith
- [9] Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, and Leo Yu Zhang. Just ask jev: Reinforcement learning for calibrated de- cisions as a zero-shot detector of AI alignment failures. arXiv preprint arXiv:2609.29429, 2026 on PithPith's reviewarXiv
- [10] David Hofer, Edoardo Debenedetti, and Florian Tramèr. Assessing automated prompt injection attacks in agentic environments.arXiv preprint arXiv:2606.10525, 2026 on PithPith's reviewarXiv
- [11] Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, and Leo Yu Zhang. Je- vAdvBench: A benchmark and black-box attacks for reinforcement learning for calibrated decisions models. arXiv preprint arXiv:2609.31142, 2026 on PithPith's reviewarXiv
- [12] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Ma- dian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023 on PithPith's reviewarXiv
- [13] Zefang Liu. Phishing email dataset. Hugging Face Datasets, https://huggingface.co/datasets/ zefang-liu/phishing-email-dataset, 2023 on Pith
- [14] Llama Team, AI @ Meta. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024 on PithPith's reviewarXiv
- [15] Todor Markov, Chong Zhang, Sandhini Agarwal, Flo- rentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intel- ligence, 2023 on Pith
- [16] Meta AI. Llama prompt guard 2. Model card, https://huggingface.co/meta-llama/Llama- Prompt-Guard-2-86M, 2025. Accessed 2026-10-12 on Pith
- [17] John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System ... on Pith
- [18] Vyas Raina, Adian Liusie, and Mark Gales. Is LLM- as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024 on Pith
- [19] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. InProceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, 2020 on Pith
- [20] Paul Röttger, Bertie Vidgen, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. InProceedings of the 2024 14 Conference of the North American Chapter of the Associ- at... on Pith
show 20 more
- [21] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Tool- former: Language models can teach themselves to use tools. InProceedings of the 37th International Confer- ence on Neural In... on Pith
- [22] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spu- rious features in prompt design, or: How i learned to start worrying about prompt formatting. InInternational Conference on Learning Representations, 2024 on Pith
- [23] Jia-Wen Shi, Zenghui Yuan, Guiyao Tie, Zhou Pan, Neil Zhenqiang Gong, and Lichao Sun. Prompt injec- tion attack to tool selection in llm agents.arXiv preprint arXiv:2504.19793, 2025 on PithPith's reviewarXiv
- [24] Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. Optimization-based prompt injection attack to LLM- as-a-judge. InProceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, 2024 on Pith
- [25] Jonathan Sneh, Ruomei Yan, Jialin Yu, Philip Torr, Yarin Gal, Sunando Sengupta, Eric Sommerlade, Alas- dair Paren, and Adel Bibi. Tooltweak: An attack on tool selection in llm-based agents.arXiv preprint arXiv:2510.02554, 2025 on PitharXiv
- [26] Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. Type- safe is not error-free: Typed decision models follow the option name, not the definition bound to it. arXiv preprint arXiv:2609.26758, 2026 on PithPith's reviewarXiv
- [27] TypeSafe AI. Introducing system one models & jev. TypeSafe AI Blog, 2026. https://typesafe.ai/ blog/introducing-system-one-models-and-jev , accessed 2026-10-12 on Pith
- [28] TypeSafe AI. Llm guardrails: Pass, review, block, or route with System One. TypeSafe AI Cookbook, on Pith
- [29] TypeSafe AI. Model jaggedness. TypeSafe AI Docu- mentation, 2026. https://docs.typesafe.ai/model- jaggedness/jev-1.13, accessed 2026-10-12 on Pith
- [30] TypeSafe AI. Typesafe api documentation: System one. TypeSafe documentation, 2026. https:// docs.typesafe.ai/api, accessed 2026-10-12 on Pith
- [31] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023 on Pith
- [32] Tiantong Wu and Wei Yang Bryan Lim. Decision hijack- ing: Prompt injection attacks on jev’s typed probabilistic decisions.arXiv preprint arXiv:2609.28613, 2026 on PithPith's reviewarXiv
- [33] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. Proceedings of the 11th International Conference on Learning Representations, 2023 on Pith
- [34] Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Bench- marking and defending against indirect prompt injection attacks on large language models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining V .1, 2025 on Pith
- [35] Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents. InFind- ings of the Association for Computational Linguistics: NAACL 2025, 2025 on Pith
- [36] Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injec- tions in tool-integrated large language model agents. In Findings of the Association for Computational Linguis- tics: ACL 2024, 2024 on Pith
- [37] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InProceedings of the 37th International C... on Pith
- [38] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Gong, and Xing Xie. PromptRobust: To- wards evaluating the robustness of large language mod- els on adversarial prompts. InProceedings of the 1st ACM Workshop on Lar... on Pith
- [40] and{"meta": {"review": ...}}(depth 3). I.3 Dilution and obfuscation (A4, A7) A4inserts a repeated benign-paragraph filler of n×4 char- acters (n = 1k, 4k, or 16k tokens): split evenly around the request for A4a–c, entirely before it (A4d), or entirely after it (A4e); the T4 va... on Pith
- [2026] https://docs.typesafe.ai/cookbooks/ llm_guardrails, accessed 2026-10-12 on Pith
Cited by 5 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Safety as a Budget: Calibrating the Control Channel of Constrained GRPO for Vision-Language Models cs.AI · Nov 2026
- FlipJev: Fast Black-Box Red Teaming Finds Universal Adversarial Suffixes for Typed Decision Models cs.CR · Oct 2026
- Cross-Primitive Consistency in Jev cs.CV · Oct 2026
- Jev as an LLM Safety Guardrail: An Empirical Study of Input Filtering, Selective Output Review, and Over-Refusal cs.CR · Nov 2026
- Escalate or Close: How Fast-Tier Models Handle Held-Out Web Attack Families in Two-Tier Triage cs.CR · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment