Pith.

Cryptography and Security · arXiv:2610.17820 v1 · submitted 13 Oct 2026 · 40 references

Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers

Xiang Li, Aichun Huang, Mingming Zhang, Zheli Liu

The paper says

“TypeSafe AI's Jev is a new kind of safety judge. It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer.”

more of the abstract

The vendor admits that adversarial input can change these answers but gives no numbers. In this paper, we present the first controlled adversarial evaluation of Jev. We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. Jev is also the fastest judge, but it flags the most benign requests (27%). The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 14 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up. First controlled comparison of Jev vs LLM judges; finds task-dependent robustness – solid methodology with honest limitations. This line is wrong

1 major objection · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 40 references say.

The paper says, at Conclusion and §9, Limitations

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.88):

“The conclusion about judgment-based tasks rests on one task, one corpus, English only, and single-turn calls.”

Pith says: Objection 1 of 1 · major

The paper's central claim that 'Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment' (Abstract, §11) rests on a single judgment-based task (phishing detection) and a single corpus.

more of objection 1

The authors acknowledge this limitation explicitly in §9: 'the conclusion about judgment-based tasks rests on one task, one corpus, English only, and single-turn calls.' While the admission is commendable, the conclusion paragraph in §11 drops this qualifier and states the task-dependence claim as a general result. To avoid overclaiming, the paper should either soften the concluding sentence to reflect the limited evidence (e.g., 'on the judgment-based task we tested') or add a forward-looking sentence that additional tasks are needed. This is a proportionate concern because the entire second half of the paper's main narrative (Jev exposed on phishing, robust on moderation) depends on this one contrast.

6 minor objections

The paper says, at Abstract and §1

Pith says: minor objection 1 of 6

The abstract contains the fragment 'On phishing detection. However' with a period and capital H, which is grammatically broken. The same phrase appears in §1. Please correct to 'On phishing detection, however,' or restructure the sentence.

The paper says, at Figure 3 caption and heatmap

Pith says: minor objection 2 of 6

The heatmap in Figure 3 uses variations of blue and orange that may be difficult for color-blind readers to distinguish. Consider adding pattern fills or text annotations for the cells with the highest ASR values.

show 4 more

The paper says, at Table 1

Pith says: minor objection 3 of 6

Row for A7: 'Language/encoding' should be 'Language/encoding' (missing slash consistency? Actually it's fine. But 'A7a Chinese translation' in the table has 'Chinese translation' but the full text says 'Chinese translation' – unify spelling.

The paper says, at §4.3 Attack Matrix

Pith says: minor objection 4 of 6

The paper describes 13 families and 50 variants, but Table 1 lists 13 families and a probe (A11). Counting the variants, A3 has 7, A13 has 8, etc. It would be helpful to add a row in Table 1 for the total number of variants per family to make the 50 count verifiable.

The paper says, at Appendix I, I.1

Pith says: minor objection 5 of 6

The template for A2c on T4 is described in the text but not given a template sentence in Appendix I. For completeness, include the exact wording of the training-example sentence used on phishing emails.

The paper says, at §6.3

Pith says: minor objection 6 of 6

The sentence 'This null is weaker than it looks' could be rephrased to 'This null result is weaker than it appears' for clarity.

What would settle it

Re-run the A5a 2x2 experiment on T1 using the OpenAI API directly (not via relay) to obtain token-logprobs for the LLM judges' severity scores, and recompute ASR using the exact logprob-derived probabilities. If the ASR gap between Jev and the GPT judges remains large (e.g., > 30 percentage points), the self-report calibration concern is empirically resolved.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 14 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 25 s · 32,086 tokens read, 2,790 written
  3. Editor wrote the verdict above 19 s · 32,687 tokens read, 1,695 written
  4. Referee minor revision: 1 major objection, 6 minor 19 s · 32,296 tokens read, 2,141 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper is the first controlled comparison of Jev, LLM judges, and purpose-built classifiers under identical payloads and policy. The headline result is that Jev's robustness depends on the task: on input moderation (rule-based) it's the hardest to fool, but on phishing detection (judgment-based) it's both less accurate and easier to attack. The work is methodical, honestly reported, and fills a gap that concurrent studies (which test Jev alone) leave open.

What's good: the harness is clean, they measure per-task flip rates as baselines, use bootstrap CIs, and they don't oversell. The 2x2 decomposition of A5a (appendix E) is particularly nice—it shows that the explanation request alone explains the LLM judges' vulnerability, while Jev requires both parts. The Llama Guard 3 result (62% bypass from a training-example statement) is a useful data point. They also flag their own limitations clearly: single judgment-based task, self-reported probabilities for LLM judges, phrase-pool bias toward Jev in the adaptive search.

Soft spots worth noting. The biggest one the paper itself acknowledges: LLM judges' probabilities are written in text, not logprobs. This could understate their robustness, but the binary pass/block comparison is robust to calibration issues, so the core ranking holds. The A4 dilution experiment is inconclusive (authors say so). The adaptive search phrase pool comes from LLM-effective attacks, so the comparison favors Jev—though the cost difference (8x faster against Jev) is still informative. More importantly, the conclusion about judgment-based tasks rests on one corpus (phishing emails). A second task would strengthen the claim, but the authors call this out.

Bottom line: solid empirical work, worth a serious referee. I'd recommend sending to review. No fatal flaws; minor concerns about generalizability that the authors handle honestly.

The strongest objection

The paper provides a controlled comparison, acknowledges its limitations (single judgment-based task, self-reported probabilities, phrase-pool favoritism), and supports its central claim with mechanistic analysis (Appendix E shows the explanation request alone reproduces the effect on LLM judges but not on Jev). The acknowledged weaknesses do not undermine the core result that Jev's robustness is task-dependent. The comparison of binary decisions (pass/block) is robust to calibration artifacts because the 2x2 experiment demonstrates a qualitative behavioral difference independent of probability magnitudes.

The reader identified self-reported probabilities as the weakest assumption. The paper acknowledges this limitation and its impact is bounded: the main ASR comparisons use binary thresholding, and the mechanistic 2x2 experiment in Appendix E shows that even the raw probabilities move differently between Jev and LLM judges. The central claim that Jev is more robust on rule-based tasks versus judgment-based tasks is supported by both clean-performance and attack-success evidence across multiple tasks. The paper's own abundant limitation statements (single judgment-based task, one corpus, phrase-pool bias, etc.) are transparent and do not invalidate the conclusion. Therefore no significant objection arises beyond what the paper already addresses.

The referee's report

If the results hold, the paper makes three significant contributions. First, it provides the first controlled comparison across three different judge designs (typed, LLM, purpose-built) under identical payloads and policy, thereby establishing a methodology that future work can reuse. Second, it demonstrates empirically that a judge's robustness is not a single property but is task-dependent: Jev is highly robust on rule-based tasks but exposed on judgment-based ones, a finding that challenges the notion of a universally better design. Third, it reveals an extreme vulnerability in Llama Guard 3 (62% ASR under a training-example statement) that is larger than any attack on the generative judges, showing that purpose-built classifiers are not inherently robust. The study's strengths include: a detailed threat model with four attacker goals, bootstrap confidence intervals on every proportion, a repeated-query stability baseline that sets the bar for attack-driven effects, and open disclosure of attack templates. Limitations (single judgment-based task, phrase-pool bias favoring Jev, self-reported LLM probabilities) are candidly stated.

Pith's reading of the paper (a machine's summary, not the authors')

This paper presents the first controlled adversarial comparison of Jev, a novel non-generative safety judge that returns probabilities for predefined questions, against five LLM judges and two purpose-built classifiers. The central finding is that Jev's robustness depends strongly on the task: on input moderation (a rule-based task) it blocks all eight structured forgeries and passes only 9% of a teaching-slide attack, compared to 56% for GPT-6 Astra. On phishing detection (a judgment-based task), the same forgeries bypass 19–29% of Jev's flagged emails, and Jev's clean detection rate (62%) lags behind the GLM judges (77–81%). The paper also finds that a purpose-built classifier can be the weakest under attack: a training-example sentence makes Llama Guard 3 miss 62% of flagged phishing. The results imply that no single judge design is universally robust, and that deployment decisions must be matched to the kind of judgment required.

What carries the argument. The central object is Jev, a typed, non-generative safety judge that returns probabilities for developer-defined questions (Noul, Choice, Score) without generating text. Its strict interpretation of question text and structured JSON state input provides robustness on rule-based tasks (input moderation, injection detection) but leaves it vulnerable to structured forgeries on judgment-based tasks (phishing detection). The paper's argument is carried by a controlled test harness that holds the same payload, policy, and metrics fixed across eight systems, so that any difference in attack success rate comes from the judge design, not from the input.

Core claim. On input moderation, Jev resists all eight forgeries of its structured input and passes only 9% of A5a teaching-slide attacks (95% CI 3–15%), while GPT-6 Astra passes 56% (44–67%) and GPT-5.6 Sol passes 54% (43–64%). On phishing detection, the same structured forgeries bypass 19–29% of Jev's flagged emails, and Jev's clean phishing detection rate is 62% (52–72%), below the 77–81% of the three GLM judges. A training-example sentence (A2) does nothing to Jev but makes Llama Guard 3 miss 62% (48–75%) of flagged phishing, the highest attack success rate in the study. Jev is the fastest judge (median 0.7 s vs. 4–24 s for LLM judges) but has the highest false-positive rate (27% vs. 9–17%). The paper concludes that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.

The premise it rests on. The comparison assumes that the LLM judges' self-reported probabilities (written as text in their answers) are a faithful measure of their confidence, which the authors acknowledge may understate their robustness because a model's stated number is weaker evidence than its generation log-probabilities.

If the paper is right (Pith's inferences, not the authors')

Deployers should treat Jev as suitable for input moderation and injection detection, where the judgment is rule-based, but not for phishing or other contextual judgments that require weighing multiple cues.

A confidence threshold on Jev's severity Score catches the few attacks that get through, because those attacks lower confidence rather than producing high-confidence wrong answers.

Jev's 27% false-positive rate on benign requests means its speed advantage comes with the operational cost of reviewing many harmless flagged cases.

Purpose-built classifiers are not inherently robust against adversarial inputs: the training-example attack disabled Llama Guard 3 on 62% of phishing cases, the largest effect in the study.

An adaptive search against Jev runs about eight times faster than against GPT-6 Astra, making Jev a more efficient target for attackers who can iterate.

The task-dependent robustness suggests that building a single universal safety judge is unlikely: the structure of the judgment determines which design is robust, not any intrinsic property of the model.

Jev's different reaction to the explanation request (both the purpose and the explanation needed together to move it, whereas LLMs moved with explanation alone) indicates that Jev's decision process is qualitatively different from an LLM's, not just a restricted generation.

The adaptive search running eight times faster against Jev means an attacker can probe more efficiently, making Jev a more attractive target for brute-force evasion attempts even if per-query success rates are similar.

The phishing result is based on one English-only corpus; multilingual phishing attacks or other judgment-based tasks (e.g., fake news, harmful intent) could shift the relative rankings.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

TypeSafe AI's Jev is a new kind of safety judge. It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer. The vendor admits that adversarial input can change these answers but gives no numbers. In this paper, we present the first controlled adversarial evaluation of Jev. We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. Jev is also the fastest judge, but it flags the most benign requests (27%). The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-14

The authors' main claim, as the classifier picked it out 1.00

“These results suggest that the classifier resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment.”

How the main claim is stated: hedged possibility 1.00

What the abstract itself states, as probabilities: claims first 0.96 · states limitation 0.38 · human participants 0.04 · controlled comparison 0.77 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.94 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
TypeSafe AI's Jev is a new kind of safety judge. background 0.97 0.05 0.01 asserted 1.00
It generates no text and no label: it answers questions the developer defines in advance and returns a probability for each answer. background 0.92 0.07 0.02 asserted 1.00
The vendor admits that adversarial input can change these answers but gives no numbers. gap 0.71 0.06 0.02 asserted 0.91
In this paper, we present the first controlled adversarial evaluation of Jev. contribution 1.00 0.91 0.04 asserted 1.00
We build a test harness that sends every system the same payload under the same policy, and run it on four public-data tasks, from rule-based ones (input moderation, injection detection, routing) to a judgment-based one (phishing detection), with fifty attack variants from thirteen families. method 0.96 0.96 0.03 asserted 0.99
The targets are Jev, five GPT and GLM judges, and two local classifiers (Llama Guard 3, Prompt Guard 2). method 0.96 0.48 0.02 asserted 0.96
We find that Jev is the most robust system on input moderation: eight forgeries of its structured input bypass none of its decisions, and a sentence asking only why someone might make a harmful request bypasses 9% of its blocks, against 56% for GPT-6 Astra. On phishing detection. result 1.00 0.96 0.12 asserted 1.00
However, the same forgeries bypass 19-29% of the emails it flags, and even without attack Jev detects only 62% of phishing emails, against 77-81% for the GLM judges. result 1.00 0.95 0.10 asserted 1.00
Jev is also the fastest judge, but it flags the most benign requests (27%). result 1.00 0.93 0.04 asserted 1.00
The local classifiers fail differently: a training-example statement makes Llama Guard 3 miss 62% of the phishing it had flagged. result 1.00 0.90 0.03 asserted 1.00
These results suggest that Jev resists attacks on tasks with a clear rule to check but is more exposed on tasks that call for an overall judgment. interpretation 1.00 0.77 0.88 hedged possibility 1.00

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-14: 1 request, 8,323 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (10)
Figure 1 of the paper
Figure 1. Evaluation design. (1) Four tasks, each with its public dataset, sample size, and answer type; (2) an attack transform
Figure 2 of the paper
Figure 2. Clean-benign FPR against median latency for the
Figure 3 of the paper
Figure 3. T1 attack success rate of the variants in six families (separated by white lines) against the six generative judges,
Figure 4 of the paper
Figure 4. T2 and T4 true-positive rates for every judge; the
Figure 5 of the paper
Figure 5. Best variant per family against JEV, on input mod￾eration (T1, small dot) and phishing detection (T4, large dot). Hollow orange: not run on T1; hollow blue: GPT-6 Astra on T4. The forgery families (A3, A10, A12, A13) reach 0.18– 0.29 on T4; A3 and A10 never work on T1. question and disturb the others. A6 appends a sentence that denies one hazard in that question’s own words (A6a–d), or denies that complying would cause harm (A6e). We measure how far the probabilities of the questions it does not target move on average. For JEV they move by 0.01–0.04 under every variant. Two variants…
Figure 6 of the paper
Figure 6. Phishing detection (T4): ASR of every variant against every system. Forgeries that never work on
Figure 7 of the paper
Figure 7. Follow-up attacks on T4. Left: putting the two forgeries that work (A3a, A10b) into one payload (A12a) gives
Figure 8 of the paper
Figure 8. A5a’s 2×2 experiment, by system. The control is zero by construction: every sample in the denominator was blocked when clean. Error bars are 95% bootstrap CIs. On every LLM judge the explanation request alone reproduces the whole effect; on JEV neither part works alone, and only the two together do. what the experiment below does. E.2 A 2×2 Experiment We test the two parts separately on the same 100 Jailbreak￾Bench harmful requests, with the same harness and routing policy as the rest of this paper. In every condition the request itself is unchanged; only the added sentence varies:…
Figure 9 of the paper
Figure 9. Margin scores of the requests each system blocks,
Figure 10 of the paper
Figure 10. A8 adaptive search on T1. Left: success on the 50

Figures are the authors', reproduced from the paper.

References (40)
  1. [1] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Se- hwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. Jailbreakbench: An open robustness benchmark for jail- breaking large... on Pith
  2. [2] Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. Struq: Defending against prompt injection with structured queries. InProceedings of the 34th USENIX Conference on Security Symposium, 2025 on Pith
  3. [3] Sizhe Chen, Arman Zharmagambetov, Saeed Mahlou- jifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. Secalign: Defending against prompt injection with preference optimization. InProceedings of the 2025 ACM SIGSAC Conference on Computer and Communi- cations Security, 2025 on Pith
  4. [4] Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents. InPro- ceedings of the 38th International Conference on Neural Informatio... on Pith
  5. [5] Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Li- dong Bing. Multilingual jailbreak challenges in large language models. InProceedings of the 2024 Interna- tional Conference on Learning Representations, 2024 on Pith
  6. [6] Bradley Efron and Robert J Tibshirani.An Introduction to the Bootstrap. Chapman & Hall/CRC, 1993 on Pith
  7. [7] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injec- tion. InProceedings of the 16th ACM Workshop on Artificial Intellige... on Pith
  8. [8] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Wein- berger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, 2017 on Pith
  9. [9] Ruoqi Guo, Yi Liu, Gelei Deng, Yuekang Li, Lida Zhao, Yutao Wu, Simin Chen, Ying Zhang, and Leo Yu Zhang. Just ask jev: Reinforcement learning for calibrated de- cisions as a zero-shot detector of AI alignment failures. arXiv preprint arXiv:2609.29429, 2026 on PithPith's reviewarXiv
  10. [10] David Hofer, Edoardo Debenedetti, and Florian Tramèr. Assessing automated prompt injection attacks in agentic environments.arXiv preprint arXiv:2606.10525, 2026 on PithPith's reviewarXiv
  11. [11] Jianyi Hu, Hangtao Zhang, Yi Liu, Yeqi Zeng, Li Zeng, Xianlong Wang, Rui Wang, and Leo Yu Zhang. Je- vAdvBench: A benchmark and black-box attacks for reinforcement learning for calibrated decisions models. arXiv preprint arXiv:2609.31142, 2026 on PithPith's reviewarXiv
  12. [12] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and Ma- dian Khabsa. Llama guard: Llm-based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674, 2023 on PithPith's reviewarXiv
  13. [13] Zefang Liu. Phishing email dataset. Hugging Face Datasets, https://huggingface.co/datasets/ zefang-liu/phishing-email-dataset, 2023 on Pith
  14. [14] Llama Team, AI @ Meta. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024 on PithPith's reviewarXiv
  15. [15] Todor Markov, Chong Zhang, Sandhini Agarwal, Flo- rentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intel- ligence, 2023 on Pith
  16. [16] Meta AI. Llama prompt guard 2. Model card, https://huggingface.co/meta-llama/Llama- Prompt-Guard-2-86M, 2025. Accessed 2026-10-12 on Pith
  17. [17] John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System ... on Pith
  18. [18] Vyas Raina, Adian Liusie, and Mark Gales. Is LLM- as-a-judge robust? investigating universal adversarial attacks on zero-shot LLM assessment. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024 on Pith
  19. [19] Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. InProceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, 2020 on Pith
  20. [20] Paul Röttger, Bertie Vidgen, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. InProceedings of the 2024 14 Conference of the North American Chapter of the Associ- at... on Pith
show 20 more
  1. [21] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettle- moyer, Nicola Cancedda, and Thomas Scialom. Tool- former: Language models can teach themselves to use tools. InProceedings of the 37th International Confer- ence on Neural In... on Pith
  2. [22] Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spu- rious features in prompt design, or: How i learned to start worrying about prompt formatting. InInternational Conference on Learning Representations, 2024 on Pith
  3. [23] Jia-Wen Shi, Zenghui Yuan, Guiyao Tie, Zhou Pan, Neil Zhenqiang Gong, and Lichao Sun. Prompt injec- tion attack to tool selection in llm agents.arXiv preprint arXiv:2504.19793, 2025 on PithPith's reviewarXiv
  4. [24] Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. Optimization-based prompt injection attack to LLM- as-a-judge. InProceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security, 2024 on Pith
  5. [25] Jonathan Sneh, Ruomei Yan, Jialin Yu, Philip Torr, Yarin Gal, Sunando Sengupta, Eric Sommerlade, Alas- dair Paren, and Adel Bibi. Tooltweak: An attack on tool selection in llm-based agents.arXiv preprint arXiv:2510.02554, 2025 on PitharXiv
  6. [26] Yu Sun, Junhao Xu, Jiajia Shi, and Zijin Yang. Type- safe is not error-free: Typed decision models follow the option name, not the definition bound to it. arXiv preprint arXiv:2609.26758, 2026 on PithPith's reviewarXiv
  7. [27] TypeSafe AI. Introducing system one models & jev. TypeSafe AI Blog, 2026. https://typesafe.ai/ blog/introducing-system-one-models-and-jev , accessed 2026-10-12 on Pith
  8. [28] TypeSafe AI. Llm guardrails: Pass, review, block, or route with System One. TypeSafe AI Cookbook, on Pith
  9. [29] TypeSafe AI. Model jaggedness. TypeSafe AI Docu- mentation, 2026. https://docs.typesafe.ai/model- jaggedness/jev-1.13, accessed 2026-10-12 on Pith
  10. [30] TypeSafe AI. Typesafe api documentation: System one. TypeSafe documentation, 2026. https:// docs.typesafe.ai/api, accessed 2026-10-12 on Pith
  11. [31] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023 on Pith
  12. [32] Tiantong Wu and Wei Yang Bryan Lim. Decision hijack- ing: Prompt injection attacks on jev’s typed probabilistic decisions.arXiv preprint arXiv:2609.28613, 2026 on PithPith's reviewarXiv
  13. [33] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. Proceedings of the 11th International Conference on Learning Representations, 2023 on Pith
  14. [34] Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. Bench- marking and defending against indirect prompt injection attacks on large language models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining V .1, 2025 on Pith
  15. [35] Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents. InFind- ings of the Association for Computational Linguistics: NAACL 2025, 2025 on Pith
  16. [36] Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. Injecagent: Benchmarking indirect prompt injec- tions in tool-integrated large language model agents. In Findings of the Association for Computational Linguis- tics: ACL 2024, 2024 on Pith
  17. [37] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuo- han Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. InProceedings of the 37th International C... on Pith
  18. [38] Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Gong, and Xing Xie. PromptRobust: To- wards evaluating the robustness of large language mod- els on adversarial prompts. InProceedings of the 1st ACM Workshop on Lar... on Pith
  19. [40] and{"meta": {"review": ...}}(depth 3). I.3 Dilution and obfuscation (A4, A7) A4inserts a repeated benign-paragraph filler of n×4 char- acters (n = 1k, 4k, or 16k tokens): split evenly around the request for A4a–c, entirely before it (A4d), or entirely after it (A4e); the T4 va... on Pith
  20. [2026] https://docs.typesafe.ai/cookbooks/ llm_guardrails, accessed 2026-10-12 on Pith
Cited by 5 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment