Effective Red-Teaming of Policy-Adherent Agents

Ateret Anaby-Tavor; George Kour; Guy Uziel; Itay Nakash; Koren Lazar; Matan Vetzler

REVIEW 3 major objections 7 minor 38 references

Effective Red-Teaming of Policy-Adherent Agents

T0 review · 3 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Policy-aware red-teaming cracks policy-abiding agents 70% of the time

desk verdict A genuinely new red-teaming framework with a plausible but under-validated benchmark; worth refereeing, but require label validation and artifacts. read the letter →

arxiv 2506.09600 v3 pith:5AMFRTNR submitted 2025-06-11 cs.MA cs.AIcs.CLcs.CR

Itay Nakash , George Kour , Koren Lazar , Matan Vetzler , Guy Uziel , Ateret Anaby-Tavor This is my paper

classification cs.MAcs.AIcs.CLcs.CR

keywords LLMagentspolicyadherencered-teamingadversarialusersimulationjailbreakattacksmulti-agentplanningtau-benchattacksuccessrate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Policy-adherent LLM agents that are supposed to refuse policy-violating requests can be reliably manipulated by an attacker who reads the policy first. The paper introduces CRAFT, a multi-agent red-teaming system whose modules extract relevant policy fragments, plan false-premise "assume that..." lines, and advise which true details to withhold. On a re-purposed version of tau-bench called tau-break, CRAFT reaches 70.0% attack success at pass@1 in the airline domain, versus 42.5% for the original non-strategic user simulation, 50.0% for emotional manipulation, and 35.0% for DAN. The paper also tests three prompt-based defenses and finds that none eliminate the vulnerability, with attack success converging above 80% at pass@4 even under the best defense. The central message is that evaluating agent safety with cooperative user simulations or generic jailbreaks produces a false sense of security.

What carries the argument

The load-bearing mechanism is the division of labor among four LLM agents that together turn a policy document into a conversation script. The PolicyAnalyzer extracts exact policy fragments relevant to the user request; the DeceptionPlanner produces an "Assume <A>" instruction that, if accepted by the target agent, would make the forbidden request look permissible; the AvoidanceAdvisor enumerates "Do not say <X>" rules so the red-teamer never volunteers disqualifying facts such as a basic-economy fare or a change of plans; and the DialogueExecutor runs the multi-turn conversation using those instructions. The evaluation machinery is tau-break, which defines attack success as a forbidden tool call identified by the set difference between a policy-free agent's actions and the gold actions ($\Delta A = A_{\text{free}} \setminus A_{\text{gold}}$), manually inspected for meaningful violations, plus retail tasks with an added authentication requirement.

What would settle it

Re-derive the tau-break airline labels using a different policy-free model (for example DeepSeek-V3) and have multiple independent annotators classify the divergent actions; if the forbidden-action set changes substantially or annotators disagree on what counts as a violation, the 70% attack success rate is not a stable measure of agent vulnerability.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that policy-adherent agents fail not because generic jailbreaks work on them, but because a red-teamer who reasons about the policy can induce specific forbidden tool calls. CRAFT decomposes the attack into four roles: a PolicyAnalyzer that copies the relevant policy fragments, a DeceptionPlanner that instructs the user to say "Assume <A>" to plant a false premise, an AvoidanceAdvisor that lists true statements the user must not say, and a DialogueExecutor that carries out the conversation. The success condition is a forbidden action, not task failure: the attack succeeds only when the agent executes an action like updating a basic-economy reservation or bypassing authentication. To build the benchmark, the paper re-labels 20 airline tau-bench tasks by comparing a policy-free agent's actions with the gold actions and manually flagging the divergent ones, and augments 30 retail tasks with an authentication rule whose violation counts as an attack. Stripping out the policy knowledge or the planning modules drops attack success toward the naive-user baseline, which is the paper's evidence that policy-aware planning, not generic prompting, is what carries the attack.

Load-bearing premise

The benchmark's attack-success labels rest on the assumption that comparing a policy-free GPT-4o agent's actions with the gold set, followed by manual inspection, correctly identifies every meaningful policy violation; if that labeling is incomplete or biased, all reported attack success rates inherit the error.

Editorial extensions

If this is right

Generic jailbreak and prompt-injection attacks (DAN, DPI, emotional manipulation, and insistence) understate the real risk to policy-constrained agents by 20 to 35 percentage points in the airline domain.
Deploying a non-strategic, cooperative user simulator as a safety evaluation gives a misleadingly optimistic picture of policy adherence.
Prompt-based defenses can lower attack success, but none removes it; under repeated trials (pass@4) even the strongest tested defense converges above 80% ASR, so safety cannot be secured by prompting alone.
The same conversion recipe, using a policy-free agent plus relabeling or adding new policy constraints, can turn other task-completion benchmarks into security benchmarks.
Being a strong attacker and being a robust agent are separate properties; Qwen2.5-70B is among the best attackers and also the most vulnerable agent in the authors' measurements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's claims: the 70% airline ASR is measured under the assumption that the attacker knows the full policy document; if real deployments keep policies confidential, CRAFT-style attacks would likely lose some of their edge, so testing with partially inferred policies is a natural next step.
Beyond the paper's claims: because the success labels come from manual inspection of the delta set, an independent re-annotation study with multiple annotators and different policy-free models would show how much of the reported ASR is an artifact of labeling choices.
Beyond the paper's claims: the "Assume <A>" mechanism suggests a targeted defense the paper does not develop, namely asking the agent to verify factual premises against the database or policy before acting, rather than merely re-prompting the policy.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper is worth reading if you work on LLM agent safety. The CRAFT decomposition into PolicyAnalyzer, DeceptionPlanner, and AvoidanceAdvisor is genuinely new, and the tau-break conversion recipe from task-completion to security evaluation is a useful idea that others will build on. The ablation showing that policy knowledge drives most of the gain is clean, and the defense experiments, though light, are a reasonable first pass. The paper is honest about its limitations, which I appreciate.

The soft spot is the airline label construction. The stress-test note is on target: the 70% ASR number and every comparison in Tables 1-3 depend on a label set derived from a single policy-free GPT-4o agent plus manual inspection. No inter-annotator agreement, no check that a different or stronger policy-free model would give the same Delta-A, and no verification that the labeled action actually produces a policy violation when executed with the target agent's arguments. That is a real validity threat, and the paper does not answer it. The retail domain is cleaner because the added authentication policy makes violations unambiguous.

The statistical concerns are real but minor by comparison. n=4 per task is thin, and the paper reports no confidence intervals. The pass@k point estimates bounce around enough that the 70% vs 42.5% gap is probably real in direction but the magnitude is shaky. That alone would not stop me from taking the paper seriously, but the label issue could.

I do not think the central argument collapses. Even if the airline labels shift, the qualitative finding that policy-aware multi-agent planning beats generic jailbreaks is plausible and supported by the retail numbers and the ablation. But the exact headline number should not be trusted until the labels are validated with multiple policy-free models, an explicit rubric, and ideally released data.

Who is this for? Researchers and practitioners building or red-teaming customer-service agents. It deserves a serious referee, not a desk reject. For acceptance I would require the code and data release, a validation of the airline labels, and at least some uncertainty quantification on the headline comparisons. If those are addable, this is a solid paper.

Referee Report

3 major / 7 minor

Summary. The paper proposes CRAFT, a multi-agent red-teaming system that extracts relevant policy fragments, plans deception strategies, and advises on avoided disclosures in order to induce policy violations in task-oriented LLM agents. It introduces τ-break, a security-focused re-labeling of τ-bench airline and retail tasks, reports attack success rates for five agent models and several attack baselines, and evaluates three prompt-based defenses. The headline result is a 70.0% attack success rate (ASR) for CRAFT in the airline domain versus 42.5% for the non-strategic τ-bench user simulation, with the claim that non-strategic simulation severely underestimates real-world attack risk.

Significance. If the benchmark labels are reliable and the reported differences are statistically robust, the paper makes a useful contribution: it demonstrates that policy-aware red-teaming substantially outperforms generic jailbreak methods at eliciting policy violations, and that non-strategic user simulation underestimates risk. The proposed methodology for converting a task-completion benchmark into a security benchmark is reusable, and the defense evaluation addresses a practical deployment concern. The authors are transparent about limitations (small task set, synthetic environment, static attacks, full-policy assumption) and include a responsible-disclosure plan. However, the central quantitative claims currently rest on a label-construction procedure that is not fully validated and on statistics computed from very small per-task samples without uncertainty quantification.

major comments (3)

[Section 4 and Appendix C.1] The airline-domain ground-truth labels are obtained by running a single policy-free GPT-4o agent and manually inspecting the divergent actions in ΔA. This is unvalidated in three respects: (i) a different or stronger policy-free model could yield a different ΔA and hence a different set of tasks and different attack successes; (ii) the manual inspection has no reported inter-annotator agreement, rubric, or independent verification; and (iii) the labels are action names, so a call to e.g., update_reservation_flights that does not actually change the database state would still be counted as a success. Because every ASR in Tables 1–3 inherits these labels, the benchmark's completeness and precision must be demonstrated, for example with multiple annotators, a second policy-free model, and tool-execution checks.
[Section 5.1 and Tables 1–3] All pass@k and ASR values are computed from n=4 trials per task over 20 airline tasks (80 trials per method), and no confidence intervals or significance tests are reported. For the headline comparison of 70.0% vs 42.5% (56/80 vs 34/80), the approximate 95% confidence intervals are 59–80% and 32–54%, so the difference is nominally significant, but for narrower comparisons (for example GPT-4o-mini as attacker: 71.2 vs 56.3) no such evidence is given. Report confidence intervals and a paired test (e.g., McNemar) across the 20 tasks for each method pair, and clarify how the four runs per task are made independent given the fixed seed mentioned in Section 5.2.
[Section 7.1 and Table 6] The claim 'By k = 4, even the strongest defense (Policy Reminder (Relevant)), exceeds 80% ASR' is not supported by the reported data. Under CRAFT, the pass@4 values for Policy Reminder (Relevant) are 55.0 (GPT-4o), 60.0 (GPT-4o-mini), 90.0 (LLaMA-3.3), 85.0 (DeepSeek-V3), and 90.0 (Qwen2.5-70B), which average to 76.0%. Please either correct the claim or describe the aggregation used to produce Figure 4 so that the reader can reproduce it.

minor comments (7)

[Abstract] The sentence 'outperforming conventional jailbreak methods such as DAN prompts, emotional manipulation, and coercive' ends with an incomplete item; 'coercive' should be 'coercive prompting' or the sentence should be rewritten.
[Section 5.2] The statement 'with seed set to 10 for reproducibility' does not specify sampling temperature or how the four trials are made independent; please clarify the sampling protocol.
[Table 3 caption] The caption says 'Attack Success Rate (ASR) at various pass @ levels' but the table reports only pass@1; please update the caption to match the contents.
[Section 6.1] The text says 'removing either the PolicyAnalyzer or the policy knowledge itself', but Table 4 has no row for removing policy knowledge explicitly; please clarify which ablation corresponds to this statement.
[Section 9 (Ethical Considerations)] The sentence 'public release will of the work artifacts will redact the attack implementation details' is grammatically incomplete and appears to conflict with the GitHub release statement in the abstract; please rewrite.
[Appendix F] The verify_user_secret tool always returns 'verified: False', so a compliant agent can never satisfy the user's request in the retail tasks; the main text should state that retail ASR measures refusal-only behavior, as this affects interpretation.
[Throughout] The spelling of 'DialogueExecuter' varies with 'DialogueExecutor'; please make the terminology consistent.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: tau-break labels are constructed independently of CRAFT, and all attack methods are scored against the same external forbidden-action labels.

full rationale

The paper's central empirical claim is that CRAFT achieves higher attack success than generic jailbreaks and the tau-bench user simulator on the tau-break benchmark. The forbidden-action labels that define attack success are constructed in Section 4 independently of CRAFT: in the airline domain, Delta-A = A_free \ A_gold is obtained by running a policy-free GPT-4o agent on each tau-bench task and then manually inspecting the divergent actions; in the retail domain, new authentication constraints and a verify_user_secret tool are added, and violations are defined as sensitive actions taken without authentication. No CRAFT component or parameter is fitted to these labels, and no equation in the paper defines CRAFT's outputs in terms of the labels or vice versa. The same precomputed labels are applied uniformly to CRAFT, DAN, emotional manipulation, DPI, insistent prompting, and the non-strategic user baseline, so the comparisons are not forced by construction. The only same-author citations (Nakash et al. 2025 and Kour et al. 2025) appear in related-work and analysis contexts as contrasts or supporting observations; they are not load-bearing justifications, and no uniqueness theorem or ansatz is imported from them. The paper's acknowledged limitations, such as full policy access, a synthetic environment, and a small number of adversarial test cases, are scope limitations rather than circularity. The derivation chain is therefore self-contained with respect to the benchmark labels, and no fitted parameter is renamed as a prediction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the paper reports empirical attack success rates with stated experimental settings (4 trials, seed 10, 30 turn limit). The central assumptions are the attacker's policy access, the correctness of the manually derived violation labels, and the representativeness of the synthetic tau-bench environment. No new theoretical entities are postulated; CRAFT is a software system, not a postulated physical or formal object.

assumptions (3)

domain assumption Attackers have full access to the agent's policy documentation.
The Limitations section (fourth point) states the attack setup assumes full access to the policy documentation, which may not hold for closed-source or proprietary deployments.
domain assumption The policy-free agent and manual inspection correctly identify all meaningful policy violations in the airline domain.
Section 4 describes deriving violations as Afree minus Agold from a policy-free GPT-4o agent, with each candidate manually inspected; the benchmark's validity depends on this labeling.
domain assumption The tau-bench simulated environment captures the essential vulnerability surface of real customer service agents.
The Limitations section states the evaluation is fully synthetic and may not capture real-world complexity; the paper generalizes from this environment to real deployments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Effective Red-Teaming of Policy-Adherent Agents." pith.science (2026). https://pith.science/paper/5AMFRTNR

@misc{pith2026250609600,
  author       = {Pith},
  title        = {Pith review of: Effective Red-Teaming of Policy-Adherent Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5AMFRTNR}},
  note         = {Machine review of arXiv:2506.09600}
}

read the original abstract

Task-oriented LLM-based agents are increasingly used in domains with strict policies, such as refund eligibility or cancellation rules. The challenge lies in ensuring that the agent consistently adheres to these rules and policies, appropriately refusing any request that would violate them, while still maintaining a helpful and natural interaction. This calls for the development of tailored design and evaluation methodologies to ensure agent resilience against malicious user behavior. We propose a novel threat model that focuses on adversarial users aiming to exploit policy-adherent agents for personal benefit. To address this, we present CRAFT, a multi-agent red-teaming system that leverages policy-aware persuasive strategies to undermine a policy-adherent agent in a customer-service scenario, outperforming conventional jailbreak methods such as DAN prompts, emotional manipulation, and coercive. Building upon the existing tau-bench benchmark, we introduce tau-break, a complementary benchmark designed to rigorously assess the agent's robustness against manipulative user behavior. Finally, we evaluate several straightforward yet effective defense strategies. While these measures provide some protection, they fall short, highlighting the need for stronger, research-driven safeguards to protect policy-adherent agents from adversarial attacks

Figures

Figures reproduced from arXiv: 2506.09600 by the authors.

**Figure 2.** CRAFT: a multi-agent red-teaming system for eliciting policy violations in policy-adherent agents. A full example and interaction trajectory is provided in Appendix A.1 and B. policies. As such, they overlook the vulnerabilities that arise when agents are expected to follow strict procedural or policy constraints, a common requirement in many real-world deployment settings. 3 Method To evaluate an agent’s ability t… view at source ↗

**Figure 3.** Dataset construction process. In the airline domain (left), we used a method based on policy-free agent to generate forbbiden actions for each task (∆A = Afree \ Agold). In the retail domain (right), we augment the policy with authentication requirements and label as violations any gold actions that bypass authentication. These violations define new attack test cases. alter the gold label to reflect the set of actio… view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Pass@k Attack Success Rate (ASR) averaged across all agent models, under both non-strategic user simulations (Naïve User, the original τ -bench baseline, left) and adversarial interactions using the CRAFT attack (ours, right). Lower ASR indicates stronger policy adher…

**Figure 6.** Figure 6: Example trajectory of the same user request [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Example of CRAFT agent generations for a given user request, compared to the non-strategic user [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 12 canonical work pages

[1]

Sahar Abdelnabi, Kai Greshake, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. https://doi.org/10.1145/3605764.3623985 Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection . In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec 2023, Cope...

arXiv 2023
[2]

Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. 2024. https://doi.org/10.48550/ARXIV.2410.09024 Agentharm: A benchmark for measuring harmfulness of LLM agents . CoRR, abs/2410.09024

arXiv doi:10.48550/arxiv.2410.09024 2024
[3]

Jayachandu Bandlamudi, Ritwik Chaudhuri, Neelamadhav Gantayat, Kushal Mukherjee, Prerna Agarwal, Renuka Sindhgatta, and Sameep Mehta. 2025. A framework for testing and adapting rest apis as llm tools. arXiv preprint arXiv:2504.15546

arXiv 2025
[4]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

arXiv 2021
[5]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024
[6]

Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123--44279

work page 2023
[7]

Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. 2024. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. CoRR

work page 2024
[8]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

arXiv 2024

Show all 38 references

[9]

Haibo Jin, Ruoxi Chen, Peiyan Zhang, Andy Zhou, Yang Zhang, and Haohan Wang. 2024. Guard: Role-playing to generate natural-language jailbreakings to test guideline adherence of large language models. arXiv preprint arXiv:2402.03299

2024
[10]

Jared Joselowitz, Arjun Jagota, Satyapriya Krishna, and Sonali Parbhoo. 2024. Insights from the inverse: Reconstructing llm training goals through inverse rl. arXiv preprint arXiv:2410.12491

2024
[11]

Rand Koualty, Nien-Ying Chou, and Suleiman Alabdallah. 2024. Generative ai agents, build a multilingual chatgpt-based customer service chatbot. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), pages 5--10. IEEE

2024
[12]

George Kour, Marcel Zalmanovici, Naama Zwerdling, Esther Goldbraich, Ora Nova Fandina, Ateret Anaby-Tavor, Orna Raz, and Eitan Farchi. 2023. Unveiling safety vulnerabilities of large language models. arXiv preprint arXiv:2311.04124

2023 arXiv
[13]

George Kour, Naama Zwerdling, Marcel Zalmanovici, Ateret Anaby Tavor, Ora Nova Fandina, and Eitan Farchi. 2025. https://aclanthology.org/2025.naacl-industry.10/ Exploring straightforward methods for automatic conversational red-teaming . In Proceedings of the 2025 Conference o...

2025
[14]

Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. 2024. St-webagentbench: A benchmark for evaluating safety and trustworthiness in web agents. arXiv preprint arXiv:2410.06703

2024 arXiv
[15]

Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, William Yang Wang, and Xifeng Yan. 2025. Agentorca: A dual-system framework to evaluate language agents on operational routine and constraint adherence. arXiv preprint...

2025 arXiv
[16]

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024 a . Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437

2024 arXiv
[17]

Na Liu, Liangyu Chen, Xiaoyu Tian, Wei Zou, Kaijiang Chen, and Ming Cui. 2024 b . https://doi.org/10.48550/ARXIV.2401.02777 From LLM to conversational agent: A memory enhanced architecture with fine-tuning of large language models . CoRR, abs/2401.02777

2024 doi arXiv
[18]

Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023 a . Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688

2023 arXiv
[19]

Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, and Yang Liu. 2023 b . https://doi.org/10.48550/ARXIV.2306.05499 Prompt injection attack against llm-integrated applications . CoRR, abs/2306.05499

2023 doi arXiv
[20]

Margherita Mori. 2021. Ai-powered virtual assistants in the realms of banking and financial services. In Artificial Intelligence in Financial Services. IntechOpen

2021
[21]

u ndler, Mark Niklas M \

Niels M \"u ndler, Mark Niklas M \"u ller, Jingxuan He, and Martin Vechev. 2024. Code agents are state of the art software testers. In ICML 2024 Workshop on LLMs and Cognition

2024
[22]

Itay Nakash, George Kour, Guy Uziel, and Ateret Anaby Tavor. 2025. https://aclanthology.org/2025.findings-naacl.363/ Breaking R e A ct agents: Foot-in-the-door attack will get you in . In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6484--6509, ...

2025
[23]

do anything now

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024 a . " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p...

2024
[24]

do anything now

Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024 b . https://doi.org/10.1145/3658644.3670388 "do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models . In Proceedings of the 2024 on ACM SIGSAC Conference on...

2024
[25]

Jingzhe Shi, Jialuo Li, Qinwei Ma, Zaiwen Yang, Huan Ma, and Lei Li. 2024. Chops: Chat with customer profile systems for customer service with llms. arXiv preprint arXiv:2404.01343

2024 arXiv
[26]

Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. 2024. https://doi.org/10.48550/ARXIV.2409.04109 Can llms generate novel research ideas? A large-scale human study with 100+ NLP researchers . CoRR, abs/2409.04109

2024 doi arXiv
[27]

Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1--8

2025
[28]

Xiongtao Sun, Deyue Zhang, Dongdong Yang, Quanchen Zou, and Hui Li. 2024. Multi-turn context jailbreak attack on large language models from first principles. arXiv preprint arXiv:2408.04686

2024 arXiv
[29]

Miles Tracy, Wayne Jansen, and Mark McLarnon. 2002. Guidelines on Securing Public Web Servers: Recommendations of the National Institute of Standards and Technology. Computer Security Division, Information Technology Laboratory, National …

2002
[30]

Rasita Vinay, Giovanni Spitale, Nikola Biller - Andorno, and Federico Germani. 2024. https://doi.org/10.48550/ARXIV.2403.03550 Emotional manipulation through prompt engineering amplifies disinformation generation in AI large language models . CoRR, abs/2403.03550

2024 doi arXiv
[31]

Rose E Wang, Ana T Ribeiro, Carly D Robinson, Susanna Loeb, and Dora Demszky. 2024. Tutor copilot: A human-ai approach for scaling real-time expertise. arXiv preprint arXiv:2410.03017

2024 arXiv
[32]

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

2025 arXiv
[33]

Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045

2024 arXiv
[34]

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR)

2023
[35]

Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of llm-based agents. arXiv preprint arXiv:2503.16416

2025 arXiv
[36]

Andy Zhou, Kai Yan, Michal Shlapentokh - Rothman, Haohan Wang, and Yu - Xiong Wang. 2024. https://openreview.net/forum?id=njwv9BsGHF Language agent tree search unifies reasoning, acting, and planning in language models . In Forty-first International Conference on Machine Learn...

2024
[37]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[38]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Sahar Abdelnabi, Kai Greshake, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. https://doi.org/10.1145/3605764.3623985 Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection . In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec 2023, Cope...

arXiv 2023

[2] [2]

Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Eric Winsor, Jerome Wynne, Yarin Gal, and Xander Davies. 2024. https://doi.org/10.48550/ARXIV.2410.09024 Agentharm: A benchmark for measuring harmfulness of LLM agents . CoRR, abs/2410.09024

arXiv doi:10.48550/arxiv.2410.09024 2024

[3] [3]

Jayachandu Bandlamudi, Ritwik Chaudhuri, Neelamadhav Gantayat, Kushal Mukherjee, Prerna Agarwal, Renuka Sindhgatta, and Sameep Mehta. 2025. A framework for testing and adapting rest apis as llm tools. arXiv preprint arXiv:2504.15546

arXiv 2025

[4] [4]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pond \' e de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bava...

arXiv 2021

[5] [5]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

arXiv 2024

[6] [6]

Neel Guha, Julian Nyarko, Daniel Ho, Christopher R \'e , Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems, 36:44123--44279

work page 2023

[7] [7]

Kung-Hsiang Huang, Akshara Prabhakar, Sidharth Dhawan, Yixin Mao, Huan Wang, Silvio Savarese, Caiming Xiong, Philippe Laban, and Chien-Sheng Wu. 2024. Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments. CoRR

work page 2024

[8] [8]

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276

arXiv 2024