Pith. sign in

REVIEW 4 major objections 5 minor 20 references

A malicious client can hijack collaborative prompt optimization by injecting instructions that survive aggregation and defeat current defenses.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:51 UTC pith:PVB4G5FD

load-bearing objection Real new attack surface, broad honest evaluation, but the headline claim overreaches and the attack's effectiveness is configuration-sensitive—still worth a careful peer review. the 4 major comments →

arxiv 2607.18622 v2 pith:PVB4G5FD submitted 2026-07-21 cs.CR cs.AI

CPInj: Uncovering Prompt Injection Risks in Textual Collaborative Prompt Optimization

Xinting Liao , Behnoosh Zamanlooy , Masoumeh Shafieinejad , David B. Emerson , Ruinan Jin , Deval Pandya , Xiaoxiao Li This is my paper
classification cs.CR cs.AI
keywords prompt injectioncollaborative prompt optimizationtextual gradientsfederated prompt learningLLM securityadversarial prompt optimizationprompt aggregation defensesmulti-round attacks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that textual collaborative prompt optimization—a scheme where multiple clients improve a shared text prompt locally and a server aggregates the updates—is a viable prompt-injection surface. The proposed attack, CPInj, rewrites a malicious client's local prompt so that the injected instruction survives server aggregation, persists through later benign optimization rounds, and redirects the final global prompt toward an adversarial objective. On five reasoning benchmarks spanning math, logic, and medicine, across three model families, the attack lowers task accuracy and raises attack success rates, while four existing detection-based defenses fail to consistently restore clean performance. The paper also offers a first defense, APAgg, an aggregation-time purification that reduces attack success but does not fully solve the problem. If the claim holds, collaborative text-prompt optimization carries an overlooked vulnerability that current single-turn defenses do not cover.

Core claim

On the paper's own terms, the central discovery is that the aggregation step in textual collaborative prompt optimization is an attack surface: a malicious client can optimize its textual update against six objectives—maliciousness, persistence, stealth, similarity, distinctness, and redirection—so that the resulting prompt is aggregated into the global prompt, shifts subsequent benign responses away from the original task objective, and resists later benign refinement. The formal attack goal is to minimize expected task utility on test inputs, and the empirical claim is that CPInj achieves this across GPT-4o, Gemini-2.5-Pro, and Qwen3-32B on AIME, AoPS, LiveBench-M, FOLIO, and PubMedQA, bea

What carries the argument

The engine of the attack is CPInj, a multi-round textual rewrite loop that treats the adversarial prompt as the variable and optimizes it with six text-based losses: a behavior loss that rewards inducing exactly one subtle reasoning error, a redirect loss that installs a silent 'latest instruction overrides earlier ones' rule, a priority loss that keeps the injection dominant under rewriting and summarization, a similarity loss that styles the payload like the global prompt, a stealth loss that runs a surrogate detector and asks for de-suspicion, and a distinctness loss that forces structural rewrite rather than simple appending. The losses are combined by weighted repetition and applied thr

Load-bearing premise

The demonstration of a general attack surface rests on a manually crafted reference instruction and a hand-picked six-loss weighting; if those choices require per-task retuning to work, the threat is a tuned demonstration rather than a general vulnerability.

What would settle it

Run CPInj on a TCPO task outside the five evaluated benchmarks using the paper's default reference prompt and default loss weights, and check whether accuracy drop and attack success rate reproduce without per-task tuning; failure to reproduce would falsify the claimed generality.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The collaborative optimization loop itself is an injection channel: even a single malicious client among fifteen measurably shifts accuracy on PubMedQA.
  • Filtering client prompts with current single-turn detectors does not restore clean performance, and some filters lower accuracy below the undefended attack setting.
  • CPInj's failures concentrate in core reasoning outputs such as proof-style and numeric answers, meaning the damage is not merely surface-level formatting.
  • The same framework transfers to a different behavioral objective, such as inducing biased abstention on a target subpopulation, while preserving the required answer format.
  • When the attacker is allowed to optimize against APAgg's purification, APAgg's protection weakens, so the proposed defense is explicitly an initial, non-adaptive baseline rather than a final solution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this attack surface generalizes, any protocol that aggregates free-form natural-language updates from multiple parties—not just TCPO—inherits a similar injection risk, including multi-agent LLM pipelines.
  • The authors' adaptive-attack result implies that defenses for this setting must be evaluated against attackers who know the defense, not only against static threat models.
  • A testable extension is to run CPInj in a deployment with heterogeneous client tasks and aggregation-by-summarization to see whether the malicious instruction still dominates when benign updates strongly diverge.
  • Because the attack uses a surrogate detector during optimization, defense designers may need to exploit cross-round consistency rather than single-prompt features to catch persistent injections.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper identifies a prompt-injection surface in textual collaborative prompt optimization (TCPO), where one malicious client can submit adversarially rewritten local prompts that survive server aggregation, persist across later benign optimization rounds, and degrade the final global prompt. The proposed attack, CPInj, combines six textual losses targeting maliciousness, persistence, and stealth, and is evaluated against CatAttack, QueryGCG, PAIR, and TAP across three model families and five benchmarks. The paper further proposes a defense, APAgg, that filters and then purifies client prompts before aggregation, and reports that existing single-turn defenses fail to restore clean-task accuracy under CPInj. The central claim is that the TCPO aggregation loop is a viable and previously unexplored prompt-injection surface.

Significance. If the central claim is established, the paper opens a genuinely new attack surface: prior prompt-injection work focuses on single-turn interactions, while prior federated poisoning focuses on parametric or continuous-gradient space. The evaluation is unusually broad for an attack paper—three LLM families, five benchmarks, multiple aggregation methods, and several defense baselines—and the appendix is commendably honest in reporting adaptive attacks against APAgg and settings where APAgg does not help. The paper also ships detailed prompt templates and algorithms, which aids reproducibility. However, the current evidence does not yet establish CPInj as a general attack on TCPO: the method depends on manually authored reference prompts and non-uniform loss weights, and the empirical claims rest on unseeded point estimates without confidence intervals or significance tests. These issues are load-bearing for the paper's central generalization claim.

major comments (4)
  1. [§3.2, §4.1, Table 2c, Appendix A.3] The generality of CPInj is not established because the attack configuration is hand-selected. Section 4.1 fixes λ=(1,1,2,1,1,3) in Eq. (16) and the attack is seeded with a manually written reference prompt p_r in Eq. (6); Appendix A.3 discloses only a math-specific injected instruction and the implicit priority cue. Table 2c shows that this choice is decisive: uniform weights reduce PubMedQA ASR from 31.43 to 11.43, while behavior-heavy and persistence-heavy variants raise ASR to 74.29 and 91.43 only at the cost of accuracy collapsing to 29 and 14. No procedure is given for deriving p_r or λ from the task description, the aggregation operator, or the target model. Since the paper claims CPInj as a broadly effective attack surface, the authors need either a principled way to set these quantities or an evaluation showing that the attack remains effective across a range of reasonable config
  2. [§4.2, Table 1, Appendix A.5] The paper reports only single unseeded point estimates for every accuracy and ASR value, with no error bars, multiple seeds, or significance tests. Given the attack temperature 0.7 and the stochasticity of LLM-based prompt rewriting, the differences between many conditions are well within plausible run-to-run noise. This is especially problematic for the ASR metric in Eq. (21), whose denominator is only the number of zero-shot-correct examples; when attacked accuracy is low, ASR is computed on a small and possibly unstable subset. For example, in Table 1 the GPT-4o FOLIO comparison between CPInj (51/41.43) and DataSentinel (51/35.71) differs only in ASR with no uncertainty estimate. The authors should report at least 3–5 independent seeds with means and standard deviations (or confidence intervals) for the main tables and figures, and ideally a paired significance test for the central ac
  3. [§1, §3.3, Table 1, Table 7] The defense claims for APAgg are stronger than the reported results. The contribution section states that APAgg “reduces attack success rates while preserving clean task accuracy,” but Table 1 contains counterexamples: on Qwen3-32B FOLIO, APAgg increases ASR from 23.94 to 30.88 while dropping accuracy from 71 to 57; on GPT-4o AoPS, APAgg ASR is 42.86 versus 23.81 for the undefended CPInj attack. The abstract and conclusion are more measured, but the contribution bullet is not. The authors should either qualify the defense claim to the settings where it holds or analyze why APAgg is not effective on these cells. Appendix Table 7 further shows that a simple ‘survive purification’ adaptive loss degrades APAgg (e.g., PubMedQA ASR rises from 1.64 to 21.31); this is honestly reported, but it means the defense is being presented as robust without an adaptive evaluation in the main body.
  4. [§4.2, Figure 3, Table 10] The generalization of the attack across aggregation methods and malicious-client ratios is only partially supported. Figure 3 shows CPInj remains effective on PubMedQA under concatenation, summarization, and SumUID, but these are all on GPT-4o-mini, a smaller model, and only with the same hand-tuned default configuration. Table 10 similarly evaluates APAgg across attack families, but the main attack-effectiveness claim across aggregation methods is not tested on the three primary backbones of Table 1. Given that the paper’s central novelty is the aggregation loop as an attack surface, the authors should either extend the aggregation-method experiments to at least one additional backbone or explicitly bound the claim to the tested settings.
minor comments (5)
  1. [§4.2, Figure 3] The text says “In Fig. 3b, we extend the aggregation to concatenation and summarization,” but aggregation results are shown in panel (c) of Figure 3; panel (b) plots ASR versus malicious ratio. Please correct the cross-reference.
  2. [§3.2, Eq. (10)] Equation (10) has a formatting error: “TextLoss(...)(pt) adv” should read “TextLoss(T_beh(C_beh^(t)))(p_adv^(t))” or similar. Please fix the superscript/argument.
  3. [Table 3] Table 3 introduces CatAttack* and QueryGCG* as “CPInj-initialized variants,” but the initialization procedure is not defined in the main text or in the table caption. Please clarify whether the only change is the seed prompt p_r or also the loss weighting.
  4. [Appendix A.3] The stealth detector prompt returns SAFE, CONFIDENCE, and REASON, but the stealth refinement prompt refers to a “STEALTH SCORE.” Please make the fields consistent.
  5. [§4.1] The phrase “attack temperature of 0.7” is used, but the paper does not specify whether this is the temperature for the rewriting LLM, the inference LLM, or both. Please clarify in the implementation details.

Circularity Check

0 steps flagged

No circular derivation: CPInj is an empirical attack/defense evaluation whose hand-tuned objectives are construction choices, not predictions derived from their own outputs.

full rationale

CPInj's load-bearing claim is empirical: malicious local prompts optimized through Eq. (16) degrade task accuracy and raise ASR (Eq. 21) relative to external baselines such as CatAttack, GCG, PAIR, and TAP. The attack objective (Eqs. 8-16) explicitly optimizes for low-accuracy behavior, instruction priority, and stealth; reporting that the resulting prompts produce low accuracy and high ASR is a measurement of the optimized object, not a prediction that is definitionally equal to the objective. ASR is computed on test examples as the fraction of zero-shot-correct examples flipped to incorrect, while the behavior loss evaluates a single attacker-owned sample; these are not the same quantity by construction. Defenses (PromptGuard, DataSentinel, AttentionTracker, LLM-based detector) are external systems; the in-paper APAgg defense is tested against the authors' own attack, but the appendix discloses an APAgg-aware adaptive loss and shows that APAgg weakens under it, and the paper explicitly frames APAgg as an initial non-adaptive defense baseline rather than a robust solution. The hand-crafted reference prompt p_r (Eq. 6) and fixed loss weights lambda=(1,1,2,1,1,3) are attack-instance choices; Table 2c shows sensitivity to them. That is a generalization/transferability limitation, not circularity. Self-citations to FedTextGrad (Chen et al., 2025a) and FOCoOp are framework/related-work references to peer-reviewed work, and are not invoked as unverified constraints that force the paper's conclusions. I therefore find no circular step that reduces a claimed result to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim borrows substantial machinery from prior work (TextGrad) and an explicit threat model. The only numbers the authors choose are loss weights, rewrite steps, attack temperature, and the manual malicious seed; no new physical or conceptual entities are introduced. The manual seed and weight choices are the main free parameters that shape the empirical results.

free parameters (4)
  • loss weights lambda=(lambda_sim, lambda_pri, lambda_beh, lambda_dis, lambda_stl, lambda_redir) = (1, 1, 2, 1, 1, 3)
    Default scalarization in Eq. 16 and Section 4.1. Sensitivity analysis in Table 2c shows ASR swings from 11.43 to 91.43 across weight configurations, so the headline attack strength is not parameter-free.
  • adversarial rewrite steps R = 2 (default)
    Algorithm 1 and Section 4.1. Table 6 shows R=3 jumps PubMedQA ASR to 84.51, so the default budget materially affects reported attack success.
  • hand-authored reference instruction p_r = the 'mathematical journey / one subtle mistake' seed in Appendix A.3
    Eq. 6 initializes every malicious prompt from a human-written malicious instruction. The attack does not discover the malicious behavior from the task description alone; it optimizes from a manually specified seed.
  • attack temperature = 0.7
    Section 4.1 sets decoding temperature for adversarial rewrites; no evidence that results are insensitive to this choice.
axioms (4)
  • domain assumption Textual 'gradients' from Eq. (2) are treated as valid optimization signals for prompt improvement.
    Both benign TCPO and CPInj rely on LLM feedback driving the prompt toward the objective. This heuristic is borrowed from TextGrad and is not mathematically proven.
  • domain assumption The attacker controls m/K clients and has the initial task prompt, LLM APIs, and a labeled query-answer sample.
    Section 2.2.1 specifies these capabilities. If a deployment denies any of them, e.g., through authenticated client updates or no access to task labels, the attack fails under the stated threat model.
  • ad hoc to paper Weighted combination of six TextLoss objectives by repeating losses according to rounded weights (Eq. 16) is a sound scalarization in text space.
    The paper realizes the weighted sum by duplicating loss terms, but provides no evidence that this repetition preserves the intended multi-objective trade-off across tasks.
  • domain assumption Server-side defenses are modeled as text filters before aggregation and do not verify text provenance.
    Section 2.2.2 and Algorithm 3 use detection and purification on text. A server that cryptographically authenticates client prompts or rejects updates with unknown provenance would invalidate the attack, but such a defense is outside the paper's scope.

pith-pipeline@v1.3.0-alltime-deepseek · 20077 in / 14517 out tokens · 155608 ms · 2026-08-03T01:51:11.674420+00:00 · methodology

0 comments
read the original abstract

Textual Collaborative Prompt Optimization (TCPO) extends TextGrad (Yuksekgonul et al., 2025) to a decentralized setting by allowing multiple clients to jointly improve prompts for large language models (LLMs) while keeping their data locally. Its reliance on free-form textual updating and aggregation introduces a new and largely unexplored attack surface, i.e., malicious instructions can be injected into local prompts and propagated through server-side prompt aggregation. Unlike conventional prompt injection attacks, attacking TCPO targets the collaborative optimization loop in TCPO. This setting is more challenging because malicious instructions must survive aggregation, persist through subsequent benign prompt optimization, and evade server-side defenses. To expose this risk, we propose Collaborative Prompt Injection (CPInj) attack that contaminates the aggregated global prompt with malicious instructions, degrades downstream task performance, resists purification by prompt optimization on benign clients, and evades advanced detection-based defenses on the server. We find that current defense methods are ineffective against CPInj. We further propose Anchored Purification Aggregation (APAgg), a defense-oriented aggregation that purifies malicious instructions without severely degrading TCPO utility. We conduct extensive experiments across three LLM families and five reasoning tasks in math, logic, and medicine, and demonstrate that our proposed attack reveals a critical vulnerability in TCPO. Although we take a first step toward mitigation, the attack remains highly effective and far from fully resolved, calling for more robust defense for TCPO.

Figures

Figures reproduced from arXiv: 2607.18622 by Behnoosh Zamanlooy, David B. Emerson, Deval Pandya, Masoumeh Shafieinejad, Ruinan Jin, Xiaoxiao Li, Xinting Liao.

Figure 1
Figure 1. Figure 1: PubMedQA performance under four FedTextGrad settings: vanilla, prompt injec [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: General TextLoss template. Specific loss templates are provided in the appendix. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: GPT-4o-mini performance under different malicious-client ratios on PubMedQA [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The performance analysis on attack. cases, PromptGuard, DataSentinel, AttTracker, and the LLM-Based detector improve over the undefended attack setting, yet they still fail to consistently recover performance to the level of vanilla FedTextGrad. This suggests that simply filtering suspicious client prompts is insufficient for fully eliminating malicious influence once poisoned instructions have been woven … view at source ↗
Figure 5
Figure 5. Figure 5: GPT-4o-mini performance under different malicious-client ratios on PubMedQA [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Server-side defense and aggregation. The server filters suspicious prompt up [PITH_FULL_IMAGE:figures/full_fig_p022_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Benign client local optimization. The client evaluates the current prompt on local [PITH_FULL_IMAGE:figures/full_fig_p023_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Server-side defense and aggregation of APAgg. The server filters suspicious prompt updates, preprocesses the remaining prompts, aggregates them, and optionally performs task-anchored purification before broadcasting the next-round global prompt [PITH_FULL_IMAGE:figures/full_fig_p023_9.png] view at source ↗
Figure 7
Figure 7. Figure 7: Overview of the CPInjapproach. The method leverages prompt optimization via text-based gradients to iteratively rewrite malicious prompts under predefined objectives. By jointly optimizing across multiple evaluation dimensions, it (i) enhances malicious effectiveness through subtle task shifts, (ii) maintains persistence via priority signals, and (iii) improves stealth by aligning with global prompt contex… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 9 linked inside Pith

  1. [4]

    Query- based adversarial prompt generation

    11 Published as a conference paper at COLM 2026 Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tram`er, and Milad Nasr. Query- based adversarial prompt generation. InAdvances in Neural Information Processing Systems, volume 37, pp. 128260–128279. Curran Associates, Inc.,

    2026

  2. [5]

    Defending against indirect prompt injection attacks with spotlighting.arXiv preprint arXiv:2403.14720,

    Keegan Hines, Gary Lopez, Matthew Hall, Federico Zarfati, Yonatan Zunger, and Emre Kiciman. Defending against indirect prompt injection attacks with spotlighting.arXiv preprint arXiv:2403.14720,

    Pith/arXiv arXiv

  3. [6]

    Attention tracker: Detecting prompt injection attacks in llms

    Kuo-Han Hung, Ching-Yun Ko, Ambrish Rawat, I-Hsin Chung, Winston H Hsu, and Pin-Yu Chen. Attention tracker: Detecting prompt injection attacks in llms. InFindings of the Association for Computational Linguistics: NAACL 2025, pp. 2309–2322,

    2025

  4. [8]

    Baseline defenses for adversarial attacks against aligned language models.arXiv preprint arXiv:2309.00614,

    Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping- yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline defenses for adversarial attacks against aligned language models.arXiv preprint arXiv:2309.00614,

    Pith/arXiv arXiv

  5. [9]

    Pubmedqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp. 2567–2577,

    2019

  6. [10]

    Instruction boundary: Quantifying biases in llm reasoning under various coverage.arXiv preprint arXiv:2509.20278,

    Zipeng Ling, Yuehao Tang, Chen Huang, Shuliang Liu, Gaoyang Jiang, Shenghong Fu, Junqi Yang, Yao Wan, Jiawan Zhang, Kejia Huang, et al. Instruction boundary: Quantifying biases in llm reasoning under various coverage.arXiv preprint arXiv:2509.20278,

    arXiv

  7. [11]

    Tree of attacks: Jailbreaking black-box llms automatically

    12 Published as a conference paper at COLM 2026 Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. InAdvances in Neural Information Processing Systems, volume 37,

    2026

  8. [12]

    Official model card, ac- cessed 2026-03-30. Inkit Padhi, Manish Nagireddy, Giandomenico Cornacchia, Subhajit Chaudhury, Tejaswini Pedapati, Pierre Dognin, Keerthiram Murugesan, Erik Miehling, Mart´ın Santill´an Cooper, Kieran Fraser, et al. Granite guardian.arXiv preprint arXiv:2412.07724,

    Pith/arXiv arXiv 2026

  9. [13]

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ: A hand-built bias bench- mark for question answering. InFindings of the Association for Computational Linguistics: ACL 2022, pp. 2086–2105, Dublin, Ireland, May

    2022

  10. [15]

    Meghana Rajeev, Rajkumar Ramamurthy, Prapti Trivedi, Vikas Yadav, Oluwanifemi Bamg- bose, Sathwik Tejaswi Madhusudan, James Zou, and Nazneen Rajani

    URLhttps://openreview.net/forum?id=gKfj7Jb1kj. Meghana Rajeev, Rajkumar Ramamurthy, Prapti Trivedi, Vikas Yadav, Oluwanifemi Bamg- bose, Sathwik Tejaswi Madhusudan, James Zou, and Nazneen Rajani. Cats confuse reasoning llm: Query agnostic adversarial triggers for reasoning models.arXiv preprint arXiv:2503.01781,

    Pith/arXiv arXiv

  11. [16]

    Livebench: A challenging, contamination-free llm benchmark.arXiv preprint arXiv:2406.19314, 4:2,

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench: A challenging, contamination-free llm benchmark.arXiv preprint arXiv:2406.19314, 4:2,

    Pith/arXiv arXiv

  12. [17]

    Black-box optimization of llm outputs by asking for directions.arXiv preprint arXiv:2510.16794,

    Jie Zhang, Meng Ding, Yang Liu, Jue Hong, and Florian Tram`er. Black-box optimization of llm outputs by asking for directions.arXiv preprint arXiv:2510.16794,

    arXiv

  13. [18]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043,

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043,

    Pith/arXiv arXiv

  14. [19]

    ,L(r) u o nu ! ,n u =max (1, round(λu))

    13 Published as a conference paper at COLM 2026 Algorithm 1CPInj: Attack on Malicious Client Require: Local prompt pl,(t) , reference prompt pr,(t), trigger τ(t), feedback batch B, rewrite stepsR Ensure:Malicious prompt ˜p l,(t) 1:Initialize attack seed: 2:p (0) adv ←p r,(t) ⊕p l,(t) 3:forr=1 toRdo 4:Build TextLoss templates and contexts: 5:{T u,C (r) u }...

    2026

  15. [20]

    CPInj achieves the highest Target ASR while maintaining perfect format compliance

    The attacker aims to make the model choose the unknown option on target-subpopulation answer cases while preserving normal behavior on non-target cases and maintaining the required A/B/C answering format. CPInj achieves the highest Target ASR while maintaining perfect format compliance. These results suggest that CPInj can extend to another prompt-level b...

    2023

  16. [2022]

    doi: 10.18653/v1/2022.findings-acl.165

    Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.165. URL https://aclanthology.org/ 2022.findings-acl.165/. Anselm Paulus, Arman Zharmagambetov, Chuan Guo, Brandon Amos, and Yuandong Tian. Advprompter: Fast adaptive adversarial prompting for llms.arXiv preprint arXiv:2404.16873,

    Pith/arXiv arXiv 2022

  17. [2023]

    Can textual gradient work in federated learning? InThe Thirteenth International Conference on Learning Representations, 2025a

    Minghui Chen, Ruinan Jin, Wenlong Deng, Yuanyuan Chen, Zhi Huang, Han Yu, and Xi- aoxiao Li. Can textual gradient work in federated learning? InThe Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id= Cy5IKvYbR3. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. {StruQ}: Defending against ...

    2025

  18. [2024]

    Folio: Natural language reasoning with first-order logic

    Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, et al. Folio: Natural language reasoning with first-order logic. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22017–22031,

    2024

  19. [2025]

    Llama guard: Llm- based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674,

    Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm- based input-output safeguard for human-ai conversations.arXiv preprint arXiv:2312.06674,

    Pith/arXiv arXiv

  20. [2026]

    Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J

    URL https://openreview.net/forum?id=RQm2KQTM5r. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries.arXiv preprint arXiv:2310.08419,

    Pith/arXiv arXiv