REVIEW 4 major objections 4 minor 28 references
Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read By forging the model's own past utterances in the conversational history, an adversary bypasses safety mechanisms and achieves a higher attack success rate than user-turn jailbreaks on Gemini.
desk verdict Plausible and cleanly formalized attack idea, but the empirical core is missing: no ASR numbers, no baseline, no API trace, and a 'Work in progress' conclusion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the "Trojan Horse Prompting" attack, which operates on the structured Content/role format of a conversational multimodal API. The attacker replaces the normal history with $H_{\mathrm{Trojan}} = [c_1, \dots, c_{n-2}, c_{\mathrm{forged}}^{n-1}, c_{\mathrm{trigger}}^{n}]$, where the forged message has role 'model' and contains the malicious payload, and the final message is a benign user prompt. The load-bearing mechanism is the role attribute itself: the model's safety policy conditions on whether the current turn comes from the user, but does not re-verify earlier model-role messages. This is the "insecure binding" between the model's identity and its recorded history.
What would settle it
Submit a request to a conversational multimodal API with a forged model-role message containing a clearly harmful instruction followed by a benign user prompt, and check whether the API returns a refusal, an error, a sanitized continuation, or the harmful output; a refusal or an error indicating that forged model history is rejected or re-generated would disprove the central claim. A stronger test is to inspect the API's request/response logs or documentation to see whether client-supplied model-role messages are preserved verbatim in the server-side conversation state.
Extended reading notes
Core claim
The paper's central claim is that conversational models treat client-supplied conversation history, and especially messages tagged with the model's own role, as trusted ground truth. By constructing a fake history in which the model appears to have already agreed to a harmful request, an attacker can make a subsequent benign user prompt produce policy-violating content, such as an image with offensive text. The authors name the underlying cause "Asymmetric Safety Alignment": models are trained with reinforced refusals for harmful user requests, but no comparable mechanism for verifying that a 'model' role message is genuinely self-generated. The attack is demonstrated against Gemini-2.0-flash-preview-image-generation through the standard API, with a higher attack success rate than attacks that only manipulate the user turn.
Load-bearing premise
The attack assumes the Gemini API actually accepts and uses client-supplied messages tagged with the model role as if they were the model's real prior outputs, and the paper gives no request trace or live demonstration proving the service does not strip or sanitize those messages.
Editorial extensions
If this is right
- Any conversational API that lets clients pass a full history object with a model-role field inherits the attack surface, so the finding is not limited to image generation.
- Safety guarantees that assume a faithful dialogue record are void unless the API validates that model-role messages were actually produced by the model.
- Defenders must shift from input-level filtering to protocol-level checks such as server-side storage of history, digital signatures on assistant messages, or anomaly detection on role sequences.
- Alignment training should include examples where the model's past statements are false or untrusted, teaching models to verify the authenticity of their own purported outputs.
Reading between the lines
- The attack likely extends beyond Gemini to any LLM API that lets clients pass a full conversation object, including open-weight models served behind a chat wrapper; the essential condition is only that the model trusts role-typed history.
- A concrete test of the mechanism would compare attack success rate with the same payload placed in a user-role message versus a model-role message; a sharp drop in ASR would confirm that the role attribute itself, not the payload wording, is the key variable.
- The "source amnesia" analogy predicts that models with stronger provenance tracking or explicit instruction about untrusted history should resist the attack; comparing versions of the same model family with and without such training would test this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces 'Trojan Horse Prompting,' a proposed jailbreak technique for conversational multimodal models in which an attacker forges messages attributed to the model role in the API-supplied conversational history, followed by a benign user prompt that triggers harmful content generation. The authors claim that this technique achieves a significantly higher Attack Success Rate (ASR) than established user-turn jailbreaking methods on Gemini-2.0-flash-preview-image-generation, and they attribute the vulnerability to an 'Asymmetric Safety Alignment' hypothesis: safety training makes models skeptical of user-role inputs but trusting of model-role history. The paper provides a taxonomy of related attack classes, a formal description of the attack on the Gemini API, and a short, qualitative discussion of the hypothesized mechanism. Section 5 is titled 'Experimental Evaluation' but contains no quantitative results, and Section 6 is explicitly marked 'Work in progress.'
Significance. The paper identifies a potentially important attack surface: if a production API accepts client-supplied conversational history containing forged model-role messages and treats those messages as ground truth, then conversation-history integrity becomes a critical security property. The proposed attack shifts the locus of manipulation from the user role to the model role, which is a conceptually interesting direction, and the paper's taxonomy helpfully situates the idea among prior work. However, the central empirical claim—that the attack 'achieves a significantly higher ASR'—is not supported by any reported measurement. There is no API trace demonstrating that Gemini accepts forged model-role messages as intended, no ASR value, no baseline comparison, no trial count, and no error analysis. The Asymmetric Safety Alignment hypothesis is plausible but is presented without independent evidence or falsifiable predictions. If substantiated with rigorous experiments, the finding would be significant; as submitted, the paper is an incomplete research proposal rather than a completed empirical study.
major comments (4)
- [§5.2] The paper's central claim, stated in the Abstract and in Contribution 3, is that Trojan Horse Prompting 'achieves a significantly higher Attack Success Rate (ASR) compared to established user-turn jailbreaking methods.' Section 5.2, titled 'Main Results: Trojan Horse Prompting Efficacy,' reports no ASR value, no sample size, no number of trials, no baseline comparison, no benchmark details, and no automated or manual evaluation counts. The only content is a figure with selected image outputs. Without quantitative measurements, the paper provides no empirical support for its headline claim. The manuscript itself acknowledges incompleteness in Section 6, which is literally 'Work in progress.' This is a load-bearing omission: the contribution is an empirical claim, and the paper contains no data.
- [§3.1] The attack's feasibility rests on the unverified premise that the Gemini API accepts a client-supplied conversational history containing forged Content objects with role='model' and treats them as the model's own prior outputs. Section 3.1 asserts that the attacker 'can construct and submit the entire conversational history payload,' and Section 3.2 formalizes this, but the paper provides no API specification excerpt, no raw request/response trace, no code, and no live demonstration. If the API rejects or sanitizes client-supplied model-role messages, the attack fails regardless of the proposed alignment asymmetry. This premise must be demonstrated before any further claims about the attack's efficacy can be evaluated.
- [§4] The Asymmetric Safety Alignment hypothesis is introduced in Section 4 as the explanation for the attack's effectiveness, yet no independent evidence is provided. The only evidence offered is the attack itself, which makes the reasoning circular. The paper does not report a behavioral experiment comparing model responses to forged model-role content versus equivalent user-role content, does not state a falsifiable prediction, and does not test whether the attack succeeds only when the malicious content is in the model role. Without such a test, the hypothesis is an assertion rather than a validated mechanism, and it cannot support the abstract's claim that the vulnerability 'stems from' this asymmetry.
- [§5.1] The experimental setup is described only in broad terms. There is no specification of the harmful prompt benchmark, no list of categories with actual prompt examples, no evaluation protocol, and no definition of how ASR was computed or how human evaluation was conducted. Section 5.1 mentions 'automated content analysis and manual human evaluation' but provides no details on either. This makes it impossible to assess the validity of any claimed ASR, even if numbers were reported.
minor comments (4)
- [§5.1] There is a typo: 'smaples' should be 'samples' in Section 5.2, and 'jailbraking' should be 'jailbreaking' in the figure caption.
- [References] Reference [28] (AdvBench) is listed in the bibliography but not cited in the text, and references [15] and [18] share the same arXiv identifier (2503.17987) despite describing different attacks; these need to be checked and corrected.
- [§3.2] The formalism uses H_standard and H_Trojan with an index n, but the relationship between the indices in the notation and the example in Figure 1 is not explained; clarifying the exact payload structure would improve reproducibility.
- [§2.3.3] The discussion of Agent Smith and Agent-in-the-Middle is useful, but the claim that Trojan Horse Prompting targets 'the fundamental, universal user-model dialogue structure' is stated rather than supported by evidence about other APIs; this should be framed as a hypothesis or tested on multiple systems.
Circularity Check
No circular derivation found; the paper's core ASR claim is empirically unsupported but not circular.
full rationale
The paper's derivation chain does not reduce to its own inputs. The attack in §3.2 is a constructive definition of a payload structure (forging role='model' Content objects), not a quantity fitted or predicted from data. The Asymmetric Safety Alignment hypothesis in §4 is a proposed mechanism: it asserts that alignment trains refusal mainly for user-role prompts and does not teach skepticism of model-role history. That hypothesis is not derived from a fitted parameter, nor is it defined in terms of the attack's success; it is a general claim that could in principle be tested independently (for example, by comparing model-role forgeries against user-role baselines in controlled settings). The paper's actual weakness is evidentiary: §5.2 reports no measured Attack Success Rate, no baseline comparison, and no sample counts, and §6 is explicitly 'Work in progress,' so the abstract's 'significantly higher ASR' claim is unverified. But missing empirical support is a completeness/correctness problem, not circular reasoning. There are no load-bearing self-citations by the authors, no imported uniqueness theorems, and no fitted inputs renamed as predictions. Therefore, the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Gemini API accepts client-supplied role='model' messages in the conversational history without integrity verification.
- ad hoc to paper Safety alignment (SFT/RLHF) creates a policy that is skeptical of user-role content but trusts model-role content.
- domain assumption The model treats the provided history H as an immutable ground-truth record.
invented entities (1)
-
Asymmetric Safety Alignment
Cite this review
Pith. "Pith review of Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message." pith.science (2026). https://pith.science/paper/3CIDEMCT
@misc{pith2026250704673,
author = {Pith},
title = {Pith review of: Trojan Horse Prompting: Jailbreaking Conversational Multimodal Models by Forging Assistant Message},
year = {2026},
howpublished = {\url{https://pith.science/paper/3CIDEMCT}},
note = {Machine review of arXiv:2507.04673}
}
read the original abstract
The rise of conversational interfaces has greatly enhanced LLM usability by leveraging dialogue history for sophisticated reasoning. However, this reliance introduces an unexplored attack surface. This paper introduces Trojan Horse Prompting, a novel jailbreak technique. Adversaries bypass safety mechanisms by forging the model's own past utterances within the conversational history provided to its API. A malicious payload is injected into a model-attributed message, followed by a benign user prompt to trigger harmful content generation. This vulnerability stems from Asymmetric Safety Alignment: models are extensively trained to refuse harmful user requests but lack comparable skepticism towards their own purported conversational history. This implicit trust in its "past" creates a high-impact vulnerability. Experimental validation on Google's Gemini-2.0-flash-preview-image-generation shows Trojan Horse Prompting achieves a significantly higher Attack Success Rate (ASR) than established user-turn jailbreaking methods. These findings reveal a fundamental flaw in modern conversational AI security, necessitating a paradigm shift from input-level filtering to robust, protocol-level validation of conversational context integrity.
Figures
Reference graph
Works this paper leans on
-
[1]
Tao, X., Zhong, S., Li, L., Liu, Q., & Kong, L. (2024). ImgTrojan: Jailbreaking Vision-Language Models with ONE Image. arXiv preprint arXiv:2403.02910. 6
arXiv 2024
- [2]
-
[3]
Dong, Y ., Li, Z., Meng, X., Yu, N., & Guo, S. (2024). Jailbreaking Text-to-Image Models with LLM-Based Agents. arXiv preprint arXiv:2408.00523
arXiv 2024
- [4]
-
[5]
Zheng, Y ., et al. (2025). Reasoning-Augmented Con- versation for Multi-Turn Jailbreak Attacks on Large Language Models. arXiv preprint arXiv:2502.11054
arXiv 2025
-
[6]
Perez, E., Ringer, S., Lukoˇsi¯ut˙e, K., Nguyen, K., Chen, E., Heiner, S.,... & Kaplan, J. (2022).Red teaming lan- guage models with language models . arXiv preprint arXiv:2202.03286
arXiv 2022
-
[7]
Ganguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y ., Kadavath, S.,... & Amodei, D. (2022). Red team- ing language models to forget they’re language mod- els. arXiv preprint arXiv:2209.07858
arXiv 2022
-
[8]
Zou, A., Wang, Z., Kolter, J. Z., & Fredrikson, M. (2023). Universal and transferable adversarial attacks on aligned language models . arXiv preprint arXiv:2307.15043
arXiv 2023
Show all 28 references
-
[9]
Shahariar, S., et al. (2024). Adversarial Attacks on Parts of Speech: An Empirical Study in Text-to-Image Generation. Findings of the Association for Computa- tional Linguistics: EMNLP 2024
2024
-
[10]
Radharapu, B., & Krishna, H. (2023). Taxonomy of Ad- versarial Attacks on Text-to-Image Generative Models. 2023 IEEE International Conference on Omni-layer Intelligent Systems (COINS)
2023
-
[11]
Liu, T., Lai, Z., Zhang, G., Torr, P., Demberg, V ., Tresp, V ., & Gu, J. (2024). Multimodal prag- matic jailbreak on text-to-image models. arXiv preprint arXiv:2409.19149
2024 arXiv
-
[12]
& Kaplan, J
Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N.,... & Kaplan, J. (2022). Train- ing a helpful and harmless assistant with reinforce- ment learning from human feedback . arXiv preprint arXiv:2204.05862
2022 arXiv
-
[13]
F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30
2017
-
[14]
Wei, A., Haghtalab, N., & Steinhardt, J. (2023). Jail- broken: How does llm safety training fail? . arXiv preprint arXiv:2307.02483
2023 arXiv
-
[16]
Jiang, F., et al. (2024). ArtPrompt: ASCII Art- based Jailbreak Attacks against Aligned LLMs . arXiv preprint arXiv:2402.11753
2024 arXiv
-
[17]
Wang, W., Gao, K., Jia, Z., Yuan, Y ., Huang, J., Liu, Q.,... & Tu, Z. (2024). Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step . arXiv preprint arXiv:2410.03869
2024 arXiv
-
[18]
Zhang, C., et al. (2025). Reason2Attack: Jailbreak- ing Text-to-Image Models via LLM Reasoning . arXiv preprint arXiv:2503.17987
2025
-
[19]
Wang, Z., et al. (2025). GenBreak: Red Teaming Text- to-Image Generators Using Large Language Models . arXiv preprint arXiv:2506.10047
2025 arXiv
-
[20]
Zhao, S., et al. (2025). Jailbreaking Multimodal Large Language Models via Shuffle Inconsistency . arXiv preprint arXiv:2501.04931
2025 arXiv
-
[21]
Gao, S., et al. (2024). HTS-Attack: Heuristic Token Search for Jailbreaking Text-to-Image Models . arXiv preprint arXiv:2408.13896
2024 arXiv
-
[22]
Jiang, Y ., et al. (2023). SneakyPrompt: Jailbreaking Text-to-Image Generative Models with A Stealthy At- tack. arXiv preprint arXiv:2305.12082
2023 arXiv
-
[23]
Wang, Z., et al. (2025). JailBound: Eliciting Latent Safety-Risks in VLMs by Breaking the Decision Bound- ary. arXiv preprint arXiv:2505.19610
2025
-
[24]
Rando, J., Korevaar, H., Brinkman, E., Evtimov, I., & Tram `er, F. (2024). Gradient-based Jailbreak Im- ages for Multimodal Fusion Models . arXiv preprint arXiv:2410.03489
2024 arXiv
-
[25]
& Lin, M
Gu, X., Zheng, X., Pang, T., Du, C., Liu, Q., Wang, Y .,... & Lin, M. (2024). Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fast. arXiv preprint arXiv:2402.08567
2024 arXiv
-
[26]
Ju, D., et al. (2025). Agent-in-the-Middle: Communi- cation Attack on Large Language Model-based Multi- Agent Systems. arXiv preprint arXiv:2502.14847
2025 arXiv
-
[27]
OpenAI. (2023). GPT-4 Technical Report . arXiv preprint arXiv:2303.08774
2023 arXiv
-
[28]
Zhu, Y ., et al. (2024). AdvBench: A compre- hensive benchmark for adversarial robustness of multimodal large language models . arXiv preprint arXiv:2307.14333
2024 arXiv
-
[29]
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthe- sis with latent diffusion models . In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695). 7
2022
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.