REVIEW 3 major objections 5 minor 40 references
CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggers
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Switching languages mid-paragraph can act as a nearly perfect LLM backdoor trigger.
desk verdict Genuinely new cross-lingual paragraph-level backdoor trigger with well-designed ablations, but the universality/stealthiness claims need a false-trigger test on natural code-mixed inputs before they hold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the cross-lingual structure trigger: a paragraph-level trigger defined by a fixed sequence of K languages (K = 2 or 3), applied by splitting a sample into segments and translating each segment into the language assigned by that sequence. Its work is to give the model a language-order pattern it can learn without any token-level anomaly; the paper's discussion shows the learned association is with the sequence itself, because changing the translated text leaves the attack at full strength while changing or reordering the languages destroys it.
What would settle it
Take a poisoned model and run it on a test set of naturally occurring code-mixed texts in which the language order follows the trigger sequence (for example, a sentence in Chinese, then English, then German from real user forums); if a substantial fraction of these unpoisoned inputs produce the target label or target content, then the trigger fires on clean data and the claim that it is stealthy and semantically inert is falsified. A cheaper version is to compute the fraction of samples in a public code-mixed corpus that match the trigger order and report the resulting ASR on that subset.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a fixed, multi-language paragraph structure functions as a universal backdoor trigger: when the model sees a text whose segments follow the trained language sequence (for example ZH-EN-DE), it outputs the target label or target content, while monolingual or differently ordered texts leave normal behavior intact. The attack achieves nearly 100% attack success rate at a 5% poisoning rate and above 90% at 3% on the harder QA task, across Llama-3-8B-Instruct, Qwen2-7B-Instruct, and Qwen2-1.5B-Instruct. The trigger is language-specific and structure-specific: replacing one language or swapping or removing a language drops ASR to zero, whereas re-translating the same segments with a different translation model leaves ASR at 100%. Compared with fixed-token and style triggers, the poisoned samples show lower perplexity and higher text similarity to clean samples, indicating less semantic shift.
Load-bearing premise
The attack's stealthiness rests on the assumption that normal, non-poisoned usage is effectively monolingual, so a fixed sequence of languages like Chinese-English-German is rare enough never to fire on clean inputs; the paper does not measure false triggers on naturally multilingual or code-mixed text.
Editorial extensions
If this is right
- At poisoning rates around 3 to 5 percent, CL-Attack reaches close to 100 percent attack success on sentiment classification, rating prediction, and extractive QA, so an attacker with fine-tuning access needs very few poisoned examples.
- ONION, which removes words that raise perplexity, barely affects CL-Attack because there is no anomalous token; supervised fine-tuning on clean data also leaves most of the attack intact.
- TranslateDefense lowers ASR from near 100 percent to roughly 33 to 36 percent on the tested models and datasets while preserving clean performance, confirming that the trigger is carried by language mixing rather than by content.
- The trigger's effect is tied to the exact language sequence: replacing a language or changing the order drops ASR to zero, which means the backdoor is not a general multilingual-input signal.
Reading between the lines
- The paper's setup deliberately uses monolingual clean samples, so the key untested risk is the false-trigger rate on naturally code-mixed input; a natural next experiment is to measure ASR on real code-mixed user prompts that happen to follow the trigger order.
- If code-mixed writing becomes more common in real LLM usage, the rarity assumption weakens, and a trigger like ZH-EN-DE may fire unintentionally; this suggests the attack's stealthiness is tied to current usage patterns, not to an intrinsic property of the trigger.
- A stronger variant could choose a language sequence that survives machine-translation normalization, which would defeat TranslateDefense; this is an extension the paper does not explore.
- The finding that structure, not text, carries the attack suggests the same paragraph-level trigger idea could transfer to other structural markers such as punctuation patterns or template order, though the paper does not test that.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CL-Attack, a backdoor attack on fine-tuned LLMs whose trigger is a fixed sequence of languages (e.g., ZH-EN-DE) spanning multiple segments of an input. An attacker translates segments of clean samples into the chosen languages and relabels them; at inference, inputs exhibiting the same language order produce the target behavior. The authors evaluate on SST-2, MARC, and MLQA with Llama-3-8B, Qwen2-7B, and Qwen2-1.5B, report near-100% ASR with a 5% poisoning rate and no drop in CP on their selected clean sets, show better fluency and semantic similarity than BadNL, SOS, and StyleBkd, find the attack survives ONION and SFT, and propose TranslateDefense as a mitigation.
Significance. If the trigger mechanism is as general as claimed, the paper identifies a new backdoor vector that is harder to filter than fixed-token triggers and more broadly applicable than syntax- or style-based triggers. The paper's strengths include a clear attack construction, code release, and a well-designed set of structural ablations: Table 6 shows that replacing the translation model leaves ASR at 1.0 while changing the language order or the segment order drops ASR to 0, which is good evidence that the language-order pattern, not the specific translated text, is what the model learns. The main gap is that the stealthiness and universality claims are evaluated only on clean sets that exclude the natural multilingual and code-mixed inputs most relevant to the trigger. Because this gap is directly addressable with additional measurement, the paper is promising but not yet ready in its current form.
major comments (3)
- [Implementation Details; Table 5] The paper's central stealthiness claim is not measured on the input regime where it matters. The authors state that for MARC and MLQA they deliberately select monolingual Chinese, English, and German samples as the clean set 'because there is a risk that text in these languages might be mistaken by LLM for poisoned text.' This is a direct acknowledgment that multilingual text in the trigger languages is a known failure mode, yet no false-trigger rate on naturally code-mixed or multilingual user inputs is reported. Given that Table 5 shows two-language sequences such as ZH-ES, DE-ZH, and ES-EN-ES achieve near-100% ASR, an attacker can choose a trigger that resembles common language alternation; the reported CPs therefore cannot support the claims of 'better stealthiness' and 'normal performance on clean samples' for natural multilingual usage. I would expect at minimum a measurement on an unmodified multilingual or code-mixed corpus, plus an analysis of near-trigger inputs such as two-of-three-language subsequences.
- [Evaluation Metrics; Table 2] The construction of the poisoned test set is underspecified, and this matters for the universality claim. The paper does not state whether the poisoned test examples are disjoint from the poisoned training examples, whether the test set is generated with the same translation model and segmentation rules as training, or whether any naturally occurring instances of the trigger language sequence were removed from the test set. If ASR is measured only on inputs produced by the same pipeline, the near-100% numbers in Table 2 may not transfer to novel inputs with the same language sequence but different segmentation or translation quality. Please report the test-poisoning procedure explicitly and, if possible, measure ASR on held-out naturally multilingual texts that contain the trigger sequence.
- [Discussion; Table 6] The structural ablation is informative but too coarse to bound the false-trigger risk. Changing one language or swapping or removing a segment drops ASR to 0, which shows that exact sequence identity matters, but it does not quantify the model's behavior on partial or noisy realizations of the trigger, such as a ZH-EN prefix of ZH-EN-DE or an inserted untranslated word. Since natural code-mixed text will rarely match a three-language sequence exactly, the paper should report ASR on such variants before claiming stealthiness under realistic usage.
minor comments (5)
- [Author block; Defenses] There are several typographical errors, including 'Univeristy' in the author affiliations and 'consdider' in the Defenses section.
- [Evaluation Metrics] The paper states that hypothesis tests with a p-value threshold of 0.05 are conducted on CP and ASR results, but no p-values, test names, or significance tables are reported in the manuscript.
- [Table 6] The 'Modification' rows in Table 6 are not fully defined in the table or the surrounding text; for reproducibility, each modification should specify exactly which languages are swapped and which segment is removed.
- [Figure 1 caption] The caption states as an assumption that 'monolingual or other multilingual inputs do not trigger the backdoor'; this should be labeled as an empirical assumption to be tested rather than a demonstrated property.
- [Methodology] The segmentation rule is described only as 'based on natural paragraphs,' but for MLQA the inputs contain instructions, questions, and contexts; the relationship between these components and the segment boundaries should be clarified to make the trigger construction reproducible.
Circularity Check
No circular derivation: all quantities are measured against independent labels and design choices are varied in ablations; the code-mixed false-trigger concern is an untested assumption, not a circular step.
full rationale
CL-Attack does not derive any predicted quantity from an assumed result. The trigger pattern (ZH-EN-DE) is a design choice; poisoning rate (5%) is a hyperparameter; and segmentation/translation are procedural choices evaluated by ASR, CP, PPL, and TS. ASR is defined as the percentage of targeted outputs on a poisoned test set with independently labeled targets, so success is not true by definition. The 'Model Change' ablation (Table 6) shows the effect persists when the translated text is replaced, which is evidence against memorization. The central stealthiness/universality claim does rely on an unmeasured assumption: natural code-mixed or multilingual user inputs might contain the trigger language sequence. The paper itself acknowledges this in Implementation Details ('because there is a risk that text in these languages might be mistaken by LLM for poisoned text') and in Figure 1 ('We regard that monolingual or other multilingual inputs do not trigger the backdoor'). That is a validity gap, but not circularity, since no equation or fitted parameter reduces to the conclusion. Self-citations (e.g., Sha et al. for SFT, He et al. for related backdoor transfer) are background or baseline methodology and are not load-bearing for the paper's central empirical claim. Accordingly, no circular step is identified and the score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Translation via GPT-4o (or OPUS-MT) preserves enough meaning that poisoned samples remain semantically equivalent to originals.
- domain assumption Normal user inputs are effectively monolingual, so the fixed language-order trigger is rare in clean data.
- domain assumption SFT on a small 4,000-sample training set is a representative victim training scenario.
- domain assumption LLMs can learn and generalize a language-structure trigger rather than memorizing specific translations.
Cite this review
Pith. "Pith review of CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggers." pith.science (2026). https://pith.science/paper/Q2WFVC6N
@misc{pith2026241219037,
author = {Pith},
title = {Pith review of: CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggers},
year = {2026},
howpublished = {\url{https://pith.science/paper/Q2WFVC6N}},
note = {Machine review of arXiv:2412.19037}
}
read the original abstract
Backdoor attacks significantly compromise the security of large language models by triggering them to output specific and controlled content. Currently, triggers for textual backdoor attacks fall into two categories: fixed-token triggers and sentence-pattern triggers. However, the former are typically easy to identify and filter, while the latter, such as syntax and style, do not apply to all original samples and may lead to semantic shifts. In this paper, inspired by cross-lingual (CL) prompts of LLMs in real-world scenarios, we propose a higher-dimensional trigger method at the paragraph level, namely CL-attack. CL-attack injects the backdoor by using texts with specific structures that incorporate multiple languages, thereby offering greater stealthiness and universality compared to existing backdoor attack techniques. Extensive experiments on different tasks and model architectures demonstrate that CL-attack can achieve nearly 100% attack success rate with a low poisoning rate in both classification and generation tasks. We also empirically show that the CL-attack is more robust against current major defense methods compared to baseline backdoor attacks. Additionally, to mitigate CL-attack, we further develop a new defense called TranslateDefense, which can partially mitigate the impact of CL-attack.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
AI@Meta. 2024. Llama 3 Model Card
2024
-
[4]
Chai, L.; Yang, J.; Sun, T.; Guo, H.; Liu, J.; Wang, B.; Liang, X.; Bai, J.; Li, T.; Peng, Q.; et al. 2024. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. arXiv preprint arXiv:2401.07037
arXiv 2024
-
[5]
Chang, Y.; Wang, X.; Wang, J.; Wu, Y.; Yang, L.; Zhu, K.; Chen, H.; Yi, X.; Wang, C.; Wang, Y.; et al. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3): 1--45
2024
-
[6]
Chen, X.; Dong, Y.; Sun, Z.; Zhai, S.; Shen, Q.; and Wu, Z. 2022. Kallima: A clean-label framework for textual backdoor attacks. In European Symposium on Research in Computer Security, 447--466. Springer
work page 2022
-
[7]
Chen, X.; Salem, A.; Chen, D.; Backes, M.; Ma, S.; Shen, Q.; Wu, Z.; and Zhang, Y. 2021. Badnl: Backdoor attacks against nlp models with semantic-preserving improvements. In Proceedings of the 37th Annual Computer Security Applications Conference, 554--569
work page 2021
-
[8]
Colla, D.; Delsanto, M.; Agosto, M.; Vitiello, B.; and Radicioni, D. P. 2022. Semantic coherence markers: The contribution of perplexity metrics. Artificial Intelligence in Medicine, 134: 102393
work page 2022
Show all 40 references
-
[9]
Dai, J.; Chen, C.; and Li, Y. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access, 7: 138872--138878
2019
-
[10]
J.; and Bing, L
Deng, Y.; Zhang, W.; Pan, S. J.; and Bing, L. 2023. Multilingual jailbreak challenges in large language models. arXiv preprint arXiv:2310.06474
2023 arXiv
-
[11]
I.; and Cohn, T
He, X.; Wang, J.; Xu, Q.; Minervini, P.; Stenetorp, P.; Rubinstein, B. I.; and Cohn, T. 2024. Transferring Troubles: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning. arXiv preprint arXiv:2404.19597
2024 arXiv
-
[12]
Keung, P.; Lu, Y.; Szarvas, G.; and Smith, N. A. 2020. The multilingual amazon reviews corpus. arXiv preprint arXiv:2010.02573
2020 arXiv
-
[13]
Kurita, K.; Michel, P.; and Neubig, G. 2020. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660
2020 arXiv
-
[14]
Lewis, P.; O g uz, B.; Rinott, R.; Riedel, S.; and Schwenk, H. 2019. MLQA: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475
2019 arXiv
-
[15]
Li, S.; Liu, H.; Dong, T.; Zhao, B. Z. H.; Xue, M.; Zhu, H.; and Lu, J. 2021. Hidden backdoors in human-centric language models. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 3123--3140
2021
-
[16]
Liu, Y.; Sun, Z.; He, X.; and Huang, X. 2024. Quantized Delta Weight Is Safety Keeper. arXiv preprint arXiv:2411.19530
2024 arXiv
-
[17]
Ormazabal, A.; Zheng, C.; d'Autume, C. d. M.; Yogatama, D.; Fu, D.; Ong, D.; Chen, E.; Lamprecht, E.; Pham, H.; Ong, I.; et al. 2024. Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models. arXiv preprint arXiv:2404.12387
2024 arXiv
-
[18]
Qi, F.; Chen, Y.; Li, M.; Yao, Y.; Liu, Z.; and Sun, M. 2020. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint arXiv:2011.10369
2020 arXiv
-
[19]
Qi, F.; Chen, Y.; Zhang, X.; Li, M.; Liu, Z.; and Sun, M. 2021 a . Mind the style of text! adversarial and backdoor attacks based on text style transfer. arXiv preprint arXiv:2110.07139
2021 arXiv
-
[20]
Qi, F.; Li, M.; Chen, Y.; Zhang, Z.; Liu, Z.; Wang, Y.; and Sun, M. 2021 b . Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint arXiv:2105.12400
2021 arXiv
-
[21]
Qi, F.; Yao, Y.; Xu, S.; Liu, Z.; and Sun, M. 2021 c . Turn the combination lock: Learnable textual backdoor attacks via word substitution. arXiv preprint arXiv:2106.06361
2021 arXiv
-
[22]
Qin, L.; Chen, Q.; Wei, F.; Huang, S.; and Che, W. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. arXiv preprint arXiv:2310.14799
2023 arXiv
-
[23]
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9
2019
-
[24]
Ran, D.; Liu, J.; Gong, Y.; Zheng, J.; He, X.; Cong, T.; and Wang, A. 2024. JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Models. arXiv preprint arXiv:2406.09321
2024 arXiv
-
[25]
Sha, Z.; He, X.; Berrang, P.; Humbert, M.; and Zhang, Y. 2022. Fine-tuning is all you need to mitigate backdoor attacks. arXiv preprint arXiv:2212.09067
2022 arXiv
-
[26]
Sheng, X.; Han, Z.; Li, P.; and Chang, X. 2022. A survey on backdoor attack and defense in natural language processing. In 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS), 809--820. IEEE
2022
-
[27]
D.; Ng, A
Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A. Y.; and Potts, C. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, 1631--1642
2013
-
[28]
Song, K.; Tan, X.; Qin, T.; Lu, J.; and Liu, T.-Y. 2020. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33: 16857--16867
2020
-
[29]
Sun, Z.; Cong, T.; Liu, Y.; Lin, C.; He, X.; Chen, R.; Han, X.; and Huang, X. 2024. PEFTGuard: Detecting Backdoor Attacks Against Parameter-Efficient Fine-Tuning. arXiv preprint arXiv:2411.17453
2024 arXiv
-
[30]
Tiedemann, J.; Aulamo, M.; Bakshandaeva, D.; Boggia, M.; Gr \"o nroos, S.-A.; Nieminen, T.; Raganato\, A.; Scherrer, Y.; Vazquez, R.; and Virpioja, S. 2023. Democratizing neural machine translation with OPUS-MT . Language Resources and Evaluation, (58): 713--755
2023
-
[31]
Tiedemann, J.; and Thottingal, S. 2020. OPUS-MT--building open translation services for the world. In Proceedings of the 22nd annual conference of the European Association for Machine Translation, 479--480
2020
-
[32]
I.; and Cohn, T
Wang, J.; Xu, Q.; He, X.; Rubinstein, B. I.; and Cohn, T. 2024. Backdoor Attack on Multilingual Machine Translation. arXiv preprint arXiv:2404.02393
2024 arXiv
-
[33]
Wendler, C.; Veselovsky, V.; Monea, G.; and West, R. 2024. Do llamas work in english? on the latent language of multilingual transformers. arXiv preprint arXiv:2402.10588
2024 arXiv
-
[34]
Yang, A.; Yang, B.; Hui, B.; Zheng, B.; Yu, B.; Zhou, C.; Li, C.; Li, C.; Liu, D.; Huang, F.; Dong, G.; Wei, H.; Lin, H.; Tang, J.; Wang, J.; Yang, J.; Tu, J.; Zhang, J.; Ma, J.; Xu, J.; Zhou, J.; Bai, J.; He, J.; Lin, J.; Dang, K.; Lu, K.; Chen, K.; Yang, K.; Li, M.; Xue, M.;...
2024 arXiv
-
[35]
Yang, W.; Lin, Y.; Li, P.; Zhou, J.; and Sun, X. 2021. Rethinking stealthiness of backdoor attack against nlp models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proc...
2021
-
[36]
Yang, Z.; He, X.; Li, Z.; Backes, M.; Humbert, M.; Berrang, P.; and Zhang, Y. 2023. Data Poisoning Attacks Against Multimodal Encoders. In International Conference on Machine Learning, ICML , volume 202 of Proceedings of Machine Learning Research, 39299--39313. PMLR
2023
-
[37]
Yao, Y.; Duan, J.; Xu, K.; Cai, Y.; Sun, Z.; and Zhang, Y. 2024. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, 100211
2024
-
[38]
Yong, Z.-X.; Menghini, C.; and Bach, S. H. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446
2023 arXiv
-
[39]
E.; Sheng, Q
Zhang, W. E.; Sheng, Q. Z.; Alhazmi, A.; and Li, C. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3): 1--41
2020
-
[40]
Zhang, X.; Zhang, Z.; Ji, S.; and Wang, T. 2021. Trojaning language models for fun and profit. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), 179--197. IEEE
2021
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.