{"id":"1bcdd8a2-67bc-428b-974f-de328528994c","arxiv_id":"2412.16555","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"JMLLM, a hybrid obfuscation framework, raises jailbreak success rates across text, image, and speech inputs of multimodal LLMs while using fewer queries than prior methods.","lead":"This paper introduces JMLLM, a jailbreak method that combines four prompt-obfuscation tricks (translation, encryption, image noise, and text-in-image) to attack text, visual, and speech inputs of multimodal LLMs. It also contributes TriJail, a new tri-modal jailbreak dataset, and reports higher attack success rates with fewer queries than four baselines across 13 models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The KW-ASR metric in §5.2 is defined inverted, making the keyword-based success rates in Tables 3–8 and the multi-metric validation of JMLLM uninterpretable until the decision rule and dictionary are specified.","rationale":"The reader's weakest assumption identifies exactly the right fault line. The paper's quantitative contribution is a set of ASR numbers, and §5.2 is the only place where the reader is told how those numbers were produced; that passage is self-contradictory. The inversion is repeated in §6.1, so it cannot be dismissed as an isolated typo, and the authors provide neither the keyword dictionary nor the matching code, making independent verification impossible. I considered whether the GPT-ASR column alone could carry the SOTA claim even if KW-ASR were void. It might, but the paper's argument for reliability is explicitly multi-metric, and Table 5 presents both GPT-ASR and KW-ASR as joint evidence. With KW-ASR uninterpretable, the combined claim is not established as written. This does not mean the method is fraudulent; the case study and ablations are suggestive, and the GPT-ASR numbers may well survive a corrected evaluation. That is why I recommend keeping the reader's CONDITIONAL verdict rather than moving to REJECT: the issue is checkable, and the authors should be required to supply the dictionary, the exact decision rule, and a reproducible evaluation script. The Limitations section in §8 concedes the lack of a unified benchmark, which further underscores why a precisely specified keyword metric matters, but it does not acknowledge the KW-ASR ambiguity itself.","tokens_in":26108,"tokens_out":8870,"duration_ms":79331,"concrete_test":"Obtain from the authors the evaluation code, the keyword dictionary, and the raw model responses for the AdvBench experiments (or reproduce JMLLM-Single on the 520 AdvBench behaviors). Run the keyword filter under both decision rules: (i) success iff the response contains at least one dictionary keyword; (ii) success iff the response contains no dictionary keyword. Compare the resulting KW-ASR values to Table 5. If rule (i) reproduces the table, the prose in §5.2 is a typo and the metric is salvageable after correction; if rule (ii) reproduces the table, the reported KW-ASR column is inverted and all keyword-based ASR claims in Tables 3–8 must be recomputed. In either case, the authors should publish the corrected metric definition and a reproducibility script before the SOTA claim is taken at face value.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim, that JMLLM-Single outperforms prior text jailbreak methods on AdvBench with one query, is supported by two automated metrics in Table 5: GPT-ASR and KW-ASR. The KW-ASR definition in §5.2 is stated backwards: 'If the model-generated response does not contain any of the keywords in the dictionary, it is considered a successful attack.' Under this literal rule, a refusal containing no keywords counts as a success, so the metric is not an attack success rate at all. The same inverted phrasing appears in §6.1 ('If no keywords are matched, it is considered a successful jailbreak'), so this is not a single typographical slip but a systematic ambiguity in the evaluation layer. The paper also does not provide the keyword dictionary, the matching procedure, or any code, so the reader cannot tell whether the reported values such as 0.967 KW-ASR for JMLLM-Single on Llama2-7B reflect successful attacks or the absence of dictionary hits. Even if the GPT-ASR column alone could carry the SOTA claim, the paper's own argument in §5.2 is that multiple metrics are needed to reduce bias, and §6.1 explicitly warns that keyword matching 'may lead to inflated performance scores.' With KW-ASR uninterpretable in either direction, the combined multi-metric validation that the paper relies on is not currently established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes JMLLM, a multimodal jailbreaking framework that combines four obfuscation techniques—alternating translation into low-resource languages, word encryption via shuffling and a Caesar cipher, feature collapse/edge-highlighting for images, and harmful text injection into noisy images—to attack text, visual, and speech inputs of large language models. The authors introduce TriJail, a tri-modal dataset with 1,250 text prompts, 150 images, and 1,250 speech prompts across six harm categories, and evaluate JMLLM on 13 LLMs using four metrics: keyword-based ASR, GPT-4-based ASR, human annotation, and toxicity scoring. On AdvBench, JMLLM-Single is reported to achieve higher ASR than ReNeLLM and other baselines with a single query and a time cost of 24.65 seconds per attack, while JMLLM-Multi uses six queries. The paper also proposes a defense strategy called the Harmful Separator that partially mitigates the attack.","tokens_in":26451,"tokens_out":5228,"duration_ms":40024,"significance":"If the empirical claims hold, the paper makes a practical contribution: it shows that simple obfuscation strategies—word-level encryption and low-resource-language alternation—can bypass alignment in several popular LLMs with only one query, a substantial efficiency improvement over iterative methods like PAIR and ReNeLLM. The TriJail dataset is a potentially reusable resource for multimodal safety evaluation, and the comparison across four evaluation metrics is a step toward more robust assessment. The central direction is corroborated by existing work on text obfuscation and by the paper's own ablations, which show meaningful drops when modules are removed. However, the evaluation layer currently prevents the reader from trusting the headline numbers: the keyword-based metric is defined in a self-contradictory way, no variance estimates are provided, the visual and speech samples are small and text-derived, and no code or data artifacts are released to verify the results.","major_comments":[{"comment":"The KW-ASR definition is stated in reverse in Section 5.2: 'If the model-generated response does not contain any of the keywords in the dictionary, it is considered a successful attack.' Taken literally, a refusal that contains no keywords is counted as a success, which contradicts the entire premise of an attack success rate and would make scores near 1 for all models. Section 6.1 compounds the confusion by writing, 'If no keywords are matched, it is considered a successful jailbreak, which may lead to inflated performance scores,' which is inconsistent both with the Section 5.2 wording and with the paper's own intent. Because KW-ASR is one of the two metrics supporting the headline state-of-the-art claim in Table 5 and is used throughout Tables 3–8, the authors must state the correct decision rule explicitly, provide the exact keyword dictionary and matching procedure, and re-report or recompute the affected tables.","section":"5.2 and 6.1"},{"comment":"All ASR values in the paper appear to be point estimates from a single run. No error bars, confidence intervals, or significance tests are reported. Since the attack model has temperature 1 and the GPT-4 judge is stochastic, differences such as JMLLM-Single's 0.921 versus ReNeLLM's 0.869 on GPT-3.5-turbo in Table 5 may be within run-to-run noise. I request means and standard deviations over at least three independent runs, or paired statistical tests for the head-to-head comparisons in Table 9, before the superiority claim is accepted.","section":"5.4, Tables 3–9"},{"comment":"The visual and speech evaluations are underpowered and not independent. The visual results in Table 6 are based on only the 150 images in TriJail, which were generated from the text prompts via DALL-E-3, and the speech results in Table 7 are based on 20 randomly chosen speech samples per scenario, i.e., 120 samples total, converted from the same text prompts via TTS-1. These samples are not independent adversarial instances, and the sample sizes are too small to support broad claims about 'strong competitiveness' in the speech modality or about the vulnerability of the visual modality. Report confidence intervals or bootstrap estimates for these results.","section":"6.3 and 6.4"},{"comment":"The paper commits to publicly sharing the TriJail dataset and JMLLM framework, but no code, data URL, keyword dictionary, GPT-4 judge prompt, or exact API parameters (e.g., TTS-1 voice settings, DALL-E-3 prompt templates, Canny thresholds) are provided anywhere in the manuscript or appendices. Without these artifacts, the four ASR metrics cannot be independently computed and the central comparison in Table 5 cannot be verified. Please release the artifacts or include a complete appendix with all evaluation details.","section":"8, Ethics and open science policy"}],"minor_comments":[{"comment":"The Caesar cipher formula in line 18 uses `ord(ch)` with `ch in {A,a}` but does not condition on the case of the character being encrypted; as written, lower-case letters will be shifted relative to `ord('a')` only when `ch='a'`, and mixed-case words may produce non-alphabetic output. Clarify the case handling and provide the exact reconstruction instruction given to the LLM.","section":"4.2, Algorithm 1"},{"comment":"The 'Words' and 'Tokens' columns each show two values concatenated without a separator or header explanation (e.g., '10.94±6.0412.11±6.56'); format these as two separate mean±std columns.","section":"Table 1"},{"comment":"The selection of 6 as the multi-round query count is justified by the ASR curve in Figure 7; state explicitly whether this endpoint was chosen a priori or on the basis of this figure, since post hoc selection can inflate the reported performance.","section":"6.7"},{"comment":"There is a typo: 'TarJail' should be 'TriJail'.","section":"Appendix D"},{"comment":"References [14] and [60] appear to refer to the same work with different titles and venues; please reconcile the duplicates and ensure each cited work is listed once.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has a useful dataset and a plausible attack direction, but the evaluation layer is not yet trustworthy. The KW-ASR inversion appears twice and affects every table, signaling a systematic problem rather than a simple typo; the authors should verify what the implemented metric actually did. The lack of error bars, the small text-derived visual/speech samples, and the absence of code/data make the SOTA claim unverifiable as it stands. I also note that the 'first tri-modal jailbreak' claim is asserted without a detailed comparison to the multimodal works already cited ([37,62]), and the ethics statement does not address the dual-use implications of releasing a working tri-modal attack beyond a generic commitment to vendor disclosure. These issues are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the SOTA-on-AdvBench result is plausible, but the keyword-based success metric is defined backwards in two places and no code or data are out, so the exact numbers should be treated as provisional.\n\nWhat is actually new: JMLLM combines alternating translation, word encryption, feature collapse, and harmful injection into one tri-modal framework, and contributes the TriJail dataset (1,250 text, 150 image, 1,250 speech prompts across six scenarios). Each module has precedent in the jailbreak literature, but the combination and the tri-modal dataset are new. The ablation study is informative and shows each module contributes. The single-query attack, at 24.65 seconds per query on AdvBench, is a meaningful efficiency improvement over ReNeLLM, and the GPT-ASR column alone shows a sizeable jump on four of five models. I think the phenomenon is real.\n\nThe soft spots are real but fixable. The KW-ASR definition in §5.2 says: 'If the model-generated response does not contain any of the keywords in the dictionary, it is considered a successful attack.' That is inverted. The same phrasing appears in §6.1, and right after that the authors warn that keyword matching can inflate scores. Taken literally, refusals count as successes. The keyword dictionary and matching procedure are never provided, so the KW-ASR columns in Tables 3–8 are not interpretable. That matters because the paper's own argument is that multiple metrics are needed to reduce bias; with KW-ASR scrambled, the multi-metric validation collapses, even if GPT-ASR alone still tells a coherent story.\n\nAlso minor but real: the visual (150) and speech (20 per class) samples are small and derived from text; no error bars or repeated runs; the multi-round query count of 6 was chosen after looking at the ASR curve; code and hyperparameters are not public; the GPT-4 judge is used without agreement statistics. None of this is fatal, but it means the specific numbers need re-verification.\n\nThis paper is for researchers working on jailbreak evaluation and multimodal safety. It deserves a serious referee, but the evaluation layer needs an explicit fix: correct the keyword rule, release the dictionary and code, and ideally rerun the headline tables. I would accept conditionally after those revisions.","headline":"The AdvBench SOTA is plausible, but the keyword-ASR metric is defined backwards in two places and no code/data are released, so the headline numbers need re-verification before I'd trust them.","tokens_in":26932,"tokens_out":3561,"would_cite":false,"duration_ms":28497,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hybrid text, image, and speech attack breaks aligned LLMs in one query","keywords":["jailbreak","multimodal large language models","attack success rate","adversarial prompts","TriJail dataset","speech modality","visual modality","low-resource language translation"],"falsifier":"Re-run the JMLLM single-query attack on a sample of 100 AdvBench prompts across GPT-4 and Claude-2, have five independent human annotators label each response as refusal, compliant harmless text, or harmful, and compare the resulting success rate with the reported GPT-ASR and KW-ASR values; if the human-verified rate is materially below the reported rate, the central claim about attack effectiveness is not established.","tokens_in":25960,"feed_emoji":"🔓","tokens_out":6575,"duration_ms":51232,"temperature":0.7,"pith_summary":"The paper claims that jailbreaking aligned multimodal large language models is most effective when the harmful instruction is split across text, visual, and speech channels and each channel is disguised separately. It proposes JMLLM, a hybrid framework combining alternating translation, word encryption, feature collapse, and harmful injection, and reports state-of-the-art attack success rates on 13 popular LLMs using the AdvBench benchmark. In single-query mode, JMLLM-Single reaches higher GPT-ASR scores than prior text-only baselines, such as 0.921 on GPT-3.5 and 0.792 on GPT-4 versus 0.869 and 0.589 for ReNeLLM, while taking roughly one-fifth of the time (24.65 seconds versus 132.03 seconds). The paper also contributes TriJail, a tri-modal dataset of 1,250 text prompts, 150 harmful images, and 1,250 speech prompts across six harm scenarios, and proposes a 'Harmful Separator' defense that lowers but does not eliminate the attack's success rate.","feed_headline":"Hybrid text, image, and speech attack breaks aligned LLMs in one query","feed_subtitle":"JMLLM claims higher success rates on 13 LLMs in one query and offers a tri-modal attack dataset.","key_machinery":"The carrier of the argument is JMLLM, a four-module disguise pipeline. Alternating Translation maps each word of a harmful prompt into one of four low-resource languages (Czech, Norwegian, Danish, Romanian) so the model's weaker multilingual comprehension lowers its guard. Word Encryption shuffles each word's letters and applies a Caesar cipher, then asks the model to decrypt and restore the text as a benign two-task prompt. Feature Collapse converts images to grayscale, applies Canny edge detection and Gaussian blur, and multiplies them so the image retains only the harmful outlines the model can still read while automated safety filters see little. Harmful Injection adds uniform noise to an image and draws harmful text as subtitles on it. For speech, the processed text is converted to audio with a text-to-speech model. These modules are applied independently to text, image, and speech inputs, and the framework runs either as a single query or as a six-round iterative attack; the ablation study attributes the largest single-query gains to the two text modules.","core_discovery":"On its own terms, the central claim is that JMLLM is the first jailbreak method to integrate text, visual, and speech modalities, and that this integration defeats aligned LLMs more reliably and more cheaply than text-only methods. The evidence is presented in Table 5: with one query, JMLLM-Single achieves GPT-ASR of 0.921 on GPT-3.5, 0.792 on GPT-4, 0.992 on Claude-1, 0.942 on Claude-2, and 0.842 on Llama2-7B, all above the corresponding ReNeLLM scores, and the six-query multi-round version pushes most scores above 0.95. The attack works by disguising the harmful instruction in four ways: translating words into low-resource languages, encrypting words with character shuffling and a Caesar cipher, stripping image features via edge detection and blurring, and injecting harmful text as subtitles into noisy images, so the target model reconstructs the harmful request and completes it. The paper treats these results as evidence that multimodal inputs create a wider attack surface and that safety alignment in one modality does not protect the others.","pith_inferences":["If the inverted KW-ASR wording in Section 5.2 is a typo and the intended rule is 'contains,' the reported keyword-based scores are likely optimistic; re-evaluating the same outputs with the corrected rule would show how much of the headline improvement is metric artifact.","The single-query success on closed models suggests that multimodal alignment is still shallow: the same safety policy does not transfer across encoders, so future models may need per-modality red-teaming before release.","The four disguise techniques could generalize to video or haptic modalities, where the same principle of splitting harmful content across a weak channel and asking the model to reconstruct it may apply, and this is directly testable with JMLLM's existing text modules converted to video subtitles or audio captions.","Because the paper reports results on specific model versions, and models are updated and patched over time, the absolute ASR numbers are time-sensitive; the durable claim is the relative vulnerability of multimodal inputs rather than the exact scores."],"forward_implications":["A single query is enough to elicit harmful completions from several closed and open models, so the attack cost drops by roughly five times relative to the strongest text-only baseline.","The six-query version reaches near-1.0 success on several models, showing that iterative refinement can push aligned models to comply almost always.","Visual and speech channels are independently vulnerable: JMLLM scores above 0.89 GPT-ASR on Qwen-vl-max in the visual modality and above 0.89 on GPT-4o-mini in speech, so defenses must inspect each modality separately.","The proposed Harmful Separator defense, which splits an instruction from its example and checks the example, cuts the attack success rate substantially but not to zero, leaving a residual risk.","Keyword-dictionary evaluation yields the highest scores among the four metrics, and the paper itself warns that this metric may inflate performance."],"supporting_citations":[{"why":"It supplies the AdvBench benchmark dataset and the GCG gradient-based attack baseline that JMLLM is compared against.","marker":"[66]"},{"why":"It provides the ReNeLLM baseline, the strongest text-only competitor, and the keyword-dictionary evaluation framework reused for KW-ASR.","marker":"[14]"},{"why":"It defines the PAIR iterative black-box jailbreak baseline and the GPT-4-based ASR evaluator used for the primary attack-success metric.","marker":"[9]"},{"why":"It contributes the AutoDAN genetic-algorithm baseline and the keyword dictionary construction followed by the paper.","marker":"[25]"},{"why":"It establishes the speech-modality jailbreak line of work and the finding that text-to-speech conversion alone is resisted, motivating JMLLM's disguised text-then-speech approach.","marker":"[39]"},{"why":"It provides the HADES visual jailbreak baseline whose response toxicity JMLLM is qualitatively compared with in the case study.","marker":"[23]"},{"why":"It shows that single visual adversarial samples can break aligned LLMs, motivating the visual modules of feature collapse and harmful injection.","marker":"[33]"},{"why":"It documents that LLMs are weaker on low-resource languages, which is the premise behind Alternating Translation.","marker":"[13]"}],"fun_headline_variants":["One-query tri-modal jailbreak outdoes 13 LLMs","Hybrid attack on text, image, speech fells aligned LLMs","JMLLM: One-query tri-modal jailbreak defeats 13 LLMs","Tri-modal jailbreak: one query, higher success on 13 LLMs","One-shot tri-modal attack cracks text, image, and speech barriers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline attack-success numbers depend on the keyword-dictionary metric (KW-ASR), and the paper states its rule as 'if the response does not contain any of the keywords, it is considered a success,' which taken literally counts refusals as successful attacks, and the paper never clarifies whether 'contains' was intended, leaving every KW-ASR figure built on an unresolved ambiguity.","fun_headline_variants_meta":{"raw":{"variants":["One-query tri-modal jailbreak outdoes 13 LLMs","Hybrid attack on text, image, speech fells aligned LLMs","JMLLM: One-query tri-modal jailbreak defeats 13 LLMs","Tri-modal jailbreak: one query, higher success on 13 LLMs","One-shot tri-modal attack cracks text, image, and speech barriers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000719,"raw_usage":{"total_tokens":3242,"prompt_tokens":975,"completion_tokens":2267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":2170}},"tokens_in":591,"tokens_out":2267,"duration_ms":15494,"temperature":1.0,"reasoning_tokens":2170,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:28:14.574185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the JMLLM single-query attack on a sample of 100 AdvBench prompts across GPT-4 and Claude-2, have five independent human annotators label each response as refusal, compliant harmless text, or harmful, and compare the resulting success rate with the reported GPT-ASR and KW-ASR values; if the human-verified rate is materially below the reported rate, the central claim about attack effectiveness is not established.","supporting_citations":[{"cited_title":"A wolf in sheep’s clothing: Generalized nested jailbreak prompts can fool large language models easily","cited_arxiv_id":null,"evidence_quote":"It provides the ReNeLLM baseline, the strongest text-only competitor, and the keyword-dictionary evaluation framework reused for KW-ASR."},{"cited_title":"Autodan: Generating stealthy jailbreak prompts on aligned large language models","cited_arxiv_id":null,"evidence_quote":"It contributes the AutoDAN genetic-algorithm baseline and the keyword dictionary construction followed by the paper."},{"cited_title":"Images are achilles’ heel of alignment: Exploiting visual vulnerabilities for jailbreaking multi- modal large language models","cited_arxiv_id":null,"evidence_quote":"It provides the HADES visual jailbreak baseline whose response toxicity JMLLM is qualitatively compared with in the case study."},{"cited_title":"Visual adversarial examples jailbreak aligned large language models","cited_arxiv_id":null,"evidence_quote":"It shows that single visual adversarial samples can break aligned LLMs, motivating the visual modules of feature collapse and harmful injection."},{"cited_title":"Multilingual jailbreak challenges in large language models","cited_arxiv_id":null,"evidence_quote":"It documents that LLMs are weaker on low-resource languages, which is the premise behind Alternating Translation."}],"review_version":1}