{"id":"74b2199c-342b-407b-9bc7-f45c547192de","arxiv_id":"2507.08898","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A LoRA-adapted SeaLLM model detects unsafe and jailbreak prompts in nine Southeast Asian languages with 97% recall and 98% F1 on the authors' new SEALSBench benchmark, far above zero-shot LlamaGuard.","lead":"Researchers built a multilingual safety filter, SEALGuard, for Southeast Asian languages by fine-tuning a regional large language model with LoRA, and they released a 260,000-plus prompt benchmark called SEALSBench. If the results hold outside their own benchmark, the work could close a known gap where English-centric guardrails miss unsafe and jailbreak prompts in languages like Lao, Khmer, and Burmese.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Machine-translated jailbreak prompts likely lose their attack structure; SEALGuard's 97% jailbreak DSR may measure translation artifacts rather than genuine multilingual jailbreak detection.","rationale":"The reader's weakest_assumption is precisely that machine translation preserves semantics and safety labels, and the paper's failure to validate translations undermines every reported metric. My concern is a sharper instance of the same assumption: not only can translation shift meaning, but the jailbreak prompts are constructed through character-level or code-level operations that translation cannot preserve. This is especially damaging because the paper's headline distinguishes SEALGuard by its jailbreak defense (RQ2 and RQ3). If the test instances are not real jailbreaks after translation, the 97.23% DSR on jailbreak prompts is an artifact of the data construction pipeline, not evidence of multilingual guardrail capability. The proposed check directly tests this by auditing the released dataset's jailbreak instances. I therefore agree with the reader's identification of the load-bearing assumption and recommend the same conditional verdict: the paper is promising but must validate translations and, if the audit fails, rebuild the jailbreak portion of SEALSBench. The other issues noted by the reader (train/test overlap, absent fine-tuned baselines, arithmetic inconsistencies) are real but secondary; correcting them would improve the comparative claims, while correcting translation validity is necessary for the benchmark itself to be meaningful.","tokens_in":17793,"tokens_out":5619,"duration_ms":63979,"concrete_test":"From the released SEALSBench, sample 20 jailbreak prompts per attack type (9 types) per language (9 languages), yielding 1,620 prompts. Have two native speakers per language independently judge: (1) Is the prompt a valid instance of the named jailbreak attack (e.g., a Caesar cipher, code chameleon, or self cipher)? (2) Is the prompt unsafe? (3) Does the English back-translation preserve the original harmful intent? Compute the proportion of prompts that fail (1) or (3) by majority vote. If the failure rate exceeds 5%, the jailbreak DSR claim is not supported; repeating the same audit on a smaller sample of the safe and unsafe prompts would test the broader translation-label-preservation assumption.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two parts: a benchmark (SEALSBench) and a detector (SEALGuard). The most load-bearing assumption is in Section 4.2 Step 2: all 29,604 English prompts—including the jailbreak prompts generated with Caesar Cipher, Self Cipher, Code Chameleon, and other character/code-level attacks from Section 4.4—are machine-translated into nine SEA languages with the unofficial googletrans wrapper. These attacks are defined at the character or code level; machine translation does not preserve cipher structure or code semantics. A Caesar-cipher prompt such as 'Nyy qnl ybat' translated into Khmer will either remain opaque characters or become meaningless text, losing the intended cipher-based jailbreak. Similarly, Self Cipher and Code Chameleon rely on exact code/cipher operations that Google Translate will corrupt. Therefore many 'multilingual jailbreak' test instances in SEALSBench are not valid instances of the jailbreak attacks they claim to represent. The model may instead learn to flag the translationese or formatting artifacts of googletrans output, inflating DSR. Section 7 acknowledges translation variation as a threat to internal validity but provides no validation: no native-speaker checks, no back-translation audit, and no attack-integrity verification. Without such validation, the reported 97.23% DSR and 98.05% F1 are not interpretable as defense against genuine multilingual jailbreak attacks.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEALGuard, a multilingual safety guardrail for Southeast Asian languages, built by LoRA fine-tuning SeaLLM-v3-7B-Chat into a LlamaGuard-style safe/unsafe classifier, and introduces SEALSBench, a multilingual safety alignment benchmark of 29,604 English prompts (safe, unsafe, jailbreak) machine-translated into nine Southeast Asian languages. The evaluation claims that SEALGuard achieves 97.23% DSR, 98.90% precision, and 98.05% F1 on the SEALSBench test set, substantially outperforming LlamaGuard-3-8B and OpenAI Moderation, and an ablation attributes the gain primarily to LoRA adaptation.","tokens_in":18139,"tokens_out":6159,"duration_ms":66346,"significance":"If the reported results are valid, the contribution would be practically valuable: a small LoRA adaptation of an accessible multilingual model would close the multilingual safety gap for low-resource Southeast Asian languages, and the released dataset would support further research. The authors are to be credited for releasing the model and benchmark, for partitioning language variants of the same prompt into the same data split to avoid leakage, and for comparing against widely used guardrails. The conditional value of the contribution, however, rests on two load-bearing assumptions: that machine translation preserves both the semantics and the safety label of every prompt, especially character-level and code-level jailbreak attacks, and that training on a 5% slice of the same benchmark used for testing provides a fair comparison with zero-shot baselines. These assumptions are not validated in the current manuscript, and the internal data counts are inconsistent.","major_comments":[{"comment":"The evaluation is self-referential: SEALGuard is trained on a 5% sample (14,800 examples) of SEALSBench and tested on the remaining 90% of the same benchmark, whose prompts share the same translation pipeline and the same nine jailbreak attack families, while LlamaGuard and OpenAI Moderation are evaluated zero-shot. The reported DSR and F1 gaps therefore largely reflect in-distribution supervised learning versus zero-shot transfer, not necessarily a multilingual guardrail advantage. Please add an out-of-benchmark evaluation, such as training on a subset of source datasets and testing on a held-out source dataset or an independent multilingual unsafe-prompt collection, and report the same zero-shot baselines on that external test set.","section":"§4.6, §5 RQ3"},{"comment":"The jailbreak test instances for Caesar Cipher, Self Cipher, Code Chameleon, and Dual use are produced by machine-translating English prompts whose attack mechanism relies on exact character-level or code-level operations. Machine translation via the googletrans library will not preserve ciphertext structure, cipher semantics, or code blocks, so many 'multilingual jailbreak' instances are not valid instances of the named attacks; the per-attack DSR exceeding 95% in Section 6.3 may be detecting translation artifacts rather than attack-specific intent. Please audit attack integrity on a sample of translated instances (for example, verify that the translation still contains the ciphertext or code and can still be decoded by the intended procedure), and either restrict the jailbreak set to attacks that survive translation or report results separately for translation-preserving and non-preserving attacks. Section 7 acknowledges translation variation as an internal-validity threat but proposes no validation beyond relying on a consistent translation tool; this is insufficient for a benchmark whose labels propagate into every reported metric.","section":"§4.2 Step 2, §4.4, §7"},{"comment":"The dataset counts are internally inconsistent. The English corpus is stated as 29,604 prompts (18,846 safe + 8,959 unsafe + 1,799 jailbreak); multiplying by ten languages gives 296,040 prompts, but the split in §4.6 sums to 14,800 + 14,800 + 266,444 = 296,044, and the RQ3 test set components sum to 169,433 + 80,601 + 16,410 = 266,444. RQ2 refers to 8,060 English unsafe prompts and 16,410 jailbreak prompts, neither of which follows from 8,959 or from 1,799 × 10 = 17,990. Section 6.3 states '10 jailbreak categories' while only nine are defined in §4.4. Please reconcile all counts and include an exact per-language and per-category breakdown, as the current inconsistencies make the reported test-set sizes and category-wise results impossible to verify.","section":"§4.2, §4.6, §5 RQ2/RQ3"},{"comment":"The external-validity discussion in Section 7 notes that the results may not generalize beyond SEALSBench, but the proposed mitigation—incorporating prompts from six source datasets—does not address the fact that all prompts are translated through the same pipeline and labeled under the same ten unsafe categories and nine jailbreak families. Since SEALGuard is trained on the same benchmark, generalization cannot be inferred from the reported numbers. A validation strategy such as back-translation agreement, native-speaker spot checks, or a human rating of translated attack instances should be reported, along with agreement statistics.","section":"§7"}],"minor_comments":[{"comment":"The abstract says 'over 260,000 prompts' while the actual total is 296,040 and the test set is 266,444; please unify the wording to the exact number.","section":"Abstract, §4.6"},{"comment":"Section 6.3 says this analysis covers '10 jailbreak categories,' but Section 4.4 defines nine jailbreak attacks; correct the typo or add the missing category.","section":"§6.3"},{"comment":"The method list in RQ4 refers to 'SeaLLM-V3-1B-Chat' while Table 1 and the surrounding text use 'SeaLLM-V3-1.5B-Chat'; make the model naming consistent.","section":"§5 RQ4, Table 1"},{"comment":"The sentence 'the LoRA module alone contributes 72% of the total F1-Score' should be phrased as a 72-percentage-point improvement (from 26% to 98%), not a 72% contribution, and the comparison conflates LoRA with the chat-template adaptation and the training data; an ablation that isolates LoRA from the template and data is needed for that attribution.","section":"§5 RQ4"},{"comment":"Reference [46] and [47] are the same Yong et al. paper, and [36] and [37] both describe the 'Do Anything Now' work; consolidate duplicate references.","section":"References"},{"comment":"The manuscript retains ACM template placeholder metadata (2018, Woodstock, placeholder DOI) and a reference to the undefined 'Conference acronym XX'; update all venue metadata before submission.","section":"Metadata"}],"recommendation":"major_revision","confidential_remarks":"The core idea of adapting a multilingual base model with LoRA for guardrail duties is reasonable and potentially useful, but the current evaluation design is self-referential and the translated character-level jailbreak instances cast doubt on the headline DSR numbers. I would require, as a condition for acceptance, an out-of-benchmark evaluation and a translation/attack-integrity audit. The paper also appears early-stage in its formatting (template placeholders, duplicate references), so the editor should ensure the submitted version is the intended camera-ready form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the dataset is a real contribution, the method is standard, and the evaluation is not yet fair enough to support the headline claims. SEALGuard's 97.23% DSR and 98.05% F1 are measured on a test set that is 90% of the same benchmark the model was trained on, with baselines run zero-shot. That gap is largely by construction.\n\nWhat is genuinely new is SEALSBench: 29,604 English prompts (safe, unsafe, jailbreak) translated into nine Southeast Asian languages, with category and attack-type labels. That is a large, useful resource for a real problem. English-centric guardrails are known to fail on low-resource languages, and the paper documents that clearly: LlamaGuard-8B drops from 58.78% to 49.63% DSR on multilingual unsafe prompts, and to 41.02% on multilingual jailbreaks. The LoRA adaptation recipe itself is not novel, but applying it to a multilingual base with a guardrail template is a sensible baseline. The authors also release the model and benchmark, which is good practice.\n\nNow the soft spots, in decreasing order of severity.\n\nFirst, evaluation fairness. SEALGuard is trained on a 5% slice of SEALSBench and tested on the remaining 90% of the same dataset. The model has seen the same distribution, including the same jailbreak families. The baselines are zero-shot. A direct comparison under these conditions is not meaningful. The fix is straightforward: fine-tune or LoRA-adapt LlamaGuard on the same training split, and evaluate everything on external multilingual safety datasets. Until then, the 97% DSR is an in-distribution number.\n\nSecond, the jailbreak translations. The stress-test note is right: Caesar Cipher, Self Cipher, and Code Chameleon operate at the character or code level. Running those through Google Translate produces text that is not a valid instance of the attack. That means many of the 16,410 'multilingual jailbreak' test items are likely not jailbreak prompts at all. The model may be learning to flag translationese or formatting cues. This undermines the jailbreak-specific claims, though it is less damaging for the plain unsafe prompts.\n\nThird, dataset arithmetic. The paper states 296,040 total prompts in Section 4.2, but the split sums to 296,044, and the test-set component counts (169,433 + 80,601 + 16,410 = 266,444) do not align cleanly with the source counts multiplied by ten. The internal numbers should reconcile. This suggests the pipeline was not run exactly as described.\n\nFourth, no error bars, no repeated runs, and no native-speaker validation of translations. The threat-to-validity section acknowledges translation variation but does not measure it.\n\nOverall, this is a solid empirical paper about a real gap, and the benchmark is worth building on. But the headline results are not yet credible as stated. A serious referee would need the authors to fix the evaluation protocol, validate (or exclude) translated jailbreak attacks, and reconcile the dataset counts. That is a major revision, not a rejection.\n\nI would send it to review, and I would want to see the revised version. For my own work, I'd cite the benchmark once the numbers are cleaned up, but not the DSR comparison.","headline":"Useful benchmark, but the headline 97% DSR is an in-distribution number and the jailbreak translations likely break the attack structure.","tokens_in":18608,"tokens_out":4951,"would_cite":false,"duration_ms":51151,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small LoRA-tuned multilingual model detects unsafe and jailbreak prompts in nine Southeast Asian languages at over 97% success, roughly doubling the defense rate of English-centric guardrails.","keywords":["multilingual safety alignment","guardrails","jailbreak detection","LoRA","Southeast Asian languages","LLM safety","safety benchmark","low-resource languages"],"falsifier":"Sample SEALSBench prompts from each language, back-translate them to English, and have native speakers judge whether the original safety label survived; if translation errors flip labels on a scale comparable to the reported performance gap, the benchmark labels and the DSR numbers built on them are unreliable.","tokens_in":17533,"feed_emoji":"🛡️","tokens_out":6949,"duration_ms":67850,"temperature":0.7,"pith_summary":"The paper claims that a parameter-efficient LoRA adaptation of a multilingual language model, prompted with a guardrail chat template, turns a general-purpose model into a multilingual safety filter that catches unsafe and jailbreak prompts in nine Southeast Asian languages. The authors build SEALSBench, a benchmark of more than 260,000 prompts translated from English into these languages, and report that their SEALGuard model detects 97.23% of unsafe prompts with 98.90% precision, far outperforming LlamaGuard-3-8B and OpenAI Moderation. If correct, the result matters because existing English-centric guardrails measurably degrade on multilingual prompts, and low-resource languages are a known jailbreak vector. The paper also shows via ablation that the LoRA adaptation drives nearly all of the gain, while model size matters less.","feed_headline":"LoRA-tuned guardrail catches 97% of Southeast Asian unsafe prompts","feed_subtitle":"LoRA-tuning a multilingual base model beats English-only guardrails on nine Southeast Asian languages, with 98% F1.","key_machinery":"The load-bearing mechanism is LoRA: trainable low-rank matrices A and B inserted into the word embedding, self-attention, and feed-forward layers of the frozen SeaLLM-v3-7B-Chat, with the LlamaGuard chat template framing the task as binary sequence generation ('safe' or 'unsafe'). Greedy decoding reads the first generated token as the decision. The other pillar is SEALSBench, built by taking English safe prompts from Alpaca, unsafe prompts from BeaverTails and four jailbreak source sets, translating all 29,604 prompts into nine Southeast Asian languages, and splitting by prompt ID so language variants stay in the same fold.","core_discovery":"The central discovery is that the multilingual safety gap is not intrinsic: a 7B multilingual model fine-tuned with LoRA, using LlamaGuard's chat template, matches or exceeds English-only guardrail performance on multilingual inputs. On the SEALSBench test set, SEALGuard achieves a Defense Success Rate of 97.23%, precision of 98.90%, and F1 of 98.05%, compared with 48.93%, 95.84%, and 63.55% for LlamaGuard-3-8B and 30.63%, 36.35%, and 40.07% for OpenAI Moderation. The same evaluation shows LlamaGuard-3-8B's DSR drops by 9 percentage points on Southeast Asian unsafe prompts and by 18 points on multilingual jailbreak prompts relative to English, quantifying the vulnerability the method addresses.","pith_inferences":["Beyond the paper's claims, the translation-based construction means the benchmark measures detection of translated English prompt distributions, not naturally occurring Southeast Asian-language speech; real-world jailbreaks in these languages may differ.","A native-speaker validation or back-translation check of SEALSBench would test whether the reported advantage partly reflects translation artifacts; the paper itself flags translation variation as a validity threat but does not carry out such a check.","The ablation shows a 1.5B model with LoRA nearly matches the 7B model, suggesting the recipe may transfer to smaller and cheaper deployments, though the paper does not claim this.","The same LoRA-plus-chat-template recipe could plausibly be applied to other regional language families if a base model with adequate multilingual coverage exists; that extension is untested."],"forward_implications":["A parameter-efficient LoRA adaptation is enough to build a multilingual guardrail, without full fine-tuning of the base model.","Existing guardrails like LlamaGuard lose 9 to 18 percentage points of DSR on multilingual unsafe and jailbreak prompts, so multilingual prompts are a practical attack surface.","SEALGuard maintains DSR above 95% across every studied unsafe category and every studied jailbreak type, according to the paper's per-category analysis.","The high precision (about 99%) means safe multilingual inputs are rarely blocked, so the guardrail does not simply trade false alarms for coverage.","The released model and benchmark allow other researchers to reproduce the numbers and test the recipe on additional languages."],"supporting_citations":[{"why":"Supplies the chat template and the main English-centric baseline, and the claim that LlamaGuard may not generalize to other languages.","marker":"[16]"},{"why":"Provides the low-rank adaptation method that is the core of SEALGuard.","marker":"[15]"},{"why":"Provides the SeaLLM-v3-7B-Chat base model that is adapted into the guardrail.","marker":"[50]"},{"why":"Supplies the 8,959 unsafe seed prompts used for the unsafe portion of SEALSBench.","marker":"[18]"},{"why":"Supplies the 18,846 safe prompts used to measure false alarms.","marker":"[39]"},{"why":"The translation tool used to create the nine language variants, on which the benchmark labels depend.","marker":"[13]"},{"why":"The OpenAI Moderation baseline that SEALGuard is compared against.","marker":"[25]"},{"why":"Motivating evidence that translated unsafe prompts bypass LLM safety, used to justify the problem.","marker":"[46]"}],"fun_headline_variants":["SEALGuard: LoRA-tuned guardrail beats LlamaGuard on Southeast Asian unsafe prompts","English-only guardrails see 18% DSR drop on multilingual jailbreaks","SEALGuard: 97% defense rate on Southeast Asian unsafe inputs","Multilingual LoRA guardrail outperforms English-only by 48% DSR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Machine translation preserves both the meaning and the safety label of every prompt, so that an unsafe English prompt remains unsafe and a safe Alpaca prompt remains safe after translation into Lao, Khmer, or Burmese.","fun_headline_variants_meta":{"raw":{"variants":["SEALGuard: LoRA-tuned guardrail beats LlamaGuard on Southeast Asian unsafe prompts","English-only guardrails see 18% DSR drop on multilingual jailbreaks","SEALGuard: 97% defense rate on Southeast Asian unsafe inputs","Multilingual LoRA guardrail outperforms English-only by 48% DSR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000937,"raw_usage":{"total_tokens":4065,"prompt_tokens":1058,"completion_tokens":3007,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2920}},"tokens_in":674,"tokens_out":3007,"duration_ms":24948,"temperature":1.0,"reasoning_tokens":2920,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:22:52.792937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample SEALSBench prompts from each language, back-translate them to English, and have native speakers judge whether the original safety label survived; if translation errors flip labels on a scale comparable to the reported performance gap, the benchmark labels and the DSR numbers built on them are unreliable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The translation tool used to create the nine language variants, on which the benchmark labels depend."}],"review_version":1}