{"id":"6885bc0d-03cc-4fbc-803e-2fa148ea57e4","arxiv_id":"2509.01444","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Strata-Sword organizes jailbreak attacks into three reasoning-complexity levels and shows that LLMs' safety failure rates rise with attack complexity, especially for new Chinese-language attacks.","lead":"This paper introduces Strata-Sword, a Chinese-English benchmark that sorts jailbreak attacks into three levels of reasoning complexity and tests 23 LLMs against them. It finds that attack success rises with complexity, and that some models fail only at the hardest level, offering a new way to locate safety boundaries.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Level labels are assigned per attack family with a disjunctive OR rubric, so the monotonic ASR pattern is confounded with attack-family identity and cannot yet be attributed to a single reasoning-complexity dimension.","rationale":"The reader's conditionality is appropriate. The paper is a useful benchmark contribution with released code and data, and the observed monotonicity is plausible, but the central construct—'reasoning complexity' as a single quantified dimension—is not established by the current design. Section 2's rubric is disjunctive, and Table 1's level assignments are properties of attack families, so the level effect in Table 2 is confounded with family identity. The paper's own Section 4.4 attributions to novelty and experimental setup show that non-complexity factors matter. Because the reader already flagged the unvalidated level labels as the weakest assumption, my analysis agrees and does not move the verdict. The proposed within-family complexity manipulation would directly settle whether the monotonic ASR pattern is caused by reasoning complexity or by the particular attack families placed at each level.","tokens_in":27471,"tokens_out":5662,"duration_ms":66883,"concrete_test":"For 3 attack families (e.g., Text Shuffle, Template Embedding, ASCII Drawing), take 100 base harmful instructions and generate Level 2 and Level 3 variants that vary only the stated complexity dimension (shuffle ratio, number of embedded reasoning steps/turns, encoding depth) while holding novelty and prompt length roughly fixed. Have at least 3 blinded annotators assign levels and rate the three rubric dimensions; report inter-rater reliability (e.g., Krippendorff's alpha). Run the same model set as Table 2 and test whether ASR increases monotonically with within-family level. If it does not, the Table 2 monotonicity is an artifact of attack-family identity rather than reasoning complexity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the L1/L2/L3 ordering reflects 'reasoning complexity,' but the paper's design does not identify that construct separately from the attack families chosen for each level. Section 2 defines Level 3 as meeting any one of three conditions OR meeting two Level 2 factors: 'When a type of attack meets any of these conditions, or meets the two different factors in level2, we attribute it to level 3.' This is a disjunctive category, not a scalar or ordinal quantity, so 'higher complexity' is not a well-defined single dimension. More concretely, Table 1 assigns each attack family to exactly one level; no family appears at multiple complexity levels. Thus the level variable is perfectly confounded with attack-family identity. The monotonic ASR increase in Table 2 could be driven by properties of the selected L3 families—novelty, prompt length, code/technical framing, or Chinese-specific formats—rather than by higher reasoning complexity. Section 4.4 itself credits 'novelty' for the Template attack's high ASR and 'experimental setup' for Multi-turn's low ASR, confirming that non-complexity factors influence these results. Additionally, the QWQ-32B judge is unvalidated and non-conforming trials are discarded (Section 4.2), which could bias level comparisons if discard rates differ by level. Without a within-family manipulation of complexity, the headline monotonic pattern is not uniquely attributable to the paper's core construct.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces Strata-Sword, a bilingual (Chinese-English) jailbreak safety benchmark organized by a proposed 'reasoning complexity' dimension with three levels: Basic Instruction (L1), Simple Reasoning (L2), and Complex Reasoning (L3), defined through logical depth, linguistic ambiguity, and task overhead. It classifies 15 attack methods (including novel Chinese-specific attacks such as character disassembly, lantern riddles, and acrostic poems), evaluates 23 open- and closed-source LLMs/LRMs, and reports that attack success rate (ASR) increases with level in both Chinese and English. The paper interprets this monotonic pattern as evidence that current models' safety alignment is shallow and that LRM-style reasoning may help close the gap.","tokens_in":27836,"tokens_out":3092,"duration_ms":35927,"significance":"If the level taxonomy is valid, Strata-Sword would fill a real gap: existing safety benchmarks mostly aggregate attacks by content or method, not by the cognitive load imposed on the target model. The publicly released dataset, the inclusion of culturally specific Chinese attacks, and the broad model coverage are concrete strengths. The central diagnostic claim—that ASR rises with reasoning complexity and hence reveals a model's safety boundary—is, however, not yet supported by the evidence as presented. The confound between level labels and attack-family identity, the unvalidated LLM judge, and the absence of statistical validation mean that the headline result is plausible but not established.","major_comments":[{"comment":"The level taxonomy is the load-bearing construct, but it is confounded with attack-family identity. Each attack family is assigned to exactly one level, and Level 3 is defined disjunctively ('meets any of these conditions, or meets the two different factors in level2'). No attack appears at multiple levels, and no within-family complexity manipulation is reported. Consequently, the monotonic ASR pattern in Table 2 could be driven by prompt length, novelty, code/technical framing, or Chinese-specific formatting rather than by a single 'reasoning complexity' dimension. The paper needs either per-instance complexity ratings with inter-rater reliability, multiple variants of the same attack family at different levels, or a covariates-controlled analysis (e.g., regression with prompt length and novelty) to attribute the effect to complexity.","section":"Section 2, Table 1"},{"comment":"The evaluation relies on QWQ-32B as an LLM judge, but no human agreement check or calibration is reported. Moreover, non-conforming judge outputs are discarded after five re-queries; if the discard rate differs across levels or models, the reported ASR values are biased in unknown directions. The authors should report per-level and per-model discard rates, provide a human-validated subset with agreement metrics, and show that the main monotonic pattern survives under alternative judgment criteria (e.g., conservative handling of refusals).","section":"Section 4.2"},{"comment":"The central claim 'ASR increases with attack level' rests on aggregate row means without confidence intervals, error bars, or significance tests. The paper itself acknowledges within-level outliers and non-complexity explanations: Template's high L2 ASR is attributed to novelty, and Multi-turn's low L3 ASR is attributed to the concatenated input setup. These admissions show that attack-family properties, not just complexity, drive the numbers. At minimum, the authors should temper the causal language and add a per-attack breakdown or a mixed-effects model separating level from family.","section":"Section 4.3, Section 4.4"}],"minor_comments":[{"comment":"The Level 3 definition begins 'reasoning complexity is simple'—this appears to be a typo for 'complex' and should be corrected.","section":"Section 2"},{"comment":"The sentence comparing DeepSeek-R1-14B with itself ('DeepSeek-R1-14B also exposes fewer vulnerabilities than DeepSeek-R1-14B') is clearly miswritten; presumably the comparison is between 7B and 14B versions.","section":"Section 4.3, Insight 3"},{"comment":"The text says Level 3 includes 'seven types' but then lists eight attack types (CodeAttack, DrAttack, Character Disassembly, Acrostic Poem, Lantern Riddle, Script+Template, Multi-turn, Text Shuffle+Template). The counts '100 Chinese and 200 English jailbreak instructions' are also not fully derived from the listed per-type counts; please clarify the dataset construction.","section":"Table 1, Section 3"},{"comment":"The sentence about 'the three Chinese attack methods we proposed: Lantern Riddle, Acrostic Poem, and Chinese Acrostic Poem Attack' likely refers to Lantern Riddle, Acrostic Poem, and Character Disassembly; the naming is inconsistent.","section":"Section 4.4, Figure 2"},{"comment":"Model names are inconsistent (e.g., 'Meta-Llama-3.1-70B-Instruct' vs 'Meta-Llama-3.1-70B'); please standardize the model identifiers.","section":"Section 4.1 and Table 2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark release and the Chinese-specific attack designs are genuinely useful contributions. The main risk is overclaiming from a confounded design; if the authors add a within-family complexity control or per-instance validation of the taxonomy, the paper could become a solid benchmark contribution. I am not suggesting rejection, but the central interpretational claim needs substantial additional evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: Strata-Sword is a usable bilingual jailbreak benchmark with three genuinely new Chinese-specific attacks, and the reported pattern (ASR rising from L1 to L3 across nearly all models) is visible in the tables. But the paper's central claim — that this rise is driven by 'reasoning complexity' — is not supported by the design, because level is assigned per attack family and the level definitions are a disjunctive rubric. In other words, L1/L2/L3 labels are perfectly confounded with which attack family generated the prompt. Novelty, token length, language-specific formatting, or code framing could just as easily produce the monotonic trend. The paper half-admits this in Section 4.4, where it attributes Template's high ASR to novelty and Multi-turn's low ASR to the concatenated experimental setup. Those are non-complexity factors.\n\nWhat's actually good: the benchmark is released on GitHub, HuggingFace, and ModelScope, the three Chinese attacks (character disassembly, lantern riddle, acrostic poem) appear to be new and clearly usable, and the evaluation spans 23 open-source and closed models with a reasonable set of risk categories. The paper is honest about some limitations, and the writing is readable.\n\nThe soft spots beyond the confound: there are no error bars or significance tests; the ASR differences are large enough that the monotonic pattern probably isn't noise, but that is an empirical judgment, not a statistical one. The QWQ-32B judge is unvalidated against human labels, and non-conforming judge outputs are discarded — if discard rates differ by level, that biases the comparison. The rubric itself isn't validated (no inter-rater reliability), and one definition line says Level 3's 'reasoning complexity is simple,' which looks like a typo but makes the taxonomy harder to trust. The references also contain a missing citation mark '?' in the introduction.\n\nNone of this makes the paper a waste of time. The benchmark is a useful resource, and the new Chinese attack suite is a real contribution. But the headline result should be treated as 'harder/novel attack families get higher ASR,' not as evidence for a single quantified complexity axis. A careful revision could fix this by adding a within-family complexity manipulation (same attack method at two levels) or, at minimum, by re-analyzing per-attack-family ASR with confounds like length and novelty controlled, and by validating the judge against human labels.\n\nFor a reader in safety evaluation or red teaming, this is worth knowing about. It deserves peer review — not a desk reject — but I'd send it back for major revision, asking for taxonomy validation and a more careful causal story.","headline":"Useful bilingual jailbreak benchmark with genuinely new Chinese attacks, but the central 'reasoning complexity' axis is confounded with attack-family identity, so the headline monotonic pattern is not yet attributable to the paper's core construct.","tokens_in":28306,"tokens_out":2753,"would_cite":true,"duration_ms":31117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that jailbreak attack success rises with the reasoning complexity of the prompt, and proposes Strata-Sword, a three-level Chinese-English benchmark that diagnoses LLM safety boundaries.","keywords":["LLM safety evaluation","jailbreak attacks","reasoning complexity","hierarchical benchmark","safety alignment","Chinese-English evaluation","attack success rate","large reasoning models"],"falsifier":"Take the 15 attack families and have independent annotators, blind to the paper's labels, sort them by the number of inference steps a reader must perform to recover the harmful intent (or measure this with a parser that counts resolution distance and dependency depth). Then re-run the 23-model evaluation. The central claim predicts attack success should increase monotonically across the annotator-derived levels; if it does not—or if a simple padded long-text version of a Level-1 prompt, matched for token count to Level-3 prompts, already produces Level-3-level success—the confound of length/n","tokens_in":27399,"feed_emoji":"🛡️","tokens_out":7531,"duration_ms":81680,"temperature":0.7,"pith_summary":"The paper argues that jailbreak instructions have a measurable property—reasoning complexity—that existing safety benchmarks ignore, and that this property determines how easily a model can be made to produce harmful output. It sorts 15 attack methods, drawn from existing jailbreak literature plus three new Chinese-language attacks, into three levels: direct harmful requests, requests needing simple reasoning, and requests needing complex, multi-step or culturally embedded reasoning. On a bilingual (Chinese and English) benchmark built this way, the paper reports attack success rates rising steadily from Level 1 to Level 3 across open- and closed-source LLMs and reasoning models. The value, if the claim holds, is diagnostic: a model that refuses a direct harmful prompt may still comply when the same intent is hidden behind enough ambiguity, task overhead, or logical indirection, so measuring safety at one complexity level is insufficient.","feed_headline":"Jailbreak success climbs with prompt complexity","feed_subtitle":"A three-level bilingual benchmark shows where safety alignment stops holding across 23 models.","key_machinery":"The three-part definition of reasoning complexity—logical depth, linguistic ambiguity, and task overhead—together with the L1/L2/L3 rubric that maps 15 attack families onto it. The rubric is the load-bearing mechanism: it turns an intuition about harder prompts into an ordered scale, and the scale is what makes a model's safety boundary visible as a curve rather than a single number.","core_discovery":"Strata-Sword's central claim is that safety behavior should be evaluated as a function of instruction complexity, not just harmful content. The paper defines reasoning complexity along three axes—logical depth (how many steps of inference), linguistic ambiguity (how much pragmatic work is needed to recover intent), and task overhead (extra computational or knowledge load layered on top)—and uses these axes to bucket 15 jailbreak attack families into Level 1 (basic, intent exposed), Level 2 (simple reasoning: one or two steps, partial ambiguity, or light overhead), and Level 3 (complex reasoning: multi-step logic, deep ambiguity, professional knowledge, or any combination of two Level-2 facto","pith_inferences":["Editorial extension: if the monotone attack-success-versus-level pattern is robust, the benchmark could be inverted into a diagnostic probe—start at Level 1 and escalate until the model fails—giving each deployment a safety ceiling that an aggregate score cannot express.","Editorial extension: the rubric could be made quantitative, for example by measuring parse-tree depth, pronoun-resolution distance, or the number of intermediate operations needed to see the harm, so the L1/L2/L3 assignments become predictions rather than manual labels; the paper's Table 1 is the first step but not the measurement.","Editorial extension: the Chinese-specific attacks suggest a broader script-level attack surface—any writing system with separable components or wordplay, such as kanji, Arabic morphology, or emoji rebuses, may admit analogous attacks not covered by English-centric benchmarks.","Editorial extension: the level structure suggests a testable training recipe—augment safety data by re-expressing the same harmful intents at higher complexity levels—which should flatten the attack-success curve if the diagnosis is correct."],"forward_implications":["Safety benchmarks that report only an aggregate attack success rate will hide the failure mode Strata-Sword exposes: a model can look aligned on direct prompts while failing badly on complex ones, so level-wise reporting becomes the informative unit.","A model that stops only Level-1 attacks has at best surface-level alignment; fixing it means training on intent reconstruction and complex-prompt refusal, not just adding more refusal examples of direct harms.","For reasoning models, the benchmark suggests that chain-of-thought can cut both ways: the same capacity that lets a model infer user intent accurately can be exploited by complex jailbreaks, so reasoning-model-specific safety alignment is needed.","Language-specific attack surfaces matter: the Chinese character-disassembly, lantern-riddle, and acrostic-poem attacks succeed against models that robustly reject the same harms stated plainly, implying safety alignment should be audited per language, not globally.","English-heavy safety fine-tuning can improve English robustness while leaving Chinese vulnerabilities—the paper reports this pattern for a fine-tuned model family—so multilingual safety requires language-balanced training data."],"supporting_citations":[{"why":"Supplies the 100 basic harmful instructions (AdvBench) used as Level-1 seeds in English and, after translation, in Chinese.","marker":"Zou et al., 2023"},{"why":"Cited as a typical Level-1 jailbreak benchmark, grounding the simplest complexity tier.","marker":"Mazeika et al., 2024"},{"why":"Cited for Level-2 DoAnythingNow prompts and used as the out-of-distribution comparison for template attacks.","marker":"Shen et al., 2024"},{"why":"Provides the ASCII-drawing attack used as a Level-2 method with simple task overhead.","marker":"Jiang et al., 2024"},{"why":"Supplies the text-shuffle attack used at Level 2 and combined into the Level-3 Shuffle+Template attack.","marker":"Zhao et al., 2025"},{"why":"Supplies the code-based attack used as the most effective Level-3 English attack.","marker":"Kang et al., 2024"},{"why":"Supplies the multi-round dialogue attack placed at Level 3 for its multi-step logical depth.","marker":"Wang et al., 2024"},{"why":"Supplies DrAttack, a two-round dialogue plus template embedding attack categorized at Level 3.","marker":"Liu et al., 2024b"},{"why":"Supplies the long-text attack used as a Level-2 method that drowns harmful content in context.","marker":"Anil et al., 2024"}],"fun_headline_variants":["Benchmark ranks jailbreak attacks by reasoning complexity","Jailbreak success tied to prompt reasoning depth","Hierarchical benchmark exposes LLM safety boundaries","Complex jailbreaks reveal LLM safety limits"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"Everything hangs on the manual L1/L2/L3 labels genuinely ordering attacks by reasoning complexity rather than by confounds such as prompt length, novelty, or language difficulty.","fun_headline_variants_meta":{"raw":{"variants":["Benchmark ranks jailbreak attacks by reasoning complexity","Jailbreak success tied to prompt reasoning depth","Hierarchical benchmark exposes LLM safety boundaries","Complex jailbreaks reveal LLM safety limits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1502,"prompt_tokens":821,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":636}},"tokens_in":565,"tokens_out":681,"duration_ms":8119,"temperature":1.0,"reasoning_tokens":636,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:30:17.784756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 15 attack families and have independent annotators, blind to the paper's labels, sort them by the number of inference steps a reader must perform to recover the harmful intent (or measure this with a parser that counts resolution distance and dependency depth). Then re-run the 23-model evaluation. The central claim predicts attack success should increase monotonically across the annotator-derived levels; if it does not—or if a simple padded long-text version of a Level-1 prompt, matched for token count to Level-3 prompts, already produces Level-3-level success—the confound of length/n","supporting_citations":[],"review_version":1}