{"id":"58c258d6-70e3-4b78-8656-ccc4a01c52d3","arxiv_id":"2508.14128","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A dual-track core-extraction defense reports near-zero jailbreak success on four attacks with preserved benign utility, but its safety-classifier component is undisclosed.","lead":"CCFC is a new prompt-level defense for large language models that first extracts the core question from a user prompt, then runs two parallel safety checks before answering. It reports near-zero jailbreak success rates on four attacks, but the paper's safety filter is unspecified and some reported numbers conflict with its own tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dual-track claim is unsupported because S in Eq. (5) is undisclosed and may replicate the Dic-Judge ASR metric; an ablation removing S is needed.","rationale":"The reader's REJECT verdict is well-founded, and the weakest assumption identified—the unspecified safety classifier S in Eq. (5)—is the most load-bearing issue. The headline claim is that the dual-track architecture itself is responsible for a 50-75% ASR reduction. But the selection logic in Eq. (5) lets a single unsafe label from S reject the entire response, and S is never defined anywhere in the manuscript. The evaluation metric is Dic-Judge, a keyword-based refusal detector; if S is implemented with similar or overlapping keywords, then the defense and the metric are effectively the same filter, and the near-zero ASR is expected regardless of core extraction or structure disruption. The absence of any ablation with S removed makes it impossible to attribute the results to the dual tracks. The internal inconsistency between the main text and Table 1 on three Vicuna attack numbers compounds the problem: it suggests the reported numbers may not be reliable enough to support the central claim. A direct ablation plus an independent judge would settle whether the dual-track mechanism carries the defense; until that is provided, the claim that CCFC outperforms baselines because of its dual-track design is not established. This does not change the reader's verdict; it reinforces the REJECT decision.","tokens_in":11204,"tokens_out":4752,"duration_ms":52708,"concrete_test":"Run the full CCFC pipeline on all four attacks and both models with the second-stage classifier from Eq. (5) disabled, i.e., set S(·)=1 for all outputs so only the two tracks and the final RCFC selection remain. Measure ASR with both Dic-Judge and an independent judge such as GPT-4 or human annotation. If ASR rises materially above the reported 0-6% under Dic-Judge, the claimed defense success is attributable to the undisclosed S rather than the dual-track structure; if ASR remains near zero under both judges, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the dual-track architecture—core-only plus core-full-core—is what reduces attack success. However, Eq. (5) includes a binary safety classifier S that can unilaterally force a refusal if either track's output is labeled unsafe. S is never specified in Section 4, in the experimental setup, or in the appendix examples, where fields such as 'core_harmful' and 'aug_harmful' appear with no indication of how they were computed. The ASR metric used in Table 1 is Dic-Judge (Zou et al., 2023), a keyword-based refusal detector that marks an output safe if it contains refusal phrases. If S shares that keyword logic, the defense and the evaluation are measuring the same signal, making the near-zero ASR partly circular. Even if S is a separate classifier, the paper provides no ablation with S removed or fixed to 'safe', so there is no evidence that the dual tracks themselves, rather than this hidden filter, drive the reported robustness. The main text also contradicts Table 1 on the Vicuna results (text: 2% GCG, 4% AutoDAN/PAIR; table: 6% GCG, 0% AutoDAN, 2% PAIR), further undermining the reliability of the headline numbers. Without specifying S and isolating its contribution, the comparison against prompt-level baselines is not a fair test of the proposed mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CCFC, a prompt-level defense for LLM jailbreak attacks. The method first extracts a \"core question\" from the user prompt using few-shot prompting, then runs two parallel tracks: a core-only track (C) and a core-full-core track (CFC) that concatenates core, full prompt, and core again. A double safety check (refusal detection plus a binary harmful-content classifier S) decides the final output, refusing if either track refuses or is flagged unsafe, and otherwise returning the CFC response. The authors evaluate on Vicuna-7B and LLaMA2-7B-chat against GCG, AutoDAN, PAIR, DeepInception, and AdvBench, reporting near-zero attack success rates and claiming 50-75% ASR reduction over state-of-the-art defenses while preserving benign utility as measured by Just-Eval.","tokens_in":11505,"tokens_out":3853,"duration_ms":40486,"significance":"If the dual-track mechanism itself were responsible for the reported robustness, CCFC would be a practically attractive, model-agnostic, prompt-level defense with low overhead. The paper's core claim, however, is not established in the current manuscript: the harmful-content classifier S in Eq. (5) is never specified, no ablation isolates the contribution of the dual-track logic from the contribution of S, and the headline numbers are inconsistent between the main text and Table 1. These are load-bearing issues because the reported near-zero ASR could be entirely attributable to an undisclosed safety filter rather than to the proposed architecture. The utility results are useful but secondary; without a credible causal attribution, the paper's central contribution is unverified.","major_comments":[{"comment":"The binary safety classifier S that implements the second stage of the double safety check is never specified. The paper does not state what model or algorithm S is, how it is trained, what features it uses, or how the fields core_harmful and aug_harmful in the Appendix examples (Tables 3 and 4) are computed. Because Eq. (5) lets S unilaterally force a refusal if either track's output is labeled unsafe, the reported 0-6% ASR in Table 1 could in principle be produced by S alone, with the dual-track architecture contributing nothing. The authors must disclose the exact implementation of S and, critically, provide an ablation in which S is removed (or fixed to 'safe') so that the reader can see what the dual-track logic alone achieves.","section":"Section 4, Eq. (5)"},{"comment":"There is a potential circularity between the defense and the evaluation. ASR is computed with Dic-Judge, a keyword-based refusal detector that classifies an output as safe if it contains refusal phrases such as 'Sorry, I cannot...'. The defense's own decision rule (Eq. 5) triggers on refusal-like patterns RD and, in the Appendix examples, the final output is the literal string 'Sorry, I cannot answer that question.' If S uses the same or overlapping keyword logic as Dic-Judge, then the defense is effectively applying the evaluation metric to its own outputs, making the near-zero ASR partly circular by construction. The authors should disclose whether S shares vocabulary or rules with Dic-Judge and re-evaluate with an independent judge (e.g., Llama Guard, GPT-4-based judge, or human annotation).","section":"Section 5, Evaluation Metrics; Section 4, Eq. (5)"},{"comment":"The main text and Table 1 disagree on the Vicuna results. The text states that CCFC achieves 2% ASR for GCG and 4% for both AutoDAN and PAIR, whereas Table 1 reports 6% for GCG, 0% for AutoDAN, and 2% for PAIR. This discrepancy undermines confidence in the reported numbers. In addition, the abstract's claim of 'cuts attack success rates by 50-75% versus state-of-the-art defenses' is not clearly supported by Table 1: for example, on Vicuna PAIR, CCFC (2%) does not improve over DATDP (2%), and on Llama2 PAIR, CCFC (2%) is worse than DATDP (0%). The authors should fix the inconsistency and specify exactly which baseline-attack pairs the 50-75% figure refers to.","section":"Section 5, Experimental Results; Table 1"}],"minor_comments":[{"comment":"The few-shot demonstration set D_examples is a key component but is not described in the main text besides two examples in the Appendix. The number of demonstrations, their selection, and their diversity should be documented, and ideally a sensitivity analysis should be provided.","section":"Section 4, Eq. (2)"},{"comment":"Just-Eval scores are reported as averages without variance, confidence intervals, or significance tests. With 500 instructions, the differences between CCFC and the undefended baseline (e.g., Vicuna average 4.353 vs. 4.339) are likely within noise; the authors should report standard deviations or statistical tests.","section":"Section 5, Table 2"},{"comment":"The appendix is titled 'Detailed configurations' but only contains two defense examples. It does not provide the configurations for the attack baselines (GCG, AutoDAN, PAIR, DeepInception), the defense baselines, or the unspecified classifier S. These should be added or referenced to a public repository.","section":"Appendix 8"},{"comment":"The notation 'S(R_C) or S(R_CFC) = 0' is ambiguous because it mixes a logical 'or' over binary outputs with a numeric equality. Rewrite to make clear that a refusal is triggered if S(R_C)=0 or S(R_CFC)=0.","section":"Section 4, Eq. (5)"},{"comment":"The abstract and introduction claim CCFC 'cuts attack success rates by 50-75%' but no precise derivation of this figure is given in the experimental section. Please specify the comparison baseline and the subset of attacks for which this holds.","section":"Section 1, Abstract"}],"recommendation":"reject","confidential_remarks":"The paper's central empirical claim is not verifiable in its current form: the classifier S is a black box that can unilaterally produce the reported safety outcomes, and no ablation separates its effect from the dual-track mechanism. The main-text/Table 1 discrepancy on Vicuna numbers and the vague 50-75% claim further reduce confidence in the authors' reporting. I would be willing to reconsider a revised version that fully specifies S, adds an ablation without S, uses an evaluation judge with no shared logic, and corrects the numerical inconsistency."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: CCFC is an interesting engineering combination, but the paper doesn't support its headline claim because the safety classifier in the second check is never specified. The near-zero ASR may be coming from that hidden filter rather than the dual-track mechanism.\n\nWhat's actually new: the assembled architecture — few-shot core extraction, a core-only track, a core-full-core sandwich, and a double-check — is a new combination, and the utility preservation on Just-Eval is genuinely decent. The core extraction examples in the appendix are nicely designed, and the idea of re-inserting the extracted core around the full prompt to break structural patterns is worth testing.\n\nWhere it falls down: the stress-test note is right. Equation (5) includes S, a binary safety classifier that can unilaterally force a refusal. S is never described: not in Section 4, not in the experimental setup, not in the appendix. The appendix examples list `core_harmful` and `aug_harmful` flags but don't say how they were computed. Meanwhile, ASR is measured with Dic-Judge, a keyword-based refusal detector. If S uses the same or overlapping refusal phrases, the defense is literally applying the evaluation metric to its own outputs. That's circular, or at least it makes the contribution of the dual tracks impossible to isolate. An ablation with S removed or fixed to \"safe\" is essential, and it's missing.\n\nThere's also a direct inconsistency: the main text says Vicuna ASR is 2% GCG, 4% AutoDAN, 4% PAIR, but Table 1 shows 6%, 0%, 2%. That kind of mismatch on the headline numbers makes me trust the tables less, not more.\n\nTo be fair: the dual-track mechanism itself is not nonsense, and the Llama-2 results mostly mirror DATDP, which suggests the added value of CCFC may be modest. But without specifying S and isolating its effect, the comparison against prompt-level baselines isn't a fair test of the proposed mechanism.\n\nWho this is for: someone working on LLM safety defenses who wants to see the landscape of prompt-level tricks. It doesn't deserve publication as-is. A serious referee could push for the missing details and a proper ablation, so I'd send it to review rather than desk-reject, but the revision needed is substantial.\n\nRecommendation: engage as a referee only if you have time to demand the S specification and an ablation; otherwise skip.","headline":"An interesting but under-specified prompt-level defense whose near-zero attack success rates are not attributable to the proposed mechanism until the hidden safety classifier S is disclosed and ablated.","tokens_in":12056,"tokens_out":3529,"would_cite":false,"duration_ms":31358,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-track defense that strips queries to their semantic core and double-checks both answers cuts jailbreak success rates to near zero while preserving response quality.","keywords":["LLM jailbreak defense","prompt-level defense","core extraction","dual-track architecture","safety consistency check","few-shot prompting","adversarial suffix attacks","roleplay jailbreak"],"falsifier":"Re-run CCFC against the same four attacks with the internal safety classifier replaced by an independent judge whose labels are never used to compute attack success, and add an adaptive attacker who has access to the core-extraction prompt; if attack success stays near zero, the dual-track mechanism is responsible, and if it rises toward the undefended baseline, the reported gains came from the classifier or from non-adaptive attack templates.","tokens_in":11013,"feed_emoji":"🛡️","tokens_out":10144,"duration_ms":90894,"temperature":0.7,"pith_summary":"CCFC is a prompt-level defense that tries to make large language models refuse harmful requests by transforming the input rather than retraining the model. It first extracts the semantic core of each user query using a small set of demonstration examples, then runs the query through two parallel tracks: the core alone, and the core wrapped around the full original prompt. A two-stage check rejects the output if either track refuses or produces harmful content, and otherwise keeps the richer core-full-core answer. The paper reports that on Vicuna and Llama 2 this reduces attack success rates by 50–75% compared with state-of-the-art prompt-level defenses, bringing GCG, AutoDAN, PAIR, and DeepInception to 0–6% while keeping Just-Eval quality scores close to the undefended baseline. The claim is that semantic distillation plus structural disruption plus consensus checking is enough to block both gradient-suffix and roleplay jailbreaks without white-box access or fine-tuning.","feed_headline":"A dual-track defense cuts LLM jailbreak success to near zero","feed_subtitle":"Stripping queries to their semantic core and double-checking answers blocks attacks without hurting benign responses.","key_machinery":"The load-bearing mechanism is the dual-track prompt construction together with the double safety check. A few-shot prompt, built from adversarial-to-benign demonstration pairs, distills the user's full prompt $P$ into a core question $Q_c$; that extraction runs on the target model itself. Track A queries $M(Q_c)$, and Track B queries $M([Q_c \\parallel P \\parallel Q_c])$, sandwiching the full prompt between two copies of the core to break structural attack patterns while preserving context. The selection rule in Eq. (5) then refuses if either track's response contains an explicit refusal, refuses if a binary safety classifier $S$ labels either response unsafe, and otherwise returns the Track B answer. The core track is the fallback against obfuscation, the core-full-core track is the fidelity-preserving structural disruptor, and the double check is what makes the conservative choice safe.","core_discovery":"The central discovery, on the paper's own terms, is that a jailbreak succeeds only when an adversarial wrapper can hide a prohibited instruction inside a context the model is willing to follow, and CCFC attacks that condition from two complementary directions. The core track answers only the extracted question $Q_c$, so adversarial suffixes, roleplay frames, and injected instructions are simply gone. The core-full-core track answers $[Q_c \\parallel P \\parallel Q_c]$, which keeps the original context for fidelity while sandwiching it between two copies of the distilled intent, a pattern the paper says disrupts structure-dependent attacks such as gradient-optimized suffixes. The final answer is selected by refusal detection followed by a harmful-content check on both responses, and only when both are safe is the full-context answer returned. In the reported experiments this brings GCG, AutoDAN, PAIR, and DeepInception to 0–6% attack success on Vicuna and to 0–2% on Llama 2, with Just-Eval averages within a few hundredths of the undefended models.","pith_inferences":["The paper does not specify the binary safety classifier $S$ used in Eq. (5); we infer that the reported near-zero attack-success rates are only interpretable if $S$ is chosen independently of the keyword-based judge used to measure attack success.","We infer that an adaptive attacker who knows the core-extraction prompt and can obfuscate the distilled core itself is the natural next stress test; the current experiments use fixed attack templates rather than attacks designed against CCFC.","The consistency-check logic generalizes to more than two tracks, so a natural extension is to specialize each track to one attack family such as roleplay, injection, or suffix; the paper lists this direction as future work, and our inference is that the double-check would still decide among any number of parallel answers."],"forward_implications":["If CCFC works as reported, prompt-level defenses can reach near-zero jailbreak success without gradient access, fine-tuning, or an external safety model, which matters for closed-source APIs.","The same two-forward-pass structure would give any instruction-following LLM a drop-in safety layer: extract the core, answer twice, compare, and reject if either side refuses.","Because the full prompt is preserved in the CFC track, benign queries should keep their original informativeness, so the defense does not force a trade-off between safety and helpfulness.","The reported robustness across four very different attack families suggests core distillation targets a common vulnerability—the wrapper that hides the intent—rather than each attack's specific surface pattern."],"supporting_citations":[{"why":"It supplies the GCG attack, the AdvBench harmful benchmark, and the Dic-Judge keyword refusal classifier used to compute attack success rates.","marker":"(Zou et al., 2023)"},{"why":"It supplies the PAIR attack, an evaluated jailbreak.","marker":"(Chao et al., 2025)"},{"why":"It supplies the AutoDAN attack, an evaluated jailbreak.","marker":"(Liu et al., 2023a)"},{"why":"It supplies the DeepInception roleplay attack, an evaluated jailbreak.","marker":"(Li et al., 2023)"},{"why":"It supplies the Paraphrase, Retokenization, and ICD baseline defenses compared in Table 1.","marker":"(Wei et al., 2023)"},{"why":"It supplies the Self-Reminder baseline defense.","marker":"(Xie et al., 2023)"},{"why":"It supplies the DATDP baseline defense, the strongest competitor in the tables.","marker":"(Armstrong et al., 2025)"},{"why":"It supplies the Vicuna-7B target model used in the experiments.","marker":"(Chiang et al., 2023)"},{"why":"It supplies the LLaMA2-7B-chat target model used in the experiments.","marker":"(Touvron et al., 2023)"},{"why":"It supplies the Just-Eval benchmark used to measure response quality on benign queries.","marker":"(Zheng et al., 2023)"}],"fun_headline_variants":["Dual track isolates core intent, slashing jailbreak success","CCFC dual-track defense cuts jailbreak attacks by up to 75%","Stripping queries to core intent blocks most LLM jailbreaks","New defense halves jailbreak success by core-only checking","Core distillation plus dual-track check neutralizes jailbreaks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result rests on an unspecified automated safety check that labels each track's answer safe or unsafe; if that check is the same keyword-based judge used to measure attack success, the near-zero attack rates could be circular rather than caused by the dual-track structure.","fun_headline_variants_meta":{"raw":{"variants":["Dual track isolates core intent, slashing jailbreak success","CCFC dual-track defense cuts jailbreak attacks by up to 75%","Stripping queries to core intent blocks most LLM jailbreaks","New defense halves jailbreak success by core-only checking","Core distillation plus dual-track check neutralizes jailbreaks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000856,"raw_usage":{"total_tokens":3727,"prompt_tokens":963,"completion_tokens":2764,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":2678}},"tokens_in":579,"tokens_out":2764,"duration_ms":19204,"temperature":1.0,"reasoning_tokens":2678,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:13:06.394319+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run CCFC against the same four attacks with the internal safety classifier replaced by an independent judge whose labels are never used to compute attack success, and add an adaptive attacker who has access to the core-extraction prompt; if attack success stays near zero, the dual-track mechanism is responsible, and if it rises toward the undefended baseline, the reported gains came from the classifier or from non-adaptive attack templates.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the PAIR attack, an evaluated jailbreak."}],"review_version":1}