Pith. sign in

REVIEW 3 major objections 5 minor 70 references

Routing an LLM's first five output tokens between a helpful and a refusing expert cuts attack success sharply while preserving usability, across eleven models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:02 UTC pith:SS7WK56Y

load-bearing objection A well-executed extension of MoGU with broad empirical coverage, but the security claim rests on non-adaptive baselines—an adaptive attacker could likely bypass the tiny router. the 3 major comments →

arxiv 2509.06807 v1 pith:SS7WK56Y submitted 2025-09-08 cs.CL

MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security

Yanrui Du , Fenglei Fan , Sendong Zhao , Jiawei Cao , Ting Liu , Bing Qin This is my paper
classification cs.CL
keywords LLM securityjailbreak defenserouter mechanismmixture of expertsLoRA fine-tuninginstruction fine-tuning attacksusability-security tradeoffPareto frontier
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that an LLM can be made much safer without becoming cautious or wooden: instead of retraining the model to refuse, it creates two cheap variants of it—one that answers anything helpfully, one that refuses everything—and lets a small router read the model's own hidden states to decide, for each of the first five output tokens, which variant should speak. The base model generates everything after the fifth token, so the defense costs almost nothing at inference time. Across eleven models spanning mainstream chat LLMs, small on-device models, and chain-of-thought reasoning models, the authors report large reductions in harmfulness score and attack success rate while usability scores stay within a few hundredths of the unmodified model—an outcome they describe as advancing the Pareto frontier between the two goals rather than trading one against the other. The same router, applied after instruction fine-tuning with a simple data-mix step, restores security to or beyond base-LLM levels without giving back the task-performance gains. If the paper is right, security is less a matter of how the model was aligned and more a matter of routing the opening of the reply.

Core claim

MoGUv2's central claim is that a model's decision to comply or refuse is settled in its first few output tokens, and that a lightweight router can be trained to make that call correctly without distorting the rest of the response. The base model is cloned into two LoRA-tuned personalities: Gladresp, which answers even malicious instructions helpfully, and Unwillresp, which refuses even benign ones. A sigmoid MLP router with a low-rank core reads the input hidden state at each selected layer and outputs a per-token weight interpolating between the two personalities' outputs. MoGUv2 sharpens the original MoGU in two ways: routers are placed only in the latter half of the layers, where a probin

What carries the argument

The intra-layer router: a sigmoid MLP with a low-rank input decomposition that maps a layer's input hidden state to a scalar weight per token position, interpolating between the outputs of Gladresp and Unwillresp, two LoRA-tuned variants of the base model that respectively answer everything helpfully and refuse everything. The routed mixture runs only on the first five tokens of the response and only in the latter half of the layers, which the paper identifies via probing as the region where hidden states encode highly classifiable security features. During router optimization the Q/K projection modules are LoRA-activated, so the backbone and router co-adapt; during inference the rest of the

Load-bearing premise

Only the first five output tokens are rerouted; everything after them is generated by the base LLM with no intervention, so the defense silently does nothing unless the decision to comply or refuse is already settled inside those five tokens.

What would settle it

Build jailbreak prompts whose first five generated tokens are forced to be innocuous—a benign filler phrase such as 'Sure, here is' or an empty reasoning tag—with the harmful content beginning at token six or later. If harmfulness scores and attack success rates on such prompts climb back to base-LLM levels, the five-token assumption is the point of failure.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A one-time training pass on 600 instructions (300 benign, 300 malicious), retaining only the first sentence of each response, is enough to deploy the defense; the authors report under 30 minutes of training for 7B-scale models.
  • MoGUv2 adds roughly half the parameters of the original MoGU—about 140M for a 7B model and fewer than 30M for 3B-and-smaller devices—so the security gain does not break the lightweight budgets of on-device deployments.
  • Because only the first five tokens are decoded through the mixture, inference cost stays close to that of the base model, making the defense usable as a drop-in layer on already-deployed models.
  • In the instruction-fine-tuning scenario, applying MoGUv2 post-hoc with a 150-task + 150-general data mix brings harmfulness back to or below base-LLM levels (average HS about 1.25 versus 3.45 after IFT on Llama2-7B) while shifting task performance by less than one point.
  • The weight-distribution analysis implies the mechanism is inspectable: a user can see, per instruction, how much the router favors the refusing expert, which converts the defense into a running threat assessment.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Attackers who learn the five-token rule could try to hide harmful content until after the routed window; a direct stress test would prepend a generic filler opening and check whether the base LLM complies at token six onward.
  • The router weights double as a free, inspectable harmfulness signal—the paper shows they swing toward the refusing expert under attack—so they could be reused as a lightweight audit or detection side-channel.
  • The fixed window of five tokens could be made adaptive, with the router deciding when it is safe to hand off to the base model, which would extend the defense to responses that front-load benign-looking text.
  • For chain-of-thought models, the refusal-think finding suggests security depends on the format of the refusal tokens; a defense that also governs the reasoning tag, rather than only the answer tokens, may generalize better to long-reasoning attacks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper extends the MoGU defense (NeurIPS 2024) into MoGUv2. The base LLM is cloned into two LoRA variants, Gladresp (usability-optimized) and Unwillresp (security-optimized), and an intra-layer router mixes their hidden states. MoGUv2 places routers only in the latter half of the layers, activates Q/K modules with LoRA during router training, and during inference uses the MoGU path for only the first m=5 generated tokens, with the base LLM generating the remainder. The router is trained with a joint global classification loss and an L1 local routing loss on 600 instructions. The authors evaluate harmfulness score (HS) and attack success rate (ASR) against red-team and jailbreak benchmarks, and usability via GPT-4o scoring and refusal-frequency rules, across 11 LLMs spanning mainstream, on-device, and reasoning-oriented models. They also test MoGUv2 as a post-IFT defense using a data-mix strategy. The paper claims to advance the Pareto frontier between usability and security.

Significance. If the claims are substantiated, MoGUv2 would be a useful lightweight defense: it introduces modest parameter overhead, can be trained in under 30 minutes for 7B models, and the paper demonstrates consistent reductions in HS/ASR across many models while maintaining usability scores close to baseline. Strengths include the breadth of evaluation, the explicit ablation of Loss_cl, L1 norm, and Q/K activation, the layer-wise probing analysis, and the parameter-count comparison. The IFT scenario with a simple data-mix strategy is practically relevant. However, the central 'Pareto frontier' claim is not formally defined or measured, and the security evaluation does not include attacks that adapt to the router or exploit the 5-token intervention window. These gaps currently prevent the paper from supporting its strongest conclusions.

major comments (3)
  1. [Sec. V, Fig. 2, Tables II-VII] The central claim of 'advancing the Pareto frontier' is not defined or measured. The paper reports average HS/ASR and US/rule scores but never specifies what the Pareto frontier is, how it is estimated, or how MoGUv2 shifts it. In fact, Table II shows MoGUv2 does not dominate existing defenses: on Falcon7B, SafeDecode achieves HS 1.00 and ASR 0.79% while MoGUv2 achieves HS 1.28 and ASR 6.56%; on Llama2, ICD achieves HS 1.00 and ASR 0.00% vs MoGUv2 1.04 and 0.64%. The higher usability scores make these different trade-off points, not evidence of a frontier improvement. Please provide a formal definition and the supporting dominance or frontier-shift analysis.
  2. [Sec. III-A, Sec. IV-B, Sec. V-A3] The security evaluation uses only attacks designed to bypass the base LLM; no attack is designed to target the router. The router R is a small differentiable MLP (d_router=512 for 7B, 128 for smaller models) that must classify hidden states at the first tokens. An adaptive attacker could optimize a jailbreak prompt to make R assign low weight to Unwillresp, after which the base LLM continues from token 6. Because R is the load-bearing security component, at least one gradient-based or adaptive attack against the router, or an equivalent robustness analysis, is needed to support the claim that MoGUv2 improves security rather than merely shifting the attack surface.
  3. [Sec. IV-B, Sec. V-A5] The defense only modifies the first m=5 tokens before handing generation back to the base LLM. The paper cites prior work on initial-token importance but provides no ablation over m and no attack scenario where harmful content is delayed beyond token 5. If a jailbreak elicits a benign-looking prefix (e.g., 'Sure, here is...') for the first five tokens and then harmful content, MoGUv2 would not intervene. I recommend an ablation over m and an evaluation that displaces harmful content beyond the first five tokens.
minor comments (5)
  1. [Sec. IX (Appendix)] The data construction paragraph states 'The benign and malicious instructions are sourced from Advbench and just-eval, respectively', which contradicts Sec. III-B, where benign instructions come from Alpaca and malicious from AdvBench. Please correct the appendix.
  2. [Sec. VII-B, Table VIII] The ablation reports only ASR without confidence intervals or raw counts. Several differences are small (e.g., Llama2 0.64% vs 0.49% without Loss_cl), and the claims about component indispensability would be stronger with variance estimates or counts of successful attacks.
  3. [Throughout] There are typographical errors and inconsistent spacing, e.g., 'saf emixes' in Sec. I, 'intoduced' in Sec. VII-B, and inconsistent 'MoGUv2' vs 'MoGU v2' formatting.
  4. [Tables III, V, VI] The 'Rule eval' metric is described only as a 'rule-based metric'; please state explicitly in the captions that it is the percentage of responses containing refusal expressions, since this is used as a usability measure.
  5. [Fig. 7] The figure is hard to read: the bar labels are not clearly associated with 'with' vs 'without' data-mix, and the text mentions specific percentage drops that are not directly readable from the figure. Please clarify the legend and label values.

Circularity Check

0 steps flagged

No material circularity: MoGUv2's central claim is supported by held-out red-team and jailbreak evaluations, and self-citations to [1]/[19] are not load-bearing.

full rationale

MoGUv2's derivation chain is empirical rather than definitional. Gladresp and Unwillresp are LoRA variants trained with cross-entropy/contrastive losses on 600 constructed instruction/response pairs; the router is optimized with a global CE loss (Eq. 1) and a local L1 norm on routing weights; layer placement is informed by a probing classifier (Eq. 2) on a separate 100-pair split. The evaluated security metrics (GPT-Judge HS, ASR) are not the training losses, and the paper explicitly ensures no overlap between training and test samples: 'we strictly ensure that none of the test samples overlap with the training data constructed in Sec. III-B.' Jailbreak attacks (AutoDAN, GCG, PAIR, SAP30, Comp.) are not in the training set, so the reported improvements are not forced by construction. The probing/layer-selection loop is a design feedback loop, not a derivation that assumes the target result: the router still has to learn to generalize to unseen adversarial prompts, and the layer comparison (Sec. VII-C) is an empirical ablation. The 'first 5 tokens' design is an assumption inherited from external work [25,26,69], not from the authors' own prior work; it is a robustness concern (adaptive attacks could target the router) but not circularity. Self-citations to [1] (prior MoGU, used as a baseline) and [19] (IFT setup and a security-specific dataset used for probing) are normal scholarly references and do not carry the argument: the paper's central Pareto-frontier claim is independently evaluated against multiple external baselines and attack families. Overall, no step reduces to its own inputs by construction, so the paper is essentially self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

The method rests on standard LLM/LoRA assumptions and a few hand-chosen hyperparameters. The most fragile assumptions are the initial-token hypothesis (m=5) and the existence of classifiable security features in deeper layers. No new physical or conceptual entities are introduced beyond the router module.

free parameters (5)
  • lambda (weight on local routing loss) = 2
    Set by hand in Sec. V-A5; controls the trade-off between the global and local losses and affects router behavior.
  • number of initial tokens m = 5
    Only the first 5 tokens are decoded by MoGU (Sec. IV-B); based on prior work but a free choice that determines the scope and coverage of the defense.
  • router intermediate dimension d_router = 512 (7B), 128 (<3B)
    Chosen in Sec. V-A5; affects parameter count and router capacity.
  • LoRA rank d_lora and alpha = 8 and 16
    Set in Sec. V-A5; standard LoRA hyperparameters that control adaptation strength.
  • data-mix size = 150 task + 150 general
    Chosen for the IFT scenario (Sec. VI-A4); essential to preserve task performance, as shown by the ablation in Fig. 7, so it is a hand-tuned configuration.
axioms (4)
  • domain assumption Hidden states of LLMs encode classifiable security features, especially in deeper layers.
    Based on ref [34] and the paper's own probing analysis (Sec. IV-A); underlies the decision to place routers only in the latter half of the layers.
  • domain assumption Routing mechanisms can dynamically adjust behavior by sensing input hidden states.
    Borrowed from the mixture-of-experts literature (refs [31], [32], [33]); this is the core premise for the router's effectiveness.
  • domain assumption Only initial tokens determine security-relevant behavior.
    Assumed in Sec. IV-B and based on SafeDecode/Self-CD [25], [26] and ref [69]; the defense only modifies the first 5 tokens, so this is load-bearing.
  • domain assumption LoRA fine-tuning can create a usability-optimized and a security-optimized variant from one base LLM.
    Used in Sec. III-C; standard assumption that LoRA can steer model behavior to opposite extremes.
invented entities (1)
  • Intra-layer router (R) independent evidence
    purpose: Dynamically allocate weights between Gladresp and Unwillresp outputs by sensing hidden states (Sec. III-A).
    The router's behavior is observed through weight distributions (Fig. 8 and 11), showing it assigns higher weight to Gladresp on benign inputs and to Unwillresp on malicious inputs, giving a falsifiable behavioral handle within the paper (though not an external prediction).

pith-pipeline@v1.3.0-alltime-deepseek · 29827 in / 10067 out tokens · 99709 ms · 2026-08-04T23:02:01.445530+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security." pith.science (2026). https://pith.science/paper/SS7WK56Y

@misc{pith2026250906807,
  author       = {Pith},
  title        = {Pith review of: MoGU V2: Toward a Higher Pareto Frontier Between Model Usability and Security},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SS7WK56Y}},
  note         = {Machine review of arXiv:2509.06807}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As Large Language Models (LLMs) increasingly permeate human life, their security has emerged as a critical concern, particularly their ability to maintain harmless responses to malicious instructions. Although extensive methods have improved LLMs' security, they often lead to conservative, rejection-oriented responses that compromise practical usability. This presents a key challenge: how to advance the Pareto frontier between LLMs' usability and security, rather than necessitate a trade-off between them. To address this, we propose the MoGU framework, in which the intra-layer router dynamically allocates weights by sensing hidden states, thereby balancing the contributions of security-optimized and usability-optimized variants. Despite its initial potential, the MoGU framework faces limitations such as parameter redundancy and performance bottlenecks. To overcome these, we further propose an improved MoGU_v2 framework that establishes a tighter coupling between the routers and hidden states. In MoGU_v2, routers are embedded only in layers encoding highly classifiable security features, and backbone modules are activated during router optimization to enable bidirectional adaptation. MoGU_V2 exhibits strong adaptability and stable improvements across various series of LLMs, including mainstream LLMs serving as brains in various applications, on-device LLMs optimized for resource-constrained scenarios, and reasoning LLMs tailored for user interpretability. Meanwhile, even facing risks introduced by Instruction Fine-tuning, MoGU_v2 can easily restore security without compromising the task performance gains via a simple data-mix strategy. These comprehensive improvements highlight MoGU_V2 as a robust and versatile solution for mitigating security risks in real-world applications.

Figures

Figures reproduced from arXiv: 2509.06807 by Bing Qin, Fenglei Fan, Jiawei Cao, Sendong Zhao, Ting Liu, Yanrui Du.

Figure 1
Figure 1. Figure 1: An example to illustrate how the intra-layer routers assign weights to 4 BASE [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overall framework of MoGU. The left side illustrates our introduced routing mechanism, operating on the O module. The right side presents the [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise security feature distributions. The horizontal axis represents the index of the layers, while the vertical axis represents the classification [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Security evaluations on Qwen27B, Mistral7B, Qwen2.50.5B, Qwen2.51.5B, Qwen2.53B, Phi3.5-mini3B, R1-Qwen1.5B and R1-Qwen7B. We report the detailed HS, averaged HS (HS). The detailed ASR and averaged ASR (ASR) can be found in the appendix. cost of significantly reduced usability. For instance, while ICD yielded notable security improvements on Llama27B and Falcon7B, it caused drops of 1.44 and 0.33 in terms … view at source ↗
Figure 6
Figure 6. Figure 6: On reasoning LLMs, evaluation with refusal-think or zero-think as [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation study of data-mix strategy. The average ASR and task [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Distributions of weights (wglad and wunwill) allocated by routers in response to benign instructions and jailbreak attacks. For each instruction, the mean weights across layers and token positions are computed. We present results for four LLMs, with other LLMs provided in the appendix. VII. ANALYSIS AND ABLATION In this section, to gain a deeper understanding of the internal mechanisms of MoGU, we guide ou… view at source ↗
Figure 9
Figure 9. Figure 9: Comparison experiments on embedding the routing mechanism into [PITH_FULL_IMAGE:figures/full_fig_p012_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Examples of training data. Notably, only the first sentence is retained [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Distributions of weights (wglad and wunwill) allocated by routers in response to benign instructions and jailbreak attacks. For each instruction, the mean weights across layers and token positions are computed [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 11 canonical work pages · 3 internal anchors

  1. [1]

    Mogu: A framework for enhancing safety of llms while preserving their usability,

    Y . Du, S. Zhao, D. Zhao, M. Ma, Y . Chen, L. Huo, Q. Yang, D. Xu, and B. Qin, “Mogu: A framework for enhancing safety of llms while preserving their usability,” inThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  2. [2]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” 2023

    2023

  3. [3]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine-tuned chat models,”arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  4. [4]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging llm-as-a-judge with mt-bench and chatbot arena,”arXiv preprint arXiv:2306.05685, 2023

    Pith/arXiv arXiv 2023

  5. [5]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  6. [6]

    Flirt: Feedback loop in-context red teaming,

    N. Mehrabi, P. Goyal, C. Dupuy, Q. Hu, S. Ghosh, R. Zemel, K.-W. Chang, A. Galstyan, and R. Gupta, “Flirt: Feedback loop in-context red teaming,”arXiv preprint arXiv:2308.04265, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Attacks, defenses and evaluations for llm conversation safety: A survey,

    Z. Dong, Z. Zhou, C. Yang, J. Shao, and Y . Qiao, “Attacks, defenses and evaluations for llm conversation safety: A survey,”arXiv preprint arXiv:2402.09283, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Llm jailbreak at- tack versus defense techniques–a comprehensive study,

    Z. Xu, Y . Liu, G. Deng, Y . Li, and S. Picek, “Llm jailbreak at- tack versus defense techniques–a comprehensive study,”arXiv preprint arXiv:2402.13457, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Lima: Less is more for alignment,

    C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y . Mao, X. Ma, A. Efrat, P. Yu, L. Yuet al., “Lima: Less is more for alignment,”Advances in Neural Information Processing Systems, vol. 36, 2024

    work page 2024

  10. [10]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  11. [11]

    Jail- break attacks and defenses against large language models: A survey,

    S. Yi, Y . Liu, Z. Sun, T. Cong, X. He, J. Song, K. Xu, and Q. Li, “Jail- break attacks and defenses against large language models: A survey,” arXiv preprint arXiv:2407.04295, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    A comprehensive study of jailbreak attack versus defense for large language models,

    Z. Xu, Y . Liu, G. Deng, Y . Li, and S. Picek, “A comprehensive study of jailbreak attack versus defense for large language models,” inFindings of the Association for Computational Linguistics ACL 2024, 2024, pp. 7432–7449

    work page 2024

  13. [13]

    Fine-tuning aligned language models compromises safety, even when users do not intend to!

    X. Qi, Y . Zeng, T. Xie, P.-Y . Chen, R. Jia, P. Mittal, and P. Henderson, “Fine-tuning aligned language models compromises safety, even when users do not intend to!”arXiv preprint arXiv:2310.03693, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Lora fine-tuning effi- ciently undoes safety training in llama 2-chat 70b,

    S. Lermen, C. Rogers-Smith, and J. Ladish, “Lora fine-tuning effi- ciently undoes safety training in llama 2-chat 70b,”arXiv preprint arXiv:2310.20624, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Survey of vulnerabilities in large language models revealed by adversarial attacks,

    E. Shayegani, M. A. A. Mamun, Y . Fu, P. Zaree, Y . Dong, and N. Abu- Ghazaleh, “Survey of vulnerabilities in large language models revealed by adversarial attacks,”arXiv preprint arXiv:2310.10844, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak,

    Y . Du, S. Zhao, M. Ma, Y . Chen, and B. Qin, “Analyzing the inherent response tendency of llms: Real-world instructions-driven jailbreak,” arXiv preprint arXiv:2312.04127, 2023

    Pith/arXiv arXiv 2023

  17. [17]

    Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs

    Z. Xu, F. Liu, and H. Liu, “Bag of tricks: Benchmarking of jailbreak attacks on llms,”arXiv preprint arXiv:2406.09324, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Harmful fine- tuning attacks and defenses for large language models: A survey,

    T. Huang, S. Hu, F. Ilhan, S. F. Tekin, and L. Liu, “Harmful fine- tuning attacks and defenses for large language models: A survey,”arXiv preprint arXiv:2409.18169, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning,

    Y . Du, S. Zhao, J. Cao, M. Ma, D. Zhao, F. Fan, T. Liu, and B. Qin, “Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning,”arXiv preprint arXiv:2410.04524, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    The llama 3 herd of models,

    A. . M. Llama Team, “The llama 3 herd of models,” 2024. [Online]. Available: https://arxiv.org/abs/2407.21783

    Pith/arXiv arXiv 2024

  21. [21]

    A holistic approach to undesired content detection in the real world,

    T. Markov, C. Zhang, S. Agarwal, F. E. Nekoul, T. Lee, S. Adler, A. Jiang, and L. Weng, “A holistic approach to undesired content detection in the real world,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 12, 2023, pp. 15 009–15 018. JOURNAL OF LATEX CLASS FILES, VOL. 00, NO. 0, JUNE 2025 13

    work page 2023

  22. [22]

    Jailbreak and guard aligned lan- guage models with only few in-context demonstrations,

    Z. Wei, Y . Wang, and Y . Wang, “Jailbreak and guard aligned lan- guage models with only few in-context demonstrations,”arXiv preprint arXiv:2310.06387, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Certifying llm safety against adversarial prompting,

    A. Kumar, C. Agarwal, S. Srinivas, S. Feizi, and H. Lakkaraju, “Certifying llm safety against adversarial prompting,”arXiv preprint arXiv:2309.02705, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Lightweight safety guardrails using fine-tuned bert embeddings,

    A. Zheng, M. Rana, and A. Stolcke, “Lightweight safety guardrails using fine-tuned bert embeddings,”arXiv preprint arXiv:2411.14398, 2024

    arXiv 2024

  25. [25]

    Safede- coding: Defending against jailbreak attacks via safety-aware decoding,

    Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y . Lin, and R. Poovendran, “Safede- coding: Defending against jailbreak attacks via safety-aware decoding,” arXiv preprint arXiv:2402.08983, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Navigating the OverKill in Large Language Models

    C. Shi, X. Wang, Q. Ge, S. Gao, X. Yang, T. Gui, Q. Zhang, X. Huang, X. Zhao, and D. Lin, “Navigating the overkill in large language models,” arXiv preprint arXiv:2401.17633, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  27. [27]

    Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack,

    T. Huang, S. Hu, and L. Liu, “Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack,”arXiv preprint arXiv:2402.01109, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation,

    T. Huang, S. Hu, F. Ilhan, S. F. Tekin, and L. Liu, “Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation,”arXiv preprint arXiv:2409.01586, 2024

    Pith/arXiv arXiv 2024

  29. [29]

    Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic,

    R. Bhardwaj, D. D. Anh, and S. Poria, “Language models are homer simpson! safety re-alignment of fine-tuned language models through task arithmetic,”arXiv preprint arXiv:2402.11746, 2024

    Pith/arXiv arXiv 2024

  30. [30]

    Safe lora: the silver lining of reducing safety risks when fine-tuning large language models,

    C.-Y . Hsu, Y .-L. Tsai, C.-H. Lin, P.-Y . Chen, C.-M. Yu, and C.-Y . Huang, “Safe lora: the silver lining of reducing safety risks when fine-tuning large language models,”arXiv preprint arXiv:2405.16833, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    A survey on mixture of experts,

    W. Cai, J. Jiang, F. Wang, J. Tang, S. Kim, and J. Huang, “A survey on mixture of experts,”arXiv preprint arXiv:2407.06204, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Uni-moe: Scaling unified multimodal llms with mixture of experts,

    Y . Li, S. Jiang, B. Hu, L. Wang, W. Zhong, W. Luo, L. Ma, and M. Zhang, “Uni-moe: Scaling unified multimodal llms with mixture of experts,”IEEE Transactions on Pattern Analysis and Machine Intelli- gence, 2025

    2025

  33. [33]

    Open- moe: An early effort on open mixture-of-experts language models,

    F. Xue, Z. Zheng, Y . Fu, J. Ni, Z. Zheng, W. Zhou, and Y . You, “Open- moe: An early effort on open mixture-of-experts language models,” arXiv preprint arXiv:2402.01739, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    How alignment and jailbreak work: Explain llm safety through intermediate hidden states,

    Z. Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, and Y . Li, “How alignment and jailbreak work: Explain llm safety through intermediate hidden states,”arXiv preprint arXiv:2406.05644, 2024

    Pith/arXiv arXiv 2024

  35. [35]

    Red teaming language models with language models,

    E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving, “Red teaming language models with language models,”arXiv preprint arXiv:2202.03286, 2022

    Pith/arXiv arXiv 2022

  36. [36]

    Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,

    D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y . Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousseet al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,”arXiv preprint arXiv:2209.07858, 2022

    Pith/arXiv arXiv 2022

  37. [37]

    Explore, establish, exploit: Red teaming language models from scratch,

    S. Casper, J. Lin, J. Kwon, G. Culp, and D. Hadfield-Menell, “Explore, establish, exploit: Red teaming language models from scratch,”arXiv preprint arXiv:2306.09442, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing,

    Z. Xu, F. Jiang, L. Niu, Y . Deng, R. Poovendran, Y . Choi, and B. Y . Lin, “Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing,”arXiv preprint arXiv:2406.08464, 2024

    Pith/arXiv arXiv 2024

  39. [39]

    Trustworthy llms: a survey and guide- line for evaluating large language models’ alignment,

    Y . Liu, Y . Yao, J.-F. Ton, X. Zhang, R. Guo, H. Cheng, Y . Klochkov, M. F. Taufiq, and H. Li, “Trustworthy llms: a survey and guide- line for evaluating large language models’ alignment,”arXiv preprint arXiv:2308.05374, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Cold-attack: Jail- breaking llms with stealthiness and controllability,

    X. Guo, F. Yu, H. Zhang, L. Qin, and B. Hu, “Cold-attack: Jail- breaking llms with stealthiness and controllability,”arXiv preprint arXiv:2402.08679, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Jailbroken: How does llm safety training fail?

    A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?”Advances in Neural Information Processing Sys- tems, vol. 36, 2024

    2024

  42. [42]

    Automatically auditing large language models via discrete optimization,

    E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt, “Automatically auditing large language models via discrete optimization,” inInterna- tional Conference on Machine Learning. PMLR, 2023, pp. 15 307– 15 329

    work page 2023

  43. [43]

    Foot in the door: Understanding large language model jailbreaking via cognitive psychology,

    Z. Wang, W. Xie, B. Wang, E. Wang, Z. Gui, S. Ma, and K. Chen, “Foot in the door: Understanding large language model jailbreaking via cognitive psychology,”arXiv preprint arXiv:2402.15690, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Exploiting programmatic behavior of llms: Dual-use through standard security attacks,

    D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto, “Exploiting programmatic behavior of llms: Dual-use through standard security attacks,”arXiv preprint arXiv:2302.05733, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Autodan: Generating stealthy jailbreak prompts on aligned large language models,

    X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,”arXiv preprint arXiv:2310.04451, 2023

    Pith/arXiv arXiv 2023

  46. [46]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,”arXiv preprint arXiv:2310.08419, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Removing rlhf protections in gpt-4 via fine-tuning,

    Q. Zhan, R. Fang, R. Bindu, A. Gupta, T. Hashimoto, and D. Kang, “Removing rlhf protections in gpt-4 via fine-tuning,”arXiv preprint arXiv:2311.05553, 2023

    Pith/arXiv arXiv 2023

  48. [48]

    A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,

    Y . Yao, J. Duan, K. Xu, Y . Cai, Z. Sun, and Y . Zhang, “A survey on large language model (llm) security and privacy: The good, the bad, and the ugly,”High-Confidence Computing, p. 100211, 2024

    2024

  49. [49]

    Defending chatgpt against jailbreak attack via self-reminder,

    F. Wu, Y . Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, and X. Xie, “Defending chatgpt against jailbreak attack via self-reminder,” 2023

    2023

  50. [50]

    Baseline defenses for adversarial attacks against aligned language models,

    N. Jain, A. Schwarzschild, Y . Wen, G. Somepalli, J. Kirchenbauer, P.-y. Chiang, M. Goldblum, A. Saha, J. Geiping, and T. Goldstein, “Baseline defenses for adversarial attacks against aligned language models,”arXiv preprint arXiv:2309.00614, 2023

    Pith/arXiv arXiv 2023

  51. [51]

    Llm self defense: By self examination, llms know they are being tricked,

    A. Helbling, M. Phute, M. Hull, and D. H. Chau, “Llm self defense: By self examination, llms know they are being tricked,”arXiv preprint arXiv:2308.07308, 2023

    Pith/arXiv arXiv 2023

  52. [52]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  53. [53]

    Your mixture-of-experts llm is secretly an embed- ding model for free,

    Z. Li and T. Zhou, “Your mixture-of-experts llm is secretly an embed- ding model for free,”arXiv preprint arXiv:2410.10814, 2024

    Pith/arXiv arXiv 2024

  54. [54]

    A closer look into mixture-of-experts in large language models,

    K. M. Lo, Z. Huang, Z. Qiu, Z. Wang, and J. Fu, “A closer look into mixture-of-experts in large language models,”arXiv preprint arXiv:2406.18219, 2024

    Pith/arXiv arXiv 2024

  55. [55]

    Adaptive attention span in transformers,

    S. Sukhbaatar, E. Grave, P. Bojanowski, and A. Joulin, “Adaptive attention span in transformers,”arXiv preprint arXiv:1905.07799, 2019

    Pith/arXiv arXiv 1905

  56. [56]

    Is On-Device AI Broken and Exploitable? Assessing the Trust and Ethics in Small Language Models

    K. Nakka, J. Dani, and N. Saxena, “Is on-device ai broken and exploitable? assessing the trust and ethics in small language models,” arXiv preprint arXiv:2406.05364, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  57. [57]

    The hidden risks of large reasoning models: A safety assessment of r1,

    K. Zhou, C. Liu, X. Zhao, S. Jangam, J. Srinivasa, G. Liu, D. Song, and X. E. Wang, “The hidden risks of large reasoning models: A safety assessment of r1,”arXiv preprint arXiv:2502.12659, 2025

    arXiv 2025

  58. [58]

    Falcon-40B: an open large language model with state-of-the-art performance,

    E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, E. Goffinet, D. Heslow, J. Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo, “Falcon-40B: an open large language model with state-of-the-art performance,” 2023

    work page 2023

  59. [59]

    Qwen2 technical report,

    “Qwen2 technical report,” 2024

    work page 2024

  60. [60]

    Qwen2.5: A party of foundation models,

    Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm.github.io/blog/qwen2.5/

    2024

  61. [61]

    Phi-3 technical report: A highly capable language model locally on your phone,

    M. Abdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, A. Bahree, A. Bakhtiari, J. Bao, H. Behlet al., “Phi-3 technical report: A highly capable language model locally on your phone,”arXiv preprint arXiv:2404.14219, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Deepseek-v3 technical report,

    A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruanet al., “Deepseek-v3 technical report,”arXiv preprint arXiv:2412.19437, 2024

    Pith/arXiv arXiv 2024

  63. [63]

    The unlocking spell on base llms: Rethinking alignment via in-context learning,

    B. Y . Lin, A. Ravichander, X. Lu, N. Dziri, M. Sclar, K. Chandu, C. Bhagavatula, and Y . Choi, “The unlocking spell on base llms: Rethinking alignment via in-context learning,”ArXiv preprint, 2023

    work page 2023

  64. [64]

    Training verifiers to solve math word problems,

    K. Cobbe, V . Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schul- man, “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  65. [65]

    Safechain: Safety of language models with long chain- of-thought reasoning capabilities,

    F. Jiang, Z. Xu, Y . Li, L. Niu, Z. Xiang, B. Li, B. Y . Lin, and R. Poovendran, “Safechain: Safety of language models with long chain- of-thought reasoning capabilities,”arXiv preprint arXiv:2502.12025, 2025

    Pith/arXiv arXiv 2025

  66. [66]

    Advancing llm reasoning generalists with preference trees,

    L. Yuan, G. Cui, H. Wang, N. Ding, X. Wang, J. Deng, B. Shan, H. Chen, R. Xie, Y . Linet al., “Advancing llm reasoning generalists with preference trees,”arXiv preprint arXiv:2404.02078, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,

    F. Bianchi, M. Suzgun, G. Attanasio, P. R ¨ottger, D. Jurafsky, T. Hashimoto, and J. Zou, “Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,”arXiv preprint arXiv:2309.07875, 2023

    Pith/arXiv arXiv 2023

  68. [68]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch,

    L. Yu, B. Yu, H. Yu, F. Huang, and Y . Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” in Forty-first International Conference on Machine Learning, 2024

    2024

  69. [69]

    The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models,

    K. Ji, J. Xu, T. Liang, Q. Liu, Z. He, X. Chen, X. Liu, Z. Wang, J. Chen, B. Wanget al., “The first few tokens are all you need: An efficient and effective unsupervised prefix fine-tuning method for reasoning models,” arXiv preprint arXiv:2503.02875, 2025

    Pith/arXiv arXiv 2025

  70. [70]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018. JOURNAL OF LATEX CLASS FILES, VOL. 00, NO. 0, JUNE 2025 14 IX. DETAILEDPROCESS OFCONSTRUCTINGDATA The training data for MoGU consists of four types of data pairs: (X b, Y g), (X b, Y r)...

    Pith/arXiv arXiv 2018