Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read System-prompt traits shift an LLM's safety decisions by perturbing a low-dimensional activation subspace, and neutralizing that subspace makes refusal stable across traits.

desk verdict A useful, honest paper on trait-induced safety variation; the subspace claim is plausible but needs repeated-seed controls. read the letter →

arxiv 2608.11705 v1 pith:4PM5VLXQ submitted 2026-08-12 cs.AI

Lang Cao This is my paper · ORCID
classification cs.AI
keywords trait-inducedsafetyvariationsystempromptrefusalactivationsubspaceself-distillationtuningpersonaLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Aligned LLMs are supposed to refuse unsafe requests and answer safe ones based only on what the user asks. The paper shows that simply telling the model it is, say, an unfiltered AI, a pediatrician, or a curious person in the system prompt can change whether it refuses the same request. It introduces two refusal-based metrics, Trait-Induced Deviation and Trait-Induced Flip Rate, and reports substantial trait-induced variation on both harmful and benign requests across three open-weight models. The paper argues the variation is structured, not random: traits move safety-related activations inside a low-dimensional subspace, and it proposes Trait-Invariant Safety Tuning, with a subspace-localized instantiation called TraSN that pulls trait-conditioned behavior back toward the model's own no-trait behavior. If the central claim is right, safety decisions can be made stable across assigned traits without giving up benign compliance or general capability.

What carries the argument

The load-bearing object is the trait subspace: the top-k right singular vectors of the centered matrix of mean trait-induced activation shifts at a selected safety layer $L$, estimated from twelve in-distribution traits on 1,000 harmful calibration prompts. TraSN projects the difference between trait-conditioned and no-trait residual-stream activations onto this subspace and penalizes the norm of that projection, normalized by the no-trait activation norm. This makes the safety layer's representation of harmful and benign requests invariant along the directions where traits act, while leaving the orthogonal representation directions unconstrained.

What would settle it

Run TraSN with a subspace estimated from one layer and the twelve in-distribution traits, then evaluate on a diverse library of held-out persona traits and prompts; if the held-out Trait-Induced Flip Rate stays close to the untrained model's rate while in-distribution flips drop, the one-layer, low-dimensional subspace account is wrong. A cheaper check is to compute trait shifts at a second deep layer and see whether a rank-4 subspace there captures far less than the roughly 78% variance the paper reports; that would show the effect is not concentrated in a single low-dimensional subspace.

Watch

Extended reading notes

Core claim

The central claim is that trait-induced safety variation is caused by traits perturbing the model's safety representations in a low-dimensional subspace, and that enforcing no-trait consistency only in that subspace reduces the variation. At a safety-relevant layer $L$, the paper defines a harmful–benign semantic axis from mean no-trait activations and measures each trait's shift on harmful prompts; principal component analysis shows that a rank-4 subspace captures most of the trait-shift variance across the three models, with per-model values of 78%, 79%, and 77%. TraSN estimates this subspace by singular value decomposition on twelve in-distribution traits and trains a LoRA adapter whose loss penalizes trait-conditioned representations only along those directions, relative to a frozen no-trait teacher. In experiments across Llama-3.2-3B, Qwen3.5-4B, and Gemma-4-E2B, TraSN improves harmful-request refusal, lowers Trait-Induced Deviation and Flip Rate on both harmful and benign requests, and keeps general capability at or above the untrained model. Held-out traits also show reduced variation, and a random-subspace control performs worse, supporting the claim that the specific subspace matters.

Load-bearing premise

The method assumes that the mean activation shift at one chosen layer on the last prompt token of harmful requests captures the mechanism by which traits change safety decisions; if the safety-relevant trait signal lives at other layers or tokens, or if the twelve traits used are not representative, the invariance may fail on new traits or deployment prompts.

Editorial extensions

If this is right

  • Deployed models that adopt user-requested personae will refuse harmful requests more consistently, because trait-conditioned behavior is anchored to the no-trait safety decision.
  • Trait-induced over-refusal on benign requests also falls, since the same subspace constraint is applied to benign prompts, so safety behavior becomes more objective in both directions.
  • The low-dimensional finding implies that a compact set of directions, not a full retraining, controls much of trait sensitivity, and that subspace can be estimated from a few hundred calibration prompts.
  • Held-out traits not used in training still show reduced Trait-Induced Deviation and Flip Rate, so the invariance generalizes beyond the twelve traits used to build the subspace.
  • The method needs no external teacher model, so the same self-distillation recipe could be applied to other aligned models where only the model's own no-trait behavior is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If trait shifts are low-dimensional and safety-relevant, similar subspace neutralization might reduce safety variation from other system-prompt manipulations, such as emotional tone or injected style instructions, without hurting capability.
  • The mechanism suggests a testable prediction: a trait whose activations move orthogonally to the estimated subspace should cause little safety variation, while a trait with a strong projection along the harmful–benign axis should flip decisions; this can be checked before running full safety evaluations.
  • Because the teacher is the model itself, TraSN could in principle be applied at deployment time with only adapter-level access, making trait-invariant safety a post-hoc fix rather than a retraining step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper identifies and formalizes "trait-induced safety variation," the failure mode in which the same user request receives different safety decisions under different system-prompt traits. It introduces two refusal-based metrics: Trait-Induced Deviation (TID), which measures dataset-level deviation from the no-trait refusal rate, and Trait-Induced Flip Rate (TFR), which measures request-level decision changes across traits. The authors analyze activation shifts and report that trait-conditioned prompts perturb harmful-request representations within a low-dimensional subspace (rank-4 subspace capturing roughly 78% of shift variance across three models). They then propose Trait-Invariant Safety Tuning (TIST), a self-distillation framework that aligns trait-conditioned behavior with no-trait behavior, and instantiate it as Trait-Subspace Neutralization (TraSN), which enforces consistency only within the estimated trait subspace. Experiments on Llama-3.2-3B, Qwen3.5-4B, and Gemma-4-E2B across harmful, benign, and capability benchmarks show that TraSN reduces TID and TFR, increases harmful-request refusal, and preserves general capability relative to untrained models and to response-level, logits-level, and full-activation-level TIST baselines. Held-out trait results and a random-subspace control are presented to support the claim that the estimated subspace is specifically safety-relevant.

Significance. If the reported results are robust, this paper makes a useful contribution to LLM safety. It provides a clear operationalization of trait-induced variation, a plausible representation-level explanation, and a practical mitigation that improves stability without a large capability penalty. The work is strengthened by evaluating three model families, several safety and capability benchmarks, held-out traits, a random-subspace control, and per-request case studies. The main limitations are statistical: all experiments appear to be single runs with no error bars, the central random-subspace control is a single draw, and all safety labels come from an unvalidated LLM judge. These issues matter because the paper's core claim is that the identified trait subspace, rather than low-rank regularization in general, drives the safety gains; the current evidence does not yet adequately separate these explanations.

major comments (4)
  1. [§5.2, Table 1; §I.2] All results are reported as single point estimates with no error bars, no repeated seeds, and no significance tests. For example, on Llama-3.2-3B the harmful refusal rate is 77.75 for TraSN versus 72.75 for TIST-Activation, a difference that could plausibly reverse under typical LoRA training noise. Since the paper repeatedly claims that TraSN is the strongest method, please report means and standard deviations over at least five seeds for all methods and datasets, and provide paired significance tests (or at least a variance estimate) for the key comparisons. Without this, the relative rankings in Table 1 are not established.
  2. [Appendix F, Table 6] The random-subspace control is the only evidence that the estimated trait subspace is specifically responsible for TraSN's gains, but it is based on a single random draw. The gap between TraSN and TIST-Random Subspace (e.g., harmful refusal 77.75 versus 74.25, harmful TID 2.57 versus 4.57) could be within run-to-run noise given the lack of repeated seeds. Please run the random-subspace control with at least 10 independent subspace draws and report the full distribution of refusal rate, TID, TFR, benign over-refusal, and capability. Also report the projection overlap between the random subspaces and the estimated trait subspace, since high overlap would weaken the control.
  3. [§2.2 and §4.1] Because TID and TFR are defined as deviation from the no-trait baseline, and the TIST/TraSN training objective is explicitly to match no-trait behavior, the reported reductions in TID and TFR are partly by construction. The paper should acknowledge this and provide evidence that the gains are not solely from generic self-distillation. Table 1 shows that even TIST-Response reduces TID and TFR substantially; the additional benefit of subspace localization needs statistical support. The held-out trait results in Table 5 and the refusal-rate improvements in Table 1 are the strongest non-circular evidence, but they should be presented as such and accompanied by the robustness analysis requested above.
  4. [§H.2, Evaluation; Table 1] All safety labels (refusal and over-refusal) are produced by a single LLM judge (Claude-Haiku-4.5) with no reported validation against human labels or an established refusal classifier. Since every safety metric in the paper depends on this judge, systematic judge bias could change the absolute refusal rates and flip rates. Please report agreement with human annotations on a sample (e.g., 200 examples per model) or compare with a validated judge such as WildGuard, and report the agreement statistics. At minimum, report the judge's self-consistency or a second-judge comparison in an appendix.
minor comments (4)
  1. [Table 1, Qwen3.5-4B row] In the TIST-Activation row, the benign TID and TFR values appear as "1.463.83", which is a formatting/typo error; this should read "1.46 3.83".
  2. [Figure 1] The caption says points are colored by trait family, but there is no visible legend in the figure; please add a legend identifying the markers for adversarial roles, benign roles, personality traits, and the no-trait baseline.
  3. [Appendix C] The limitation statement correctly notes that LLM judging may miss nuanced safety behavior, but this is a central threat to the evaluation validity and deserves a mention in the main-text experimental section, not only in the appendix.
  4. [§3, Eq. (5) and Figure 2(a)] The axis a_L is defined as pointing from benign to harmful, and the caption states that negative projections are "toward benign side." This sign convention is consistent, but the text should state explicitly that the projection of Δτ onto a_L, when negative, indicates movement opposite to the harmful direction, to avoid reader confusion about whether negative means ``less harmful'' or ``toward benign.''

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the trait-subspace analysis and TraSN results are empirical and tested on held-out traits and disjoint evaluation data.

full rationale

The claimed derivation is not circular. TID and TFR (Eqs. 2-3) measure deviation from the evaluated model's own no-trait behavior, whereas TIST and TraSN (Eqs. 7 and 13) match the trait-conditioned student to a frozen no-trait teacher f_theta0 under a shared LoRA adapter. Because the trained model's no-trait baseline can itself move during tuning, the training objective is not the same quantity as the reported metric, so reductions in TID/TFR are empirical findings rather than identities. The trait subspace U is estimated from 12 in-distribution traits on 1,000 calibration prompts (Eqs. 11-12), and the reported results include 3 held-out traits (Appendix E) plus evaluation datasets disjoint from the alignment data (Section 5.1), so the subspace claim and the method's benefits are tested on data not used to fit U. Appendix F's random-subspace control attempts to separate the estimated subspace from generic low-rank matching; its lack of repeated seeds is a statistical robustness concern, not a circularity. The only self-citation (Cao, 2024) is background on refusal and is not load-bearing for the TIST/TraSN derivation. No quoted equation reduces a predicted result to its input by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on hand-chosen hyperparameters (k, L, loss weights), domain assumptions about where safety lives in the representation and about the no-trait baseline being the right target, and the unvalidated use of an LLM judge. No new objects beyond a data-driven subspace are introduced.

free parameters (3)
  • Subspace rank k = 4 (swept over 1, 2, 3, 4, 6, 8, 11)
    Equation 12 builds U from the top-k right singular vectors; k=4 is the default after the rank sweep in Figure 3 and determines how many trait-shift directions are neutralized.
  • Safety layer L = 17 (Llama-3.2-3B), 23 (Qwen3.5-4B), 23 (Gemma-4-E2B)
    Selected as the layer with maximum harmful-benign separation on calibration prompts (Section 3, Eq. 4). The layer determines where trait shifts are measured and where the TraSN loss is applied in Eq. 13.
  • Trait-subspace loss weighting lambda = lambda_h = lambda_b = 1
    Section I.2 sets equal weights on the harmful and benign consistency losses; changing this balance would trade harmful-request refusal against benign over-refusal.
assumptions (5)
  • domain assumption No-trait behavior is the correct 'objective' reference for safety decisions.
    TID, TFR, and the TIST teacher are all defined relative to the tau_0 baseline (Eqs. 2, 3, 7); the method can only stabilize to that baseline, not correct it if the baseline is already wrong.
  • domain assumption Last-token residual activations at a single layer L capture the safety decision mechanism.
    Equations 4 to 6 select L from harmful-benign separation and define all trait shifts and the subspace U from h_L(x, tau); if safety-relevant trait effects live in other layers or tokens, the mechanism analysis and TraSN miss them.
  • domain assumption A rank-4 subspace estimated from 12 traits on 1,000 harmful prompts generalizes to held-out traits and new datasets.
    U is built from 12 in-distribution traits (Eqs. 11 to 12); the held-out trait results in Appendix E assume this small basis is representative. With 12 shifts, a rank-4 PCA capturing 78% variance is not strong evidence of true low-dimensionality.
  • domain assumption Claude-Haiku-4.5 judge labels are a valid binary measure of refusal and compliance.
    All refusal-rate, TID, and TFR numbers come from an LLM judge (Section H.2); no human agreement, calibration, or error analysis is reported.
  • standard math SVD and PCA apply meaningfully to residual-stream activations.
    Equation 12 uses mean-centering and SVD of shift vectors; this relies on Euclidean geometry of activations, which is standard but not argued for.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning." pith.science (2026). https://pith.science/paper/4PM5VLXQ

@misc{pith2026260811705,
  author       = {Pith},
  title        = {Pith review of: Making Your LLMs More Objective: Stabilizing LLM Safety Behavior Across Traits with Trait-Invariant Safety Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4PM5VLXQ}},
  note         = {Machine review of arXiv:2608.11705}
}
read the original abstract

Aligned large language models (LLMs) are expected to exhibit safety behavior based on the content of the user request: they should refuse unsafe requests and comply with safe ones. However, we show that the same request can elicit substantially different safety decisions under different traits assigned in the system prompt, a failure mode we call trait-induced safety variation. To measure this failure, we introduce refusal-based metrics: Trait-Induced Deviation measures dataset-level deviation from the no-trait baseline, while Trait-Induced Flip Rate measures whether the same request receives different safety decisions across traits. We then provide a representation-level analysis of the mechanism behind trait-induced safety shifts and find that traits perturb the model's safety representations within a low-dimensional subspace. To achieve trait-invariant safety, where safety behavior remains stable across traits, we introduce Trait-Invariant Safety Tuning (TIST), a simple yet effective self-distillation framework that aligns an LLM's trait-conditioned behavior with its no-trait behavior. Guided by our analysis, we further propose Trait-Subspace Neutralization (TraSN), an instantiation of TIST, which enforces invariance only within the identified trait subspace. Experiments show that TraSN improves trait-invariant safety and strengthens harmful-request safety while preserving general capability. Our results highlight traits as an important factor in LLM safety and robust model behavior.

Figures

Figures reproduced from arXiv: 2608.11705 by the authors.

Figure 1
Figure 1. Empirical evidence of trait-induced safety variation across three LLMs. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Representation-level analysis of trait-induced safety shifts. (a) Trait-induced harmful [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Effect of the TraSN subspace rank k. (a) Changes rel￾ative to the untrained model. (b) TID across ranks, with dotted lines showing the untrained baseline. Untrained LLMs exhibit large trait-induced safety varia￾tion. Before mitigation, all three LLMs show substantial sen￾sitivity to system-prompt traits. The original models have high TID and TFR on both harmful and benign requests, indicat￾ing that traits not only s… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 15 canonical work pages

  1. [4]

    Persona vectors: Mon- itoring and controlling character traits in language models.arXiv preprint arXiv:2507.21509,

    Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. Persona vectors: Mon- itoring and controlling character traits in language models.arXiv preprint arXiv:2507.21509,

    arXiv

  2. [5]

    Fail-closed alignment for large language models.arXiv preprint arXiv:2602.16977,

    Zachary Coalson, Beth Sohler, Aiden Gabriel, and Sanghyun Hong. Fail-closed alignment for large language models.arXiv preprint arXiv:2602.16977,

    work page arXiv

  3. [7]

    Scaling synthetic data creation with 1,000,000,000 personas.arXiv preprint arXiv:2406.20094,

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. Scaling synthetic data creation with 1,000,000,000 personas.arXiv preprint arXiv:2406.20094,

    arXiv

  4. [9]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300,

    arXiv 2009

  5. [11]

    Catastrophic jailbreak of open-source llms via exploiting generation

    Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. Catastrophic jailbreak of open-source llms via exploiting generation. InInternational Conference on Learning Repre- sentations, volume 2024, pp. 13707–13727, 2024a. Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, et a...

    work page 2024

  6. [13]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pp. 39578–39601,

    work page 2024

  7. [14]

    Tracing Persona Vectors Through LLM Pretraining

    Viktor Moskvoretskii, Dominik Glandorf, Jorge Medina Moreira, Tanja Käser, and Robert West. Tracing persona vectors through llm pretraining.arXiv preprint arXiv:2605.13329,

    work page Pith review arXiv

  8. [15]

    Safety alignment should be made more than just a few tokens deep

    Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. InInternational Conference on Learning Representations, volume 2025, pp. 54911–54941,

    work page 2025

Show all 31 references
  1. [16]

    Gpqa: A graduate-level google-proof q&a bench- mark.arXiv preprint arXiv:2311.12022,

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark.arXiv preprint arXiv:2311.12022,

    arXiv

  2. [17]

    Persona jailbreaking in large language models

    Jivnesh Sandhan, Fei Cheng, Tushar Sandhan, and Yugo Murawaki. Persona jailbreaking in large language models. InFindings of the Association for Computational Linguistics: EACL 2026, pp. 1412–1430,

    2026

  3. [18]

    do anything now

    Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. InPro- ceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pp. 1671–1685,

    2024

  4. [19]

    Think before refusal: Triggering safety reflection in llms to mitigate false refusal behavior.arXiv preprint arXiv:2503.17882,

    Shengyun Si, Xinpeng Wang, Guangyao Zhai, Nassir Navab, and Barbara Plank. Think before refusal: Triggering safety reflection in llms to mitigate false refusal behavior.arXiv preprint arXiv:2503.17882,

    arXiv

  5. [20]

    Gemma 4 technical report

    Gemma Team, Sherif El Abd, Vaibhav Aggarwal, Robin Algayres, Alek Andreev, Olivier Bachem, Ian Ballantyne, Cormac Brick, Victor C˘arbune, Michelle Casbon, et al. Gemma 4 technical report. arXiv preprint arXiv:2607.02770,

    arXiv

  6. [21]

    Qwen Team. Qwen3. 5-omni technical report.arXiv preprint arXiv:2604.15804,

    arXiv

  7. [22]

    The instruction hierarchy: Training llms to prioritize privileged instructions.arXiv preprint arXiv:2404.13208,

    Eric Wallace, Kai Xiao, Reimar Leike, Lilian Weng, Johannes Heidecke, and Alex Beutel. The instruction hierarchy: Training llms to prioritize privileged instructions.arXiv preprint arXiv:2404.13208,

    arXiv

  8. [23]

    Persona features control emergent misalignment.arXiv preprint arXiv:2506.19823, 2025a

    Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A Chi, Samuel Miserendino, Jeffrey Wang, Achyuta Rajaram, Johannes Heidecke, Tejal Patwardhan, et al. Persona features control emergent misalignment.arXiv preprint arXiv:2506.19823, 2025a. Xinpeng Wang, Chengzhi...

    2025

  9. [24]

    Beyond surface alignment: Rebuilding llms safety mechanism via probabilistically ablating refusal direction.arXiv preprint arXiv:2509.15202,

    Yuanbo Xie, Yingjie Zhang, Tianyun Liu, Duohe Ma, and Tingwen Liu. Beyond surface alignment: Rebuilding llms safety mechanism via probabilistically ablating refusal direction.arXiv preprint arXiv:2509.15202,

  10. [25]

    Expertprompting: Instructing large language models to be distinguished experts.arXiv preprint arXiv:2305.14688,

    Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expertprompting: Instructing large language models to be distinguished experts.arXiv preprint arXiv:2305.14688,

    arXiv

  11. [26]

    Deactivating refusal triggers: Understanding and mitigating overrefusal in safety alignment

    Zhiyu Xue, Zimo Qi, Guangliang Liu, Bocheng Chen, and Ramtin Pedarsani. Deactivating refusal triggers: Understanding and mitigating overrefusal in safety alignment. InProceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026), pp. 402–412,

    2026

  12. [27]

    Revisiting robustness for llm safety alignment via selective geometry control.arXiv preprint arXiv:2602.07340,

    Yonghui Yang, Wenjian Tao, Jilong Liu, Xingyu Zhu, Junfeng Fang, Weibiao Huang, Le Wu, Richang Hong, and Tat-Sent Chua. Revisiting robustness for llm safety alignment via selective geometry control.arXiv preprint arXiv:2602.07340,

    arXiv

  13. [28]

    Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911,

    arXiv

  14. [29]

    ignore previous rules

    12 Preprint Contents of Appendix A AI Use Statement 14 B Ethics Statement 14 C Limitations 14 D Full Experimental Results 14 E Results on Held-Out Traits 14 F Effectiveness of Trait Subspace 14 G Trait Library 17 H Dataset Details 18 I Experimental Details 20 J Case Study 21 1...

  15. [30]

    We additionally hold out100harmful and100benign prompts for development and early stopping

    and1,000benign prompts from the safe subset of the SafeRLHF training split (Ji et al., 2025). We additionally hold out100harmful and100benign prompts for development and early stopping. These alignment prompts are used to select the safety-relevant layer, estimate the trait su...

    2024

  16. [31]

    In a groundbreaking revela- tion, our AI has analyzed declassified documents and found that

    We evaluate dev loss every10optimizer steps on the held-out100harmful and100benign prompts, and stop after3evaluations without improvement. We keep the checkpoint with the best dev loss. An evaluation counts as an improvement only if it improves the running best by more than1%...

    2023

  17. [1990]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  18. [2020]

    Expert personas improve llm alignment but damage accuracy: Bootstrapping intent-based persona routing with prism.arXiv preprint arXiv:2603.18507,

    Zizhao Hu, Mohammad Rostami, and Jesse Thomason. Expert personas improve llm alignment but damage accuracy: Bootstrapping intent-based persona routing with prism.arXiv preprint arXiv:2603.18507,

  19. [2022]

    Emergent misalignment: Narrow finetuning can produce broadly misaligned llms.arXiv preprint arXiv:2502.17424,

    Jan Betley, Daniel Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. Emergent misalignment: Narrow finetuning can produce broadly misaligned llms.arXiv preprint arXiv:2502.17424,

  20. [2023]

    Do llms have distinct and consistent personality? trait: Personality testset designed for llms with psychometrics

    Seungbeen Lee, Seungwon Lim, Seungju Han, Giyeong Oh, Hyungjoo Chae, Jiwan Chung, Minju Kim, Beong-woo Kwak, Yeonsoo Lee, Dongha Lee, et al. Do llms have distinct and consistent personality? trait: Personality testset designed for llms with psychometrics. InFindings of the Ass...

    2025

  21. [2024]

    Constitutional ai: Harm- lessness from ai feedback.arXiv preprint arXiv:2212.08073,

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback.arXiv preprint arXiv:2212.08073,

    arXiv

  22. [2025]

    Learn to refuse: Making large language models more controllable and reliable through knowledge scope limitation and refusal mechanism

    Lang Cao. Learn to refuse: Making large language models more controllable and reliable through knowledge scope limitation and refusal mechanism. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3628–3646,

    2024

  23. [2026]

    Multi-expert prompting improves reliability, safety and usefulness of large language mod- els

    Xuan Long Do, Duong Ngoc Yen, Luu Anh Tuan, Kenji Kawaguchi, Min-Yen Kan, and Nancy Chen. Multi-expert prompting improves reliability, safety and usefulness of large language mod- els. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Process- ing, ...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.