Extracting and Understanding the Superficial Knowledge in Alignment

Bhavya Kailkhura; Gabriel Jacob Perin; Junyuan Hong; Nina S. T. Hirata; Runjin Chen; Xilun Chen; Xuxi Chen; Yan Han

REVIEW 3 major objections 5 minor 1 cited by

Extracting and Understanding the Superficial Knowledge in Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single linear adjustment to a language model's final output layer can reproduce most of the safety and detoxification gains of full alignment, plus a substantial share of its math and truthfulness improvements.

desk verdict A real, useful decomposition with a label that outruns the evidence: the linear-capturable component of alignment is portable and recoverable, but calling it 'superficial' is an interpretation, not a measurement. read the letter →

arxiv 2502.04602 v1 pith:NNEVGTUC submitted 2025-02-07 cs.CL cs.AI

Runjin Chen , Gabriel Jacob Perin , Xuxi Chen , Xilun Chen , Yan Han , Nina S. T. Hirata , Junyuan Hong , Bhavya Kailkhura This is my paper · ORCID

classification cs.CLcs.AI

keywords alignmentsuperficialknowledgelinearprojectionheaddistillationLLMsafetytransferfine-tuningattackstokenrestyling

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether alignment of large language models is mostly a surface phenomenon. It defines superficial knowledge as knowledge that can be acquired through simple token restyling—changing how the model phrases and structures responses—without altering the transformer's internal understanding of token relationships. To test this, the authors train only a linear residual on the base model's final projection head, using the aligned model's output logits as supervision, and compare the patched base model with the fully aligned model across math, safety, toxicity, and truthfulness benchmarks. They find that this single linear patch eliminates safety and toxicity risks and recovers a substantial fraction of math and truthfulness gains, though a clear gap remains on knowledge-intensive tasks. The result matters because it quantifies how much of alignment is cheap and transferable, and it suggests two practical uses: transferring alignment from a small model to a large one, and restoring safety after fine-tuning attacks.

What carries the argument

The carrying object is a learnable linear residual $\Delta W$ applied to the final linear projection head of the frozen base model (or, in the transferable variant, a linear map $W_{\text{trans}}$ on the top-k logits). The base model's transformer layers $f_b(\cdot)$ and its base projection $W_b$ stay fixed, so the only learned change is a shallow adjustment to token selection. Training uses KL divergence between the softmax of the patched base logits and the aligned model's logits, so no new knowledge is injected; the resulting $\Delta W$ is the extracted superficial knowledge. Positional analysis shows that the first tokens of each response carry most of the alignment shift, and that this front-of-response shift is almost perfectly captured by the linear head, while later positions retain a KL gap that resists linear capture.

What would settle it

Measure whether the extracted $\Delta W$ changes the model's top-k token distribution on content words tied to the correct answer rather than only on style words (connectives, enumerators, hedging). If the linear head reproduces substantive first-word shifts on factual questions where the aligned model's first word is a number or a named entity, then the residual is capturing content, not just restyling, and the 'superficial' label would be wrong.

Watch

Extended reading notes

Core claim

The central discovery is that a large portion of alignment-induced behavior change lives in the final token-selection step and can be captured by a linear map trained by distillation. Concretely, for a base model with a frozen transformer backbone, the authors learn a residual matrix $\Delta W$ such that $(W_b + \Delta W) f_b(x)$ approximates the aligned model's logits; this residual is the extracted superficial knowledge. Evaluated on LLaMA2, applying only this residual eliminates toxic and harmful outputs entirely (ToxiScore and HarmRate drop to 0) and recovers average improvements of 58% on GSM math and 78% on TruthfulQA relative to the base model, outperforming simpler baselines like LIMA and Urial. The same residual, re-expressed as a linear map on top-k logits, transfers from LLaMA2-7B-Chat to LLaMA2-13B and restores 88% of safety after a benign fine-tuning attack. The authors conclude that alignment is largely, but not entirely, superficial: the residual cannot close the gap on reasoning-heavy tasks, which they attribute to the aligned model's deeper capacity for reasoning and context integration.

Load-bearing premise

The whole analysis rests on the assumption that a change confined to the final output layer is automatically superficial—that it only restyles tokens and cannot alter the model's actual understanding.

Editorial extensions

If this is right

Safety and toxicity alignment can be added to a base model by a plug-in linear layer, with no further fine-tuning of the transformer.
Superficial alignment extracted from a smaller model transfers to a larger model from the same family, enabling offsite alignment without access to the large model's weights.
After a benign fine-tuning attack destroys a chat model's safety, re-attaching the extracted superficial layer restores most of the safety behavior without degrading MMLU accuracy.
Because later response positions retain a gap not capturable by the linear head, alignment includes a deeper, reasoning-related component that simple restyling cannot provide.
The extracted superficial knowledge outperforms earlier low-cost alignment methods (LIMA, Urial) on the tested benchmarks, indicating it covers more of the full alignment signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

As an extension beyond this paper, the same linear-isolation technique could be applied to other alignment attributes—instruction following, refusal style, persona—to build a library of plug-and-play behavioral patches for a frozen base model.
The positional result (alignment shifts concentrated in the first tokens) suggests a testable hypothesis: prompting interventions that only rewrite the opening of a response may achieve much of the safety benefit, an idea the paper's Urial comparison points toward but does not itself establish.
The recoverability result implies that fine-tuning attacks delete or overwrite a shallow, extractable component rather than erasing deep safety knowledge; if so, attack defenses could focus on preserving or reattaching the output-layer map rather than retraining the whole model.
A caution the authors leave implicit: if a downstream task's 'aligned' behavior is itself shallow, then measuring alignment by these benchmarks may overstate how much of alignment is superficial in general.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this one. It has a genuinely new empirical result, and the new result is not the headline interpretation. The method is simple: fit a linear map, either on the last hidden state or on top-k logits, that mimics an aligned model's logit shifts from the base model, then bolt it on. That recovers most safety and toxicity alignment, a good chunk of GSM and TruthfulQA gains, transfers LLaMA-2-7B chat behavior to 13B, and restores most safety after a fine-tuning attack. Those are real demonstrations, and the paper is honest that a gap remains and that the non-superficial part is not understood.

Where I part company with the paper is the word 'superficial.' The linear-head-capturable component is defined as superficial in Section 2.2, so the headline claim that a significant portion of alignment is superficial is in part a restatement. The stress-test concern is right: a linear map on the hidden state is not a small stylistic knob, and one early token like 'To find' can flip the model into calculation mode. The paper's own GSM example shows exactly that. So the quantitative decomposition is real, but calling the extracted component superficial is an interpretation, not a measurement. The safety results are especially sensitive to this: emitting one refusal-style token can redirect the whole continuation.

Lesser issues: all tables are single runs, with no error bars or repeats. The transfer result is close in spirit to proxy-tuning (Liu et al., 2024), which they cite; the novelty is the linear distillation and the decomposition framing. For a 100-sample Advbench evaluation, a 0% harm rate with no variance is fragile.

That said, the practical claims survive most of the interpretive critique. Even if the extracted map is not purely stylistic, it is portable and re-attachable, and the recovery result after a fine-tuning attack is worth knowing. The paper deserves a serious referee. The fix is controls: token-level analyses separating content shifts from format shifts, repeated runs, and at least one control showing that a linear probe on the same hidden states does not recover deep knowledge to the same degree. With those, the superficiality claim would be testable.

Recommendation: send it out. The referee should insist on the controls, but this is a useful contribution regardless of how the 'superficial' label shakes out.

Referee Report

3 major / 5 minor

Summary. The paper asks whether alignment of LLMs is predominantly 'superficial.' It formalizes superficial knowledge as token-restyling information that can be captured by a linear modification of the final projection head (Eq. 3) or a linear map on top-k logits (Eq. 5), extracted by distillation from an aligned model's logits. Experiments on LLaMA-2, Mistral, and Qwen evaluate a base model augmented only with this extracted knowledge on GSM, Toxigen, AdvBench, and TruthfulQA. The authors report that this linear residual eliminates safety and toxicity risks, recovers on average 58% of GSM and 78% of TruthfulQA gains (Table 1), transfers from LLaMA-2-7B-Chat to LLaMA-2-13B (Table 3), and can restore most safety after catastrophic fine-tuning (Table 4). They conclude that superficial knowledge is a large but incomplete part of alignment, with remaining gaps attributed to reasoning and contextual integration.

Significance. The question of how much of alignment is surface-level restyling is timely and practically relevant. If the identification is valid, the paper provides a concrete, quantitative framework for testing the Superficial Alignment Hypothesis and a lightweight mechanism for alignment transfer and safety restoration. The manuscript has clear strengths: it states a precise operational definition, releases code, evaluates across three model families and multiple tasks, and includes quantitative comparisons against LIMA and Urial. The safety and detoxification results, in particular, are striking. The main risk is construct validity: because 'superficial knowledge' is defined as what a shallow linear projection can capture, the headline percentages partly restate the fact that a linear head fits much of the aligned distribution. The paper needs control experiments that separate linear decodability from superficiality, and repeated runs to support the precise numeric claims. With those additions, the contribution could be significant for the alignment interpretation literature.

major comments (3)

[§2.2, Eq. (3)] The central claim in the abstract and §2.3.1 that superficial knowledge constitutes a significant portion of alignment is partly definitional. Section 2.2 defines superficial knowledge as what the linear residual ΔW_b captures ('To ensure the extracted knowledge remains superficial, we restrict our modifications to shallow, simple structures'). The later statement that the base model plus superficial knowledge recovers 58% of GSM gains and 78% of TruthfulQA gains (Table 1) is then equivalent to saying a linear head trained on aligned logits fits part of the aligned output distribution. The paper does not provide an independent measure that the captured shifts are stylistic rather than content-bearing. The qualitative evidence (word clouds, selected token shifts) is suggestive but not a quantitative separation. I recommend adding control analyses, for example: (a) train linear probes on the same hidden states to predict task-relevant semantic content and show that the extracted residual does not carry such content; (b) compare the residual's effect to a linear map fitted to a deliberately deep transformation (e.g., a fine-tuned LoRA on the same data) to show the two are distinguishable; and (c) report the proportion of shift tokens that are function words versus content words. Without such controls, the quantitative 'superficial portion' is not established.
[§2.3.3, Table 2; §2.2, Eq. (1)] The autoregressive nature of decoding undermines the claim that a linear projection head only affects 'the final token selection process.' Changing the predicted token at step t changes the input to all subsequent transformer layers, so a single linear change can trigger deep computation and elicit knowledge stored in the frozen backbone. The paper's own example shows this: the shift '## → To' at the first token can steer the model into a calculation mode, and the later calculation error in the base-model-plus-superficial example shows that subsequent deep processing matters. Thus a 'shallow' modification can produce effects that are not shallow in origin. The manuscript should quantify the contribution of early-position versus later-position residual effects, for example by applying the residual only at the first few tokens and freezing it afterwards, and by measuring how much of the safety/toxicity elimination is due to a single early refusal-format token versus sustained stylistic control. This is load-bearing for the interpretation of the 0% harm rates in Table 1 as evidence of 'superficial' safety knowledge.
[§2.3.1, Tables 1, 3, 4; Appendix B] All quantitative claims rest on single runs with no variance or significance information. Appendix B states 'we evaluate in a single run' with do_sample=False. The headline percentages (58%, 78%, 88% recovery) are reported as precise numbers, but with a single run and small evaluation sets (100 samples for AdvBench and TruthfulQA) the uncertainty is likely large. For example, a change of a few correct answers on a 100-sample TruthfulQA evaluation changes the reported percentage by several points. I ask the authors to provide multiple random seeds or bootstrap confidence intervals for the main tables, and to state the exact number of test samples per metric. This is necessary for the quantitative 'portion of alignment' claim to be credible.

minor comments (5)

[§4, Conclusion] The sentence 'enabling us to quantify the the superficial portion of alignment' contains a duplicated article 'the the'; please fix.
[Tables 1 and 3] In Table 1 the entry for 13B-Chat reads '0.324+(0.258)' where the plus sign is outside the parenthesis, unlike the other rows; this is a formatting inconsistency. Please standardize the notation for deltas.
[Appendix B, Evaluation metrics] The text refers to 'two open-source fine-tuned LLaMA12' judges; this appears to be a typo for 'LLaMA2' judges, and the footnote anchors for the two judges are placed in a confusing way. Please clarify.
[Appendix C] The authors note that Urial fails on Qwen and therefore omit that baseline, but the main text (Tables 1 and 3) presents Urial as a comparison without mentioning that it is not applicable to all model families. A brief qualification in the main text would help readers interpret the cross-model comparisons.
[References] Several references are duplicated (Cobbe et al. 2021a/b, Lin et al. 2021a/b, Qi et al. 2023a/b); please merge them into single entries.

Circularity Check

1 steps flagged · score 5.0 of 10

Central label 'superficial' is built into the extraction constraint: the quantified 'superficial portion' of alignment is, by construction, the portion capturable by a linear head on frozen hidden states.

self definitional [Section 2.2, Eqs. (1)-(3); interpreted in Section 2.3.1 and Table 1]
"To ensure the extracted knowledge remains superficial, we restrict our modifications to shallow, simple structures, specifically, the linear projection head of the LLM. ... The resulting [∆Wb serves as an approximation of the superficial knowledge in the alignment process."

The abstract defines superficial knowledge as knowledge acquired through 'easily token restyling, without affecting the model's ability to capture underlying causal relationships between tokens.' Section 2.2 operationalizes that notion by fiat: the extraction method is restricted to a linear residual on the final projection head, and the fitted residual is then called 'the superficial knowledge.' The later quantification—'superficial knowledge indeed takes a large proportion of the alignment'—is therefore partly a restatement of the fitting procedure: it reports the behavioral alignment captured by a linear map on frozen hidden states and labels that linear-capturable portion 'superficial.' The benchmark numbers themselves are not tautological (the fit might have recovered little), and…

full rationale

The paper contains a genuine empirical contribution: a linear residual trained by distillation against aligned-model logits, evaluated on held-out benchmark splits, transferred across LLaMA-2 sizes, and used to restore safety after fine-tuning attacks. These results are not forced by the equations and would be informative even if the word 'superficial' were replaced by 'linearly capturable.' The circularity is concentrated in the interpretive claim that the method isolates superficial knowledge and that the measured recovery is the superficial portion of alignment. Section 2.2 guarantees by construction that the extracted component is shallow in the sense of touching only the final linear projection, but no evidence is supplied that shallow linear structure is equivalent to the paper's own definition of superficial as 'simple token restyling without affecting causal token relationships.' In fact, the paper's own GSM example shows that starting with 'To find' switches the model into a different reasoning mode, indicating that a final-head residual can trigger deeper autoregressive computation, so the identification is not harmless. The transfer and recovery experiments are self-contained against external benchmarks and do not depend on self-citations; there is no load-bearing self-citation. The paper's Section 5 limitation—'the non-superficial part in alignment is not fully understood'—is honest but does not repair the definitional conflation that drives the headline percentage interpretation. Overall, the central quantification is partially circular, but the experimental findings have independent content, so a moderate score of 5 is appropriate rather than a higher score that would imply the results are wholly forced.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The central claim rests on a small number of modeling assumptions: that a linear head captures only superficial behavior, that distillation adds no new knowledge, that the chosen benchmarks represent alignment, and that same-family logit spaces are transferable. The only fitted objects are the extracted residual matrices and the top-k choice, but the definitional link between 'superficial' and 'linear-head-capturable' is the biggest burden.

free parameters (3)

Linear residual delta_Wb = Trained via KL distillation against aligned model logits on roughly 1000 samples per task
This matrix is the extracted 'superficial knowledge' itself; the entire quantification depends on what this fitted residual captures.
Top-k logit truncation k = Selected around 500 via validation transfer accuracy (Figure 4)
The choice of k balances informativeness and transferability; it is chosen by hand from a validation trade-off and directly affects the black-box transfer results.
Learning rate = 1e-4
Reported in Appendix B with no sensitivity analysis; standard choice but still a hyperparameter.

assumptions (4)

domain assumption Alignment knowledge that can be modeled by changing only the final linear projection head is superficial.
Stated in Section 2.2; this is the operational definition and the load-bearing premise of the entire quantification.
domain assumption KL distillation from the aligned model into a frozen-backbone head introduces no new knowledge.
Section 2.2: 'we avoid standard fine-tuning... Instead, we apply distillation.' This assumes distillation transfers only aligned behavior without adding information beyond the aligned model.
domain assumption The four benchmarks (GSM, Toxigen, Advbench, TruthfulQA) adequately measure alignment.
Appendix B uses these datasets to quantify the superficial portion; alignment is broader than these four tasks, so the measured 'portion' is relative to these benchmarks only.
domain assumption Models from the same family share a transferable logit space.
Section 3.1: 'models from the same family typically share the same vocabulary... the logits space offers a consistent input structure.' This is required for the weak-to-strong transfer claim.

invented entities (1)

Superficial knowledge as a separable, storable component of alignment independent evidence
purpose: Names and quantifies the portion of alignment attributable to token restyling, and enables transfer and recovery experiments.
The paper provides behavioral evidence on held-out benchmarks and cross-model transfer, though the entity is defined by the extraction method rather than by an independent measurement outside the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Extracting and Understanding the Superficial Knowledge in Alignment." pith.science (2026). https://pith.science/paper/NNEVGTUC

@misc{pith2026250204602,
  author       = {Pith},
  title        = {Pith review of: Extracting and Understanding the Superficial Knowledge in Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNEVGTUC}},
  note         = {Machine review of arXiv:2502.04602}
}

read the original abstract

Alignment of large language models (LLMs) with human values and preferences, often achieved through fine-tuning based on human feedback, is essential for ensuring safe and responsible AI behaviors. However, the process typically requires substantial data and computation resources. Recent studies have revealed that alignment might be attainable at lower costs through simpler methods, such as in-context learning. This leads to the question: Is alignment predominantly superficial? In this paper, we delve into this question and provide a quantitative analysis. We formalize the concept of superficial knowledge, defining it as knowledge that can be acquired through easily token restyling, without affecting the model's ability to capture underlying causal relationships between tokens. We propose a method to extract and isolate superficial knowledge from aligned models, focusing on the shallow modifications to the final token selection process. By comparing models augmented only with superficial knowledge to fully aligned models, we quantify the superficial portion of alignment. Our findings reveal that while superficial knowledge constitutes a significant portion of alignment, particularly in safety and detoxification tasks, it is not the whole story. Tasks requiring reasoning and contextual understanding still rely on deeper knowledge. Additionally, we demonstrate two practical advantages of isolated superficial knowledge: (1) it can be transferred between models, enabling efficient offsite alignment of larger models using extracted superficial knowledge from smaller models, and (2) it is recoverable, allowing for the restoration of alignment in compromised models without sacrificing performance.

Figures

Figures reproduced from arXiv: 2502.04602 by the authors.

**Figure 2.** KL divergence between the base model and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Trade-off between informativeness(Shift to [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: K vs Token Accuracy (e.g., 500, as shown in the figure), increasing k enriches the information base, thereby enhancing both validation accuracy and validation transfer accuracy. However, surpassing this threshold, the test accuracy plateaus and the transfer accuracy de…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding
cs.CL 2025-06 conditional novelty 6.0 of 10

FoReaL-Decoding lets a strong reasoning model generate the first few tokens of each sentence and a weaker model complete the sentence, cutting theoretical FLOPs by 30-55% while retaining 86-100% of accuracy on four ma...

Reference graph

Works this paper leans on

13 extracted references · 8 canonical work pages · cited by 1 Pith paper

[1]

The alarm rings three times a day

work page
[2]

Advances in neural in- formation processing systems, 35:27730–27744

Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023a. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310...

arXiv
[3]

The second time it went off, it rang for three times as long as the first time, which means it rang for 4 x 3 = 12 times

work page
[4]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al

Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. Johann...

arXiv 2022
[6]

demonstrated that alignment can be achieved through in-context learning. However, these studies only show that alignment can be accomplished using superficial methods to a certain degree, without fully validating the hypothesis or assessing the extent to which alignment is superficial. In this paper, we explore the superficiality of knowledge introduced d...

work page 2022
[8]

The first time it went off today, it rang four times

work page
[10]

So, the alarm rang a total of 4 + 12 + 6 = 22 times today

The third time it went off, it rang for half as long as the second time, which means it rang for 12 / 2 = 6 times. So, the alarm rang a total of 4 + 12 + 6 = 22 times today. Alignment Token Shift Examples: (## → Let) ( $ → ’ ) (say → break) (problem → information) (question→ problem ) (.→: ) (The → 1 ) (When→ The ) (alarm→ first ) (.→ , ) (so→ which ) (is...

work page
[11]

The first time the Alarm went off, it rang for 4 times

work page

Show all 13 references

[12]

The second time the Alarm went off, it rang for 3 times as long as the first time, which is 3 x 4 = 12 seconds
[13]

So, in total, the Alarm rang for 4 + 12 + 6 = 22 seconds

The third time the Alarm went off, it rang for half as long as the second time, which is 12 / 2 = 6 seconds. So, in total, the Alarm rang for 4 + 12 + 6 = 22 seconds. </s> Shallow Align Token Shift Examples: (## → Let) ( $ → ’ ) (say→ break), (down→ this) (into→ step ), (.→: )...
[2022]

arXiv preprint arXiv:2210.10683

Why should adversarial perturbations be im- perceptible? rethink the research paradigm in adver- sarial nlp. arXiv preprint arXiv:2210.10683. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024. Self-play fine-tuning con- verts weak language models to stro...

2024 arXiv
[2023]

arXiv preprint arXiv:2301.11774

Reinforcement learning from diverse human preferences. arXiv preprint arXiv:2301.11774. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:23...

2023 arXiv
[2024]

Advances in Neu- ral Information Processing Systems, 36

Direct preference optimization: Your language model is secretly a reward model. Advances in Neu- ral Information Processing Systems, 36. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2024. Pref- erence ranking optimization for human al...

2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

The alarm rings three times a day

work page

[2] [2]

Advances in neural in- formation processing systems, 35:27730–27744

Training language models to follow instruc- tions with human feedback. Advances in neural in- formation processing systems, 35:27730–27744. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023a. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310...

arXiv

[3] [3]

The second time it went off, it rang for three times as long as the first time, which means it rang for 4 x 3 = 12 times

work page

[4] [4]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al

Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. Johann...

arXiv 2022

[5] [6]

demonstrated that alignment can be achieved through in-context learning. However, these studies only show that alignment can be accomplished using superficial methods to a certain degree, without fully validating the hypothesis or assessing the extent to which alignment is superficial. In this paper, we explore the superficiality of knowledge introduced d...

work page 2022

[6] [8]

The first time it went off today, it rang four times

work page

[7] [10]

So, the alarm rang a total of 4 + 12 + 6 = 22 times today

The third time it went off, it rang for half as long as the second time, which means it rang for 12 / 2 = 6 times. So, the alarm rang a total of 4 + 12 + 6 = 22 times today. Alignment Token Shift Examples: (## → Let) ( $ → ’ ) (say → break) (problem → information) (question→ problem ) (.→: ) (The → 1 ) (When→ The ) (alarm→ first ) (.→ , ) (so→ which ) (is...

work page

[8] [11]

The first time the Alarm went off, it rang for 4 times

work page