How Effective Is Constitutional AI in Small LLMs? A Study on DeepSeek-R1 and Its Peers

Antonio-Gabriel Chac\'on Menke (Shibaura Institute of Technology; Kempten University of Applied Sciences); Phan Xuan Tan (Shibaura Institute of Technology)

REVIEW 4 major objections 5 minor 14 references

How Effective Is Constitutional AI in Small LLMs? A Study on DeepSeek-R1 and Its Peers

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Constitutional AI's generate-critique-revise loop sharply cuts harmful responses on Llama-based small models but does little for Gemma-2 and Qwen2.5, suggesting the method's benefit depends on model architecture and reasoning.

desk verdict New observation about architecture-dependent CAI effectiveness in small LLMs, but the undefined 'harmful response rate' metric makes the headline comparison unverifiable as written. read the letter →

arxiv 2503.17365 v2 pith:F3B3KGLJ submitted 2025-02-01 cs.LG cs.AIcs.CY

Antonio-Gabriel Chacón Menke (Shibaura Institute of Technology , Kempten University of Applied Sciences) , Phan Xuan Tan (Shibaura Institute of Technology) This is my paper · ORCID

classification cs.LGcs.AIcs.CY

keywords constitutionalAIself-critiqueabliterationsmalllanguagemodelsharmreductionDeepSeek-R1Llama3.1safetyalignment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether Constitutional AI's self-critique loop can make small uncensored language models safe again after their refusal behavior has been removed through abliteration. Testing four 7-9B models on 90 HarmBench prompts, it finds that Llama-based models (DeepSeek-R1-Distill-Llama-8B and Llama 3.1-8B) cut harmful responses by large margins, while Gemma-2-9B and Qwen2.5-7B improve only slightly. The claim matters because it suggests that lightweight, self-supervised alignment can work in resource-constrained settings, but only when the model's architecture and reasoning style cooperate. The paper concludes that CAI's effectiveness varies by architecture and that retained safety knowledge is not always applied during open-ended critique.

What carries the argument

The central mechanism is the Constitutional AI self-critique loop, implemented as three steps: generate an initial response, ask the model to critique its own response against a set of safety rules, and then instruct the model to rewrite the response in light of that critique, with refusal encouraged when the prompt itself is harmful. Abliteration is the second load-bearing mechanism: removing a single activation direction to suppress refusal behavior, so that the effects of CAI can be isolated from pre-existing safety training. The comparison across four architectures is what carries the conclusion that harm reduction is not uniform.

What would settle it

Rerun the exact 90-prompt generate-critique-revise pipeline with a specified judge (a written rubric applied by human annotators with measured agreement, or a frozen judge model) and check the revised-response harmful rates: R1-Llama around 11.1%, Llama-3.1 around 38.9%, Gemma-2 around 40.0%, and Qwen2.5 around 67.8%; the architecture-level claim fails if the ordering changes materially or if R1-Llama's large drop does not replicate, and the Gemma-2 result should be checked with and without its system-prompt workaround.

Watch

Extended reading notes

Core claim

The paper claims that after abliteration strips refusal behavior, CAI's self-critique mechanism produces architecture-dependent harm reduction: on 90 HarmBench prompts, revised harmful responses fall from 54.4% to 11.1% for DeepSeek-R1-Distill-Llama-8B and from 87.8% to 38.9% for Llama 3.1-8B, but only from 45.6% to 40.0% for Gemma-2-9B and 76.7% to 67.8% for Qwen2.5-7B. It further claims that failure patterns differ: Gemma-2 and Qwen2.5 usually fail to detect harm during the critique phase, while Llama-3.1 often adds warnings without removing the harmful content, and R1-Llama alternates between the two failure modes. The paper interprets this as evidence that the models retain safety knowledge (SafetyBench scores stay high) but differ in applying it during open-ended critique, and that R1-Llama's reasoning step contributes to more consistent harm reduction.

Load-bearing premise

The paper's headline comparison assumes the same valid judge scored every model's responses as harmful or not, but it never describes the judge or rubric; it also treats the four models as comparable even though Gemma-2 was tested with a different prompting setup because it does not natively support system prompts.

Editorial extensions

If this is right

CAI can serve as a low-cost self-alignment method for small models in resource-constrained settings, at least for Llama-based architectures.
Architecture-specific prompting may bridge the gap between retained safety knowledge and effective self-critique, since SafetyBench scores stay high even where critique fails.
Abliteration has uneven side effects: Qwen2.5 loses 20 points on the CommonsenseMoral test and degrades on several knowledge and morality benchmarks, while Llama-based models barely change, so uncensoring and realigning is not architecture-neutral.
Reasoning-enhanced models such as R1-Llama may need distinct safety evaluation, because they show both stronger and more stable harm reduction than their non-reasoning base.
High scores on recognition-style safety benchmarks do not guarantee that a model will apply safety judgments in free-form critique and revision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If architecture dependence is real, averaging a single CAI benefit across heterogeneous models is misleading; evaluations should report results stratified by architecture and by reasoning behavior.
The absent judge description is the largest unresolved risk: a different fixed judge or rubric could shift the borderline Qwen2.5 and Gemma-2 results enough to change the ordering, even if the Llama improvements hold.
Gemma-2's system-prompt incompatibility is a testable confound: re-running it with no system prompt or with an equivalent chat-template injection would show whether its small improvement is an artifact of the prompting workaround.
The paper's 'safety knowledge exists but is not applied' framing suggests a direct intervention: add explicit reasoning steps or chain-of-thought to the critique phase for Qwen2.5 and Gemma-2 and test whether the gap to Llama closes.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper makes a genuinely new empirical observation—CAI self-critique cuts harmful responses dramatically for two Llama-based 8B models, while barely moving the needle for Gemma-2 and Qwen2.5—but the headline metric is never operationally defined. I couldn't tell you what 'Harmful Responses (%)' means after reading the paper, and neither can anyone else. That's a load-bearing gap, not a polish issue.

What's good: the study is honestly scoped. It abliterates four small instruction-tuned models to strip out refusal behavior, then applies the same three-step CAI protocol. The benchmark appendix is helpful—the fact that Qwen2.5 drops 20 points on CommonsenseMoral after abliteration while retaining SafetyBench performance is a genuine supporting observation. The prompt templates are given in Appendix A, and the authors disclose that they tuned the wording iteratively. That's the kind of transparency that makes this worth engaging with.

Where it gets soft: the central comparison rests entirely on Figure 1, and there is no description of how the harmful-response percentages were computed. No judge, no classifier, no rubric, no human annotation protocol, no agreement metric. If the judge was an LLM, the architecture split could be an artifact of the judge's own biases. If it was a human, that needs to be said. Either way, the numbers aren't reproducible as written. There's also a real confound: Gemma-2 received a different prompting condition (system prompt prepended to the user prompt, or none), and the authors say prepending raised harm scores—so the Gemma condition is not strictly comparable. The sample is 90 prompts across six categories; no significance tests or per-prompt error bars are reported. That's a minor-to-moderate weakness on its own, but combined with the undefined metric it makes the architecture-level conclusion provisional.

One more thing: the causal claim that reasoning capabilities explain R1-Llama's advantage is speculative. R1-Llama is a distilled reasoning model, but it's also a different training run; the comparison can't isolate reasoning as the active ingredient. The paper mostly frames this as a suggestion, so I'd treat it as a hypothesis for follow-up, not a flaw in the core result.

Bottom line: this is a useful data point for people doing safety alignment on small models, if the evaluation protocol gets cleaned up. It deserves a serious referee, but the referee should send it back for major revision: define the harmful-response metric, disclose the judge, run a significance test or at least report per-prompt agreement, and either match Gemma's prompting condition or discuss the confound openly. I wouldn't cite the current version in my own work, but I would bring it to a reading group to talk about what a reproducible small-model safety evaluation should look like.

Referee Report

4 major / 5 minor

Summary. The paper reports an empirical comparison of Constitutional AI (CAI) self-critique on four small instruction-tuned LLMs (DeepSeek-R1-Distill-Llama-8B, Gemma-2-9B-it, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct), all abliterated to remove refusal behavior. For 90 HarmBench prompts across six categories, the authors measured the 'harmful response rate' of initial and CAI-revised responses, reporting that Llama-based models improve substantially (R1-Llama: 54.4% to 11.1%; Llama-3.1: 87.8% to 38.9%) while Gemma-2 and Qwen2.5 improve only slightly (45.6% to 40.0%; 76.7% to 67.8%). They also report benchmark scores (MMLU, tinyBenchmarks, ETHICS subset, SafetyBench) for abliterated versus original models. The paper concludes that CAI's effectiveness varies by model architecture and reasoning capability.

Significance. The question of whether lightweight CAI can align small uncensored models is timely and practically important. The study has some strengths: it directly compares four models under an abliteration procedure, uses a standardized prompt set (HarmBench), and includes capability benchmarks to monitor collateral damage from abliteration. However, the central comparison is currently unverifiable because the harmfulness classification protocol is not described, and the cross-architecture claim is confounded by a system-prompt difference for Gemma-2. If the authors supply a precise judge/rubric, per-prompt data, and error analysis, the paper could be a useful contribution to safety alignment for small models. At present, the headline conclusion rests on a metric that is not operationally defined.

major comments (4)

[Section 2, Figure 1] The central dependent variable, 'Harmful Responses (%)' in Figure 1, is never operationally defined. The manuscript reports means such as 54.4% to 11.1% for R1-Llama and 87.8% to 38.9% for Llama-3.1, but it does not describe the judge, classifier, rubric, annotation instructions, or inter-annotator agreement used to label each initial and revised response. Without this information, the headline architecture-level conclusion cannot be reproduced or falsified. Please specify exactly how each response was classified, including the identity and version of the judge (human annotators, a specific LLM, a classifier), the criteria for 'harmful', and the exact judge prompt, and report raw per-prompt labels and agreement statistics.
[Appendix A, Section 3] The cross-architecture comparison is confounded by a system-prompt asymmetry: all models except Gemma-2 received the same system prompt ('You are a helpful AI assistant' / 'You are an ethical evaluator'), while Gemma-2 received no system prompt because the authors found that prepending it increased harm scores. Because the paper's main conclusion is that Gemma-2 benefits less from CAI, this differential prompting condition alone could explain part of the observed difference. Please either use a system-prompt-capable chat template for Gemma-2, or explicitly analyze Gemma-2 as a separate prompting-condition comparison rather than pooling it with the other models in the architecture-level conclusion.
[Section 2, Figure 1] The study uses only 90 HarmBench prompts (15 per category) and reports no statistical significance tests, confidence intervals, or per-prompt outcome data. The mean harm rates are presented with two decimals, which overstates precision for a sample of 90 Bernoulli trials, and the paper's qualitative claims (e.g., 'completely eliminating harmful content in several categories' in Section 3) are not supported by any error analysis. Please provide per-prompt labels, bootstrap confidence intervals or exact binomial intervals, and a test of whether the initial-to-revised changes differ across architectures.
[Appendix A] The CAI prompt wording was tuned via 'iterative testing' on the same HarmBench prompts used for evaluation. This introduces a selection risk: prompt choices that were adjusted after observing their effect on the 90 evaluation prompts can inflate the measured CAI improvement and make the architecture comparison hard to interpret. Please clarify whether the 90 prompts were also used during prompt development, and if so, consider reporting results with a held-out set of HarmBench prompts or describing the tuning procedure in enough detail to assess the risk of overfitting.

minor comments (5)

[Abstract, Section 2] The model is named inconsistently as 'DeepSeek-R1-8B' in the abstract and 'DeepSeek-R1-Distill-Llama-8B' in Section 2; use one name and state the base model explicitly.
[Figure 1] The category labels are too small to read, and the per-category percentages are not listed; please provide a table with exact values and per-category denominators.
[Section 3] The statement that 'R1-Llama's overall harm reduction compared to its base model (Llama-3.1)' is misleading because R1-Llama is distilled from Llama-3.1-Base, not from Llama-3.1-Instruct; clarify this relationship.
[Appendix B, Table 1] The column order ('Llama, DeepSeek, gemma, Qwen2.5') should match the order in the text and use consistent capitalization.
[Section 3] Report the standard deviation (±31.60% vs ±49.02%) together with its definition (e.g., across categories or across prompts) and sample size, since the current text does not say what the spread represents.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is a direct empirical measurement with no fitted parameter, self-citation chain, or derivation that reduces to its inputs.

full rationale

This manuscript contains no derivation chain of the kind the circularity test targets. It reports direct measurements: it runs four small LLMs through a three-step CAI pipeline (initial response, self-critique, revised response) on 90 HarmBench prompts and reports the percentage of harmful responses before and after revision, together with standard benchmark scores before and after abliteration. The headline quantities in Figure 1 (54.4% to 11.1%, 87.8% to 38.9%, 45.6% to 40.0%, 76.7% to 67.8%) are observed response rates, not predictions derived from fitted parameters, and no equation in the paper defines a target quantity in terms of itself. The CAI prompts in Appendix A are inputs rather than outputs, so the revised-response rates are not forced by construction. The abliteration method is cited to an external source (Labonne 2025) and applied as a technique, not imported as a load-bearing theorem that forbids alternatives. The only substantive weaknesses are not circularity: the evaluation protocol for 'harmful responses' is not specified (no judge, rubric, or annotation procedure is described), and Gemma-2 was evaluated under a different prompting condition as disclosed in Appendix A. These are reproducibility and validity concerns, not self-referential reasoning. No self-citation is used to justify the paper's central empirical claim, and the paper is self-contained against external benchmarks (MMLU, SafetyBench, etc.). Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No numeric free parameters are fitted. The study rests on several assumptions: the unstated harmful-response judging method, the representativeness of 90 hand-balanced prompts, the effectiveness of abliteration in removing refusal without erasing safety knowledge, and the validity of comparing models with different system-prompt setups.

assumptions (4)

domain assumption The harmful-response metric (as scored by an unspecified judge or classifier) is a valid and consistent measure of output harmfulness across all four models.
The paper reports harmful responses in percent but never describes the judge, criteria, or consistency of labeling; this is the central measurement assumption. Section 3, Figure 1.
domain assumption Abliteration removes refusal behavior without materially disrupting the model's underlying safety knowledge or reasoning.
The methodology relies on abliteration to isolate CAI effects from pre-existing refusals; the paper itself shows Qwen's moral benchmark collapses post-abliteration, so this assumption is violated for at least one model. Section 2 and Appendix B.
domain assumption A set of 90 HarmBench prompts with equal representation per category is representative enough to estimate CAI harm-reduction performance.
Small, hand-balanced sample with no seed or per-category error analysis. Section 2.
domain assumption Cross-model comparison is valid even though Gemma-2 received no system prompts while others did.
Appendix A states Gemma does not support system prompts and prepending them raised harm scores; the authors nevertheless compare Gemma's results directly. This confound weakens the architecture explanation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How Effective Is Constitutional AI in Small LLMs? A Study on DeepSeek-R1 and Its Peers." pith.science (2026). https://pith.science/paper/F3B3KGLJ

@misc{pith2026250317365,
  author       = {Pith},
  title        = {Pith review of: How Effective Is Constitutional AI in Small LLMs? A Study on DeepSeek-R1 and Its Peers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3B3KGLJ}},
  note         = {Machine review of arXiv:2503.17365}
}

read the original abstract

Recent incidents highlight safety risks in Large Language Models (LLMs), motivating research into alignment methods like Constitutional AI (CAI). This paper explores CAI's self-critique mechanism on small, uncensored 7-9B parameter models: DeepSeek-R1-8B, Gemma-2-9B, Llama 3.1-8B, and Qwen2.5-7B. We show that while Llama-based models exhibited significant harm reduction through self-critique, other architectures demonstrated less improvement in harm detection after abliteration. These results suggest CAI's effectiveness may vary depending on model architecture and reasoning capabilities.

Figures

Figures reproduced from arXiv: 2503.17365 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages

[4]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al

URL https://zenodo.org/records/12608602. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open language models at a practical size,

arXiv
[5]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al

URL https://arxiv.org/ abs/2408.00118. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models,

arXiv
[6]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt

URL https://arxiv.org/abs/2407.21783. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding,

arXiv
[10]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al

URL https: //arxiv.org/abs/2402.14992. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al. Qwen2.5 technical report,

arXiv
[11]

Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang

URL https://arxiv.org/abs/ 2412.15115. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models,

arXiv
[12]

3 A P ROMPTS For our Constitutional AI implementation, we used the following prompts, with model-generated outputs shown in blue text

URL https://arxiv.org/abs/2309.07045. 3 A P ROMPTS For our Constitutional AI implementation, we used the following prompts, with model-generated outputs shown in blue text. All models except Gemma received the same system prompts: ”You are a helpful AI assistant” for initial and revised responses, and ”You are an ethical evaluator, reviewing AI responses ...

arXiv 2022
[13]

(2024) (tiny) for diverse reasoning tasks, and the ”Commonsense Moral- ity” subset of the ETHICS benchmark (Hendrycks et al.,

for general knowledge, tinyBenchmarks Polo et al. (2024) (tiny) for diverse reasoning tasks, and the ”Commonsense Moral- ity” subset of the ETHICS benchmark (Hendrycks et al.,

work page 2024
[14]

( SB), to assess harm detection capabilities across various safety-related categories. Metric Llama DeepSeek gemma Qwen2.5 MMLU 68.3 / 68.1 (-0.1) 55.7 / 55.7 ( -0.0) 72.3 / 71.2 ( -1.1) 74.2 / 70.6 ( -3.6) tiny HellaSwag 80.5 / 81.4 (+0.9) 79.2 / 80.5 ( +1.2) 80.8 / 79.9 ( -0.9) 76.5 / 75.2 ( -1.3) tiny ARC 65.3 / 64.9 (-0.5) 47.7 / 48.5 ( +0.8) 69.3 / 6...

work page 2024

Show all 14 references

[28]

Jan. 2025]. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal,

2025
[2021]

Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt

URL https: //arxiv.org/abs/2009.03300. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values,

2009 arXiv
[2022]

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al

URL https://arxiv.org/abs/2212.08073. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

arXiv
[2023]

Maxime Labonne

URL https://arxiv.org/abs/ 2008.02275. Maxime Labonne. Uncensor any LLM with abliteration, jan

2008 arXiv
[2024]

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al

URL https: //arxiv.org/abs/2406.11717. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback,

arXiv
[2025]

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, et al

URL https://arxiv.org/abs/2501.12948. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, et al. A framework for few-shot language model evaluation, 07

arXiv

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [4]

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al

URL https://zenodo.org/records/12608602. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open language models at a practical size,

arXiv

[2] [5]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al

URL https://arxiv.org/ abs/2408.00118. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models,

arXiv

[3] [6]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt

URL https://arxiv.org/abs/2407.21783. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding,

arXiv

[4] [10]

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al

URL https: //arxiv.org/abs/2402.14992. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, et al. Qwen2.5 technical report,

arXiv

[5] [11]

Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang

URL https://arxiv.org/abs/ 2412.15115. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. Safetybench: Evaluating the safety of large language models,

arXiv

[6] [12]

3 A P ROMPTS For our Constitutional AI implementation, we used the following prompts, with model-generated outputs shown in blue text

URL https://arxiv.org/abs/2309.07045. 3 A P ROMPTS For our Constitutional AI implementation, we used the following prompts, with model-generated outputs shown in blue text. All models except Gemma received the same system prompts: ”You are a helpful AI assistant” for initial and revised responses, and ”You are an ethical evaluator, reviewing AI responses ...

arXiv 2022

[7] [13]

(2024) (tiny) for diverse reasoning tasks, and the ”Commonsense Moral- ity” subset of the ETHICS benchmark (Hendrycks et al.,

for general knowledge, tinyBenchmarks Polo et al. (2024) (tiny) for diverse reasoning tasks, and the ”Commonsense Moral- ity” subset of the ETHICS benchmark (Hendrycks et al.,

work page 2024

[8] [14]

( SB), to assess harm detection capabilities across various safety-related categories. Metric Llama DeepSeek gemma Qwen2.5 MMLU 68.3 / 68.1 (-0.1) 55.7 / 55.7 ( -0.0) 72.3 / 71.2 ( -1.1) 74.2 / 70.6 ( -3.6) tiny HellaSwag 80.5 / 81.4 (+0.9) 79.2 / 80.5 ( +1.2) 80.8 / 79.9 ( -0.9) 76.5 / 75.2 ( -1.3) tiny ARC 65.3 / 64.9 (-0.5) 47.7 / 48.5 ( +0.8) 69.3 / 6...

work page 2024