Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs

Kai Jun Teh; Qibing Ren; Shei Pern Chua; Xiao Li; Xiaolin Hu; Zhen Leng Thai

arxiv: 2509.05367 · v5 · pith:TDD5C6FQnew · submitted 2025-09-04 · 💻 cs.CR · cs.AI

Between a Rock and a Hard Place: The Tension Between Ethical Reasoning and Safety Alignment in LLMs

Shei Pern Chua , Zhen Leng Thai , Kai Jun Teh , Xiao Li , Qibing Ren , Xiaolin Hu This is my paper

Pith reviewed 2026-05-18 19:32 UTC · model grok-4.3

classification 💻 cs.CR cs.AI

keywords LLM safety alignmentred-teamingethical dilemmasadversarial attacksLoRA architecturemoral reasoningharmful request framing

0 comments

The pith

Ethical reasoning in LLMs opens a vulnerability where harmful requests framed as moral dilemmas can bypass safety alignments.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows that binary safe-or-unsafe classifications in LLM alignment fail when models face ethical dilemmas requiring moral trade-off reasoning. This creates an exploitable surface because the model can be led to treat harmful actions as necessary compromises in a moral sense. The authors introduce TRIAL as a multi-turn method to red-team by embedding harmful requests in ethical contexts and demonstrate high success rates on tested models. They counter this with ERR, a defense that separates enabling responses from pure analysis using a specialized LoRA-based architecture to preserve utility.

Core claim

Safety alignment in large language models assumes requests are either safe or unsafe, but ethical dilemmas expose a gap where reasoning about moral trade-offs allows framing harmful actions as morally necessary. TRIAL exploits this by systematically presenting harmful requests within ethical framings to achieve high attack success rates. ERR addresses it by distinguishing instrumental responses that enable harm from explanatory ones that analyze without endorsing, implemented through a Layer-Stratified Harm-Gated LoRA to maintain model performance.

What carries the argument

The TRIAL red-teaming method that embeds harmful requests in ethical framings to exploit moral reasoning, paired with the ERR defense framework that partitions responses into instrumental and explanatory categories using Layer-Stratified Harm-Gated LoRA.

If this is right

Binary safety classifications are insufficient and must be expanded to handle ethical dilemma scenarios.
Models may endorse harm when it is presented as a moral necessity in a dilemma.
Targeted defenses can block enabling ethical responses while allowing analysis of ethical issues.
Overall model utility can be preserved during defense implementation through stratified training approaches.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Improving a model's ethical reasoning depth could heighten this vulnerability if not accompanied by response-type guards.
Alignment techniques might need to include dilemma simulation during training to build resistance.
Similar tensions could appear in other reasoning domains like legal or medical advice where trade-offs are common.

Load-bearing premise

Ethical reasoning outputs can be reliably sorted into those that enable harmful actions versus those that merely discuss ethics without supporting harm.

What would settle it

Running the TRIAL attacks on a model trained with ERR and observing whether attack success rates drop substantially while performance on standard ethical reasoning benchmarks remains high.

Figures

Figures reproduced from arXiv: 2509.05367 by Kai Jun Teh, Qibing Ren, Shei Pern Chua, Xiao Li, Xiaolin Hu, Zhen Leng Thai.

**Figure 2.** Figure 2: TRIAL’s pipeline consists of two stages: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: Extending TRIAL interactions (up to K=10 rounds) for 30 JBB-Behaviors prompts initially failing against [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

read the original abstract

Large Language Model safety alignment predominantly operates on a binary assumption that requests are either safe or unsafe. This classification proves insufficient when models encounter ethical dilemmas, where the capacity to reason through moral trade-offs creates a distinct attack surface. We formalize this vulnerability through TRIAL, a multi-turn red-teaming methodology that embeds harmful requests within ethical framings. TRIAL achieves high attack success rates across most tested models by systematically exploiting the model's ethical reasoning capabilities to frame harmful actions as morally necessary compromises. Building on these insights, we introduce ERR (Ethical Reasoning Robustness), a defense framework that distinguishes between instrumental responses that enable harmful outcomes and explanatory responses that analyze ethical frameworks without endorsing harmful acts. ERR employs a Layer-Stratified Harm-Gated LoRA architecture, achieving robust defense against reasoning-based attacks while preserving model utility.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper flags how ethical reasoning creates a distinct opening for multi-turn attacks on aligned LLMs and offers a defense layer, but the evidence for both the attack novelty and the fix is still thin on controls and numbers.

read the letter

The main takeaway is that safety alignments built on binary safe/unsafe labels leave models open when they have to reason through moral trade-offs. The authors turn this into TRIAL, a multi-turn red-teaming method that frames harmful requests as necessary ethical compromises, and they report it succeeds across several models. They then propose ERR, which tries to steer responses toward analysis rather than action by using a Layer-Stratified Harm-Gated LoRA setup that keeps utility intact while blocking enabling outputs. That distinction between instrumental and explanatory responses is a practical way to think about the problem and is not something standard alignment papers usually isolate. The work is empirical and introduces a concrete attack plus a targeted defense, which gives it a clear hook for people working on real deployments where models might face gray-area queries. The soft spot is exactly the one the stress-test note raises: there is no clear ablation that swaps out the moral-framing language while holding turn count and request content fixed. Without that, it is hard to tell whether the high success rates come from the ethical angle or from any persistent multi-turn structure. The abstract also gives no model list, exact metrics, or baseline comparisons, so the central claims stay hard to judge until the full results are checked. This is the kind of paper that belongs in a reading group for AI safety folks who already track jailbreak and alignment work. Readers who want to test whether current defenses hold up under moral reasoning will get something out of it, even if they end up running their own controls. It deserves to go to peer review because the underlying concern is real and the proposed framing is new enough to warrant referee input, but the referees should insist on the missing ablations and quantitative details before any stronger claims are accepted.

Referee Report

2 major / 2 minor

Summary. The paper argues that binary safe/unsafe classification in LLM safety alignment is insufficient for ethical dilemmas, where moral trade-off reasoning creates an attack surface. It introduces TRIAL, a multi-turn red-teaming method that embeds harmful requests in ethical framings to achieve high attack success rates by exploiting models' ethical reasoning to justify harmful actions as moral compromises. It then proposes ERR, a defense framework using a Layer-Stratified Harm-Gated LoRA architecture to distinguish instrumental (harm-enabling) from explanatory (analysis-only) responses while preserving utility.

Significance. If the empirical results hold with proper controls, the work identifies a novel, reasoning-based vulnerability in aligned LLMs and supplies a corresponding defense (ERR) that targets the identified failure mode. The introduction of the TRIAL methodology and the ERR framework are concrete contributions that could inform future alignment research; the empirical focus on ethical framing as a distinct attack vector is a strength if ablations confirm causality.

major comments (2)

[§4] §4 (Experimental Evaluation) and abstract: The central claim that TRIAL achieves high attack success rates specifically by exploiting ethical reasoning to frame harmful actions as morally necessary compromises lacks an ablation that replaces the moral-trade-off language with neutral multi-turn scaffolding while holding request content, turn count, and context length fixed. Without this control, it is impossible to isolate ethical framing as the causal driver versus generic persistence or multi-turn effects.
[Abstract, §4] Abstract and §4: The abstract asserts 'high attack success rates across most tested models' and 'robust defense' for ERR but supplies no quantitative metrics, model list, baselines, or ablation details. This renders the primary empirical claims unverifiable from the provided summary and undermines assessment of whether the results support the stated conclusions.

minor comments (2)

[§3.2] Clarify the precise definition and decision criteria used to partition responses into 'instrumental' versus 'explanatory' categories in the ERR framework, including any inter-annotator agreement or automated classification details.
[§5] Add explicit discussion of potential new failure modes introduced by the ERR defense, such as over-refusal on legitimate ethical analysis queries.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We have revised the manuscript to strengthen the experimental controls and to make the quantitative claims more explicit in the abstract and main text.

read point-by-point responses

Referee: [§4] §4 (Experimental Evaluation) and abstract: The central claim that TRIAL achieves high attack success rates specifically by exploiting ethical reasoning to frame harmful actions as morally necessary compromises lacks an ablation that replaces the moral-trade-off language with neutral multi-turn scaffolding while holding request content, turn count, and context length fixed. Without this control, it is impossible to isolate ethical framing as the causal driver versus generic persistence or multi-turn effects.

Authors: We agree that isolating the contribution of ethical framing requires a tightly controlled comparison. Our original evaluation included multi-turn baselines without explicit moral-trade-off language, but these did not hold every variable (including exact phrasing length and context) perfectly fixed. In the revised §4 we have added the requested ablation: we replace the moral-trade-off framing with neutral multi-turn scaffolding while keeping request content, turn count, and context length identical. The new results show a substantial drop in attack success rate under the neutral condition, supporting the claim that ethical reasoning is the primary driver rather than generic multi-turn persistence. revision: yes
Referee: [Abstract, §4] Abstract and §4: The abstract asserts 'high attack success rates across most tested models' and 'robust defense' for ERR but supplies no quantitative metrics, model list, baselines, or ablation details. This renders the primary empirical claims unverifiable from the provided summary and undermines assessment of whether the results support the stated conclusions.

Authors: We acknowledge that the original abstract was too high-level. The full §4 already contains the requested details (attack success rates for GPT-4, Claude-3, Llama-3, and Mistral models; comparison to standard jailbreak baselines; and ERR ablations). We have now updated the abstract to report the key quantitative figures (TRIAL ASR >75 % on most models, ERR reducing ASR to <12 % with negligible utility loss) and to list the models and main baselines, making the claims directly verifiable. revision: yes

Circularity Check

0 steps flagged

No significant circularity; empirical methodology is self-contained

full rationale

The paper introduces TRIAL as a novel multi-turn red-teaming approach that embeds harmful requests in ethical framings and ERR as a Layer-Stratified Harm-Gated LoRA defense that separates instrumental from explanatory responses. All central claims rest on direct experimental measurements of attack success rates and utility preservation across tested models rather than any mathematical derivation, parameter fitting presented as prediction, or load-bearing self-citation. No equations, uniqueness theorems, or ansatzes are invoked that reduce to prior inputs by construction, so the work remains independent of circular self-referential structures.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 2 invented entities

The central claims rest on the assumption that LLMs exhibit exploitable ethical reasoning and that response types can be cleanly separated during training. No free parameters or invented physical entities are introduced; the new methods themselves constitute the primary additions.

axioms (2)

domain assumption LLMs possess ethical reasoning capabilities that can be systematically exploited in multi-turn interactions to bypass safety alignment.
Invoked as the basis for TRIAL's effectiveness in the abstract.
domain assumption Responses can be partitioned into instrumental (harm-enabling) and explanatory (non-endorsing) categories without loss of model utility.
Foundational premise for the ERR defense framework.

invented entities (2)

TRIAL methodology no independent evidence
purpose: Multi-turn red-teaming that embeds harmful requests in ethical framings.
Newly proposed attack technique.
ERR defense framework no independent evidence
purpose: Distinguishes response types using Layer-Stratified Harm-Gated LoRA.
Newly proposed defense architecture.

pith-pipeline@v0.9.0 · 5684 in / 1460 out tokens · 41859 ms · 2026-05-18T19:32:34.630351+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Foundation/AbsoluteFloorClosure.lean reality_from_one_distinction unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

TRIAL embeds adversarial goals within ethical dilemmas modeled on the trolley problem... frames the harmful action as necessary to prevent a greater catastrophe by specifically leveraging utilitarian decision-making

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

HARC: Coupling Harmfulness and Refusal Directions for Robust Safety Alignment
cs.AI 2026-07 unverdicted novelty 6.0

HARC couples harmfulness and refusal directions across prompt and response positions via subspace fine-tuning, achieving better robustness-capability-usability trade-off than six baselines while transferring across mo...