Pith. sign in

REVIEW 4 major objections 6 minor 63 references

Deontological Keyword Bias: The Impact of Modal Expressions on Normative Judgments of Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that adding an obligation modal such as 'must' or 'ought to' to a prompt makes large language models label ordinary commonsense sentences as obligations.

desk verdict A solid empirical mapping of modal-expression effects on LLM obligation judgments, undermined by an overstated abstract and a human-model comparison that measures two different things. read the letter →

arxiv 2506.11068 v1 pith:25IYBMME submitted 2025-06-01 cs.CL

Bumjin Park , Jinsil Lee , Jaesik Choi This is my paper
classification cs.CL
keywords deonticmodalitymodalexpressionsobligationjudgmentlanguagemodelbiasnormativealignmentfew-shotlearningin-contextreasoningcommonsense
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that large language models make deontic judgments primarily from surface modal keywords rather than from contextual meaning. Across seven model families, inserting an obligation modal ('must', 'ought to', 'should', 'have to') into a commonsense sentence raises the proportion of 'this is deontic' judgments to high levels—for most models over 90%—even though the sentences describe ordinary, non-obligatory actions. Humans also show a smaller version of this modal effect, but the human–model gap is large: on commonsense items with a modal, GPT-4o scores 4.90 on a 0–5 obligation scale versus 3.33 for humans. The authors define this over-reliance as Deontological Keyword Bias, distinguish it from the shared Deontological Keyword Effect, and show that a few labeled examples combined with an explicit reasoning prompt reduces the bias without retraining. If correct, the finding matters because LLMs are increasingly used for normative judgments and agent decisions, where mistaking advice or preference for obligation could distort behavior.

What carries the argument

The key machinery is the pair of definitions Deontological Keyword Effect (DKE) and Deontological Keyword Bias (DKB), tied to valuation functions $f_{\mathrm{binary}}: L \to \{0,1\}$ and $f_{\mathrm{continuous}}: L \to [0,1]$ that score generated responses. The paper samples outputs with and without modal augmentation, $Y_{\text{with ME}} \sim P_\theta(Y \mid S, Z, Q)$ versus $Y_{\text{without ME}} \sim P_\theta(Y \mid S, \varnothing, Q)$, and calls the gap DKE; DKB is the special case where the semantic framing $S$ is non-obligatory and the modal still raises the score. The debiasing machinery is in-context reasoning: few-shot demonstrations labeled by semantic obligation rather than by keyword presence, combined with an explicit instruction to reason, which the paper shows lowers positive judgments in commonsense contexts.

What would settle it

Have the same human raters answer the paper's exact binary LLM question ('Is this a deontic sentence? yes/no') on the exact commonsense sentences with modals; if human 'yes' rates also climb toward the models' high rates, the human-model misalignment is a question-format artifact, while if they stay well below, DKB is confirmed as a model-side effect. Also record whether any modal commonsense item is rated 1 ('no obligation') by a majority of humans, since the 'non-obligatory' label requires the modal version to remain at the floor.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that LLM obligation judgments are anchored to the presence of modal expressions of obligation rather than to the semantics of the situation. Defining DKB formally as the case where the semantic framing $S$ carries no obligation yet $f(Y_{\text{with ME}}) > f(Y_{\text{without ME}})$ for the model's valuation functions, the paper reports rates above 90% for most models on commonsense sentences with modals, consistent across question phrasings and binary/score answer formats, and notes that even negated modals ('must not') trigger the bias. The paper also reports that a training-free debiasing strategy—few-shot examples labeled by semantics plus an instruction to reason—reduces obligation over-generation on the commonsense set, with GPT-4o falling from 0.62 to 0.08 positive rate in the 2-positive/2-negative reasoning condition.

Load-bearing premise

The load-bearing assumption is that the binary LLM question ('Is this a deontic sentence?') and the human 5-point question ('Does the context imply an obligation?') measure the same construct, and that commonsense sentences with a modal inserted remain genuinely non-obligatory—a premise the paper's own human data (3.33 out of 5 with the modal) strain; if either assumption fails, the claimed human-model misalignment weakens.

Editorial extensions

If this is right

  • Adding 'must', 'ought to', 'should', or 'have to' to a commonsense sentence raises LLM positive deontic judgments from below about 20% to high levels in the authors' binary task, with most models above 90%.
  • The bias persists across general, explicit, and strict question phrasings and across binary and 0–100 continuous answer formats, and negated modals such as 'must not' also trigger it.
  • Replacing modals with weak directive expressions such as 'recommend' or 'suggest' drops obligation judgments sharply even in genuinely deontic contexts, indicating the model keys on the modal form itself.
  • In the authors' obligation-conflict reasoning scenarios, the presence or absence of modals makes little consistent difference, so the keyword effect appears in direct deontic classification more than in multi-step conflict reasoning.
  • Two few-shot demonstrations plus an explicit reasoning instruction lower commonsense positive-judgment rates (GPT-4o from 0.62 to 0.08), while zero-shot chain-of-thought prompts do not reliably help.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the paper's own human ratings (3.33 out of 5 for modal commonsense items) suggest those items are not cleanly non-obligatory, so a stronger DKB test would use sentences that remain near the floor even after a modal is inserted; if no such sentences exist, the bias claim needs reframing.
  • Inference: the human-model gap may be partly a question-format artifact, since humans answered a 5-point 'imply an obligation' question while models answered a binary 'is this deontic' question; rerunning the models on the human wording and scale would settle this cheaply.
  • Inference: because negated modals produce the bias, the operative cue may be any salient deontic vocabulary rather than directive force; probing with permission modals ('may', 'can') and modal-free normative nouns ('duty', 'obligation') would map the actual trigger set.
  • Inference: the success of few-shot examples plus reasoning and the failure of zero-shot CoT is consistent with a label-prior effect in the binary task, which would predict that telling models the base rate of obligations, or rebalancing positive and negative demonstrations, should further reduce the bias.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces and investigates "Deontological Keyword Bias" (DKB): the tendency of large language models (LLMs) to judge sentences as conveying obligations when they contain modal expressions such as 'must,' 'should,' 'ought to,' and 'have to,' even in contexts that the authors regard as non-obligatory. The authors define DKE (the general effect of modal expressions on obligation judgments) and DKB (the allegedly erroneous over-application of this effect), and they report experiments across several LLM families, datasets (deontology, commonsense, morality high/low ambiguity), question types (general, explicit, strict), and answer formats (binary, continuous). They further propose a training-free debiasing method that combines few-shot examples with in-context reasoning, and report that this method reduces the bias. The headline claim is that LLMs judge over 90% of commonsense scenarios as obligations when modal expressions are present and that this tendency is consistent across families, question types, and formats.

Significance. If the central claims are sustained, the paper would document a practically important and under-studied type of linguistic framing bias in LLM normative judgments, with direct implications for alignment, safe deployment of LLM agents, and the design of debiasing prompts. The paper has clear strengths: it evaluates a broad set of models (GPT-4o, GPT-4o-mini, Llama-3.1-70B/8B, Gemma-9B, Qwen-7B, Exaone-8B), covers multiple datasets and prompt variants, includes a human evaluation, and tests a debiasing intervention. The few-shot plus reasoning mitigation is a useful empirical contribution, and the paper is generally transparent about its limitations (Section 8). However, the paper's headline quantitative claims and the construct validity of the measurement are problematic, as detailed below.

major comments (4)
  1. [Abstract; Section 5.1, Table 2] The abstract claims that "LLMs judge over 90% of commonsense scenarios as obligations when modal expressions are present." Table 2 reports Commonsense With ME proportions of 0.98 (GPT-4o), 0.96 (GPT-4o-mini), 0.86 (Llama-3.1-70B), 0.54 (Llama-3.1-8B), 0.89 (Gemma-9B), and 0.88 (Qwen-7B). Only two of the six models exceed 90%, and Llama-3.1-8B is at 54%, which is close to chance. The abstract's "over 90%" claim is therefore not supported by the paper's own results. The claim should be restricted to the models that actually achieve that level, or replaced by an aggregate statistic with variance information.
  2. [Section 3.1, Section 4, Appendix D] There is a construct mismatch between the LLM evaluation and the human evaluation. The LLM prompt (Section 4) asks "Determine whether the following sentence is deontic... Answer with 1 if it is a deontic sentence, 0 if not," while the human evaluation (Appendix D) asks "Does the context of the sentence imply an obligation?" on a 5-point Likert scale. These are different constructs: a sentence that literally contains "must" or "ought to" is, by standard linguistic criteria, a deontic modal sentence, so an LLM answering 1 may be correctly identifying grammatical modality rather than erroneously asserting a normative obligation. The DKB definition in Section 3.1 requires that the base context "lacks obligation-related semantics," but this condition is never validated; in fact, the human ratings for commonsense sentences with modals are 3.33 on a 0-to-5 obligation scale (Appendix D), near the midpoint, so those items are not clearly non-obligatory. The claimed human-model misalignment (Figure 2, Table 1) may therefore be an artifact of comparing grammatical deontic classification with normative obligation judgment. Please either use the same question for humans and LLMs, or provide independent evidence that the base commonsense contexts are non-obligatory.
  3. [Section 5.2, Figure 5a] The claim that DKB is consistent across question types is contradicted by the paper's own results. Under the Strict question prompt ("mandates compliance in all cases"), the text states that GPT-4o, GPT-4o-mini, and ExaOne-8B show near-zero scores on the commonsense dataset even when modal expressions are present. Thus the bias does not survive a stricter normative criterion for several models. The conclusion of Section 5.2 ("DKB remains consistent across different question levels") should be revised to describe the dependence on prompt strictness rather than an across-the-board consistency.
  4. [Definition 1; Tables 2 and 3; Section 5.1] Definition 1 defines DKE/DKB as holding "consistently or statistically across instances," but the paper reports no significance tests, confidence intervals, or per-item variance measures for the headline proportions. The results in Table 2 show large model-to-model variation (e.g., commonsense with ME ranges from 0.54 to 0.98), and Table 1 reports variance only for a different (Likert) measure for a single model. Without paired per-item tests or bootstrap intervals, the claim that the effect is systematic and robust across instances is not statistically substantiated, even though the qualitative pattern is visible in most models.
minor comments (6)
  1. [Abstract and throughout] The word "consist" should be "consistent" in the abstract and in several other places.
  2. [Table 1 caption and Appendix D] The human evaluation description in Appendix D should state the total number of ratings per item (100 participants, 10 items each) and whether any participants were excluded; Table 1 reports variance but not N, making it hard to assess the reliability of the human means.
  3. [Section 4 prompt] The prompt displayed as "Sentence: Context: {ctx} Input: {input}" appears malformed; the placeholders and formatting should be corrected so that the exact prompt is reproducible.
  4. [Table 4] The column labeled "D" is not defined in the caption or text; presumably it stands for dataset but this should be stated explicitly.
  5. [Appendix F.2, Table 16] The expression-substitution results show that with weak expressions (WEs) in the Deontology dataset, GPT-4o's obligation judgment drops to 0.00. This is worth discussing: it suggests the model may fail to recognize deontic semantics at all without modal keywords, which is a different (and arguably equally important) failure mode that the current framing of DKB does not cover.
  6. [References] Several citations are incomplete or contain typos (e.g., "V on Wright" in the text and references, "Nous" in a reference, the Alpaca dataset reference lacks a year-page). Please copyedit the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the central DKB claim is an empirical measurement against external benchmark labels and a human evaluation, not a fit or self-citation chain.

full rationale

The paper's central claim is defined and tested against external benchmarks and human judgments, not against parameters fitted by the paper. Definition 1 operationalizes DKE and DKB as a difference in model outputs, f(Y_with_ME) > f(Y_without_ME), and the experiments measure this difference on the Hendrycks et al. deontology/commonsense datasets, the Scherrer et al. morality datasets, and a new high-order reasoning dataset. The commonsense dataset's negative label is inherited from prior work, and the human evaluation in Appendix D independently rates the same sentence pairs, so the 'non-obligatory' status of the base contexts is not manufactured by the paper's own equations. There are no self-citations at all: none of the references are by the present authors, and no uniqueness theorem or prior result by the authors is invoked to force the conclusion. The debiasing section uses few-shot examples labeled by the authors according to deontic semantics, but that is a standard in-context-learning intervention rather than a fitted parameter renamed as a prediction. The skeptical concern that the LLM prompt 'Determine whether the following sentence is deontic' may not measure the same construct as the human question 'Does the context of the sentence imply an obligation?' is a validity or correctness objection, not a circularity objection: the model's high positive rate is not entailed by the paper's definitions or by a self-citation chain, and the paper does not derive its headline number from its own assumptions. Therefore no circular step is identified, and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central measurement rests on the benchmark's ground-truth labeling, the comparability of human and LLM tasks, the choice of trigger words, and the author-assigned few-shot labels. These are domain assumptions, not fitted parameters.

free parameters (2)
  • Obligation-modal expression set = must, ought to, should, have to
    Hand-chosen triggers; the measured DKB rate and the cross-expression comparison depend directly on which modals count as obligation markers. May, can, and would are excluded.
  • Few-shot example counts (Npos, Nneg) = Npos,Nneg in {0,1,2}
    The debiasing result is a function of how many positive and negative examples are shown; the paper reports the best-performing 2+2 setting but the drop in obligation rate is not monotonic for all models.
assumptions (4)
  • domain assumption Sentences in the Commonsense dataset with inserted MEs remain non-obligatory in ground truth, so positive LLM judgments are errors.
    This is the core labeling assumption behind DKB. It is supported by human ratings (Commonsense with ME mean 3.33 on 0-5), but these ratings are on a different scale than the LLM binary task, and the human mean is above the midpoint, so the non-obligatory status is not unambiguous.
  • domain assumption LLM binary deontic classification and human 5-point obligation rating measure the same construct and are comparable.
    The paper directly compares GPT-4o scores (0-5) with human scores in Table 1, despite the LLM being prompted for a 0/1 decision in the main experiments; the mapping between the two scales is not established.
  • domain assumption The four modal expressions (must, ought to, should, have to) are the operative triggers of the effect, and weak expressions are neutral controls.
    The choice of triggers sets the scope of DKB; the paper justifies exclusion of may/can/would but does not test per-trigger ground truth.
  • domain assumption The few-shot example labels assigned by the authors are correct deontic labels and effective teaching signals.
    The debiasing evaluation measures how much LLM judgments move toward the authors' labels; if those labels are wrong, the mitigation result is not meaningful.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deontological Keyword Bias: The Impact of Modal Expressions on Normative Judgments of Language Models." pith.science (2026). https://pith.science/paper/25IYBMME

@misc{pith2026250611068,
  author       = {Pith},
  title        = {Pith review of: Deontological Keyword Bias: The Impact of Modal Expressions on Normative Judgments of Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/25IYBMME}},
  note         = {Machine review of arXiv:2506.11068}
}
read the original abstract

Large language models (LLMs) are increasingly engaging in moral and ethical reasoning, where criteria for judgment are often unclear, even for humans. While LLM alignment studies cover many areas, one important yet underexplored area is how LLMs make judgments about obligations. This work reveals a strong tendency in LLMs to judge non-obligatory contexts as obligations when prompts are augmented with modal expressions such as must or ought to. We introduce this phenomenon as Deontological Keyword Bias (DKB). We find that LLMs judge over 90\% of commonsense scenarios as obligations when modal expressions are present. This tendency is consist across various LLM families, question types, and answer formats. To mitigate DKB, we propose a judgment strategy that integrates few-shot examples with reasoning prompts. This study sheds light on how modal expressions, as a form of linguistic framing, influence the normative decisions of LLMs and underscores the importance of addressing such biases to ensure judgment alignment.

Figures

Figures reproduced from arXiv: 2506.11068 by the authors.

Figure 1
Figure 1. Graphical illustration of the Deontological [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. This figure illustrates the results of Table [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Effect of modal expressions of obligation on the proportion of positive judgments for the deontology and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: The proportion of positive predictions for [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Consistency of the Deontological Keyword [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Conflict Scenario Reasoning example. GPT-4o GPT-4o-mini LLaMA3.1-8B Gemma2-9B ExaOne-8B Qwen2-7B 0 10 20 30 40 50 60 70 80 90 100 Obligation Judgment Rate (%) Keyword-Free MEs of Obligation (e.g., "Must") [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Proportion of positive predictions in Conflict [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: While DKB remains relatively stable in the Deontology dataset, it tends to decrease with longer prompts [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 44 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Thomas gotnes, Wiebe Van Der Hoek, Juan A Rodr \' guez-Aguilar, Carles Sierra, and Michael Wooldridge. 2009. A temporal logic of normative systems. In Towards Mathematical Philosophy: Papers from the Studia Logica conference Trends in Logic IV

    work page 2009

  4. [4]

    Emmanuel Ameisen, Jack Lindsey, Adam Pearce, Wes Gurnee, Nicholas L. Turner, Brian Chen, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivo...

    work page 2025

  5. [5]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  6. [6]

    Albert Bandura. 1969. Social-learning theory of identificatory processes. Handbook of socialization theory and research

    work page 1969

  7. [7]

    Guido Boella, Leendert Van Der Torre, and Harko Verhagen. 2006. Introduction to normative multiagent systems. Computational & Mathematical Organization Theory

    work page 2006

  8. [8]

    Adam Bouyamourn. 2023. Why LLM s hallucinate, and how to get (evidential) closure: Perceptual, intensional, and extensional learning for faithful natural language generation. In Proceedings of Empirical Methods in Natural Language Processing

    work page 2023

Show all 63 references
  1. [9]

    Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen,...

    2023

  2. [10]

    Ilias Chalkidis, Ion Androutsopoulos, and Achilleas Michos. 2018. Obligation and prohibition extraction using hierarchical rnns. In Proceedings of the Association for Computational Linguistics

    2018

  3. [11]

    Ruizhe Chen, Yichen Li, Jianfei Yang, Yang Feng, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu. 2025. Identifying and mitigating social bias knowledge in language models. In Findings of the Association for Computational Linguistics:NAACL

    2025

  4. [12]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research

    2024

  5. [13]

    Jennifer Coates. 1983. The semantics of the modal auxiliaries. English Language and Linguistics Studies

    1983

  6. [14]

    John M Doris. 1998. Persons, situations, and virtue ethics. Nous

    1998

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Xiao Fang, Shangkun Che, Minjia Mao, Hongzhe Zhang, Ming Zhao, and Xiaohang Zhao. 2024. Bias of ai-generated content: an examination of news produced by large language models. Scientific Reports

    2024

  9. [17]

    Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. 2024. Bias and fairness in large language models: A survey. Computational Linguistics

    2024

  10. [18]

    Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ricardo Mu \ n oz S \'a nchez, Mugdha Pandya, and Adam Lopez. 2020. Intrinsic bias metrics do not correlate with application bias. arXiv preprint arXiv:2012.15859

    2020 arXiv

  11. [19]

    Kurt Gray, Liane Young, and Adam Waytz. 2012. Mind perception is the essence of morality. Psychological inquiry

    2012

  12. [20]

    Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embeddings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society

    2021

  13. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations

    2021

  14. [22]

    Jaakko Hintikka. 1971. Some main problems of deontic logic. In Deontic logic: Introductory and systematic readings

    1971

  15. [23]

    Wesley H Holliday, Matthew Mandelkern, and Cedegao E Zhang. 2024. Conditional and modal reasoning in large language models. arXiv preprint arXiv:2401.17169

    2024 arXiv

  16. [24]

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Proceedings of International Conference on Machine Learning

    2019

  17. [25]

    Rodney Huddleston and Geoffrey K Pullum. 2005. The cambridge grammar of the english language. Zeitschrift für Anglistik und Amerikanistik

    2005

  18. [26]

    Ken Hyland. 2005. Metadiscourse: Exploring interaction in writing. Journal of Academic Writing and Discourse Studies

    2005

  19. [27]

    Vivek Joshi, Preethu Rose Anish, and Smita Ghaisas. 2021. Domain adaptation for an automated classification of deontic modalities in software engineering contracts. In In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foun...

    2021

  20. [28]

    Daniel Kasenberg and Matthias Scheutz. 2018. Norm conflict resolution in stochastic domains. In Proceedings of the AAAI Conference on Artificial Intelligence

    2018

  21. [29]

    Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. 2022. Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937

    2022 arXiv

  22. [30]

    Deepak Kumar, Oleg Lesota, George Zerveas, Daniel Cohen, Carsten Eickhoff, Markus Schedl, and Navid Rekabsaz. 2023. Parameter-efficient modularised bias mitigation via adapterfusion. arXiv preprint arXiv:2302.06321

    2023 arXiv

  23. [31]

    Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang, Dan Jurafsky, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. When do pre-training biases propagate to downstream tasks? a case study in text summarization. In Proceedings of the European Chapter of the Association for...

    2023

  24. [32]

    Lvxue Li, Jiaqi Chen, Xinyu Lu, Yaojie Lu, Hongyu Lin, Shuheng Zhou, Huijia Zhu, Weiqiang Wang, Zhongyi Liu, Xianpei Han, et al. 2024. Debiasing in-context learning by instructing llms how to follow demonstrations. In Findings of the Association for Computational Linguistics:ACL

    2024

  25. [33]

    Nikos Nikiforakis, Charles N Noussair, and Tom Wilkening. 2012. Normative conflict and feuds: The limits of self-enforcement. Journal of Public Economics

    2012

  26. [34]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, and Others. 2024. Gpt-4 technical report

    2024

  27. [35]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems

    2022

  28. [36]

    Frank Robert Palmer. 2001. Mood and modality. Cambridge University

    2001

  29. [37]

    Herbert James Paton. 1971. The categorical imperative: A study in Kant's moral philosophy

    1971

  30. [38]

    Fabio Petroni, Tim Rockt \"a schel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? arXiv preprint arXiv:1909.01066

    2019 arXiv

  31. [39]

    Gabriel Poesia, Kanishk Gandhi, Eric Zelikman, and Noah Goodman. 2024. Certified deductive reasoning with language models. Transactions on Machine Learning Research

    2024

  32. [40]

    Leonardo Ranaldi, Elena Ruzzetti, Davide Venditti, Dario Onorati, and Fabio Massimo Zanzotto. 2024. A trip towards fairness: Bias and de-biasing in large language models. In Proceedings of Lexical and Computational Semantics

    2024

  33. [41]

    Abhinav Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. 2023. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in llms. arXiv preprint arXiv:2310.07251

    2023 arXiv

  34. [42]

    LG Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, et al. 2024. Exaone 3.0 7.8 b instruction tuned language model. arXiv preprint arXiv:2408.03541

    2024

  35. [43]

    Alf Ross. 1944. Imperatives and logic. Philosophy of Science

    1944

  36. [44]

    Pratik S Sachdeva and Tom van Nuenen. 2025. Normative evaluation of large language models with everyday moral dilemmas. arXiv preprint arXiv:2501.18081

    2025 arXiv

  37. [45]

    Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. 2023. Evaluating the moral beliefs encoded in llms. Advances in Neural Information Processing Systems

    2023

  38. [46]

    Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, et al. 2025. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming. arXiv preprint arXiv:2501.18837

    2025 arXiv

  39. [47]

    Chenglei Si, Zhe Gan, Zhengyuan Yang, Shuohang Wang, Jianfeng Wang, Jordan Boyd-Graber, and Lijuan Wang. 2023. Prompting gpt-3 to be reliable. In Proceedings of the International Conference on Learning Representations

    2023

  40. [48]

    Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203

    2019 arXiv

  41. [49]

    Jingyun Sun, Shaobin Huang, and Chi Wei. 2023. A bert-based deontic logic learner. Information Processing & Management

    2023

  42. [50]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  43. [51]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  44. [52]

    Wamberto W Vasconcelos, Martin J Kollingbaum, and Timothy J Norman. 2009. Normative conflict resolution in multi-agent systems. Autonomous agents and multi-agent systems

    2009

  45. [53]

    Jean-Christophe Verstraete. 2005. Scalar quantity implicatures and the interpretation of modality: Problems in the deontic domain. Journal of pragmatics

    2005

  46. [54]

    Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems

    2020

  47. [55]

    Kai von Fintel. 2012. The best we can (expect to) get? challenges to the classic semantics for deontic modals. In Central Meeting of the American Philosophical Association

    2012

  48. [56]

    Georg Henrik Von Wright. 1951. Deontic logic. Mind

    1951

  49. [57]

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652

    2021 arXiv

  50. [58]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  51. [59]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  52. [60]

    Kai-Ching Yeh, Jou-An Chi, Da-Chen Lian, and Shu-Kai Hsieh. 2023. Evaluating interfaced llm bias. In Proceedings of Computational Linguistics and Speech Processing

    2023

  53. [61]

    Yudi Zhang, Pei Xiao, Lu Wang, Chaoyun Zhang, Meng Fang, Yali Du, Yevgeniy Puzyrev, Randolph Yao, Si Qin, Qingwei Lin, Mykola Pechenizkiy, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. 2025. Ruag: Learned-rule-augmented generation for large language models. In Proceedings of ...

    2025

  54. [62]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning

    2021

  55. [63]

    Jingyan Zhou, Minda Hu, Junan Li, Xiaoying Zhang, Xixin Wu, Irwin King, and Helen Meng. 2023. Rethinking machine ethics--can llms perform moral reasoning through the lens of moral theories? arXiv preprint arXiv:2308.15399

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.