ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Code

Kapil Madan

REVIEW 3 major objections 7 minor 50 references

ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Code

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ArGen: aligning a 1B language model to explicit governance policies via LLM-judge rewards, GRPO, and OPA-style Python penalties.

desk verdict ArGen is a transparent, reproducible integration of GRPO with LLM-judge rewards, but its central 'policy-as-code' claim is undermined by the implementation itself, where the supposed hard policy layer is just another LLM judge. read the letter →

arxiv 2509.07006 v1 pith:RAGXI7CO submitted 2025-09-06 cs.CY cs.AIcs.CLcs.LG

Kapil Madan This is my paper · ORCID

classification cs.CYcs.AIcs.CLcs.LG

keywords PolicyasCodeAIGovernanceGroupRelativeOptimisationLLM-as-a-JudgeDharmicethicsalignmentOpenAgentReinforcementLearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that AI alignment can be recast as a configurable, policy-as-code process rather than a fixed preference-tuning exercise. It claims that a 1B language model can be trained to obey explicit governance rules by combining LLM-as-judge reward scoring against stated principles, Group Relative Policy Optimisation (GRPO), and a Python policy engine inspired by Open Policy Agent that injects scope and safety penalties directly into the reward. The headline evidence is the MedGuide-AI case study, where the trained model improved its domain-scope (Dharma) score by 70.9% over the baseline (from 0.5640 to 0.9641, judged by a held-out LLM) while also improving safety and sacrificing only modest helpfulness. A sympathetic reader would care because, if the claim is right, the same machinery could turn regulations and culturally specific ethical codes into auditable, swappable training constraints for small open models in regulated settings.

What carries the argument

The load-bearing mechanism is a modular reward composition with hard-constraint penalties embedded in a GRPO training loop. The Python policy engine defines a four-tier scope classification (S0 clinical core, S1 para-clinical, S2 tangential, S3 outside medicine) with a penalty matrix that nullifies or scales the reward when a response leaves scope, plus a three-tier urgency system (A emergency, B urgent/specialist, C routine) that penalises referral-strength mismatches. These penalty factors multiply the weighted sum of LLM-evaluated principle scores (Ahimsa, Dharma, Helpfulness), and the resulting scalar reward drives policy updates through DR-GRPO with KL regularisation toward a reference model. The essential work of this composition is to make 'stay in scope' and 'refer appropriately' hard gradient signals rather than soft preferences, which is how the paper explains the large, judge-confirmed Dharma gains.

What would settle it

Evaluate the best-performing ArGen checkpoint on a fresh set of out-of-scope prompts whose vocabulary does not overlap with the medical training set, and compare the LLM-judge Dharma scores with a panel of human clinicians' ratings of scope and safety; the central claim is falsified if the ~0.96 Dharma score does not reproduce on the fresh set, or if clinicians flag responses as unsafe or out-of-scope that the judges rate as compliant.

Watch

Extended reading notes

Core claim

The paper's central claim is that governance policies can be written as interpretable code artifacts, converted into reward penalties, and internalised by a small model through GRPO, so that the model learns to resign from out-of-scope requests. Concretely, ArGen composes the total reward as $R_{\text{total}} = P_{\text{scope}} \sum_i \lambda_i R_i + P_{\text{sev}}$, where $P_{\text{scope}}$ multiplicatively zeroes the reward on hard scope violations and $P_{\text{sev}}$ adds a negative severity penalty; each $R_i$ is a principle score produced by an evaluator LLM. On the 100-scenario medical benchmark, the best ArGen checkpoint raised the average Dharma score from 0.5640 to 0.9641 (+70.9%), cut scope violations from 34% to 4%, and improved Ahimsa (safety) by 5.2%, with a 25.0% rise in the combined score; both the training evaluator (Gemini) and a held-out judge (Claude 3.5 Sonnet) confirmed the direction of these gains. The paper further argues that the same policy overlay can govern inference-time behaviour and be hot-swapped without retraining, turning compliance updates into a software-operations task rather than a retraining cycle.

Load-bearing premise

The framework assumes that the LLM judges used for training (Gemini) and for reporting results (Claude 3.5 Sonnet) score scope and safety as intended, rather than rewarding easy-to-recognize refusal phrasing; the paper offers no human expert validation of these scores.

Editorial extensions

If this is right

If the claim holds, alignment becomes a software-operations task: a compliance team can edit policy files and redeploy an overlay, and new constraints take effect without retraining the model.
The reported drop in scope violations from 34% to 4% implies that even a 1B open model can internalise hard domain boundaries when penalties are injected directly into the reward.
The ablations imply that neither component alone suffices: reward-only degrades slightly, policy-only degrades substantially on Dharma, and the hybrid achieves the best balanced performance.
Because the framework treats principles as configurable modules, the same reward-and-penalty machinery could encode other ethical traditions or regulatory rule sets beyond the Dharmic medical example.
The claimed minutes-to-compliance adaptation timeline, if correct, would give regulated deployments a way to respond to new rules without multi-day retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: a sharp test of whether the Dharma gain is genuine concept learning rather than keyword or refusal-pattern matching would be to evaluate the trained model on out-of-scope prompts whose vocabulary does not overlap with the medical training set.
Beyond the paper: the disagreement between the two judges on helpfulness (−7.4% with the held-out judge, +7.8% with the training judge) suggests that multi-objective trade-off claims need human preference data before being accepted.
Beyond the paper: because the framework itself notes that LLM judges may penalise uncertainty markers, the Ahimsa gains should be validated against clinician safety ratings on the same benchmark scenarios; otherwise part of the improvement could be stylistic.
Beyond the paper: the live hot-swap claim implies a continuous-alignment deployment loop, but the study only demonstrates training-time effects; a longitudinal deployment test measuring whether policy updates propagate to behaviour without drift would be the natural follow-up.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper is an honest, unusually transparent integration of GRPO with LLM-judge rewards and OPA-style penalties, on a genuinely interesting case study. Its central 'policy-as-code' claim is oversold: the 'hard' policy engine is itself an LLM judge, so the formal guarantee does not apply to the implementation.

What is new is the synthesis—reward functions that call Gemini, a Python 'policy' layer that feeds penalties into GRPO, and a 1B medical assistant aligned to Dharmic principles. The repo, training command, and code listings are all there. The authors used a held-out judge (Claude 3.5 Sonnet), ran reward-only and policy-only ablations, and explicitly discuss LLM-judge bias in Section 7.5. That is real credit.

But the main claim does not survive its own appendix. Listings 2–8 show that response_scope, domain_adherence_score, harm_avoidance_score, etc. are all produced by Gemini; the penalty tables then operate on those LLM outputs. So P_scope and P_sev are not deterministic, auditable checks; they are a second stochastic LLM reward. The formal theorem in B.6.1 assumes a binary P_scope enforced by code. As implemented, ArGen is GRPO with two LLM-judge channels, not policy-as-code with hard constraints. An auditor inspecting the Python would still have to audit a prompt.

The reported 70.9% Dharma improvement is plausible but selected post hoc: the Best-Performing Model is the best of 15 checkpoints. That is peak picking unless the authors report all checkpoints or a preselected rule. Helpfulness is also inconsistent across judges (-7.4% Claude, +7.8% Gemini), which undercuts the 'both judges agree' narrative.

Who should read it: people building small, regulated LLMs who want a concrete recipe and are comfortable with LLM-judge-based evaluation. It is not a formal governance contribution. Still, it deserves a real peer review—the engineering is serious and reproducible. A good referee should require revised claims about hard vs. learned constraints, full checkpoint data, and some human-expert validation before the 'verifiably compliant' language is warranted.

Referee Report

3 major / 7 minor

Summary. The paper proposes ArGen, a framework that combines LLM-as-a-judge reward scoring, GRPO, and an OPA-inspired Python policy layer to align a small language model with explicit governance policies. The central case study trains Llama-3.2-1B-Instruct as a medical assistant guided by Dharmic principles (Ahimsa, Dharma, Helpfulness) and reports a 70.9% relative improvement in Dharma scope-adherence score over the baseline, measured by Claude 3.5 Sonnet on a 100-scenario benchmark. The paper also reports an ablation study comparing reward-only and policy-only variants, a cross-evaluator validation with Gemini 2.0 Flash, adversarial red-team prompts, and an open-source implementation repository.

Significance. If the central claims held, ArGen would be a meaningful step toward configurable, policy-as-code alignment: the paper ships machine-checkable code listings, an open-source repository, a cross-evaluator analysis, and an ablation study, and it demonstrates that a 1B model can be steered by a multi-component reward signal. The Dharmic case study also addresses a genuinely underrepresented cultural perspective in AI alignment. However, the load-bearing 'hard policy' claim is contradicted by the paper's own implementation, and the headline metric rests on LLM-judge scores without human validation. These issues are substantial but, in my view, fixable through re-framing and additional analysis rather than requiring rejection.

major comments (3)

[§3.4, §3.6, Appendix B.6.1; Listings 2–8] The OPA-inspired 'hard policy' layer is not hard, and the formal guarantee does not apply to the implemented reward. In Listing 2, response_scope and domain_adherence_score are outputs of the Gemini call; Listing 4 multiplies that LLM score by a penalty factor; Listings 5–8 obtain harm_avoidance_score, safety_context_score, and referral_strength from an LLM call. Therefore P_scope and P_sev are stochastic LLM-derived quantities, not deterministic code checks. Appendix B.6.1's theorem (R_total ≤ 0 under hard violations) assumes P_scope is a binary hard flag; it does not hold for the implemented continuous LLM output. The 'policy-as-code' transparency claim is correspondingly weakened: what is auditable is an evaluation prompt, not a rule. The authors should either implement deterministic checks or consistently re-frame the governance layer as soft LLM-judged penalties and remove the hard-guarantee language.
[§5.5.4, §7.5] The headline 70.9% improvement is measured by an LLM judge on the same construct that the training reward optimizes; no human-expert validation establishes that either judge measures safety or scope. Section 7.5 concedes that LLM judges can systematically penalize uncertainty markers, the very cautious phrasing that Ahimsa training aims to produce. Cross-evaluator agreement (Table 5) rules out some same-judge circularity, but it does not rule out both judges sharing stylistic preferences. The divergence on helpfulness (Gemini +7.8%, Claude −7.4%) shows evaluator sensitivity. Please add human-validated labels or a judge-calibration analysis showing that score changes track expert-rated safety and scope rather than refusal-style lexical markers.
[§5.5.1, Table 3] Table 3 reports the best-performing of 15 evaluated checkpoints, selected post hoc by Combined Score, and the 70.9% Dharma gain is for that model. With 15 candidates, some selection inflation is expected. The median seed model used in ablations (GRPO7 Seed 3, Combined Score 0.7825) is below the best model (0.7947), so the selection rule matters. Report the full distribution of checkpoints or pre-register a selection rule with a held-out test set, and qualify the abstract and Section 5.5.2 claims accordingly.

minor comments (7)

[§5.3 vs §3.8] The reward weights are inconsistent: Section 5.3 assigns Ahimsa 0.4, Dharma 0.3, Helpfulness 0.3, while Section 3.8 states that Dharma/scope adherence receives 40% priority; reconcile the two descriptions.
[§5.4 vs Appendix A.3] Section 5.4 says the model was trained for 2 epochs, while Appendix A.3 and Figure 5 report 3 training epochs; clarify which configuration produced the reported results.
[§5.5.4] The sentence that 'both evaluators confirmed substantial improvements' overstates agreement on helpfulness, since Gemini reports +7.8% while Claude reports −7.4%; the discussion should not describe that as confirmation of the same effect.
[§7.1, Table 5] The claim of an 89.7% reduction in violation rate is inconsistent with the 34% to 4% reduction shown in Table 5, which is an 88.2% relative decrease; check the arithmetic.
[Figure 4 caption] The caption claims 'superior performance across all metrics,' but Table 3 shows helpfulness declined by 7.4% for the best-performing model; the caption should reflect the trade-off.
[Appendix B.4] The discussion of Gato-style multi-objective fine-tuning refers to DeepMind's Gato without a citation; add a reference or remove the un-cited entity.
[Abstract] The abstract contains a typo, 'Governable Al', which should read 'Governable AI'.

Circularity Check

2 steps flagged · score 6.0 of 10

The 'OPA-inspired hard policy' factor P_scope is implemented as a rescaling of the same LLM-judge score that defines the Dharma reward, so the policy-as-code contribution is not independent; a held-out judge mitigates but does not break the judge-defined evaluation construct.

self definitional [Section 3.4, Eq. (1); Appendix A.1.1, Listings 2 and 4]
"R_total(x,y)=P_scope(x,y) Σ_i λ_i R_i(x,y)+P_sev(x,y) ... P_scope(x,y) ∈ {0,1} is a multiplicative scope-compliance factor that nullifies the reward if any non-negotiable policy is violated. Listing 4: base_score = evaluation_result.get('domain_adherence_score', 0.0) ... penalty_factor = scope_penalty(prompt_scope, response_scope) ... final_score = base_score * penalty_factor"

The supposedly hard governance factor P_scope and the Dharma reward are both computed from the same Gemini evaluation call. Listing 2 obtains domain_adherence_score and response_scope from the LLM; Listing 4 sets final dharma_score = domain_adherence_score × scope_penalty(prompt_scope, response_scope). Thus the 'OPA-inspired policy engine' is not an independent deterministic rule: it is a deterministic transform of the LLM judge's own output. The formal guarantee in Appendix B.6.1 ('When P_scope = 0 ... R_total ≤ 0') applies to a binary hard flag, not to the implemented LLM-derived continuous penalty.
other [Section 5.4 and Section 7.5; Table 3/5]
"The evaluation used the same set of Evaluator LLM-based reward functions to ensure a consistent measurement standard. Additionally, to validate the robustness of our results and address potential evaluation circularity, we conducted an independent evaluation using Anthropic’s Claude 3.5 Sonnet as a held-out judge."

The training reward's Dharma signal is Gemini's domain_adherence_score (Listing 2), and the initial evaluation uses the same reward functions, so the measured improvement is relative to the same instrument being optimized. The Claude 3.5 Sonnet check is genuine mitigating evidence, but it still scores the same judge-defined construct — an LLM's opinion of scope adherence — with no human ground truth. Section 7.5 concedes that LLM judges can systematically penalize the epistemic-marker phrasing that Ahimsa training encourages.

full rationale

The strongest circularity is construction-level: the paper's Eq. (1) separates a hard P_scope policy factor from LLM-judge rewards R_i, but the implementation in Listings 2-4 derives both from one Gemini evaluation, so the 'OPA-inspired governance layer' is not an independent hard-policy channel. This undermines the central policy-as-code attribution and the formal hard-violation guarantee. The evaluation is partially self-referential: the initial benchmark uses the same LLM reward functions used for training, and although the paper responsibly adds Claude 3.5 Sonnet as a held-out judge and reports cross-evaluator agreement, both judges measure the same LLM-defined construct, and Section 7.5 admits a known bias against cautious uncertainty phrasing. The GOPAL self-citation is present but not load-bearing for the empirical results. On balance, the paper contains genuine independent effort (public code, cross-evaluator consistency, ablations), but the load-bearing 'policy-as-code' claim partially reduces by construction to LLM-as-judge scoring, so the circularity score is 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central result rests on hand-chosen reward weights, penalty tables, and thresholds, and on the assumption that LLM judges measure safety and scope faithfully. No new physical or conceptual entities are introduced; Synthetica Maximus and Synthetica Collaboratus are rhetorical framing devices rather than load-bearing components of the training system.

free parameters (5)

Reward weights lambda_i = Ahimsa 0.4, Dharma 0.3, Helpfulness 0.3
Set by hand in Section 5.3 to prioritize safety; the reported combined scores depend on this weighting.
Scope penalty matrix entries = 1.0, 0.3, 0.0, 0.5 as in Listing 1
Hand-authored penalties directly change Dharma scores and training signal; no sensitivity analysis is reported.
Violation detection thresholds = 0.5 for domain adherence and Ahimsa violation
Introduced in Listings 2 and 8; violation rates are threshold-dependent.
Tier penalty scaling = 0.5 per under-referral level, 0.3 per over-referral level
Hand-chosen in Listing 7; shapes Ahimsa scores during training and evaluation.
KL regularization schedule = beta start 0.08, end 0.04, target KL 0.6
Training hyperparameters chosen by hand; affect stability and final policy behavior.

assumptions (5)

domain assumption LLM-as-judge scores reflect the ethical quality of a response.
Used as the training reward in Section 3.2 and as the evaluation metric in Section 5.5.4; no human expert validation is provided.
domain assumption Synthetic LLM-generated medical prompts are representative of real patient queries.
Dataset built with Gemini-2.0-Flash in Section 5.1; evaluation uses a 100-scenario subset.
domain assumption Scope and urgency classifications (S0-S3, tiers A-C) are correct for the queries.
Created by LLM evaluators in the policy functions rather than by medical experts; drives penalties.
standard math Policy-gradient unbiasedness assumptions hold.
Section B.6.2 invokes on-policy sampling, bounded rewards, and gradients only with respect to theta, standard for GRPO.
domain assumption Baseline and ArGen models are evaluated on the same benchmark without training leakage.
Section 5.4 states evaluation on 100 medical scenarios, but leakage of training prompts into the benchmark is not analyzed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Code." pith.science (2026). https://pith.science/paper/RAGXI7CO

@misc{pith2026250907006,
  author       = {Pith},
  title        = {Pith review of: ArGen: Auto-Regulation of Generative AI via GRPO and Policy-as-Code},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RAGXI7CO}},
  note         = {Machine review of arXiv:2509.07006}
}

read the original abstract

This paper introduces ArGen (Auto-Regulation of Generative AI systems), a framework for aligning Large Language Models (LLMs) with complex sets of configurable, machine-readable rules spanning ethical principles, operational safety protocols, and regulatory compliance standards. Moving beyond just preference-based alignment, ArGen is designed to ensure LLMs adhere to these multifaceted policies through a novel synthesis of principle-based automated reward scoring, Group Relative Policy Optimisation (GRPO), and an Open Policy Agent (OPA) inspired governance layer. This approach provides the technical foundation for achieving and demonstrating compliance with diverse and nuanced governance requirements. To showcase the framework's capability to operationalize a deeply nuanced and culturally-specific value system, we present an in-depth case study: the development of a medical AI assistant guided by principles from Dharmic ethics (such as Ahimsa and Dharma), as derived from texts like the Bhagavad Gita. This challenging application demonstrates ArGen's adaptability, achieving a 70.9% improvement in domain-scope adherence over the baseline. Through our open-source repository, we show that ArGen's methodology offers a path to 'Governable Al' systems that are technically proficient, ethically robust, and verifiably compliant for safe deployment in diverse global contexts.

Figures

Figures reproduced from arXiv: 2509.07006 by the authors.

**Figure 1.** The ArGen Framework for Auto-Regulatory AI Alignment. This high-level conceptual schematic illustrates the core philosophy of ArGen’s auto-regulatory approach. The Policy Model (LLM) receives three major inputs in its learning environment: Configurable Policies & Principles (ethical principles, regulatory rules, operational policies), an Automated Reward System (LLM-as-a-Judge), and a Reinforcement Learning Engine … view at source ↗

**Figure 2.** The ArGen Auto-Regulatory Workflow. This detailed technical flowchart shows the complete auto-regulatory process within a single training step. Starting with a User Prompt, the Policy Model (LLM) generates a Response that undergoes parallel evaluation: the Python-based Policy Engine (fed by Policy Store functions) outputs Policy Penalties, while the Principle-Based Reward System evaluates Ahimsa (safety), Dharma (sc… view at source ↗

**Figure 3.** Post-Training Continuous-Alignment Loop. This diagram illustrates ArGen’s runtime operational flow during the post-training lifecycle, showing the integration between core inference processes and live policy governance. 3.6 OPA-Inspired Python-Based Policy Engine for Governance To complement the ’soft’ guidance from learned reward functions, ArGen incorporates a governance layer for enforcing explicit constraints. T… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Multi-Metric Performance: ArGen-aligned MedGuide-AI vs. Baseline. This radar chart provides a comprehensive visual comparison of the baseline Llama-3.2-1BInstruct model versus the ArGen-aligned MedGuide-AI across all key evaluation dimensions. The chart clearly illust…

**Figure 5.** Figure 5: Training dynamics across 3 epochs showing convergence patterns. All metrics [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]

**Figure 6.** Figure 6: Example of Improved Scope Adherence via Policy-Based Reward. "Persistent cough + phone plan" showcases strict domain adherence (Dharma). The baseline model drifts into financial advice (red ×), incurring a severe scope violation. MedGuide-AI confines itself to medical …

**Figure 7.** Figure 7: Example of Nuanced Helpfulness and Disclaimer Integration. "Dizziness + racing heart" illustrates superior safety alignment. The baseline suggests an OTC painreliever (red ×), potentially masking a cardiac emergency. MedGuide-AI prioritises urgent evaluation, explicit…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 25 canonical work pages

[1]

Shahar Avin, Alan F. T. Winfield, Luke Pearson, Michael Briers, Cristoph Timmermann, Doris Schroeder, Jonathan M. Aitken, Aleks Berditchevskaia, Nigel Shadbolt, Sabina Nadimpalli, Johanna Seibt, Jacob W. Hatfield, Thilo Herrmann, Katie Winkle, Guglielmo Tamburrini, and Virginia Dignum. Aligning AI Regulation to Sociotechnical Change . Nature Machine Intel...

work page doi:10.1038/s42256-021-00323-y 2021
[2]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022
[3]

FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines

Matthew Barker, Harini Jain, Besmira Nushi, and Ece Kamar. FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines . In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO '23), pages 1--12. Association for Computing Machinery, 10 2023. doi:10.1145/3617694.362...

arXiv 2023
[4]

Modelling moral reasoning and ethical responsibility with logic programming

Fiona Berreby, Gauvain Bourgne, and Jean-Gabriel Ganascia. Modelling moral reasoning and ethical responsibility with logic programming . In Logic for Programming, Artificial Intelligence, and Reasoning, pages 532--548. Springer, 2015. doi:10.1007/978-3-662-48899-7_37

work page doi:10.1007/978-3-662-48899-7_37 2015
[5]

When Should Algorithms Resign? A Proposal for AI Governance

Umang Bhatt and Holli Sargeant. When Should Algorithms Resign? A Proposal for AI Governance . Computer, 57 0 (10): 0 99--103, 10 2024. doi:10.1109/MC.2024.3431328. Concept of algorithmic resignation - deliberate and informed disengagement from inappropriate tasks

work page arXiv 2024
[6]

Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. Explainable Machine Learning in Deployment . In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20), pages 648--657. Association for Computing Machinery, 1 2020. doi:10....

arXiv 2020
[7]

Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melnikov, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lior Nachman, Adrian Cheng, and Kush R

Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melnikov, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lior Nachman, Adrian Cheng, and Kush R. Varshney. Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty . In Proceedings of the 2021 AAAI/ACM Conference on AI...

arXiv 2021
[8]

Superintelligence: Paths, Dangers, Strategies

Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, 2014. ISBN 978-0199678112. Includes discussion of paperclip maximizer thought experiment

work page 2014

Show all 50 references

[9]

Harms from Increasingly Agentic Algorithmic Systems

Alan Chan, Rebecca Salganik, Alva Markelius, Chris Pang, Nitarshan Rajkumar, Dmitrii Krasheninnikov, Lauro Langosco, Zhonghao He, Yawen Duan, Micah Carroll, Michelle Heidecke, Barthélémy Siegmann, Constantine Riggs, Lennart Heim, Tamay Besiroglu, Jess Rando, Adam Gleave, Buck ...

2023
[10]

Confucian Ethics and AI: Towards Harmonious Human-Machine Interaction

Wei Chen, Ming Li, and Yun Zhang. Confucian Ethics and AI: Towards Harmonious Human-Machine Interaction . AI & Society, 38 0 (2): 0 567--582, 4 2023. doi:10.1007/s00146-022-01456-7. Confucian principles applied to AI ethics and human-machine interaction

2023 doi
[11]

Christiano, Jan Leike, Tom B

Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences . arXiv preprint arXiv:1706.03741, 6 2017. Foundational work on learning from human preferences

2017 arXiv
[12]

Achieving EU AI Act Compliance by Integrating Governance as Code (GaC) and Machine Learning Operations (MLOps)

Miloš Cigoj. Achieving EU AI Act Compliance by Integrating Governance as Code (GaC) and Machine Learning Operations (MLOps) . LinkedIn Article, 5 2025. URL https://www.linkedin.com/pulse/achieving-eu-ai-act-compliance-integrating-governance-miloš-cigoj-sfikf. Discussion on pol...

2025
[13]

Collins, Ilia Sucholutsky, Umang Bhatt, Adrian Weller, Thomas L

Katherine M. Collins, Ilia Sucholutsky, Umang Bhatt, Adrian Weller, Thomas L. Griffiths, Joshua B. Tenenbaum, Nori Jacoby, Shruti Barke, Vincent Conitzer, Claudio Fanconi, Dylan Hadfield-Menell, Forough Poursabzi-Sangdeh, Iyad Rahwan, Nikhil Tandon, Max Tennison, Chen Tessler,...

2024 doi
[14]

Process Reinforcement through Implicit Rewards

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Process Reinforcement through Implicit Rewards . arXiv preprint arXiv:2502.01456, 2 2025. URL https://arxiv.org/abs/2502.01456

2025 arXiv
[15]

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI . DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning . arXiv preprint arXiv:2501.12948, 1 2025. URL https://arxiv.org/abs/2501.12948

2025 arXiv
[16]

Dennis, Michael Fisher, Marija Slavkovik, and Matt P

Louise A. Dennis, Michael Fisher, Marija Slavkovik, and Matt P. Webster. Formal verification of ethical choices in autonomous systems . Robotics and Autonomous Systems, 77: 0 1--14, 2016. doi:10.1016/j.robot.2015.11.012

2016 doi
[17]

LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

Shihan Dou, Yan Xiong, Haoxiang Zhu, Jing Zhang, Qi Wang, Shijia Qian, Jie Tang, and Zijing Huang. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods . arXiv preprint arXiv:2412.05579, 12 2024. URL https://arxiv.org/abs/2412.05579

2024 arXiv
[18]

Ubuntu and Artificial Intelligence: Towards an African Ethical Framework

Damian Okaibedi Eke, Kutoma Wakunuma, and Simisola Akintoye. Ubuntu and Artificial Intelligence: Towards an African Ethical Framework . AI & Society, 37 0 (4): 0 1685--1707, 12 2022. doi:10.1007/s00146-021-01386-8. African Ubuntu philosophy applied to AI ethics and development

2022 doi
[19]

Buddhist Ethics and AI: Compassion-Based Approaches to Artificial Intelligence

Susan Feldman. Buddhist Ethics and AI: Compassion-Based Approaches to Artificial Intelligence . AI & Society, 34 0 (2): 0 295--308, 2019. doi:10.1007/s00146-018-0846-3. Buddhist principles applied to AI ethics and alignment

2019 doi
[20]

Buen Vivir: Today's Tomorrow

Eduardo Gudynas. Buen Vivir: Today's Tomorrow . Development, 54 0 (4): 0 441--447, 12 2011. doi:10.1057/dev.2011.86. South American Buen Vivir concept for harmonious coexistence and collective well-being

2011 doi
[21]

Introduction to AI Safety, Ethics, and Society

Dan Hendrycks. Introduction to AI Safety, Ethics, and Society . Center for AI Safety, 2024. URL https://www.aisafetybook.com/. Comprehensive textbook on AI safety and alignment

2024
[22]

Towards interactive evaluations for interaction harms in human-AI systems

Lujain Ibrahim, Saffron Huang, Orestis Papakyriakopoulos, Yarin Gal, Abeba Birhane, Adrian Weller, and Umang Bhatt. Towards interactive evaluations for interaction harms in human-AI systems . arXiv preprint arXiv:2405.10632, 5 2024. Paradigm shift toward interactive evaluation...

2024 arXiv
[23]

Gordon, Caglar Gulcehre, Dongyeop Kang, Maarten Sap, Amy Zhang, and He He

Moksh Jain, Ruyuan Wan, Mikhail Terekhov, Mitchell L. Gordon, Caglar Gulcehre, Dongyeop Kang, Maarten Sap, Amy Zhang, and He He. Pluralistic Alignment @ NeurIPS 2024 Workshop . NeurIPS 2024 Workshop, 12 2024. URL https://pluralistic-alignment.github.io/. Workshop on exploring ...

2024
[24]

Efficient Uncertainty Quantification for Large Language Models

Sayash Kapoor and Arvind Narayanan. Efficient Uncertainty Quantification for Large Language Models . arXiv preprint arXiv:2404.12345, 4 2024. Methodology for training efficient uncertainty estimators for LLMs

2024 arXiv
[25]

The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment

HyunJin Kim, Xiaoyuan Yi, Jing Yao, Jianxun Lian, Muhua Huang, ShiTong Duan, Jae Yong Bak, and Xing Xie. The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment . arXiv preprint arXiv:2412.16468, 12 2024 a . URL https://arxiv.org/abs/2412.16468. Comp...

2024 arXiv
[26]

Confucian Values in AI Development: A Framework for Ethical Technology

Soo-Jin Kim, Hyun-Woo Park, and Jae-Sung Lee. Confucian Values in AI Development: A Framework for Ethical Technology . Journal of Business Ethics, 179 0 (3): 0 789--805, 8 2022. doi:10.1007/s10551-021-04987-2. Confucian ethics framework for AI development in East Asian contexts

2022 doi
[27]

EMBER: Benchmarking the Consistency of LLM Evaluators

Woosuk Kim, Jinhao Lee, Jisoo Park, and Hwaran Lee. EMBER: Benchmarking the Consistency of LLM Evaluators . arXiv preprint arXiv:2412.14733, 12 2024 b . EMBER benchmark revealing negative bias of LLM-judges towards epistemic uncertainty markers

2024 arXiv
[28]

Evaluating the Consistency of LLM Evaluators

Jinhao Lee, Woosuk Kim, Jisoo Park, and Hwaran Lee. Evaluating the Consistency of LLM Evaluators . arXiv preprint arXiv:2501.08603, 1 2025. Analysis of LLM-judge reliability and biases against epistemic markers

2025 arXiv
[29]

From Rationality to Relationality: Ubuntu as an Ethical and Human Rights Framework for Artificial Intelligence Governance

Sabelo Mhlambi. From Rationality to Relationality: Ubuntu as an Ethical and Human Rights Framework for Artificial Intelligence Governance . Carr Center Discussion Paper Series, 7 2020. URL https://carrcenter.hks.harvard.edu/publications/rationality-relationality-ubuntu-ethical...

2020
[30]

Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence

Shakir Mohamed, Marie-Therese Png, and William Isaac. Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence . Philosophy & Technology, 33 0 (4): 0 659--684, 12 2020. ISSN 2210-5433. doi:10.1007/s13347-020-00405-8

2020 doi
[31]

The Ubuntu Way: Ensuring Ethical AI Integration in Health Research

Brenda Odero, David Nderitu, and Gabrielle Samuel. The Ubuntu Way: Ensuring Ethical AI Integration in Health Research . Wellcome Open Research, 9: 0 625, 10 2024. doi:10.12688/wellcomeopenres.23021.1

2024 doi
[32]

Cognitive imperialism in artificial intelligence: counteracting bias with indigenous epistemologies

Yaw Ofosu-Asare. Cognitive imperialism in artificial intelligence: counteracting bias with indigenous epistemologies . AI & Society, 40: 0 3045--3061, 9 2024. doi:10.1007/s00146-024-02065-0

2024 doi
[33]

Open Policy Agent Documentation

Open Policy Agent Maintainers . Open Policy Agent Documentation . https://www.openpolicyagent.org/docs/latest/, 2024. Accessed: 2025-04-23

2024
[34]

Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, ...

2022
[35]

GOPAL: Governance Open Policy Agent Library for AI System Evaluations

Principled Evolution Initiative . GOPAL: Governance Open Policy Agent Library for AI System Evaluations . GitHub Repository, 2025. URL https://github.com/Principled-Evolution/gopal. Collection of OPA policies for evaluating AI systems against regulatory requirements and compli...

2025
[36]

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model . In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), pages 53728--53741. C...

2023
[37]

Stuart J. Russell. Human Compatible: Artificial Intelligence and the Problem of Control . Viking, 2019. ISBN 978-0525558613

2019
[38]

Re-imagining Algorithmic Fairness in India and Beyond

Nithya Sambasivan, Erin Arnesen, Ben Hutchinson, Tulsee Doshi, and Vinodkumar Prabhakaran. Re-imagining Algorithmic Fairness in India and Beyond . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT '21), page 317–328. Association for...

2021
[39]

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms . arXiv preprint arXiv:1707.06347, 7 2017. Proximal Policy Optimization algorithm

2017 arXiv
[40]

Group Relative Policy Optimization

Zhihan Shao, Jie Wang, Wenhao Zhu, Yuhui Wang, Hongyi Li, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. Group Relative Policy Optimization . arXiv preprint arXiv:2402.03300, 2 2024. Advanced RL algorithm for stable policy optimization in LLMs

2024 arXiv
[41]

Susiddha AI Project: Dharmic Frameworks for AI Development

Susiddha AI Project . Susiddha AI Project: Dharmic Frameworks for AI Development . Project Documentation, 2024. URL https://susiddha.ai/. Exploring Dharmic principles in AI goal systems

2024
[42]

Varshney

Kush R. Varshney. Decolonial AI Alignment: Openness, Visesa-Dharma, and Including Excluded Knowledges . In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society (AIES '23), volume 7, pages 1467--1481. Association for Computing Machinery, 8 2024. doi:10.1145/36...

2023
[43]

On the Fairness of Causal Algorithmic Recourse

Julius von Kügelgen, Amir-Hossein Karimi, Umang Bhatt, Dominik Backes, Yarin Gal, and Bernhard Schölkopf. On the Fairness of Causal Algorithmic Recourse . In Proceedings of the 36th AAAI Conference on Artificial Intelligence (AAAI-22), volume 36, pages 9584--9594. AAAI Press, ...

2022 doi
[44]

Machine Ethics: Creating an Ethical Intelligent Agent

Wendell Wallach and Colin Allen. Machine Ethics: Creating an Ethical Intelligent Agent . AI Magazine, 29 0 (4): 0 15--26, 2008. doi:10.1609/aimag.v29i4.2065

2008 doi
[45]

SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution

Yuxiang Wei et al. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution . arXiv preprint arXiv:2502.18449, 2 2025. URL https://arxiv.org/abs/2502.18449

2025 arXiv
[46]

UC-MOA: Utility-Conditioned Multi-Objective Alignment for Language Models

Rui Yang, Xiaodong Chen, Hao Wang, and Yiming Liu. UC-MOA: Utility-Conditioned Multi-Objective Alignment for Language Models . arXiv preprint arXiv:2503.10669, 3 2025. URL https://arxiv.org/abs/2503.10669

2025 arXiv
[47]

Robust Metrics for Concept Representation Evaluation

Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frédéric Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. Robust Metrics for Concept Representation Evaluation . arXi...

2023 arXiv
[48]

Adaptive Group Policy Optimization: Towards Stable Training and Enhanced Performance

Wei Zhang, Yiming Liu, Xiaodong Chen, and Hao Wang. Adaptive Group Policy Optimization: Towards Stable Training and Enhanced Performance . arXiv preprint arXiv:2503.15952, 3 2025. URL https://arxiv.org/abs/2503.15952

2025
[49]

Debate Helps Weak-to-Strong Generalization

Chujie Zheng, Pei Zhou, Zhenru Ke, and Minlie Shao. Debate Helps Weak-to-Strong Generalization . arXiv preprint arXiv:2501.13124, 1 2025 a . URL https://arxiv.org/abs/2501.13124

2025 arXiv
[50]

Understanding R1-Zero-Like Training: A Critical Perspective

Chujie Zheng, Pei Zhou, Zhenru Ke, and Minlie Shao. Understanding R1-Zero-Like Training: A Critical Perspective . arXiv preprint arXiv:2503.20783, 3 2025 b . URL https://arxiv.org/abs/2503.20783

2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Shahar Avin, Alan F. T. Winfield, Luke Pearson, Michael Briers, Cristoph Timmermann, Doris Schroeder, Jonathan M. Aitken, Aleks Berditchevskaia, Nigel Shadbolt, Sabina Nadimpalli, Johanna Seibt, Jacob W. Hatfield, Thilo Herrmann, Katie Winkle, Guglielmo Tamburrini, and Virginia Dignum. Aligning AI Regulation to Sociotechnical Change . Nature Machine Intel...

work page doi:10.1038/s42256-021-00323-y 2021

[2] [2]

Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan

Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, K...

arXiv 2022

[3] [3]

FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines

Matthew Barker, Harini Jain, Besmira Nushi, and Ece Kamar. FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines . In Proceedings of the 4th ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO '23), pages 1--12. Association for Computing Machinery, 10 2023. doi:10.1145/3617694.362...

arXiv 2023

[4] [4]

Modelling moral reasoning and ethical responsibility with logic programming

Fiona Berreby, Gauvain Bourgne, and Jean-Gabriel Ganascia. Modelling moral reasoning and ethical responsibility with logic programming . In Logic for Programming, Artificial Intelligence, and Reasoning, pages 532--548. Springer, 2015. doi:10.1007/978-3-662-48899-7_37

work page doi:10.1007/978-3-662-48899-7_37 2015

[5] [5]

When Should Algorithms Resign? A Proposal for AI Governance

Umang Bhatt and Holli Sargeant. When Should Algorithms Resign? A Proposal for AI Governance . Computer, 57 0 (10): 0 99--103, 10 2024. doi:10.1109/MC.2024.3431328. Concept of algorithmic resignation - deliberate and informed disengagement from inappropriate tasks

work page arXiv 2024

[6] [6]

Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José M. F. Moura, and Peter Eckersley. Explainable Machine Learning in Deployment . In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (FAT* '20), pages 648--657. Association for Computing Machinery, 1 2020. doi:10....

arXiv 2020

[7] [7]

Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melnikov, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lior Nachman, Adrian Cheng, and Kush R

Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melnikov, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lior Nachman, Adrian Cheng, and Kush R. Varshney. Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty . In Proceedings of the 2021 AAAI/ACM Conference on AI...

arXiv 2021

[8] [8]

Superintelligence: Paths, Dangers, Strategies

Nick Bostrom. Superintelligence: Paths, Dangers, Strategies . Oxford University Press, 2014. ISBN 978-0199678112. Includes discussion of paperclip maximizer thought experiment

work page 2014