Pith Number

pith:TFGRUFSJ

pith:2024:TFGRUFSJMKIAYD5QZ4N4SBWZ6D

not attested not anchored not stored refs resolved

KTO: Model Alignment as Prospect Theoretic Optimization

Dan Jurafsky, Douwe Kiela, Kawin Ethayarajh, Niklas Muennighoff, Winnie Xu

KTO aligns LLMs by maximizing prospect-theoretic utility from binary desirability signals rather than paired preferences.

arxiv:2402.01306 v4 · 2024-02-02 · cs.LG · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{TFGRUFSJMKIAYD5QZ4N4SBWZ6D}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do. We call this approach KTO, and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B, despite only learning from a binary signal of whether an output is desirable.

C2weakest assumption

That the specific utility function taken from prospect theory literature accurately captures human judgments of LLM outputs and that optimizing it with only binary desirability labels is sufficient without additional modeling assumptions or reference-point choices.

C3one line summary

KTO aligns LLMs by directly maximizing prospect-theoretic utility on binary signals and matches or exceeds preference-based methods like DPO from 1B to 30B parameters.

References

31 extracted · 31 resolved · 16 Pith anchors

[1] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback · arXiv:2204.05862

[2] Human irrationality: both bad and good for reward inference

[3] Evaluating Large Language Models Trained on Code · arXiv:2107.03374

[4] Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models · arXiv:2401.01335

[5] Training Verifiers to Solve Math Word Problems · arXiv:2110.14168

Formal links

2 machine-checked theorem links

Cited by

109 papers in Pith

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

Weak-to-Strong Generalization via Direct On-Policy Distillation

Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment

Towards Spec Learning: Inference-Time Alignment from Preference Pairs

The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

Receipt and verification

First computed	2026-07-05T09:37:35.138392Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

994d1a164962900c0fb0cf1bc906d9f0ec34fe3690ca2946dae217505bdd947c

Aliases

arxiv: 2402.01306 · arxiv_version: 2402.01306v4 · doi: 10.48550/arxiv.2402.01306 · pith_short_12: TFGRUFSJMKIA · pith_short_16: TFGRUFSJMKIAYD5Q · pith_short_8: TFGRUFSJ

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/TFGRUFSJMKIAYD5QZ4N4SBWZ6D \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 994d1a164962900c0fb0cf1bc906d9f0ec34fe3690ca2946dae217505bdd947c

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "94efdd36e188101bee0587c220581f370ce6a2771feaf9014f32fa1ee0a992f8",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://creativecommons.org/licenses/by-sa/4.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2024-02-02T10:53:36Z",
    "title_canon_sha256": "8a08c1d7a8bd0d9b6fcb32559ffffe8c804c7d7c9713ba1fe5f0aa40351edd9e"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2402.01306",
    "kind": "arxiv",
    "version": 4
  }
}