Pith Number

pith:O5FGSQF5

pith:2023:O5FGSQF52JYQJG6M4EJUGWR4RU

not attested not anchored not stored refs resolved

The Linear Representation Hypothesis and the Geometry of Large Language Models

Kiho Park, Victor Veitch, Yo Joong Choe

High-level concepts in large language models are linear directions under a causal inner product built from counterfactual pairs.

arxiv:2311.03658 v2 · 2023-11-07 · cs.CL · cs.AI · cs.LG · stat.ML

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{O5FGSQF52JYQJG6M4EJUGWR4RU}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Using this causal inner product, we show how to unify all notions of linear representation. In particular, this allows the construction of probes and steering vectors using counterfactual pairs.

C2weakest assumption

The assumption that the identified non-Euclidean inner product respects language structure in the precise sense required to unify probing and steering, and that counterfactual pairs can be reliably constructed or approximated in the model.

C3one line summary

Linear representations of high-level concepts in LLMs are formalized via counterfactuals in input and output spaces, unified under a causal inner product that enables consistent probing and steering.

References

30 extracted · 30 resolved · 8 Pith anchors

[1] doi: 10.18653/v1/K16-1002 2022 · doi:10.18653/v1/k16-1002

[2] Word embed- dings, analogies, and machine learning: Beyond king - man + woman = queen 2016

[3] Toy Models of Superposition · arXiv:2209.10652

[4] How contextual are contextualized word rep- resentations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings 2019

[5] doi: 10.18653/v1/2020.conll-1.29 2020 · doi:10.18653/v1/2020.conll-1.29

Cited by

45 papers in Pith

Steered Generation via Gradient-Based Optimization on Sparse Query Features

Is Dimensionality a Barrier for Retrieval Models?

Manifold-Guided Attention Steering

Relational Linear Properties in Language Models: An Empirical Investigation

Under Pressure: Emotional Framing Induces Measurable Behavioral Shifts and Structured Internal Geometry in Small Language Models

Receipt and verification

First computed	2026-05-20T00:00:14.503329Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

774a6940bdd271049bcce113435a3c8d3c31947ca6c396b22ef91cc32f9ea2f9

Aliases

arxiv: 2311.03658 · arxiv_version: 2311.03658v2 · doi: 10.48550/arxiv.2311.03658 · pith_short_12: O5FGSQF52JYQ · pith_short_16: O5FGSQF52JYQJG6M · pith_short_8: O5FGSQF5

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/O5FGSQF52JYQJG6M4EJUGWR4RU \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 774a6940bdd271049bcce113435a3c8d3c31947ca6c396b22ef91cc32f9ea2f9

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "52fc5acc1032b265edd952df1098bf1b816a082f7237490d8038ab7862d67fac",
    "cross_cats_sorted": [
      "cs.AI",
      "cs.LG",
      "stat.ML"
    ],
    "license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2023-11-07T01:59:11Z",
    "title_canon_sha256": "4f38a4423afec1c2192b83aca612444daf27dedf0b6ae368025085f67b69be7f"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2311.03658",
    "kind": "arxiv",
    "version": 2
  }
}