Pith Number

pith:DJHSBN54

pith:2022:DJHSBN54HU2VV2B7SIQRIVL5FE

not attested not anchored not stored refs pending

Measuring and Narrowing the Compositionality Gap in Language Models

Ludwig Schmidt, Mike Lewis, Muru Zhang, Noah A. Smith, Ofir Press, Sewon Min

Larger language models improve single-fact recall faster than they improve the ability to compose multiple facts into answers.

arxiv:2210.03350 v3 · 2022-10-07 · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{DJHSBN54HU2VV2B7SIQRIVL5FE}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

In the GPT-3 family of models, as model size increases we show that the single-hop question answering performance improves faster than the multi-hop performance does, therefore the compositionality gap does not decrease.

C2weakest assumption

That the multi-hop questions are built from facts unlikely to have been observed together during pretraining, so that correct answers to the full question must come from composition rather than direct memorization of the combined fact.

C3one line summary

Larger language models improve faster at single facts than at composing them, but self-ask prompting reduces the compositionality gap by forcing explicit intermediate questions.

Formal links

2 machine-checked theorem links

Cited by

31 papers in Pith

EVE-Agent: Evidence-Verifiable Self-Evolving Agents

Holistic Evaluation of Language Models

ZeroSearch: Incentivize the Search Capability of LLMs without Searching

Artificial Phantasia: Emergent Mental Imagery in Large Language Models

EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle

Receipt and verification

First computed	2026-05-17T23:38:13.479942Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

1a4f20b7bc3d355ae83f922114557d291eb3fffea673138ac709f19448d57925

Aliases

arxiv: 2210.03350 · arxiv_version: 2210.03350v3 · doi: 10.48550/arxiv.2210.03350 · pith_short_12: DJHSBN54HU2V · pith_short_16: DJHSBN54HU2VV2B7 · pith_short_8: DJHSBN54

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/DJHSBN54HU2VV2B7SIQRIVL5FE \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 1a4f20b7bc3d355ae83f922114557d291eb3fffea673138ac709f19448d57925

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "b6f926ce0d2d9a6fb0798b7b877f894c9f7ab05f8cabc5a4854025fc477756e1",
    "cross_cats_sorted": [],
    "license": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
    "primary_cat": "cs.CL",
    "submitted_at": "2022-10-07T06:50:23Z",
    "title_canon_sha256": "b51c2e7d513e3bc51f957d92064bdf2b30f25b753209fbaa8ff4a6266f98bb0d"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2210.03350",
    "kind": "arxiv",
    "version": 3
  }
}