Pith Number

pith:BMM5EYN7

pith:2026:BMM5EYN7NAZNSW7VKT2LCMP7T4

not attested not anchored not stored refs pending

The Topological Trouble With Transformers

Michael C. Mozer, Rosanne Liu, Shoaib Ahmed Siddiqui

Transformers push evolving state representations deeper into their layers with each new input, exhausting depth and limiting dynamic tracking.

arxiv:2604.17121 v3 · 2026-04-18 · cs.LG · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{BMM5EYN7NAZNSW7VKT2LCMP7T4}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables reflecting an evolving environment -- involves inherently sequential dependencies that feedforward networks struggle to maintain. Consequently, feedforward models push evolving state representations deeper into their layer stack with each new input step, rendering information inaccessible in shallow layers and ultimately exhausting the model's depth.

C2weakest assumption

That dynamic depth models, explicit thinking traces, and latent thinking are inherently too computationally and memory inefficient to serve as scalable solutions, and that recurrent architectures will integrate state tracking more effectively without introducing comparable costs.

C3one line summary

Transformers face a topological limitation in dynamic state tracking because their feedforward architecture pushes evolving state representations deeper into layers until depth is exhausted, requiring a shift to recurrent architectures for implicit activation dynamics.

Receipt and verification

First computed	2026-06-05T01:14:38.907997Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

0b19d261bf6832d95bf554f4b131ff9f3902f55dbceb5e559f14ceb4abf67051

Aliases

arxiv: 2604.17121 · arxiv_version: 2604.17121v3 · doi: 10.48550/arxiv.2604.17121 · pith_short_12: BMM5EYN7NAZN · pith_short_16: BMM5EYN7NAZNSW7V · pith_short_8: BMM5EYN7

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/BMM5EYN7NAZNSW7VKT2LCMP7T4 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 0b19d261bf6832d95bf554f4b131ff9f3902f55dbceb5e559f14ceb4abf67051

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "86cc7ddcbfb88ccb7429888589ecae5153cb07ffe5f9b45849786d86c5c71403",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.LG",
    "submitted_at": "2026-04-18T19:46:30Z",
    "title_canon_sha256": "dc9ab62c6a6a71e3f67e75c33518c75c8f4a3620545822a512863e335212cefc"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2604.17121",
    "kind": "arxiv",
    "version": 3
  }
}