Pith Number

pith:WKYJLQMB

pith:2026:WKYJLQMBAUVRUFRACMHMKGQWP4

not attested not anchored not stored refs pending

Prompt Optimization Is a Coin Flip: Diagnosing When It Helps in Compound AI Systems

Bing Zhu, Guanghui Wang, Peiyang He, Wei Qiu, Xing Zhang, Yanwei Cui, Ziyuan Li

Prompt optimization in compound AI systems performs no better than random chance on most tasks.

arxiv:2604.14585 v2 · 2026-04-16 · cs.AI · cs.CL

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{WKYJLQMBAUVRUFRACMHMKGQWP4}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Prompt optimization in compound AI systems is statistically indistinguishable from a coin flip: across 72 optimization runs on Claude Haiku (6 methods × 4 tasks × 3 repeats), 49% score below zero-shot; ... optimization helps only when the task has exploitable output structure -- a format the model can produce but does not default to.

C2weakest assumption

That the tested tasks, models (Claude Haiku, Amazon Nova Lite), and optimization methods are representative enough for the conclusions about interaction effects (p > 0.52) and exploitable output structure to generalize to other compound AI systems.

C3one line summary

Prompt optimization in compound AI systems is statistically indistinguishable from random chance except when tasks have exploitable output structure; a two-stage diagnostic predicts success.

Receipt and verification

First computed	2026-05-28T01:04:40.145303Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

b2b095c181052b1a1620130ec51a167f28d185e94de65f8064e60be64198f54e

Aliases

arxiv: 2604.14585 · arxiv_version: 2604.14585v2 · doi: 10.48550/arxiv.2604.14585 · pith_short_12: WKYJLQMBAUVR · pith_short_16: WKYJLQMBAUVRUFRA · pith_short_8: WKYJLQMB

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/WKYJLQMBAUVRUFRACMHMKGQWP4 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: b2b095c181052b1a1620130ec51a167f28d185e94de65f8064e60be64198f54e

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "ff912bb80d5464568b224fd1c42b15b927036bbb6e95114c81bb049248c97f08",
    "cross_cats_sorted": [
      "cs.CL"
    ],
    "license": "http://creativecommons.org/licenses/by-nc-sa/4.0/",
    "primary_cat": "cs.AI",
    "submitted_at": "2026-04-16T03:23:46Z",
    "title_canon_sha256": "662f573db131e5cd7e7c7cd80bc165db3f7d6623bfbf3d3f4723bd87495016e4"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2604.14585",
    "kind": "arxiv",
    "version": 2
  }
}