Pith Number

pith:UFR2BXRU

pith:2026:UFR2BXRU5E456SURIPLSQFXGUX

not attested not anchored not stored refs pending

Measuring LLM Trust Allocation Across Conflicting Software Artifacts

Ahsanul Ameen Sabit, Noshin Ulfat, Soneya Binta Hossain

LLMs penalize documentation bugs more than implementation faults when artifacts conflict, but overlook code drift when docs remain plausible.

arxiv:2604.03447 v2 · 2026-04-03 · cs.SE · cs.AI

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{UFR2BXRU5E456SURIPLSQFXGUX}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

Across all models, quality penalties are largely localized to the perturbed artifact and increase with severity, but sensitivity is asymmetric across artifact types: documentation bugs induce a substantially larger heavy-to-subtle gap than implementation faults (0.152-0.253 vs. 0.049-0.123). Models detect explicit documentation bugs well (67-94%) and Javadoc and implementation contradictions at 50-91%, yet show a systematic blind spot when only the implementation drifts while the documentation remains plausible, with detection dropping by 7-42 percentage points.

C2weakest assumption

That the curated Java method bundles and blind perturbations sufficiently represent real-world conflicting artifacts and that the elicited trust traces accurately capture internal model reasoning rather than surface-level pattern matching.

C3one line summary

TRACE reveals that LLMs detect documentation bugs and contradictions better than subtle implementation drift, with asymmetric sensitivity and poor confidence calibration across seven models on 22k traces.

Formal links

2 machine-checked theorem links

Receipt and verification

First computed	2026-07-23T00:23:39.656787Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

a163a0de34e939df4a9143d72816e6a5c42e7a08e8e4b0f16b926b74cc409b28

Aliases

arxiv: 2604.03447 · arxiv_version: 2604.03447v2 · doi: 10.48550/arxiv.2604.03447 · pith_short_12: UFR2BXRU5E45 · pith_short_16: UFR2BXRU5E456SUR · pith_short_8: UFR2BXRU

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/UFR2BXRU5E456SURIPLSQFXGUX \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: a163a0de34e939df4a9143d72816e6a5c42e7a08e8e4b0f16b926b74cc409b28

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "f090aedc264eba67c88433b4d6fd447629943d15497275b362cd2625e6b9b015",
    "cross_cats_sorted": [
      "cs.AI"
    ],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.SE",
    "submitted_at": "2026-04-03T20:38:14Z",
    "title_canon_sha256": "e4c95a618f4d421f830c2fab73754c0592c0781ce3253a69d79c3576b1dbce6f"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2604.03447",
    "kind": "arxiv",
    "version": 2
  }
}