Pith Number

pith:SGPTAII7

pith:2026:SGPTAII74CWGOZG3F43LOCFY53

not attested not anchored not stored refs pending

ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise Java

Advait Pavuluri, Ashita Saxena, Baishakhi Ray, Bridget McGinn, George Safta, Michele Merler, Rahul Krishna, Raju Pavuluri, Srikanth Tamilselvam

Current coding agents succeed on only 15 percent of behavior-preserving cross-framework migrations in enterprise Java.

arxiv:2605.06754 v2 · 2026-05-07 · cs.SE

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{SGPTAII74CWGOZG3F43LOCFY53}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

The strongest agent achieves only 15.3% aggregate test pass on focused-layer migrations and 12.2% on whole applications, and only one of the 204 tasks yields a fully behaviorally equivalent target.

C2weakest assumption

That the 34 expert-written application triples and their associated test oracles are representative of real-world cross-framework migration difficulty and that passing the oracles guarantees behavior preservation outside the tested interface.

C3one line summary

ScarfBench supplies 204 cross-framework Java migration tasks where the best agent passes only 15.3% of focused and 12.2% of whole-application tests.

Formal links

2 machine-checked theorem links

Receipt and verification

First computed	2026-05-20T00:05:46.360763Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

919f30211fe0ac6764db2f36b708b8eeec0b15092959780b99f130bebdeaadf0

Aliases

arxiv: 2605.06754 · arxiv_version: 2605.06754v2 · doi: 10.48550/arxiv.2605.06754 · pith_short_12: SGPTAII74CWG · pith_short_16: SGPTAII74CWGOZG3 · pith_short_8: SGPTAII7

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/SGPTAII74CWGOZG3F43LOCFY53 \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 919f30211fe0ac6764db2f36b708b8eeec0b15092959780b99f130bebdeaadf0

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "89b95648366ba9972ed07ff2c097015f3408bba83125a424dc0e0f4a96603636",
    "cross_cats_sorted": [],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.SE",
    "submitted_at": "2026-05-07T16:05:35Z",
    "title_canon_sha256": "6374c34f88af8b23f0a2f66ed70f1117573f354813749da088f8d75b947169e3"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.06754",
    "kind": "arxiv",
    "version": 2
  }
}