Pith Number

pith:Q6JYRBA5

pith:2026:Q6JYRBA52K2D2R3IBPLEI6NOCW

not attested not anchored not stored refs pending

Evaluating LLM-Generated Code: A Benchmark and Developer Study

Anne Schwerk, Joanna Szych

Developer reviews uncover production-readiness issues in LLM code that standard correctness benchmarks overlook.

arxiv:2605.09059 v2 · 2026-05-09 · cs.SE

Open paper page JSON Open Graph Bundle Merged state Verified badge What is a Pith Number?

Add to your LaTeX paper

\usepackage{pith}
\pithnumber{Q6JYRBA52K2D2R3IBPLEI6NOCW}

Prints a linked badge after your title and injects PDF metadata. Compiles on arXiv. Learn more · Embed verified badge

Record completeness

1 Bitcoin timestamp

2 Internet Archive

3 Author claim open · sign in to claim

4 Citations open

5 Replications open

✓ Portable graph bundle live · download bundle · merged state

The bundle contains the canonical record plus signed events. A mirror can host it anywhere and recompute the same current state with the deterministic merge algorithm.

Claims

C1strongest claim

reviews gathered from developers can yield many new findings, especially those related to the code being in a production-ready state, that would not be possible to obtain using the standard correctness-focused benchmark approach.

C2weakest assumption

That a structured code-review process with (unspecified number of) developers produces generalizable, unbiased insights into production readiness that standard benchmarks miss.

C3one line summary

A custom three-fold methodology combining a complex-project correctness benchmark, code quality verification, and structured developer reviews to evaluate LLM-generated code beyond correctness alone.

Receipt and verification

First computed	2026-06-11T00:08:12.678488Z
Builder	pith-number-builder-2026-05-17-v1
Signature	Pith Ed25519 (`pith-v1-2026-05`) · public key
Schema	pith-number/v1.0

Canonical hash

879388841dd2b43d47680bd64479ae15b5d9d59df3640a661e05d4f6c921f58c

Aliases

arxiv: 2605.09059 · arxiv_version: 2605.09059v2 · doi: 10.48550/arxiv.2605.09059 · pith_short_12: Q6JYRBA52K2D · pith_short_16: Q6JYRBA52K2D2R3I · pith_short_8: Q6JYRBA5

Agent API

Resolver JSON Graph JSON Events JSON Schema Signing key

Verify this Pith Number yourself

curl -sH 'Accept: application/ld+json' https://pith.science/pith/Q6JYRBA52K2D2R3IBPLEI6NOCW \
  | jq -c '.canonical_record' \
  | python3 -c "import sys,json,hashlib; b=json.dumps(json.loads(sys.stdin.read()), sort_keys=True, separators=(',',':'), ensure_ascii=False).encode(); print(hashlib.sha256(b).hexdigest())"
# expect: 879388841dd2b43d47680bd64479ae15b5d9d59df3640a661e05d4f6c921f58c

Canonical record JSON

{
  "metadata": {
    "abstract_canon_sha256": "4b89152d2063d8e7594c94602a53c3c7b9a3d6b5ead2bc613d34e5dc4121b420",
    "cross_cats_sorted": [],
    "license": "http://creativecommons.org/licenses/by/4.0/",
    "primary_cat": "cs.SE",
    "submitted_at": "2026-05-09T17:05:44Z",
    "title_canon_sha256": "7a1170a886a839d58bc029648a6f567746c011c4ad1a7d8271d38087b9aa7b5e"
  },
  "schema_version": "1.0",
  "source": {
    "id": "2605.09059",
    "kind": "arxiv",
    "version": 2
  }
}