{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2026:AARY37WBSXTPWMW5TVTXEYHQQX","short_pith_number":"pith:AARY37WB","schema_version":"1.0","canonical_sha256":"00238dfec195e6fb32dd9d677260f085f8cad29fd1b6df629cba53839b9cfbfb","source":{"kind":"arxiv","id":"2605.15222","version":1},"attestation_state":"computed","paper":{"title":"PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Current LLMs produce code that is functionally correct but far from expert-optimized on system-level performance tasks.","cross_cats":["cs.CL","cs.PL"],"primary_cat":"cs.SE","authors_text":"Hanyu Yang, Haochen Shi, Haoran Li, Huihao Jing, Shaojin Chen, Sirui Zhang, Wenbin Hu, Yangqiu Song","submitted_at":"2026-05-13T08:10:26Z","abstract_excerpt":"Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenec"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":false},"canonical_record":{"source":{"id":"2605.15222","kind":"arxiv","version":1},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.SE","submitted_at":"2026-05-13T08:10:26Z","cross_cats_sorted":["cs.CL","cs.PL"],"title_canon_sha256":"73db000b4f0a474b1bde5d52111578c40706987ac468084b07681e8f9b247a65","abstract_canon_sha256":"3379dd3feb7dc3a4a4c597fdbe4e5579af83915fd86765d7a61389721213c942"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-20T00:00:47.042573Z","signature_b64":"cy55gW0C26Ew4bTQWYhBzoeuDSd79DUB069588IFkX+fUy/ddjuok+UHyU3JaQQN3zqrK18cvBUDJPr5P7KhDg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"00238dfec195e6fb32dd9d677260f085f8cad29fd1b6df629cba53839b9cfbfb","last_reissued_at":"2026-05-20T00:00:47.041876Z","signature_status":"signed_v1","first_computed_at":"2026-05-20T00:00:47.041876Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Current LLMs produce code that is functionally correct but far from expert-optimized on system-level performance tasks.","cross_cats":["cs.CL","cs.PL"],"primary_cat":"cs.SE","authors_text":"Hanyu Yang, Haochen Shi, Haoran Li, Huihao Jing, Shaojin Chen, Sirui Zhang, Wenbin Hu, Yangqiu Song","submitted_at":"2026-05-13T08:10:26Z","abstract_excerpt":"Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-critical systems tasks remains limited. Existing code benchmarks mainly emphasize correctness or algorithmic problem solving, while realistic systems-level optimization is still underexplored. To address this gap, we introduce PerfCodeBench, an executable benchmark for evaluating LLMs on high-performance code optimization. The tasks require system-level implementation choices, hardware-aware optimization, and careful handling of performance bottlenec"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The selected tasks accurately capture realistic system-level implementation choices, hardware-aware optimizations, and performance bottlenecks that matter in practice.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"PerfCodeBench reveals that state-of-the-art LLMs produce functionally correct but significantly slower code than expert-optimized versions on system-level tasks, especially those involving parallelism and GPUs.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Current LLMs produce code that is functionally correct but far from expert-optimized on system-level performance tasks.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"ce9f3414af28d8893fc1a2f1c87a7f262d6036acd0ada58b77999e174696a3b1"},"source":{"id":"2605.15222","kind":"arxiv","version":1},"verdict":{"id":"56bb79b6-e790-4b12-8c5b-d5dd3b17467a","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-19T17:58:50.521493Z","strongest_claim":"Our evaluation on a broad set of state-of-the-art LLMs shows a clear gap between model-generated code and expert-optimized implementations. The gap is especially large on tasks involving parallelism and GPU operations.","one_line_summary":"PerfCodeBench reveals that state-of-the-art LLMs produce functionally correct but significantly slower code than expert-optimized versions on system-level tasks, especially those involving parallelism and GPUs.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The selected tasks accurately capture realistic system-level implementation choices, hardware-aware optimizations, and performance bottlenecks that matter in practice.","pith_extraction_headline":"Current LLMs produce code that is functionally correct but far from expert-optimized on system-level performance tasks."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2605.15222/integrity.json","findings":[],"available":true,"detectors_run":[{"name":"claim_evidence","ran_at":"2026-05-19T20:21:57.158243Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_title_agreement","ran_at":"2026-05-19T18:31:18.819737Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"doi_compliance","ran_at":"2026-05-19T18:10:16.387371Z","status":"completed","version":"1.0.0","findings_count":0},{"name":"ai_meta_artifact","ran_at":"2026-05-19T13:33:22.835500Z","status":"skipped","version":"1.0.0","findings_count":0}],"snapshot_sha256":"17bae781a0bec8d0c469624434c6193720fefc03dae918aa25262292e1530d19"},"references":{"count":52,"sample":[{"doi":"","year":2025,"title":"Introducing Claude Opus 4.5","work_id":"785c69cd-ef52-4c1c-80f5-3c8645ec6dbc","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"Claude Code Overview","work_id":"dcfdde82-d743-457f-b8f4-1f23552dc365","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"Claude model overview","work_id":"114f4f5b-3359-458c-ba85-d8e90a92db5e","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2025,"title":"Understanding software engineering agents: A study of thought-action-result trajectories","work_id":"ddc723de-4578-4bab-a626-5da26d2d16f7","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2026,"title":"ByteDance Seed. Seed2.0.https://seed.bytedance.com/en/seed2, 2026","work_id":"d73a7341-aa04-4bbf-8412-53ceacca5313","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":52,"snapshot_sha256":"948508292d948b51f13bfef70f9292e6f97b56534db1d54f33a298acfb604e6c","internal_anchors":4},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2605.15222","created_at":"2026-05-20T00:00:47.041983+00:00"},{"alias_kind":"arxiv_version","alias_value":"2605.15222v1","created_at":"2026-05-20T00:00:47.041983+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2605.15222","created_at":"2026-05-20T00:00:47.041983+00:00"},{"alias_kind":"pith_short_12","alias_value":"AARY37WBSXTP","created_at":"2026-05-20T00:00:47.041983+00:00"},{"alias_kind":"pith_short_16","alias_value":"AARY37WBSXTPWMW5","created_at":"2026-05-20T00:00:47.041983+00:00"},{"alias_kind":"pith_short_8","alias_value":"AARY37WB","created_at":"2026-05-20T00:00:47.041983+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":0,"internal_anchor_count":0,"sample":[]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX","json":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX.json","graph_json":"https://pith.science/api/pith-number/AARY37WBSXTPWMW5TVTXEYHQQX/graph.json","events_json":"https://pith.science/api/pith-number/AARY37WBSXTPWMW5TVTXEYHQQX/events.json","paper":"https://pith.science/paper/AARY37WB"},"agent_actions":{"view_html":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX","download_json":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX.json","view_paper":"https://pith.science/paper/AARY37WB","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2605.15222&json=true","fetch_graph":"https://pith.science/api/pith-number/AARY37WBSXTPWMW5TVTXEYHQQX/graph.json","fetch_events":"https://pith.science/api/pith-number/AARY37WBSXTPWMW5TVTXEYHQQX/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX/action/timestamp_anchor","attest_storage":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX/action/storage_attestation","attest_author":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX/action/author_attestation","sign_citation":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX/action/citation_signature","submit_replication":"https://pith.science/pith/AARY37WBSXTPWMW5TVTXEYHQQX/action/replication_record"}},"created_at":"2026-05-20T00:00:47.041983+00:00","updated_at":"2026-05-20T00:00:47.041983+00:00"}