{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:ODHAUUJ4W6KXXLMBS5A5DX6NNE","short_pith_number":"pith:ODHAUUJ4","schema_version":"1.0","canonical_sha256":"70ce0a513cb7957bad819741d1dfcd6919d880547dbe7becf8ab4e2b15317b7d","source":{"kind":"arxiv","id":"2502.14739","version":4},"attestation_state":"computed","paper":{"title":"SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines","license":"http://creativecommons.org/publicdomain/zero/1.0/","headline":"SuperGPQA benchmark shows top LLMs reach only 61.82 percent accuracy across 285 graduate disciplines.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bingli Wang, Chengdong Lin, Chenghua Zhong, Chenglin Cai, Chengtuo Cheng, Chenqing Wang, Chujie Zheng, Chun Zhang, David Ma, Dayiheng Liu, Ge Zhang, Guoyin Wang, Haoran Que, Hao Wang, Hongquan Lin, Jiaheng Liu, Jiajun Xu, Jian Yang, Jinyang Zhang, Junran Peng, Junting Zhou, Kaijing Ma, Kaixin Deng, Kexin Yang, Keyi Ding, King Zhu, Liang Chen, M-A-P Team, Meng Cao, Minghao Liu, Ming Xu, Min Yang, Qian Liu, Qige Qi, Qinrui Li, Qiyao Wang, Qunshu Lin, Ruibin Yuan, Rui Li, Shanghaoran Quan, Shawn Gavin, Shian Jia, Shi Qiu, Shi Wang, Shiwen Ni, Sichao Jiang, Siming Huang, Sirun Li, Siwei Wu, Tianhao Cheng, Tianhao Liang, Tianyang Pang, Tianyang Zhan, Tianyu Liu, Tianyu Zheng, Tyshawn Hsing, Wangchunshu Zhou, Wenbo Su, Wenhao Huang, Xiang Yue, Xiangyu Zheng, Xiaolong Jin, Xingjian Zhang, Xingwei Qu, Xingyuan Bu, Xinrun Du, Xiyue Zhang, Yang Gao, Yaoru Li, Yifan Chen, Yifan Yao, Yiming Liang, Yinghao Ma, Yiyan Liao, Yiya Wang, Yizhe Li, Yizhi Li, Yizhou Tan, Yongchi Zhao, Yuanhao Yue, Yuansheng Ni, Yubo Wang, Yuelin Bai, Yue Zhang, Yujia Qin, Yun Huang, Yunwen Li, Zekun Moore Wang, Zhaoqun Li, Zhaoxiang Zhang, Zhenlin Wei, Zhenzhu Yang, Zhongyuan Peng, Zhoufutu Wen, Zhoujun Li, Zifan Peng, Zili Wang","submitted_at":"2025-02-20T17:05:58Z","abstract_excerpt":"Large language models (LLMs) have demonstrated remarkable proficiency in mainstream academic disciplines such as mathematics, physics, and computer science. However, human knowledge encompasses over 200 specialized disciplines, far exceeding the scope of existing benchmarks. The capabilities of LLMs in many of these specialized fields-particularly in light industry, agriculture, and service-oriented disciplines-remain inadequately evaluated. To address this gap, we present SuperGPQA, a comprehensive benchmark that evaluates graduate-level knowledge and reasoning capabilities across 285 discipl"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2502.14739","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/publicdomain/zero/1.0/","primary_cat":"cs.CL","submitted_at":"2025-02-20T17:05:58Z","cross_cats_sorted":[],"title_canon_sha256":"3d35ab16412f2a2d744cf7d1cdecfbf17234113ad2edc0b1e829f639d42a3ab9","abstract_canon_sha256":"5e4221a4235efe16896596b5e18106ddc45ba53a8f135b2e1478ccf4344aabd6"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:49.563452Z","signature_b64":"uT4vp/Tg9p9YuSVEOptfo3oYZLY+wYyFFtDk8+O5WxN0I8JlcQ3fhAggBu7y3odDYwcZP7ItKEELgBEQfAO+Aw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"70ce0a513cb7957bad819741d1dfcd6919d880547dbe7becf8ab4e2b15317b7d","last_reissued_at":"2026-05-17T23:38:49.562806Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:49.562806Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines","license":"http://creativecommons.org/publicdomain/zero/1.0/","headline":"SuperGPQA benchmark shows top LLMs reach only 61.82 percent accuracy across 285 graduate disciplines.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bingli Wang, Chengdong Lin, Chenghua Zhong, Chenglin Cai, Chengtuo Cheng, Chenqing Wang, Chujie Zheng, Chun Zhang, David Ma, Dayiheng Liu, Ge Zhang, Guoyin Wang, Haoran Que, Hao Wang, Hongquan Lin, Jiaheng Liu, Jiajun Xu, Jian Yang, Jinyang Zhang, Junran Peng, Junting Zhou, Kaijing Ma, Kaixin Deng, Kexin Yang, Keyi Ding, King Zhu, Liang Chen, M-A-P Team, Meng Cao, Minghao Liu, Ming Xu, Min Yang, Qian Liu, Qige Qi, Qinrui Li, Qiyao Wang, Qunshu Lin, Ruibin Yuan, Rui Li, Shanghaoran Quan, Shawn Gavin, Shian Jia, Shi Qiu, Shi Wang, Shiwen Ni, Sichao Jiang, Siming Huang, Sirun Li, Siwei Wu, Tianhao Cheng, Tianhao Liang, Tianyang Pang, Tianyang Zhan, Tianyu Liu, Tianyu Zheng, Tyshawn Hsing, Wangchunshu Zhou, Wenbo Su, Wenhao Huang, Xiang Yue, Xiangyu Zheng, Xiaolong Jin, Xingjian Zhang, Xingwei Qu, Xingyuan Bu, Xinrun Du, Xiyue Zhang, Yang Gao, Yaoru Li, Yifan Chen, Yifan Yao, Yiming Liang, Yinghao Ma, Yiyan Liao, Yiya Wang, Yizhe Li, Yizhi Li, Yizhou Tan, Yongchi Zhao, Yuanhao Yue, Yuansheng Ni, Yubo Wang, Yuelin Bai, Yue Zhang, Yujia Qin, Yun Huang, Yunwen Li, Zekun Moore Wang, Zhaoqun Li, Zhaoxiang Zhang, Zhenlin Wei, Zhenzhu Yang, Zhongyuan Peng, Zhoufutu Wen, Zhoujun Li, Zifan Peng, Zili Wang","submitted_at":"2025-02-20T17:05:58Z","abstract_excerpt":"Large language models (LLMs) have demonstrated remarkable proficiency in mainstream academic disciplines such as mathematics, physics, and computer science. However, human knowledge encompasses over 200 specialized disciplines, far exceeding the scope of existing benchmarks. The capabilities of LLMs in many of these specialized fields-particularly in light industry, agriculture, and service-oriented disciplines-remain inadequately evaluated. To address this gap, we present SuperGPQA, a comprehensive benchmark that evaluates graduate-level knowledge and reasoning capabilities across 285 discipl"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our experimental results reveal significant room for improvement in the performance of current state-of-the-art LLMs across diverse knowledge domains (e.g., the reasoning-focused model DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting the considerable gap between current model capabilities and artificial general intelligence.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The assumption that the Human-LLM collaborative filtering process produces questions that are genuinely graduate-level, unambiguous, and representative of each discipline without introducing selection bias or over-filtering difficult items.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"SuperGPQA is a new benchmark that tests LLMs on graduate questions from 285 disciplines after human-LLM filtering, with current best models scoring 61.82 percent.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"SuperGPQA benchmark shows top LLMs reach only 61.82 percent accuracy across 285 graduate disciplines.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"e3d3e6348f6b9bf142118ad950ca4109a3f58bf2af5bc648ad85092317d57906"},"source":{"id":"2502.14739","kind":"arxiv","version":4},"verdict":{"id":"75de24d0-ec88-4bf6-8e9e-706c6830f652","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-16T00:41:53.807346Z","strongest_claim":"Our experimental results reveal significant room for improvement in the performance of current state-of-the-art LLMs across diverse knowledge domains (e.g., the reasoning-focused model DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting the considerable gap between current model capabilities and artificial general intelligence.","one_line_summary":"SuperGPQA is a new benchmark that tests LLMs on graduate questions from 285 disciplines after human-LLM filtering, with current best models scoring 61.82 percent.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The assumption that the Human-LLM collaborative filtering process produces questions that are genuinely graduate-level, unambiguous, and representative of each discipline without introducing selection bias or over-filtering difficult items.","pith_extraction_headline":"SuperGPQA benchmark shows top LLMs reach only 61.82 percent accuracy across 285 graduate disciplines."},"references":{"count":121,"sample":[{"doi":"10.48550/arxiv.2412.03205","year":2024,"title":"U-math: A university-level benchmark for evaluating mathematical skills in llms","work_id":"4fe452e9-89bf-48b7-9ab9-743ed3f4e445","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"10.18653/v1/d18-1259","year":2024,"title":"Yi: Open Foundation Models by 01.AI","work_id":"8efee8a1-5e3c-4851-9c65-18e3d1d9e769","ref_index":2,"cited_arxiv_id":"2403.04652","is_internal_anchor":true},{"doi":"","year":null,"title":"According to Danto’s definition, context is an art world with modern aspects","work_id":"ec9df59e-9f2c-47c3-ba2a-70421ba7c367","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"“La Bayadère” is a ballet created during the French July Revolution","work_id":"efed4f45-0889-4bb3-8be6-dda7e0f5a4f9","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":null,"title":"The ballet “Sylvia” is a dance drama created during the Paris Commune period in 1871","work_id":"f96be600-e5fe-42e5-ab21-d6b456ec0256","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":121,"snapshot_sha256":"370c1cd06c40c112bb0dc325c51cae27d5120fdd3b7f7d5d1780f7d900e7fc61","internal_anchors":1},"formal_canon":{"evidence_count":1,"snapshot_sha256":"da884ea85f9410c8b2816ccd15462e36129fc28f8f324243e98ba7c81293b7d9"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2502.14739","created_at":"2026-05-17T23:38:49.562914+00:00"},{"alias_kind":"arxiv_version","alias_value":"2502.14739v4","created_at":"2026-05-17T23:38:49.562914+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2502.14739","created_at":"2026-05-17T23:38:49.562914+00:00"},{"alias_kind":"pith_short_12","alias_value":"ODHAUUJ4W6KX","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"ODHAUUJ4W6KXXLMB","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"ODHAUUJ4","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":36,"internal_anchor_count":36,"sample":[{"citing_arxiv_id":"2603.06610","citing_title":"CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training","ref_index":45,"is_internal_anchor":true},{"citing_arxiv_id":"2605.23872","citing_title":"Training-Free Looped Transformers","ref_index":84,"is_internal_anchor":true},{"citing_arxiv_id":"2505.17123","citing_title":"MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2601.10348","citing_title":"Training-Trajectory-Aware Token Selection","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2605.21413","citing_title":"Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work","ref_index":23,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22064","citing_title":"Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild","ref_index":42,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22138","citing_title":"Efficient Agentic Reasoning Through Self-Regulated Simulative Planning","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2508.08636","citing_title":"InternBootcamp Technical Report: Boosting LLM Reasoning with Verifiable Task Scaling","ref_index":10,"is_internal_anchor":true},{"citing_arxiv_id":"2605.21413","citing_title":"Teaching AI Through Benchmark Construction: QuestBench as a Course-Based Practice for Accountable Knowledge Work","ref_index":23,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08738","citing_title":"SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training","ref_index":61,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17037","citing_title":"D$^2$Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning","ref_index":33,"is_internal_anchor":true},{"citing_arxiv_id":"2507.00432","citing_title":"Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning","ref_index":40,"is_internal_anchor":true},{"citing_arxiv_id":"2508.17784","citing_title":"Proximal Supervised Fine-Tuning","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2509.17677","citing_title":"EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2511.04570","citing_title":"Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2511.09907","citing_title":"Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2512.03847","citing_title":"DVPO: Distributional Value Modeling-based Policy Optimization for LLM Post-Training","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2504.21776","citing_title":"WebThinker: Empowering Large Reasoning Models with Deep Research Capability","ref_index":46,"is_internal_anchor":true},{"citing_arxiv_id":"2602.12705","citing_title":"MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2603.01589","citing_title":"SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2603.20633","citing_title":"Seed1.8 Model Card: Towards Generalized Real-World Agency","ref_index":19,"is_internal_anchor":true},{"citing_arxiv_id":"2510.25741","citing_title":"Scaling Latent Reasoning via Looped Language Models","ref_index":63,"is_internal_anchor":true},{"citing_arxiv_id":"2605.10973","citing_title":"Rotation-Preserving Supervised Fine-Tuning","ref_index":37,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11887","citing_title":"Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12500","citing_title":"SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecture","ref_index":33,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":1,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE","json":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE.json","graph_json":"https://pith.science/api/pith-number/ODHAUUJ4W6KXXLMBS5A5DX6NNE/graph.json","events_json":"https://pith.science/api/pith-number/ODHAUUJ4W6KXXLMBS5A5DX6NNE/events.json","paper":"https://pith.science/paper/ODHAUUJ4"},"agent_actions":{"view_html":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE","download_json":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE.json","view_paper":"https://pith.science/paper/ODHAUUJ4","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2502.14739&json=true","fetch_graph":"https://pith.science/api/pith-number/ODHAUUJ4W6KXXLMBS5A5DX6NNE/graph.json","fetch_events":"https://pith.science/api/pith-number/ODHAUUJ4W6KXXLMBS5A5DX6NNE/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE/action/timestamp_anchor","attest_storage":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE/action/storage_attestation","attest_author":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE/action/author_attestation","sign_citation":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE/action/citation_signature","submit_replication":"https://pith.science/pith/ODHAUUJ4W6KXXLMBS5A5DX6NNE/action/replication_record"}},"created_at":"2026-05-17T23:38:49.562914+00:00","updated_at":"2026-05-17T23:38:49.562914+00:00"}