{"work":{"id":"50652ac6-fb7c-4675-a2c2-159c241feb17","openalex_id":null,"doi":null,"arxiv_id":"2103.03874","raw_key":null,"title":"Measuring Mathematical Problem Solving With the MATH Dataset","authors":null,"authors_text":"Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang","year":2021,"venue":"cs.LG","abstract":"Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are able to increase accuracy on MATH, our results show that accuracy remains relatively low, even with enormous Transformer models. Moreover, we find that simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue. While scaling Transformers is automatically solving most other text-based tasks, scaling is not currently solving MATH. To have more traction on mathematical problem solving we will likely need new algorithmic advancements from the broader research community.","external_url":"https://arxiv.org/abs/2103.03874","cited_by_count":null,"metadata_source":"pith","metadata_fetched_at":"2026-06-29T11:53:23.708646+00:00","pith_arxiv_id":"2103.03874","created_at":"2026-05-09T05:55:30.023550+00:00","updated_at":"2026-06-29T11:53:23.708646+00:00","title_quality_ok":true,"display_title":"Measuring Mathematical Problem Solving With the MATH Dataset","render_title":"Measuring Mathematical Problem Solving With the MATH Dataset"},"hub":{"state":{"work_id":"50652ac6-fb7c-4675-a2c2-159c241feb17","tier":"super_hub","tier_reason":"100+ Pith inbound or 10,000+ external citations","pith_inbound_count":335,"external_cited_by_count":null,"distinct_field_count":15,"first_pith_cited_at":"2021-05-20T17:58:42+00:00","last_pith_cited_at":"2026-06-15T20:54:04+00:00","author_build_status":"needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-06-29T11:58:41.941158+00:00","tier_text":"super_hub"},"tier":"super_hub","role_counts":[{"context_role":"dataset","n":47},{"context_role":"background","n":32},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"use_dataset","n":43},{"context_polarity":"background","n":32},{"context_polarity":"unclear","n":4},{"context_polarity":"use_method","n":1}],"runs":{"ask_index":{"job_type":"ask_index","status":"succeeded","result":{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","claims":[{"claim_text":"Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Measuring Mathematical Problem Solving With the MATH Dataset because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-13T22:13:35.795320+00:00"},"author_expand":{"job_type":"author_expand","status":"succeeded","result":{"authors_linked":[{"id":"9a245225-c7c3-48a7-b813-bdc1644e9016","orcid":null,"display_name":"Dan Hendrycks"},{"id":"3b4ceaf7-2386-40e2-b124-322be5048cad","orcid":null,"display_name":"Collin Burns"},{"id":"155c969c-3372-4b52-b881-15f051250399","orcid":null,"display_name":"Saurav Kadavath"},{"id":"31346449-d329-4cf1-a738-0c27d84e4673","orcid":null,"display_name":"Akul Arora"},{"id":"7a82e646-0403-4c2b-85ba-72d9e0f173cd","orcid":null,"display_name":"Steven Basart"},{"id":"2a7badf4-c82d-49a5-bc1f-5e4a9b95cadb","orcid":null,"display_name":"Eric Tang"}]},"error":null,"updated_at":"2026-05-13T22:13:39.764378+00:00"},"context_extract":{"job_type":"context_extract","status":"succeeded","result":{"enqueued_papers":25},"error":null,"updated_at":"2026-05-13T22:13:36.748064+00:00"},"graph_features":{"job_type":"graph_features","status":"succeeded","result":{"co_cited":[{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":85},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":63},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":56},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":55},{"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","shared_citers":49},{"title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","work_id":"64019d00-0b11-4bbd-b173-b46c8fad0157","shared_citers":38},{"title":"Proximal Policy Optimization Algorithms","work_id":"240c67fe-d14d-4520-91c1-38a4e272ca19","shared_citers":36},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":34},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":32},{"title":"Program Synthesis with Large Language Models","work_id":"fd241a05-03b9-4de2-9588-9d77ce176125","shared_citers":32},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":27},{"title":"OpenAI o1 System Card","work_id":"68d3c334-0fc9-49e3-b7b0-a69afae933e2","shared_citers":25},{"title":"Measuring Massive Multitask Language Understanding","work_id":"e87ec49a-544b-4ec8-8991-75298c64ff5e","shared_citers":24},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":22},{"title":"LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code","work_id":"ea9e51ce-1e75-4182-92d8-4d25f70d2ee4","shared_citers":22},{"title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models","work_id":"8c6d5a6b-b5cc-4105-9c84-9c34bb9375bb","shared_citers":22},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":20},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":19},{"title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","work_id":"a097c5d4-6d32-46ee-9826-57d532bbfc9c","shared_citers":18},{"title":"Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them","work_id":"513eb205-04ca-4722-9a43-a74e8cbe7e85","shared_citers":17},{"title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","work_id":"d854765a-e664-41c0-8655-21c4bf2e0cc4","shared_citers":17},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":16},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":16},{"title":"Mistral 7B","work_id":"eb5e1305-ad11-4875-ad8d-ad8b8f697599","shared_citers":16}],"time_series":[{"n":1,"year":2021},{"n":3,"year":2022},{"n":4,"year":2023},{"n":11,"year":2024},{"n":4,"year":2025},{"n":129,"year":2026}]},"error":null,"updated_at":"2026-05-13T22:13:39.831383+00:00"},"identity_refresh":{"job_type":"identity_refresh","status":"succeeded","result":{"fixed":1,"items":[{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","resolver":"local_arxiv","confidence":0.98,"old_work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e"}],"errors":[],"attempted":1},"error":null,"updated_at":"2026-05-13T22:13:42.247637+00:00"},"role_polarity":{"job_type":"role_polarity","status":"succeeded","result":{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","claims":[{"claim_text":"Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Measuring Mathematical Problem Solving With the MATH Dataset because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-13T22:13:39.834308+00:00"},"summary_claims":{"job_type":"summary_claims","status":"succeeded","result":{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","claims":[{"claim_text":"Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Measuring Mathematical Problem Solving With the MATH Dataset because it crossed a citation-hub threshold.","role_counts":[]},"error":null,"updated_at":"2026-05-13T22:13:39.071417+00:00"}},"summary":{"title":"Measuring Mathematical Problem Solving With the MATH Dataset","claims":[{"claim_text":"Many intellectual endeavors require mathematical problem solving, but this skill remains beyond the capabilities of computers. To measure this ability in machine learning models, we introduce MATH, a new dataset of 12,500 challenging competition mathematics problems. Each problem in MATH has a full step-by-step solution which can be used to teach models to generate answer derivations and explanations. To facilitate future research and increase accuracy on MATH, we also contribute a large auxiliary pretraining dataset which helps teach models the fundamentals of mathematics. Even though we are ","claim_type":"abstract","evidence_strength":"source_metadata"}],"why_cited":"Pith tracks Measuring Mathematical Problem Solving With the MATH Dataset because it crossed a citation-hub threshold.","role_counts":[]},"graph":{"co_cited":[{"title":"Training Verifiers to Solve Math Word Problems","work_id":"acab1aa8-b4d6-40e0-a3ee-25341701dca2","shared_citers":85},{"title":"DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models","work_id":"c5006563-f3ec-438a-9e35-b7b484f34828","shared_citers":63},{"title":"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning","work_id":"e6b75ad5-2877-4168-97c8-710407094d20","shared_citers":56},{"title":"Qwen3 Technical Report","work_id":"25a4e30c-1232-48e7-9925-02fa12ba7c9e","shared_citers":55},{"title":"Evaluating Large Language Models Trained on Code","work_id":"042493e9-b26f-4b4e-bbde-382072ca9b08","shared_citers":49},{"title":"DAPO: An Open-Source LLM Reinforcement Learning System at Scale","work_id":"64019d00-0b11-4bbd-b173-b46c8fad0157","shared_citers":38},{"title":"Proximal Policy Optimization Algorithms","work_id":"240c67fe-d14d-4520-91c1-38a4e272ca19","shared_citers":36},{"title":"The Llama 3 Herd of Models","work_id":"1549a635-88af-4ac1-acfe-51ae7bb53345","shared_citers":34},{"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","shared_citers":32},{"title":"Program Synthesis with Large Language Models","work_id":"fd241a05-03b9-4de2-9588-9d77ce176125","shared_citers":32},{"title":"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge","work_id":"28ea1282-d657-4c61-a83c-f1249be6d6b1","shared_citers":27},{"title":"OpenAI o1 System Card","work_id":"68d3c334-0fc9-49e3-b7b0-a69afae933e2","shared_citers":25},{"title":"Measuring Massive Multitask Language Understanding","work_id":"e87ec49a-544b-4ec8-8991-75298c64ff5e","shared_citers":24},{"title":"DeepSeek-V3 Technical Report","work_id":"57d2791d-2219-4c31-a077-afc04b12a75c","shared_citers":22},{"title":"LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code","work_id":"ea9e51ce-1e75-4182-92d8-4d25f70d2ee4","shared_citers":22},{"title":"Self-Consistency Improves Chain of Thought Reasoning in Language Models","work_id":"8c6d5a6b-b5cc-4105-9c84-9c34bb9375bb","shared_citers":22},{"title":"Qwen2.5 Technical Report","work_id":"d8432992-4980-4a81-85c7-9fa2c2b87f85","shared_citers":20},{"title":"Scaling Laws for Neural Language Models","work_id":"b7dd8749-9c45-4977-ab9b-64478dce1ae8","shared_citers":19},{"title":"Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement","work_id":"a097c5d4-6d32-46ee-9826-57d532bbfc9c","shared_citers":18},{"title":"Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them","work_id":"513eb205-04ca-4722-9a43-a74e8cbe7e85","shared_citers":17},{"title":"Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?","work_id":"d854765a-e664-41c0-8655-21c4bf2e0cc4","shared_citers":17},{"title":"Llama 2: Open Foundation and Fine-Tuned Chat Models","work_id":"68a5177f-d644-44c1-bd4f-4e5278c22f5d","shared_citers":16},{"title":"LLaMA: Open and Efficient Foundation Language Models","work_id":"c018fc23-6f3f-4035-9d02-28a2173b2b9d","shared_citers":16},{"title":"Mistral 7B","work_id":"eb5e1305-ad11-4875-ad8d-ad8b8f697599","shared_citers":16}],"time_series":[{"n":1,"year":2021},{"n":3,"year":2022},{"n":4,"year":2023},{"n":11,"year":2024},{"n":4,"year":2025},{"n":129,"year":2026}]},"authors":[{"id":"31346449-d329-4cf1-a738-0c27d84e4673","orcid":null,"display_name":"Akul Arora","source":"manual","import_confidence":0.72},{"id":"3b4ceaf7-2386-40e2-b124-322be5048cad","orcid":null,"display_name":"Collin Burns","source":"manual","import_confidence":0.72},{"id":"9a245225-c7c3-48a7-b813-bdc1644e9016","orcid":null,"display_name":"Dan Hendrycks","source":"manual","import_confidence":0.72},{"id":"2a7badf4-c82d-49a5-bc1f-5e4a9b95cadb","orcid":null,"display_name":"Eric Tang","source":"manual","import_confidence":0.72},{"id":"155c969c-3372-4b52-b881-15f051250399","orcid":null,"display_name":"Saurav Kadavath","source":"manual","import_confidence":0.72},{"id":"7a82e646-0403-4c2b-85ba-72d9e0f173cd","orcid":null,"display_name":"Steven Basart","source":"manual","import_confidence":0.72}]}}