{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:34FHKX42YDQIVGLZGAH2Z6VLK3","short_pith_number":"pith:34FHKX42","schema_version":"1.0","canonical_sha256":"df0a755f9ac0e08a9979300facfaab56e1f5a0dbac3ba95bb90ff66abfeecacc","source":{"kind":"arxiv","id":"2409.12183","version":3},"attestation_state":"computed","paper":{"title":"To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Dongwei Jiang, Fangcong Yin, Greg Durrett, Juan Diego Rodriguez, Kyle Mahowald, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Zayne Sprague","submitted_at":"2024-09-18T17:55:00Z","abstract_excerpt":"Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2409.12183","kind":"arxiv","version":3},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2024-09-18T17:55:00Z","cross_cats_sorted":["cs.AI","cs.LG"],"title_canon_sha256":"3694254e490b8480e54fe52674fb3a7c3c593ed75d33ad92cc59e9e1d08498f8","abstract_canon_sha256":"580e915f36ab184649effba2cba322837a2d70a73347dda5746f240b73e9b7db"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T11:00:00.196628Z","signature_b64":"mgWsN72QbmtSeCYx2+aqxj/6LxHy6x7y38grlkDwdlLBfIEqV+pizb+7xaJv4kFJ5sYlDz8nlJhqCgDWnzFQCw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"df0a755f9ac0e08a9979300facfaab56e1f5a0dbac3ba95bb90ff66abfeecacc","last_reissued_at":"2026-07-05T11:00:00.196109Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T11:00:00.196109Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI","cs.LG"],"primary_cat":"cs.CL","authors_text":"Dongwei Jiang, Fangcong Yin, Greg Durrett, Juan Diego Rodriguez, Kyle Mahowald, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Zayne Sprague","submitted_at":"2024-09-18T17:55:00Z","abstract_excerpt":"Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thinking'' really helpful? To analyze this, we conducted a quantitative meta-analysis covering over 100 papers using CoT and ran our own evaluations of 20 datasets across 14 models. Our results show that CoT gives strong performance benefits primarily on tasks involving math or logic, with much smaller gains on other types of tasks. On MMLU, directly generating the answer without CoT leads to almost identical accuracy as"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2409.12183","kind":"arxiv","version":3},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2409.12183/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2409.12183","created_at":"2026-07-05T11:00:00.196174+00:00"},{"alias_kind":"arxiv_version","alias_value":"2409.12183v3","created_at":"2026-07-05T11:00:00.196174+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2409.12183","created_at":"2026-07-05T11:00:00.196174+00:00"},{"alias_kind":"pith_short_12","alias_value":"34FHKX42YDQI","created_at":"2026-07-05T11:00:00.196174+00:00"},{"alias_kind":"pith_short_16","alias_value":"34FHKX42YDQIVGLZ","created_at":"2026-07-05T11:00:00.196174+00:00"},{"alias_kind":"pith_short_8","alias_value":"34FHKX42","created_at":"2026-07-05T11:00:00.196174+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":13,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.10142","citing_title":"DB-3DME: From Dataset to Benchmark for Human-aligned Automatic 3D Mesh Evaluation","ref_index":31,"is_internal_anchor":false},{"citing_arxiv_id":"2606.05315","citing_title":"LoRi: Low-Rank Distillation for Implicit Reasoning","ref_index":29,"is_internal_anchor":false},{"citing_arxiv_id":"2605.22873","citing_title":"When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions","ref_index":2,"is_internal_anchor":false},{"citing_arxiv_id":"2504.05605","citing_title":"ShadowCoT: Cognitive Hijacking for Stealthy Reasoning Backdoors in LLMs","ref_index":3,"is_internal_anchor":false},{"citing_arxiv_id":"2605.14344","citing_title":"CrystalReasoner: Reasoning and RL for Property-Conditioned Crystal Structure Generation","ref_index":14,"is_internal_anchor":false},{"citing_arxiv_id":"2601.06993","citing_title":"Can Textual Reasoning Improve the Performance of MLLMs on Fine-grained Visual Classification?","ref_index":31,"is_internal_anchor":false},{"citing_arxiv_id":"2605.11260","citing_title":"Curriculum Learning-Guided Progressive Distillation in Large Language Models","ref_index":38,"is_internal_anchor":false},{"citing_arxiv_id":"2604.27143","citing_title":"Enhancing Linux Privilege Escalation Attack Capabilities of Local LLM Agents","ref_index":25,"is_internal_anchor":false},{"citing_arxiv_id":"2604.25155","citing_title":"Rethinking Wireless Communications through Formal Mathematical AI Reasoning","ref_index":77,"is_internal_anchor":false},{"citing_arxiv_id":"2605.05715","citing_title":"Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimes","ref_index":60,"is_internal_anchor":false},{"citing_arxiv_id":"2604.08232","citing_title":"HiRO-Nav: Hybrid ReasOning Enables Efficient Embodied Navigation","ref_index":29,"is_internal_anchor":false},{"citing_arxiv_id":"2604.09237","citing_title":"ScheMatiQ: From Research Question to Structured Data through Interactive Schema Discovery","ref_index":4,"is_internal_anchor":false},{"citing_arxiv_id":"2604.15994","citing_title":"ReactBench: A Benchmark for Topological Reasoning in MLLMs on Chemical Reaction Diagrams","ref_index":2,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3","json":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3.json","graph_json":"https://pith.science/api/pith-number/34FHKX42YDQIVGLZGAH2Z6VLK3/graph.json","events_json":"https://pith.science/api/pith-number/34FHKX42YDQIVGLZGAH2Z6VLK3/events.json","paper":"https://pith.science/paper/34FHKX42"},"agent_actions":{"view_html":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3","download_json":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3.json","view_paper":"https://pith.science/paper/34FHKX42","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2409.12183&json=true","fetch_graph":"https://pith.science/api/pith-number/34FHKX42YDQIVGLZGAH2Z6VLK3/graph.json","fetch_events":"https://pith.science/api/pith-number/34FHKX42YDQIVGLZGAH2Z6VLK3/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3/action/timestamp_anchor","attest_storage":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3/action/storage_attestation","attest_author":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3/action/author_attestation","sign_citation":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3/action/citation_signature","submit_replication":"https://pith.science/pith/34FHKX42YDQIVGLZGAH2Z6VLK3/action/replication_record"}},"created_at":"2026-07-05T11:00:00.196174+00:00","updated_at":"2026-07-05T11:00:00.196174+00:00"}