{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2022:QCBSLD4SWIGU7TWOXAP7TUKAQ7","short_pith_number":"pith:QCBSLD4S","schema_version":"1.0","canonical_sha256":"8083258f92b20d4fceceb81ff9d14087ddf2422d0c6725e5be4213be28631b5e","source":{"kind":"arxiv","id":"2209.00840","version":3},"attestation_state":"computed","paper":{"title":"FOLIO: Natural Language Reasoning with First-Order Logic","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Alexander R. Fabbri, Alex Wardle-Solano, Ansong Ni, Arman Cohan, Brian Wong, Caiming Xiong, David Peng, Dragomir Radev, Ekaterina Zubova, Hailey Schoelkopf, Hannah Szabo, James Coady, Jonathan Fan, Jungo Kasai, Linyong Nan, Lucy Sun, Luke Benson, Malcolm Sailor, Martin Riddell, Matthew Burtell, Rex Ying, Rui Zhang, Semih Yavuz, Shafiq Joty, Simeng Han, Tao Yu, Wenfei Zhou, Wojciech Kryscinski, Xi Victoria Lin, Ye Liu, Yilun Zhao, Yingbo Zhou, Yixin Liu, Yujie Qiao, Zhenting Qi","submitted_at":"2022-09-02T06:50:11Z","abstract_excerpt":"Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. The logical correctness of the premise"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2209.00840","kind":"arxiv","version":3},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.CL","submitted_at":"2022-09-02T06:50:11Z","cross_cats_sorted":[],"title_canon_sha256":"4e5189ba1f9be9acd5013358e3293fed2e1e0dba59f878c61cf46219402c7b66","abstract_canon_sha256":"c06ef0bc42bd72adad9d480230fccd0e0f2d2fcb252357014f0d620db73f3a06"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T09:19:22.554593Z","signature_b64":"Z4dxh3pA9wnc0gsbymLLXs8E+OzpbKLYcqwWLAoXUMrMk7+dvppFg4zFmdbLsRON7Vf4T+KfjOzy8AWG5CuWCQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"8083258f92b20d4fceceb81ff9d14087ddf2422d0c6725e5be4213be28631b5e","last_reissued_at":"2026-07-05T09:19:22.554046Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T09:19:22.554046Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"FOLIO: Natural Language Reasoning with First-Order Logic","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Alexander R. Fabbri, Alex Wardle-Solano, Ansong Ni, Arman Cohan, Brian Wong, Caiming Xiong, David Peng, Dragomir Radev, Ekaterina Zubova, Hailey Schoelkopf, Hannah Szabo, James Coady, Jonathan Fan, Jungo Kasai, Linyong Nan, Lucy Sun, Luke Benson, Malcolm Sailor, Martin Riddell, Matthew Burtell, Rex Ying, Rui Zhang, Semih Yavuz, Shafiq Joty, Simeng Han, Tao Yu, Wenfei Zhou, Wojciech Kryscinski, Xi Victoria Lin, Ye Liu, Yilun Zhao, Yingbo Zhou, Yixin Liu, Yujie Qiao, Zhenting Qi","submitted_at":"2022-09-02T06:50:11Z","abstract_excerpt":"Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. The logical correctness of the premise"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2209.00840","kind":"arxiv","version":3},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2209.00840/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2209.00840","created_at":"2026-07-05T09:19:22.554107+00:00"},{"alias_kind":"arxiv_version","alias_value":"2209.00840v3","created_at":"2026-07-05T09:19:22.554107+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2209.00840","created_at":"2026-07-05T09:19:22.554107+00:00"},{"alias_kind":"pith_short_12","alias_value":"QCBSLD4SWIGU","created_at":"2026-07-05T09:19:22.554107+00:00"},{"alias_kind":"pith_short_16","alias_value":"QCBSLD4SWIGU7TWO","created_at":"2026-07-05T09:19:22.554107+00:00"},{"alias_kind":"pith_short_8","alias_value":"QCBSLD4S","created_at":"2026-07-05T09:19:22.554107+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":11,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.20873","citing_title":"SciLens: Multi-modal Scientific Claim Verification with Agentic Entailment and Grounding","ref_index":4,"is_internal_anchor":false},{"citing_arxiv_id":"2606.16118","citing_title":"Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning","ref_index":21,"is_internal_anchor":false},{"citing_arxiv_id":"2607.01612","citing_title":"Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling","ref_index":50,"is_internal_anchor":false},{"citing_arxiv_id":"2606.02907","citing_title":"Linear Probes Detect Task Format, Not Reasoning Mode in Language Model Hidden States","ref_index":6,"is_internal_anchor":false},{"citing_arxiv_id":"2605.24305","citing_title":"ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale","ref_index":4,"is_internal_anchor":false},{"citing_arxiv_id":"2509.24765","citing_title":"Semantic-Aware Logical Reasoning via a Semiotic Framework","ref_index":17,"is_internal_anchor":false},{"citing_arxiv_id":"2305.14992","citing_title":"Reasoning with Language Model is Planning with World Model","ref_index":65,"is_internal_anchor":false},{"citing_arxiv_id":"2604.02733","citing_title":"DeltaLogic: Minimal Premise Edits Reveal Belief-Revision Failures in Logical Reasoning Models","ref_index":5,"is_internal_anchor":false},{"citing_arxiv_id":"2605.08905","citing_title":"Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMs","ref_index":40,"is_internal_anchor":false},{"citing_arxiv_id":"2604.05396","citing_title":"Reason Analogically via Cross-domain Prior Knowledge: An Empirical Study of Cross-domain Knowledge Transfer for In-Context Learning","ref_index":3,"is_internal_anchor":false},{"citing_arxiv_id":"2604.05383","citing_title":"Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrieval","ref_index":3,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7","json":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7.json","graph_json":"https://pith.science/api/pith-number/QCBSLD4SWIGU7TWOXAP7TUKAQ7/graph.json","events_json":"https://pith.science/api/pith-number/QCBSLD4SWIGU7TWOXAP7TUKAQ7/events.json","paper":"https://pith.science/paper/QCBSLD4S"},"agent_actions":{"view_html":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7","download_json":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7.json","view_paper":"https://pith.science/paper/QCBSLD4S","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2209.00840&json=true","fetch_graph":"https://pith.science/api/pith-number/QCBSLD4SWIGU7TWOXAP7TUKAQ7/graph.json","fetch_events":"https://pith.science/api/pith-number/QCBSLD4SWIGU7TWOXAP7TUKAQ7/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7/action/timestamp_anchor","attest_storage":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7/action/storage_attestation","attest_author":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7/action/author_attestation","sign_citation":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7/action/citation_signature","submit_replication":"https://pith.science/pith/QCBSLD4SWIGU7TWOXAP7TUKAQ7/action/replication_record"}},"created_at":"2026-07-05T09:19:22.554107+00:00","updated_at":"2026-07-05T09:19:22.554107+00:00"}