{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:WCHBCUAJDPJA3BI2DVHH5U5GUG","short_pith_number":"pith:WCHBCUAJ","schema_version":"1.0","canonical_sha256":"b08e1150091bd20d851a1d4e7ed3a6a1b85728467986a54b1264c50b5ba05ea7","source":{"kind":"arxiv","id":"2501.00663","version":1},"attestation_state":"computed","paper":{"title":"Titans: Learning to Memorize at Test Time","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Titans combine attention with a learnable neural long-term memory to handle contexts over two million tokens more effectively than Transformers or linear recurrent models.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.LG","authors_text":"Ali Behrouz, Peilin Zhong, Vahab Mirrokni","submitted_at":"2024-12-31T22:32:03Z","abstract_excerpt":"Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a new neural long-term memory module that learns to memorize historical context and helps attention to attend to the current con"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2501.00663","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.LG","submitted_at":"2024-12-31T22:32:03Z","cross_cats_sorted":["cs.AI","cs.CL"],"title_canon_sha256":"68ab678edefb0c80939e9ec6ad62f8f70af0a8957580f19f811a11a8a0a22891","abstract_canon_sha256":"2e206822891bb75ad3edfac5f675ae2117dd8dc18a8e770fe3e7037c8bcd6d5b"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:39:21.526157Z","signature_b64":"n47+gfqpZdiDc9+ju32LQ7iWVFeMjvzP7VpSuaWdEJ9c4YA5i3ptPcLuWKF8Ymq2tLQPzqVU+EWmMqV4hgSHAg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"b08e1150091bd20d851a1d4e7ed3a6a1b85728467986a54b1264c50b5ba05ea7","last_reissued_at":"2026-05-17T23:39:21.525493Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:39:21.525493Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Titans: Learning to Memorize at Test Time","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Titans combine attention with a learnable neural long-term memory to handle contexts over two million tokens more effectively than Transformers or linear recurrent models.","cross_cats":["cs.AI","cs.CL"],"primary_cat":"cs.LG","authors_text":"Ali Behrouz, Peilin Zhong, Vahab Mirrokni","submitted_at":"2024-12-31T22:32:03Z","abstract_excerpt":"Over more than a decade there has been an extensive research effort on how to effectively utilize recurrent models and attention. While recurrent models aim to compress the data into a fixed-size memory (called hidden state), attention allows attending to the entire context window, capturing the direct dependencies of all tokens. This more accurate modeling of dependencies, however, comes with a quadratic cost, limiting the model to a fixed-length context. We present a new neural long-term memory module that learns to memorize historical context and helps attention to attend to the current con"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models. They further can effectively scale to larger than 2M context window size with higher accuracy in needle-in-haystack tasks compared to baselines.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the neural memory module can reliably learn to store and retrieve relevant historical information without catastrophic forgetting or introducing new failure modes that offset the claimed gains, especially when the training objective does not explicitly supervise the memory contents.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Titans combine attention for current context with a learnable neural memory for long-term history, achieving better performance and scaling to over 2M-token contexts on language, reasoning, genomics, and time-series tasks.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Titans combine attention with a learnable neural long-term memory to handle contexts over two million tokens more effectively than Transformers or linear recurrent models.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"67a82dcc8b8ecbb016e27fde910cc989bf23e7b353be2516a1cbf9b331bbe6ba"},"source":{"id":"2501.00663","kind":"arxiv","version":1},"verdict":{"id":"d0ff25ec-78b9-47b8-b1c5-e015af330588","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T22:03:30.442238Z","strongest_claim":"Our experimental results on language modeling, common-sense reasoning, genomics, and time series tasks show that Titans are more effective than Transformers and recent modern linear recurrent models. They further can effectively scale to larger than 2M context window size with higher accuracy in needle-in-haystack tasks compared to baselines.","one_line_summary":"Titans combine attention for current context with a learnable neural memory for long-term history, achieving better performance and scaling to over 2M-token contexts on language, reasoning, genomics, and time-series tasks.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the neural memory module can reliably learn to store and retrieve relevant historical information without catastrophic forgetting or introducing new failure modes that offset the claimed gains, especially when the training objective does not explicitly supervise the memory contents.","pith_extraction_headline":"Titans combine attention with a learnable neural long-term memory to handle contexts over two million tokens more effectively than Transformers or linear recurrent models."},"references":{"count":139,"sample":[{"doi":"","year":2023,"title":"GPT-4 Technical Report","work_id":"b928e041-6991-4c08-8c81-0359e4097c7b","ref_index":1,"cited_arxiv_id":"2303.08774","is_internal_anchor":true},{"doi":"","year":2024,"title":"Linear Transformers with Learnable Kernel Functions are Better In-Context Models","work_id":"deea05cd-a116-4eb0-8665-d098e04d0402","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2016,"title":"Learning to learn by gradient descent by gradient descent","work_id":"c52938bb-b5b3-447f-a5cd-0f1078f63fa7","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2022,"title":"Exploring length generalization in large language models","work_id":"6b3a6ccc-4c8a-4249-b864-98b23b2b57e7","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2024,"title":"Simple linear attention language models balance the recall-throughput tradeoff","work_id":"e1002884-0294-462b-8c41-f124aa0e9c3f","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":139,"snapshot_sha256":"d89273418ef7806fc5c088b4751c8b4e6d84380f868f825f41aaed887666def8","internal_anchors":24},"formal_canon":{"evidence_count":2,"snapshot_sha256":"0b17f49521d6fff7313e3a679483c52c4f10eeb96057e83477867d6d036cde68"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2501.00663","created_at":"2026-05-17T23:39:21.525589+00:00"},{"alias_kind":"arxiv_version","alias_value":"2501.00663v1","created_at":"2026-05-17T23:39:21.525589+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2501.00663","created_at":"2026-05-17T23:39:21.525589+00:00"},{"alias_kind":"pith_short_12","alias_value":"WCHBCUAJDPJA","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"WCHBCUAJDPJA3BI2","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"WCHBCUAJ","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":45,"internal_anchor_count":45,"sample":[{"citing_arxiv_id":"2605.23559","citing_title":"PathNavigate: A Training-Free Pathology Agent with Surprise-Guided Scan and Shared Slide Memory for Whole-Slide Image VQA","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.23603","citing_title":"Preisach Attention: A Hysteretic Model of Sequential Memory","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2605.23603","citing_title":"Preisach Attention: A Hysteretic Model of Sequential Memory","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2410.13846","citing_title":"LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2602.08686","citing_title":"CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilation","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2510.04800","citing_title":"Hybrid Architectures for Language Models: Systematic Analysis and Design Insights","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2510.26083","citing_title":"Nirvana: A Specialized Generalist Model With Task-Aware Memory Mechanism","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2510.27258","citing_title":"Higher-order Linear Attention","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2511.03092","citing_title":"SnapStream: Efficient Long Sequence Decoding on Dataflow Accelerators","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2511.07328","citing_title":"Q-RAG: Long Context Multi-step Retrieval via Value-based Embedder Training","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2511.14823","citing_title":"Dynamic Nested Hierarchies: Pioneering Self-Evolution in Machine Learning Architectures for Lifelong Intelligence","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2504.15965","citing_title":"From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs","ref_index":152,"is_internal_anchor":true},{"citing_arxiv_id":"2509.26645","citing_title":"TTT3R: 3D Reconstruction as Test-Time Training","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2512.01643","citing_title":"ViT$^3$: Unlocking Test-Time Training in Vision","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2601.06803","citing_title":"Forest Before Trees: Latent Superposition for Efficient Visual Reasoning","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2601.14724","citing_title":"HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2505.23884","citing_title":"Test-Time Training Done Right","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2601.22766","citing_title":"Sparse Attention as Compact Kernel Regression","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2602.01219","citing_title":"Mixture-of-Top-k Attention: Efficient Attention via Scalable Fast Weights","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2602.21204","citing_title":"Test-Time Training with KV Binding Is Secretly Linear Attention","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2603.23516","citing_title":"MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03263","citing_title":"LPC-SM: Local Predictive Coding and Sparse Memory for Long-Context Language Modeling","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2507.02259","citing_title":"MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2507.03724","citing_title":"MemOS: A Memory OS for AI System","ref_index":76,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12770","citing_title":"WriteSAE: Sparse Autoencoders for Recurrent State","ref_index":36,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG","json":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG.json","graph_json":"https://pith.science/api/pith-number/WCHBCUAJDPJA3BI2DVHH5U5GUG/graph.json","events_json":"https://pith.science/api/pith-number/WCHBCUAJDPJA3BI2DVHH5U5GUG/events.json","paper":"https://pith.science/paper/WCHBCUAJ"},"agent_actions":{"view_html":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG","download_json":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG.json","view_paper":"https://pith.science/paper/WCHBCUAJ","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2501.00663&json=true","fetch_graph":"https://pith.science/api/pith-number/WCHBCUAJDPJA3BI2DVHH5U5GUG/graph.json","fetch_events":"https://pith.science/api/pith-number/WCHBCUAJDPJA3BI2DVHH5U5GUG/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG/action/timestamp_anchor","attest_storage":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG/action/storage_attestation","attest_author":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG/action/author_attestation","sign_citation":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG/action/citation_signature","submit_replication":"https://pith.science/pith/WCHBCUAJDPJA3BI2DVHH5U5GUG/action/replication_record"}},"created_at":"2026-05-17T23:39:21.525589+00:00","updated_at":"2026-05-17T23:39:21.525589+00:00"}