{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:B7KD4YH37PEWBU7OB26KIDX3WY","short_pith_number":"pith:B7KD4YH3","schema_version":"1.0","canonical_sha256":"0fd43e60fbfbc960d3ee0ebca40efbb60b9ec47ea7bf337cdada864afbe7879b","source":{"kind":"arxiv","id":"2404.05892","version":4},"attestation_state":"computed","paper":{"title":"Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Matrix-valued states and dynamic recurrence let updated RWKV models reach competitive benchmark performance while keeping RNN inference speed.","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Alon Albalak, Atsushi Saito, Bart{\\l}omiej Koptyra, Bingchen Zhao, Bo Peng, Cahya Wirawan, Daniel Goldstein, Eric Alcaide, Eugene Cheah, Fares Obeid, Guangyu Song, Haoqin Tu, Haowen Hou, Jan Koco\\'n, Jiaju Lin, Jian Zhu, Kranthi Kiran GV, Niklas Muennighoff, Peng Zhou, Przemys{\\l}aw Kazienko, Qihang Zhao, Quentin Anthony, Ronald McClelland Jr., Ruichong Zhang, Rui-Jie Zhu, Satyapriya Krishna, Stanis{\\l}aw Wo\\'zniak, Stella Biderman, Teddy Ferdinan, Xingjian Du","submitted_at":"2024-04-08T22:20:59Z","abstract_excerpt":"We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-valued states and a dynamic recurrence mechanism that improve expressivity while maintaining the inference efficiency characteristics of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a fast tokenizer based on greedy matching for enhanced multilinguality. We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2404.05892","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2024-04-08T22:20:59Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"2f2257ddd3304956e8d5d6057363a5d9905c3bc11b60c4a7be995006f13926b2","abstract_canon_sha256":"64fa49491abe0152dd19bd1c90a34ac52150a7c69a047b9e9385d85718405466"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:12.646907Z","signature_b64":"A06I9beSPi/I3yJUGoNu1dukBxZna0c8SZbxd1iR0VPxTWjJTCtrpkPdRcoVCO6iLojAHh383udEaHz6FgbxDQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"0fd43e60fbfbc960d3ee0ebca40efbb60b9ec47ea7bf337cdada864afbe7879b","last_reissued_at":"2026-05-17T23:38:12.646019Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:12.646019Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Matrix-valued states and dynamic recurrence let updated RWKV models reach competitive benchmark performance while keeping RNN inference speed.","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Alon Albalak, Atsushi Saito, Bart{\\l}omiej Koptyra, Bingchen Zhao, Bo Peng, Cahya Wirawan, Daniel Goldstein, Eric Alcaide, Eugene Cheah, Fares Obeid, Guangyu Song, Haoqin Tu, Haowen Hou, Jan Koco\\'n, Jiaju Lin, Jian Zhu, Kranthi Kiran GV, Niklas Muennighoff, Peng Zhou, Przemys{\\l}aw Kazienko, Qihang Zhao, Quentin Anthony, Ronald McClelland Jr., Ruichong Zhang, Rui-Jie Zhu, Satyapriya Krishna, Stanis{\\l}aw Wo\\'zniak, Stella Biderman, Teddy Ferdinan, Xingjian Du","submitted_at":"2024-04-08T22:20:59Z","abstract_excerpt":"We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon the RWKV (RWKV-4) architecture. Our architectural design advancements include multi-headed matrix-valued states and a dynamic recurrence mechanism that improve expressivity while maintaining the inference efficiency characteristics of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a fast tokenizer based on greedy matching for enhanced multilinguality. We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find that they achieve competitive performance across a wide variety of benchmarks.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That observed benchmark performance stems primarily from the matrix-valued states and dynamic recurrence rather than from the scale of the new 1.12-trillion-token corpus, tokenizer changes, or other unablated training choices.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Eagle and Finch enhance RWKV with matrix-valued states and dynamic recurrence, trained on a 1.12-trillion-token multilingual corpus, and report competitive performance on standard benchmarks.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Matrix-valued states and dynamic recurrence let updated RWKV models reach competitive benchmark performance while keeping RNN inference speed.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"81661ab7769ee6eb92e6b3f818317ec9832e26ad2f73b62a247decb12c28bd36"},"source":{"id":"2404.05892","kind":"arxiv","version":4},"verdict":{"id":"78cd7ba4-6180-4e93-a528-0e12037a08d8","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-17T23:15:46.657659Z","strongest_claim":"We trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two Finch models with 1.6 and 3.1 billion parameters and find that they achieve competitive performance across a wide variety of benchmarks.","one_line_summary":"Eagle and Finch enhance RWKV with matrix-valued states and dynamic recurrence, trained on a 1.12-trillion-token multilingual corpus, and report competitive performance on standard benchmarks.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That observed benchmark performance stems primarily from the matrix-valued states and dynamic recurrence rather than from the scale of the new 1.12-trillion-token corpus, tokenizer changes, or other unablated training choices.","pith_extraction_headline":"Matrix-valued states and dynamic recurrence let updated RWKV models reach competitive benchmark performance while keeping RNN inference speed."},"references":{"count":59,"sample":[{"doi":"10.18653/v1/2022.bigscience-1.9","year":2022,"title":"Generating Long Sequences with Sparse Transformers","work_id":"c5b81688-45ee-4a9a-b095-e6290f45cb6c","ref_index":1,"cited_arxiv_id":"1904.10509","is_internal_anchor":true},{"doi":"10.18653/v1/2022.findings-naacl.55","year":2021,"title":"Teddy Ferdinan, Jan Koco ´ n, and Przemysław Kazienko","work_id":"1b1857e4-3fda-4d47-b1ed-acfd838e2edd","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"10.1162/neco.1992.4.1.131","year":2024,"title":"Zhen Qin, Dong Li, Weigao Sun, Weixuan Sun, Xuyang Shen, Xiaodong Han, Yunshen Wei, Baohong Lv, Xiao Luo, Yu Qiao, and Yiran Zhong","work_id":"4d506627-035c-4c55-8ce2-e6caf494d405","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2018,"title":"Group normalization","work_id":"19a9604a-9e6c-4e87-a353-712030317c6d","ref_index":4,"cited_arxiv_id":"1803.08494","is_internal_anchor":true},{"doi":"","year":2023,"title":"The most recent single-timestep input to the Time-mixing module, denoted as xt −1 ∈ RD , useful for the Token Shift. 33 Dataset Domain Wikipediaa Encyclopedia SlimPajama Web peS2o Academia BigPatent P","work_id":"d52ea90d-be93-4f45-a81d-aebdbabfb3ec","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":59,"snapshot_sha256":"c834c169d7b4041cd0ae7ca34e1cb8b735902fb5ab1d2c44d00a6c25ac93e5de","internal_anchors":2},"formal_canon":{"evidence_count":3,"snapshot_sha256":"4b4e203967ce3084064ec2791dfb5432250202e2313f47854ff2d16567529554"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2404.05892","created_at":"2026-05-17T23:38:12.646135+00:00"},{"alias_kind":"arxiv_version","alias_value":"2404.05892v4","created_at":"2026-05-17T23:38:12.646135+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2404.05892","created_at":"2026-05-17T23:38:12.646135+00:00"},{"alias_kind":"pith_short_12","alias_value":"B7KD4YH37PEW","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_16","alias_value":"B7KD4YH37PEWBU7O","created_at":"2026-05-18T12:33:37.589309+00:00"},{"alias_kind":"pith_short_8","alias_value":"B7KD4YH3","created_at":"2026-05-18T12:33:37.589309+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":20,"internal_anchor_count":20,"sample":[{"citing_arxiv_id":"2605.19049","citing_title":"KVBuffer: IO-aware Serving for Linear Attention","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2509.22630","citing_title":"StateX: Enhancing RNN Recall via Post-training State Expansion","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2510.27258","citing_title":"Higher-order Linear Attention","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2406.11794","citing_title":"DataComp-LM: In search of the next generation of training sets for language models","ref_index":142,"is_internal_anchor":true},{"citing_arxiv_id":"2405.14782","citing_title":"Lessons from the Trenches on Reproducible Evaluation of Language Models","ref_index":221,"is_internal_anchor":true},{"citing_arxiv_id":"2502.13189","citing_title":"MoBA: Mixture of Block Attention for Long-Context LLMs","ref_index":58,"is_internal_anchor":true},{"citing_arxiv_id":"2602.21204","citing_title":"Test-Time Training with KV Binding Is Secretly Linear Attention","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2407.04620","citing_title":"Learning to (Learn at Test Time): RNNs with Expressive Hidden States","ref_index":59,"is_internal_anchor":true},{"citing_arxiv_id":"2312.06635","citing_title":"Gated Linear Attention Transformers with Hardware-Efficient Training","ref_index":64,"is_internal_anchor":true},{"citing_arxiv_id":"2603.23032","citing_title":"Generative Event Pretraining with Foundation Model Alignment","ref_index":41,"is_internal_anchor":true},{"citing_arxiv_id":"2501.00663","citing_title":"Titans: Learning to Memorize at Test Time","ref_index":86,"is_internal_anchor":true},{"citing_arxiv_id":"2604.09671","citing_title":"Belief-State RWKV for Reinforcement Learning under Partial Observability","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2506.13585","citing_title":"MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2605.05066","citing_title":"The Impossibility Triangle of Long-Context Modeling","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2605.05838","citing_title":"MDN: Parallelizing Stepwise Momentum for Delta Linear Attention","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2405.21060","citing_title":"Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality","ref_index":72,"is_internal_anchor":true},{"citing_arxiv_id":"2604.19826","citing_title":"Co-Located Tests, Better AI Code: How Test Syntax Structure Affects Foundation Model Code Generation","ref_index":26,"is_internal_anchor":true},{"citing_arxiv_id":"2604.10946","citing_title":"Learning to Adapt: In-Context Learning Beyond Stationarity","ref_index":34,"is_internal_anchor":true},{"citing_arxiv_id":"2604.07658","citing_title":"Optimal Decay Spectra for Linear Recurrences","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2604.05030","citing_title":"Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space","ref_index":105,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":3,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY","json":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY.json","graph_json":"https://pith.science/api/pith-number/B7KD4YH37PEWBU7OB26KIDX3WY/graph.json","events_json":"https://pith.science/api/pith-number/B7KD4YH37PEWBU7OB26KIDX3WY/events.json","paper":"https://pith.science/paper/B7KD4YH3"},"agent_actions":{"view_html":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY","download_json":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY.json","view_paper":"https://pith.science/paper/B7KD4YH3","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2404.05892&json=true","fetch_graph":"https://pith.science/api/pith-number/B7KD4YH37PEWBU7OB26KIDX3WY/graph.json","fetch_events":"https://pith.science/api/pith-number/B7KD4YH37PEWBU7OB26KIDX3WY/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY/action/timestamp_anchor","attest_storage":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY/action/storage_attestation","attest_author":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY/action/author_attestation","sign_citation":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY/action/citation_signature","submit_replication":"https://pith.science/pith/B7KD4YH37PEWBU7OB26KIDX3WY/action/replication_record"}},"created_at":"2026-05-17T23:38:12.646135+00:00","updated_at":"2026-05-17T23:38:12.646135+00:00"}