{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:6I5VPMOKRLCGX3VCK67XQAXUYY","short_pith_number":"pith:6I5VPMOK","schema_version":"1.0","canonical_sha256":"f23b57b1ca8ac46beea257bf7802f4c6201855efd791fbade1d9f7e1e7321635","source":{"kind":"arxiv","id":"2412.17739","version":4},"attestation_state":"computed","paper":{"title":"Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.CL"],"primary_cat":"cs.AI","authors_text":"Biqing Qi, Bowen Zhou, Che Jiang, Ermo Hua, Kaiyan Zhang, Ning Ding, Xingtai Lv, Xuekai Zhu, Youbang Sun, Yuchen Fan","submitted_at":"2024-12-23T17:44:01Z","abstract_excerpt":"Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within attention, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs. Using Discrete Signal Processing theory, we show that RoPE enables periodic attention by implicitly achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions; 2) insufficiently trained frequency components b"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2412.17739","kind":"arxiv","version":4},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.AI","submitted_at":"2024-12-23T17:44:01Z","cross_cats_sorted":["cs.CL"],"title_canon_sha256":"fcc63b11546079b4d466f36c0b8fcd4a091caa842316b60bfb0e9c3b1ec3423a","abstract_canon_sha256":"0c632b26fd681f536b10fb9ba9bf9c5dc145b6782fcce6873e35106e39045f42"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T11:36:26.485336Z","signature_b64":"xUx8ntw9gnrM2MCn4Ag0kS1rkEbQoxVuU7IAl+xMVZcPjxF+lUhNb3Gagjw1hQr1lrqNWRKgYtQD/ngmtggZAg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"f23b57b1ca8ac46beea257bf7802f4c6201855efd791fbade1d9f7e1e7321635","last_reissued_at":"2026-07-05T11:36:26.484814Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T11:36:26.484814Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.CL"],"primary_cat":"cs.AI","authors_text":"Biqing Qi, Bowen Zhou, Che Jiang, Ermo Hua, Kaiyan Zhang, Ning Ding, Xingtai Lv, Xuekai Zhu, Youbang Sun, Yuchen Fan","submitted_at":"2024-12-23T17:44:01Z","abstract_excerpt":"Extending the context length of Language Models (LMs) by improving Rotary Position Embedding (RoPE) has become a trend. While prior works mainly address RoPE's limitations within attention, this paper uncovers the adverse effects on length generalization from nearly all parts of LMs. Using Discrete Signal Processing theory, we show that RoPE enables periodic attention by implicitly achieving Non-Uniform Discrete Fourier Transform. However, this periodicity is undermined by the spectrum damage caused by: 1) linear layers and activation functions; 2) insufficiently trained frequency components b"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2412.17739","kind":"arxiv","version":4},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2412.17739/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2412.17739","created_at":"2026-07-05T11:36:26.484880+00:00"},{"alias_kind":"arxiv_version","alias_value":"2412.17739v4","created_at":"2026-07-05T11:36:26.484880+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2412.17739","created_at":"2026-07-05T11:36:26.484880+00:00"},{"alias_kind":"pith_short_12","alias_value":"6I5VPMOKRLCG","created_at":"2026-07-05T11:36:26.484880+00:00"},{"alias_kind":"pith_short_16","alias_value":"6I5VPMOKRLCGX3VC","created_at":"2026-07-05T11:36:26.484880+00:00"},{"alias_kind":"pith_short_8","alias_value":"6I5VPMOK","created_at":"2026-07-05T11:36:26.484880+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":7,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2605.24681","citing_title":"Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs","ref_index":43,"is_internal_anchor":false},{"citing_arxiv_id":"2606.04302","citing_title":"LazyAttention: Efficient Retrieval-Augmented Generation with Deferred Positional Encoding","ref_index":69,"is_internal_anchor":false},{"citing_arxiv_id":"2605.23719","citing_title":"Weierstrass Positional Encoding for Vision Transformers","ref_index":7,"is_internal_anchor":false},{"citing_arxiv_id":"2510.22102","citing_title":"Mitigating Coordinate Prediction Bias from Positional Encoding Failures","ref_index":10,"is_internal_anchor":false},{"citing_arxiv_id":"2603.11680","citing_title":"UCAN: Unified Convolutional Attention Network for Expansive Receptive Fields in Lightweight Super-Resolution","ref_index":11,"is_internal_anchor":false},{"citing_arxiv_id":"2510.26692","citing_title":"Kimi Linear: An Expressive, Efficient Attention Architecture","ref_index":42,"is_internal_anchor":false},{"citing_arxiv_id":"2604.20817","citing_title":"Convergent Evolution: How Different Language Models Learn Similar Number Representations","ref_index":7,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY","json":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY.json","graph_json":"https://pith.science/api/pith-number/6I5VPMOKRLCGX3VCK67XQAXUYY/graph.json","events_json":"https://pith.science/api/pith-number/6I5VPMOKRLCGX3VCK67XQAXUYY/events.json","paper":"https://pith.science/paper/6I5VPMOK"},"agent_actions":{"view_html":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY","download_json":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY.json","view_paper":"https://pith.science/paper/6I5VPMOK","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2412.17739&json=true","fetch_graph":"https://pith.science/api/pith-number/6I5VPMOKRLCGX3VCK67XQAXUYY/graph.json","fetch_events":"https://pith.science/api/pith-number/6I5VPMOKRLCGX3VCK67XQAXUYY/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY/action/timestamp_anchor","attest_storage":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY/action/storage_attestation","attest_author":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY/action/author_attestation","sign_citation":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY/action/citation_signature","submit_replication":"https://pith.science/pith/6I5VPMOKRLCGX3VCK67XQAXUYY/action/replication_record"}},"created_at":"2026-07-05T11:36:26.484880+00:00","updated_at":"2026-07-05T11:36:26.484880+00:00"}