{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2026:D427NFWTIR5VANRDQYPIG5URD2","short_pith_number":"pith:D427NFWT","schema_version":"1.0","canonical_sha256":"1f35f696d3447b503623861e8376911e9b2e824c19e0253b90b9c7ec9db463a4","source":{"kind":"arxiv","id":"2603.15031","version":1},"attestation_state":"computed","paper":{"title":"Attention Residuals","license":"http://creativecommons.org/licenses/by-nc-nd/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bohong Yin, Chao Hong, Enzhe Lu, Fanqing Meng, Guanduo Chen, Guokun Lai, Haiqing Guo, Haoyu Lu, Jianlin Su, Jinguo Zhu, Junjie Yan, Kimi Team: Guangyu Chen, Lin Sui, Ming Wei, Shaowei Liu, Siyuan Pan, Weiran He, Weixiao Huang, Weixin Xu, Xiaotong Xie, Xin Men, Xinran Xu, Xinyu Zhou, Yanru Chen, Yaoyu Wang, Y. Charles, Yucheng Wang, Yulun Du, Yunpeng Tai, Yutian Chen, Yuxin Wu, Yu Zhang, Yuzhi Wang, Y. Zhang, Zaida Zhou, Zhilin Yang","submitted_at":"2026-03-16T09:32:21Z","abstract_excerpt":"Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2603.15031","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by-nc-nd/4.0/","primary_cat":"cs.CL","submitted_at":"2026-03-16T09:32:21Z","cross_cats_sorted":[],"title_canon_sha256":"3a4952b6e6649bc244b4b5d32faf5e8c8a0172792b7af6d2f252f7c7ffd2da7a","abstract_canon_sha256":"cc11a89c0db75b3580840775cb9fb52aec66ab52f3b186b51245daa13fbf7a81"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-21T06:32:11.552152Z","signature_b64":"zVGXC/A2CctSyoBqj9qx0jUhrtigWZOhX4NzkV6kwRtUEudtfF4ijmrjJzcekKrrGdxZO5XOTMPby9QQPYBkBQ==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"1f35f696d3447b503623861e8376911e9b2e824c19e0253b90b9c7ec9db463a4","last_reissued_at":"2026-05-21T06:32:11.549586Z","signature_status":"signed_v1","first_computed_at":"2026-05-21T06:32:11.549586Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Attention Residuals","license":"http://creativecommons.org/licenses/by-nc-nd/4.0/","headline":"","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Bohong Yin, Chao Hong, Enzhe Lu, Fanqing Meng, Guanduo Chen, Guokun Lai, Haiqing Guo, Haoyu Lu, Jianlin Su, Jinguo Zhu, Junjie Yan, Kimi Team: Guangyu Chen, Lin Sui, Ming Wei, Shaowei Liu, Siyuan Pan, Weiran He, Weixiao Huang, Weixin Xu, Xiaotong Xie, Xin Men, Xinran Xu, Xinyu Zhou, Yanru Chen, Yaoyu Wang, Y. Charles, Yucheng Wang, Yulun Du, Yunpeng Tai, Yutian Chen, Yuxin Wu, Yu Zhang, Yuzhi Wang, Y. Zhang, Zaida Zhou, Zhilin Yang","submitted_at":"2026-03-16T09:32:21Z","abstract_excerpt":"Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2603.15031","kind":"arxiv","version":1},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2603.15031/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2603.15031","created_at":"2026-05-21T06:32:11.549690+00:00"},{"alias_kind":"arxiv_version","alias_value":"2603.15031v1","created_at":"2026-05-21T06:32:11.549690+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2603.15031","created_at":"2026-05-21T06:32:11.549690+00:00"},{"alias_kind":"pith_short_12","alias_value":"D427NFWTIR5V","created_at":"2026-05-21T06:32:11.549690+00:00"},{"alias_kind":"pith_short_16","alias_value":"D427NFWTIR5VANRD","created_at":"2026-05-21T06:32:11.549690+00:00"},{"alias_kind":"pith_short_8","alias_value":"D427NFWT","created_at":"2026-05-21T06:32:11.549690+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":29,"internal_anchor_count":29,"sample":[{"citing_arxiv_id":"2605.23259","citing_title":"Multi-Gate Residuals","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12374","citing_title":"Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20798","citing_title":"Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2605.20708","citing_title":"Rethinking Cross-Layer Information Routing in Diffusion Transformers","ref_index":53,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06317","citing_title":"NavOne: One-Step Global Planning for Vision-Language Navigation on Top-Down Maps","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06501","citing_title":"Cubit: Token Mixer with Kernel Ridge Regression","ref_index":77,"is_internal_anchor":true},{"citing_arxiv_id":"2605.18855","citing_title":"Delta Attention Residuals","ref_index":1,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17887","citing_title":"Attention Sinks and Outliers in Attention Residuals","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12374","citing_title":"Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2605.15514","citing_title":"RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably","ref_index":40,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03297","citing_title":"XAttnRes: Cross-Stage Attention Residuals for Medical Image Segmentation","ref_index":30,"is_internal_anchor":true},{"citing_arxiv_id":"2604.03190","citing_title":"Gradient Boosting within a Single Attention Layer","ref_index":3,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11011","citing_title":"LoopUS: Recasting Pretrained LLMs into Looped Latent Refinement Models","ref_index":72,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12374","citing_title":"Fill the GAP: A Granular Alignment Paradigm for Visual Reasoning in Multimodal Large Language Models","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08806","citing_title":"L2A: Learning to Accumulate Pose History for Accurate 3D Human Pose Estimation","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08423","citing_title":"Queryable LoRA: Instruction-Regularized Routing Over Shared Low-Rank Update Atoms","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08806","citing_title":"L2A: Learning to Accumulate Pose History for Accurate 3D Human Pose Estimation","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2605.08234","citing_title":"When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09196","citing_title":"RigidFormer: Learning Rigid Dynamics using Transformers","ref_index":20,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09992","citing_title":"Attention Drift: What Autoregressive Speculative Decoding Models Learn","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2605.03953","citing_title":"Transformers with Selective Access to Early Representations","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06501","citing_title":"Cubit: Token Mixer with Kernel Ridge Regression","ref_index":77,"is_internal_anchor":true},{"citing_arxiv_id":"2605.06317","citing_title":"NavOne: One-Step Global Planning for Vision-Language Navigation on Top-Down Maps","ref_index":35,"is_internal_anchor":true},{"citing_arxiv_id":"2605.05394","citing_title":"BARFI-Q: Quantum-Enhanced Block Attention Residual Fusion Framework for Multivariate Time-Series Forecasting in Atom Interferometry","ref_index":38,"is_internal_anchor":true},{"citing_arxiv_id":"2605.01376","citing_title":"A Cellular Doctrine of Morality: Intrinsic Active Precision and the Mind-Reality Overload Dilemma","ref_index":38,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2","json":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2.json","graph_json":"https://pith.science/api/pith-number/D427NFWTIR5VANRDQYPIG5URD2/graph.json","events_json":"https://pith.science/api/pith-number/D427NFWTIR5VANRDQYPIG5URD2/events.json","paper":"https://pith.science/paper/D427NFWT"},"agent_actions":{"view_html":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2","download_json":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2.json","view_paper":"https://pith.science/paper/D427NFWT","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2603.15031&json=true","fetch_graph":"https://pith.science/api/pith-number/D427NFWTIR5VANRDQYPIG5URD2/graph.json","fetch_events":"https://pith.science/api/pith-number/D427NFWTIR5VANRDQYPIG5URD2/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2/action/timestamp_anchor","attest_storage":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2/action/storage_attestation","attest_author":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2/action/author_attestation","sign_citation":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2/action/citation_signature","submit_replication":"https://pith.science/pith/D427NFWTIR5VANRDQYPIG5URD2/action/replication_record"}},"created_at":"2026-05-21T06:32:11.549690+00:00","updated_at":"2026-05-21T06:32:11.549690+00:00"}