{"id":"9b8532a9-7ea4-4b4f-a141-d7e9d58d1248","arxiv_id":"2412.12465","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CCA-Attention pools token groups into core tokens and adds a local window to replace self-attention, claiming long-context speedups with minimal fine-tuning.","lead":"A new attention module for large language models compresses chunks of context into core tokens and adds a local window, aiming to cut computation for very long inputs. The authors report up to 7.9x faster prefill and about 45% less memory, but the claimed linear complexity is not supported by their own formulas, and the method often scores below standard attention.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'linear complexity' claim is internally inconsistent with the paper's own derivation: with fixed group size g, global attention costs O(L^2/g), which is still quadratic in L, so CCA-Attention does not achieve the claimed linear scaling.","rationale":"I read the central claim as a plug-and-play efficiency replacement: minimal fine-tuning, linear complexity, reduced KV cache, superior long-context modeling, and a 7.9x speedup. The weakest point in that chain is the complexity-class claim, because it is contradicted by the paper's own equations rather than by external benchmark variance. The reader's weakest_assumption (last-token attention as an importance signal) is a legitimate effectiveness risk, but the ablations in Tables 7-9 partially address it, and even an imperfect pooling rule could be adapted during fine-tuning; the complexity contradiction has no such remedy. The performance evidence is also weaker than the abstract suggests: in Tables 1 and 2, CCA's average LongBench-E score is below the vanilla self-attention row in several configurations (for example, 21.86 vs. 22.11 for LLaMA2-7B-32K and 37.81 vs. 37.93 for LLaMA3.1-8B-128K), and the baselines are largely training-free while CCA is fine-tuned, so 'superior long-context modeling' needs qualification. Those issues reinforce a conditional verdict but are addressable with fairer comparisons and corrected claims. I therefore keep the reader's CONDITIONAL verdict rather than moving to REJECT, because the method still provides a real constant-factor speedup and competitive accuracy in several settings; the mandatory revision is to correct the complexity claim and qualify the performance comparisons.","tokens_in":24852,"tokens_out":8718,"duration_ms":80628,"concrete_test":"Use Algorithm 1's structure to compute prefill FLOPs for g = 16 and s = 1024: the global module performs roughly 2 * L * floor(L/g) * d multiply-adds. Evaluate at L = 32K, 64K, 128K, and 256K, then fit log(FLOPs) versus log(L); a slope near 2 confirms O(L^2/g) scaling and falsifies the linear-complexity claim. As a complementary check, instrument the released Triton kernel at these lengths and compare measured scaling; if the measured slope is approximately 1, the concern would be resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.6 reports O(Lm + Ls) for fixed-length sequences and O(L^2/g + Ls) for autoregressive generation, while m = floor(L/g) is defined in Eqn. (2). The abstract and Section 1 claim the method 'reduces computational complexity to linear.' These claims are incompatible unless g grows with L. In the implemented regime g is fixed (Table 8 sweeps g = 2..64 and uses g = 16), so m = Theta(L) and the globality-aware pooling module alone costs Theta(L^2/g) multiply-adds in the prefill pass: each of the L queries attends to the m core keys. For any fixed g, Theta(L^2/g) is still Theta(L^2), a quadratic complexity with a reduced constant, not linear. The locality module adds O(Ls), which is linear in L for fixed s. The KV-cache claim O(L/g + s) is also Theta(L) for fixed g, so it is a constant-factor memory reduction, not a new asymptotic class. The only way to obtain O(L) is to treat m as a constant independent of L, which contradicts Eqn. (2) and the experimental fixed-g setting. Because the paper's headline efficiency contribution is specifically 'linear complexity,' this internal inconsistency is load-bearing for the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Core Context Aware (CCA) Attention, a plug-and-play replacement for the self-attention module in pretrained LLMs. The mechanism partitions the input into groups of size g, compresses each group into a single 'core' token using an attention-weighted pooling driven by the group's last token, and then computes attention over the resulting core tokens while also attending to a local window of preceding tokens. The authors claim linear-time complexity, reduced KV cache, minimal fine-tuning, and superior long-context modeling, reporting up to a 7.9x first-token latency speedup at 128K context on LLaMA2-7B-80K. Experiments cover LongBench-E, RULER, multi-document EM, MMLU, and models LLaMA2, LLaMA3.1, and Qwen2.5.","tokens_in":25088,"tokens_out":13539,"duration_ms":109555,"significance":"If the stated complexity and accuracy claims held, the method would be a practical low-cost retrofit for long-context inference. The paper's strengths include a simple architecture, public source code, and evaluation across several benchmarks and model families. However, the central 'linear complexity' claim is internally inconsistent with the paper's own complexity analysis, and the aggregate accuracy results do not support 'superior' long-context modeling over vanilla self-attention. The contribution is best viewed as a subquadratic approximation with large constant-factor speedups, pending corrective revisions.","major_comments":[{"comment":"The abstract and Section 1 state that CCA-Attention 'reduces computational complexity to linear complexity,' but Section 3.6 derives O(L^2/g + Ls) for autoregressive decoding when the group size g is a constant, and the experiments use g=16 (Table 8). Because m = floor(L/g) in Eqn. (2), the global-attention term is Theta(L^2/g), which is quadratic for any fixed g; the 7.9x speedup reported in Table 3 is a constant-factor reduction rather than a change of asymptotic class. The fixed-length case O(Lm + Ls) is only linear if m is treated as a constant, which contradicts the definition of m and the implemented setting. The efficiency claims should be restated for two regimes: fixed g (subquadratic, O(L^2/g + Ls)) and fixed m (linear, O(Lm + Ls)), with the latter requiring g to grow with L and a separate validation.","section":"Section 3.6, Eqn. (2), Table 8, Abstract"},{"comment":"The abstract claims 'superiority of our method in both long-context modeling and computational efficiency,' and Section 4.2 claims CCA-LLM 'attains the highest average score' on LongBench-E, but the tables show the opposite. In Table 1, the CCA-LLM average is 21.86 vs 22.11 for vanilla self-attention on LLaMA2-7B-32K and 22.24 vs 22.42 on LLaMA2-7B-80K; in Table 3 the average EM score at 4K-128K is 34.4 vs 36.0 for vanilla attention. The specific claim in Section 4.2 that 'our CCA-LLM consistently achieves the highest EM score' for short contexts is also contradicted at 4K (39.3 vs 39.4). The modeling-superiority claim should be replaced by a competitive-accuracy claim, or supported by significance tests at the lengths (e.g., 64K/128K) where the paper claims improvement.","section":"Tables 1 and 3, Section 4.2, Abstract"},{"comment":"The globality-aware pooling in Eqn. (2) uses the last token of each group as the query to assign importance weights to the group's tokens. This choice is load-bearing because the pooled core tokens are the only global information channel for all later queries; an unreliable importance measure would discard context needed downstream. The paper motivates this choice solely with attention-map visualizations on a single 32-token sentence (Section C.4). The authors should either quantify how well the last token's attention predicts the attention of later positions (e.g., by computing the correlation across a sample of long documents) or add an ablation that varies the query token (first, middle, group-average, or a learned query), since the existing Table 7 ablation varies only the pooling strategy, not the query source.","section":"Section C.4, Eqn. (2), Table 7"}],"minor_comments":[{"comment":"Section 4.2 contains several typos: 'ur CCA-LLM' should read 'Our CCA-LLM,' 'MInferencce' should read 'MInference,' and 'accerlerate' should read 'accelerate.'","section":"Section 4.2"},{"comment":"Section 3.6 states that the locality-preserving module's upper bound is O(L(s+g)) but then reports the total as O(Lm+Ls); the g-dependent term should be included or explicitly absorbed, for example as O(Lm + L(s+g)).","section":"Section 3.6"},{"comment":"The Table 3 caption says 'FTL denotes the latency to generate the first token,' but this is the prefill latency, not the per-token generation latency; labeling it 'prefill latency' or 'time to first token' would avoid confusion with the inter-token latency reported elsewhere.","section":"Table 3 caption"},{"comment":"The sentence in Section 4.2 claiming that 'our CCA-LLM consistently achieves the highest EM score' for short contexts is not supported by Table 3, where CCA-LLM scores 39.3 versus 39.4 for vanilla self-attention at 4K.","section":"Section 4.2"},{"comment":"Calling the fine-tuning costs 'minimal' may be misleading, as Section B.2 reports 2.1B tokens for 32K and 5B tokens for 80K; please clarify the comparison baseline or rephrase the claim.","section":"Section 3.5"},{"comment":"The phrase 'tokens in close proximity to the query Qi are likely to be more relevant' should specify 'preceding tokens within the local window,' since causal masking prevents attending to future tokens.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The complexity inconsistency is the main scientific concern; if the authors cannot substantiate linear complexity for the implemented regime, the paper should be reframed as a subquadratic efficient-attention method. The manuscript appears to be a camera-ready ICML publication; the journal version should resolve these issues regardless of prior acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look: the core idea is a practical attention replacement that gives real speedups, but the paper's headline 'linear complexity' claim is flatly contradicted by its own complexity analysis, and its main comparisons are against training-free baselines while the method itself is fine-tuned. Read it for the method and the useful ablations, not for the asymptotics as written.\n\nThe new thing here is the way CCA-Attention builds one 'core token' per group via attention-weighted pooling, using the group's last token as the query, and then fuses those core tokens with a sliding local window in a single attention computation. That combination is not in the cited LongLoRA, StreamingLLM, or MInference line of work, and the reachability property in Proposition 1 is a clean formal guarantee that all earlier tokens still get positive weight. The practical measurements look credible: 7.9x prefill speedup and roughly 45% memory reduction at 128K are the kind of numbers you can get from this architecture, and the ablations on group size g and window size s are informative. Code is public, which is a plus.\n\nThe soft spots are real, though. Most importantly, the linear complexity claim fails on the paper's own terms. For fixed g, which is the implemented regime (g=16 default, swept 2..64), the global module computes L queries against m=L/g core keys, so it costs O(L^2/g) multiply-adds -- still quadratic. Section 3.6 derives exactly this O(L^2/g + Ls) for autoregressive generation, then the abstract and introduction say 'linear.' You can get O(L) only if m is constant independent of L, which contradicts Eq. (2) and the experiments. The 7.9x is a real constant-factor reduction, but it is not a change in asymptotic complexity class.\n\nAlso: StreamingLLM, LM-Infinite, and MInference are training-free; CCA-LLM is fine-tuned for 1000 steps on 2-5B tokens. That is not an even comparison. The paper has one training-based comparison (LongLoRA) buried in the supplement, and there CCA wins, but one setting is thin. And the 'superior performance' claim is not fully supported by the main table: average LongBench-E is below vanilla self-attention for both 32K and 80K (21.86 vs 22.11; 22.24 vs 22.42), with summarization dropping from 12.43 to 7.79 at 32K. The method only clearly beats vanilla at very long EM contexts (64K/128K).\n\nThis paper is for people working on efficient attention, long-context serving, or context compression. It deserves a serious referee: the idea is plausible, the practical gains are real, and the main flaws are fixable (correct the complexity claim, add training-based baselines, report variance). But as written, the efficiency claims overstate the contribution. I would engage, and I'd ask for a revision before accepting.","headline":"Useful attention-replacement idea with real speedups, but the 'linear complexity' claim is wrong as stated and the comparison protocol is unfair; still worth a referee's time.","tokens_in":25689,"tokens_out":4412,"would_cite":false,"duration_ms":38023,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Core Context Aware Attention replaces self-attention in pretrained LLMs with a plug-and-play two-branch module — group-wise pooling into core tokens plus a local window — reducing complexity to linear, KV cache to O(L/g + s), and…","keywords":["core context aware attention","long context language modeling","efficient attention","KV cache compression","globality-aware pooling","locality-preserving attention","plug-and-play LLM adaptation","linear attention complexity"],"falsifier":"Build a multi-document question-answering set with known answer sentences and move an answer-bearing sentence from the end of its group to the beginning, then compare CCA's exact-match score with full self-attention's; if CCA's score drops sharply while full attention's stays flat, the last-token importance measure of Eqn. (2) is discarding exactly the tokens that matter. Running the same experiment across group sizes g would bracket where the method's ceiling sits.","tokens_in":24605,"feed_emoji":"⚡","tokens_out":12339,"duration_ms":97938,"temperature":0.7,"pith_summary":"The paper claims that the quadratic self-attention inside a pretrained large language model can be swapped out, with only about a thousand steps of fine-tuning, for a \"Core Context Aware\" attention that compresses each fixed-size block of tokens into one core token and then attends over the few core tokens plus a sliding local window. The two branches are fused in a single softmax, so every earlier token remains reachable from every later token while the context is stored as far fewer key-value pairs. If the claim holds, long-context inference becomes near-linear in time and memory: the authors report a 7.9x first-token speedup at 128K context, a KV cache reduction from 64GB to 4.5GB, and question-answering accuracy at 64K-128K that matches or exceeds full self-attention. The reason this matters is that very long contexts would become affordable on existing hardware without training new models from scratch, and a single trained model could be dialed between speed and accuracy at inference time.","feed_headline":"Core-token attention runs long-context LLMs 7.9x faster","feed_subtitle":"Compresses token blocks into core tokens to keep 128K-context accuracy while slashing compute and memory.","key_machinery":"The load-bearing object is the group-wise weighted pooling of Eqn. (2): the query vector of a block's last token scores every token in the block, a softmax turns those scores into pooling weights, and the core token is the weighted sum of the block's token embeddings; that one token then stands in for the whole block in the global attention branch. The locality-preserving module, which keeps the preceding s tokens in full resolution, supplies the fine-grained detail the pooling discards, and the differentiable fusion of Eqn. (5) concatenates global and local key-value pairs into a single softmax so the two sources of evidence are weighted against each other within one attention computation. The supporting theoretical result is the reachability argument of Proposition 1, which expands the fused attention element-wise into the structure of full attention and shows every earlier token contributes, through its block's core token, to every later query.","core_discovery":"The central discovery the paper argues for is that a pretrained LLM's dense attention can be replaced by a sparser structure that keeps the information full attention uses: divide the sequence into blocks of g tokens, judge each block's important content by the attention its last token pays within the block (the weighted pooling of Eqn. 2), and use the resulting core tokens as keys and values for a global branch, while a local branch keeps the most recent s tokens in full detail. Concatenating the two branches' keys and values into one causal softmax (Eqn. 5) gives every query a positive attention weight toward every preceding token — the reachability property proved in Proposition 1 — so the compression cannot sever long-range dependencies even as it drops what the paper calls redundant context. On this basis the authors report that, after minimal continued training, CCA versions of LLaMA2-7B, LLaMA3.1-8B, and Qwen2.5-7B match or beat the efficient-attention baselines on LongBench-E and RULER, beat full self-attention at 64K-128K on multi-document question answering, and prefill at 128K runs 7.9x faster with roughly one-fourteenth the KV cache.","pith_inferences":["An implication the paper leaves implicit: the last-token-as-judge principle could be replaced by other importance measures at equal cost, such as the average query of the group or a learned importance head, and whether accuracy improves would isolate whether the pooling premise or the compression idea is doing the work.","Because core tokens are produced for every block, they double as a readable summary of the context; checking whether their pooling weights concentrate on entities, numbers, or verbs would show whether CCA can also serve as an attribution tool for long-context QA.","The reachability proof guarantees influence but not information retention; quantifying how much block-level signal survives the g-to-1 pooling would predict the context lengths and task types where CCA diverges from full attention.","The inference-time dial between g and s implies a deployment pattern — one checkpoint serving multiple latency targets — that fixed-pattern sparse attention methods cannot offer."],"forward_implications":["Long-context prefill and decoding both become sub-quadratic: O(Lm + Ls) for fixed-length inputs and O(L^2/g + Ls) for autoregressive generation, with KV cache storage cut from O(L) to O(L/g + s).","A single trained CCA checkpoint can be served at different group sizes g and window sizes s at inference time, trading a little accuracy for large throughput gains as traffic changes.","The self-attention replacement transfers across model families — LLaMA2, LLaMA3.1, and Qwen2.5 — with the same minimal fine-tuning budget, and it is orthogonal to RoPE-based context-extension methods.","At 64K-128K contexts the authors report CCA matching or beating full self-attention on multi-document QA, implying that compressing redundant context helps rather than hurts very-long-context reasoning.","Because it cuts KV cache memory as well as compute, CCA's gains extend to the decoding stage, where baselines such as MInference fall back to full attention."],"supporting_citations":[{"why":"StreamingLLM is the attention-sink baseline CCA must beat; it keeps only initial and recent tokens and discards the rest.","marker":"(Xiao et al., 2024b)"},{"why":"MInference is the strongest efficiency baseline, with dynamic sparse prefill kernels; the paper claims CCA is faster and also cuts KV cache where MInference does not.","marker":"(Jiang et al., 2024)"},{"why":"LM-Infinite supplies the local-plus-global branch structure against which CCA compares and motivates the local window design.","marker":"(Han et al., 2023)"},{"why":"LongBench and LongBench-E are the main accuracy benchmarks where the paper reports CCA's long-context scores.","marker":"(Bai et al., 2023)"},{"why":"The multi-document exact-match protocol from Lost in the Middle supports the 4K-128K EM evaluations.","marker":"(Liu et al., 2024b)"},{"why":"Provides the LLaMA2-7B-80K long-context base model that the paper fine-tunes CCA into.","marker":"(Fu et al., 2024)"},{"why":"RULER is the second long-context benchmark used to compare CCA against MInference in the supplementary results.","marker":"(Hsieh et al., 2024)"},{"why":"SlimPajama is the corpus used for the minimal fine-tuning that makes CCA plug-and-play.","marker":"(Cerebras, 2024)"},{"why":"LLaMA2 is the base architecture whose self-attention is replaced to build CCA-LLM.","marker":"(Touvron et al., 2023b)"}],"fun_headline_variants":["Core tokens compress context for 7.9x faster LLM prefill","Keep 128K-context accuracy with core-token attention","7.9x faster long-context prefill via core-token compression","KV cache shrinks 14x with core-token attention","Core-context attention speeds LLMs without losing long-range reach"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the premise that the last token of each group is a reliable judge of which tokens in the group matter for all future queries, because that single token's attention scores decide how the group is compressed into one core token, and the paper supports this premise only with attention-map visualizations.","fun_headline_variants_meta":{"raw":{"variants":["Core tokens compress context for 7.9x faster LLM prefill","Keep 128K-context accuracy with core-token attention","7.9x faster long-context prefill via core-token compression","KV cache shrinks 14x with core-token attention","Core-context attention speeds LLMs without losing long-range reach"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000291,"raw_usage":{"total_tokens":1739,"prompt_tokens":1023,"completion_tokens":716,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":639,"completion_tokens_details":{"reasoning_tokens":624}},"tokens_in":639,"tokens_out":716,"duration_ms":6316,"temperature":1.0,"reasoning_tokens":624,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:03:45.111257+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a multi-document question-answering set with known answer sentences and move an answer-bearing sentence from the end of its group to the beginning, then compare CCA's exact-match score with full self-attention's; if CCA's score drops sharply while full attention's stays flat, the last-token importance measure of Eqn. (2) is discarding exactly the tokens that matter. Running the same experiment across group sizes g would bracket where the method's ceiling sits.","supporting_citations":[{"cited_title":"16 Core Context Aware Transformers for Long Context Language Modeling C","cited_arxiv_id":null,"evidence_quote":"MInference is the strongest efficiency baseline, with dynamic sparse prefill kernels; the paper claims CCA is faster and also cuts KV cache where MInference does not."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LM-Infinite supplies the local-plus-global branch structure against which CCA compares and motivates the local window design."}],"review_version":1}