Pith. sign in

REVIEW 3 major objections 6 minor 27 references

Core Context Aware Transformers for Long Context Language Modeling

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Core Context Aware Attention replaces self-attention in pretrained LLMs with a plug-and-play two-branch module — group-wise pooling into core tokens plus a local window — reducing complexity to linear, KV cache to O(L/g + s), and…

desk verdict Useful attention-replacement idea with real speedups, but the 'linear complexity' claim is wrong as stated and the comparison protocol is unfair; still worth a referee's time. read the letter →

arxiv 2412.12465 v3 pith:KHVCRWAB submitted 2024-12-17 cs.CL cs.LG

Yaofo Chen , Zeng You , Shuhai Zhang , Haokun Li , Yirui Li , Yaowei Wang , Mingkui Tan This is my paper · ORCID
classification cs.CLcs.LG
keywords corecontextawareattentionlonglanguagemodelingefficientKVcachecompressionglobality-awarepoolinglocality-preservingplug-and-playLLMadaptationlinearcomplexity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the quadratic self-attention inside a pretrained large language model can be swapped out, with only about a thousand steps of fine-tuning, for a "Core Context Aware" attention that compresses each fixed-size block of tokens into one core token and then attends over the few core tokens plus a sliding local window. The two branches are fused in a single softmax, so every earlier token remains reachable from every later token while the context is stored as far fewer key-value pairs. If the claim holds, long-context inference becomes near-linear in time and memory: the authors report a 7.9x first-token speedup at 128K context, a KV cache reduction from 64GB to 4.5GB, and question-answering accuracy at 64K-128K that matches or exceeds full self-attention. The reason this matters is that very long contexts would become affordable on existing hardware without training new models from scratch, and a single trained model could be dialed between speed and accuracy at inference time.

What carries the argument

The load-bearing object is the group-wise weighted pooling of Eqn. (2): the query vector of a block's last token scores every token in the block, a softmax turns those scores into pooling weights, and the core token is the weighted sum of the block's token embeddings; that one token then stands in for the whole block in the global attention branch. The locality-preserving module, which keeps the preceding s tokens in full resolution, supplies the fine-grained detail the pooling discards, and the differentiable fusion of Eqn. (5) concatenates global and local key-value pairs into a single softmax so the two sources of evidence are weighted against each other within one attention computation. The supporting theoretical result is the reachability argument of Proposition 1, which expands the fused attention element-wise into the structure of full attention and shows every earlier token contributes, through its block's core token, to every later query.

What would settle it

Build a multi-document question-answering set with known answer sentences and move an answer-bearing sentence from the end of its group to the beginning, then compare CCA's exact-match score with full self-attention's; if CCA's score drops sharply while full attention's stays flat, the last-token importance measure of Eqn. (2) is discarding exactly the tokens that matter. Running the same experiment across group sizes g would bracket where the method's ceiling sits.

Watch

Extended reading notes

Core claim

The central discovery the paper argues for is that a pretrained LLM's dense attention can be replaced by a sparser structure that keeps the information full attention uses: divide the sequence into blocks of g tokens, judge each block's important content by the attention its last token pays within the block (the weighted pooling of Eqn. 2), and use the resulting core tokens as keys and values for a global branch, while a local branch keeps the most recent s tokens in full detail. Concatenating the two branches' keys and values into one causal softmax (Eqn. 5) gives every query a positive attention weight toward every preceding token — the reachability property proved in Proposition 1 — so the compression cannot sever long-range dependencies even as it drops what the paper calls redundant context. On this basis the authors report that, after minimal continued training, CCA versions of LLaMA2-7B, LLaMA3.1-8B, and Qwen2.5-7B match or beat the efficient-attention baselines on LongBench-E and RULER, beat full self-attention at 64K-128K on multi-document question answering, and prefill at 128K runs 7.9x faster with roughly one-fourteenth the KV cache.

Load-bearing premise

Everything rests on the premise that the last token of each group is a reliable judge of which tokens in the group matter for all future queries, because that single token's attention scores decide how the group is compressed into one core token, and the paper supports this premise only with attention-map visualizations.

Editorial extensions

If this is right

  • Long-context prefill and decoding both become sub-quadratic: O(Lm + Ls) for fixed-length inputs and O(L^2/g + Ls) for autoregressive generation, with KV cache storage cut from O(L) to O(L/g + s).
  • A single trained CCA checkpoint can be served at different group sizes g and window sizes s at inference time, trading a little accuracy for large throughput gains as traffic changes.
  • The self-attention replacement transfers across model families — LLaMA2, LLaMA3.1, and Qwen2.5 — with the same minimal fine-tuning budget, and it is orthogonal to RoPE-based context-extension methods.
  • At 64K-128K contexts the authors report CCA matching or beating full self-attention on multi-document QA, implying that compressing redundant context helps rather than hurts very-long-context reasoning.
  • Because it cuts KV cache memory as well as compute, CCA's gains extend to the decoding stage, where baselines such as MInference fall back to full attention.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the last-token-as-judge principle could be replaced by other importance measures at equal cost, such as the average query of the group or a learned importance head, and whether accuracy improves would isolate whether the pooling premise or the compression idea is doing the work.
  • Because core tokens are produced for every block, they double as a readable summary of the context; checking whether their pooling weights concentrate on entities, numbers, or verbs would show whether CCA can also serve as an attribution tool for long-context QA.
  • The reachability proof guarantees influence but not information retention; quantifying how much block-level signal survives the g-to-1 pooling would predict the context lengths and task types where CCA diverges from full attention.
  • The inference-time dial between g and s implies a deployment pattern — one checkpoint serving multiple latency targets — that fixed-pattern sparse attention methods cannot offer.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Core Context Aware (CCA) Attention, a plug-and-play replacement for the self-attention module in pretrained LLMs. The mechanism partitions the input into groups of size g, compresses each group into a single 'core' token using an attention-weighted pooling driven by the group's last token, and then computes attention over the resulting core tokens while also attending to a local window of preceding tokens. The authors claim linear-time complexity, reduced KV cache, minimal fine-tuning, and superior long-context modeling, reporting up to a 7.9x first-token latency speedup at 128K context on LLaMA2-7B-80K. Experiments cover LongBench-E, RULER, multi-document EM, MMLU, and models LLaMA2, LLaMA3.1, and Qwen2.5.

Significance. If the stated complexity and accuracy claims held, the method would be a practical low-cost retrofit for long-context inference. The paper's strengths include a simple architecture, public source code, and evaluation across several benchmarks and model families. However, the central 'linear complexity' claim is internally inconsistent with the paper's own complexity analysis, and the aggregate accuracy results do not support 'superior' long-context modeling over vanilla self-attention. The contribution is best viewed as a subquadratic approximation with large constant-factor speedups, pending corrective revisions.

major comments (3)
  1. [Section 3.6, Eqn. (2), Table 8, Abstract] The abstract and Section 1 state that CCA-Attention 'reduces computational complexity to linear complexity,' but Section 3.6 derives O(L^2/g + Ls) for autoregressive decoding when the group size g is a constant, and the experiments use g=16 (Table 8). Because m = floor(L/g) in Eqn. (2), the global-attention term is Theta(L^2/g), which is quadratic for any fixed g; the 7.9x speedup reported in Table 3 is a constant-factor reduction rather than a change of asymptotic class. The fixed-length case O(Lm + Ls) is only linear if m is treated as a constant, which contradicts the definition of m and the implemented setting. The efficiency claims should be restated for two regimes: fixed g (subquadratic, O(L^2/g + Ls)) and fixed m (linear, O(Lm + Ls)), with the latter requiring g to grow with L and a separate validation.
  2. [Tables 1 and 3, Section 4.2, Abstract] The abstract claims 'superiority of our method in both long-context modeling and computational efficiency,' and Section 4.2 claims CCA-LLM 'attains the highest average score' on LongBench-E, but the tables show the opposite. In Table 1, the CCA-LLM average is 21.86 vs 22.11 for vanilla self-attention on LLaMA2-7B-32K and 22.24 vs 22.42 on LLaMA2-7B-80K; in Table 3 the average EM score at 4K-128K is 34.4 vs 36.0 for vanilla attention. The specific claim in Section 4.2 that 'our CCA-LLM consistently achieves the highest EM score' for short contexts is also contradicted at 4K (39.3 vs 39.4). The modeling-superiority claim should be replaced by a competitive-accuracy claim, or supported by significance tests at the lengths (e.g., 64K/128K) where the paper claims improvement.
  3. [Section C.4, Eqn. (2), Table 7] The globality-aware pooling in Eqn. (2) uses the last token of each group as the query to assign importance weights to the group's tokens. This choice is load-bearing because the pooled core tokens are the only global information channel for all later queries; an unreliable importance measure would discard context needed downstream. The paper motivates this choice solely with attention-map visualizations on a single 32-token sentence (Section C.4). The authors should either quantify how well the last token's attention predicts the attention of later positions (e.g., by computing the correlation across a sample of long documents) or add an ablation that varies the query token (first, middle, group-average, or a learned query), since the existing Table 7 ablation varies only the pooling strategy, not the query source.
minor comments (6)
  1. [Section 4.2] Section 4.2 contains several typos: 'ur CCA-LLM' should read 'Our CCA-LLM,' 'MInferencce' should read 'MInference,' and 'accerlerate' should read 'accelerate.'
  2. [Section 3.6] Section 3.6 states that the locality-preserving module's upper bound is O(L(s+g)) but then reports the total as O(Lm+Ls); the g-dependent term should be included or explicitly absorbed, for example as O(Lm + L(s+g)).
  3. [Table 3 caption] The Table 3 caption says 'FTL denotes the latency to generate the first token,' but this is the prefill latency, not the per-token generation latency; labeling it 'prefill latency' or 'time to first token' would avoid confusion with the inter-token latency reported elsewhere.
  4. [Section 4.2] The sentence in Section 4.2 claiming that 'our CCA-LLM consistently achieves the highest EM score' for short contexts is not supported by Table 3, where CCA-LLM scores 39.3 versus 39.4 for vanilla self-attention at 4K.
  5. [Section 3.5] Calling the fine-tuning costs 'minimal' may be misleading, as Section B.2 reports 2.1B tokens for 32K and 5B tokens for 80K; please clarify the comparison baseline or rephrase the claim.
  6. [Section 3.2] The phrase 'tokens in close proximity to the query Qi are likely to be more relevant' should specify 'preceding tokens within the local window,' since causal masking prevents attending to future tokens.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline 'linear complexity' claim is forced by redefining the group count m as constant, contradicting Eqn. (2)'s m = L/g; empirical benchmark claims are independent and non-circular.

  1. self definitional [Section 3.6 'Computational and Storage Complexity Analysis' (also abstract, Contribution bullet 1, and Section C.3); m is defined in Section 3.2, Eqn. (2)]
    "we segment the input sequence X, each group containing g tokens, in total m=L/g groups ... For tasks with fixed-length sequences ... our CCA-Attention exhibits a linear computational complexity of O(Lm + Ls) ... Here, we define the number of group m as a constant."

    By the paper's own definitions the linearity claim is stipulated, not derived: Section 3.2/Eqn. (2) fixes m = L/g, so the globality module cost O(Lm) (Figure 2) equals O(L^2/g) for any fixed g, which is quadratic in L. Section 3.6 obtains 'linear' only by adding 'Here, we define the number of group m as a constant,' a redefinition contradicted by Eqn. (2) and by the implemented regime (g=16 default, Table 8; Section C.5 sweeps g=2..64). The same section's variable-length analysis fixes g and honestly reports O(L^2/g + Ls), still quadratic.

full rationale

One definition-forced step was found, and it is load-bearing for the paper's headline efficiency claim. In Section 3.6 the paper claims 'our CCA-Attention exhibits a linear computational complexity of O(Lm + Ls)' and obtains this by stipulating 'Here, we define the number of group m as a constant,' while Section 3.2/Eqn. (2) defines m = floor(L/g). With the paper's own definitions, the globality module costs O(Lm) = O(L^2/g), which is quadratic for the fixed group sizes used in every experiment (default g=16, Table 8); the paper's own autoregressive analysis states O(L^2/g + Ls), also quadratic. The 'linear' conclusion therefore follows only from redefining m, i.e., the central complexity-class claim is forced by definition rather than derived from the method. This constitutes partial circularity, not a fully circular paper: all modeling-accuracy claims rest on external benchmarks (LongBench-E, multi-document EM, RULER, MMLU, Proof-pile PPL), hyperparameters g and s are selected by PPL ablations (Tables 8 and 9) rather than fit to target results, the measured speedups and memory savings are empirical constant-factor gains, and no fitted-input-called-prediction pattern appears. No self-citation appears in the reference list, so the self-citation and uniqueness-import patterns do not apply. The reachability proof (Proposition 1) is a formal restatement of the construction (positive softmax weights over prefix groups plus local window) and is not circular, though its binary notion of 'access' via a g-way pooled proxy is much weaker than full self-attention, which is a scope caveat rather than a circularity. Apart from the complexity-class claim, the derivation chain is self-contained and the empirical claims are independently grounded.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claims depend on two hand-chosen hyperparameters (g, s), a reported but underspecified m for the linear-complexity argument, and three domain assumptions about attention sparsity, last-token queries, and fine-tuning sufficiency. No new physical entities are introduced.

free parameters (4)
  • group size g = 16 (default)
    Chosen from PPL ablations (Table 8); larger g reduces latency but degrades perplexity. Used in all main experiments.
  • local window size s = 1024 (default)
    Chosen from PPL ablations (Table 9); larger s improves PPL at higher cost.
  • number of groups m (for fixed-length tasks) = set to a constant in MMLU experiments; exact value not reported
    Introduced in Section 3.6 to claim linear complexity O(Lm + Ls); in practice m = floor(L/g) for other experiments.
  • RoPE base frequency = 500000
    Changed from 10000 to 500000 following prior long-context scaling work; affects position encoding and is a design choice.
assumptions (4)
  • standard math softmax attention weights are positive and normalize to 1
    Used throughout, e.g., Eqn (2) and Proposition 1, to show all tokens are reachable.
  • domain assumption Attention scores in pretrained LLMs are highly sparse, with a few tokens receiving most mass
    Motivates the entire compression approach; cited in Section 3.1 and supported only by visualizations (Section C.4), not by quantitative evidence in this paper.
  • domain assumption The last token of a group is a sufficient query to measure the importance of all tokens in that group for future tokens
    Used in Eqn (2) to compute core tokens; justified only by attention map visualizations, with no theoretical or empirical demonstration that this proxy is reliable.
  • domain assumption Fine-tuning for 1000 steps on SlimPajama preserves pretrained knowledge and adapts the model to CCA-Attention
    Central to the plug-and-play claim; the paper shows convergence curves but no comparison of knowledge retention beyond benchmark scores.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Core Context Aware Transformers for Long Context Language Modeling." pith.science (2026). https://pith.science/paper/KHVCRWAB

@misc{pith2026241212465,
  author       = {Pith},
  title        = {Pith review of: Core Context Aware Transformers for Long Context Language Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KHVCRWAB}},
  note         = {Machine review of arXiv:2412.12465}
}
read the original abstract

Transformer-based Large Language Models (LLMs) have exhibited remarkable success in extensive tasks primarily attributed to self-attention mechanism, which requires a token to consider all preceding tokens as its context to compute attention. However, when the context length L becomes very large (e.g., 128K), the amount of potentially redundant information in the context tends to increase. The redundant context not only hampers the modeling representation performance but also incurs unnecessary computational and storage overhead. In this paper, we propose a plug-and-play Core Context Aware (CCA) Attention for efficient long-context modeling, comprising two complementary modules: 1) Globality-aware pooling module groups input tokens and dynamically compresses each group into one core token based on their significance. In this way, our method automatically focuses and strengthens core context while diminishing redundancy during the learning process, leading to effective long-term dependency modeling. 2) Locality-preserving module incorporates neighboring tokens to preserve local context for detailed representation. Notably, our CCA-Attention is able to replace the self-attention module in existing LLMs with minimal fine-tuning cost. Extensive experimental results show the superiority of our method in both long-context modeling and computational efficiency over state-of-the-art methods.

Figures

Figures reproduced from arXiv: 2412.12465 by the authors.

Figure 1
Figure 1. Illustration of core contexts and redundant contexts. We show attention scores of the last token relative to the other tokens in LLaMA2-7B (darker shadows indicate higher attention scores). The last token exhibits high attention scores towards core contexts. The remains are considered as redundant contexts, introducing unnecessary computational overhead for attention. and final tokens, effectively disregarding the i… view at source ↗
Figure 2
Figure 2. Illustration of CCA-Attention, which includes two components: 1) Globality-aware pooling module encapsulates the input tokens X into core tokens C according to the importance (Eqn. (2)). The core tokens C serve as representative proxies of X for attention, thereby reducing computational costs. 2) Locality-preserving module incorporates the local context from neighboring tokens, acting as supplement for the globality… view at source ↗
Figure 3
Figure 3. Illustration of inference flexibility by adjusting the group size g and local window size s to generate various CCA-LLM models with different latency and accuracies in the test time. This architectural flexibility allows for precise control over the trade-off between inference latency and accuracy, particularly beneficial for real-world applications with varying user traffic patterns. and KV cache usage among all co… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparisons with state-of-the-art methods in terms of both computational and storage overhead on LLaMA2-7B-80K. “FTL” (first token latency) is the time taken to generate the first token after receiving the input in the pre-filling stage. “ITL” (inter token latency) is …
Figure 5
Figure 5. Figure 5: A visualization of attention scores in LLaMA2-7B with a sentence of 32 input tokens. The attention map reveals a distinct pattern: the majority of tokens exhibit minimal attention scores. Conversely, a minority of tokens are associated with significantly higher attenti…
Figure 6
Figure 6. Figure 6: Convergence curves of our CCA-LLM models under different contexts. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 13 canonical work pages

  1. [1]

    Same as demonstrated in existing methods (Beltagy et al., 2020; Xiao et al., 2024b)

    As shown in the figure, these attention scores show consistent sparsity from shallow to deep layers. Same as demonstrated in existing methods (Beltagy et al., 2020; Xiao et al., 2024b). 17 Core Context Aware Transformers for Long Context Language Modeling Figure

    work page 2020

  2. [4]

    Extending context window of large language models via positional interpolation

    Chen, S., Wong, S., Chen, L., and Tian, Y . Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595,

    arXiv

  3. [5]

    Masked language modeling for proteins via linearly scalable long-context transformers

    Choromanski, K., Likhosherstov, V ., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Belanger, D., Colwell, L., et al. Masked language modeling for proteins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03555,

    arXiv 2006

  4. [6]

    Longnet: Scaling transformers to 1,000,000,000 tokens

    Ding, J., Ma, S., Dong, L., Zhang, X., Huang, S., Wang, W., Zheng, N., and Wei, F. Longnet: Scaling transformers to 1,000,000,000 tokens. arXiv preprint arXiv:2307.02486,

    arXiv

  5. [7]

    Data engineering for scaling language models to 128k context.arXiv preprint arXiv:2402.10171,

    Fu, Y ., Panda, R., Niu, X., Yue, X., Hajishirzi, H., Kim, Y ., and Peng, H. Data engineering for scaling language models to 128k context.arXiv preprint arXiv:2402.10171,

    arXiv

  6. [8]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,

    arXiv

  7. [9]

    s 256 512 1024 2048 4096 PPL ↓ 2.98 2.92 2.86 2.79 2.73 Latency ↓ (ms) 457.4 460.1 461.4 462.8 473.1 Effect of Different Updating Strategies

    Effect of local window size s. s 256 512 1024 2048 4096 PPL ↓ 2.98 2.92 2.86 2.79 2.73 Latency ↓ (ms) 457.4 460.1 461.4 462.8 473.1 Effect of Different Updating Strategies. As mentioned in Section 3.4, we have two updating strategies:

    work page

  8. [12]

    Mu, J., Li, X., and Goodman, N. D. Learning to com- press prompts with gist tokens. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023,

    work page 2023

Show all 27 references
  1. [13]

    GPT-4 technical report

    OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv

  2. [15]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozi `ere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. T...

    arXiv

  3. [16]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115,

    arXiv

  4. [17]

    The MMLU benchmark spans 57 diverse subjects, ranging from elementary mathematics to professional law

    dataset is designed to assess the capabilities of language models across a wide array of subjects, delving deeper into their academic and professional understanding. The MMLU benchmark spans 57 diverse subjects, ranging from elementary mathematics to professional law. The ques...

    2021

  5. [18]

    base frequency

    dataset, an open-source replication of the LLaMA pretraining data mixture. This dataset comprises 82% web data, split between 67% from CommonCrawl and 15% from C4, alongside 4.5% from GitHub code, 4.5% from Wikipedia, 4.5% from books, 2.5% from Arxiv, and 2.0% from Stack Excha...

    2024

  6. [19]

    This enables us to integrate our CCA-Attention as a standalone, cache-friendly operator, effectively eliminating redundant computations

    to perform low-level operator fusion between our globality-aware pooling and locality-preserving modules. This enables us to integrate our CCA-Attention as a standalone, cache-friendly operator, effectively eliminating redundant computations. Compared Methods. We conduct compr...

    2023

  7. [20]

    Our experiments are based on the LLaMA-2 7B model fine-tuned on sequences of length 32K and 80K (Fu et al., 2024)

    benchmarks. Our experiments are based on the LLaMA-2 7B model fine-tuned on sequences of length 32K and 80K (Fu et al., 2024). For StreamingLLM (Xiao et al., 2024b), we use the official implementation, adjusting the attention sink to 4 and setting the attention context size to

    2024

  8. [22]

    16 Core Context Aware Transformers for Long Context Language Modeling C

    In the case of MInference (Jiang et al., 2024), we also employ the official code implementations, configured with the official settings. 16 Core Context Aware Transformers for Long Context Language Modeling C. More Experimental Results C.1. Experiments on More Long Context Ben...

    2024

  9. [23]

    67.78 65.32 61.43 52.77 61.83 CCA-LLM (Ours) 68.15 66.31 60.89 54.88 62.56 C.2. Comparisons with More Efficient Attention Methods To further evaluate our method, we compare it with LongLoRA (Chen et al., 2024), a recently proposed training-based approach with PI techniques. As...

    2024

  10. [25]

    Strategy Mean Pooling Max Pooling CCA-Attention (Ours) PPL ↓ 2.99 2.99 2.85 Effect of Group Size g

    Effect of pooling strategy. Strategy Mean Pooling Max Pooling CCA-Attention (Ours) PPL ↓ 2.99 2.99 2.85 Effect of Group Size g. To investigate the effect of different group sizesg, we implement the proposed CCA-Attention with different g ∈ {2, 4, 8, 16, 32, 64}. In Table 8, as...

  11. [27]

    The perplexity rapidly converges within approximately the first 100 iterations and remains stable over 1,000 iterations

    From the results, by minimizing the training loss, both LLaMA2-7B-32K and LLaMA2-7B-80K models are able to converge very fast. The perplexity rapidly converges within approximately the first 100 iterations and remains stable over 1,000 iterations. These results not only demons...

    2024

  12. [2000]

    Similarly, for LM-infinite (Han et al., 2023), we follow the official code, configuring the local branch size to 1024 and the global branch size to

    2023

  13. [2019]

    Deepseek- v3 technical report

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. 10 Core Context Aware Transformers for Long Context Language Modeling Liu, N. F., Lin, K., Hewitt, J., P...

    arXiv

  14. [2020]

    Retentive network: A successor to transformer for large language models

    Sun, Y ., Dong, L., Huang, S., Ma, S., Xia, Y ., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621,

    arXiv

  15. [2021]

    Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., Zhang, Y ., and Ginsburg, B. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654,

    arXiv

  16. [2022]

    Longbench: A bilingual, multitask benchmark for long context understanding

    Bai, Y ., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y ., Tang, J., and Li, J. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508,

    arXiv

  17. [2023]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150,

    2004 arXiv

  18. [2024]

    Chang, Y ., Wang, X., Wang, J., Wu, Y ., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y ., Ye, W., Zhang, Y ., Chang, Y ., Yu, P

    Accessed: 2025-01-20. Chang, Y ., Wang, X., Wang, J., Wu, Y ., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y ., Ye, W., Zhang, Y ., Chang, Y ., Yu, P. S., Yang, Q., and Xie, X. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems an...

    2025

  19. [2025]

    LM-infinite: Simple on-the-fly length generalization for large language models

    Han, C., Wang, Q., Xiong, W., Chen, Y ., Ji, H., and Wang, S. LM-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arXiv:2308.16137,

    arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.