SEAL: Scaling to Emphasize Attention for Long-Context Retrieval

Changhun Lee; Eunhyeok Park; Jun-gyu Jin; Minsang Seok; Younghyun Cho

REVIEW 4 major objections 5 minor 28 references

SEAL: Scaling to Emphasize Attention for Long-Context Retrieval

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that scaling attention head and channel strengths, learned from 50 synthetic samples, recovers long-context retrieval accuracy and can extend effective context without inference overhead.

desk verdict SEAL is a cheap, surprisingly effective attention-scaling fix for long-context retrieval, with broad evidence across models; the main caveat is reproducibility. read the letter →

arxiv 2501.15225 v2 pith:D2QEH4SG submitted 2025-01-25 cs.CL cs.AIcs.LG

Changhun Lee , Minsang Seok , Jun-gyu Jin , Younghyun Cho , Eunhyeok Park This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords long-contextretrievalattentionheadschannel-wisescalingsyntheticdataLLMcontextwindowextensionRULERSEAL

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SEAL, Scaling to Emphasize Attention for Long-context Retrieval, proposes that long-context retrieval failures are not fixed capacity limits but misallocated attention: some attention heads and channels help retrieval while others actively hurt it. The method learns a small set of per-head or per-channel scaling factors for attention outputs, tuned with only 50 synthetic samples that match the target task's format rather than its content. After tuning, the scales are merged into the model weights offline, so inference cost is unchanged. Across line retrieval, needle-in-a-haystack, and RULER tasks, SEAL restores much of the lost accuracy, and combined with training-free context extension it lets models retrieve beyond their native window.

What carries the argument

The central mechanism is a set of learnable multiplicative scales applied to attention outputs: one scalar per head for SEAL-H, or one scalar per hidden channel for SEAL-C. These scales are trained by gradient-based optimization on the synthetic samples and afterward folded into the v_proj or o_proj weight matrices so inference is unchanged. The paper's justification rests on a pruning analysis showing that the effect of removing an attention head or channel is consistent at different context lengths, which makes the scale a stable task-level knob rather than a per-sample artifact.

What would settle it

Take a model after SEAL tuning for line retrieval and run it on a new set of 50 independently generated line-retrieval prompts with different keys and numbers; if the retrieval gain over baseline vanishes or turns negative, the claimed transfer fails. A more direct test is to re-run SEAL twice from different random synthetic samples and compare the learned scale vectors; if they disagree strongly while both improve the training samples, the tuning is overfitting the synthetic set rather than finding stable attention components.

Watch

Extended reading notes

Core claim

The paper establishes, on its own terms, that the strength of individual attention heads and channels is a controllable dial for long-context retrieval. Head-wise pruning shows a spread of roughly ±20% accuracy effects that remain consistent across context lengths; jointly scaling down heads that hurt and scaling up heads that help improves line retrieval from 0.32 to 0.88 at 31K tokens for LongChat-7B. Channel-wise scaling refines this further, with SEAL-C giving near-perfect results on RULER common-word extraction at 32K. The learned scales are derived from synthetic samples that share the task format but not the content, and the paper's transferability analysis shows the scales travel within task categories better than across them.

Load-bearing premise

The scales only help if the attention components that matter for retrieval behave the same across different content of the same task format, so that patterns learned from 50 synthetic examples transfer to unseen needles and lines.

Editorial extensions

If this is right

For LongChat-7B, SEAL-C raises 31K line retrieval from 0.32 to 0.88, and SEAL-H reaches 0.80 with only 1,024 learnable parameters.
Near-perfect common-word extraction at 32K on Llama-3.1-8B indicates that what changes is the effective context range where reliable retrieval is possible, not just the nominal window.
Combining SEAL with NTK or Self-Extend lets Llama-2-7B-Chat retrieve beyond its 4K native window, providing an alternative to costly fine-tuning-based context extension.
Scales tuned on common-word extraction transfer to frequent-word extraction and vice versa, but not to variable tracking, suggesting that task-category coverage rather than per-task tuning may be enough.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the tuning data is format-matched but content-random, the results suggest that long-context retrieval failures are induced attention biases rather than missing knowledge; a direct test would be whether the same scales survive when the haystack corpus changes from synthetic text to real documents of the same format.
SEAL-C's higher cross-task transfer suggests channel-level scales act as reusable task-format lenses; one could imagine selecting among a small set of precomputed scales at inference time, which the paper does not explore.
The method's dependence on format alignment predicts that a mismatch between training format and test format should erase gains; this is testable by applying CWE-trained scales to a variant with different quoting or line conventions.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: SEAL tunes per-head or per-channel attention scales on fifty synthetic samples, merges the scales offline, and reports large retrieval gains across six models and several benchmarks. The method is simple, practical, and the gains are big enough to take seriously. The main soft spot is that the paper gives us no code, no data, and no variance estimates, so I can't tell whether the exact numbers are robust or partly a lucky 50-sample draw.

What's genuinely new: head-wise attention scaling itself isn't new, but learning channel-wise and head-wise scales on tiny format-matched synthetic data and then merging them into the weights is a clean, useful combination. The experiments are broad—LongEval, Needle-in-a-Haystack, RULER, plus LongBench doc QA—and the comparison against LoRA/DoRA shows SEAL gets comparable or better retrieval with roughly four orders of magnitude fewer parameters. The circuit-analysis section (direct effect) is a nice addition: it shows why scaling helps rather than just reporting numbers. The authors are also honest about limits: transfer across task formats is poor (Fig. 8), learning rate needs per-task tuning (Tables 4–5), and long training samples stress GPU memory. The pruning analysis in Section 3 motivates the design, but the final scales are learned on synthetic data and tested on held-out examples from the same task format—that's a standard supervised setup, not circular reasoning.

The soft spots are real but not disqualifying. No seeds, no error bars, and every reported number appears to come from a single tuning run. Given that the method changes just 1K–131K parameters, the dramatic jumps (LongChat line retrieval from 0.32 to 0.88) could partly be an artifact of learning-rate selection. The paper's own sweep in Table 7 shows sample count and LR both matter, but each cell is one run. The LongBench results are encouraging but use triviaqa_e as the tuning set—good for avoiding contamination, but the scale isn't learned on the target task format, so those gains are more about general retrieval-head emphasis. The transferability analysis is honest but limits the claim: you likely need a few samples per task category.

Who is this for? Anyone working on long-context efficiency, parameter-efficient fine-tuning, or attention interpretability. It deserves a serious referee. The method is simple enough to reproduce, the claimed gains are large enough to justify verification, and the paper would be substantially stronger with released code, seeds, and a few repeats per condition. Send it to review.

Referee Report

4 major / 5 minor

Summary. SEAL proposes to improve long-context retrieval by learning multiplicative scaling factors for attention heads (SEAL-H) or attention-output channels (SEAL-C). The scales are trained for one epoch on 50 synthetic examples that mimic the format of the target retrieval task, then merged offline into the v_proj/o_proj weights so that inference cost is unchanged. The paper reports large improvements over baselines on LongEval line retrieval, Needle-in-a-Haystack, and three RULER tasks across six open LLMs, and in the appendix on LongBench Document QA; it also shows that SEAL can be combined with NTK and Self-Extend context extension. A pruning study and a direct-effect analysis are used to motivate and interpret the method.

Significance. If the reported results are reproducible, this is a practically valuable and conceptually interesting result: a very small, task-format-specific reweighting of attention outputs substantially recovers long-context retrieval accuracy without model fine-tuning or added inference overhead. The paper's strengths are the breadth of evaluation (six models, several benchmarks, an additional real-world benchmark), the clean offline-merging argument, the extremely small parameter counts (Table 8), and the qualitative direct-effect analysis. The main weaknesses are that no code, data, or learned scales are provided, all tuning runs are single-seed, and the learning rate is selected per model/task without an explicit protocol, leaving the central '50 samples are sufficient' claim insufficiently supported for replication.

major comments (4)

[Section 4.1; Tables 1-2, 7] The central claim that 50 synthetic samples yield stable attention scales is not yet supported because every result comes from one draw of the training samples and one optimization run. With up to 131K tunable parameters (Table 8) fit to 50 examples, the headline improvements (e.g., LongChat-7B line retrieval at 31K from 0.32 to 0.88 in Table 1; Llama-3.1 CWE at 64K from 0.1 to 95.7 in Table 2) could be partly due to a favorable draw. Please report repeated independent draws of the 50 training samples (or a bootstrap), with means and confidence intervals, for at least the main benchmark points. The existing Table 7 and Figure 9 are helpful but are also single-seed and use a different random sample set than Table 1.
[Appendix A.1, Tables 4-5, Limitations] The learning rate, described as the only hyperparameter, is varied per model, per task, and per SEAL variant, and the selection protocol is not stated. Table 7 shows that on the same task and sample count, scores at 31K range from 0.78 (LR 1e-2) to 0.90 (LR 3e-2), while Table 1 reports 0.80 for SEAL-H; without knowing whether the reported numbers come from a pre-specified grid or from test-set-based selection, the gains are difficult to interpret. Please state the selection procedure and report results for a fixed LR or across the full grid, preferably with multiple seeds.
[Reproducibility (all experiments)] The manuscript does not release code, the exact 50 synthetic samples per task, or the learned scales. Since SEAL is an offline modification of open models, providing these artifacts is feasible and would allow independent verification of the dramatic gains. I request that the revision include a public repository with tuning scripts, data-generation code, and learned scale files.
[Section 7, Eq. (2), Figure 8] The transferability analysis is based on single-run scales and uses a normalized metric whose denominator is the baseline's score range per task, making the values in Figure 8 hard to compare across tasks. Please report the raw scores and state whether the conclusions (CWE/FWE transfer, VT does not) are stable across random tuning draws. This is secondary to the main claim, but it is used to recommend future directions.

minor comments (5)

[Section 1 / Section 6] The abstract and Section 6 say SEAL 'extends the contextual limits of LLMs while maintaining highly reliable outputs'; the evidence is for retrieval tasks only, so 'reliable outputs' is too broad.
[Section 5.1, Eq. (1)] The quantity in Eq. (1) is called a 'direct effect', but in circuit analysis a direct effect usually requires an intervention. The text says 'without the normalization term' but should clarify that this is a logit attribution rather than the standard direct effect, to avoid terminological confusion.
[Section 5.2] The MMLU check is reported for LongChat-7B only, with a decrease from 42.53 to 42.17 for SEAL-C; this is small but not necessarily negligible, and no variance is reported. Please either report more models or soften the claim that MMLU 'remains nearly unchanged'.
[Appendix E, Table 7 vs Table 1] Table 7 was generated from a new set of 100 random samples, so the 0.78 at LR 1e-2 (50 samples) is not the same run as the 0.80 in Table 1; the authors should note this and discuss the observed sample-to-sample variation.
[Tables 4-5] The formatting of Tables 4 and 5 makes it difficult to tell which learning rate corresponds to which model and SEAL variant; please use explicit rows and columns.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SEAL's gains come from held-out evaluations, not from quantities forced by the tuning procedure or by self-citation.

full rationale

Walking the derivation chain: the pruning experiments in Section 3 are diagnostic and motivate the design, but the final SEAL-H/SEAL-C scales are obtained by gradient-based tuning on 50 synthetic samples generated in the target task format (Section 4), not by reusing the pruning results as the reported improvement. All headline results in Tables 1-2 and Figures 5-7 are computed on held-out test needles, lines, and RULER samples after merging the learned scales into the model, so the reported retrieval scores are not algebraically determined by the fitted scalars; they are independent empirical evaluations. The offline merging in Section 4.3 is an exact reparameterization of the scaling operation into v_proj/o_proj weights and does not create a new quantity that is equal to its input by construction. The direct-effect analysis in Section 5.1 is a post-hoc explanation on one example, not the source of the quantitative gains. The transferability metric in Section 7 is a normalized measurement of cross-task score differences, not a derivation that presupposes its conclusion. There are no load-bearing self-citations by the authors and no imported uniqueness theorem; the cited works are external benchmarks, models, and interpretability studies. The paper's own Sections 7 and Appendix E disclose that cross-task transfer is limited and that sample count and learning rate matter, which is an honest robustness limitation rather than a circularity. The remaining concerns about single-seed tuning and 50-sample sensitivity are reproducibility/statistical questions, not cases where the output is equivalent to the input by definition.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

No new physical or architectural entities are introduced; the method relies on empirical assumptions about the stability of attention head/channel importance across content and the sufficiency of format-matched synthetic samples. The trained scales and the per-task learning rate are the main tunable quantities.

free parameters (2)

Attention scale values (SEAL-H/SEAL-C) = Learned; not reported as fixed values (SEAL-H is L*H scalars, e.g., 1,024 for LongChat-7B; SEAL-C is L*H*d_h, e.g.…
These are the method's trained parameters, fitted to 50 synthetic samples per task. The central claim of improvement depends on these fitted values being appropriate; they are not fixed constants.
Learning rate = 1e-2 to 5e-2 depending on model/task
The only hyperparameter, chosen per model and task (Tables 4-5). The paper states 'the optimal learning rate varies slightly depending on the difficulty or characteristics of the downstream task.'

assumptions (3)

domain assumption Attention head/channel importance is consistent across content and context lengths within a task format.
The method learns scales on 50 synthetic samples and expects them to transfer to held-out examples. Evidence is Fig. 2(b) showing consistency between 28K and 31K; transferability analysis (Sec. 7) shows limited transfer across task formats, so this assumption is task-format specific.
domain assumption Synthetic samples with the same format as the target task contain enough signal to learn transferable scales.
Section 4.1 uses synthetic data with 'only the format remained consistent while generating random content'. The success depends on format matching; the Needle-in-a-Haystack results use ChatGPT-generated needles that differ from test needles, supporting this in one setting.
domain assumption Language modeling loss on synthetic answers drives scales toward retrieval-favorable values without destructive forgetting.
The tuning uses standard cross-entropy loss on generated answers (Sec. 4.2). MMLU scores remain nearly unchanged (Sec. 5.2), providing empirical support, though the mechanism is not proven.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SEAL: Scaling to Emphasize Attention for Long-Context Retrieval." pith.science (2026). https://pith.science/paper/D2QEH4SG

@misc{pith2026250115225,
  author       = {Pith},
  title        = {Pith review of: SEAL: Scaling to Emphasize Attention for Long-Context Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D2QEH4SG}},
  note         = {Machine review of arXiv:2501.15225}
}

read the original abstract

While many advanced LLMs are designed to handle long sequence data, we can still observe notable quality degradation even within the sequence limit. In this work, we introduce a novel approach called Scaling to Emphasize Attention for Long-context retrieval (SEAL), which enhances the retrieval performance of large language models (LLMs) over long contexts. We observe that specific attention heads are closely tied to long-context retrieval, showing positive or negative correlation with retrieval scores, and adjusting the strength of these heads boosts the quality of LLMs in long context by a large margin. Built on this insight, we propose a learning-based mechanism that leverages generated data to emphasize these heads. By applying SEAL, we achieve significant improvements in long-context retrieval performance across various tasks and models. Additionally, when combined with existing training-free context extension techniques, SEAL extends the contextual limits of LLMs while maintaining highly reliable outputs.

Figures

Figures reproduced from arXiv: 2501.15225 by the authors.

**Figure 2.** Changes in retrieval scores (%) with different [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** (a) The overview of the proposed SEAL method. SEAL-H (head) or SEAL-C (channel) can be used [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Effects of attention heads on logits: Left: Di [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Comparison of Needle-in-a-Haystack perfor [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Line retrieval task scores for context length [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: The results of Needle-in-a-Haystack in Llama [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 9.** Figure 9: Line retrieval results when using fewer sam [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

**Figure 10.** Figure 10: Comparison of Needle-in-a-Haystack perfor [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 5 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

arXiv 2023
[4]

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119--3137

2024
[5]

bloc97. 2023. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/ NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

2023
[6]

Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 36:75067--75096

work page 2023
[7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS, 33:1877--1901

2020
[8]

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

arXiv 2023

Show all 28 references

[9]

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6

2023
[10]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[11]

Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12

2021
[12]

Javier Ferrando, Gabriele Sarti, Arianna Bisazza, and Marta R Costa-juss \`a . 2024. A primer on the inner workings of transformer-based language models. arXiv preprint arXiv:2405.00208

2024 arXiv
[13]

Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qianguosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang. 2024. Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Pr...

2024
[14]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

2020 arXiv
[15]

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654

2024 arXiv
[16]

Lora: Low-rank adaptation of large language models

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations
[17]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[18]

Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, and Xia Hu. 2024. Llm maybe longlm: Self-extend llm context window without tuning. arXiv preprint arXiv:2401.01325

2024 arXiv
[19]

G Kamradt. 2023. Needle in a haystack - pressure testing llms. https://github.com/gkamradt/LLMTest_NeedleInAHaystack

2023
[20]

Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. How long can context length of open-source llms truly promise? In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following

2023
[21]

Tom Lieberum, Matthew Rahtz, J \'a nos Kram \'a r, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. 2023. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla. arXiv preprint arXiv:2307.09458

2023 arXiv
[22]

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 a . Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157--173

2024
[23]

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. 2024 b . Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.09353

2024 arXiv
[24]

Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071

2023 arXiv
[25]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

2019
[26]

Outlier dimensions encode task specific knowledge

William Rudman, Catherine Chen, and Carsten Eickhoff. Outlier dimensions encode task specific knowledge. In The 2023 Conference on Empirical Methods in Natural Language Processing

2023
[27]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[28]

A Vaswani. 2017. Attention is all you need. Advances in Neural Information Processing Systems

2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebr \'o n, and Sumit Sanghai. 2023. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245

arXiv 2023

[4] [4]

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2024. Longbench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119--3137

2024

[5] [5]

bloc97. 2023. https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkaware_scaled_rope_allows_llama_models_to_have/ NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation

2023

[6] [6]

Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. 2023. Quantizable transformers: Removing outliers by helping attention heads do nothing. Advances in Neural Information Processing Systems, 36:75067--75096

work page 2023

[7] [7]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS, 33:1877--1901

2020

[8] [8]

Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595

arXiv 2023