Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

GEM: Empowering LLM for both Embedding Generation and Language Understanding

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A decoder-only LLM can produce high-quality text embeddings and keep its reasoning, by training it to compress text into special bottleneck tokens.

desk verdict GEM has a clean recipe and strong small-model results, but its own 8B MMLU drop contradicts the abstract's 'minimal impact' claim. read the letter →

arxiv 2506.04344 v1 pith:JDCZWBYN submitted 2025-06-04 cs.CL cs.LG

Caojin Zhang , Qiang Zhang , Ke Li , Sai Vidyaranya Nuthalapati , Benyu Zhang , Jason Liu , Serena Li , Lizhu Zhang
show 1 more author
Xiangjun Fan
This is my paper
classification cs.CLcs.LG
keywords textembeddingsdecoder-onlyLLMsattentionmaskingbottleneckcompressioncontrastivelearningnext-tokenpredictionMTEBretrievalaugmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a decoder-only LLM can be taught to produce high-quality text embeddings without giving up its ability to generate text or answer questions. The recipe is a small change to training: insert one or more special tokens into the input, let the attention mask forbid later tokens from looking back at the original text, and train the model to predict the remaining text from the compressed special tokens. On the MTEB embedding benchmark, the same model goes from 18.29 to 54.35 average on 15 tasks for a 1B model, while MMLU over language understanding drops only from 31.7 to 28.3. The appeal is practical: if true, retrieval-augmented systems no longer need a separate embedding model, and the transformation requires only 32,000 rows of self-supervised data.

What carries the argument

The load-bearing piece is a modified causal attention mask that creates a bottleneck at the special token(s). The mask is the usual lower-triangular causal mask with all entries before the first special token zeroed out, so suffix tokens can attend only to the special tokens and to themselves; the special tokens cannot attend to one another. This forces the model to route all information from the prefix through the special tokens, and the mixed training loss (mostly ordinary next-token prediction, plus contrastive learning on the special-token embeddings) teaches the resulting hidden state to act as a summary embedding. The paper's ablations show the mix ratio and the number of special tokens both matter, with more tokens helping retrieval and a roughly 80/20 mix preserving MMLU.

What would settle it

A controlled ablation can settle it: train with the same data and the same losses but with the attention bottleneck removed, letting suffix tokens attend to the prefix normally; if the MTEB retrieval score does not drop, the bottleneck is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The central discovery is that the bottleneck should be created by the attention mask rather than by changing the architecture. Given a text, GEM inserts a special token at a random position, splitting the input into prefix and suffix; the mask blocks suffix tokens from attending to prefix tokens, so the only path from the prefix to the suffix runs through the special tokens. Training with next-token prediction then forces the model to compress the prefix into those tokens, and their final-layer hidden states serve as the text embedding. A self-supervised contrastive loss, using a randomly dropped-out version of the same prefix as a positive pair, sharpens the embedding further. The paper reports that on the 1B Llama model this raises the 15-task MTEB average from 18.29 to 54.35, with retrieval climbing from 4.38 to 25.6, while MMLU falls only from 31.7 to 28.3.

Load-bearing premise

The assumption the method rests on is that a one- or few-token bottleneck, trained by next-token prediction, actually carries the semantic content of the prefix; the appendix's reconstruction examples show the compression becomes visibly lossy for inputs much longer than about 30 tokens.

Editorial extensions

If this is right

  • A single model can serve both retrieval and generation, so a RAG pipeline no longer needs a separately trained embedding model with a different notion of what a query means.
  • The training recipe is small and self-supervised: about 32,000 rows, with no MTEB or MMLU data exposed, which makes the transformation cheap to apply during post-training or fine-tuning.
  • The trick transfers across model families and scales; the paper shows consistent MTEB gains on two families from 1B to 8B, with the 3B Llama variant reaching a 59.06 average.
  • The same bottleneck state can act as a compressed representation of its prefix, so text reconstruction and context compression come out of the same mechanism that produces embeddings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension not tested in the paper: the bottleneck representation could be combined with supervised contrastive pairs on domain-specific data, which would likely sharpen retrieval on that domain without retraining the whole model.
  • The Appendix B reconstruction results suggest that a single special token is a lossy summary beyond roughly 30 tokens, so a hierarchical or chunked special-token design would be a natural next step for long-document retrieval and question answering.
  • If the method transfers, one model could embed documents and answer queries in a single deployment, which would simplify agent systems and streaming long-context services that today keep an encoder and a decoder running side by side.
  • A controlled test the paper does not run is to isolate the bottleneck mask from the contrastive loss; comparing models trained with and without the mask on identical data would show how much of the embedding gain is due to compression rather than to contrastive training.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes GEM, a self-supervised fine-tuning method that inserts special token(s) into the input sequence and modifies the attention mask to create a bottleneck between prefix and suffix tokens, forcing the model to compress the prefix into the special tokens. The model is trained with a mix of standard next-token prediction (on raw text) and next-token prediction on 'prefix special suffix' sequences, plus a contrastive loss that treats the embeddings of the special tokens from randomly masked prefixes as positive pairs. The resulting model can be used as both a generative LLM and a text embedding model. The authors evaluate GEM on Llama 3.2 1B/3B, Llama 3 8B, and Mistral 7B on MTEB and MMLU. They report large MTEB improvements, e.g., the 15-task average on Llama 3.2 1B rises from 18.29 to 54.35, while MMLU drops modestly at 1B/3B but significantly at 8B (66.7 to 57.24).

Significance. If the results are reproducible, GEM is a simple, data-efficient (32k rows) recipe for unifying embedding and generation, which is practically relevant for RAG and agentic applications. The ablations on mix ratio, contrastive loss, and number of special tokens are informative, and the cross-family transfer of hyperparameters is a strength. However, the paper currently lacks a full MTEB aggregate score, the reproduced baselines for two models are far below cited values, and the 8B MMLU drop directly contradicts the abstract's claim of minimal impact. These issues need to be resolved before the central claim of preserving language understanding 'at scale' can be accepted.

major comments (3)
  1. [§5.4, Table 5 (and Table 1)] The 8B MMLU drop from 66.7 to 57.24 (≈14% relative) is described by the authors themselves as 'a significant drop' (§5.4). This directly contradicts the abstract's claim that GEM has 'minimal impact on MMLU' and 'maintains original NLP performance' (abstract, introduction). The speculation that different hyperparameters would help is not supported by any experiment in the paper. Since the paper claims the method generalizes across model sizes ('ranging from 1B to 8B') and to 'any existing LLMs' (Section 1), the 8B result is a concrete counterexample to the central claim. Please either provide 8B-specific training that preserves MMLU within a small margin, or substantially revise the claim to acknowledge that language understanding is not preserved at the largest scale tested.
  2. [Table 1, §4.3] The reproduced baselines for Llama 3 8B and Mistral 7B are far below the values cited from Muennighoff et al. and BehnamGhader et al. For example, Llama 3 8B clustering is 25.57 (reproduced) versus 36.84 (cited), and retrieval is 11.86 versus 15.17 (cited). The paper attributes this to 'differences in the protocol or changes of access to the benchmark' but does not specify what protocol was used for either the baselines or GEM. Because the improvement from GEM is computed against these reproduced baselines, a systematic protocol difference (e.g., pooling method, prompt format, MTEB version) could inflate the reported gains. The paper must report the exact evaluation protocol for all models, including pooling, instructions, and benchmark version, and demonstrate that GEM and the reproduced baselines are evaluated under identical conditions.
  3. [Table 1, §5, abstract] The paper never reports the standard overall MTEB average score. Table 1 provides task-group averages (clustering, reranking, pair classification, STS, retrieval, summarization, classification) and Table 5 reports a 15-task average used for ablations, but the abstract's claims of 'state-of-the-art text embedding capabilities' and the introduction's claim of results 'comparable to SoTA embedding models' require the full MTEB aggregate over all datasets, which is the metric used by the MTEB leaderboard. Without an overall MTEB score, the reader cannot determine whether the improvements on the selected task groups translate to a state-of-the-art result. Please report full MTEB averages for GEM and the comparison models, or clearly state the task subset and justify its use as the primary metric.
minor comments (6)
  1. [Equation (1)] The denominator should read Σ_{d−∈N} exp(λ s(q, d−)); as written, the second term repeats s(q, d+) instead of using the negative embeddings.
  2. [§4.1] MMLU stands for 'Massive Multitask Language Understanding' and covers 57 tasks, not '57 languages'; the current description mischaracterizes both the acronym and the benchmark.
  3. [§3.1, Table 1] The number of special tokens used in the main experiments is not stated. Table 4 ablates 1, 2, 5, and 10 tokens, but Table 1 does not indicate which configuration produced the reported numbers.
  4. [§3.2] The 'random dropout in prefix' used to create contrastive positive pairs is not precisely defined; please specify the dropout probability and granularity (token-level, span-level, or embedding dropout) used in the experiments.
  5. [§4.2] The learning-rate schedule for the two losses is ambiguous: the text says 'learning rate 1e−4 for NTP fintuning and 1e−5 for contrastive learning with cosine scheduler,' but it does not clarify whether these are separate optimizers or how the α schedule interacts with them.
  6. [Table 1, §4.3] The table uses 'GEM LLAMA 3.1-1B' while the text and Table 5 refer to 'Llama 3.2-1B'; similarly, 'S-Llama-1.3B' is abbreviated inconsistently. Please unify model naming throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: GEM's MTEB/MMLU results are out-of-sample transfer evaluations, not reductions to training targets; the 8B MMLU drop is a correctness/consistency issue, not a circularity issue.

full rationale

The paper's derivation chain is self-contained against its evidence. GEM is trained on the public SIMCSE dataset with a mixed next-token-prediction and contrastive objective, then evaluated on held-out MTEB and MMLU benchmarks; the paper explicitly states that no MTEB or MMLU data is exposed during fine-tuning, so the reported embedding gains are genuine out-of-sample transfer results rather than fitted predictions. The special-token bottleneck attention mask is an architectural design choice, not a parameter fitted to the MTEB target, and the contrastive objective uses self-supervised dropout positives rather than benchmark labels. Hyperparameters are tuned on Llama 3.2-1B and then transferred to other model families; this is a generalization concern, but it does not make the reported results equivalent to their inputs by construction. The cited prior work (Gist tokens, VoCo-Llama, ICAE, Llm2vec, GritLM) is external, and no load-bearing premise is justified solely by a self-citation or an imported uniqueness theorem. The notable discrepancy between the abstract's claim of 'minimal impact on MMLU' and the paper's own Table 5 and Section 5.4, where the 8B model shows a significant 9.46-point MMLU drop, is an internal consistency and correctness issue, not circular reasoning, because the evaluation is still an independent measurement of a claimed capability trade-off. No circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method relies on standard transformer mechanisms and the SimCSE positive-pair assumption. All hyperparameters (p, alpha schedule, learning rates, special-token count) are tuned on a single model without sensitivity analysis, and the paper does not report the key detail of special-token count used in the main results. No new physical or conceptual entities are introduced beyond trainable special token embeddings.

free parameters (6)
  • special token mix ratio p = 0.8
    Chosen from ablation on Llama 3.2-1B (Table 2); balances MTEB gain against MMLU preservation.
  • alpha schedule = 0 for first 100 iterations, then ramps to 1
    Selected via ablations (Table 3); NTP-only start prevents catastrophic forgetting.
  • contrastive temperature lambda = learnable, init log 20, clamped to [0, log 100]
    Follows Radford et al. 2021; used in Eq. (1).
  • learning rates = 1e-4 for NTP, 1e-5 for contrastive
    Stated in Section 4.2 without sensitivity analysis.
  • batch size and max sequence length = 32 and 512
    Memory convenience choices in Section 4.2.
  • number of special tokens in main results = not stated; ablation shows 10 tokens best on MTEB
    Table 4 shows 10 tokens yield best retrieval, but main results do not specify the count used.
assumptions (5)
  • standard math Causal decoder-only transformers can be repurposed as encoders by extracting hidden states at special token positions.
    Foundation of the method, Section 3.
  • domain assumption Blocking suffix attention to the prefix forces the model to compress prefix information into the special tokens.
    Core mechanism, Section 3.1 and Figure 2; not proven, tested via downstream performance.
  • domain assumption Random dropout on the prefix gives valid positive pairs for contrastive learning (SimCSE assumption).
    Section 3.2; standard SimCSE premise.
  • domain assumption Pretrained LLM weights from Llama and Mistral initialize the model, and their tokenizer can be extended with new special tokens.
    Section 4.2 uses public checkpoints; vocabulary extension details omitted.
  • domain assumption MMLU five-shot under the Llama 3.1 protocol approximates language understanding and preservation of generation.
    Section 4.1; generation quality itself is not evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GEM: Empowering LLM for both Embedding Generation and Language Understanding." pith.science (2026). https://pith.science/paper/JDCZWBYN

@misc{pith2026250604344,
  author       = {Pith},
  title        = {Pith review of: GEM: Empowering LLM for both Embedding Generation and Language Understanding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JDCZWBYN}},
  note         = {Machine review of arXiv:2506.04344}
}
read the original abstract

Large decoder-only language models (LLMs) have achieved remarkable success in generation and reasoning tasks, where they generate text responses given instructions. However, many applications, e.g., retrieval augmented generation (RAG), still rely on separate embedding models to generate text embeddings, which can complicate the system and introduce discrepancies in understanding of the query between the embedding model and LLMs. To address this limitation, we propose a simple self-supervised approach, Generative Embedding large language Model (GEM), that enables any large decoder-only LLM to generate high-quality text embeddings while maintaining its original text generation and reasoning capabilities. Our method inserts new special token(s) into a text body, and generates summarization embedding of the text by manipulating the attention mask. This method could be easily integrated into post-training or fine tuning stages of any existing LLMs. We demonstrate the effectiveness of our approach by applying it to two popular LLM families, ranging from 1B to 8B parameters, and evaluating the transformed models on both text embedding benchmarks (MTEB) and NLP benchmarks (MMLU). The results show that our proposed method significantly improves the original LLMs on MTEB while having a minimal impact on MMLU. Our strong results indicate that our approach can empower LLMs with state-of-the-art text embedding capabilities while maintaining their original NLP performance

Figures

Figures reproduced from arXiv: 2506.04344 by the authors.

Figure 1
Figure 1. An overview of the proposed method. The proposed method takes tokenized text as input, then insert special [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the next token prediction with special tokens. The attention mask allows the suffix tokens [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Illustration of the attention mask for adding k special tokens after input of m tokens. Here we have m tokens [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HT-Transformer: Event Sequences Classification by Accumulating Prefix Information with History Tokens

    cs.LG 2025-08 conditional novelty 6.0 of 10

    Introducing history tokens with sparse attention masks during next-token pretraining improves transformer event-sequence classification, but the effect is confounded by the use of an appended token at inference.

Reference graph

Works this paper leans on

34 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    Retrieval- Augmented Generation for Knowledge - Intensive NLP Tasks , April 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval- Augmented Generation for Knowledge - Intensive NLP Tasks , April 2021. URL http://arxiv.org/abs/2005.11401. arXiv:2005.11401 [cs]

    arXiv 2021

  2. [2]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, V...

    2019

  3. [3]

    Text and code embeddings by contrastive pre-training

    Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005, 2022

    arXiv 2022

  4. [4]

    Fine-tuning llama for multi-stage text retrieval

    Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421--2425, 2024

    work page 2024

  5. [5]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533, 2022

    arXiv 2022

  6. [6]

    SimCSE : Simple Contrastive Learning of Sentence Embeddings , May 2022

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE : Simple Contrastive Learning of Sentence Embeddings , May 2022. URL http://arxiv.org/abs/2104.08821. arXiv:2104.08821 [cs]

    arXiv 2022

  7. [7]

    C-pack: Packaged resources to advance general chinese embedding, 2023

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged resources to advance general chinese embedding, 2023

    2023

  8. [8]

    Repetition Improves Language Model Embeddings , February 2024

    Jacob Mitchell Springer, Suhas Kotha, Daniel Fried, Graham Neubig, and Aditi Raghunathan. Repetition Improves Language Model Embeddings , February 2024. URL http://arxiv.org/abs/2402.15449. arXiv:2402.15449 [cs]

    arXiv 2024

Show all 34 references
  1. [9]

    Generative representational instruction tuning

    Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. arXiv preprint arXiv:2402.09906, 2024

    2024 arXiv

  2. [10]

    Llm2vec: Large language models are secretly powerful text encoders

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961, 2024

    2024 arXiv

  3. [11]

    MTEB : Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Lo \" c Magne, and Nils Reimers. MTEB : Massive text embedding benchmark. arXiv preprint arXiv:2210.07316, 2022

    2022 arXiv

  4. [12]

    Smarter, Better , Faster , Longer : A Modern Bidirectional Encoder for Fast , Memory Efficient , and Long Context Finetuning and Inference , December 2024

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, Better , Faster , Longer : A Modern Bidirectiona...

    2024 arXiv

  5. [13]

    Learning to Compress Prompts with Gist Tokens , February 2024

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to Compress Prompts with Gist Tokens , February 2024. URL http://arxiv.org/abs/2304.08467. arXiv:2304.08467 [cs]

    2024 arXiv

  6. [14]

    Voco-llama: Towards vision compression with large language models

    Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. Voco-llama: Towards vision compression with large language models. arXiv preprint arXiv:2406.12275, 2024

    2024 arXiv

  7. [15]

    Vasilakos, and Thippa Reddy Gadekallu

    Gokul Yenduri, Ramalingam M, Chemmalar Selvi G, Supriya Y, Gautam Srivastava, Praveen Kumar Reddy Maddikunta, Deepti Raj G, Rutvij H Jhaveri, Prabadevi B, Weizheng Wang, Athanasios V. Vasilakos, and Thippa Reddy Gadekallu. Generative pre-trained transformer: A comprehensive re...

    2023 arXiv

  8. [16]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  9. [17]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  10. [18]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  11. [19]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  12. [20]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  13. [21]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing...

    2020

  14. [22]

    Li, Madian Khabsa, Han Fang, and Hao Ma

    Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity, 2020. URL https://arxiv.org/abs/2006.04768

    2020 arXiv

  15. [23]

    Ring attention with blockwise transformers for near-infinite context, 2023

    Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023. URL https://arxiv.org/abs/2310.01889

    2023 arXiv

  16. [24]

    Efficient streaming language models with attention sinks, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv.org/abs/2309.17453

    2024 arXiv

  17. [26]

    SepLLM : Accelerate Large Language Models by Compressing One Segment into One Separator , December 2024

    Guoxuan Chen, Han Shi, Jiawei Li, Yihang Gao, Xiaozhe Ren, Yimeng Chen, Xin Jiang, Zhenguo Li, Weiyang Liu, and Chao Huang. SepLLM : Accelerate Large Language Models by Compressing One Segment into One Separator , December 2024. URL http://arxiv.org/abs/2412.12094. arXiv:2412....

    2024 arXiv

  18. [27]

    Adapting Language Models to Compress Contexts , November 2023

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting Language Models to Compress Contexts , November 2023. URL http://arxiv.org/abs/2305.14788. arXiv:2305.14788 [cs]

    2023 arXiv

  19. [28]

    A Silver Bullet or a Compromise for Full Attention ? A Comprehensive Study of Gist Token -based Context Compression , December 2024

    Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu, and Zhicheng Dou. A Silver Bullet or a Compromise for Full Attention ? A Comprehensive Study of Gist Token -based Context Compression , December 2024. URL http://arxiv.org/abs/2412.17483. arXiv:2412....

    2024 arXiv

  20. [29]

    In-context autoencoder for context compression in a large language model

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945, 2023

    2023 arXiv

  21. [30]

    Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish

    Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats L. Richter, Quentin Anthony, Timothée Lesort, Eugene Belilovsky, and Irina Rish. Simple and Scalable Strategies to Continually Pre -train Large Language Models , March 2024. URL http://arxiv.org/abs/2403.08763. arXiv:2403.08763 [cs]

    2024 arXiv

  22. [31]

    Dense Passage Retrieval for Open - Domain Question Answering , September 2020

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open - Domain Question Answering , September 2020. URL http://arxiv.org/abs/2004.04906. arXiv:2004.04906 [cs]

    2020 arXiv

  23. [32]

    Learning Transferable Visual Models From Natural Language Supervision , February 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision , February 2021. URL ...

    2021 arXiv

  24. [33]

    Aligning ai with shared human values

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Representations (ICLR), 2021 a

    2021

  25. [34]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021 b

    2021

  26. [35]

    Efficient continual pre-training by mitigating the stability gap, 2024

    Yiduo Guo, Jie Fu, Huishuai Zhang, Dongyan Zhao, and Yikang Shen. Efficient continual pre-training by mitigating the stability gap, 2024. URL https://arxiv.org/abs/2406.14833

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.