Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

A trainable encoder compresses prompts to a quarter of their tokens, and an unmodified LLM still answers with nearly full-text accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 08:23 UTC pith:M5VNWOCU

load-bearing objection Solid plug-and-play context compression paper with released artifacts; main 4x compression claim holds, but the context-extension result is confounded by task-matched synthetic fine-tuning data and needs a full-text control. the 3 major comments →

arxiv 2510.20535 v2 pith:M5VNWOCU submitted 2025-10-23 cs.CL cs.AI

ARC-Encoder: learning compressed text representations for large language models

Hippolyte Pilchen , Edouard Grave , Patrick P\'erez This is my paper
classification cs.CL cs.AI
keywords context compressioncontinuous representationsfrozen decodersoft compressionpoolingfew-shot learninglong-context understandingmulti-decoder adaptation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

ARC-Encoder is a separate, trainable encoder that turns a block of text into a smaller set of continuous vectors and hands those vectors to a frozen decoder LLM in place of normal token embeddings. The paper argues that this arrangement preserves most of the model's ability to answer, translate, and summarize — at 4x compression, the frozen decoder nearly matches its own performance on the full text, while cutting prefill FLOPs by roughly 1.8x. The same design extends a small chat decoder's usable context window by 8x without changing its weights, and a single shared encoder can serve multiple decoders with about 15M adapter parameters each. If this holds, context compression no longer requires sacrificing a model's general abilities through decoder fine-tuning.

Core claim

On the paper's own terms, the discovery is that a frozen decoder LLM can consume learned, pooled continuous vectors almost as fluently as its own embeddings. ARC-Encoder is a non-causal transformer encoder (about 3 billion parameters) whose final self-attention layer merges consecutive queries by averaging, so n tokens become n/x vectors; a two-layer MLP with a dimensionality bottleneck projects those vectors into the decoder's hidden space. Two alternating pretraining tasks — reconstructing the original tokens and continuing text after compressed segments — align the encoder's outputs with the decoder, and fine-tuning on mixed QA, translation, and summarization data adapts it to tasks. Repo

What carries the argument

The load-bearing mechanism is pooled-query soft compression plus a bottleneck projector. In the encoder's last self-attention layer, queries are averaged over consecutive tokens while keys and values stay untouched, so the output sequence is exactly n/x vectors that still carry a locality-aware summary of the whole context. A 2-layer MLP without activation (3072→2048→4096) maps those vectors into the frozen decoder's embedding dimension, and learned task-marker tokens tell the decoder whether to reconstruct or continue. This keeps the pooling factor constant regardless of input length, unlike fixed-size memory tokens, and makes the compressed vectors look like ordinary embeddings to the deco

Load-bearing premise

The claim assumes the compressed vectors are genuinely what the decoder benefits from — not the synthetic fine-tuning data or benchmark overlap — and that a frozen decoder can treat learned vectors as ordinary embeddings across tasks.

What would settle it

Fine-tune the same decoder on the same synthetic data with ordinary full-text contexts and compare on the long-context benchmarks: if full-text matches or beats ARC-Encoder, the context-extension result is explained by the data, not by compression. For the core compression claim, remove the final pooling step from the encoder but keep everything else; if downstream accuracy is unchanged, pooling is not the operative mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Corpora used repeatedly in RAG can be encoded once and stored: compressed English Wikipedia fits in roughly 80 GiB at default settings, with only a small accuracy hit, or 20 GiB at stronger compression while still beating the closed-book baseline.
  • A decoder's effective context window expands by the pooling factor without any change to its weights — chunks are compressed in parallel and concatenated — so context extension becomes a training-time property of the encoder, not the LLM.
  • Adapting the system to a new decoder requires training only an MLP and special tokens (about 15M parameters), so a single encoder can serve multiple models at once.
  • The largest accuracy gains over closed-book appear on tasks where the decoder cannot fall back on parametric memory (reading comprehension, summarization), suggesting the compression preserves the task-relevant content rather than relying on memorized knowledge.
  • Higher pooling factors (8x) trade a few points of accuracy for more compute savings, and a 32x factor degrades sharply, marking a practical compression-efficiency frontier.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because a full transformer encoder computes the compressed tokens, system-level FLOPs still include the encoder pass; the 1.8x gain is for decoder prefill, so the net benefit approaches that only when contexts are reused or when comparing decoder-side costs. A fair end-to-end accounting would compare encoder-plus-decoder against the full-text decoder.
  • The long-context improvement is not separately identified from the synthetic fine-tuning data: the paper only compares against a chat baseline that never saw that data, so a controlled full-text baseline trained on the same samples would isolate the compression mechanism.
  • If pooling relies on locality, adaptive pooling that respects semantic boundaries rather than fixed consecutive groups could improve fidelity at the same ratio — though the paper's k-means experiment failed, suggesting naive adaptive grouping is not enough.
  • The multi-decoder result points toward a shared hidden-state geometry across similar LLMs; a direct test would be whether an encoder pretrained on two decoders transfers to a third with only projector training better than training from scratch, which the OLMo adaptation partially supports.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. ARC-Encoder is an external text encoder that compresses an input context into continuous pooled vectors, which are injected into a frozen decoder LLM in place of token embeddings. The encoder uses a truncated non-causal LLM (Llama3.2 3B) and pools queries in its last self-attention layer to achieve a fixed compression factor (typically 4x or 8x), followed by a small MLP projector to the decoder's embedding dimension. Training alternates reconstruction and continuation objectives, followed by task-specific fine-tuning with in-context examples. The authors report experiments on six context-compression benchmarks with two base decoders (Mistral 7B, Llama3.1 8B), long-context benchmarks with Llama2 Chat, ablations of design choices, multi-decoder sharing, and a memory/storage analysis of precomputed compressed Wikipedia. The main claims are that ARC-Encoder nearly matches an open-book baseline at 4x compression without modifying the decoder, achieves state-of-the-art among methods that do not fine-tune the decoder, and can extend a decoder's effective context window.

Significance. If the empirical claims hold, the paper makes a useful practical contribution: it offers a plug-and-play, decoder-agnostic context compression method with released code, data, and models, and the multi-decoder sharing result is interesting. The paper is also careful in several ways: evaluation train sets are excluded from fine-tuning, exact match is used rather than free-form generation, and a common protocol is applied to reimplemented baselines. However, the headline 'context extension' result is currently confounded, and the state-of-the-art claim depends on reimplementations whose fidelity is not externally validated. The paper's strengths include a consistent evaluation protocol, a thorough ablation study, and a willingness to release artifacts, all of which raise the value of the work if the concerns below are addressed in revision.

major comments (3)
  1. [§4.3, Table 3] The long-context contribution is not separately identified. ARC8-Encoder + Llama2 Chat is compared with Llama2 Chat (4k), Llama2-32k Instruct, and CEPED, none of which were trained on the synthetic QA/summarization/paraphrase data described in Appendix C.4.2. The paper itself concedes that the advantage 'may stem partly from our synthesized fine-tuning dataset, which matches the answer-length distributions of evaluation benchmarks.' Because the comparison changes both the compression mechanism and the training data, the observed gains (e.g., NQA 27.5 vs. 16.1) cannot be attributed to compression. To support the claim that an external encoder extends a frozen decoder's context, the authors need a control that receives the same fine-tuning data without compression, or an ablation that keeps the data fixed while varying only the compression mechanism. Without such a control, Table 3 only sh
  2. [§4.2, Table 1, Appendix D.1] The 'state-of-the-art' claim rests on reimplemented baselines (ICAE-like, xRAG-like, PISCO-like), and PISCO's official code is unavailable. The reimplementations use the authors' fine-tuning dataset and template, and Appendix D.1 describes several deviations from the original methods (e.g., PISCO without sequence-level distillation, ICAE with a different fine-tuning dataset). A reader cannot tell whether the gap between ARC-Encoder and these baselines reflects the method or the reimplementation. Please report, for at least the baselines with available official checkpoints, the original models under the same evaluation protocol, or provide a validation of the reimplementations against originally reported numbers on a common setting. The main results in Table 1 are also single-seed; the ablations in Tables 5 and 8 average over three seeds, and the same practice should be applied to the hea
  3. [§4.2 vs. §B.1/B.2] The abstract and §4.2 state a '×1.8 gain of prefilling FLOPs' for ARC4, while Appendix B.1's formula yields approximately 1.5× smaller FLOPs for ARC4 and 1.9× for ARC8 in the main-table setting; §B.2 profiling shows time/TFLOPs but does not directly report the 1.8× figure. Please reconcile the theoretical estimate, the profiling measurement, and the main-text claim, and clearly state whether the factor refers to FLOPs, wall-clock time, or prefilling tokens.
minor comments (4)
  1. [Throughout] Please copyedit for typos, e.g., 'Simarly' in Related Work and 'looses' in §3.3. Also consider rephrasing '×1.8 gains' to '1.8× gain'.
  2. [Table 3] The row 'ARC8-Encoder + Llama2 Chat, 4k (32k//8)' is confusing: specify that 32k input tokens are compressed to 4k decoder tokens. 'Max. Tokens' currently mixes input-side and decoder-side token counts, which obscures the comparison with CEPED.
  3. [Abstract/§4.2] 'State-of-the-art on several benchmarks' should be qualified: among decoder-frozen methods under the authors' evaluation protocol, with the specific benchmarks listed. As written, the abstract overstates the scope of the comparison.
  4. [§4.1] The statement that the MLP has 'less than 1% of the encoder parameters' is useful, but please also give the absolute parameter count for the standard configuration (15M is mentioned only for the OLMo adaptation).

Circularity Check

0 steps flagged

No significant circularity: the central claims are empirical benchmark results with external baselines and excluded train sets; the long-context confound is a validity limitation, not a definitional reduction.

full rationale

The paper's main contributions are empirical: an encoder is pretrained and fine-tuned, then evaluated on standard benchmarks. The Section 4.2 claims are supported by comparisons against external baselines that the authors re-implemented ('We re-implemented the last three baselines using our decoder, fine-tuning dataset, and interleaved fine-tuning task format'), and the fine-tuning data 'explicitly excluding the training sets of our evaluation benchmarks.' This prevents the main context-compression results from being circular fits to the evaluation labels. Hyperparameter choices such as pooling-factor pairs are selected using the evaluation suite ('For our final ARC-Encoder we follow the best pretraining / fine-tuning pooling factor pairs as shown in Fig. 5'), but this is model selection/tuning rather than a prediction that reduces to a fitted parameter by construction. The long-context section contains a genuine confound, which the paper itself flags: 'This advantage may stem partly from our synthesized fine-tuning dataset, which matches the answer-length distributions of evaluation benchmarks, as other models do not fine-tune specifically on these tasks.' Because ARC8-Encoder is compared only to baselines that did not receive this task-matched synthetic data, the improvement over Llama2 Chat is not uniquely attributable to compression; a full-text baseline trained on the same data would be needed. However, this is an identification/validity problem, not circularity: the synthetic data does not contain the evaluation answers, the encoder must still generalize to new documents and questions, and no equation or construction makes the reported score equal to the training objective. There is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via prior work. The FLOPs analysis in Appendix B.1 is a standard complexity calculation, not a fit. Overall, the central derivation is self-contained and the long-context limitation should be addressed experimentally rather than treated as circular reasoning.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The method is empirical: no derivation is claimed. The free parameters are hyperparameters tuned on the evaluation suite; the axioms are design assumptions validated by experiments rather than theorems. The invented entities are trained representations, not physical postulates.

free parameters (5)
  • pooling factor pair (pretrain/finetune) = 8/4 for ARC4; 8/8 for ARC8
    Chosen from Fig. 5 ablation over {2,4,8,16,32} on the same benchmark suite; final models use pretrain PF=8 and finetune PF=4 for ARC4.
  • reconstruction ratio in pretraining = 0.2
    Tab. 4 shows 20% reconstruction maximizes downstream average; 0% and 100% degrade.
  • MLP bottleneck dimension = 2048
    Hand-chosen middle dimension between encoder 3072 and decoder 4096; memory analysis explores other values but main results use 2048.
  • encoder truncation = 2 layers removed from Llama3.2 3B
    Tab. 5 ablation shows truncate-2 balances params and performance; truncate-0 performs slightly better but is larger.
  • number of in-context examples in fine-tuning = 5 (with 3-4 for some subsets)
    Evaluations are 5-shot; fine-tuning uses the same template with up to # in-context examples, aligning train and eval.
axioms (5)
  • domain assumption A frozen decoder can directly consume continuous vectors as token embeddings without any architectural change.
    Central to the method; empirically validated by Tables 1-3, but assumed rather than proven.
  • domain assumption Averaging consecutive query vectors in the last self-attention layer preserves enough information for downstream tasks.
    Design choice from the systematic study; supported by ablations (Tab. 8), not theoretically grounded.
  • domain assumption Reconstruction and continuation pretraining align the encoder's output space with the decoder's embedding space.
    Inherited from memory-token work (Ge et al. 2024); the paper shows pretraining is essential (Tab. 4).
  • domain assumption The hidden-state spaces of different decoder LLMs (Llama3.1, Mistral, OLMo) are sufficiently compatible for a shared encoder with per-decoder MLP projectors.
    Supported by Tab. 1, 2 and A.3, but not guaranteed beyond tested models.
  • domain assumption Transformer FLOP estimates in Appendix B.1 assume d proportional to sqrt(N/n_layers) and ignore norms.
    Used to claim 1.8x prefill speedup; a rough approximation, not a bound.
invented entities (2)
  • Pooled continuous tokens (compressed text representations) independent evidence
    purpose: Replace token embeddings in a frozen decoder to shorten context.
    Postulated representational interface; downstream benchmark performance provides falsifiable evidence.
  • <Cont>/<Rec> special tokens independent evidence
    purpose: Tell the decoder whether a compressed sequence is for continuation or reconstruction; also decoder-specific in multi-decoder training.
    Learned task markers; ablations show they help generalization to new decoders.

pith-pipeline@v1.3.0-alltime-deepseek · 22015 in / 11946 out tokens · 103418 ms · 2026-08-04T08:23:57.599368+00:00 · methodology

0 comments
read the original abstract

Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs. Context compression techniques can reduce these costs, but the most effective approaches require fine-tuning the target model or even modifying its architecture. This can degrade its general abilities when not used for this specific purpose. Here we explore an alternative approach: an encoder that compresses the context into continuous representations which replace token embeddings in decoder LLMs. First, we perform a systematic study of training strategies and architecture choices for the encoder. Our findings led to the design of an Adaptable text Representations Compressor, named ARC-Encoder, which outputs $x$-times fewer continuous representations (typically $x\!\in\!\{4,8\}$) than text tokens. We evaluate ARC-Encoder across a variety of LLM usage scenarios, ranging from in-context learning to context window extension, on both instruct and base decoders. Results show that ARC-Encoder achieves state-of-the-art performance on several benchmarks while improving computational efficiency at inference. Finally, we demonstrate that our models can be adapted to multiple decoders simultaneously, allowing a single encoder to generalize across different decoder LLMs. This makes ARC-Encoder a flexible and efficient solution for portable encoders that work seamlessly with multiple LLMs. We release a training code at https://github.com/kyutai-labs/ARC-Encoder , fine-tuning dataset and pretrained models are available at https://huggingface.co/collections/kyutai/arc-encoders-68ee18787301407d60a57047 .

Figures

Figures reproduced from arXiv: 2510.20535 by Edouard Grave, Hippolyte Pilchen, Patrick P\'erez.

Figure 1
Figure 1. Figure 1: , pooling is performed in the self-attention module. We aver￾age consecutive queries to reach the targeted pooling factor, while keys and values remain unchanged. For a PF of 2 for example (denoting the encoder as ‘ARC2-Encoder’), we group the tokens of the sequence two-by-two. We merge their queries in the last self-attention module, by averaging their continuous hidden states. Then, these pooled queries … view at source ↗
Figure 2
Figure 2. Figure 2: ARC-Encoder pretraining tasks. The encoder, special tokens and the MLP are trained through two alternating tasks: a) Reconstruction: compressed tokens are given to the decoder which is teacher-forced to replicate the full text tokens; b) Continuation: a subpart of the tokens in the sequence are compressed and the decoder is teacher-forced to continue starting from the partially compressed sequence. Illustr… view at source ↗
Figure 3
Figure 3. Figure 3: Extending an ARC￾Encoder to a new decoder [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Compression results with varying MLP dimensional bottlenecks and number of bits per dimension (B/d). 5 CONCLUSION We introduce ARC-Encoder, a novel method to compute compressed text representations that can replace the raw text input in large language models. By reducing the context length, our method 4https://github.com/facebookresearch/faiss/wiki 9 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: demonstrates that ARC-Encoder pretrained at a cer￾tain pooling factor can still be fine-tuned at another, some￾times even improving results on downstream tasks. This transfer works best as we use a smaller pooling factor than the one previously pretrained on. Notably, pretraining at 8× seems to be particularly effective since we can then out￾perform any other pair on pooling factors of 4× and 8×. In contra… view at source ↗
Figure 6
Figure 6. Figure 6: Measured computational costs. (a) Number of TFLOPs and (b) CUDA time in seconds for the continuation of a book from PG19 for various prompt lengths and numbers of tokens to generate on one NVIDIA H100. C TRAINING DETAILS C.1 DEFAULT SETTING Trainings are performed on 8×H100 NVIDIA GPUs using PyTorch’s FSDP framework6 . The ab￾lations follow the parameters and architectural choices from our best ARC8-Encode… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. End-to-End Context Compression at Scale

    cs.CL 2026-06 unverdicted novelty 6.0

    LCLMs are scaled 0.6B-encoder 4B-decoder compressors pre-trained on over 350B tokens that improve the Pareto frontier for general-task performance, compression speed, and peak memory in long-context language model inference.

  2. Efficient Listwise Reranking with Compressed Document Representations

    cs.IR 2026-04 unverdicted novelty 5.0

    RRK compresses documents to multi-token embeddings for efficient listwise reranking, enabling an 8B model to achieve 3x-18x speedups over smaller models with comparable or better effectiveness.

Reference graph

Works this paper leans on

52 extracted references · 1 canonical work pages · cited by 2 Pith papers

  1. [1]

    Beat the ai: Investigating adversarial human annotation for reading comprehension

    Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. Beat the ai: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8: 0 662–678, December 2020. ISSN 2307-387X. doi:10.1162/tacl_a_00338. URL http://dx.doi.org/10.1162/tacl_a_00338

    doi:10.1162/tacl_a_00338 2020

  2. [2]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    Pith/arXiv arXiv 2020

  3. [3]

    Extending context window of large language models via positional interpolation, 2023

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL https://arxiv.org/abs/2306.15595

    Pith/arXiv arXiv 2023

  4. [4]

    D ialog S um: A real-life scenario dialogue summarization dataset

    Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. D ialog S um: A real-life scenario dialogue summarization dataset. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp.\ 5062--5074, Online, August 2021. Association for Computational Linguistics. doi:10.18653/v1/...

    doi:10.18653/v1/2021.findings-acl.449 2021

  5. [5]

    xrag: Extreme context compression for retrieval-augmented generation with one token, 2024

    Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. xrag: Extreme context compression for retrieval-augmented generation with one token, 2024. URL https://arxiv.org/abs/2405.13792

    Pith/arXiv arXiv 2024

  6. [6]

    W iki S um: Coherent summarization dataset for efficient human-evaluation

    Nachshon Cohen, Oren Kalinsky, Yftah Ziser, and Alessandro Moschitti. W iki S um: Coherent summarization dataset for efficient human-evaluation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural La...

    doi:10.18653/v1/2021 2021

  7. [7]

    P ara SCI : A large scientific paraphrase dataset for longer paraphrase generation

    Qingxiu Dong, Xiaojun Wan, and Yue Cao. P ara SCI : A large scientific paraphrase dataset for longer paraphrase generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp.\ 424--434, Online, April 2021. Association for Co...

    work page doi:10.18653/v1/2021.eacl-main.33 2021

  8. [8]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https://arxiv.org/abs/1903.00161

    Pith/arXiv arXiv 2019

  9. [9]

    Cartridges: Lightweight and general-purpose long context representations via self-study, 2025

    Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Will Tennien, Atri Rudra, James Zou, Azalia Mirhoseini, and Christopher Re. Cartridges: Lightweight and general-purpose long context representations via self-study, 2025. URL https://arxiv.org/abs/2506.06266

    Pith/arXiv arXiv 2025

  10. [10]

    In-context autoencoder for context compression in a large language model, 2024

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model, 2024. URL https://arxiv.org/abs/2307.06945

    Pith/arXiv arXiv 2024

  11. [11]

    SAMS um corpus: A human-annotated dialogue dataset for abstractive summarization

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMS um corpus: A human-annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp.\ 70--79, Hong Kong, China, November 2019. Association for Computa...

    doi:10.18653/v1/d19-5409 2019

  12. [12]

    The flores-101 evaluation benchmark for low-resource and multilingual machine translation, 2021

    Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzman, and Angela Fan. The flores-101 evaluation benchmark for low-resource and multilingual machine translation, 2021. URL https://arxiv.org/abs/2106.03193

    Pith/arXiv arXiv 2021

  13. [13]

    The llama 3 herd of models, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava S...

    Pith/arXiv arXiv 2024

  14. [14]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Nai...

    2024

  15. [15]

    Raven: In-context learning with retrieval-augmented encoder-decoder language models, 2024

    Jie Huang, Wei Ping, Peng Xu, Mohammad Shoeybi, Kevin Chen-Chuan Chang, and Bryan Catanzaro. Raven: In-context learning with retrieval-augmented encoder-decoder language models, 2024. URL https://arxiv.org/abs/2308.07922

    Pith/arXiv arXiv 2024

  16. [16]

    Atlas: Few-shot learning with retrieval augmented language models, 2022

    Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models, 2022. URL https://arxiv.org/abs/2208.03299

    Pith/arXiv arXiv 2022

  17. [17]

    Product Quantization for Nearest Neighbor Search

    Herv \'e J \'e gou, Matthijs Douze, and Cordelia Schmid. Product Quantization for Nearest Neighbor Search . IEEE Transactions on Pattern Analysis and Machine Intelligence , 33 0 (1): 0 117--128, January 2011. doi:10.1109/TPAMI.2010.57. URL https://inria.hal.science/inria-00514462

    doi:10.1109/tpami.2010.57 2011

  18. [18]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023 a . URL https://a...

    Pith/arXiv arXiv 2023

  19. [19]

    Llmlingua: Compressing prompts for accelerated inference of large language models, 2023 b

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models, 2023 b . URL https://arxiv.org/abs/2310.05736

    Pith/arXiv arXiv 2023

  20. [20]

    F reebase QA : A new factoid QA data set matching trivia-style question-answer pairs with F reebase

    Kelvin Jiang, Dekun Wu, and Hui Jiang. F reebase QA : A new factoid QA data set matching trivia-style question-answer pairs with F reebase. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo...

    doi:10.18653/v1/n19-1028 2019

  21. [21]

    Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017

    2017

  22. [22]

    T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 1601--1611, Vancouver, Canada, July 2017....

    doi:10.18653/v1/p17-1147 2017

  23. [23]

    Cramming 1568 tokens into a single vector and back again: Exploring the limits of embedding space capacity, 2025

    Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, and Mikhail Burtsev. Cramming 1568 tokens into a single vector and back again: Exploring the limits of embedding space capacity, 2025. URL https://arxiv.org/abs/2502.13063

    Pith/arXiv arXiv 2025

  24. [24]

    Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transac...

    2019

  25. [25]

    Nv-embed: Improved techniques for training llms as generalist embedding models, 2025

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nv-embed: Improved techniques for training llms as generalist embedding models, 2025. URL https://arxiv.org/abs/2405.17428

    Pith/arXiv arXiv 2025

  26. [26]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL https://arxiv.org/abs/2005.11401

    Pith/arXiv arXiv 2021

  27. [27]

    Shifting ai efficiency from model-centric to data-centric compression, 2025

    Xuyang Liu, Zichen Wen, Shaobo Wang, Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, Xu Zheng, Honggang Chen, Weijia Li, Xuming Hu, Conghui He, and Linfeng Zhang. Shifting ai efficiency from model-centric to data-centric compression, 2025. URL https://arxiv.org/abs/2505.19147

    arXiv 2025

  28. [28]

    Decoupled weight decay regularization, 2019

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxiv.org/abs/1711.05101

    Pith/arXiv arXiv 2019

  29. [29]

    Pisco: Pretty simple compression for retrieval-augmented generation, 2025 a

    Maxime Louis, Hervé Déjean, and Stéphane Clinchant. Pisco: Pretty simple compression for retrieval-augmented generation, 2025 a . URL https://arxiv.org/abs/2501.16075

    Pith/arXiv arXiv 2025

  30. [30]

    Oscar: Online soft compression and reranking, 2025 b

    Maxime Louis, Thibault Formal, Hervé Dejean, and Stéphane Clinchant. Oscar: Online soft compression and reranking, 2025 b . URL https://arxiv.org/abs/2504.07109

    arXiv 2025

  31. [31]

    Learning to compress prompts with gist tokens, 2024

    Jesse Mu, Xiang Lisa Li, and Noah Goodman. Learning to compress prompts with gist tokens, 2024. URL https://arxiv.org/abs/2304.08467

    Pith/arXiv arXiv 2024

  32. [32]

    MS MARCO: A human generated machine reading comprehension dataset

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268, 2016. URL http://arxiv.org/abs/1611.09268

    Pith/arXiv arXiv 2016

  33. [33]

    Vicky Zhao, Lili Qiu, and Dongmei Zhang

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, H. Vicky Zhao, Lili Qiu, and Dongmei Zhang. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression, 2024. URL https://arxiv.org/abs/2403.12968

    Pith/arXiv arXiv 2024

  34. [34]

    Kilt: a benchmark for knowledge intensive language tasks, 2021

    Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. Kilt: a benchmark for knowledge intensive language tasks, 2021. URL https://arxiv.org/abs/2009.02252

    Pith/arXiv arXiv 2021

  35. [35]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint, 2019. URL https://arxiv.org/abs/1911.05507

    Pith/arXiv arXiv 2019

  36. [36]

    Squad: 100,000+ questions for machine comprehension of text, 2016

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250

    Pith/arXiv arXiv 2016

  37. [37]

    Zeroscrolls: A zero-shot benchmark for long text understanding, 2023

    Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding, 2023. URL https://arxiv.org/abs/2305.14196

    Pith/arXiv arXiv 2023

  38. [38]

    Asqa: Factoid questions meet long-form answers, 2023

    Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. Asqa: Factoid questions meet long-form answers, 2023. URL https://arxiv.org/abs/2204.06092

    Pith/arXiv arXiv 2023

  39. [39]

    Adapting decoder-based language models for diverse encoder downstream tasks, 2025

    Paul Suganthan, Fedor Moiseev, Le Yan, Junru Wu, Jianmo Ni, Jay Han, Imed Zitouni, Enrique Alfonseca, Xuanhui Wang, and Zhe Dong. Adapting decoder-based language models for diverse encoder downstream tasks, 2025. URL https://arxiv.org/abs/2503.02656

    Pith/arXiv arXiv 2025

  40. [40]

    Gmsa: Enhancing context compression via group merging and layer semantic alignment, 2025

    Jiwei Tang, Zhicheng Zhang, Shunlong Wu, Jingheng Ye, Lichen Bai, Zitai Wang, Tingwei Lu, Jiaqi Chen, Lin Hai, Hai-Tao Zheng, and Hong-Gee Kim. Gmsa: Enhancing context compression via group merging and layer semantic alignment, 2025. URL https://arxiv.org/abs/2505.12215

    arXiv 2025

  41. [41]

    Efficient transformers: A survey, 2022

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022. URL https://arxiv.org/abs/2009.06732

    Pith/arXiv arXiv 2022

  42. [42]

    Gemma 3 technical report, 2025

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, and Gaël Liu et al. Gemma 3 technical report, 2025. UR...

    Pith/arXiv arXiv 2025

  43. [43]

    Llama‑2‑7B‑32K‑Instruct — and fine‑tuning for Llama‑2 models with Together API

    Together AI . Llama‑2‑7B‑32K‑Instruct — and fine‑tuning for Llama‑2 models with Together API . https://www.together.ai/blog/llama-2-7b-32k-instruct, August 2023. Accessed: 2025-07-17

    2023

  44. [44]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    Pith/arXiv arXiv 2023

  45. [45]

    Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models. NeurIPS Datasets and ...

    2024

  46. [46]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600

    Pith/arXiv arXiv 2018

  47. [47]

    Long-context language modeling with parallel context encoding, 2024

    Howard Yen, Tianyu Gao, and Danqi Chen. Long-context language modeling with parallel context encoding, 2024. URL https://arxiv.org/abs/2402.16617

    Pith/arXiv arXiv 2024

  48. [48]

    Encoder-decoder gemma: Improving the quality-efficiency trade-off via adaptation, 2025

    Biao Zhang, Fedor Moiseev, Joshua Ainslie, Paul Suganthan, Min Ma, Surya Bhupatiraju, Fede Lebron, Orhan Firat, Armand Joulin, and Zhe Dong. Encoder-decoder gemma: Improving the quality-efficiency trade-off via adaptation, 2025. URL https://arxiv.org/abs/2504.06225

    Pith/arXiv arXiv 2025

  49. [49]

    Hashimoto

    Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. Benchmarking large language models for news summarization, 2023. URL https://arxiv.org/abs/2301.13848

    Pith/arXiv arXiv 2023

  50. [50]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  51. [51]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  52. [52]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...