RECORD
Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression
Computation and Language · Submitted 25 Feb 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Large language models (LLMs) require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. In this paper, we propose the Gist COnditioned deCOding (Gist-COCO) model, introducing a novel method for compressing prompts which also can assist the prompt interpretation and engineering. Gist-COCO employs an encoder-decoder based language model and then incorporates an additional encoder as a plugin module to compress prompts with inputs using gist tokens. It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla language model. By verbalizing the representations of gist tokens into gist prompts, the compression ability of Gist-COCO can be generalized to different LLMs with high compression rates. Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks. Further analysis on gist verbalization results suggests that our gist prompts serve different functions in aiding language models. They may directly provide potential answers, generate the chain-of-thought, or simply repeat the inputs. All data and codes are available at https://github.com/OpenMatch/Gist-COCO .
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Large language models (LLMs) require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. | background 1.00 | 0.08 | 0.02 | asserted 1.00 |
| s1 | In this paper, we propose the Gist COnditioned deCOding (Gist-COCO) model, introducing a novel method for compressing prompts which also can assist the prompt interpretation and engineering. | contribution 1.00 | 0.86 | 0.12 | asserted 0.95 |
| s2 | Gist-COCO employs an encoder-decoder based language model and then incorporates an additional encoder as a plugin module to compress prompts with inputs using gist tokens. | method 1.00 | 0.82 | 0.03 | asserted 1.00 |
| s3 | It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla language model. | method 1.00 | 0.87 | 0.04 | asserted 1.00 |
| s4 | By verbalizing the representations of gist tokens into gist prompts, the compression ability of Gist-COCO can be generalized to different LLMs with high compression rates. | method 0.71 | 0.70 | 0.12 | asserted 0.95 |
| s5 | Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks. | result 1.00 | 0.96 | 0.60 | asserted 1.00 |
| s6 | Further analysis on gist verbalization results suggests that our gist prompts serve different functions in aiding language models. | result 0.48 | 0.92 | 0.10 | hedged possibility 0.66 |
| s7 | They may directly provide potential answers, generate the chain-of-thought, or simply repeat the inputs. | interpretation 0.93 | 0.77 | 0.07 | hedged possibility 1.00 |
| s8 | All data and codes are available at https://github.com/OpenMatch/Gist-COCO . | contribution 0.51 | 0.31 | 0.01 | asserted 0.98 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Large language models (LLMs) require lengthy prompts as the input context to produce output aligned with user intentions, a process that incurs extra costs during inference. In this paper, we propose the Gist COnditioned deCOding (Gist-COCO) model, introducing a novel method for compressing prompts which also can assist the prompt interpretation and engineering. Gist-COCO employs an encoder-decoder based language model and then incorporates an additional encoder as a plugin module to compress prompts with inputs using gist tokens. It finetunes the compression plugin module and uses the representations of gist tokens to emulate the raw prompts in the vanilla language model. By verbalizing the representations of gist tokens into gist prompts, the compression ability of Gist-COCO can be generalized to different LLMs with high compression rates. Our experiments demonstrate that Gist-COCO outperforms previous prompt compression models in both passage and instruction compression tasks. Further analysis on gist verbalization results suggests that our gist prompts serve different functions in aiding language models. They may directly provide potential answers, generate the chain-of-thought, or simply repeat the inputs. All data and codes are available at https://github.com/OpenMatch/Gist-COCO .
Human review
How the literature uses it
6 arXiv papers cite this in 7 sentences
Papers citing it · 3 sentences without a confident label
Compares with it 1 paper
-
Unlike them and other architecturally divergent approaches like T5-based encoder-decoders [ref] and LoCoCo's convolution-based memory [ref] , we focus on compressing the KV-cache within the standard decoder-only transformer architecture, avoiding these complexities.
Long Context In-Context Compression by Getting to the Gist of Gisting · Related Work · 0.90
Cites it as background 3 papers
-
Recently, with the development of Large language models, more and more researchers have paid attention to improving the RAG performance based on large language models by considering the retrieval frequency [ref] , designing delicate CoTs [ref] , training the retrieval models and language models together [ref] , comprising the retrieved references to fit the input length limit [ref] .
DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generation · Retrieval-augmented Generation Models · 0.89
-
Recent researchers also explored fixing LLMs and optimizing retrievers as plug-in modules [ref] or introducing post-retrieval components, , compressors and rankers [ref] , to reduce the training cost.
RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation · Retrieval-augmented Generation · 0.93
-
Gist tokens have been used in various use-cases, including for better selection of in-context examples [ref] , conditioned decoding [ref] , hierarchical compression for longer contexts (by gisting the gist tokens) [ref] , and in task-specific gisting [ref] .
Compositional Steering of Large Language Models with Steering Tokens · Related Work · 0.99
What it cites · 22 sentences without a confident label
Uses the method or tool of 2 papers
-
Following previous work [ref] , we used the ROUGE-L metric to evaluate the performance of different models on instruction compression tasks.
Learning to Compress Prompts with Gist Tokens · Experimental Methodology · 1.00
-
We initialize Gist-COCO model with FlanT5-base and FlanT5-large checkpoints from Hugginface Transformers [ref] .
HuggingFace's Transformers: State-of-the-art Natural Language Processing · Experimental Methodology · 0.98
Compares with 1 paper
-
Two embedding based compression models are compared in our experiments, including AutoCompressor [ref] and Gist [ref] .
Learning to Compress Prompts with Gist Tokens · Experimental Methodology · 0.99
Cites as background 11 papers
-
Given an input $x$, existing work usually uses lengthy task instructions [ref] or retrieved passages [ref] as prompts, denoted as $c$, to aid LLMs for the generation.
Scaling Instruction-Finetuned Language Models · Preliminary of Prompt Compression · 0.84
-
Given an input $x$, existing work usually uses lengthy task instructions [ref] or retrieved passages [ref] as prompts, denoted as $c$, to aid LLMs for the generation.
REPLUG: Retrieval-Augmented Black-Box Language Models · Preliminary of Prompt Compression · 0.84
-
[ref] propose the Black-box Prompt Optimization (BPO) method, which optimizes the prompts to bridge the gap between humans and LLMs.
Black-Box Prompt Optimization: Aligning Large Language Models without Model Training · Related Work · 0.83
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.