Pith. sign in

REVIEW 3 major objections 6 minor 77 references

Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hybrid context compression matches full-context QA while cutting tokens by 88.8%.

desk verdict A plausible hybrid compression architecture that undercuts its own 'matches uncompressed' claim; near-parity with RAG at ~11% tokens is the real result. read the letter →

arxiv 2505.15774 v1 pith:4DYBCLVO submitted 2025-05-21 cs.CL cs.LG

Huanxuan Liao , Wen Hu , Yao Xu , Shizhu He , Jun Zhao , Kang Liu This is my paper
classification cs.CLcs.LG
keywords contextcompressionhybridretrieval-augmentedgenerationlong-contextreasoningtokenimportanceselectionlatentrepresentationinstructiontuningmixtureofexperts
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that hard and soft context compression are not competing alternatives but two halves of a single solution: a compressor can keep the exact tokens that carry local details while separately encoding the whole context into instruction-aware latent tokens, and the two views together preserve more of what a question-answering system needs than either view alone. Its HyCo2 system adds a token-retention classifier and a hybrid adapter to a frozen LLM, reporting that this dual path beats both hard-only and soft-only compressors on seven knowledge-intensive QA benchmarks across three LLM families. The headline numbers are an average 13.1% improvement over the no-retrieval baseline and parity with uncompressed retrieval-augmented generation at an 88.8% reduction in context tokens, with only 168M trainable parameters. If the claim holds, it means the standard trade-off between detail preservation and semantic abstraction in prompt compression is not a law but a design choice.

What carries the argument

The central mechanism is the dual-path compressor with a gated hybrid adapter and a token-retention classifier. The hard path is a classification layer $p = \sigma(WV+b)$ over the encoder's token features that assigns a retention probability to each token and keeps the top $k\%$ (default 10%). The soft path is a hybrid adapter whose noisy mixture-of-experts router $G(V)$ blends an MLP branch performing grouped local attention with a QFormer branch performing global attention over learnable query tokens; both branches use cross-attention with the instruction embedding $C$. The machinery is held together by the alternating training strategy (Stage 1 paraphrase, Stage 2 completion, Stage 3 instruction tuning with $L_{kl}$ against a RAG teacher), which the paper argues prevents the model from collapsing onto the easier global path. The router and the concatenation of local and global outputs are what let the frozen LLM read both verbatim details and abstracted semantics in a single prompt.

What would settle it

Run HyCo2's compression on a QA benchmark that has answer-span or supporting-fact annotations, then compute recall of the answer-bearing tokens among the top-10% retained tokens. If the recall is at or below chance while exact-match scores stay high, the hard-selection path is not doing the evidence-retention work the paper attributes to it, and the performance would have to be explained by the soft path and the router instead.

Watch

Extended reading notes

Core claim

The central claim, stated in the paper's own frame, is that local-detail loss and global-semantic loss are complementary failure modes, and a dual-level compression frame can retain both. HyCo2 routes the context through two parallel paths: a classification layer that scores every token and hard-selects the top 10% retained tokens, and a hybrid adapter that soft-encodes the global semantics, with an MLP branch doing grouped local attention and a QFormer branch using learnable query tokens, blended by a noisy mixture-of-experts router that is conditioned on the instruction via cross-attention. The local and global outputs are concatenated and fed to a frozen LLM. The paper reports that on seven QA benchmarks across Mistral-7B, LLaMA-3.1-8B, and Qwen-2.5-7B, HyCo2 achieves the best average exact-match score among compression methods, matches or exceeds uncompressed RAG on several datasets, and degrades only 1.2 EM points when the number of retrieved documents grows from 1 to 10. The paper further claims the alternating pretraining schedule—paraphrase, then completion, then instruction tuning with KL self-distillation against RAG—is necessary, since end-to-end training or omitting either pretraining stage drops average performance by several points.

Load-bearing premise

The load-bearing premise is that Stage 2 completion pretraining on randomly split RedPajama text segments teaches the classification layer a general sense of which tokens matter, and that this sense transfers to instruction-conditioned question answering, even though the paper does not show the retained tokens overlap with actual answer evidence.

Editorial extensions

If this is right

  • If the central claim holds, retrieval-augmented QA can run at roughly one ninth of the context tokens with accuracy close to the full-context RAG oracle, directly reducing attention cost and memory.
  • The result implies that a frozen instruction-tuned LLM can consume a mixture of verbatim tokens and learned latent tokens in the same prompt without any LLM fine-tuning, since only the 168M-parameter compressor is trained.
  • The consistency across Mistral, LLaMA-3.1, and Qwen backbones suggests the hybrid design transfers to different LLM families, not just to the model it was tuned on.
  • The robustness result—only a 1.2-point EM drop from K=1 to K=10 retrieved documents—implies hybrid compression is the most graceful of the tested methods as context length grows past 1k tokens.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test the paper does not run: measure whether the retained top-10% tokens actually overlap with answer-bearing spans on the evaluation sets; if recall is near chance, the local path's contribution to QA accuracy is not through evidence retention, and the reported gains would be carried by the soft path and router.
  • The KL self-distillation against a RAG teacher suggests the same compression head could be distilled against a stronger reader (a larger LLM or an oracle with answer spans), which would likely raise the ceiling of the reported numbers.
  • The architecture is not specific to text: grouping-based local attention plus learnable-query global attention with noisy gating is a natural fit for video or code compression, where local syntax and global structure both matter; nothing in the method constrains it to QA.
  • Since the paper's own limitations note that the static 10% keep-ratio can discard critical details, an obvious extension is a budget-aware classifier that lets the router or the instruction decide how many tokens to keep per document, rather than a fixed global ratio.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes HyCo2, a hybrid context compression method that combines soft global compression (a mixture of MLP and QFormer adapters with instruction cross-attention and a learned router) with hard local compression (a classification layer that scores tokens and retains a top-k% subset). The authors introduce a three-stage training recipe: paraphrase pretraining for the adapter, completion pretraining for the classifier, and instruction tuning with KL distillation against a RAG teacher. Experiments on seven QA benchmarks across Mistral-7B, LLaMA-3.1-8B, and Qwen-2.5-7B report average EM improvements over the non-retrieval baseline, strong token reductions, ablation results supporting each component, and efficiency measurements. The code is promised but not yet released.

Significance. If the reported results hold, HyCo2 is a meaningful contribution to prompt/context compression: it demonstrates that a lightweight 168M-parameter trainable compressor can outperform established hard and soft baselines (EXIT, LLMLingua2, LongLLMLingua, xRAG) on knowledge-intensive QA while cutting tokens by roughly 89%. The paper contains substantive engineering: the hybrid adapter design, the alternating pretraining strategy, and the ablations in Table 3 are informative and internally consistent. However, the central claim that HyCo2 'matches' uncompressed RAG is not supported by Table 1, where HyCo2 is below RAG by 5.5, 2.6, and 2.3 average EM points on the three backbones. The main load-bearing mechanism, the training signal for the discrete top-k token selection, is not fully specified, and the transfer of the completion-pretraining objective to question-relevant retention is not verified. These issues are addressable but require additional evidence and a recalibration of the claims.

major comments (3)
  1. [Section 3.3 and Appendix B.2, Eq. (6)] The training of the classification layer through the discrete top-k selection is not specified. Equation (6) writes p_phi(b_i | H(F_phi(a)), ...) as if H is differentiable in the scoring parameters, but retaining the top-10% tokens is a non-differentiable operation. The paper never states whether a straight-through estimator, Gumbel-top-k relaxation, REINFORCE-style surrogate, or some other gradient approximation is used. Because the entire local-compression contribution depends on this gradient path, the authors must specify the estimator and, ideally, show that gradients actually flow into W and b of the classification layer.
  2. [Section 3.3, Stage 2, and Appendix D] The transfer assumption behind completion pretraining is unverified. Stage 2 trains the classifier to predict a random continuation segment b from a compressed prefix a on randomly split RedPajama documents. Success at this task requires retaining local coherence cues, not necessarily the entity names, dates, or roles that a later question will ask about. The paper provides no analysis showing that the retained top-10% tokens overlap answer-bearing spans in the QA evaluation sets. Appendix D itself admits that the static 10% ratio 'might discard critical details.' Without overlap evidence, the claimed local-global balance could reduce to a soft compressor plus a blind filter, and the ablation 'w/o Local' does not isolate this mechanism because it removes the entire classification pathway. Please add a token-overlap or answer-span-recall analysis on a sample of the evaluation data.
  3. [Abstract and Section 4.2, Table 1] The abstract's claim that 'HyCo2 matches the performance of uncompressed methods' is contradicted by Table 1. Averaged over the seven datasets, HyCo2 scores 44.96 versus RAG's 50.51 for Mistral, 43.96 versus 46.51 for LLaMA-3.1, and 42.11 versus 44.41 for Qwen-2.5. The paper should either soften the abstract to say that HyCo2 approaches, and in a few datasets (e.g., WQ and CWQ) exceeds, full RAG, or provide a statistical basis for 'matches.' The current wording overstates the empirical result.
minor comments (6)
  1. [Abstract and Section 4.2] The 'average of 13.1%' improvement should be reported with the reference point made explicit (relative EM improvement over the non-retrieval Vanilla baseline, averaged across backbones), because the same number is easy to misread as an improvement over RAG.
  2. [Section 4.4 and Checklist item 7] The paper states that hyperparameter effects are examined, but no error bars, confidence intervals, or significance tests are reported anywhere. Even a small number of repeated seeds would strengthen the claim that the average gaps in Table 1 are not within run-to-run noise.
  3. [Appendix B.2 and C.2] There is a typo: 'casual attention mask' should be 'causal attention mask.' Also, in Table 6 the hyperparameter table is labelled 'Assignment' rather than 'Value.' These are minor but should be fixed.
  4. [Figure 1] The label 'Classfic Layer' should be 'Classification Layer.' The figure is otherwise informative, but this typo is distracting.
  5. [Section 4.3, Efficiency and Memory] The sentence 'EXIT incurres the highest computational and time costs' contains a typo ('incurres' should be 'incurs'). Also, Table 2 reports single-run measurements without specifying variance; a brief note on measurement methodology would help.
  6. [Appendix C.3] The Information Loss metric is defined as H_x - H_hat{x}, but it is not stated how entropy is estimated for token sequences. Please clarify the estimator (e.g., empirical token-level entropy, model-based surprisal) so the metric is reproducible.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: HyCo2's central QA results are produced by held-out benchmark evaluation against external baselines, and the self-citations and hand-set hyperparameters are not load-bearing.

full rationale

HyCo2's central claim is an empirical result rather than a derivation: the compressor is trained with a paraphrase objective (Eq. 5), a random-continuation completion objective (Eq. 6), and an instruction-tuning loss with a KL term against a RAG teacher (Eqs. 7-8), then evaluated with Exact Match on seven knowledge-intensive QA benchmarks against external baselines. No equation defines the reported EM gains in terms of the training losses; the completion-pretraining objective is trained on random RedPajama segments rather than on QA labels, so transfer to the evaluation benchmarks is an empirical hypothesis rather than a construction. The RAG-teacher KL term is a training signal, not a fitted test-set quantity. Self-citations [38, 39, 40, 43] support peripheral claims such as the existence of complex reasoning frameworks or generation cost and do not carry the method; hand-set hyperparameters (k=10%, N_L=16, alpha=2.0) are choices rather than fitted predictions. Two evaluation caveats are worth noting but are not circularity: NQ and TQA appear in the instruction-tuning corpus and are marked IID in Table 1, and the Section 4.3 reconstruction-based information-preservation metric resembles the Stage 1 paraphrase objective. These affect interpretation of specific numbers but do not make the central held-out comparison equivalent to its inputs. Appendix D explicitly discloses that the static 10% retention may discard critical details, and the paper does not invoke any uniqueness theorem or author-imported constraint to force its design choices.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on empirical training dynamics, not a derivation. The main hand-set quantities are k=10%, N_L=16, and alpha=2.0; these affect the compression rate and loss balance and are not swept in the paper. The framework assumes the frozen LLM's hidden states encode both global semantics and local token importance, and that the synthetic completion pretraining induces a transferable notion of relevance. No new physical or mathematical entities are introduced; the hybrid adapter and classification layer are learned modules with no falsifiable handle outside the reported benchmarks.

free parameters (3)
  • Token retention ratio k% = 10%
    Default hard-compression ratio (Section 4.1 and Tables 6-7). It directly sets how many local tokens survive and is not swept per dataset, even though the paper itself lists static granularity as a limitation in Appendix D.
  • Number of learnable query tokens N_L = 16
    Default number of global tokens in the QFormer branch (Section 4.1). It fixes the size of the soft global stream and is not varied in the main experiments.
  • KL distillation weight alpha = 2.0
    Weight of the self-distillation term in instruction tuning (Appendix C.2, Table 7). Ablations show removing L_kl hurts, but no sweep over alpha is reported.
assumptions (3)
  • domain assumption The frozen LLM's final hidden states contain sufficient signal for both global semantic compression and token-level importance scoring.
    Section 3.2 builds both the hybrid adapter and the classification layer on the encoder features V from the frozen LLM, without any task-specific supervision for this premise.
  • domain assumption Completion pretraining on randomly split RedPajama segments transfers to instruction-relevant token selection.
    Stage 2 (Section 3.3, Eq. 6) trains the selector to keep tokens that predict the next text segment b. The paper assumes this synthetic objective produces retention probabilities that are useful for downstream QA, but does not measure token overlap with answer evidence.
  • ad hoc to paper The top-k selection is trainable via backpropagation.
    The paper never specifies a straight-through estimator, Gumbel relaxation, or REINFORCE for the discrete mask, yet Eq. 6 and Eq. 7 optimize L_nll through H. The implied trainability assumption is ad hoc and unstated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention." pith.science (2026). https://pith.science/paper/4DYBCLVO

@misc{pith2026250515774,
  author       = {Pith},
  title        = {Pith review of: Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4DYBCLVO}},
  note         = {Machine review of arXiv:2505.15774}
}
abstract

Large Language Models (LLMs) encounter significant challenges in long-sequence inference due to computational inefficiency and redundant processing, driving interest in context compression techniques. Existing methods often rely on token importance to perform hard local compression or encode context into latent representations for soft global compression. However, the uneven distribution of textual content relevance and the diversity of demands for user instructions mean these approaches frequently lead to the loss of potentially valuable information. To address this, we propose $\textbf{Hy}$brid $\textbf{Co}$ntext $\textbf{Co}$mpression (HyCo$_2$) for LLMs, which integrates both global and local perspectives to guide context compression while retaining both the essential semantics and critical details for task completion. Specifically, we employ a hybrid adapter to refine global semantics with the global view, based on the observation that different adapters excel at different tasks. Then we incorporate a classification layer that assigns a retention probability to each context token based on the local view, determining whether it should be retained or discarded. To foster a balanced integration of global and local compression, we introduce auxiliary paraphrasing and completion pretraining before instruction tuning. This promotes a synergistic integration that emphasizes instruction-relevant information while preserving essential local details, ultimately balancing local and global information retention in context compression. Experiments show that our HyCo$_2$ method significantly enhances long-text reasoning while reducing token usage. It improves the performance of various LLM series by an average of 13.1\% across seven knowledge-intensive QA benchmarks. Moreover, HyCo$_2$ matches the performance of uncompressed methods while reducing token consumption by 88.8\%.

Figures

Figures reproduced from arXiv: 2505.15774 by the authors.

Figure 1
Figure 1. Different paradigms for processing long-text inputs: (a) original input, (b) hard compression, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. (a) Hybrid Context Compression Framework. We employ a classification layer for local tokens selection and use a hybrid adapter to extract instruction-relevant representation. Additionally, a router optimizes the global context through soft integration, thereby optimizing overall context representation. (b) Alternating Training Method. (1) Refining the hybrid adapter with paraphrase pretraining, (2) optimizing the cl… view at source ↗
Figure 3
Figure 3. Significance of Soft MoE. The reported values represent the performance ratio of baselines to the best one: Gate. This suggests that a simpler structure may facilitate more effec￾tive assimilation of compressed context by LLMs. However, in specific tasks, such as multi-document reasoning on 2WIKI, the QFormer demonstrates an advantage. Through learnable query tokens and attention mechanisms, it can dynamically prior… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: We employ Mistral-7B to investigate two aspects: [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 24 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  3. [3]

    Ms marco: A human generated machine reading comprehension dataset, 2018

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: A human generated machine reading comprehension dataset, 2018

    2018

  4. [4]

    Semantic parsing on freebase from question-answer pairs

    Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. InProceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013

    2013

  5. [5]

    Efficient prompting methods for large language models: A survey.arXiv preprint arXiv:2404.01077, 2024

    Kaiyan Chang, Songcheng Xu, Chenglong Wang, Yingfeng Luo, Xiaoqian Liu, Tong Xiao, and Jingbo Zhu. Efficient prompting methods for large language models: A survey.arXiv preprint arXiv:2404.01077, 2024

    arXiv 2024

  6. [6]

    Dialogsum: A real-life scenario dialogue summarization dataset, 2021

    Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset, 2021

    work page 2021

  7. [7]

    xrag: Extreme context compression for retrieval-augmented generation with one token.arXiv preprint arXiv:2405.13792, 2024

    Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, and Dongyan Zhao. xrag: Extreme context compression for retrieval-augmented generation with one token.arXiv preprint arXiv:2405.13792, 2024

    arXiv 2024

  8. [8]

    Adapting language models to compress contexts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 3829–3846, 2023

    2023

Show all 77 references
  1. [9]

    Learning to compress prompt in natural language formats

    Yu-Neng Chuang, Tianwei Xing, Chia-Yuan Chang, Zirui Liu, Xun Chen, and Xia Hu. Learning to compress prompt in natural language formats. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo...

    2024

  2. [10]

    Selection-p: Self-supervised task-agnostic prompt compression for faithfulness and transfer- ability

    Tsz Ting Chung, Leyang Cui, Lemao Liu, Xinting Huang, Shuming Shi, and Dit-Yan Yeung. Selection-p: Self-supervised task-agnostic prompt compression for faithfulness and transfer- ability. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 11057–11070, 2024

    2024

  3. [11]

    Improve student’s reasoning generalizability through cascading decomposed cots distillation.arXiv preprint arXiv:2405.19842, 2024

    Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. Improve student’s reasoning generalizability through cascading decomposed cots distillation.arXiv preprint arXiv:2405.19842, 2024

    2024 arXiv

  4. [12]

    A silver bullet or a compromise for full attention? a comprehensive study of gist token-based context compression.arXiv preprint arXiv:2412.17483, 2024

    Chenlong Deng, Zhisong Zhang, Kelong Mao, Shuaiyi Li, Xinting Huang, Dong Yu, and Zhicheng Dou. A silver bullet or a compromise for full attention? a comprehensive study of gist token-based context compression.arXiv preprint arXiv:2412.17483, 2024

    2024 arXiv

  5. [13]

    Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019

    Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019

    2019

  6. [14]

    Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and et al

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, An- thony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Asto...

    2024 arXiv

  7. [15]

    Unifying demonstration selection and compression for in-context learning.arXiv preprint arXiv:2405.17062, 2024

    Jun Gao, Ziqiang Cao, and Wenjie Li. Unifying demonstration selection and compression for in-context learning.arXiv preprint arXiv:2405.17062, 2024. 10

    2024 arXiv

  8. [16]

    In-context autoencoder for context compression in a large language model.arXiv preprint arXiv:2307.06945, 2023

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model.arXiv preprint arXiv:2307.06945, 2023

    2023 arXiv

  9. [17]

    In-context autoencoder for context compression in a large language model, 2023

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model, 2023

    2023

  10. [18]

    Samsum corpus: A human-annotated dialogue dataset for abstractive summarization

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. InProceedings of the 2nd Workshop on New Frontiers in Summarization. Association for Computational Linguistics, 2019

    2019

  11. [19]

    Hipporag: Neurobiologically inspired long-term memory for large language models

    Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  12. [20]

    Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps

    Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. InProceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, 2020

    2020

  13. [21]

    Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model.arXiv preprint arXiv:2408.09559, 2024

    Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model.arXiv preprint arXiv:2408.09559, 2024

    2024 arXiv

  14. [22]

    Exit: Context-aware extractive compression for enhancing retrieval-augmented generation

    Taeho Hwang, Sukmin Cho, Soyeong Jeong, Hoyun Song, SeungYoon Han, and Jong C Park. Exit: Context-aware extractive compression for enhancing retrieval-augmented generation. arXiv preprint arXiv:2412.12559, 2024

    2024 arXiv

  15. [23]

    Unsupervised dense information retrieval with contrastive learning, 2021

    Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning, 2021

    2021

  16. [24]

    Characteriz- ing prompt compression methods for long context inference.arXiv preprint arXiv:2407.08892, 2024

    Siddharth Jha, Lutfi Eren Erdogan, Sehoon Kim, Kurt Keutzer, and Amir Gholami. Characteriz- ing prompt compression methods for long context inference.arXiv preprint arXiv:2407.08892, 2024

    2024 arXiv

  17. [25]

    Mistral 7b.ArXiv, abs/2310.06825, 2023

    Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril,...

    arXiv

  18. [26]

    Llmlingua: Compress- ing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compress- ing prompts for accelerated inference of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13358–13376, 2023

    2023

  19. [27]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguis...

    2024

  20. [28]

    FreebaseQA: A new factoid QA data set matching trivia- style question-answer pairs with Freebase

    Kelvin Jiang, Dekun Wu, and Hui Jiang. FreebaseQA: A new factoid QA data set matching trivia- style question-answer pairs with Freebase. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Associa...

    2019

  21. [29]

    PubMedQA: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors,Proceedings of the 2019 Conference on Empirical Methods in Natural Language...

    2019

  22. [30]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16...

    2017

  23. [31]

    The narrativeqa reading comprehension challenge, 2017

    Tomáš Koˇciský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge, 2017

    2017

  24. [32]

    Natural questions: A benchmark for question answering research.Transactions of the Association for Computational Linguistics, 7:452–466, 2019

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: A benchmark for question answering research.Transactions of the Association for Computat...

    2019

  25. [33]

    Understanding and improving information preservation in prompt compression for llms.arXiv preprint arXiv:2503.19114, 2025

    Weronika Łajewska, Momchil Hardalov, Laura Aina, Neha Anna John, Hang Su, and Lluís Màrquez. Understanding and improving information preservation in prompt compression for llms.arXiv preprint arXiv:2503.19114, 2025

    2025

  26. [34]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  27. [35]

    Compressing context to enhance inference efficiency of large language models

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. Compressing context to enhance inference efficiency of large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6342–6353, 2023

    2023

  28. [36]

    Prompt compression for large language models: A survey.arXiv preprint arXiv:2410.12388, 2024

    Zongqian Li, Yinhong Liu, Yixuan Su, and Nigel Collier. Prompt compression for large language models: A survey.arXiv preprint arXiv:2410.12388, 2024

    2024 arXiv

  29. [37]

    500xcompressor: Generalized prompt compression for large language models.arXiv preprint arXiv:2408.03094, 2024

    Zongqian Li, Yixuan Su, and Nigel Collier. 500xcompressor: Generalized prompt compression for large language models.arXiv preprint arXiv:2408.03094, 2024

    2024 arXiv

  30. [38]

    Skintern: Internalizing symbolic knowledge for distilling better cot capabilities into small language models

    Huanxuan Liao, Shizhu He, Yupu Hao, Xiang Li, Yuanzhe Zhang, Jun Zhao, and Kang Liu. Skintern: Internalizing symbolic knowledge for distilling better cot capabilities into small language models. InProceedings of the 31st International Conference on Computational Linguistics, p...

    2025

  31. [39]

    Neural-symbolic collaborative distillation: Advancing small language models for complex reasoning tasks

    Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Kang Liu, and Jun Zhao. Neural-symbolic collaborative distillation: Advancing small language models for complex reasoning tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 24567–24575, 2025

    2025

  32. [40]

    Awakening augmented generation: Learning to awaken internal knowledge of large language models for question answering

    Huanxuan Liao, Shizhu He, Yao Xu, Yuanzhe Zhang, Shengping Liu, Kang Liu, and Jun Zhao. Awakening augmented generation: Learning to awaken internal knowledge of large language models for question answering. InProceedings of the 31st International Conference on Computational Li...

    2025

  33. [41]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  34. [42]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  35. [43]

    A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407, 2025

    Jiaheng Liu, Dawei Zhu, Zhiqi Bai, Yancheng He, Huanxuan Liao, Haoran Que, Zekun Wang, Chenchen Zhang, Ge Zhang, Jiebin Zhang, et al. A comprehensive survey on long context language modeling.arXiv preprint arXiv:2503.17407, 2025

    2025

  36. [44]

    Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024

    2024

  37. [45]

    Hybrid-level instruction injection for video token compression in multi-modal large language models.arXiv preprint arXiv:2503.16036, 2025

    Zhihang Liu, Chen-Wei Xie, Pandeng Li, Liming Zhao, Longxiang Tang, Yun Zheng, Chuanbin Liu, and Hongtao Xie. Hybrid-level instruction injection for video token compression in multi-modal large language models.arXiv preprint arXiv:2503.16036, 2025. 12

    2025 arXiv

  38. [46]

    When not to trust language models: Investigating effectiveness of parametric and non-parametric memories

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Ha- jishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. InProceedings of the 61st Annual Meeting of the Association for Computatio...

    2023

  39. [47]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36:19327–19352, 2023

    2023

  40. [48]

    Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression.arXiv preprint arXiv:2403.12968, 2024

    Zhuoshi Pan, Qianhui Wu, Huiqiang Jiang, Menglin Xia, Xufang Luo, Jue Zhang, Qingwei Lin, Victor Rühle, Yuqing Yang, Chin-Yew Lin, et al. Llmlingua-2: Data distillation for efficient and faithful task-agnostic prompt compression.arXiv preprint arXiv:2403.12968, 2024

    2024 arXiv

  41. [49]

    Pytorch: An imperative style, high-performance deep learning library.arXiv preprint arXiv:1912.01703, 2019

    A Paszke. Pytorch: An imperative style, high-performance deep learning library.arXiv preprint arXiv:1912.01703, 2019

    1912 arXiv

  42. [50]

    Know what you don’t know: Unanswerable questions for squad, 2018

    Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad, 2018

    2018

  43. [51]

    Siva Reddy, Danqi Chen, and Christopher D. Manning. Coqa: A conversational question answering challenge, 2019

    2019

  44. [52]

    Prompt programming for large language models: Beyond the few-shot paradigm

    Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. InExtended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1–7, 2021

    2021

  45. [53]

    Getting closer to ai complete question answering: A set of prerequisite real tasks

    Anna Rogers, Olga Kovaleva, Matthew Downey, and Anna Rumshisky. Getting closer to ai complete question answering: A set of prerequisite real tasks. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 8722–8731, 2020

    2020

  46. [54]

    Liu, and Christopher D

    Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks, 2017

    2017

  47. [55]

    Taco-rl: Task aware prompt compression optimization with reinforcement learning.arXiv preprint arXiv:2409.13035, 2024

    Shivam Shandilya, Menglin Xia, Supriyo Ghosh, Huiqiang Jiang, Jue Zhang, Qianhui Wu, and Victor Rühle. Taco-rl: Task aware prompt compression optimization with reinforcement learning.arXiv preprint arXiv:2409.13035, 2024

    2024 arXiv

  48. [56]

    Large language models can be easily distracted by irrelevant context

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. InInternational Conference on Machine Learning, pages 31210–31227. PMLR, 2023

    2023

  49. [57]

    The web as a knowledge-base for answering complex questions

    Alon Talmor and Jonathan Berant. The web as a knowledge-base for answering complex questions. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651, 2018

    2018

  50. [58]

    Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A question answering challenge targeting commonsense knowledge, 2019

    2019

  51. [59]

    Memoryllm: Towards self-updatable large language models

    Yu Wang, Yifan Gao, Xiusi Chen, Haoming Jiang, Shiyang Li, Jingfeng Yang, Qingyu Yin, Zheng Li, Xian Li, Bing Yin, et al. Memoryllm: Towards self-updatable large language models. arXiv preprint arXiv:2402.04624, 2024

    2024 arXiv

  52. [60]

    Learning to filter context for retrieval-augmented generation.arXiv preprint arXiv:2311.08377, 2023

    Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. Learning to filter context for retrieval-augmented generation.arXiv preprint arXiv:2311.08377, 2023

    2023 arXiv

  53. [61]

    Maurice Weber, Daniel Y . Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. R...

    2024

  54. [62]

    Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models

    David Wingate, Mohammad Shoeybi, and Taylor Sorensen. Prompt compression and contrastive conditioning for controllability and toxicity reduction in language models. InFindings of the Association for Computational Linguistics: EMNLP 2022, pages 5621–5634, 2022. 13

    2022

  55. [63]

    Transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020

  56. [64]

    Instructing large language models to identify and ignore irrelevant conditions

    Zhenyu Wu, Chao Shen, and Meng Jiang. Instructing large language models to identify and ignore irrelevant conditions. InNorth American Chapter of the Association for Computational Linguistics, 2024

    2024

  57. [65]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Finding...

    2024

  58. [66]

    From 128k to 4m: Efficient training of ultra-long context large language models.arXiv preprint arXiv:2504.06214, 2025

    Chejian Xu, Wei Ping, Peng Xu, Zihan Liu, Boxin Wang, Mohammad Shoeybi, Bo Li, and Bryan Catanzaro. From 128k to 4m: Efficient training of ultra-long context large language models.arXiv preprint arXiv:2504.06214, 2025

    2025 arXiv

  59. [67]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  60. [68]

    WikiQA: A challenge dataset for open-domain question answering

    Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors,Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013– 2018, Li...

    2015

  61. [69]

    Hotpotqa: A dataset for diverse, explainable multi-hop question answering

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Process...

    2018

  62. [70]

    Extending llama-3’s context ten-fold overnight.arXiv preprint arXiv:2404.19553, 2024

    Peitian Zhang, Ninglu Shao, Zheng Liu, Shitao Xiao, Hongjin Qian, Qiwei Ye, and Zhicheng Dou. Extending llama-3’s context ten-fold overnight.arXiv preprint arXiv:2404.19553, 2024

    2024 arXiv

  63. [71]

    Adacomp: Extractive context compression with adaptive predictor for retrieval-augmented large language models.arXiv preprint arXiv:2409.01579, 2024

    Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. Adacomp: Extractive context compression with adaptive predictor for retrieval-augmented large language models.arXiv preprint arXiv:2409.01579, 2024

    2024 arXiv

  64. [72]

    Beyond llava-hd: Diving into high-resolution large multimodal models.arXiv preprint arXiv:2406.08487, 2024

    Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang, Zhang Zhang, Liang Wang, and Rong Jin. Beyond llava-hd: Diving into high-resolution large multimodal models.arXiv preprint arXiv:2406.08487, 2024

    2024 arXiv

  65. [73]

    Longrag: A dual-perspective retrieval-augmented generation paradigm for long-context question answering.arXiv preprint arXiv:2410.18050, 2024

    Qingfei Zhao, Ruobing Wang, Yukuo Cen, Daren Zha, Shicheng Tan, Yuxiao Dong, and Jie Tang. Longrag: A dual-perspective retrieval-augmented generation paradigm for long-context question answering.arXiv preprint arXiv:2410.18050, 2024

    arXiv

  66. [74]

    George Claus Rankin Sir George Claus Rankin PC (12 August 1877 – 8 April 1946) was a British judge in India

    Wenbo Zhao, Arpit Gupta, Tagyoung Chung, and Jing Huang. Spc: Soft prompt construction for cross domain generalization. InProceedings of the 8th Workshop on Representation Learning for NLP (RepL4NLP 2023), pages 118–130, 2023. A Case Study for Critical Interplay between Local ...

    2023

  67. [76]

    Background: [X] means the same as [D]

    and deepspeed 5 v0.14.0. All experiments were conducted on 8 A100 NVIDIA GPUs, each equipped with 80GB of memory. In Table 6 and Table 7, we list the hyperparameters for Pretraining and Instruction Tuning. 5https://github.com/microsoft/DeepSpeed 17 Table 6: Hyperparameters for...

  68. [77]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

  69. [1957]

    Question:

    was an Australian soldier and politician. He served in both the House of Representatives. . . " Question: "What is George Rankin ’s occupation?" rather than a single person. A compression method that conflates these entities or represents only the first instance fails to prese...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.