pith. sign in

arxiv: 2409.06679 · v3 · submitted 2024-09-10 · 💻 cs.CL

E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

Zihan Liao , Jun Wang , Hang Yu , Lingxiao Wei , Jianguo Li , Wei Zhang This is my paper

Pith reviewed 2026-05-23 20:33 UTC · model grok-4.3

classification 💻 cs.CL
keywords long-context LLMscontext compressionsoft promptsencoder alignmentdocument summarizationquestion answeringLongBenchinstruction fine-tuning
0
0 comments X

The pith

E2LLM divides long contexts into chunks, compresses each into a soft prompt with a pretrained text encoder, and aligns the prompts to a decoder-only LLM via an adapter to handle long inputs efficiently.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents E2LLM to let large language models manage long inputs for tasks such as document summarization and question answering. It splits the input into chunks, encodes each chunk into a compact soft prompt, and passes the prompts to the main model through an adapter without changing the pretrained weights. Training combines reconstruction of the encoder outputs with instruction fine-tuning on long-context examples. This setup is reported to surpass eight existing methods in both accuracy and speed on summarization and question answering while leading results on LongBench v2 for models of similar size. Readers would care because many current models hit compute or memory walls on extended contexts, and a compatible extension method could widen their practical use.

Core claim

E2LLM navigates the impossible triangle of high long-context performance, low computational complexity, and compatibility with pretrained models by dividing long contexts into chunks, compressing each into soft prompts using a pretrained text encoder, aligning these representations with a decoder-only LLM via an adapter, and applying two training objectives of encoder output reconstruction and long-context instruction fine-tuning, which yields better effectiveness and efficiency than prior approaches.

What carries the argument

Chunk-wise soft prompt compression by a pretrained text encoder followed by adapter alignment to the LLM decoder.

If this is right

  • Outperforms eight state-of-the-art methods in both effectiveness and efficiency on document summarization and question answering.
  • Achieves the best performance on LongBench v2 among models of comparable size.
  • Preserves compatibility with existing pretrained decoder-only LLMs.
  • Lowers computational complexity for processing extended contexts compared with direct long-sequence handling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same chunk-and-encode pattern could be tested on tasks beyond text, such as long code repositories or multi-document collections.
  • Deployment costs for long-document applications might drop because the LLM sees only short prompt sequences after compression.
  • Pairing different encoders with the same LLM might reveal how much the choice of encoder affects final reasoning quality.

Load-bearing premise

The compressed soft prompts retain enough detail from the original long context to support accurate understanding and reasoning in the LLM.

What would settle it

A controlled experiment on a long-context benchmark where key facts are distributed across distant chunks and E2LLM produces measurably lower accuracy than baselines that process the full text directly.

Figures

Figures reproduced from arXiv: 2409.06679 by Hang Yu, Jianguo Li, Jun Wang, Lingxiao Wei, Wei Zhang, Zihan Liao.

Figure 1
Figure 1. Figure 1: E2LLM solves the “impossible tri￾angle” challenge of Performance, Efficiency, and Compatibility. Length Extension: The first group of methods adjust the position embeddings of LLMs to accommodate longer context extensions Peng et al. (2023); Ding et al. (2024a). This typically involves selecting a large base value for RoPE (Su et al., 2024) followed by con￾tinued pretraining or fine-tuning Zhao et al. (202… view at source ↗
Figure 2
Figure 2. Figure 2: The E2LLM architecture. answers. Moreover, pre-trained encoder models are inherently crafted to produce chunk-level representations. As a result, this design allows E2LLM to leverage the strengths of both pre-trained encoders and decoders, minimizing the need for extensive additional training (T3). Additionally, compressing each original chunk into a single vector (the chunk token) not only enhances traini… view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of all methods on training and inference efficiency. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Case study on extremely long-context input in LongBench v2. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Effect of the hyperparameter. (a) the loss weight of “understanding” task. (b) the lora rank [PITH_FULL_IMAGE:figures/full_fig_p036_5.png] view at source ↗
read the original abstract

Processing long contexts is increasingly important for Large Language Models (LLMs) in tasks like multi-turn dialogues, code generation, and document summarization. This paper addresses the challenges of achieving high long-context performance, low computational complexity, and compatibility with pretrained models -- collectively termed the ``impossible triangle''. We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. E2LLM divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. To enhance the LLM's reasoning with these soft prompts, we employ two training objectives: encoder output reconstruction and long-context instruction fine-tuning. Extensive experiments reveal that E2LLM not only outperforms 8 state-of-the-art (SOTA) methods in effectiveness and efficiency for document summarization and question answering, but also achieves the best performance on LongBench v2 among models of comparable size.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces E2LLM to address the 'impossible triangle' of high long-context performance, low computational complexity, and pretrained-model compatibility. Long inputs are chunked and each chunk is compressed into soft prompts by a pretrained text encoder; these are aligned to a decoder-only LLM via an adapter. Training combines encoder-output reconstruction with long-context instruction tuning. The central empirical claim is that E2LLM outperforms eight prior SOTA methods in both effectiveness and efficiency on document summarization and QA while also achieving the best LongBench v2 score among models of comparable size.

Significance. If the reported gains are reproducible and the information-retention properties of the encoder-compression step are confirmed, the work would offer a practical route to long-context modeling that re-uses existing pretrained components without quadratic attention costs, which is a meaningful engineering contribution in the current landscape of long-context LLM research.

major comments (2)
  1. [Abstract, §4] Abstract and §4 (Experiments): the headline claims of outperforming eight SOTA baselines and topping LongBench v2 rest on experimental results whose design, datasets, metrics, controls, and statistical tests are not described in sufficient detail to allow assessment of whether the data actually support the stated superiority.
  2. [§3] §3 (Method): the central modeling assumption—that chunk-wise encoder compression followed by adapter alignment preserves task-relevant information for downstream reasoning—is load-bearing for all performance claims, yet the manuscript provides no quantitative analysis (e.g., reconstruction fidelity per chunk, information-loss ablations, or attention-map comparisons) that would substantiate retention of reasoning-critical content.
minor comments (2)
  1. [§3] Notation for the soft-prompt tensor and the adapter module should be introduced with explicit dimensionalities and a diagram that distinguishes the frozen encoder, adapter, and LLM components.
  2. [§3.2] The two training objectives (reconstruction and instruction tuning) are mentioned but their relative weighting, scheduling, and data mixtures are not specified; a short paragraph or table would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive feedback. We address the two major comments point by point below, indicating planned revisions where the manuscript requires strengthening.

read point-by-point responses

  1. Referee: [Abstract, §4] Abstract and §4 (Experiments): the headline claims of outperforming eight SOTA baselines and topping LongBench v2 rest on experimental results whose design, datasets, metrics, controls, and statistical tests are not described in sufficient detail to allow assessment of whether the data actually support the stated superiority.

    Authors: We agree that the current level of detail in the experimental section is insufficient to allow independent assessment of the reported gains. In the revised manuscript we will expand §4 with full specifications of all datasets (including sizes, sources, and preprocessing), exact evaluation metrics and their implementations, baseline reproduction details, experimental controls, and any statistical tests performed. This will directly address the concern about substantiating the superiority claims. revision: yes

  2. Referee: [§3] §3 (Method): the central modeling assumption—that chunk-wise encoder compression followed by adapter alignment preserves task-relevant information for downstream reasoning—is load-bearing for all performance claims, yet the manuscript provides no quantitative analysis (e.g., reconstruction fidelity per chunk, information-loss ablations, or attention-map comparisons) that would substantiate retention of reasoning-critical content.

    Authors: The referee correctly notes the absence of direct quantitative support for information retention. Although the training objective includes encoder-output reconstruction, we did not report per-chunk fidelity metrics or targeted ablations in the submitted version. We will add these analyses to the revised manuscript, including reconstruction error statistics across chunks and ablations measuring the impact of compression on downstream task performance. revision: yes

Circularity Check

0 steps flagged

No significant circularity; architectural claims rest on empirical evaluation

full rationale

The paper presents E2LLM as an engineering architecture: chunking long inputs, pretrained-encoder compression to soft prompts, adapter alignment, plus reconstruction + instruction-tuning objectives. All performance numbers (outperformance on summarization/QA, LongBench v2) are reported from direct experiments against external baselines. No derivation reduces a claimed prediction to a fitted parameter by construction, no uniqueness theorem is invoked via self-citation, and no ansatz is smuggled through prior work. The central claim is therefore an empirical statement about the proposed pipeline, not a tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

The abstract does not specify any free parameters, axioms, or invented entities.

pith-pipeline@v0.9.0 · 5716 in / 1132 out tokens · 39110 ms · 2026-05-23T20:33:51.297468+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

50 extracted references · 50 canonical work pages · 11 internal anchors

  1. [1]

    Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, et al. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. 2024

    work page 2024

  2. [2]

    Repocoder: Repository-level code completion through iterative retrieval and generation

    Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471--2484, 2023

    work page 2023

  3. [3]

    Open domain multi-document summarization: A comprehensive study of model brittleness under retrieval

    John Michael Giorgi, Luca Soldaini, BO WANG, Gary D Bader, Kyle Lo, Lucy Lu Wang, and Arman Cohan. Open domain multi-document summarization: A comprehensive study of model brittleness under retrieval. In The 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  4. [4]

    End-to-end training of multi-document reader and retriever for open-domain question answering

    Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering. Advances in Neural Information Processing Systems, 34: 0 25968--25981, 2021

    work page 2021

  5. [5]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    work page 2022

  6. [6]

    A Survey on In-context Learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  7. [7]

    A survey on rag meets llms: Towards retrieval-augmented large language models

    Yujuan Ding, Wenqi Fan, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meets llms: Towards retrieval-augmented large language models. arXiv preprint arXiv:2405.06211, 2024

    work page arXiv 2024

  8. [8]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    work page 2024

  9. [9]

    Longrope: Extending llm context window beyond 2 million tokens

    Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. Longrope: Extending llm context window beyond 2 million tokens. In Forty-first International Conference on Machine Learning

    work page

  10. [10]

    Llmlingua: Compressing prompts for accelerated inference of large language models

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Llmlingua: Compressing prompts for accelerated inference of large language models. arXiv preprint arXiv:2310.05736, 2023 a

    work page arXiv 2023

  11. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2, 2019

    work page 2019

  12. [12]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations

    work page

  13. [13]

    A length-extrapolatable transformer

    Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A length-extrapolatable transformer. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023

    work page 2023

  14. [14]

    Llama 2: Open Foundation and Fine-Tuned Chat Models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [15]

    Qwen Technical Report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    Extending Context Window of Large Language Models via Positional Interpolation

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023 a

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Ntk-aware scaled rope allows llama models to have extended(8k+) context size without any fine-tuning and minimal perplexity degradation

    bloc97. Ntk-aware scaled rope allows llama models to have extended(8k+) context size without any fine-tuning and minimal perplexity degradation. 2023

    work page 2023

  18. [18]

    YaRN: Efficient Context Window Extension of Large Language Models

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Efficient streaming language models with attention sinks

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. The Twelfth International Conference on Learning Representations, 2024

    work page 2024

  20. [20]

    Dynamic context pruning for efficient and interpretable autoregressive transformers

    Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurelien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers. Advances in Neural Information Processing Systems, 36, 2023

    work page 2023

  21. [21]

    Sparser is faster and less is more: Efficient sparse attention for long-range transformers

    Chao Lou, Zixia Jia, Zilong Zheng, and Kewei Tu. Sparser is faster and less is more: Efficient sparse attention for long-range transformers. arXiv preprint arXiv:2406.16747, 2024

    work page arXiv 2024

  22. [22]

    Lm-infinite: Zero-shot extreme length generalization for large language models

    Chi Han, Qifan Wang, Hao Peng, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3991--4008, 2024

    work page 2024

  23. [23]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021

    work page 2021

  24. [24]

    Lloco: Learning long contexts offline

    Sijun Tan, Xiuyu Li, Shishir Patil, Ziyang Wu, Tianjun Zhang, Kurt Keutzer, Joseph E Gonzalez, and Raluca Ada Popa. Lloco: Learning long contexts offline. arXiv preprint arXiv:2404.07979, 2024

    work page arXiv 2024

  25. [25]

    Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering

    Yucheng Li. Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering. arXiv preprint arXiv:2304.12102, 2023

    work page arXiv 2023

  26. [26]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression,

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839, 2023 b

    work page arXiv 2023

  27. [27]

    Learning to compress prompts with gist tokens

    Jesse Mu, Xiang Li, and Noah Goodman. Learning to compress prompts with gist tokens. Advances in Neural Information Processing Systems, 36, 2023

    work page 2023

  28. [28]

    In-context autoencoder for context compression in a large language model

    Tao Ge, Jing Hu, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei. In-context autoencoder for context compression in a large language model. arXiv preprint arXiv:2307.06945, 2023

    work page arXiv 2023

  29. [29]

    Adapting language models to compress contexts

    Alexis Chevalier, Alexander Wettig, Anirudh Ajith, and Danqi Chen. Adapting language models to compress contexts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2023

    work page 2023

  30. [30]

    Towards General Text Embeddings with Multi-stage Contrastive Learning

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL https://arxiv.org/abs/2308.03281

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  31. [31]

    C-Pack: Packed Resources For General Chinese Embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  32. [32]

    Gaussian Error Linear Units (GELUs)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  33. [33]

    Vision-language models for vision tasks: A survey

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    work page 2024

  34. [34]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations

    work page

  35. [35]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    work page 2024

  36. [36]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023 b

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  37. [37]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185--24198, 2024

    work page 2024

  38. [38]

    A Survey on Optical Character Recognition System

    Noman Islam, Zeeshan Islam, and Nazia Noor. A survey on optical character recognition system. arXiv preprint arXiv:1710.05703, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  39. [39]

    Unraveling and mitigating retriever inconsistencies in retrieval-augmented large language models

    Mingda Li, Xinyu Li, Yifan Chen, Wenfeng Xuan, and Weinan Zhang. Unraveling and mitigating retriever inconsistencies in retrieval-augmented large language models. arXiv preprint arXiv:2405.20680, 2024

    work page arXiv 2024

  40. [40]

    Longlora: Efficient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309.12307, 2023 b

    work page arXiv 2023

  41. [41]

    Retrieval-Augmented Generation for Large Language Models: A Survey

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024. URL https://arxiv.org/abs/2312.10997

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Qmsum: A new benchmark for query-based multi-domain meeting summarization,

    Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, et al. Qmsum: A new benchmark for query-based multi-domain meeting summarization. arXiv preprint arXiv:2104.05938, 2021

    work page arXiv 2021

  43. [43]

    Efficient attentions for long document summarization

    Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, pages 1419--1436, 2021

    work page 2021

  44. [44]

    Quality: Question answering with long input texts, yes! NAACL 2022, 2022

    Samuel R Bowman, Angelica Chen, He He, Nitish Joshi, Johnny Ma, Nikita Nangia, Vishakh Padmakumar, Richard Yuanzhe Pang, Alicia Parrish, Jason Phang, et al. Quality: Question answering with long input texts, yes! NAACL 2022, 2022

    work page 2022

  45. [45]

    The NarrativeQA reading comprehension challenge

    Tomas Kovcisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G\'abor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 2018

    work page 2018

  46. [46]

    Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601--1611, 2017

    work page 2017

  47. [47]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81, 2004

    work page 2004

  48. [48]

    Zeroscrolls: A zero-shot benchmark for long text understanding

    Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977--7989, 2023

    work page 2023

  49. [49]

    How easily do irrelevant inputs skew the responses of large language models? arXiv preprint arXiv:2404.03302, 2024

    Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. How easily do irrelevant inputs skew the responses of large language models? arXiv preprint arXiv:2404.03302, 2024

    work page arXiv 2024

  50. [50]

    Context embeddings for efficient answer generation in rag

    David Rau, Shuai Wang, Herv \'e D \'e jean, and St \'e phane Clinchant. Context embeddings for efficient answer generation in rag. arXiv preprint arXiv:2407.09252, 2024

    work page arXiv 2024