Pith. sign in

REVIEW 5 major objections 5 minor 48 references

Efficient Long Context Language Model Retrieval with Compression

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small trained compressor can shrink LCLM retrieval inputs by 1.91x and still improve retrieval accuracy by 6%.

desk verdict Valuable compression method for LCLM retrieval, but the length-regularized loss has a real technical gap and the consistency claim is overstated. read the letter →

arxiv 2412.18232 v2 pith:KCYKOR7H submitted 2024-12-24 cs.IR

Minju Seo , Jinheon Baek , Seongyun Lee , Sung Ju Hwang This is my paper · ORCID
classification cs.IR
keywords long-contextlanguagemodelsinformationretrievalpassagecompressionpreferenceoptimizationORPOin-contextsyntheticdataefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Long-context language models can retrieve by reading a whole corpus in one context window, but every extra token costs compute and time. This paper claims that a small passage-compression model trained specifically for retrieval—CoLoR—cuts that cost: across nine retrieval benchmarks it reduces the in-context token count by a factor of 1.91 while improving average retrieval performance by 6% over raw passages. The training recipe uses synthetic preference data: several language models summarize each passage, the LCLM's retrieval outcome labels each summary as chosen or rejected, and the shortest successful summary is preferred. If the claim holds, compression is not a quality trade-off for LCLM retrieval but a way to make it cheaper and slightly more accurate at the same time.

What carries the argument

The load-bearing object is the training objective: Odds Ratio Preference Optimization (ORPO), a preference-learning method that raises the odds of a chosen output over a rejected one without a separate reference model, augmented by a length-regularization term. CoLoR multiplies the ORPO odds-ratio loss by $(|y_l|-|y_w|)$, the token-length difference between rejected and chosen compressions, so pairs where the chosen compression is much shorter produce larger updates. The synthetic preference data is equally central: multiple LLMs summarize each passage, and the LCLM's retrieval outcome labels each summary chosen or rejected, with the shortest successful summary chosen. That data-generation loop is what lets the compressor optimize retrieval success and brevity simultaneously without human-written summaries.

What would settle it

Inspect the 3,418 training pairs and count how often the rejected compression is shorter than the chosen one; if a non-negligible share has negative $(|y_l|-|y_w|)$, the stated length-regularization objective is not the one being optimized, and the reported 1.91x compression would need re-explanation. Re-running the same training with a sign-corrected factor would show whether the brevity gains come from the claimed mechanism.

Watch

Extended reading notes

Core claim

CoLoR is an abstractive passage compressor that rewrites each corpus document into a shorter passage and lets the LCLM retrieve over the compressed corpus. The paper's central claim is that this compressor, trained with Odds Ratio Preference Optimization on automatically labeled pairs, produces passages that are both shorter and more retrieval-effective than the originals. The data-generation loop is the key: multiple LLMs summarize each passage, retrieval success with a fixed LCLM labels each summary chosen or rejected, and among successful summaries the shortest is chosen. The training objective adds a length-regularization term to the ORPO odds-ratio loss, scaling updates by $|y_l| - |y_w|$, the token-length difference between rejected and chosen compressions. On nine single- and multi-document datasets, CoLoR reports a 1.91x average compression with a 6% average retrieval gain over raw passages, and the paper further reports that it outperforms extractive, prompt-based, and RAG-oriented compression baselines, generalizes to out-of-domain datasets, and mitigates the lost-in-the-middle degradation.

Load-bearing premise

The whole training procedure relies on rejected compressions always being longer than chosen ones; if a failed compression is shorter, the brevity bonus reverses and pushes the model toward worse preferences.

Editorial extensions

If this is right

  • If CoLoR's results hold, LCLM retrieval can run at roughly half the context length without losing accuracy, lowering compute and memory per query and allowing larger corpora inside a fixed context window.
  • Compression becomes a one-time indexing step: passages are compressed and cached once, so the added cost is amortized across all future queries, similar to building an index.
  • The benefit is not confined to LCLMs: the paper reports that dense retrieval also improves on compressed passages, so the same compressor could serve both retrieval paradigms.
  • CoLoR mitigates the lost-in-the-middle effect: performance degrades less when relevant passages sit in the middle of the context, suggesting compression removes distracting content.
  • The approach transfers to unseen datasets and even to very long passages, where compressed inputs reached a 216x size reduction with a 42% accuracy gain over truncated raw passages in the paper's long-context benchmark.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the preference-labeling loop is what carries the gains, the same recipe could be applied to other in-context tasks where output quality is easily evaluated—for example compressing documents for text-to-SQL or multi-document reasoning—rather than only retrieval.
  • The 1.91x compression is an average over datasets; the paper's own tables show larger gains on some datasets and near-parity on others, so a practical deployment would likely compress selectively, only where the trade-off is favorable.
  • A direct test of the brevity mechanism would be to compute $|y_l|-|y_w|$ on all 3,418 training pairs; if rejected compressions are sometimes shorter than chosen ones, the sign of the regularization flips and the stated objective does not match the implemented one.
  • Because compression is trained on retrieval outcomes from a specific LCLM, the compressor implicitly inherits that model's failure modes; using a different LCLM at inference time may not preserve the same gains without retraining or adaptation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes CoLoR, a passage-compression model for long-context language model (LCLM) retrieval. Synthetic compressed passages are generated with several LLMs, labeled chosen/rejected based on whether the LCLM retrieves the correct passage and on relative length, and used to train a Phi-3-mini compressor with ORPO plus a length-difference regularization. Evaluation on nine retrieval datasets reports an average relative improvement of 5.9% in retrieval performance and a 1.91x average compression rate over raw passages, with ablations and out-of-domain tests. Code is released.

Significance. If the reported average gains are robust, CoLoR is a practically useful and inexpensive way to reduce the inference cost of LCLM retrieval while preserving or improving accuracy. The paper provides a complete pipeline and a released implementation, which is a strength. However, the evidence has several gaps: the length-regularization objective is not guaranteed to have the sign implied by Eq. (3), the 'consistently outperforms' claim is contradicted by Table 1, and the long-context benchmark result in Table 11 is not produced by CoLoR.

major comments (5)
  1. [§3.2.1, Eq. (3)] The length-regularization term in Eq. (3) multiplies the ORPO odds-ratio loss by (|yl|−|yw|), with the claim that yl is always longer than yw based on the data collection criteria. However, the labeling procedure in §3.2.1 labels every failed compression as rejected, without any requirement that failed compressions be longer than the chosen one. A failed compression can be shorter than the chosen compression, making |yl|−|yw| negative and reversing the preference gradient for that pair. Table 9 reports only average token counts (rejected 135.61 vs. chosen 91.36), which does not rule out per-pair negative differences; no filtering or verification is described. Because Eq. (3) is the stated training objective, the paper needs to either verify the length ordering in the actual data or code, or modify the objective (e.g., use an absolute value or only pairs satisfying the ordering).
  2. [§5.1, Table 1] The claim that CoLoR 'consistently outperforms all baseline methods' is not supported by Table 1. CoLoR underperforms Raw Passage on FEVER (0.94 vs. 0.95), HotpotQA (0.86 vs. 0.87), and QAMPARI (0.55 vs. 0.56); it also underperforms Comp. w/ GPT on FEVER (0.94 vs. 0.96) and NQ (0.98 vs. 0.99), and COMPACT on FEVER (0.94 vs. 0.96). While the average and compression-rate gains are real, the 'consistently' phrasing should be replaced with a statement about average improvements and per-dataset outcomes.
  3. [Appendix B, Table 11] The long-context retrieval result in Table 11 is not produced by CoLoR: the appendix states that the compressed passages are generated using GPT-4o-mini prompted for summaries under 200 words, and that training CoLoR on those datasets is infeasible. This table therefore does not support the claim that CoLoR is effective in long-context retrieval scenarios; it should be relabeled as a compression-prompting baseline or removed from the CoLoR evaluation.
  4. [Table 1 and Table 2] No error bars, variance estimates, or significance tests are reported for any of the main results. Given the small absolute differences on several datasets (e.g., 0.98 vs. 0.99 on NQ, 0.55 vs. 0.56 on QAMPARI) and the regressions noted above, the robustness of the headline 6% improvement is unclear. The paper should report results over multiple seeds (or at least standard deviations) and state whether the average gain is statistically reliable.
  5. [§4.2 and §5] The preference data are labeled using GPT-4o-mini retrieval, and the final evaluation also uses GPT-4o-mini as the LCLM. This shared-LCLM design means CoLoR could be overfitting to the idiosyncrasies of GPT-4o-mini's retrieval behavior. The paper's generalizability experiments vary the compression base model and datasets but never vary the retrieval LCLM. A simple validation would be to evaluate the same compressed passages with a different LCLM (e.g., Llama-3.1-70B or Qwen2.5) to confirm that the gains transfer.
minor comments (5)
  1. [Conclusion] The word 'Langauge' should be 'Language'.
  2. [References] Several reference entries contain formatting artifacts, e.g., 'F ound. Trends Inf. Retr.' and 'V olume' should be 'Found.' and 'Volume'.
  3. [Figure 3] The scatter plot does not clearly label which point corresponds to which method for all methods; consider adding point labels or a table with the same data.
  4. [Table 2] The asterisk on CoLoR* is not defined in the table caption; it is explained in the text, but a footnote would help.
  5. [§3.2.1] The text says 'reject' in a few places where 'rejected' is intended; please standardize the terminology.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CoLoR's reported gains are empirical results measured against external benchmarks, not derivations that reduce to the training objective or to self-citations.

full rationale

The paper's central claims—6% retrieval improvement and 1.91x compression—are experimental measurements on nine retrieval benchmarks, including out-of-domain splits, rather than quantities derived from the training objective by construction. The loss in Eq. (3) combines ORPO with a length-difference multiplier, but this does not algebraically force the observed retrieval scores or compression ratios; the model could fail to improve either metric, and the ablation study treats the components as empirical contributors. The preference labels are generated by the same LCLM (GPT-4o-mini) used for evaluation, which is a self-referential evaluation setup, but it is not a derivation-level circularity: the evaluation passages and queries are distinct from the training pairs, and out-of-domain experiments provide independent evidence. The stated assumption that y_l is always longer than y_w is not guaranteed by the data-collection rules for failed compressions, and this is a legitimate correctness risk about the objective's implementation, not a circular reduction of the results to the inputs. The benchmark and prompting setup cite Lee et al. (2024), an external prior work with no author overlap, and ORPO is an externally cited method; no load-bearing self-citations or imported uniqueness claims are present. The derivation chain is therefore self-contained as an empirical training-and-evaluation pipeline, and no circular step can be exhibited.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new formal entities. It depends on standard deep learning hyperparameters and a specific data labeling assumption: retrieval success of the evaluation LCLM is the reward signal. The most consequential assumption is that the shortest successful compression is the best compression target, which the preference data encodes.

free parameters (4)
  • ORPO loss weight lambda = 2.5
    Chosen by hand or validation; balances the SFT term against the odds-ratio term in Eq. (2). The central results depend on this trade-off.
  • Learning rate for ORPO training = 1e-6 (Phi, Llama), 5e-6 (Mistral)
    Reported in Appendix A; selected by validation. Affects the compression model and therefore the measured gains.
  • Training epochs = 10 (best epoch by validation)
    Reported in Appendix A; a hyperparameter that controls underfitting and overfitting of the compression model.
  • Batch size = 8
    Reported in Appendix A; standard training hyperparameter.
assumptions (4)
  • domain assumption Retrieval success of compressed passages on the evaluation LCLM is a valid and sufficient label for compression quality.
    Section 3.2.1 defines chosen and rejected labels based solely on whether the LCLM retrieves the passage; if the LCLM's retrieval is noisy or biased, the training signal inherits that noise.
  • domain assumption ORPO without a reference model is an effective objective for this task.
    Adopted from Hong et al. (2024) in Section 3.2.1; the paper does not compare alternative preference objectives.
  • domain assumption Corpus-in-context prompting from Lee et al. (2024) is an appropriate and transferable evaluation setting.
    Used for all 9 benchmarks in Section 4.4; the compression results are measured only within this prompt format.
  • domain assumption The synthetic summaries generated by Phi-3, Mistral, Llama, and GPT-4o-mini cover the space of effective compressions.
    Data collection in Section 3.2.1 relies on these four LLMs; a compression that none of them produces cannot be learned as a chosen target.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Long Context Language Model Retrieval with Compression." pith.science (2026). https://pith.science/paper/KCYKOR7H

@misc{pith2026241218232,
  author       = {Pith},
  title        = {Pith review of: Efficient Long Context Language Model Retrieval with Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KCYKOR7H}},
  note         = {Machine review of arXiv:2412.18232}
}
read the original abstract

Long Context Language Models (LCLMs) have emerged as a new paradigm to perform Information Retrieval (IR), which enables the direct ingestion and retrieval of information by processing an entire corpus in their single context, showcasing the potential to surpass traditional sparse and dense retrieval methods. However, processing a large number of passages within in-context for retrieval is computationally expensive, and handling their representations during inference further exacerbates the processing time; thus, we aim to make LCLM retrieval more efficient and potentially more effective with passage compression. Specifically, we propose a new compression approach tailored for LCLM retrieval, which is trained to maximize the retrieval performance while minimizing the length of the compressed passages. To accomplish this, we generate the synthetic data, where compressed passages are automatically created and labeled as chosen or rejected according to their retrieval success for a given query, and we train the proposed Compression model for Long context Retrieval (CoLoR) with this data via preference optimization while adding the length regularization loss on top of it to enforce brevity. Through extensive experiments on 9 datasets, we show that CoLoR improves the retrieval performance by 6% while compressing the in-context size by a factor of 1.91. Our code is available at: https://github.com/going-doer/CoLoR.

Figures

Figures reproduced from arXiv: 2412.18232 by the authors.

Figure 1
Figure 1. Comparison of different IR approaches. (A) Dense Retrieval. To identify relevant documents to the given query, it first embeds them into the vector space and then calculates their semantic similarity. (B) LCLM Retrieval. The LCLM takes and processes the raw passages from the corpus along with the query in the input context, and identifies the relevant passages. (C) CoLoR. We compress the raw passages, and use the co… view at source ↗
Figure 2
Figure 2. Overview of Training Processes for CoLoR. 1. We first create the training data for CoLoR by generating multiple compressed passages from their original passages with multiple LMs. 2. The compressed passages and their associated query are used as input to the LCLM, and their retrieval performance is measured to label them as either chosen or rejected based on retrieval results. 3. CoLoR is trained using the pairs of … view at source ↗
Figure 3
Figure 3. The trade-off of different methods, showing their compression rate (x-axis) and retrieval performance (y-axis). trieval category) and evaluating its performance on datasets from other domains (other retrieval cate￾gories). As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Results with varying the position of (compressed) passages associated with the query within the corpus, where 0% (on the x-axis) represents beginning. of successful retrieval over the ones with unsuccess￾ful retrieval (while enforcing brevity as well). Analysis on Comp…
Figure 5
Figure 5. Figure 5: Results with varying the position of (compressed) passages for all datasets. Specifically, we arbitrarily adjust the positions of the gold and few-shot passages within the corpus relative to the query (0% represents the beginning). The figures at the top, middle, and b…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 8 canonical work pages

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  2. [2]

    Samuel Amouyal, Tomer Wolfson, Ohad Rubin, Ori Yoran, Jonathan Herzig, and Jonathan Berant. 2023. https://aclanthology.org/2023.gem-1.9/ QAMPARI : A benchmark for open-domain questions with many answers . In Proceedings of the Third Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 97--110, Singapore. Association for Computatio...

    work page 2023

  3. [3]

    Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.776 L-eval: Instituting standardized evaluation for long context language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2...

    doi:10.18653/v1/2024.acl-long.776 2024

  4. [4]

    Anthropic. 2024. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf The claude 3 model family: Opus, sonnet, haiku

    2024

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. https://doi.org/10.18653/V1/2024.ACL-LONG.172 Longbench: A bilingual, multitask benchmark for long context understanding . In Proceedings of the 62nd Annual Meeting of the Association for Com...

    doi:10.18653/v1/2024.acl-long.172 2024

  6. [6]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. https://arxiv.org/abs/2404.05961 Llm2vec: Large language models are secretly powerful text encoders . Preprint, arXiv:2404.05961

    arXiv 2024

  7. [7]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. https://arxiv.org/abs/2004.05150 Longformer: The long-document transformer . Preprint, arXiv:2004.05150

    arXiv 2020

  8. [8]

    Gormley, and Graham Neubig

    Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2025. https://aclanthology.org/2025.naacl-long.605/ In-context learning with long-context models: An in-depth exploration . In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: H...

    work page 2025

Show all 48 references
  1. [9]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. https://aclanthology.org/2024.emnlp-main.626 ORPO: monolithic preference optimization without reference model . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, ...

    2024

  2. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  3. [11]

    Huiqiang Jiang, Qianhui Wu, Chin - Yew Lin, Yuqing Yang, and Lili Qiu. 2023 b . https://doi.org/10.18653/V1/2023.EMNLP-MAIN.825 Llmlingua: Compressing prompts for accelerated inference of large language models . In Proceedings of the 2023 Conference on Empirical Methods in Nat...

    2023 doi

  4. [12]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen - tau Yih. 2020. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empir...

    2020 doi

  5. [13]

    Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. https://arxiv.org/abs/2404.01261 Fables: Evaluating faithfulness and content selection in book-length summarization . Preprint, arXiv:2404.01261

    2024 arXiv

  6. [15]

    Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, a...

    2024 arXiv

  7. [16]

    Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. 2024 a . https://doi.org/10.18653/V1/2024.ACL-LONG.859 Loogle: Can long-context language models understand long contexts? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume ...

    2024 doi

  8. [17]

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024 b . https://arxiv.org/abs/2404.02060 Long-context llms struggle with long in-context learning . Preprint, arXiv:2404.02060

    2024 arXiv

  9. [18]

    Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.391 Compressing context to enhance inference efficiency of large language models . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, ...

    2023 doi

  10. [19]

    Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. 2024 a . https://arxiv.org/abs/2406.06025 Repoqa: Evaluating long context code understanding . Preprint, arXiv:2406.06025

    2024 arXiv

  11. [20]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 b . https://doi.org/10.1162/TACL\_A\_00638 Lost in the middle: How language models use long contexts . Trans. Assoc. Comput. Linguistics, 12:157--173

    2024 doi

  12. [21]

    AI @ Meta Llama Team. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  13. [22]

    Macedo Maia, Siegfried Handschuh, Andr \' e Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. https://doi.org/10.1145/3184558.3192301 Www'18 open challenge: Financial opinion mining and question answering . In Companion of the The Web Conference...

    2018

  14. [23]

    Chaitanya Malaviya, Peter Shaw, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2023. https://doi.org/10.18653/v1/2023.acl-long.784 QUEST : A retrieval dataset of entity-seeking queries with implicit set operations . In Proceedings of the 61st Annual Meeting of the Associa...

    2023 doi

  15. [24]

    Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. https://ceur-ws.org/Vol-1773/CoCoNIPS\_2016\_paper9.pdf MS MARCO: A human generated machine reading comprehension dataset . In Proceedings of the Workshop on Cognitive Comput...

    2016

  16. [25]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  17. [26]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  18. [27]

    Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K \" o pf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Stei...

    2019

  19. [28]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. https://openreview.net/forum?id=wHBfxhZu1u Yarn: Efficient context window extension of large language models . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria,...

    2024

  20. [29]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html Direct preference optimization: Your language model is sec...

    2023

  21. [31]

    Nils Reimers and Iryna Gurevych. 2019 b . https://doi.org/10.18653/V1/D19-1410 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference ...

    2019 doi

  22. [32]

    Robertson and Hugo Zaragoza

    Stephen E. Robertson and Hugo Zaragoza. 2009. https://doi.org/10.1561/1500000019 The probabilistic relevance framework: BM25 and beyond . Found. Trends Inf. Retr., 3(4):333--389

    2009 doi

  23. [33]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf Learning to summarize with human fee...

    2020

  24. [34]

    Gemini Team. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context . Preprint, arXiv:2403.05530

    2024 arXiv

  25. [35]

    Nandan Thakur, Nils Reimers, Andreas R \" u ckl \' e , Abhishek Srivastava, and Iryna Gurevych. 2021. https://arxiv.org/abs/2104.08663 BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models . arXiv preprint arXiv:2104.08663, abs/2104.08663

    2021 arXiv

  26. [36]

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. https://doi.org/10.18653/V1/N18-1074 FEVER: a large-scale dataset for fact extraction and verification . In Proceedings of the 2018 Conference of the North American Chapter of the Association f...

    2018 doi

  27. [37]

    Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. https://doi.org/10.1162/tacl_a_00475 M u S i Q ue: Multihop questions via single-hop question composition . Transactions of the Association for Computational Linguistics, 10:539--554

    2022 doi

  28. [38]

    David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.609 Fact or fiction: Verifying scientific claims . In Proceedings of the 2020 Conference on Empirical Methods in Na...

    2020 doi

  29. [39]

    Chonghua Wang, Haodong Duan, Songyang Zhang, Dahua Lin, and Kai Chen. 2024. https://doi.org/10.18653/V1/2024.NAACL-LONG.205 Ada-leval: Evaluating long-context llms with length-adaptable benchmarks . In Proceedings of the 2024 Conference of the North American Chapter of the Ass...

    2024 doi

  30. [40]

    Zhiruo Wang, Jun Araki, Zhengbao Jiang, Md Rizwan Parvez, and Graham Neubig. 2023. https://arxiv.org/abs/2311.08377 Learning to filter context for retrieval-augmented generation . Preprint, arXiv:2311.08377

    2023 arXiv

  31. [41]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \' e mi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger,...

    2020 doi

  32. [42]

    Fangyuan Xu, Weijia Shi, and Eunsol Choi. 2024 a . https://openreview.net/forum?id=mlJLVigNHp RECOMP: improving retrieval-augmented lms with context compression and selective augmentation . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna,...

    2024

  33. [43]

    Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2024 b . https://openreview.net/forum?id=xw5nxFWMlo Retrieval meets long context large language models . In The Twelfth Interna...

    2024

  34. [44]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. https://doi.org/10.18653/V1/D18-1259 Hotpotqa: A dataset for diverse, explainable multi-hop question answering . In Proceedings of the 2018 Conference...

    2018 doi

  35. [45]

    Chanwoong Yoon, Taewhoo Lee, Hyeon Hwang, Minbyul Jeong, and Jaewoo Kang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1194 C omp A ct: Compressing retrieved documents actively for question answering . In Proceedings of the 2024 Conference on Empirical Methods in Natural ...

    2024 doi

  36. [46]

    Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. 2024. https://arxiv.org/abs/2402.05136 Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k . Prepr...

    2024

  37. [47]

    Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. 2023. https://arxiv.org/abs/2305.10425 Slic-hf: Sequence likelihood calibration with human feedback . Preprint, arXiv:2305.10425

    2023 arXiv

  38. [48]

    Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. https://aclanthology.org/2024.emnlp-main.47 Longembed: Extending embedding models for long context retrieval . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language ...

    2024

  39. [49]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [50]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.