Pith. sign in

REVIEW 3 major objections 5 minor 54 references

GeAR: Generation Augmented Retrieval

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read GeAR claims that training a bi-encoder retriever with a generation objective and a cross-attention fusion module gives it simultaneous global and local retrieval ability, at no additional retrieval cost.

desk verdict Solid local-retrieval mechanism, but the global-retrieval claim is undercut by the paper's own ablation data. read the letter →

arxiv 2501.02772 v2 pith:SYI2U3FU submitted 2025-01-06 cs.IR cs.CL

Haoyu Liu , Shaohan Huang , Jianfeng Liu , Yuefeng Zhan , Hao Sun , Weiwei Deng , Feng Sun , Furu Wei
show 1 more author
Qi Zhang
This is my paper · ORCID
classification cs.IRcs.CL
keywords GeARgeneration-augmentedretrievalbi-encoderlocalinformationcross-attentionfusioncontrastivelearninglanguage-modelinglosssyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a bi-encoder retriever can learn global document relevance and local sentence-level relevance at the same time if it is trained with a generation objective alongside contrastive learning. The proposed model, GeAR, adds a fusion encoder and a lightweight text decoder to the classic query-document bi-encoder. At inference, document ranking uses only the bi-encoder, so the global retrieval step costs the same as a standard bi-encoder, while the query-relevant sentences inside a document are ranked by cross-attention weights. On their benchmarks, GeAR reports a 3.5% relative improvement over a retrained BGE baseline for global document retrieval and a 12.9% relative improvement over BGE-Reranker-L for local information retrieval. A sympathetic reader would take this as evidence that retrieval and fine-grained localization can be one learned model rather than a pipeline of separate systems.

What carries the argument

The central machinery is a fusion encoder that shares nearly all parameters with the query encoder but adds a lightweight cross-attention module at each layer, letting document tokens attend to query tokens. A causal text decoder with a dedicated [Decode] token consumes the fusion embeddings and is trained to generate the fine-grained unit (sentence or answer) that the query points to inside the document. At inference the cross-attention weights serve as a per-token relevance map, and sentences are ranked by aggregating those weights; the decoder and fusion encoder are not needed for global document ranking, so the bi-encoder alone keeps the retrieval cost unchanged.

What would settle it

A direct test would be to run GeAR's cross-attention sentence ranking on a held-out, human-annotated local-relevance set built from real search queries and sentence-level judgments; if its sentence-level Recall@1 is no better than a chunk-then-BGE baseline or BGE-Reranker-L on that set, the central claim of superior local retrieval would be refuted.

Watch

Extended reading notes

Core claim

GeAR's central claim is that optimizing a retriever to generate the query-relevant text inside a document, rather than only to score the document against the query, makes the retriever better at both tasks. The model is trained with a combined loss $\mathcal{L}_{\mathrm{GeAR}} = \mathcal{L}_{\mathrm{CL}} + \alpha \mathcal{L}_{\mathrm{LM}}$, where the contrastive loss aligns query and document embeddings and the language-modeling loss trains a causal decoder to reproduce the ground-truth sentence or answer given fused query-document representations. The same cross-attention weights that carry the fused representation are then used at inference to rank sentences for local retrieval. The paper reports consistent gains across SQuAD, NQ, TriviaQA, PAQ, and a synthetic RIR dataset, and shows via ablation that removing the language-modeling loss degrades both global and local retrieval.

Load-bearing premise

The load-bearing premise is that the synthetic RIR benchmark, built by having an LLM rewrite Wikipedia sentences into search queries and treating the original sentences as ground truth, faithfully represents real user information needs; if it does not, the reported local-retrieval gains may measure the model's ability to invert the data-generation pipeline rather than to locate relevant content in real documents.

Editorial extensions

If this is right

  • Web search and retrieval-augmented generation systems can show, alongside each retrieved document, the sentence or phrase that actually answers the query, without running a separate reranker or chunking pipeline.
  • Because global retrieval still uses only the bi-encoder, deployment does not require storing extra vectors or paying the delayed-interaction costs associated with ColBERT-style late interaction.
  • The generation output doubles as an interpretable explanation of why a document was retrieved, replacing a bare scalar similarity score with concrete, query-specific text.
  • The joint retrieval-and-generation objective can be trained end to end, so improvements to the fine-grained localization signal also feed back into the global document-ranking quality.
  • The same architecture could extend to long-document retrieval, where finding the needle inside the haystack is as important as retrieving the haystack itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims: the RIR benchmark is generated by the same LLM rewriting pipeline used to build training data, so the 12.9% local-retrieval gain should be re-measured on human-annotated query-sentence pairs before being taken as a statement about real search behavior.
  • Beyond the paper's claims: the finding that the best cross-attention layer is not the deepest one suggests that for other fusion-retrieval designs, probing intermediate layers may yield better localization than using the final layer.
  • Beyond the paper's claims: the same architecture could be tested with a decoder that generates a short rationale or a highlighted span instead of the full sentence, which would tighten the connection to fine-grained citation and grounded-generation tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GeAR (Generation Augmented Retrieval), a bi-encoder retriever augmented with a fusion encoder and a lightweight text decoder. Training combines a contrastive loss for global document retrieval with a language-modeling loss that conditions on fused query/document representations to generate query-relevant local information, such as answer sentences or relevant sentences. The authors construct a synthetic RIR dataset by using Llama-3.1-70B to rewrite Wikipedia sentences into queries, and evaluate both global document retrieval and local sentence-level retrieval on QAR and RIR scenarios. They report a 3.5% relative global Recall@5 improvement over a retrained BGE baseline and a 12.9% relative local-retrieval improvement over BGE-Reranker-L, and emphasize that global retrieval at inference uses only the bi-encoder, so it adds no computational cost over a standard bi-encoder.

Significance. If the claims are supported, GeAR is a useful step toward retrieval models that return not only relevant documents but also the specific sentences that justify the match, with the global retrieval stage costing no more than a bi-encoder. The paper has several strengths: it retrains strong baselines (SBERT_RT, BGE_RT) on aligned data, evaluates on multiple datasets, reports an ablation with and without the LM loss, measures inference cost, and provides qualitative attention analysis and layer-wise local-retrieval results. The local-retrieval gains from the LM loss are large and internally consistent. However, the paper's causal claim that the generation objective improves global retrieval is not supported by its own ablation table, and the RIR benchmark's synthetic construction raises validity concerns for the local-retrieval headline. Both issues are load-bearing and need to be addressed before the paper can be accepted.

major comments (3)
  1. [§4.3, Table 1, Table 4] Section 4.3 states that removing the LM loss reduces both global and local retrieval performance, but Table 1 does not support this for global retrieval. GeARw/oLLM outperforms GeAR on SQuAD (R@5 0.889 vs 0.887; M@5 0.776 vs 0.766) and PAQ (R@5 0.955 vs 0.952; M@5 0.877 vs 0.872), and the two are essentially tied on RIR R@5 (0.963 vs 0.964). Across the five datasets the average R@5 difference is about 0.001, and Table 4 shows only a 0.002 difference in global average Recall between α=0 and α=0.25, with α=0.5 matching α=0. No significance testing or variance across runs is reported. Consequently the causal claim that the generation objective improves global retrieval is unsupported: the global improvement over BGE_RT appears to come from the contrastive/momentum training recipe rather than from the generation module. The paper should either provide per-dataset significance tests and a justified effect size, or reframe the global contribution accordingly.
  2. [§3.2, Appendix A] The RIR benchmark is generated by sampling a Wikipedia sentence u, rewriting it into a query q with Llama-3.1-70B, and then treating the original sentence u as ground truth. GeAR's LM objective trains the decoder to reproduce exactly that u given q and the document. Local-retrieval scores on RIR therefore partly measure how well the model inverts the benchmark-construction pipeline, rather than how well it satisfies naturally occurring user information needs. The paper does not include any human-authored or otherwise externally sourced RIR test set, so the claimed 12.9% local-retrieval improvement over BGE-Reranker-L should be interpreted as in-distribution performance on synthetic data. I recommend adding an evaluation on naturally occurring queries (for example, MS MARCO passages with human judgments, or human rewrites of the same documents) and an analysis of whether the cross-attention localization transfers to that setting.
  3. [Abstract, §3.5, Appendix D] The abstract and Section 3.5 state that when used as a retriever, GeAR does not incur any additional computational cost over bi-encoders. This is accurate for global document retrieval, which uses only the bi-encoder, but the local-information-retrieval mode runs the fusion encoder and is measured at 1.28x on CPU and 1.65x on GPU relative to BGE (Table 9). The cost-free claim should be explicitly qualified to global retrieval, or the local-retrieval overhead should be disclosed in the abstract and Section 1, since the paper's framing otherwise overstates the efficiency benefit.
minor comments (5)
  1. [Table 2 and §4.2] For local-information retrieval, the reranker baselines are applied to each sentence independently, while GeAR attends to the full document through cross-attention. This gives GeAR additional document-level context and is a potential confound in the comparison; the paper should discuss why sentence-level scoring is the appropriate baseline or include a reranker variant that also conditions on document context.
  2. [§4.3, Table 4] The phrase 'optimal values at 0.25 and 0.5 respectively' should specify that 0.25 is the global-retrieval optimum and 0.5 is the local-retrieval optimum; the current wording is ambiguous.
  3. [Figure 4] The x-axis is labeled 'Number of Layers' but appears to be a layer index; please relabel it as 'Layer index' for clarity.
  4. [Appendix A] There is a typo in the paragraph describing the alternative data-construction attempt: 'we fine that other sentences...' should be 'we find that other sentences...'. Similar typos elsewhere include 'intutive' (Section 1), 'attnetion' (Figure 3 caption), and 'Visulization' (Appendix E).
  5. [Abstract and Section 5] The statement that code, data, and models will be released at the general LMOps repository should be replaced with a dedicated project link or repository path; a general organization URL is not actionable for reproducing the specific method.

Circularity Check

1 steps flagged · score 4.0 of 10

RIR local-retrieval benchmark is generated by the same sentence-to-query pipeline that GeAR is trained to invert, so part of the reported local-retrieval gain is in-distribution; global and QAR local results remain independent.

  1. self definitional [Sec. 3.2, 3.4, 4.1, 4.2; Appendix A]
    "we sampled sentences of appropriate length and whose subject is not a pronoun as u. Then we leverage LLM to rewrite these sentences as queries q. ... LM optimizes the cross-entropy loss over the entire vocabulary, maximizing the likelihood of the ground truth text. ... we leverage the synthesized 5.8M data, of which 95% is used for training and 5% is reserved for the test set."

    In the RIR scenario, the test query q is produced by LLM-rewriting the ground-truth sentence u, using the same pipeline that generated the training triples. GeAR's LM loss trains the decoder to output exactly that u given q and the document. Thus RIR local-retrieval performance measures the model's ability to invert the LLM rewrite function on held-out examples of the same synthetic process: the target u is, by construction, the variable from which q was generated, and the training objective is to predict that variable from q. The RIR local-retrieval 'prediction' therefore reduces to in-distribution inversion of the dataset-construction process rather than an independent test against real user queries.

full rationale

The main circularity concern is confined to the RIR benchmark. The paper generates a query by asking an LLM to rewrite a sampled Wikipedia sentence, treats that sentence as the ground-truth local information, and then trains GeAR with a language-modeling loss to generate exactly that sentence given the query and document. The held-out RIR test set is produced by the same pipeline, so high RIR local-retrieval scores partly reflect the model's learned inversion of the query-generation process. This is a real but partial circularity: the QAR local-retrieval evaluations on SQuAD, NQ, TriviaQA, and PAQ use externally constructed question-answer data and are not generated by this pipeline, and the global-retrieval evaluations on those datasets provide independent evidence. Section 4.3's statement that removing the LM loss reduces global retrieval is not supported by Table 1 (GeARw/oLLM is slightly better on SQuAD and PAQ global metrics), but that is an internal inconsistency or correctness issue, not a circularity of the derivation. Self-citations in the paper (e.g., Liu et al. 2024a) are not load-bearing. Overall, the central generation-augmented-retrieval claim does not collapse into its inputs; only the RIR portion of the local-retrieval evidence is substantially weakened by the self-definitional benchmark construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claims rest on standard contrastive learning machinery, plus several domain assumptions about the synthetic data and attention interpretation. No physical or mathematical constants are fitted; the main selected parameters are alpha, the BGE filtering threshold, and the layer used for local retrieval.

free parameters (5)
  • Language-modeling loss weight alpha = 0.25
    Set by the authors; Table 4 shows alpha=0.25 and 0.5 give the best average recall on the two retrieval tasks. This controls how much the generation objective contributes.
  • Local retrieval layer index = 10
    Footnote 4 and Figure 4: the 10th layer cross-attention weights are used for sentence ranking because peak performance occurs in the last 3 to 4 layers; this is a selection made on evaluation curves.
  • BGE relevance filtering threshold for synthetic RIR data = 0.5
    Appendix A: rewritten queries with BGE similarity below 0.5 to the source document are discarded; this threshold shapes the distribution of both training and test triples.
  • RIR candidate sentence length filter = 8 to 20 tokens
    Appendix A: only sentences with 8 to 20 tokens and a non-pronoun subject are selected for rewriting into queries, defining the granularity of 'local information' in the benchmark.
  • Momentum queue size and momentum coefficient = queue 57600, momentum 0.995
    Appendix C: standard MoCo and BLIP hyperparameters used for the momentum bi-encoder; manually chosen and potentially affecting contrastive learning quality.
assumptions (4)
  • domain assumption Cross-attention weights between query tokens and document tokens indicate sentence-level relevance
    Section 3.5 uses these weights to rank sentences for local retrieval without direct supervision on attention; the paper assumes attention strength equals relevance.
  • domain assumption A sentence rewritten into a query remains the unique relevant unit for that query
    Section 3.2 and Appendix A treat the original sentence as ground truth for RIR; if rewriting creates other relevant content or ambiguity, the labels are wrong.
  • domain assumption PAQ training data does not overlap with the SQuAD, NQ, TriviaQA evaluation sets
    Section 4.1 calls these held-out, but no decontamination is documented; PAQ is built from Wikipedia, as are the evaluation sets.
  • domain assumption Momentum contrastive learning provides a valid soft-label training signal for retrieval
    Section 3.4 adopts MoCo and BLIP momentum encoders; standard in the field but an unproved background design assumption.
invented entities (1)
  • [Decode] special token
    purpose: Marks the start of the sequence for the text decoder, separating the encoding phase from autoregressive generation
    A new vocabulary token introduced by the architecture; it has no testable prediction outside the model's own benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GeAR: Generation Augmented Retrieval." pith.science (2026). https://pith.science/paper/SYI2U3FU

@misc{pith2026250102772,
  author       = {Pith},
  title        = {Pith review of: GeAR: Generation Augmented Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SYI2U3FU}},
  note         = {Machine review of arXiv:2501.02772}
}
abstract

Document retrieval techniques are essential for developing large-scale information systems. The common approach involves using a bi-encoder to compute the semantic similarity between a query and documents. However, the scalar similarity often fail to reflect enough information, hindering the interpretation of retrieval results. In addition, this process primarily focuses on global semantics, overlooking the finer-grained semantic relationships between the query and the document's content. In this paper, we introduce a novel method, $\textbf{Ge}$neration $\textbf{A}$ugmented $\textbf{R}$etrieval ($\textbf{GeAR}$), which not only improves the global document-query similarity through contrastive learning, but also integrates well-designed fusion and decoding modules. This enables GeAR to generate relevant context within the documents based on a given query, facilitating learning to retrieve local fine-grained information. Furthermore, when used as a retriever, GeAR does not incur any additional computational cost over bi-encoders. GeAR exhibits competitive retrieval performance across diverse scenarios and tasks. Moreover, qualitative analysis and the results generated by GeAR provide novel insights into the interpretation of retrieval results. The code, data, and models will be released at \href{https://github.com/microsoft/LMOps}{https://github.com/microsoft/LMOps}.

Figures

Figures reproduced from arXiv: 2501.02772 by the authors.

Figure 1
Figure 1. Comparison of functionality between classical retriever and GeAR. GeAR is designed to handle both global [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. GeAR. It consists of a bi-encoder, a fusion encoder, and a text decoder. It contains two training objectives, CL represents contrastive learning loss, which aims to optimize the similarity between documents and queries. LM represents the language modeling loss for generating relevant information given documents and queries. and enable the model to learn to focus on fine￾grained query-related information in documents… view at source ↗
Figure 3
Figure 3. Visualization of local information retrieval of GeAR . In the two scenarios, we pose two different queries [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Local information retrieval performance of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: More Visulization results [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 6 canonical work pages

  1. [1]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. https://www.microsoft.com/en-us/research/publication/make-your-llm-fully-utilize-the-context/ Make your llm fully utilize the context . In NeurIPS 2024

    work page 2024

  2. [2]

    Manoj Ghuhan Arivazhagan, Lan Liu, Peng Qi, Xinchi Chen, William Yang Wang, and Zhiheng Huang. 2023. https://doi.org/10.18653/v1/2023.findings-acl.679 Hybrid hierarchical retrieval for open-domain question answering . In Findings of the Association for Computational Linguistics: ACL 2023, pages 10680--10689, Toronto, Canada. Association for Computational ...

    doi:10.18653/v1/2023.findings-acl.679 2023

  3. [3]

    Eduardo Brito and Henri Iser. 2023. https://doi.org/10.1145/3539618.3592017 Maxsime: Explaining transformer-based semantic similarity via contextualized best matching token pairs . In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '23, page 2154–2158, New York, NY, USA. Association fo...

    arXiv 2023

  4. [4]

    Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. https://doi.org/10.18653/v1/P17-1171 Reading W ikipedia to answer open-domain questions . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870--1879, Vancouver, Canada. Association for Computational Linguistics

    doi:10.18653/v1/p17-1171 2017

  5. [5]

    Hong-You Chen, Zhengfeng Lai, Haotian Zhang, Xinze Wang, Marcin Eichner, Keen You, Meng Cao, Bowen Zhang, Yinfei Yang, and Zhe Gan. 2024. Contrastive localized language-image pre-training. arXiv preprint arXiv:2410.02746

    arXiv 2024

  6. [6]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  7. [7]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. http://arxiv.org/abs/2401.08281 The faiss library

    arXiv 2024

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 54 references
  1. [9]

    https://dumps.wikimedia.org Wikimedia downloads

    Wikimedia Foundation. https://dumps.wikimedia.org Wikimedia downloads

  2. [10]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.552 S im CSE : Simple contrastive learning of sentence embeddings . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894--6910, Online ...

    2021 doi

  3. [11]

    Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.398 Enabling large language models to generate text with citations . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465--648...

    2023 doi

  4. [12]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. http://arxiv.org/abs/2312.10997 Retrieval-augmented generation for large language models: A survey

    2024 arXiv

  5. [13]

    Michael G \"u nther, Isabelle Mohr, Daniel James Williams, Bo Wang, and Han Xiao. 2024. Late chunking: contextual chunk embeddings using long-context embedding models. arXiv preprint arXiv:2409.04701

    2024 arXiv

  6. [14]

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    2020

  7. [15]

    Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2333--2338

    2013

  8. [16]

    Ganesh Jawahar, Beno \^ t Sagot, and Djam \'e Seddah. 2019. https://doi.org/10.18653/v1/P19-1356 What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651--3657, Florence, Italy....

    2019 doi

  9. [17]

    Jeff Johnson, Matthijs Douze, and Herv \'e J \'e gou. 2019. Billion-scale similarity search with GPUs . IEEE Transactions on Big Data, 7(3):535--547

    2019

  10. [18]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  11. [19]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empiric...

    2020 doi

  12. [20]

    Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. https://doi.org/10.1162/tacl_a_00405 Relevance-guided supervision for O pen QA with C ol BERT . Transactions of the Association for Computational Linguistics, 9:929--944

    2021 doi

  13. [21]

    Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39--48

    2020

  14. [22]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  15. [23]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating S...

    2023

  16. [24]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K\" u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt\" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/6b493230205f78...

    2020

  17. [25]

    Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich K \"u ttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. https://doi.org/10.1162/tacl_a_00415 PAQ : 65 million probably-asked questions and what you can do with them . Transactions of the...

    2021 doi

  18. [26]

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888--12900. PMLR

    2022

  19. [27]

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694--9705

    2021

  20. [28]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281

    2023 arXiv

  21. [29]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  22. [30]

    Haoyu Liu, Jianfeng Liu, Shaohan Huang, Yuefeng Zhan, Hao Sun, Weiwei Deng, Furu Wei, and Qi Zhang. 2024 a . https://aclanthology.org/2024.findings-acl.312 se^2 : Sequential example selection for in-context learning . In Findings of the Association for Computational Linguistic...

    2024

  23. [31]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 b . https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics,...

    2024 doi

  24. [32]

    Ye Liu, Kazuma Hashimoto, Yingbo Zhou, Semih Yavuz, Caiming Xiong, and Philip Yu. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.19 Dense hierarchical retrieval for open-domain question answering . In Findings of the Association for Computational Linguistics: EMNLP 2021...

    2021 doi

  25. [33]

    I Loshchilov. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

    2017 arXiv

  26. [34]

    Kun Luo, Zheng Liu, Shitao Xiao, and Kang Liu. 2024. Bge landmark embedding: A chunking-free embedding method for retrieval augmented long-context large language models. arXiv preprint arXiv:2402.11573

    2024 arXiv

  27. [35]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26

    2013

  28. [36]

    Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. 2024. http://arxiv.org/abs/2402.09906 Generative representational instruction tuning

    2024 arXiv

  29. [37]

    Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543

    2014

  30. [38]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

    2021

  31. [39]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  32. [40]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  33. [41]

    Ozan Sener and Vladlen Koltun. 2018. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31

    2018

  34. [42]

    M Seo. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603

    2016 arXiv

  35. [43]

    Oscar Skean, Md Rifat Arefin, Yann LeCun, and Ravid Shwartz-Ziv. 2024. Does representation matter? exploring intermediate layers in large language models. arXiv preprint arXiv:2412.09563

    2024 arXiv

  36. [44]

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533

    2022 arXiv

  37. [45]

    Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024. https://aclanthology.org/2024.emnlp-main.322 Leave no document behind: Benchmarking long-context LLM s w...

    2024

  38. [46]

    S Wang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905

    2016 arXiv

  39. [47]

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. https://doi.org/10.1145/3626772.3657878 C-pack: Packed resources for general chinese embeddings . In Proceedings of the 47th International ACM SIGIR Conference on Research and Develop...

    2024

  40. [48]

    Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. https://www.microsoft.com/en-us/research/publication/approximate-nearest-neighbor-negative-contrastive-learning-for-dense-text-retrieval/ Approximate nearest nei...

    2021

  41. [49]

    Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. https://doi.org/10.18653/v1/N19-1242 BERT post-training for review reading comprehension and aspect-based sentiment analysis . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computation...

    2019 doi

  42. [50]

    Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. https://doi.org/10.18653/v1/N16-1174 Hierarchical attention networks for document classification . In Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Co...

    2016 doi

  43. [51]

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, et al. 2024. Longcite: Enabling llms to generate fine-grained citations in long-context qa. arXiv preprint arXiv:2409.02897

    2024 arXiv

  44. [52]

    Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.47 L ong E mbed: Extending embedding models for long context retrieval . In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan...

    2024 doi

  45. [53]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.