Pith. sign in

REVIEW 4 major objections 4 minor 43 references

Dehallucinating Parallel Context Extension for Retrieval-Augmented Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DePaC cuts fact fabrication and fact omission in RAG with refusal training and information-calibrated aggregation.

desk verdict Sign error in the main equations inverts DePaC's rejection-token filter; the empirical story is strong but the method as written cannot work. read the letter →

arxiv 2412.14905 v1 pith:JOMXZ6TC submitted 2024-12-19 cs.CL cs.AI

Zexiong Ma , Shengnan An , Zeqi Lin , Yanzhen Zou , Jian-Guang Lou , Bing Xie This is my paper · ORCID
classification cs.CLcs.AI
keywords retrieval-augmentedgenerationhallucinationparallelcontextextensionnegativetraininginformation-calibratedaggregationfactfabricationomissionlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation still lets large language models hallucinate, and parallel context extension (PCE) makes this worse in two specific ways: a model can fabricate a fact that no window supports, or an uninformative window can dominate the aggregation and cause the model to omit a fact another window supports. DePaC targets both. It fine-tunes the backbone with a context-aware negative training loss: when a document is unrelated to the question, the model is trained to output a rejection token (UNK) instead of an answer; when the document is relevant, it is trained on the ground-truth answer. It then aggregates windows with an information-calibrated rule that prefers a window with both high output certainty and high KL divergence from the no-document distribution, making it robust to both fabrication and omission. On six information-seeking tasks and three document-question-answering tasks, the paper reports that DePaC outperforms vanilla inference, the YaRN context-extension method, average-aggregation PCE, and NBCE, and that its inference cost grows linearly with the number of documents rather than quadratically. The paper claims both components are necessary, since ablations removing either one lower accuracy.

What carries the argument

Two mechanisms carry DePaC. First, context-aware negative training (NegTrain) builds a dataset in which some question-document pairs are positive (the document is the oracle that supports the answer) and others are negative (the document is an unrelated distractor), and the fine-tuning loss is cross-entropy against the gold answer for positives and against a rejection token $t_d=$UNK for negatives; this gives the model a built-in abstain signal. Second, information-calibrated aggregation (ICA) scores every parallel window at each generation step by combining the model's certainty in its top token with the KL divergence from the context-free distribution $p_{i,c}$, and the selected window is the one that is both confident and informative while windows that have learned to emit the rejection token are skipped. The complexity analysis rests on the observation that DePaC processes all documents in parallel with effective sequence length $n$ per window, so attention cost is $O(k n^2 d m)$ rather than vanilla's $O(k^2 n^2 d m)$.

What would settle it

Build a test set in which every retrieved document is topically related to the question but none supports an answer, and compare how often DePaC produces a confident fabricated answer versus emitting its refusal token; if the refusal rate is no higher than vanilla's, the negative-training component has not generalized to partially relevant distractors and the paper's hallucination claim would not stand for realistic retrieval noise.

Watch

Extended reading notes

Core claim

DePaC's central claim is that the two common failure modes of PCE for RAG are separable and fixable. Fact fabrication is treated as a training problem: DePaC fine-tunes the model so that an irrelevant context forces the first output token to be a rejection token $t_d$ (set to UNK), while relevant contexts continue with the gold answer; this teaches the model to abstain rather than invent. Fact omission is treated as an aggregation problem: rather than trusting the window with lowest entropy or the majority vote, DePaC scores each window by the certainty of its most likely token plus $\beta$ times the KL divergence between the window's distribution and the context-free distribution $\Delta(p_{i,j}, p_{i,c}) = D_{\mathrm{KL}}(p_{i,j}\,\|\,p_{i,c})$, and it discards windows whose top token is the rejection token. The paper argues this selection rule lets the model switch windows token by token, so it can assemble answers from multiple documents, and the reported results support that DePaC reduces the measured hallucination percentages on FuncNR, EntLR, and MVIH, in one case to zero, while improving F1 and accuracy across the nine tasks.

Load-bearing premise

The method depends on a clean binary split between fully relevant and fully irrelevant documents; real retrievers often return partially relevant or hard-negative documents, and the learned refuse-to-answer signal was never trained to handle that middle ground.

Editorial extensions

If this is right

  • RAG pipelines can run on models with modest context limits: instead of truncating a concatenated mega-document, the system can score many candidate documents independently and carry the best window forward at every generated token.
  • Fine-tuning with a rejection token gives a calibrated 'not answerable from context' behavior, so downstream applications can route unanswered queries to a retriever refresh or a human rather than trusting a fabricated response.
  • Because ICA explicitly favors windows whose distribution shifts most relative to the context-free prior, it is naturally suited to multi-hop questions that need evidence from different documents assembled token by token.
  • The linear-in-documents inference cost makes PCE-style RAG practical for large candidate sets, where vanilla concatenation becomes quadratic.
  • Ablations imply that negative training is not a regularization aside: removing it, or reverting aggregation to lowest-uncertainty, measurably reduces accuracy, so both components should be present in a deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's binary relevant/irrelevant training suggests an untested boundary: real retrievers produce partially relevant documents, and one straightforward extension would be to train a graded refusal output, so the rejection token's strength correlates with answerability rather than being a hard switch.
  • Because ICA only uses the top-token probability plus a scalar KL divergence, it may underweight rare but correct tokens; a natural follow-up would be to add a query-specific beam over the union of top-$k$ tokens across windows, which could further cut fact omission.
  • The rejection-token idea could transfer to general abstention in QA: the same UNK-based fine-tuning could give LLMs an explicit 'I do not know' behavior when they are prompted with context that is relevant-looking but uninformative.
  • DePaC's evaluation measures hallucination on exact-match tasks with clear answers; extending the same two-type taxonomy to open-ended generation would require a stricter entailment-based detector, which the paper leaves for future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes DePaC, a method for retrieval-augmented generation with parallel context extension, combining two components: context-aware negative training (NegTrain) that fine-tunes a backbone LLM to emit a rejection token for irrelevant documents, and information-calibrated aggregation (ICA) that selects context windows based on a combination of output certainty and KL divergence from the non-document distribution. The method is evaluated on six information-seeking tasks and three document QA tasks, with claims that it reduces fact fabrication and fact omission hallucinations relative to vanilla inference, YaRN, AVP, and NBCE baselines. The paper also contributes a complexity analysis showing linear rather than quadratic scaling with the number of documents.

Significance. If the proposed mechanism were correctly specified and the empirical results held up under fair comparison, DePaC would be a useful contribution to RAG and parallel context extension, especially because it combines negative supervision with an information-theoretic selection criterion and demonstrates gains on nine tasks with two backbone models. The complexity analysis is a strength. However, the central aggregation equation contains an inverted rejection-token filter that contradicts the stated design, and the experimental protocol compares a fine-tuned model against non-fine-tuned baselines, so the significance of the reported improvements cannot currently be assessed.

major comments (4)
  1. [Section 3, Eq. (11) and Appendix B, Eq. (15)] The rejection-token filter in the aggregation objective is inverted. In Eq. (11), the minimization objective subtracts γ·I(argmax_k p_{i,jk}=td) with γ>0, so a window whose top token is the rejection token receives a smaller objective value and is therefore more likely to be selected. In the simplified form of Appendix B, Eq. (15) maximizes \hat{C}(p_{i,j},p_{i,c})·I(argmax_k p_{i,jk}=td); the indicator is nonzero only for rejection-token windows, forcing the argmax onto exactly the windows the text says should be excluded. This directly contradicts the sentence in Section 3 that selected windows 'should not be aligned to the rejection token.' If implemented literally, DePaC would aggregate 'UNK' contexts, which cannot produce the accuracy gains reported in Tables 1 and 3. The condition should be I(argmax_k p_{i,jk} ≠ td) (or equivalently a +γ penalty in the argmin form), and the corresponding simplification must be corrected. As printed, the central mechanism is internally inconsistent.
  2. [Section 3 (Implementation Details) and Section 5 (Tables 1-3)] The experimental comparison is unfair because DePaC fine-tunes the backbone on 19K constructed examples, while all baselines (Vanilla, YaRN, AVP, NBCE) are used without any fine-tuning. The reported gains could therefore be attributable to the additional training rather than to NegTrain and ICA. To support the claim that DePaC's aggregation and negative training are responsible for the improvements, the paper needs a control baseline fine-tuned on a comparable positive-only or instruction-following dataset, or an evaluation where all methods use the same fine-tuned backbone. This applies to Tables 1, 2, 3, and 5.
  3. [Appendix G (Hallucination Definition and Evaluation Setup)] The hallucination metric is defined post hoc and is not a principled measure. Incorrect outputs containing any of 27 manually selected omission phrases are counted as fact omission, and all other errors are counted as fact fabrication. This rule can over-attribute errors to omission (e.g., a fabricated answer that happens to include 'not mentioned') and under-attribute errors that express omission without a trigger phrase. There is no inter-annotator agreement or human validation. Consequently, the strong claims in Figure 5, such as 'DePaC completely avoids fact omission on EntLR and fact fabrication on MVIH,' rest on an unvalidated automatic classifier. Please provide a human-annotated sample or a more rigorous evaluation protocol.
  4. [Section 3 (Context-aware Negative Training) and Figure 2] NegTrain assumes a binary relevance split in the training data: a document is either fully relevant (paired with the gold answer) or fully irrelevant (paired with the rejection token). Real RAG retrievers often return partially relevant documents or hard negatives that contain a mixture of useful and misleading information. The experiments in Tables 4 and 5 include multi-hop QA, where the required information is spread across documents, but the training data construction described in the Implementation Details samples only 'unrelated fragment' as distractors. The paper does not test whether the rejection token reliably transfers to partially overlapping or hard-negative documents, which is central to the claim that DePaC works in RAG scenarios.
minor comments (4)
  1. [Throughout] There are numerous typos and inconsistent notations, e.g., 'independ' in the Introduction, 'conducte' in Section 5.2, 'implicate' in Appendix B, 'A VP' instead of 'AVP' in Tables 1 and 2, and 'MVH' instead of 'MVIH' in Figure 6. A careful proofreading is needed.
  2. [Tables 1-5] All results are reported as single point estimates without error bars, multiple seeds, or significance tests. For a paper that makes claims of 'significant alleviation' and 'consistent' improvements, the absence of variance estimates is a substantial presentation gap.
  3. [Section 4 (Complexity Analysis)] The complexity comparison assumes all documents have the same length n, but in practice documents vary; the O(k·n²·d·m) bound should be stated with this assumption made explicit, and the 'linear in k' claim is only linear for fixed n.
  4. [Appendix D (Figure 8)] The training-loss curves in Figure 8 are described as showing that NegTrain increases the probability difference between refusing unrelated and related documents, but the figure only displays loss values and no quantitative comparison is given; please add a numerical metric or a clearer analysis.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: DePaC's derivation is self-contained; the self-citations are non-load-bearing, and the Eq. 15 indicator issue is a correctness concern, not a circularity.

full rationale

DePaC's core components are independently specified: NegTrain is a supervised fine-tuning loss (Eq. 8) on synthetically constructed C4 data, and ICA is an aggregation rule (Eqs. 9-12 and 15-16) defined in terms of the model's own output distributions. Evaluation is against external benchmarks (FuncNR, EntLR, MVIH, APIBench, Qasper, MultifieldQA, NarrativeQA, etc.), and no evaluated quantity is fitted and then reported as a prediction. The only tuned hyperparameter, beta = 0.2, is selected on EntLR and fixed for other tasks, which is ordinary hyperparameter selection rather than result-fitting. The self-citations to An et al. (2024) for the C4-based data construction recipe and for the FuncNR/EntLR tasks are methodological and benchmark citations, not load-bearing derivational premises, so they do not make the central claim circular. One real issue noted in review is that Eq. 15 as printed writes the indicator as I(argmax_k p_{i,jk} = td), which selects the rejection-token windows that the text says should be excluded; this is an internal sign/indicator inconsistency and a correctness risk, but it is not circularity, because the formula is not equivalent to its input by construction.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method relies on one fitted hyperparameter (beta) plus two hand-chosen constants (alpha, the UNK rejection token). The key domain assumptions are that KL divergence between with-document and no-document distributions captures answer-relevant information, and that binary synthetic negative training transfers to real RAG scenarios. No new physical or symbolic entities are introduced.

free parameters (3)
  • beta (ICA weight) = 0.2
    Tuned via ablation on EntLR (Appendix C); controls the trade-off between output certainty and KL information increment.
  • alpha (calibration weight) = 0.2
    Borrowed from NBCE (Su et al., 2024), used in the final calibrated distribution in Equation 3.
  • Rejection token td = UNK
    Chosen by hand to minimize interference with normal tokens during training; used as the refusal signal in Equations 11 and 15.
assumptions (5)
  • domain assumption Aggregation on output token distributions is a valid representation of parallel context extension (Su et al., 2024).
    Section 2 adopts the distribution-level formalization without independent validation of its equivalence to hidden-state aggregation.
  • domain assumption KL divergence between the with-document and no-document output distributions measures the useful information increment of a document.
    Section 3, Equation 9; this is the core heuristic behind ICA and is not derived from an information-theoretic guarantee of answer usefulness.
  • domain assumption The maximum output probability max_k p reflects model certainty.
    Appendix B, Equation 16, citing Ghoshal and Tucker (2022); used to estimate certainty in the simplified form.
  • domain assumption Binary relevance training on C4 fragments with GPT-4-generated QA transfers to the nine evaluation tasks.
    Section 3, Implementation Details; no robustness analysis for distribution shift from synthetic distractors to real RAG retrievers.
  • domain assumption GPT-4-Turbo generated questions and answers are sufficiently accurate to serve as supervision.
    Section 3, Implementation Details; no quality audit of the generated training data is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dehallucinating Parallel Context Extension for Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/JOMXZ6TC

@misc{pith2026241214905,
  author       = {Pith},
  title        = {Pith review of: Dehallucinating Parallel Context Extension for Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JOMXZ6TC}},
  note         = {Machine review of arXiv:2412.14905}
}
read the original abstract

Large language models (LLMs) are susceptible to generating hallucinated information, despite the integration of retrieval-augmented generation (RAG). Parallel context extension (PCE) is a line of research attempting to effectively integrating parallel (unordered) contexts, while it still suffers from hallucinations when adapted to RAG scenarios. In this paper, we propose DePaC (Dehallucinating Parallel Context Extension), which alleviates the hallucination problem with context-aware negative training and information-calibrated aggregation. DePaC is designed to alleviate two types of in-context hallucination: fact fabrication (i.e., LLMs present claims that are not supported by the contexts) and fact omission (i.e., LLMs fail to present claims that can be supported by the contexts). Specifically, (1) for fact fabrication, we apply the context-aware negative training that fine-tunes the LLMs with negative supervisions, thus explicitly guiding the LLMs to refuse to answer when contexts are not related to questions; (2) for fact omission, we propose the information-calibrated aggregation which prioritizes context windows with higher information increment from their contexts. The experimental results on nine RAG tasks demonstrate that DePaC significantly alleviates the two types of hallucination and consistently achieves better performances on these tasks.

Figures

Figures reproduced from arXiv: 2412.14905 by the authors.

Figure 1
Figure 1. DePaC significantly reduces the occurrence [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Existing PCE approaches face two types of in-context hallucination issues when applied to RAG: (1) Fact [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. DePaC consists of two key components: (1) a context-aware negative training technique to alleviate fact [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Attention pattern and execution time comparison between DePaC and vanilla inference. The execution [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Hallucination percentage in responses for the information seeking tasks. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance of DePaC without NegTrain or ICA. w/o NegTrain refers to DePaC with positive training, [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: DePaC performance with different beta 100 200 300 400 500 600 steps 6 8 10 12 14 loss PosEval NegEval [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Rejection token prediction loss on PosEval [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: DePaC can switch context window for multi-hop questions. [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Fact omission phrases. G Hallucination Definition and Evaluation Setup Previous work (Weng, 2024) categorizes halluci￾nation into two types: (1) extrinsic hallucination, where the output of LLM is not grounded by the pre-training dataset or external world knowledge. (…
Figure 11
Figure 11. Figure 11: DePaC performance at different degrees of [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 7 canonical work pages

  1. [1]

    Shengnan An, Zexiong Ma, Zeqi Lin, Nanning Zheng, and Jian-Guang Lou. 2024. https://arxiv.org/abs/2404.16811 Make your llm fully utilize the context . Preprint, arXiv:2404.16811

    arXiv 2024

  2. [2]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511

    arXiv 2023

  3. [3]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. 2023. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508

    arXiv 2023

  4. [4]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754--17762

    2024

  5. [5]

    Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2024. Lift yourself up: Retrieval-augmented text generation with self-memory. Advances in Neural Information Processing Systems, 36

    work page 2024

  6. [6]

    Tri Dao. 2023. Flash A ttention-2: Faster attention with better parallelism and work partitioning

    2023

  7. [7]

    Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4599--4610

    2021

  8. [8]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. arXiv preprint arXiv:2309.11495

    arXiv 2023

Show all 43 references
  1. [9]

    Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. 2024. Don't hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration. arXiv preprint arXiv:2402.00367

    2024 arXiv

  2. [10]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  3. [11]

    Biraja Ghoshal and Allan Tucker. 2022. On calibrated model uncertainty in deep learning. arXiv preprint arXiv:2206.07795

    2022 arXiv

  4. [12]

    Yaru Hao, Yutao Sun, Li Dong, Zhixiong Han, Yuxian Gu, and Furu Wei. 2022. Structured prompting: Scaling in-context learning to 1,000 examples. arXiv preprint arXiv:2212.06713

    2022 arXiv

  5. [13]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. https://arxiv.org/abs/2404.06654 Ruler: What's the real context size of your long-context language models? Preprint, arXiv:2404.06654

    2024 arXiv

  6. [14]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1--38

    2023

  7. [15]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023 a . Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  8. [16]

    Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023 b . Active retrieval augmented generation. arXiv preprint arXiv:2305.06983

    2023 arXiv

  9. [17]

    Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551

    2017 arXiv

  10. [18]

    Tom \'a s Ko c isk \`y , Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, G \'a bor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317--328

    2018

  11. [19]

    Kullback and R

    S. Kullback and R. A. Leibler. 1951. https://doi.org/10.1214/aoms/1177729694 On Information and Sufficiency . The Annals of Mathematical Statistics, 22(1):79 -- 86

    1951

  12. [20]

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for C...

    2019

  13. [21]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Proc...

    2020

  14. [22]

    Raymond Li, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, LI Jia, Jenny Chim, Qian Liu, et al. 2023. Starcoder: may the source be with you! Transactions on Machine Learning Research

    2023

  15. [23]

    Xingxuan Li, Xuan-Phi Nguyen, Shafiq Joty, and Lidong Bing. 2024. Paraicl: Towards robust parallel in-context learning. arXiv preprint arXiv:2404.00570

    2024 arXiv

  16. [24]

    Zexiong Ma, Shengnan An, Bing Xie, and Zeqi Lin. 2024. Compositional api recommendation for library-oriented code generation. arXiv preprint arXiv:2402.19431

    2024 arXiv

  17. [25]

    OpenAI. 2023. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2023 arXiv

  18. [26]

    Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. 2023. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334

    2023 arXiv

  19. [27]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations

    2023

  20. [28]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  21. [29]

    Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. Parallel context windows for large language models. In Proceedings of the 61st Annual Meeting of the Association for Computationa...

    2023

  22. [30]

    Vipula Rawte, Amit Sheth, and Amitava Das. 2023. A survey of hallucination in large foundation models. arXiv preprint arXiv:2309.05922

    2023 arXiv

  23. [31]

    Zhihong Shao, Yeyun Gong, Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. arXiv preprint arXiv:2305.15294

    2023 arXiv

  24. [32]

    Kaize Shi, Xueyao Sun, Qing Li, and Guandong Xu. 2024. Compressing long context for enhancing rag with amr-based concept distillation. arXiv preprint arXiv:2405.03085

    2024 arXiv

  25. [33]

    Jianlin Su, Murtadha Ahmed, Luo Ao, Mingren Zhu, Yunfeng Liu, et al. 2024. Naive bayes-based context extension for large language models. arXiv preprint arXiv:2403.17552

    2024 arXiv

  26. [34]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  27. [35]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368

    2023 arXiv

  28. [36]

    Lilian Weng. 2024. https://lilianweng.github.io/posts/2024-07-07-hallucination/ Extrinsic hallucinations in llms

    2024

  29. [37]

    Kejuan Yang, Xiao Liu, Kaiwen Men, Aohan Zeng, Yuxiao Dong, and Jie Tang. 2023. Revisiting parallel context windows: A frustratingly simple alternative and chain-of-thought deterioration. arXiv preprint arXiv:2305.15262

    2023 arXiv

  30. [38]

    Howard Yen, Tianyu Gao, and Danqi Chen. 2024. Long-context language modeling with parallel context encoding. arXiv preprint arXiv:2402.16617

    2024 arXiv

  31. [39]

    Huimin Zeng, Zhenrui Yue, Qian Jiang, and Dong Wang. 2024. Federated recommendation via hybrid retrieval augmented generation. arXiv preprint arXiv:2403.04256

    2024 arXiv

  32. [40]

    Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. 2024. Raft: Adapting language model to domain specific rag. arXiv preprint arXiv:2403.10131

    2024 arXiv

  33. [41]

    Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. Docprompting: Generating code by retrieving the docs. arXiv preprint arXiv:2207.05987

    2022 arXiv

  34. [42]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  35. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.