Pith. sign in

REVIEW 5 major objections 6 minor 39 references

CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CrEst estimates the credibility of each retrieved context document from inter-document agreement alone, without annotations or model reasoning, and uses the scores to steer LLM generation.

desk verdict Useful repurposing of weak supervision for RAG credibility, but the empirical claims outrun the evidence and the high-noise experiment doesn't actually test the stated assumption. read the letter →

arxiv 2506.14912 v1 pith:EWFYBZI5 submitted 2025-06-17 cs.CL cs.LG

Dyah Adila , Shuai Zhang , Boran Han , Bonan Min , Yuyang Wang This is my paper
classification cs.CLcs.LG
keywords credibilityestimationweaksupervisionretrieval-augmentedgenerationcontextdenoisingattentionmaskinginter-documentagreementLLMinferencequestionanswering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CrEst claims that the credibility of a retrieved context document can be estimated without any manual annotation, fine-tuning, or reliance on the LLM's own reasoning: a credible document is simply one whose embedding lies close to most other documents in the retrieved set. The paper develops this into a weak-supervision score, proves it recoverable from observable pairwise distances via a triplet identity, and injects the scores into generation by prompting (black-box) or attention scaling (white-box). Across three model families and five question-answering datasets, CrEst reports consistent gains over vanilla RAG and two denoising baselines, up to 26.86% accuracy and 3.49% F1, with the advantage persisting when up to 80% of retrieved documents are corrupted. If correct, this means trust in a document can be read off from agreement structure alone, independent of the model's capacity to judge its own context.

What carries the argument

The load-bearing machinery is the weak-supervision triplet model: a probabilistic graphical model over document embeddings, $Pr(\lambda_1,\dots,\lambda_n|\lambda^*) = \frac{1}{Z}\exp(-\sum_i \theta_i\|\lambda_i-\lambda^*\|^2)$, in which each retrieved document is a noisy observation of one unobserved true document. The pairwise-distance identity that follows from this model—expected distance between two documents equals the sum of their expected distances to the true document—lets CrEst solve for each document's distance to $\lambda^*$ from observable pairwise distances alone. That turns the intuition 'credible documents agree with other credible documents' into a closed-form score, which is then denoised by ensembling multiple embedding functions and consumed either as prompt text or as attention-mask scaling.

What would settle it

Construct a retrieval set for a single query where over half of the retrieved documents share a plausible-sounding but wrong answer and are mutually similar in embedding space; if CrEst assigns those documents the highest credibility scores and the attention- or prompt-integrated model's accuracy falls below vanilla RAG, the majority-agreement premise is falsified. A cleaner quantitative version: across the five datasets, plot CrEst's accuracy against the fraction of queries where the majority of retrieved documents do not contain the golden answer; the claim predicts a sharp accuracy collapse as that fraction crosses 50%, and a system that maintains gains in that regime would contradict the paper's stated limitation.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that document-level credibility in retrieval-augmented generation is recoverable from the geometry of the retrieved set itself. CrEst defines each document's credibility $s_i$ as the reciprocal of the expected squared distance between its embedding $\lambda_i$ and an unobserved true document embedding $\lambda^*$, then shows through a probabilistic graphical model that this expected distance can be computed using only pairwise embedding distances: from $E[\|\lambda_i-\lambda_j\|^2] = E[\|\lambda_i-\lambda^*\|^2]+E[\|\lambda_j-\lambda^*\|^2]$, a triple of documents yields $E[\|\lambda_i-\lambda^*\|^2] = \frac{1}{2}(\delta_{ij}+\delta_{ik}-\delta_{jk})$. The resulting scores are ensembled over several embedding functions and fed into the LLM either as credibility labels in a multiple-prompting black-box scheme or as multipliers on attention masks in a white-box scheme. The paper validates the approach on HotpotQA, PopQA, NaturalQuestions, TriviaQA, and ASQA with Mistral, Mistral-Nemo, and Gemma, and reports that the black-box variant outperforms InstructRAG and CLeHe, while the white-box variant composes with InstructRAG for the strongest results.

Load-bearing premise

CrEst assumes an unobserved 'true' document exists whose embedding is close to most retrieved documents, so a document's credibility is its semantic closeness to the majority of the retrieved set; when more than half of the retrieved documents are wrong, the scores invert and CrEst amplifies the unreliable majority.

Editorial extensions

If this is right

  • CrEst transfers across retrievers and LLMs with no retraining or human labels, because it operates entirely on post-retrieval embeddings.
  • The black-box variant supports much larger document sets than per-document LLM methods, since it never calls the model once per passage.
  • The white-box variant is additive with prompting-based denoising: CrEst–wbx + InstructRAG gives the best accuracy in the main results table.
  • Using more embedding functions improves scores up to roughly seven embedders, and more retrieved documents help up to roughly 12–18, after which performance slowly declines.
  • Under synthetic noise that corrupts up to 80% of retrieved documents, CrEst keeps its advantage, most clearly on the smaller Gemma model where reasoning-dependent baselines degrade.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If credibility is recoverable from agreement geometry, the same score could serve as a retriever training signal; fine-tuning a retriever to rank by CrEst score on held-out queries is a direct test the paper names as future work.
  • The single-centroid assumption may be the binding constraint for multi-hop queries such as HotpotQA; a mixture-of-centroids extension could separate several credible answer strands rather than collapsing them into one.
  • The score distribution could double as a confidence gate: when CrEst's golden-vs-distractor separation shrinks, downstream accuracy should drop, so the score itself could decide whether to use the context at all.
  • Because the method is model-agnostic and annotation-free, it should transfer to any setting where several candidate passages are retrieved for one claim, such as fact-checking or citation selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces CrEst, an annotation-free method that estimates the credibility of retrieved context documents from pairwise embedding distances, under the assumption that credible documents form a semantic majority around an unobserved true document. It derives credibility scores via a triplet-based weak-supervision estimator and integrates them into LLM inference either by prompting multiple times (black-box) or by scaling attention weights (white-box). Experiments cover three LLMs and five QA datasets, comparing with vanilla RAG, InstructRAG, and CLeHe, together with noise-robustness and ablation analyses. The headline claim is that CrEst consistently outperforms strong baselines and remains robust under high noise.

Significance. If fully supported, the paper would offer a practical, model-agnostic way to reduce the impact of unreliable retrieved passages, which is especially relevant for black-box LLM APIs. The core idea of using inter-document agreement as a credibility signal is simple and appealing, and the black-box/white-box separation is a useful framing. The paper also provides helpful ablations on the number of embedders and documents, and analyses of score distributions and attention redistribution. However, the current evidence falls short of the central claims: the mathematical derivation omits required assumptions, the reported gains are not consistent across Table 1, the definition of the noise experiment is ambiguous, and the combination method that produces the strongest results is not described. These issues are load-bearing rather than cosmetic.

major comments (5)
  1. [Section 2.2, Eq. (3)] The identity E[||lambda_i - lambda_j||^2] = E[||lambda_i - lambda*||^2] + E[||lambda_j - lambda*||^2] requires lambda_i and lambda_j to be conditionally independent given lambda* and to have zero-mean conditional deviations. Neither condition is stated or justified in the paper. Without it, a cross term -2 E[(lambda_i - lambda*) . (lambda_j - lambda*)] remains, so Eq. (4) is not derived from Eq. (1). The paper should state the required distributional assumptions and, if they are not guaranteed by the embedding model, provide evidence that the cross term is negligible.
  2. [Section 2.2, Eq. (4) and computation of s_i] Even granting Eq. (3), the manuscript does not specify how the expectation deltas are estimated or how triplets are combined when n > 3. In practice, observed pairwise distances replace the expectations without any finite-sample estimator or error analysis, and the resulting values of E[||lambda_i - lambda*||^2] can be negative for an inconsistent triplet. The authors should specify the exact estimator (e.g., averaging over all triples, clipping negative values, or solving a least-squares system) and report the frequency of negative estimates.
  3. [Table 1 and Section 1] The claim that CrEst consistently outperforms strong baselines is contradicted by several cells in Table 1. For example, on Gemma-7B HPQA, CrEst-bbx (22.60) is below both Vanilla RAG (26.60) and InstructRAG (25.60); on Gemma-7B TQA, CrEst-wbx (57.92) is below Vanilla (58.75); and on Gemma-7B ASQA F1, CrEst-bbx (15.73) is far below Vanilla (23.40) and InstructRAG (31.93). The abstract and introduction should either be revised to state the scope of the improvement or these counterexamples should be explained. In addition, no error bars or statistical significance tests are reported, so the word 'significant' in the abstract is unsupported.
  4. [Section 4.1, Figure 2] The stress-test concern about this experiment is valid. The noise injection only corrupts documents that contain the golden answer, so the reported '80% noise' is not defined as 80% of the full retrieved set. With PopQA's five-document retrieval, this setup need not ever reach a state in which more than half of the retrieved documents are non-credible, which is the regime where the authors themselves concede the method's assumption fails. The claim that CrEst maintains robust performance under high-noise conditions is therefore not established for the setting that matters. Please report, per query, the fraction of the full retrieved set that is actually wrong after injection, and include a condition in which the majority of retrieved documents are demonstrably wrong.
  5. [Section 3.2, Table 1, and Section 2.3] The strongest results in Table 1 come from 'CrEst-wbx + InstructRAG', but the paper never states how the two methods are combined. It also does not specify whether the attention scaling Attn(x_di) = s_i * Attn(x_di) * C applies to pre-softmax logits or post-softmax attention, nor how the normalization constant C is computed. Without these details, the main empirical contribution is not reproducible and the attribution of the headline gain to CrEst is unclear.
minor comments (6)
  1. [Section 2.1, Eq. (1)] The summation index 'i=0' is inconsistent with the retrieval set {d_1, ..., d_n}; it should presumably be 'i=1'. Also, Z is described as 'the log partition functions' but should be 'the log partition function' or simply 'the partition function'.
  2. [Section 2.2, Eq. (4)] The notation 'forall (i,j,k) in [n]' is imprecise: it should specify ordered distinct triples of indices, since the equation is not meaningful when i, j, k are not distinct.
  3. [Section 4.2, Figures 3 and 4] The ablation text says the number of embedders is varied 'from one to five', but the results are said to 'plateau after seven embedders'; the x-axis labels are missing from both figures. Additionally, PopQA is described as using five retrieved documents, so it is unclear how the 'number of retrieved documents' is varied up to twenty in Figure 4. There is also a typo: 'CrEst-qbx' should be 'CrEst-wbx'.
  4. [Section 4.1, Figure 2] The y-axis of Figure 2 is on a log scale but this is not mentioned in the text, and the x-axis label 'percentage of corrupted documents' is ambiguous for the reasons given in the major comments.
  5. [Section 3.2, Table 1] The CLeHe baseline achieves 0.00 on Gemma-7B HPQA, which is likely an artifact of the generation or evaluation pipeline; this should be checked and discussed, or the baseline should be re-run with a more robust setting.
  6. [Throughout] There are several typos, including 'Mistral-7B-Insruct-v0.2' and 'approaches'/'appraoch' in Section 3.2. These do not affect the technical content but should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central derivation defines credibility via inter-document agreement, but the headline claims are validated against external QA labels, so no fitted input is renamed as a prediction.

full rationale

CrEst's derivation chain is not circular. The credibility score in Eq. (2) is defined as the inverse expected distance to an unobserved true document, and Eqs. (3)-(4) recover that quantity from pairwise embedding distances under the weak-supervision model adopted from Shin et al. (2021) and Fu et al. (2020). This is a modeling choice with an openly stated assumption and an acknowledged failure mode (Section 2.2, Limitations), not a result baked into the evaluation. The headline improvements are measured against external benchmark answer labels (HotpotQA, PopQA, NQ, TriviaQA, ASQA) with no parameter fitted to those labels, so the accuracy and F1 claims are independently falsifiable. The white-box attention scaling and black-box output aggregation do not encode answer presence. The only self-citation (Adila et al., 2024, for attention steering) is corroborated by Zhang et al. (2023) and Deng et al. (2024) and is not load-bearing. The skeptical concern about the 80%-noise experiment is a possible correctness or experimental-design issue (ambiguity in the noise denominator), not a circular reduction of Eq. (2) to the benchmark result.

Assumptions & free parameters 3 free parameters · 5 assumptions · 2 invented entities

The central claim rests on a latent true document and the majority-agreement assumption, plus the unstated conditional independence of embeddings. No answer labels are used during scoring, which keeps the method label-free, but the reliability of the scores depends entirely on the embedding quality and on the retrieved set being majority-reliable.

free parameters (3)
  • Number of embedding functions M = M=5 in main results; gains plateau after 7 embedders
    Chosen by hand and affects the variance of credibility scores and the inference cost of the black-box variant.
  • Embedding model ensemble = NV-Embed-v1, SimCSE, stella_en_1.5B_v5, MS-Marco reranker
    Specific embedders are selected without a sensitivity analysis; the estimated scores depend on this choice.
  • Attention normalization constant C = unspecified
    Required by the formula Attn(xdi)=si*Attn(xdi)*C to keep total attention unchanged; implementation details are not given.
assumptions (5)
  • domain assumption There exists an unobserved true document d* whose embedding lambda* is close to most retrieved documents.
    Core of Eq 1 and Section 2.2; if false, scores assign high credibility to the noisy majority.
  • domain assumption Document embeddings are conditionally independent given lambda* (product form of Eq 1).
    Needed for Eq 3, E||lambda_i-lambda_j||^2 = E||lambda_i-lambda*||^2 + E||lambda_j-lambda*||^2, but this assumption is not stated.
  • domain assumption Semantic closeness in embedding space measures credibility or answer relevance.
    The entire score is built on Euclidean distances between embeddings.
  • domain assumption Observed pairwise distances of embeddings are accurate estimates of the model expectations in Eq 3.
    Finite retrieved sets and imperfect embedders introduce noise; ensembling mitigates but does not eliminate bias.
  • domain assumption In practice, most retrieved documents are reliable, so the majority is a good proxy for the true document.
    Stated in Section 2.2 and the Limitations; the method's failure when the majority is unreliable is acknowledged.
invented entities (2)
  • Unobserved true document d* and its embedding lambda*
    purpose: Anchor of the weak supervision graphical model; credibility scores are defined as reciprocal distances to lambda*.
    It is a latent modeling construct with no observable handle; the paper provides no way to validate lambda* outside the majority-agreement assumption.
  • Canonical parameters theta_i in Eq 1
    purpose: Specify the probabilistic graphical model and encode penalties for disagreements between lambda* and each document.
    The theta parameters are never estimated or used in the final score computation, so they are implicit invented quantities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision." pith.science (2026). https://pith.science/paper/EWFYBZI5

@misc{pith2026250614912,
  author       = {Pith},
  title        = {Pith review of: CrEst: Credibility Estimation for Contexts in LLMs via Weak Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EWFYBZI5}},
  note         = {Machine review of arXiv:2506.14912}
}
read the original abstract

The integration of contextual information has significantly enhanced the performance of large language models (LLMs) on knowledge-intensive tasks. However, existing methods often overlook a critical challenge: the credibility of context documents can vary widely, potentially leading to the propagation of unreliable information. In this paper, we introduce CrEst, a novel weakly supervised framework for assessing the credibility of context documents during LLM inference--without requiring manual annotations. Our approach is grounded in the insight that credible documents tend to exhibit higher semantic coherence with other credible documents, enabling automated credibility estimation through inter-document agreement. To incorporate credibility into LLM inference, we propose two integration strategies: a black-box approach for models without access to internal weights or activations, and a white-box method that directly modifies attention mechanisms. Extensive experiments across three model architectures and five datasets demonstrate that CrEst consistently outperforms strong baselines, achieving up to a 26.86% improvement in accuracy and a 3.49% increase in F1 score. Further analysis shows that CrEst maintains robust performance even under high-noise conditions.

Figures

Figures reproduced from arXiv: 2506.14912 by the authors.

Figure 1
Figure 1. First, documents are embedded into the latent space. Next, we measure the pairwise distance between the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. CrEst with increasing noise. X-axis: % of corrupted documents, Y-axis: Accuracy % (log scale). and InstrRAG achieves optimal performance on larger models (Mistral and Nemotron), CrEst-wbx alone performs better on Gemma. These results not only underscore CrEst’s enhanced robustness in high-noise settings but also highlight a crucial insight: for smaller models, methods that rely on the model’s inherent reasoning capa… view at source ↗
Figure 3
Figure 3. Impact of the number of embedder func￾tions. CrEst-bbx benefits from using more embed￾ders [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: CrEst score distribution on NaturalQuestions (NQ) for both documents containing the golden answer (blue) and distractor documents (orange) [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 8
Figure 8. Figure 8: Frequency distribution of the correct answer’s [PITH_FULL_IMAGE:figures/full_fig_p007_8.png]
Figure 7
Figure 7. Figure 7: Attention score distribution before CrEst-wbx (orange) and after (blue). CrEst re-distributes the at￾tention score of each documents token based on the document’s credibility. wbx (right, blue). Without CrEst-wbx, most tokens receive uniform attention scores of 1, prev…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 10 canonical work pages

  1. [1]

    Dyah Adila, Shuai Zhang, Boran Han, and Bernie Wang. 2024. Discovering bias in latent space: An unsupervised debiasing approach. In International Conference on Machine Learning, pages 246--261. PMLR

    work page 2024

  2. [2]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511

    arXiv 2023

  3. [3]

    Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024. The power of noise: Redefining retrieval for rag systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 719--729

    2024

  4. [4]

    Boyi Deng, Wenjie Wang, Fengbin Zhu, Qifan Wang, and Fuli Feng. 2024. Cram: Credibility-aware attention modification in llms for combating misinformation in rag. arXiv preprint arXiv:2406.11497

    work page Pith review arXiv 2024

  5. [5]

    Daniel Fu, Mayee Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher R \'e . 2020. Fast and three-rious: Speeding up weak supervision with triplet methods. In International conference on machine learning, pages 3280--3291. PMLR

    work page 2020

  6. [6]

    Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE : Simple contrastive learning of sentence embeddings. In Empirical Methods in Natural Language Processing (EMNLP)

    2021

  7. [7]

    Izacard Gautier, Lewis Patrick, Lomeli Maria, Hosseini Lucas, Petroni Fabio, Schick Timo, Dwivedi-Yu Jane, Joulin Armand, Riedel Sebastian, and Grave Edouard. 2022. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv: 2208.03299

    arXiv 2022

  8. [8]

    Neel Guha, Mayee F Chen, Trevor Chow, Ishan S Khare, and Christopher R \'e . 2024. Smoothie: Label free language model routing. arXiv preprint arXiv:2412.04692

    arXiv 2024

Show all 39 references
  1. [9]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929--3938. PMLR

    2020

  2. [10]

    Sarah Hooper, Michael Wornow, Ying Hang Seah, Peter Kellman, Hui Xue, Frederic Sala, Curtis Langlotz, and Christopher Re. 2020. Cut out the annotator, keep the cutout: better segmentation with weak supervision. In International Conference on Learning Representations

    2020

  3. [11]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  4. [12]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  5. [13]

    Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019

  6. [14]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. https://arxiv.org/abs/2405.17428 Nv-embed: Improved techniques for training llms as generalist embedding models . Preprint, arXiv:2405.17428

    2024 arXiv

  7. [15]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Informati...

    2020

  8. [16]

    Christina Lioma, Birger Larsen, Wei Lu, and Yong Huang. 2016. A study of factuality, objectivity and relevance: three desiderata in large-scale information retrieval? In Proceedings of the 3rd IEEE/ACM International Conference on Big Data Computing, Applications and Technologi...

    2016

  9. [17]

    Liyuan Liu, Xiang Ren, Qi Zhu, Shi Zhi, Huan Gui, Heng Ji, and Jiawei Han. 2017. Heterogeneous supervision for relation extraction: A representation learning approach. arXiv preprint arXiv:1707.00166

    2017 arXiv

  10. [18]

    Hongyin Luo, Tianhua Zhang, Yung-Sung Chuang, Yuan Gong, Yoon Kim, Xixin Wu, Helen Meng, and James Glass. 2023. Search augmented instruction learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3717--3729

    2023

  11. [19]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Investigating effectiveness and limitations of parametric and non-parametric memories. arXiv preprint

    2022

  12. [20]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. https://doi.org/10.18653/v1/2023.acl-long.546 When not to trust language models: Investigating effectiveness of parametric and non-parametric memories . In Proceedings of the 6...

    2023 doi

  13. [21]

    Ruotong Pan, Boxi Cao, Hongyu Lin, Xianpei Han, Jia Zheng, Sirui Wang, Xunliang Cai, and Le Sun. 2024. https://arxiv.org/abs/2404.06809 Not all contexts are equal: Teaching llms credibility-aware generation . Preprint, arXiv:2404.06809

    2024 arXiv

  14. [22]

    Zexuan Qiu, Zijing Ou, Bin Wu, Jingjing Li, Aiwei Liu, and Irwin King. 2024. Entropy-based decoding for retrieval-augmented large language models. arXiv preprint arXiv:2406.17519

    2024 arXiv

  15. [23]

    Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher R \'e . 2017. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the VLDB endowment. International conference on very large data bases, volume 11, page 269. NIH ...

    2017

  16. [24]

    Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher R \'e . 2016. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29

    2016

  17. [25]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    2019 arXiv

  18. [26]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess \` , Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36

    2024

  19. [27]

    Chi, Nathanael Sch\" a rli, and Denny Zhou

    Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H. Chi, Nathanael Sch\" a rli, and Denny Zhou. 2023. https://proceedings.mlr.press/v202/shi23a.html Large language models can be easily distracted by irrelevant context . In Proceedings of the 40th Internat...

    2023

  20. [28]

    Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. 2021. Universalizing weak supervision. arXiv preprint arXiv:2112.03865

    2021 arXiv

  21. [29]

    Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022. Asqa: Factoid questions meet long-form answers. arXiv preprint arXiv:2204.06092

    2022 arXiv

  22. [30]

    Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan \"O Ar k. 2024. Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. arXiv preprint arXiv:2410.07176

    2024 arXiv

  23. [31]

    Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2024. Instructrag: Instructing retrieval-augmented generation with explicit denoising. arXiv preprint arXiv:2406.13629

    2024 arXiv

  24. [32]

    Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. 2024. Certifiably robust rag against retrieval corruption. arXiv preprint arXiv:2405.15556

    2024

  25. [33]

    Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884

    2024 arXiv

  26. [34]

    Cohen, Ruslan Salakhutdinov, and Christopher D

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA : A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing ( EMNLP )

    2018

  27. [35]

    Qingru Zhang, Chandan Singh, Liyuan Liu, Xiaodong Liu, Bin Yu, Jianfeng Gao, and Tuo Zhao. 2023. Tell your model where to attend: Post-hoc attention steering for llms. arXiv preprint arXiv:2311.02262

    2023 arXiv

  28. [36]

    Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. 2024. Raft: Adapting language model to domain specific rag. arXiv preprint arXiv:2403.10131

    2024 arXiv

  29. [37]

    Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2024. Poisonedrag: Knowledge corruption attacks to retrieval-augmented generation of large language models. arXiv preprint arXiv:2402.07867

    2024 arXiv

  30. [38]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  31. [39]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.