Pith. sign in

REVIEW 10 cited by

RA-DIT: Retrieval-Augmented Dual Instruction Tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.01352 v4 pith:YJU6A6BZ submitted 2023-10-02 cs.CL cs.AI

Xi Victoria Lin , Xilun Chen , Mingda Chen , Weijia Shi , Maria Lomeli , Rich James , Pedro Rodriguez , Jacob Kahn
show 4 more authors
Gergely Szilvasy Mike Lewis Luke Zettlemoyer Scott Yih
This is my paper · ORCID
classification cs.CLcs.AI
keywords performancefine-tuningra-ditretrieval-augmentedapproachesdatadualexisting
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Retrieval-augmented language models (RALMs) improve performance by accessing long-tail and up-to-date knowledge from external data stores, but are challenging to build. Existing approaches require either expensive retrieval-specific modifications to LM pre-training or use post-hoc integration of the data store that leads to suboptimal performance. We introduce Retrieval-Augmented Dual Instruction Tuning (RA-DIT), a lightweight fine-tuning methodology that provides a third option by retrofitting any LLM with retrieval capabilities. Our approach operates in two distinct fine-tuning steps: (1) one updates a pre-trained LM to better use retrieved information, while (2) the other updates the retriever to return more relevant results, as preferred by the LM. By fine-tuning over tasks that require both knowledge utilization and contextual awareness, we demonstrate that each stage yields significant performance improvements, and using both leads to additional gains. Our best model, RA-DIT 65B, achieves state-of-the-art performance across a range of knowledge-intensive zero- and few-shot learning benchmarks, significantly outperforming existing in-context RALM approaches by up to +8.9% in 0-shot setting and +1.4% in 5-shot setting on average.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 15 citations worldwide. Full citation record

  1. Inference Scaling for Bridging Retrieval and Augmented Generation

    cs.CL 2024-12 conditional novelty 7.0 of 10

    MOI estimates a debiased utility for each retrieved passage from multiple permuted reads and reranks by it, yielding large RAG quality gains at the cost of extra LLM calls.

  2. PKAG-DDI: Pairwise Knowledge-Augmented Language Model for Drug-Drug Interaction Event Text Generation

    q-bio.BM 2025-07 conditional novelty 6.0 of 10

    A pairwise knowledge selector and a marginalization-based integration strategy let a language model generate drug-drug interaction event descriptions that outperform MolTC and MolT5 on BLEU, ROUGE, and METEOR across t...

  3. MaskSearch: A Universal Pre-Training Framework to Enhance Agentic Search Capability

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A pre-training task called RAMP, where models practice searching to fill masked text spans, improves downstream agentic open-domain QA performance across Qwen and LLaMA models.

  4. GainRAG: Preference Alignment in Retrieval-Augmented Generation through Gain Signal Synthesis

    cs.IR 2025-05 conditional novelty 6.0 of 10

    GainRAG aligns retriever and LLM preferences by training a selector on contrastive-perplexity 'gain' signals plus a pseudo-passage fallback, improving RAG accuracy on six QA datasets.

  5. Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models

    cs.IR 2025-05 conditional novelty 6.0 of 10

    DRO jointly trains a generative document selector and an LLM generator by treating document order as a latent variable and using importance-sampled expectation-maximization, beating prior RAG systems on five benchmarks.

  6. Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

    cs.CV 2025-01 conditional novelty 6.0 of 10

    StreamChat uses hierarchical memory and three parallel threads to enable real-time multi-round video question answering, and StreamBench is a new benchmark for evaluating such streaming systems.

  7. EvidenceMap: Learning Evidence Analysis to Unleash the Power of Small Language Models for Biomedical Question Answering

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Small fine-tuned encoders that emit evidence-analysis soft prompts can make frozen 3B generative models outperform 8B retrieval-augmented baselines on biomedical QA.

  8. Efficient Knowledge Injection in LLMs via Self-Distillation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Self-distillation from a model's own in-context answers injects factual knowledge into LLM weights more efficiently than supervised fine-tuning and is competitive with RAG.

  9. Investigating the Robustness of Retrieval-Augmented Generation at the Query Level

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Retrieval-augmented generation performance drops noticeably under minor query perturbations, with end-to-end results often tracking retriever behavior.

  10. LAW: Legal Agentic Workflows for Custody and Fund Services Contracts

    cs.AI 2024-12 reject novelty 4.0 of 10

    An agentic workflow using reusable legal tools beats a raw GPT-3.5 baseline on contract retrieval, but the evaluation is compromised because the ground truth was generated with the same tools.

Pith tools