Pith. sign in

REVIEW 13 cited by

ChatQA: Surpassing GPT-4 on Conversational QA and RAG

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.10225 v5 pith:QLQYQCDA submitted 2024-01-18 cs.CL cs.AIcs.IRcs.LG

Zihan Liu , Wei Ping , Rajarshi Roy , Peng Xu , Chankyu Lee , Mohammad Shoeybi , Bryan Catanzaro This is my paper · ORCID
classification cs.CLcs.AIcs.IRcs.LG
keywords benchchatragconversationalgpt-4modelmodelsscorechatqa
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

In this work, we introduce ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question answering (QA). To enhance generation, we propose a two-stage instruction tuning method that significantly boosts the performance of RAG. For effective retrieval, we introduce a dense retriever optimized for conversational QA, which yields results comparable to the alternative state-of-the-art query rewriting models, while substantially reducing deployment costs. We also present the ChatRAG Bench, which encompasses ten datasets covering comprehensive evaluations on RAG, table-related QA, arithmetic calculations, and scenarios involving unanswerable questions. Our ChatQA-1.0-70B (score: 54.14), built on Llama2, a weaker foundation model than GPT-4, can slightly outperform GPT-4-0613 (score: 53.90) and GPT-4-Turbo-2024-04-09 (score: 54.03) on the ChatRAG Bench, without relying on any synthetic data from OpenAI GPT models. Notably, the Llama3-ChatQA-1.5-70B model surpasses the accuracy of GPT-4-Turbo-2024-04-09, achieving a 4.4% improvement. To advance research in this field, we open-sourced the model weights, instruction tuning data, ChatRAG Bench, and retriever for the community: https://chatqa-project.github.io/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning

    cs.AI 2026-06 conditional novelty 6.0 of 10

    Per-question database-style plan search over multi-LLM DAGs improves QA quality under budgets by ~58% (MMLU-Pro) and ~41% (SimpleQA) versus reimplemented baselines.

  2. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

    cs.DC 2026-02 unverdicted novelty 6.0 of 10

    SPEED-Bench is a new standardized benchmark for speculative decoding that supplies semantically diverse qualitative data and throughput-oriented splits across concurrency levels, integrated with vLLM and TensorRT-LLM.

  3. HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A hierarchical chain-of-thought instruction-tuning curriculum for filtering, combination, and reasoning improves zero-shot retrieval-augmented QA.

  4. JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A multi-agent LLM framework with rule enforcement, compiler feedback, and retrieval achieves 92/93/81% pass@1 on three self-built EDA benchmarks, up from 67/62/43% for the best single model.

  5. DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation

    cs.CL 2025-05 conditional novelty 6.0 of 10

    DynamicRAG trains a reranker as an RL agent, using the generator's answer quality as reward to dynamically choose how many and which retrieved documents to pass forward.

  6. Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models

    cs.IR 2025-05 conditional novelty 6.0 of 10

    DRO jointly trains a generative document selector and an LLM generator by treating document order as a latent variable and using importance-sampled expectation-maximization, beating prior RAG systems on five benchmarks.

  7. DeepThink: Aligning Language Models with Domain-Specific User Intents

    cs.CL 2025-02 conditional novelty 6.0 of 10

    DeepThink improves domain-specific QA by synthesizing conversation-based training data and refining answers with retrieval-augmented feedback, beating a GPT-4-turbo+RAG assistant by 7.92% on advertising-domain real us...

  8. LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A two-stage, English-only training method aligns XLM-R representations with Mistral-7B to produce multilingual embeddings, improving low-resource language scores by up to 22 points while losing about 10 points on English.

  9. Efficient Knowledge Injection in LLMs via Self-Distillation

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Self-distillation from a model's own in-context answers injects factual knowledge into LLM weights more efficiently than supervised fine-tuning and is competitive with RAG.

  10. UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversations

    cs.CL 2025-07 reject novelty 5.0 of 10

    A single LLM jointly fine-tuned for conversational dense retrieval and retrieval-augmented generation beats separate retriever-plus-generator pipelines on most test collections, though its headline benchmark was conta...

  11. Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers

    cs.CL 2025-05 conditional novelty 5.0 of 10

    EXSEARCH trains LLMs for agentic search by treating search trajectories as latent variables and optimizing a weighted likelihood via expectation-maximization, yielding gains on NQ, HotpotQA, MuSiQue, and 2WikiQA.

  12. Retrieval Augmented Generation Evaluation for Health Documents

    cs.IR 2025-05 conditional novelty 4.0 of 10

    A European Commission team built and evaluated RAGEv, a RAG pipeline for health documents, reporting high yes/no and long-answer scores but recommending human verification of outputs.

  13. A Survey of the State-of-the-Art in Conversational Question Answering Systems

    cs.CL 2025-09 conditional novelty 2.0 of 10

    A review that categorizes ConvQA components, techniques, models, and datasets, with no new experimental result.

Pith tools