Pith. sign in

REVIEW 7 cited by

ChatQA: Surpassing GPT-4 on Conversational QA and RAG

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2401.10225 v5 pith:QLQYQCDA submitted 2024-01-18 cs.CL cs.AIcs.IRcs.LG

Zihan Liu , Wei Ping , Rajarshi Roy , Peng Xu , Chankyu Lee , Mohammad Shoeybi , Bryan Catanzaro This is my paper
classification cs.CLcs.AIcs.IRcs.LG
keywords benchchatragconversationalgpt-4modelmodelsscorechatqa
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this work, we introduce ChatQA, a suite of models that outperform GPT-4 on retrieval-augmented generation (RAG) and conversational question answering (QA). To enhance generation, we propose a two-stage instruction tuning method that significantly boosts the performance of RAG. For effective retrieval, we introduce a dense retriever optimized for conversational QA, which yields results comparable to the alternative state-of-the-art query rewriting models, while substantially reducing deployment costs. We also present the ChatRAG Bench, which encompasses ten datasets covering comprehensive evaluations on RAG, table-related QA, arithmetic calculations, and scenarios involving unanswerable questions. Our ChatQA-1.0-70B (score: 54.14), built on Llama2, a weaker foundation model than GPT-4, can slightly outperform GPT-4-0613 (score: 53.90) and GPT-4-Turbo-2024-04-09 (score: 54.03) on the ChatRAG Bench, without relying on any synthetic data from OpenAI GPT models. Notably, the Llama3-ChatQA-1.5-70B model surpasses the accuracy of GPT-4-Turbo-2024-04-09, achieving a 4.4% improvement. To advance research in this field, we open-sourced the model weights, instruction tuning data, ChatRAG Bench, and retriever for the community: https://chatqa-project.github.io/.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Opti-Q: A Constraint-Based Optimization Framework for Multi-LLM Question Planning

    cs.AI 2026-06 conditional novelty 6.0 of 10

    Per-question database-style plan search over multi-LLM DAGs improves QA quality under budgets by ~58% (MMLU-Pro) and ~41% (SimpleQA) versus reimplemented baselines.

  2. SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

    cs.DC 2026-02 unverdicted novelty 6.0 of 10

    SPEED-Bench is a new standardized benchmark for speculative decoding that supplies semantically diverse qualitative data and throughput-oriented splits across concurrency levels, integrated with vLLM and TensorRT-LLM.

  3. HIRAG: Hierarchical-Thought Instruction-Tuning Retrieval-Augmented Generation

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A hierarchical chain-of-thought instruction-tuning curriculum for filtering, combination, and reasoning improves zero-shot retrieval-augmented QA.

  4. JARVIS: A Multi-Agent Code Assistant for High-Quality EDA Script Generation

    cs.SE 2025-05 conditional novelty 6.0 of 10

    A multi-agent LLM framework with rule enforcement, compiler feedback, and retrieval achieves 92/93/81% pass@1 on three self-built EDA benchmarks, up from 67/62/43% for the best single model.

  5. UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversations

    cs.CL 2025-07 reject novelty 5.0 of 10

    A single LLM jointly fine-tuned for conversational dense retrieval and retrieval-augmented generation beats separate retriever-plus-generator pipelines on most test collections, though its headline benchmark was conta...

  6. Iterative Self-Incentivization Empowers Large Language Models as Agentic Searchers

    cs.CL 2025-05 conditional novelty 5.0 of 10

    EXSEARCH trains LLMs for agentic search by treating search trajectories as latent variables and optimizing a weighted likelihood via expectation-maximization, yielding gains on NQ, HotpotQA, MuSiQue, and 2WikiQA.

  7. A Survey of the State-of-the-Art in Conversational Question Answering Systems

    cs.CL 2025-09 conditional novelty 2.0 of 10

    A review that categorizes ConvQA components, techniques, models, and datasets, with no new experimental result.

Pith tools