REVIEW 25 cited by
PaperQA: Retrieval-Augmented Generative Agent for Scientific Research
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Language Models (LLMs) generalize well across language tasks, but suffer from hallucinations and uninterpretability, making it difficult to assess their accuracy without ground-truth. Retrieval-Augmented Generation (RAG) models have been proposed to reduce hallucinations and provide provenance for how an answer was generated. Applying such models to the scientific literature may enable large-scale, systematic processing of scientific knowledge. We present PaperQA, a RAG agent for answering questions over the scientific literature. PaperQA is an agent that performs information retrieval across full-text scientific articles, assesses the relevance of sources and passages, and uses RAG to provide answers. Viewing this agent as a question answering model, we find it exceeds performance of existing LLMs and LLM agents on current science QA benchmarks. To push the field closer to how humans perform research on scientific literature, we also introduce LitQA, a more complex benchmark that requires retrieval and synthesis of information from full-text scientific papers across the literature. Finally, we demonstrate PaperQA's matches expert human researchers on LitQA.
Forward citations
Cited by 25 Pith papers
-
NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?
NatureBench evaluates ten frontier AI coding agents on 90 tasks from Nature papers under web-search-disabled conditions and finds the strongest agent surpasses published SOTA on only 17.8% of tasks, succeeding mainly ...
-
LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering
LitTraceQA is a scientific QA benchmark that requires systems to retrieve relevant papers, ground answers in typed evidence, and produce verifiable answers, with each stage scored separately.
-
AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs
AISE-Bench is a real-user-query benchmark with annotated API trajectories and grounded answers that exposes LLM agents' weak performance in multi-step academic information seeking.
-
EXP-Bench: Can AI Conduct AI Research Experiments?
EXP-Bench is a new benchmark of 461 end-to-end AI research experiments, and leading AI agents complete fewer than 1 percent of them successfully.
-
Single-agent or Multi-agent Systems? Why Not Both?
On 15 agentic benchmarks, the accuracy advantage of multi-agent LLM systems over single-agent systems mostly disappears with stronger base models, and a hybrid single/multi-agent cascade improves accuracy and cuts cost.
-
Toward Reliable Scientific Hypothesis Generation: Evaluating Truthfulness and Hallucination in Large Language Models
A new benchmark (TruthHypo) and a knowledge-grounded hallucination detector (KnowHD) show that grounding scores can partially select truthful LLM-generated biomedical hypotheses, but the result is at risk from knowled...
-
Unveiling Knowledge Utilization Mechanisms in LLM-based Retrieval-Augmented Generation
RAG knowledge use in LLMs proceeds through four stages (refinement, elicitation, expression, contestation), and a new entropy metric (KAPE) identifies neurons whose deactivation shifts reliance between internal and ex...
-
Aviary: training language agents on challenging scientific tasks
A small open-source LLM trained in the new Aviary environments with expert iteration and majority voting matches or exceeds a frontier LLM agent on SeqQA and LitQA2 at far lower inference cost.
-
VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities
VISION is a modular LLM-based assistant that demonstrated voice-controlled operation of an X-ray scattering beamline, converting natural language into executable beamline code.
-
SynthLens: Visual Analytics for Facilitating Multi-step Synthetic Route Design
SynthLens combines AI-based paper reading with a tree visualization and weighted ranking to let chemists build and compare multiple synthetic routes.
-
VecTree-RAG: An Agentic Retrieval-Augmented Generation Framework Combining Vector and Tree Retrieval for Efficiency and Accuracy
An agentic RAG framework that splits corpus-level discovery (vector retrieval) from within-paper evidence localization (tree navigation) reports top scores on QASPER, LitQA2, and a new MOSAIC benchmark.
-
VASP Agent: An Agentic Framework for Autonomous First-principles Calculations
An LLM-driven agent with predefined VASP workflows and parameter-checking tools completes DFT simulation tasks more reliably and accurately than standalone LLMs, with a new 80-task benchmark.
-
GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models
An LLM can memorize a knowledge graph into LoRA weights and answer relation/reasoning queries about it without graph context, but the evaluation partly trains on the test task.
-
DeepResearch$^{\text{Eco}}$: A Recursive Agentic Workflow for Complex Scientific Question Answering in Ecology
A recursive agentic pipeline for literature synthesis showing a 21-fold source increase and 14.9-fold density gain when depth and breadth are raised.
-
HiPerRAG: High-Performance Retrieval Augmented Generation for Scientific Insights
A scalable HPC retrieval-augmented generation pipeline using a YOLO/Texify parser (Oreo) and a contrastive plus late-interaction encoder fine-tuning method (ColTrast) achieves 90% on SciQ and up to 76% on PubMedQA.
-
WindVE: Collaborative CPU-NPU Vector Embedding
WindVE offloads peak vector-embedding queries to idle CPUs through a regression-calibrated queue manager, achieving up to 22.3% higher concurrency than a GPU/NPU-only baseline under the same SLOs.
-
Making Sense of Data in the Wild: Data Analysis Automation at Scale
A multi-agent LLM system with retrieval-augmented generation automatically curates datasets from Zenodo and Hugging Face, yielding small retrieval gains and a confounded synthetic-data improvement.
-
ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
ASRank re-ranks retrieved documents by scoring how well each document supports a zero-shot answer scent generated by a large LLM, beating UPR and RankGPT on several QA datasets.
-
AI for Auto-Research: Roadmap & User Guide
The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.
-
How Far Are AI Scientists from Changing the World?
This survey proposes a four-level capability framework for AI Scientist systems and, using an AI reviewer, finds that current systems produce papers rated well below normal scientific standards.
-
Open-Source Agentic Hybrid RAG Framework for Scientific Literature Review
A DPO-tuned agentic hybrid RAG system that routes queries between a knowledge graph and a vector store beat a static baseline on a self-generated benchmark.
-
Reading Between the Timelines: RAG for Answering Diachronic Questions
TA-RAG uses LLM-extracted time intervals, time-filtered retrieval with averaged temporal query embeddings, and chronologically ordered context to beat standard RAG by 13-27 points on the new ADQAB benchmark of 525 mul...
-
Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey
A survey that classifies AI agent evaluation benchmarks along environment and capability axes, and proposes five traits that distinguish agents from chatbots.
-
On the Comprehensibility of Multi-structured Financial Documents using LLMs and Pre-processing Tools
Preprocessing financial PDFs into text, tables, and chart data with existing tools improves LLM question-answering accuracy over direct GPT-4o image input in a small private evaluation.
-
Spark: A System for Scientifically Creative Idea Generation
An LLM pipeline for generating and filtering science ideas is described, but its central claims are unverified.
Discussion (0). Continue with ORCID to comment.