REVIEW 8 cited by
DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems
read the original abstract
Recently, there has been a growing interest among large language model (LLM) developers in LLM-based document reading systems, which enable users to upload their own documents and pose questions related to the document contents, going beyond simple reading comprehension tasks. Consequently, these systems have been carefully designed to tackle challenges such as file parsing, metadata extraction, multi-modal information understanding and long-context reading. However, no current benchmark exists to evaluate their performance in such scenarios, where a raw file and questions are provided as input, and a corresponding response is expected as output. In this paper, we introduce DocBench, a new benchmark designed to evaluate LLM-based document reading systems. Our benchmark involves a meticulously crafted process, including the recruitment of human annotators and the generation of synthetic questions. It includes 229 real documents and 1,102 questions, spanning across five different domains and four major types of questions. We evaluate both proprietary LLM-based systems accessible via web interfaces or APIs, and a parse-then-read pipeline employing open-source LLMs. Our evaluations reveal noticeable gaps between existing LLM-based document reading systems and human performance, underscoring the challenges of developing proficient systems. To summarize, DocBench aims to establish a standardized benchmark for evaluating LLM-based document reading systems under diverse real-world scenarios, thereby guiding future advancements in this research area.
Forward citations
Cited by 8 Pith papers
-
GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents
Frontier multimodal models achieve at most a 15% strict pass rate on an adversarially filtered 100-item benchmark of grounded professional PDF reasoning tasks.
-
Hybrid Retriever Evolution for Multimodal Document Reasoning Agents
A meta-agent uses failure analysis to evolve a task agent's instructions for coordinating lexical, semantic, and multimodal retrievers, leading to up to 19.6 point gains on document QA benchmarks.
-
GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents
On GDP.pdf, 100 expert-authored professional PDF tasks, seventeen frontier multimodal models pass at most 30.7% of items, and most failures come from missed footnotes, exclusions, tables, and spatial details.
-
GDP.pdf: Benchmarking Grounded Multimodal Reasoning over Professional PDF Documents
Frontier multimodal models pass at most 15% of a 100-item expert-authored PDF reasoning benchmark that requires grounded multi-capability answers on real professional documents.
-
ATTAIN: Automated Exploit Failure Analysis through Trace-Driven Diff Analysis
ATTAIN is a three-module trace-driven framework that combines exploit execution, LLM-guided diff search, and evidence-based judgment to identify affected library versions for CVEs, reporting 93.24% F1 on 224 CVEs acro...
-
DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark
DocRetriever introduces a framework using layout-aware sparse embeddings for hybrid encoding without OCR and a generalizable reasoning-augmented reranker for few-shot settings, plus the MultiDocR benchmark for evaluation.
-
Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Training
Cognitive Kernel-Pro provides an open-source agent framework with curated training data across web, file, code, and reasoning domains plus test-time reflection and voting, achieving SOTA results on GAIA among free agents.
-
Human-AI Collaborative Game Testing with Vision Language Models
An experiment with 276 participants finds that vision language model assistance improves human game testers' defect identification, especially with design documentation, while AI errors create challenges.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.