Pith. sign in

REVIEW 15 cited by

LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2408.10343 v1 pith:5SIASFVY submitted 2024-08-19 cs.AI

Nicholas Pipitone , Ghita Houir Alami This is my paper · ORCID
classification cs.AI
keywords legallegalbench-ragbenchmarkretrievalcontextdomainllmsrelevant
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Retrieval-Augmented Generation (RAG) systems are showing promising potential, and are becoming increasingly relevant in AI-powered legal applications. Existing benchmarks, such as LegalBench, assess the generative capabilities of Large Language Models (LLMs) in the legal domain, but there is a critical gap in evaluating the retrieval component of RAG systems. To address this, we introduce LegalBench-RAG, the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space. LegalBench-RAG emphasizes precise retrieval by focusing on extracting minimal, highly relevant text segments from legal documents. These highly relevant snippets are preferred over retrieving document IDs, or large sequences of imprecise chunks, both of which can exceed context window limitations. Long context windows cost more to process, induce higher latency, and lead LLMs to forget or hallucinate information. Additionally, precise results allow LLMs to generate citations for the end user. The LegalBench-RAG benchmark is constructed by retracing the context used in LegalBench queries back to their original locations within the legal corpus, resulting in a dataset of 6,858 query-answer pairs over a corpus of over 79M characters, entirely human-annotated by legal experts. We also introduce LegalBench-RAG-mini, a lightweight version for rapid iteration and experimentation. By providing a dedicated benchmark for legal retrieval, LegalBench-RAG serves as a critical tool for companies and researchers focused on enhancing the accuracy and performance of RAG systems in the legal domain. The LegalBench-RAG dataset is publicly available at https://github.com/zeroentropy-cc/legalbenchrag.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark

    cs.IR 2025-08 conditional novelty 7.0 of 10

    State-of-the-art LLMs with retrieval answer simplified boolean questions about state unemployment insurance law with at best 0.69 F1, well short of reliable end-to-end code simplification.

  2. On Path to Multimodal Historical Reasoning: HistBench and HistAgent

    cs.AI 2025-05 conditional novelty 7.0 of 10

    HistAgent, a history-specialized agent, scores 27.54% pass@1 and 36.47% pass@2 on the new 414-question HistBench benchmark, surpassing generalist agents tested on the same data.

  3. NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO

    cs.CL 2026-07 conditional novelty 6.5 of 10

    Solver-verified NormWorlds-CF and MR-GRPO show that answer-only training is an unsafe proxy and that class-conditioned metamorphic rewards improve balanced counterfactual change structure.

  4. RAG-Zeval: Towards Robust and Interpretable Evaluation on RAG Responses through End-to-End Rule-Guided Reasoning

    cs.CL 2025-05 conditional novelty 6.0 of 10

    RAG-Zeval uses rule-guided RL with ranking rewards on synthetic responses to train a 7B model that evaluates RAG faithfulness and correctness competitively with 70B-class judges.

  5. CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A black-box poisoning framework, CPA-RAG, generates fluent fake documents that steer retrieval-augmented language models toward attacker-chosen wrong answers, achieving over 90% success in the reported experiments.

  6. Hypercube-Based Retrieval-Augmented Generation for Scientific Question-Answering

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A document hypercube with LLM-extracted labels and hybrid sparse/dense matching improves retrieval and QA accuracy over RAG baselines on three datasets.

  7. Evaluating RAG for French immigration law: a benchmark and baseline study

    cs.IR 2026-07 conditional novelty 5.0 of 10

    Dense RAG improves French immigration permit-type accuracy over parametric Qwen baselines on a 52-profile public benchmark, with weaker gains on documents and citations.

  8. SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in India

    cs.MA 2025-09 conditional novelty 5.0 of 10

    A multi-agent system simulates Indian judicial deliberation using LLM agents grounded in legal texts via retrieval-augmented generation, with early tests suggesting RAG improves consistency.

  9. ASP2LJ : An Adversarial Self-Play Laywer Augmented Legal Judgment Framework

    cs.CL 2025-06 conditional novelty 5.0 of 10

    ASP2LJ combines synthetic case generation with adversarial self-play for lawyer agents, improving legal judgment prediction on a Chinese benchmark and on a new rare-case dataset.

  10. Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning

    cs.CL 2025-06 conditional novelty 5.0 of 10

    State-of-the-art text embeddings lag far behind on tasks requiring pragmatic inference, stance detection, and social meaning, relative to their strong performance on surface semantic benchmarks.

  11. Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains

    cs.CL 2025-05 reject novelty 5.0 of 10

    METEORA uses DPO-tuned rationales to select and verify evidence chunks in RAG, and claims better recall, precision, evidence efficiency, and poisoning defense, though key evaluation details are missing.

  12. RAG-TESTER: Automated End-to-End Testing of Retrieval-Augmented Large Language Models

    cs.AI 2026-07 conditional novelty 4.0 of 10

    A coverage-oriented test generator that adds negative-rejection and complex-passage queries to RAG evaluation, reported to detect 6.6% more failures than a strategy-free baseline in 24 model configurations.

  13. Optimizing Hypergraph-Based RAG: Toward Better Fact Extraction and Chunk Retrieval

    cs.AI 2026-07 conditional novelty 4.0 of 10

    Combining self-consistency fact-extraction with Personalized PageRank over hypergraphs improves retrieval recall and answer completeness in RAG systems.

  14. From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEM

    cs.AI 2025-07 conditional novelty 4.0 of 10

    Uni-RAG combines a style-adaptive multimodal retriever with a small language model to retrieve STEM education materials and generate explanations from varied query inputs.

  15. Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression

    cs.LG 2025-05 conditional novelty 4.0 of 10

    ACBench tests compressed LLMs on agentic tasks and finds 4-bit quantization keeps tool use and workflow generation strong while hurting real-world application performance.

Pith tools