LLMs show severe staleness after training cutoffs and recency bias on historical German statutes; RAG with version filtering mitigates both better than web search.
hub
Lexam: Benchmarking legal reasoning on 340 law exams
18 Pith papers cite this work. Polarity classification is still indexing.
18
Pith papers citing it
hub tools
citation-role summary
background 1
citation-polarity summary
years
2026 18roles
background 1polarities
background 1representative citing papers
Magis-Bench is a new benchmark of 74 magistrate-level legal writing tasks from Brazilian exams where the strongest LLMs reach only 6.97/10, showing judicial reasoning remains difficult for current models.
Expert evaluation of LLMs on Japanese bar exam writing tasks shows clear limitations in open-ended legal reasoning and frequent hallucinations unsupported by law or precedent.
Creates TF-RefusalBench to quantify over-alignment in LLMs on criminal-law tasks across four languages and shows abliteration mitigates refusals with little performance loss.
LegalHalluLens provides typed hallucination profiles over CUAD, a Risk Direction Index, and a calibrated debate pipeline that reveals 38-40 pp category gaps hidden by aggregate 52% error rates and reduces fabricated detections by 45%.
Introduces NormBench benchmark and Span-Grounded Deontic Trees (SG-DT) for defeasible scope parsing to reduce Silent Scope Omission in LLMs on statutes and policies.
HARVE removes the component of the reward-head vector aligned with a multi-directional hacking subspace from residual streams using a small set of contrastive examples, improving robustness on RewardHackBench across eight models without fine-tuning while preserving general capability.
LegalCiteBench reveals that current LLMs achieve under 7% accuracy on closed-book legal citation retrieval and completion tasks, with misleading answer rates above 94% for nearly all tested models.
EcoGym is a new open benchmark with three economic environments that reveals no leading LLM dominates at sustained plan-and-execute decision making across scenarios.
Multi-agent deliberation frameworks for legal reasoning with LLMs match baseline performance but yield distinct answers that cover cases single models miss.
PolicyAlign aligns LLMs to natural-language safety policies by synthesizing violating instructions and performing on-policy self-distillation with policy-sensitive filtering, improving safety without high-quality supervision data.
Legal AI benchmarks must evaluate robustness to pro se litigant inputs rather than expert-preprocessed ones to support access-to-justice claims.
EP-HUBO treats CoT evidence selection as higher-order unconstrained binary optimization over per-hypothesis pools with quality weights to improve aggregation on legal benchmarks.
Reasoning-oriented LLMs reach up to 0.91 quadratic weighted kappa agreement with experts on public law cases when given sample solutions and grading rubrics, but only 0.60 on criminal law cases.
Automatic prompt optimization using lenient LLM judges improves performance and transferability in legal QA evaluations compared to human design or strict judges.
Rulemapping uses expert symbolic scaffolds to constrain LLMs, raising precision on §130(1) German hate-speech classification from 0.34-0.49 to 0.80-0.86 while preserving recall of 0.82-0.89.
NyayaMind combines RAG retrieval with domain-specific LLMs to generate transparent, structured legal reasoning and judgment predictions for Indian court cases.
BenGER integrates task creation, annotation, configurable LLM runs, and lexical/semantic/factual/judge metrics into a multi-tenant web platform for German legal benchmarking.
citing papers explorer
-
Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answering
LLMs show severe staleness after training cutoffs and recency bias on historical German statutes; RAG with version filtering mitigates both better than web search.
-
Magis-Bench: Evaluating LLMs on Magistrate-Level Legal Tasks
Magis-Bench is a new benchmark of 74 magistrate-level legal writing tasks from Brazilian exams where the strongest LLMs reach only 6.97/10, showing judicial reasoning remains difficult for current models.
-
Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task
Expert evaluation of LLMs on Japanese bar exam writing tasks shows clear limitations in open-ended legal reasoning and frequent hallucinations unsupported by law or precedent.
-
Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts
Creates TF-RefusalBench to quantify over-alignment in LLMs on criminal-law tasks across four languages and shows abliteration mitigates refusals with little performance loss.
-
LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI
LegalHalluLens provides typed hallucination profiles over CUAD, a Risk Direction Index, and a calibrated debate pipeline that reveals 38-40 pp category gaps hidden by aggregate 52% error rates and reduces fabricated detections by 45%.
-
From Statute to Control Flow: Span-Grounded Deontic Trees for Defeasible Scope Parsing
Introduces NormBench benchmark and Span-Grounded Deontic Trees (SG-DT) for defeasible scope parsing to reduce Silent Scope Omission in LLMs on statutes and policies.
-
HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
HARVE removes the component of the reward-head vector aligned with a multi-directional hacking subspace from residual streams using a small set of contrastive examples, improving robustness on RewardHackBench across eight models without fine-tuning while preserving general capability.
-
LegalCiteBench: Evaluating Citation Reliability in Legal Language Models
LegalCiteBench reveals that current LLMs achieve under 7% accuracy on closed-book legal citation retrieval and completion tasks, with misleading answer rates above 94% for nearly all tested models.
-
EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies
EcoGym is a new open benchmark with three economic environments that reveals no leading LLM dominates at sustained plan-and-execute decision making across scenarios.
-
Investigating Multi-Agent Deliberation in Law
Multi-agent deliberation frameworks for legal reasoning with LLMs match baseline performance but yield distinct answers that cover cases single models miss.
-
PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
PolicyAlign aligns LLMs to natural-language safety policies by synthesizing violating instructions and performing on-policy self-distillation with policy-sensitive filtering, improving safety without high-quality supervision data.
-
Legal Reasoning Is Not Lawyering: Rethinking Legal Benchmarks for Pro Se Access to Justice
Legal AI benchmarks must evaluate robustness to pro se litigant inputs rather than expert-preprocessed ones to support access-to-justice claims.
-
Quantum-Inspired Trace-Augmented Evidence Selection for Reasoning over Structured Hypothesis Spaces
EP-HUBO treats CoT evidence selection as higher-order unconstrained binary optimization over per-hypothesis pools with quality weights to improve aggregation on legal benchmarks.
-
GradeLegal: Automated Grading for German Legal Cases
Reasoning-oriented LLMs reach up to 0.91 quadratic weighted kappa agreement with experts on public law cases when given sample solutions and grading rubrics, but only 0.60 on criminal law cases.
-
Exploiting LLM-as-a-Judge Disposition on Free Text Legal QA via Prompt Optimization
Automatic prompt optimization using lenient LLM judges improves performance and transferability in legal QA evaluations compared to human design or strict judges.
-
Beyond Imperfect Alternatives with Rulemapping: A Neuro-Symbolic Case Study on Online Hate Speech
Rulemapping uses expert symbolic scaffolds to constrain LLMs, raising precision on §130(1) German hate-speech classification from 0.34-0.49 to 0.80-0.86 while preserving recall of 0.82-0.89.
-
NyayaMind- A Framework for Transparent Legal Reasoning and Judgment Prediction in the Indian Legal System
NyayaMind combines RAG retrieval with domain-specific LLMs to generate transparent, structured legal reasoning and judgment predictions for Indian court cases.
-
BenGER Platform: A Collaborative Web Platform for End-to-End Benchmarking of German Legal Tasks
BenGER integrates task creation, annotation, configurable LLM runs, and lexical/semantic/factual/judge metrics into a multi-tenant web platform for German legal benchmarking.