Pith. sign in

REVIEW 1 cited by

Legal Evalutions and Challenges of Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2411.10137 v1 pith:PGVJJUKW submitted 2024-11-15 cs.CL cs.AI

Jiaqi Wang , Huan Zhao , Zhenyuan Yang , Peng Shu , Junhao Chen , Haobo Sun , Ruixi Liang , Shixin Li
show 14 more authors
Pengcheng Shi Longjun Ma Zongjia Liu Zhengliang Liu Tianyang Zhong Yutong Zhang Chong Ma Xin Zhang Tuo Zhang Tianli Ding Yudan Ren Tianming Liu Xi Jiang Shu Zhang
This is my paper · ORCID
classification cs.CLcs.AI
keywords legalmodelsllmslanguagelargeapplyingcaseschallenges
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

In this paper, we review legal testing methods based on Large Language Models (LLMs), using the OPENAI o1 model as a case study to evaluate the performance of large models in applying legal provisions. We compare current state-of-the-art LLMs, including open-source, closed-source, and legal-specific models trained specifically for the legal domain. Systematic tests are conducted on English and Chinese legal cases, and the results are analyzed in depth. Through systematic testing of legal cases from common law systems and China, this paper explores the strengths and weaknesses of LLMs in understanding and applying legal texts, reasoning through legal issues, and predicting judgments. The experimental results highlight both the potential and limitations of LLMs in legal applications, particularly in terms of challenges related to the interpretation of legal language and the accuracy of legal reasoning. Finally, the paper provides a comprehensive analysis of the advantages and disadvantages of various types of models, offering valuable insights and references for the future application of AI in the legal field.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LaQual: An Automated Framework for LLM App Quality Evaluation

    cs.SE 2025-08 reject novelty 5.0 of 10

    LaQual automates LLM app-store quality evaluation through scenario classification, static indicator filtering, and LLM-generated dynamic metrics, with Spearman correlations of about 0.6 against human ratings.

Pith tools