A pilot study finds GPT-4o extracts 73% of fields from photos of a lease form, with accuracy dropping from 98% on typed copies to 60% on low-quality handwritten photos.
Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
The evolution of Generative Pre-trained Transformer (GPT) models has led to significant advancements in various natural language processing applications, particularly in legal textual entailment. We present an analysis of GPT-3.5 (ChatGPT) and GPT-4 performances on COLIEE Task 4 dataset, a prominent benchmark in this domain. The study encompasses data from Heisei 18 (2006) to Reiwa 3 (2021), exploring the models' abilities to discern entailment relationships within Japanese statute law across different periods. Our preliminary experimental results unveil intriguing insights into the models' strengths and weaknesses in handling legal textual entailment tasks, as well as the patterns observed in model performance. In the context of proprietary models with undisclosed architectures and weights, black-box analysis becomes crucial for evaluating their capabilities. We discuss the influence of training data distribution and the implications on the models' generalizability. This analysis serves as a foundation for future research, aiming to optimize GPT-based models and enable their successful adoption in legal information extraction and entailment applications.
citation-role summary
background 1
citation-polarity summary
fields
cs.CL 1years
2024 1verdicts
CONDITIONAL 1roles
background 1polarities
unclear 1representative citing papers
citing papers explorer
-
Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice
A pilot study finds GPT-4o extracts 73% of fields from photos of a lease form, with accuracy dropping from 98% on typed copies to 60% on low-quality handwritten photos.