Pith. sign in

REVIEW 3 major objections 5 minor 37 references

Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In a small test, GPT-4o extracts 73% of fields correctly from images of a paper lease form, and the model almost always finds the right field even when it misreads a value.

desk verdict A transparent pilot showing multimodal LLMs can read handwritten form fields, but its per-format accuracy ordering is noise-level with one image per cell. read the letter →

arxiv 2412.15260 v1 pith:5WU7B7HC submitted 2024-12-16 cs.CL cs.CVcs.MM

Hannes Westermann , Jaromir Savelka This is my paper · ORCID
classification cs.CLcs.CVcs.MM
keywords multimodallargelanguagemodelslegaldocumentsgovernmentformsaccesstojusticeinformationextractionhandwritingrecognitionGPT-4oformparsing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether a multimodal large language model—a model that can take images as well as text as input—can replace manual transcription as the bridge between paper legal documents and AI-based legal help. Using 15 images of a standard Ontario residential lease form, covering three scenarios of increasing difficulty and five capture formats, it reports that GPT-4o extracts the correct value for 73% of 14 tracked fields on average: nearly perfect from typed PDFs (98%), and progressively worse with neat modern-phone photos (74%), neat old-phone photos (69%), sloppy modern-phone photos (64%), and sloppy old-phone photos (60%). The authors read this as evidence that the image modality is usable for access-to-justice tools, with the clear caveat that image quality and uncommon names are the main failure points.

What carries the argument

The mechanism is a zero-shot extraction pipeline: a base64-encoded image of the form is sent to GPT-4o (gpt-4o-2024-08-06) with a system prompt specifying a JSON schema for 14 fields and instructing the model to extract values exactly as they appear, using '-' for missing values, with temperature set to 0. Success is scored as exact match to the gold labels, ignoring only capitalization. The design deliberately varies scenario difficulty (common versus uncommon names, missing fields) and capture format (typed PDF, neat or sloppy handwriting, modern or old phone) so that the accuracy differences across the 15 images can be traced to image quality and the model's prior expectations about names rather than to OCR-style preprocessing.

What would settle it

Take the three scenarios and five formats, generate roughly 20 independently photographed versions of each cell, run the same prompt and matching rule, and check whether the 73% average and the format ordering (98, 74, 69, 64, 60) reproduce within a few points; if the per-condition averages scatter widely or the ordering flips, the central quantitative claim is an artifact of the single-image design.

Watch

Extended reading notes

Core claim

The central claim is that a single general-purpose multimodal LLM can locate and read the relevant fields on a photographed paper form without document-specific tuning. Across the three scenarios, 89%, 71%, and 59% of fields matched the gold standard, and the errors are not random: the model nearly always finds the correct field on the page, but misreads values, especially short numeric strings such as street numbers (27% correct) and uncommon names, which it tends to correct toward more common spellings (Jame to Jane, Wane to Wayne). The authors conclude that the capability is strong enough to motivate integration into access-to-justice systems, particularly with a human in the loop to verify extracted values.

Load-bearing premise

Each per-condition accuracy figure is computed from a single photographed form (15 images total, 14 fields each), so the paper's percentages and difficulty ordering are assumed to generalize from one example per cell.

Editorial extensions

If this is right

  • A photo-based intake system could let a user photograph a paper lease and have the landlord's name, tenant names, rental address, and condominium flag extracted automatically, removing the need to type those facts into a computer.
  • For electronic documents (typed PDFs), the 98% field accuracy suggests near-term practical deployment is plausible.
  • Because the model almost always locates the right field even when it misreads it, an interface that asks the user to confirm highlighted values could catch most errors cheaply.
  • Field-level differences imply that handwritten or low-resolution numeric fields such as street number, postal code, and condominium flag should be treated as high-risk and routed to human verification.
  • The gap between modern-phone and old-phone capture (74% versus 60% on sloppy forms) implies that deployment must account for the hardware of the target population, not just model capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this result extends beyond leases, the same prompting pattern could be applied to court claim forms, benefit applications, and certificates; the paper's design suggests the main barrier is not the document type but the legibility of handwriting and the length of numeric strings.
  • The name-correction failure points toward a trade-off: a model with a strong language prior will repair messy text but also overwrite legitimate minority names, so an access-to-justice deployment should probably surface a confidence score for name fields rather than silently submitting them.
  • A direct comparison against traditional OCR plus a spelling-correcting language model would clarify whether end-to-end vision is the right technology; the paper's data predicts OCR would win on street numbers and the LLM would win on contextual fields like city and province.
  • A natural next step is a larger benchmark with repeated images per condition and confidence intervals; without it, the reported per-condition ordering (98%, 74%, 69%, 64%, 60%) is best read as suggestive rather than settled.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an initial study of using GPT-4o to extract 14 structured fields from photographs and screenshots of a standard Ontario residential tenancy agreement. Three scenarios of increasing expected difficulty (common names, less common names, and confusable/missing values) are crossed with five formats (typed PDF, neat and sloppy handwriting photographed with a modern phone, and neat and sloppy handwriting photographed with an old phone), yielding 15 images total. The overall field-level accuracy is 73%, with format accuracies of 98%, 74%, 69%, 64%, and 60% (typed HD, neat HD, neat SD, sloppy HD, sloppy SD) and scenario accuracies of 89%, 71%, and 59%. The authors conclude that the results are promising for access-to-justice applications while noting limitations such as image quality and the model's tendency to correct uncommon names to more common ones.

Significance. If the quantitative results were robust, this would be a useful contribution to an important problem: helping laypeople and self-represented litigants extract information from paper documents using multimodal LLMs. The paper's strengths are its clearly described experimental pipeline, the decision to use a real legal form, the release of code and data, and a scoring protocol (exact match modulo capitalization) that is transparent and easy to reproduce. The temperature-0 setting and the use of 14 well-defined fields make the measurements straightforward to audit. The main significance is as a proof-of-concept that multimodal LLMs can perform this extraction task at all, which the paper does demonstrate; however, the quantitative comparisons between formats and scenarios are not statistically supported.

major comments (3)
  1. [§3.2/§3.4, Tables 1–2] The experimental design uses exactly one image per scenario-format combination (15 images total). Each format-level average in Table 2 is computed from 42 field-level observations, but these observations are clustered within only three images, and the differences between adjacent formats are very small: neat-HD (0.74) versus neat-SD (0.69) corresponds to about two correct fields out of 42, and sloppy-HD (0.64) versus sloppy-SD (0.60) is even smaller. With an effective sample size of a few images per condition, these gaps are within sampling noise. The paper nevertheless draws substantive conclusions in §4.3 ("the quality of the results decreased when working with the printed versions") and §4.4 (the digital-divide discussion) from this ordering. The authors should either add repeated images per condition (e.g., multiple filled copies and multiple photographs) with cluster-aware confidence intervals, or explicitly reframe the format-comparison results as exploratory and avoid making quantitative comparative claims about image quality.
  2. [§3.2, Table 1, RQ2] The scenario variable is confounded. S2 differs from S1 in at least three ways: less common names, an additional tenant, and a missing field; S3 additionally includes names that resemble common names and multiple missing fields. Consequently, the scenario-level accuracy differences in Table 1 (89%, 71%, 59%) cannot be attributed to any single factor such as "complex data" or "missing fields" as listed in RQ2. Since RQ2 asks specifically about the effects of these factors, the current design cannot answer it. A factorial design varying one factor at a time (e.g., name frequency with all fields filled, or missing fields with common names) would be needed to support the claims in §4.2 about the causes of the performance drop.
  3. [§4.2, Table 1] The field-level interpretation in §4.2 also ignores the small denominators. For example, rental_unit_street_number has an average accuracy of 0.27, which is computed over 15 images total, and the perfect scores for rental_unit_city_town and rental_unit_province are based on the same 15 images; a single misclassification would change these estimates by roughly 7 percentage points. The claim that the model benefits from pre-training to "guess" city and province names "even if the image quality is lacking" is a plausible hypothesis but not one that can be supported by these counts. The field-level results should be presented as observations subject to high uncertainty, not as established findings about which fields are intrinsically easy or hard.
minor comments (5)
  1. [§3.2] The text says "poor lightning conditions" in items 4 and 5; "lighting" is the correct word (also appears in §4.4).
  2. [Figure 4 caption] The caption contains typos: "Referst to" should be "refers to," and the definition of T/P would be clearer as "T: target value, P: predicted value."
  3. [Figure 5 caption] The caption contains typos: "informationd" and "targer" should be "information" and "target."
  4. [§4.1] The claim that the model "had no trouble locating the information on the page" is supported only by the informal observation that erroneous outputs share some letters or numbers. A quantitative localization metric (e.g., whether the model's errors are attributable to reading a correctly identified field versus mislocating the field) would make this claim testable; as stated, it is an anecdotal inference.
  5. [§3.4] The scoring procedure says capitalization is ignored, but the paper does not state whether other formatting variations (e.g., extra spaces, punctuation, or different postal-code separators) were normalized. Clarifying this would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the reported accuracy is a direct measurement against externally created gold labels, with no fitted parameters or self-citation chain supporting the quantitative claims.

full rationale

The paper reports a direct empirical measurement: GPT-4o was prompted to extract 14 fields from 15 images of a deliberately constructed Ontario lease form, and the outputs were compared against gold-standard values manually entered when the forms were created (Section 3.4: 'we compared the returned data to the gold standard labels that were used to create the forms'). No parameter is fitted, no equation is derived, and no target quantity is defined in terms of the model's output; the gold labels are external to the model. The related-work section contains many self-citations by the authors, but none is load-bearing for the extraction experiment: the accuracy numbers would stand identically if those citations were replaced by external work or removed. The closest thing to a circularity concern is that the authors chose realistic city and street names and then note the model may 'guess' them from pretraining; this is an interpretive claim about model behavior, not a reduction of the result to the input. The small sample size (one image per scenario-format cell) and lack of confidence intervals are a statistical-power limitation, which the paper itself flags as an 'initial investigation' and a plan for 'more large-scale studies' (Section 5); it does not make the measurement circular. Accordingly, no circular step can be quoted or exhibited, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper does not derive anything; it reports an empirical measurement. The main implicit assumptions are that the hand-built dataset is representative, that exact-match scoring is a fair metric, and that the model's use of pre-trained knowledge does not distort accuracy. No free parameters are fitted, and no new entities are postulated.

assumptions (3)
  • domain assumption The 15 images and 14 fields are representative of real-world documents and enough to support conclusions about feasibility and limitations.
    The paper draws general conclusions about image quality and handwriting effects from one image per condition (Section 3.2).
  • domain assumption Exact string match after capitalization normalization is an appropriate accuracy measure for all fields.
    Some fields have restricted formats, but names may have multiple valid transcriptions; partial credit and human agreement are not considered (Section 3.4).
  • domain assumption The model's pre-trained knowledge (e.g., recognizing real city names) does not distort the reported extraction accuracy.
    The authors note the model may 'guess' known values such as cities, which could overstate reading ability (Section 4.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice." pith.science (2026). https://pith.science/paper/5WU7B7HC

@misc{pith2026241215260,
  author       = {Pith},
  title        = {Pith review of: Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5WU7B7HC}},
  note         = {Machine review of arXiv:2412.15260}
}
read the original abstract

Interacting with the legal system and the government requires the assembly and analysis of various pieces of information that can be spread across different (paper) documents, such as forms, certificates and contracts (e.g. leases). This information is required in order to understand one's legal rights, as well as to fill out forms to file claims in court or obtain government benefits. However, finding the right information, locating the correct forms and filling them out can be challenging for laypeople. Large language models (LLMs) have emerged as a powerful technology that has the potential to address this gap, but still rely on the user to provide the correct information, which may be challenging and error-prone if the information is only available in complex paper documents. We present an investigation into utilizing multi-modal LLMs to analyze images of handwritten paper forms, in order to automatically extract relevant information in a structured format. Our initial results are promising, but reveal some limitations (e.g., when the image quality is low). Our work demonstrates the potential of integrating multi-modal LLMs to support laypeople and self-represented litigants in finding and assembling relevant information.

Figures

Figures reproduced from arXiv: 2412.15260 by the authors.

Figure 1
Figure 1. Excerpts from filled-out form in the different formats used for experiments, Scenario 2. scenarios, we created various images (see [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Diagram showing the experimental design. Overall, there are 3 scenarios and 5 variants of each in our experimental dataset, bringing the total amount of data to 15 samples, each with 14 possible fields. The scenarios and formats were chosen to simulate real-world situations, where users may not have access to high-quality camera equipment, and forms may not be filled out in the expected ways. 3.3. Model & Prompt To … view at source ↗
Figure 3
Figure 3. Accuracy heatmap for different scenarios and formats. 4.2. Differences between fields The difficulty of accurately capturing the data varied substantially between the different fields. Certain fields (such as town and province) were perfectly captured across the formats. Looking at the data, this is likely due to the fact that the province is predefined and mentioned in multiple places on the form. The cities were p… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Excerpts from data for street numbers. T referst to the target value, while P shows the value extracted by the model. Other values were less likely to be extracted correctly. For example, street number was only correct 27% of the time [PITH_FULL_IMAGE:figures/full_fig…
Figure 5
Figure 5. Figure 5: Excerpts from selected samples where the model was able to correctly extract the informationd despite poor data quality. T refers to the targer value, while P refers to the prediction of the model. 4.4. Analysis Overall, it appears there is considerable potential for t…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages

  1. [1]

    Currie, The legal problems of everyday life, in: Access to justice, Emerald Group Publishing Limited, 2009

    A. Currie, The legal problems of everyday life, in: Access to justice, Emerald Group Publishing Limited, 2009

    work page 2009

  2. [2]

    Semple, The cost of seeking civil justice in canada, Can

    N. Semple, The cost of seeking civil justice in canada, Can. B. Rev. 93 (2015) 639

    work page 2015

  3. [3]

    Justice For All - Final Report, Technical Report, The Task Force on Justice, New York: Center on International Cooperation, 2019

    work page 2019

  4. [4]

    Macfarlane, The national self-represented litigants project: Identifying and meeting the needs of self-represented litigants final report (2013)

    J. Macfarlane, The national self-represented litigants project: Identifying and meeting the needs of self-represented litigants final report (2013)

    work page 2013

  5. [5]

    Branting, C

    K. Branting, C. Balhana, C. Pfeifer, J. S. Aberdeen, B. Brown, Judges are from mars, pro se litigants are from venus: Predicting decisions from lay text., in: JURIX, 2020, pp. 215–218

    work page 2020

  6. [6]

    C. R. Sunstein, Sludge and ordeals, Duke LJ 68 (2018) 1843

    work page 2018

  7. [7]

    Westermann, K

    H. Westermann, K. Benyekhlef, JusticeBot: A Methodology for Building Augmented Intelligence Tools for Laypeople to Increase Access to Justice, in: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL ’23, Association for Computing Machinery, New York, NY, USA, 2023, pp. 351–360. URL: https://doi.org/10.1145/3594...

    arXiv 2023

  8. [8]

    Westermann, et al., Bridging the gap: Mapping layperson narratives to legal issues with language models, in: ASAIL 2023, 2023

    H. Westermann, et al., Bridging the gap: Mapping layperson narratives to legal issues with language models, in: ASAIL 2023, 2023

    work page 2023

Show all 37 references
  1. [9]

    Steenhuis, D

    Q. Steenhuis, D. Colarusso, B. Willey, Weaving pathways for justice with gpt: Llm-driven automated drafting of interactive legal applications, 2023. arXiv:2312.09198

    2023 arXiv

  2. [10]

    Westermann, Dallma: Semi-structured legal reasoning and drafting with large language models, 2nd Workshop on Generative AI and Law (2024)

    H. Westermann, Dallma: Semi-structured legal reasoning and drafting with large language models, 2nd Workshop on Generative AI and Law (2024)

    2024

  3. [11]

    Paquin, F

    L.-C. Paquin, F. Blanchard, C. Thomasset, Loge–expert: from a legal expert system to an information system for non-lawyers, in: ICAIL 1991, 1991, pp. 254–259

    1991

  4. [12]

    L. K. Branting, Advisory systems for pro se litigants, in: Proceedings of the 8th international conference on Artificial intelligence and law, 2001, pp. 139–146

    2001

  5. [13]

    Zeleznikow, Using web-based legal decision support systems to improve access to justice, Information & Communications Technology Law 11 (2002) 15–33

    J. Zeleznikow, Using web-based legal decision support systems to improve access to justice, Information & Communications Technology Law 11 (2002) 15–33

    2002

  6. [14]

    Westermann, Using artificial intelligence to increase access to justice, Ph.D

    H. Westermann, Using artificial intelligence to increase access to justice, Ph.D. thesis, Université de Montréal, 2023. URL: https://papyrus.bib.umontreal.ca/xmlui/handle/1866/32168

    2023

  7. [15]

    Westermann, S

    H. Westermann, S. Meeùs, M. Godet, A. Troussel, J. Tan, J. Savelka, K. Benyekhlef, Bridging the gap: Mapping layperson narratives to legal issues with language models, in: ASAIL/LegalAIIA@ ICAIL, 2023

    2023

  8. [16]

    Steenhuis, B

    Q. Steenhuis, B. Willey, D. Colarusso, Beyond readability with ratemypdf: A combined rule-based and machine learning approach to improving court forms, in: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, 2023, pp. 287–296

    2023

  9. [17]

    Branting, S

    K. Branting, S. McLeod, Narrative-driven case elicitation., in: AI4AJ@ ICAIL, 2023

    2023

  10. [18]

    arXiv:2303.08774

    OpenAI, GPT-4 technical report, 2023. arXiv:2303.08774

    2023 arXiv

  11. [19]

    F. Yu, L. Quartey, F. Schilder, Legal prompting: Teaching a language model to think like a lawyer,

  12. [20]

    D. M. Katz, et al., GPT-4 passes the bar exam, SSRN 4389233 (2023)

    2023

  13. [21]

    D. M. Katz, M. J. Bommarito, S. Gao, P. Arredondo, Gpt-4 passes the bar exam, Philosophical Transactions of the Royal Society A 382 (2024) 20230254

    2024

  14. [22]

    Blair-Stanek, N

    A. Blair-Stanek, N. Holzenberger, B. Van Durme, Can GPT-3 perform statutory reasoning?, Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law (2023) 22–31

    2023

  15. [23]

    Nguyen, R

    H.-T. Nguyen, R. Goebel, F. Toni, K. Stathis, K. Satoh, Black-box analysis: GPTs across time in legal textual entailment task, arXiv preprint arXiv:2309.05501 (2023)

    2023 arXiv

  16. [24]

    Hamilton, Blind judgement: Agent-based supreme court modelling with gpt, arXiv preprint arXiv:2301.05327 (2023)

    S. Hamilton, Blind judgement: Agent-based supreme court modelling with gpt, arXiv preprint arXiv:2301.05327 (2023)

    2023 arXiv

  17. [25]

    Savelka, K

    J. Savelka, K. D. Ashley, M. A. Gray, H. Westermann, H. Xu, Can GPT-4 support analysis of textual data in tasks requiring highly specialized domain expertise?, Proceedings of ASAIL’23 (2023)

    2023

  18. [26]

    M. A. Gray, J. Savelka, W. M. Oliver, K. D. Ashley, Empirical legal analysis simplified: reducing com- plexity through automatic identification and evaluation of legally relevant factors, Philosophical Transactions of the Royal Society A 382 (2024) 20230155

    2024

  19. [27]

    Savelka, Unlocking practical applications in legal domain: Evaluation of GPT for zero-shot semantic annotation of legal texts, arXiv preprint arXiv:2305.04417 (2023)

    J. Savelka, Unlocking practical applications in legal domain: Evaluation of GPT for zero-shot semantic annotation of legal texts, arXiv preprint arXiv:2305.04417 (2023)

    2023 arXiv

  20. [28]

    Savelka, K

    J. Savelka, K. D. Ashley, The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts, Frontiers in Artificial Intelligence 6 (2023) 1279794

    2023

  21. [29]

    Savelka, K

    J. Savelka, K. D. Ashley, M. A. Gray, H. Westermann, H. Xu, Explaining legal concepts with augmented large language models (GPT-4), arXiv preprint arXiv:2306.09525 (2023)

    2023 arXiv

  22. [30]

    Westermann, J

    H. Westermann, J. Savelka, K. Benyekhlef, LLMediator: Gpt-4 assisted online dispute resolution, Artificial Intelligence for Access to Justice (AI4AJ 2023) (2023)

    2023

  23. [31]

    J. Tan, H. Westermann, N. R. Pottanigari, J. Šavelka, S. Meeùs, M. Godet, K. Benyekhlef, Robots in the middle: Evaluating llms in dispute resolution, 2024. URL: https://arxiv.org/abs/2410.07053. arXiv:2410.07053

    2024 arXiv

  24. [32]

    J. Tan, H. Westermann, K. Benyekhlef, ChatGPT as an artificial lawyer?, Artificial Intelligence for Access to Justice (AI4AJ 2023) (2023)

    2023

  25. [33]

    Janatian, H

    S. Janatian, H. Westermann, J. Tan, J. Savelka, K. Benyekhlef, From text to structure: Using large language models to support the development of legal expert systems, in: Legal Knowledge and Information Systems, IOS Press, 2023, pp. 167–176

    2023

  26. [34]

    Steenhuis, H

    Q. Steenhuis, H. Westermann, Getting in the door: Streamlining intake in civil legal services with large language models, 2024. URL: https://arxiv.org/abs/2410.03762. arXiv:2410.03762

    2024 arXiv

  27. [35]

    Drápal, H

    J. Drápal, H. Westermann, J. Savelka, Using large language models to support thematic analysis in empirical legal studies, arXiv preprint arXiv:2310.18729 (2023)

    2023 arXiv

  28. [36]

    M. Gray, J. Savelka, W. Oliver, K. Ashley, Using llms to discover legal factors, arXiv preprint arXiv:2410.07504 (2024)

    2024 arXiv

  29. [2022]

    doi:10.48550/ARXIV.2212.01326

    URL: https://arxiv.org/abs/2212.01326. doi:10.48550/ARXIV.2212.01326

    doi arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.