Pith. sign in

REVIEW 5 major objections 5 minor 20 references

Logical Lease Litigation: Prolog and LLMs for Rental Law Compliance in New York

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LogicLease claims that separating LLM-based fact extraction from hand-coded Prolog legal reasoning makes New York landlord-tenant compliance analysis fully transparent and, on ten test cases, 100 percent accurate.

desk verdict Sane architecture, honest writing, but the 100% accuracy claim is a closed loop and the anti-hallucination claim is unguarded at the extraction stage. read the letter →

arxiv 2502.09204 v1 pith:YHHQ6MUO submitted 2025-02-13 cs.AI cs.LO

Sanskar Sehgal , Yanhong A. Liu This is my paper · ORCID
classification cs.AIcs.LO
keywords legalreasoningProloglargelanguagemodelslandlord-tenantlawNewYorkinformationextractioncompliancecheckingdefeasiblelogic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LogicLease is a system for New York landlord-tenant compliance that splits the job in two: an out-of-the-box LLM turns a natural-language case description into structured attribute-value pairs, and hand-coded Prolog rules use those pairs to decide whether the landlord's action is lawful and to print the relevant legal requirements. The paper claims the separation yields 100 percent accuracy on ten test cases, an average processing time of 2.57 seconds, and step-by-step citations of applicable law, while avoiding the hallucinations that plague end-to-end legal chatbots. A sympathetic reader would care because if true, it gives tenants who face eviction without counsel a transparent, fast, checkable route to see which rules apply to their situation, in language they can act on.

What carries the argument

The load-bearing architecture is the hand-off between two formally different tools. The LLM is confined to information extraction: it reads plain-English case text and emits a small dictionary of attribute-value pairs, with no legal conclusion attached. Those values are fed as arguments to Prolog predicates, where a hand-coded knowledge base of roughly 500 lines encodes lease validity, rent stabilization, eviction, habitability, and related rules drawn from the Tenants' Rights Guide. A defeasible-logic layer lets one rule override another, so e.g. a tenant in a protected category can defeat an owner-occupancy eviction ground. Prolog's deterministic evaluation is what makes the final list of cited laws and the verdict reproducible and inspectable.

What would settle it

Take 50 landlord-tenant disputes from published New York court opinions, especially ones turning on statutes or defenses not mentioned in the Tenants' Rights Guide, run LogicLease on anonymized versions, and compare its verdicts with the courts' outcomes. Any mismatch on a case whose controlling law is absent from the guide would show that the knowledge base is incomplete, and the 100 percent accuracy claim would not extend past the original ten test cases.

Watch

Extended reading notes

Core claim

The central claim is that legal compliance analysis for rental disputes can be made transparent and reliable by enforcing a clean separation: the LLM is used only to extract facts, never to reason about law, and a manually constructed Prolog knowledge base built from the state Tenants' Rights Guide performs all legal reasoning. Given a case description, LogicLease's LLM returns attribute-value pairs such as eviction cause, whether a court ruling has been issued, and whether the tenant is in a protected category; these instantiate Prolog predicates such as eviction(EvictionCause, CourtRuling, Executioner, TenantCategory). The Prolog engine evaluates the predicates, applies defeasible logic so that protective rules can override general grounds for eviction, and outputs the numbered legal statements that apply plus a final judgment. The paper reports that this pipeline answered all ten test cases correctly, averaged 2.57 seconds per case, and beat general-purpose LLMs, which it says reached the correct conclusion in only seven of the ten cases.

Load-bearing premise

The load-bearing premise is that the hand-coded Prolog rules, based on the state Tenants' Rights Guide, are a complete and correct formalization of New York landlord-tenant law for every case the system will encounter; if that premise fails, the confident and transparent answers will still be wrong.

Editorial extensions

If this is right

  • A user who submits a lease dispute in plain English receives not only a verdict but the numbered legal statements that drive it, so the reasoning can be checked line by line.
  • Because reasoning is separated from extraction, any error has a single component to blame: either the LLM missed or misread a fact, or a Prolog rule does not match the law, narrowing the debugging surface.
  • The Prolog computation itself is nearly free, so the system's perceived speed is controlled by LLM API latency; caching and shorter inputs would make it faster.
  • The paper's comparison suggests that general-purpose LLMs, used end to end, can be wrong on cases where logic-based rule application gets them right, which is the practical case for the hybrid design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the 'no hallucinations' claim should be read narrowly: the LLM can still mis-extract a fact, and if it does, the Prolog rules will reason confidently from the wrong premise. The architecture moves hallucinations from legal reasoning to the extraction stage rather than eliminating the failure mode.
  • Editorial inference: the 100 percent figure is measured on ten self-created cases, so a natural stress test is to run the same pipeline against a published corpus of New York landlord-tenant decisions; accuracy would likely be below 100 percent wherever the Tenants' Rights Guide is silent.
  • Editorial inference: because the law is encoded as rules, the same design could be ported to other states by swapping the Prolog knowledge base, provided the attribute vocabulary and the LLM extraction prompt are aligned with the new jurisdiction's requirements.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. LogicLease is a system for landlord-tenant legal compliance analysis in New York State. It uses an out-of-the-box LLM to extract attribute-value pairs from natural-language case descriptions and a hand-coded SWI-Prolog knowledge base, based on the New York Attorney General's Tenants' Rights Guide, to reason about lease validity, rent stabilization, eviction, and habitability. The paper claims that LogicLease achieves 100% accuracy on a manually compiled dataset of 10 cases, an average processing time of 2.57 seconds, and that separating information extraction from legal reasoning lets it avoid hallucinations while providing transparent, law-cited output. A comparison with ChatGPT and Gemini is reported in which those LLMs fail on three of ten cases.

Significance. The proposed separation of concerns--LLM for extraction and Prolog for legal reasoning--is a reasonable architecture for transparent legal analytics, and the paper provides a concrete implementation, a full running example, and explicit runtime measurements. If validated with an independently audited legal knowledge base and a substantially larger, externally verified evaluation set, the system would be a useful demonstration of combining logic programming with LLMs. However, the evaluation presented here is far too weak to support the accuracy and hallucination claims, and the legal knowledge base is not audited against primary sources. As it stands, the contribution is chiefly a prototype description rather than an established result.

major comments (5)
  1. [Abstract; Section 4.2] The central claim 'achieving 100% accuracy' is not supported by the evaluation design. The dataset consists of 10 manually compiled cases, and the ground-truth verdicts were derived from the same New York Attorney General's Tenants' Rights Guide [12] used to code the Prolog knowledge base, with informal online forums such as Reddit as additional checks. This is a closed evaluation loop: the system is essentially tested against its own source of rules rather than against an independent legal ground truth. There is no legal expert review, no inter-annotator agreement, no report of repeated LLM runs or variance, and no error analysis. The unconditional 100% claim in the abstract is therefore not justified; at most the paper can claim agreement with the authors' labels on those 10 cases.
  2. [Section 3, NLP; Listing 1; Section 4.2] The paper's claim that LogicLease avoids hallucinations is not supported by the architecture. The LLM extraction stage runs out-of-the-box with no verification, consistency check, or confidence threshold, and the Prolog engine treats every extracted attribute-value pair as a fact. If the LLM omits a legally relevant attribute or fabricates one (e.g., a hallucinated CourtRuling: true), the Prolog rule in Listing 1 will emit a transparent, well-cited, but legally wrong verdict. Section 4.2 only states that the LLM 'functioned effectively' on ten cases; it gives no extraction precision/recall per attribute, no failure taxonomy, and no adversarial or perturbation tests. The hallucination risk is thus relocated to the extraction stage rather than eliminated.
  3. [Section 3, Prolog Knowledge Base; Section 4.1] The knowledge base is claimed to contain 'an exhaustive set of rules' and to cover 'the full New York State landlord-tenant legal framework,' but the only legal source cited is the Attorney General's guide [12]. No statutes (such as the Real Property Actions and Proceedings Law), regulations, or case law are cited, and no independent legal audit is described. The one rule shown in Listing 1 references auxiliary predicates (eviction_legal, eviction_warrant_execution, overrides) whose definitions are not given, so even the illustrated rule cannot be checked by the reader. A completeness claim of this magnitude requires either the full Prolog code with statutory annotations or a systematic audit against primary New York legal sources.
  4. [Section 4.4] The robustness section does not evaluate the robustness of LogicLease. It compares the system with ChatGPT and Gemini on the running example and reports that these LLMs missed the correct conclusion in three of ten cases, but it does not identify the three cases, list the prompts, report multiple runs, or analyze failure modes. There are no experiments varying case phrasing, injecting missing or contradictory attributes, or using out-of-distribution scenarios. The comparison also does not measure LogicLease's own behavior under such perturbations, so the section's title and the 'robustness' claim are not supported.
  5. [Table 1; Section 4.3] The efficiency table contains an arithmetic inconsistency: the average of the ten Prolog running times listed in Table 1 is approximately 0.00052 seconds, not the reported 0.00011 seconds. Additionally, the total time is dominated by the LLM API call, so the 2.572-second average is not a stable performance measure unless extraction quality is simultaneously reported. This should be corrected and clarified.
minor comments (5)
  1. [Section 4.4] The text refers to Figures 4 and 5 but does not describe their contents; the reader cannot tell what the figures show or how the outputs were produced.
  2. [References] Reference [15] duplicates reference [14] with the same title and DOI; one citation should be removed or corrected.
  3. [Section 4.3] The paper reports a 'strong positive correlation (approx 68%)' between query length and Llama response time without giving the correlation coefficient, the number of data points, or a scatterplot.
  4. [Section 1] The statistic '122.8 million households in the United States are renters' [2] appears to conflate households with individuals; the cited Pew Research source should be checked.
  5. [Section 4.1] The paper should state explicitly which model and version of LLaMA was used through the llamaapi service, since the service is not a fixed model and reproducibility depends on this detail.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: LogicLease's Prolog reasoning is hand-coded, and its accuracy claim, while based on a small in-house validation set, is not forced by construction.

full rationale

The paper's derivation chain is not circular. The Prolog knowledge base is manually coded from the New York Renters' Rights Handbook, and the LLM extracts attribute-value pairs that are passed as arguments to Prolog predicates. There are no fitted parameters, no learned weights, and no quantity defined in terms of the target output. The claimed 100% accuracy is an empirical result on ten cases, which could in principle have failed due to coding errors or extraction mistakes; it is not guaranteed by the construction of the system. The human-in-the-loop evaluation uses the same legal handbook and online forums as the source of ground truth, which is a limitation on external validity, but this does not make the accuracy result equivalent to its inputs by definition. The paper cites no self-authored uniqueness results or ansatz-smuggling prior work, and the anti-hallucination claim is an architectural argument about separating extraction from reasoning, not a circular reduction. Overall, no specific circular step can be exhibited from the text.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted parameters are present; the system is rule-based. The main burdens are domain assumptions about LLM extraction reliability, completeness of the hand-coded legal rules, representativeness of the 10-case test set, and adequacy of human-in-the-loop review. No new entities are introduced.

assumptions (5)
  • domain assumption The LLM reliably extracts all legally relevant attribute-value pairs from natural language case descriptions.
    Section 3 (NLP component) and Section 4.2 assume the extracted dictionary is complete and correct; no failure analysis is reported.
  • domain assumption The Prolog knowledge base hand-coded from the New York Tenants' Rights Guide is an exhaustive and correct formalization of NY landlord-tenant law.
    Section 3, Prolog Knowledge Base calls the rules 'exhaustive'; Section 4 claims a full framework, but only one handbook is cited and no statutes or case law are given.
  • ad hoc to paper The 10 manually compiled test cases are representative of real landlord-tenant disputes and sufficient to validate 100% accuracy.
    Section 4.2 says the dataset is 'currently small' and was constrained by API credits, with condensed documents.
  • ad hoc to paper Human-in-the-loop review using handbooks and online legal forums provides valid ground truth for legal correctness.
    Section 4.2; Reddit is not a reliable legal authority and no court decisions or statutes were used to verify outputs.
  • domain assumption Defeasible logic in Prolog correctly resolves conflicts between legal principles.
    Section 4.1 says defeasible logic was incorporated, but no examples of conflict resolution or formal verification are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Logical Lease Litigation: Prolog and LLMs for Rental Law Compliance in New York." pith.science (2026). https://pith.science/paper/YHHQ6MUO

@misc{pith2026250209204,
  author       = {Pith},
  title        = {Pith review of: Logical Lease Litigation: Prolog and LLMs for Rental Law Compliance in New York},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YHHQ6MUO}},
  note         = {Machine review of arXiv:2502.09204}
}
read the original abstract

Legal cases require careful logical reasoning following the laws, whereas interactions with non-technical users must be in natural language. As an application combining logical reasoning using Prolog and natural language processing using large language models (LLMs), this paper presents a novel approach and system, LogicLease, to automate the analysis of landlord-tenant legal cases in the state of New York. LogicLease determines compliance with relevant legal requirements by analyzing case descriptions and citing all relevant laws. It leverages LLMs for information extraction and Prolog for legal reasoning. By separating information extraction from legal reasoning, LogicLease achieves greater transparency and control over the legal logic applied to each case. We evaluate the accuracy, efficiency, and robustness of LogicLease through a series of tests, achieving 100% accuracy and an average processing time of 2.57 seconds. LogicLease presents advantages over state-of-the-art LLM-based legal analysis systems by providing clear, step-by-step reasoning, citing specific laws, and distinguishing itself by its ability to avoid hallucinations -- a common issue in LLMs.

Figures

Figures reproduced from arXiv: 2502.09204 by the authors.

Figure 1
Figure 1. LogicLease front-end interface As shown in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Design diagram of LogicLease To illustrate the workflow of LogicLease and to facilitate a deeper understanding of the system, we provide an overview of the input processing, attribute-value pair extraction using an LLM, compliance checking with the Prolog backend system, and the final output. User input of case in natural language. The input for a case is taken in natural language. An example is the following: In a … view at source ↗
Figure 3
Figure 3. Total Time Taken for Each Test Case case. According to New York law, the situation clearly favors the tenant if they live in a rent-stabilized apartment and belong to a protected category. For our test dataset, these LLMs did not reach the correct conclusion in three out of ten cases. This underscores the importance of our system and highlights the unreliability of LLMs for this application [PITH_FULL_IMAGE:figures… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: ChatGPT output 5 Related work and conclusion Existing systems for legal analysis employ various techniques, including text classification, machine learning, and rule-based reasoning [1, 9, 6, 7, 8]. However, these approaches often have limitations in handling nuanced l…
Figure 5
Figure 5. Figure 5: Gemini output accuracy. However, these systems often operate as black boxes, raising concerns about transparency and fairness. They may also be limited by data quality and biases. Rule-based legal reasoning systems like PROLEG [14, 15] offer support for judges in civil…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 18 canonical work pages

  1. [12]

    https://ag.ny.gov/sites/default/ files/tenants_rights.pdf

    New Y ork State Attorney General (2024): T enants’ Rights Guide. https://ag.ny.gov/sites/default/ files/tenants_rights.pdf

    work page 2024

  2. [1]

    PyEuroVoc: A Tool for Multilingual Legal Document Classification with EuroVoc Descriptors

    Andrei-Marius Avram, V asile Pais & Dan Tufis (2021): PyEuroV oc: A T ool for Multilingual Legal Document Classification with EuroV oc Descriptors. ArXiv preprint arXiv:2108.01139

    work page Pith review arXiv 2021

  3. [2]

    https://www.pewresearch.org/short-reads/2021/08/02/ as-national-eviction-ban-expires-a-look-at-who-rent s-and-who-owns-in-the-u-s/

    Drew Desilver (2021): As national eviction ban expires, a look at who rents and who owns in the US . https://www.pewresearch.org/short-reads/2021/08/02/ as-national-eviction-ban-expires-a-look-at-who-rent s-and-who-owns-in-the-u-s/

    work page 2021

  4. [3]

    Large language model

    Google (2024): Gemini Large Language Model . Large language model. Available at https://gemini. google.com/app. 68 Logical Lease Litigation

    work page 2024

  5. [4]

    Proceedings of the National Academy of Sciences 120(41), doi:10.1073/pnas.2305860120

    Nick Graetz, Carl Gershenson, Peter Hepburn, Sonya R Por ter, Danielle H Sandler & Matthew Desmond (2023): A comprehensive demographic profile of the US evicted popula tion. Proceedings of the National Academy of Sciences 120(41), doi:10.1073/pnas.2305860120

    work page doi:10.1073/pnas.2305860120 2023

  6. [5]

    https://evictionlab.org/ us-eviction-filing-patterns-2021/

    Peter Hepburn, Olivia Jin, Joe Fish, Emily Lemmerman, An ne Kat Alexander & Matthew Desmond (2022): Preliminary analysis: Eviction filing patterns in 2021 . https://evictionlab.org/ us-eviction-filing-patterns-2021/

    work page 2022

  7. [6]

    https://kirasystems.com/

    KiraSystems (2015): Kira. https://kirasystems.com/

    work page 2015

  8. [7]

    https://www.lawgeex.com

    LawGeex (2014): LawGeex. https://www.lawgeex.com

    work page 2014

Show all 20 references
  1. [8]

    https://lexmachina.com/

    Lemley Mark, Gregory George & Walker Joshua (2006): Lex Machina. https://lexmachina.com/

    2006

  2. [9]

    https://github.com/LexPredict/lexpredict-lexnlp

    LexPredict (2024): LexNLP by LexPredict. https://github.com/LexPredict/lexpredict-lexnlp

    2024

  3. [10]

    https://github.com/llamaapi/ llamaapi-python

    llamaapi, Reis, Eduardo & Hu, Jesse (2023): llamaapi-python. https://github.com/llamaapi/ llamaapi-python

    2023

  4. [11]

    https://swish

    Jason Morris (2024): Rules as Code Demonstration in SWISH and s(CASP) . https://swish. swi-prolog.org/p/r34v2.swinb. Accessed Mar. 17, 2024

    2024

  5. [13]

    Large language model

    OpenAI (2023): ChatGPT (Mar 14 version) . Large language model. Available at https://chat.openai. com/chat

    2023

  6. [15]

    In Takashi Onada, Daisuke Bekki & Elin McCready, editors: New Frontiers in Artificial Intelligence , Springer Berlin Heidelberg, Berlin, Heidelberg, pp

    Ken Satoh, Kento Asai, Takamune Kogawa, Masahiro Kubot a, Megumi Nakamura, Y oshiaki Nishigai, Kei Shirakawa & Chiaki Takano (2011): PROLEG: An Implementation of the Presupposed Ultimate Fact Theory of Japanese Civil Code by PROLOG T echnology . In Takashi Onada, Daisuke Bekki...

    2011 doi

  7. [16]

    https://www

    Heidi Schultheis & Caitlin Rooney (2019): A right to counsel is a right to a fighting chance . https://www. americanprogress.org/article/right-counsel-right-fighting-chance/

    2019

  8. [17]

    https://www.swi-prolog.org

    (2024): SWI Prolog. https://www.swi-prolog.org. Accessed Mar. 17, 2024

    2024

  9. [18]

    (2023): LlaMa: Open and efficient foun- dation language models, doi:10.48550/arXiv.2302.13971

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ ee Lacroix, Baptiste Rozi` ere, Naman Goyal, Eric Hambro, Faisal Azhar e t al. (2023): LlaMa: Open and efficient foun- dation language models, doi:10.48550/arXiv.2302.13971. ArXiv prepri...

    2023 doi arXiv

  10. [19]

    In: Logic Programming: 25th International Conference, ICLP 2009, Pasadena, CA, USA, July 14-17, 2009

    Hui Wan, Benjamin Grosof, Michael Kifer, Paul Fodor & Se nlin Liang (2009): Logic programming with defaults and argumentation theories . In: Logic Programming: 25th International Conference, ICLP 2009, Pasadena, CA, USA, July 14-17, 2009. Proceedings 25 , Springer, pp. 432–448...

    2009

  11. [20]

    Semantic Web 6(1), pp

    Hui Wan, Michael Kifer & Benjamin Grosof (2015): Defeasibility in answer set programs with defaults and argumentation rules. Semantic Web 6(1), pp. 81–98, doi: 10.3233/SW-140140

    2015 doi

  12. [21]

    Theory and Practice of Logic Programming 12(1-2), pp

    Jan Wielemaker, Tom Schrijvers, Markus Triska & Torbj¨orn Lager (2012): SWI-Prolog. Theory and Practice of Logic Programming 12(1-2), pp. 67–96, doi: 10.1017/S1471068411000494

    2012 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.