Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art

T0 review · 5 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models can judge patent novelty by comparing claims with cited prior art, and the best configuration outperforms a human benchmark on a new examination-based dataset.

desk verdict Useful dataset idea, but the evaluation is confounded: the test set keeps a claim-length/provenance shortcut and the claim-only baseline beats the claim–cited-text condition, so the headline empirical claims are not supported as written. read the letter →

arxiv 2502.06316 v1 pith:KSMESKG6 submitted 2025-02-10 cs.CL

Hayato Ikoma , Teruko Mitamura This is my paper · ORCID
classification cs.CL
keywords patentnoveltylargelanguagemodelspriorartcomparisonclaimanalysisevaluationdatasetexplanationgenerationexamination
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models can judge whether a patent claim is novel by comparing it with the prior art that a patent examiner actually cited in a first rejection. The authors build the first dataset for this task from real patent examination records: a claim labeled Non-Novel is an original claim that drew a lack-of-novelty rejection, and a claim labeled Novel is the amended claim from the granted patent, each paired with the same cited paragraphs. They find that classification models stay near chance when both claim and cited text are supplied, while large generative models reach meaningful accuracy; Llama3 70B with an explanation-before-prediction prompt and few-shot examples surpasses the human benchmark on the test set. The model's explanations are mostly faithful to the inputs and often mirror the examiner's stated reasoning, which is why the authors conclude that LLMs can assist in patent evaluation even if they cannot replace examiners.

What carries the argument

The central mechanism is the Claim-Cited Texts input format, which pairs a patent claim with the specific paragraphs that the examiner cited in the first Non-Final Rejection, so the model sees exactly the evidence the examiner used. The second mechanism is the prompt, revised from MPEP § 2131 (the examination manual's anticipation rule), instructing the model to check whether each and every claim element appears expressly or inherently in the cited text; the Explain-Predict variant forces the model to state its reasoning before outputting the label. This turns the open-ended legal question into a constrained element-matching task, and it is what lets the best model both predict and explain novelty.

What would settle it

Have independent patent attorneys re-annotate a random sample of the claim–cited-text pairs under the MPEP § 2131 standard, blind to whether the claim was rejected or amended; if their labels match the recorded labels about as often as Llama3 70B does, then the reported accuracy is measuring the label source rather than true novelty judgment.

Watch

Extended reading notes

Core claim

The central discovery is that the correspondence test from patent law—whether each and every element of a claim is found, expressly or inherently, in a single prior-art reference—is tractable for large generative models but not for standard classification models. On the claim-cited-text condition, Llama3 70B reaches 0.624 accuracy using a few-shot Explain-Predict prompt, above the single-author human benchmark of 0.600, while Longformer and classification-headed Llama models hover near 0.5. In the qualitative sample, explanations for correct predictions track the actual non-final rejection documents, no hallucinations were found, and errors cluster on implied disclosures and hyponym/hypernym relationships. The paper also reports a systematic 'Novel' bias in generative models on the paired-input condition, which it interprets as the models detecting differences between inputs while struggling to assess content similarity.

Load-bearing premise

The load-bearing premise is that the recorded labels reflect true novelty: an original claim that drew a first non-final rejection really is non-novel, and an amended claim from a granted patent really is novel, judged against the same cited paragraphs; if examiner judgment, arguments, or unrelated amendments influenced those outcomes, the labels contain noise the models are simply learning.

Editorial extensions

If this is right

  • Classification models with a linear classification head do not capture the claim–cited-text correspondence; adding cited text can push their accuracy toward chance, suggesting the text acts as noise for them.
  • Large generative models under few-shot Explanation-Predict conditions can exceed the human benchmark on this dataset, establishing a baseline for AI novelty screening.
  • The explanations generated by the best model are usable: in the sampled cases they are grounded in the inputs, match the structure of real rejections, and contain no hallucinations.
  • The consistent 'Novel' bias on paired inputs indicates that generative models are better at detecting that something differs between claim and prior art than at judging whether the difference is legally sufficient.
  • Prompt design and label naming matter: the authors changed the label 'Non-Novel' to 'Found' to avoid model confusion, and the explain-first condition changed both accuracy and output quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the recorded examiner labels contain noise—allowance can come from arguments or other amendments—then human-beating accuracy may partly reflect predicting examiner behavior, not legal novelty; a multi-annotator version of the dataset would test this.
  • The fact that claim-only input sometimes beats claim-cited input in smaller models suggests a shortcut risk: models may lean on claim wording and length rather than on true correspondence, so swapping cited paragraphs with unrelated text would show whether the paired input is actually used.
  • The same claim–cited-text pairing with the MPEP instruction could be extended to other rejection grounds, such as obviousness, by changing the legal test in the prompt; this is a direct testable extension the paper does not run.
  • An immediate practical rollout is as a triage tool: use the model's 'Non-Novel' predictions with high precision to flag claims that likely need redrafting, and send the rest to human review.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a binary patent-novelty evaluation task and a dataset derived from USPTO non-final rejections. Non-Novel examples pair an original rejected claim with the cited paragraphs from the first rejection; Novel examples pair the amended claim from a granted patent with the same cited paragraphs. The authors evaluate classification-head models (Longformer, Llama2, Llama3) and generative models (Llama2, Llama3, GPT-4o) under claim-only and claim-cited-text inputs, across zero-shot, few-shot, and supervised fine-tuning. They report that large generative models achieve moderate accuracy on the claim-cited-text condition, with Llama3 70B reaching 0.624 accuracy under Explain-Predict few-shot, above a 20-example human benchmark of 0.600, and that the model explanations are mostly reasonable. The paper concludes that LLMs may assist novelty screening.

Significance. The dataset construction from real USPTO office actions is a useful contribution, and the task is well-motivated by actual examination practice. The inclusion of a claim-only baseline is the correct control, and the qualitative analysis gives an informative picture of where the model fails, such as hypernym/hyponym correspondences and implicit disclosure. If the headline result were robust, the paper would provide practical evidence that large generative models can support patent novelty screening. However, the current experimental evidence does not establish the central claim that the models evaluate the claim-prior-art correspondence, because the test set retains a known length/provenance confound and the headline accuracies are single runs without statistical testing.

major comments (5)
  1. [Section 3.2 / Table 1] The test set is not length-debiased. The pre-processing that equalizes claim length and label ratio is applied only to training and validation data; Table 1 leaves the test set at 398 instances with the original skew. Since Novel examples are amended claims from granted patents and Non-Novel examples are original claims from applications that received a first rejection, claim length and provenance are systematically correlated with the label on the test set. The best C-T result (Llama3 70B, Explain-Predict, few-shot, 0.624) is below the same model's Claim-only result (0.704), and Table 4 shows a strong bias toward Novel predictions (0.92 recall vs. 0.33 for Non-Novel). This pattern is precisely what a claim-side shortcut would produce. The conclusion in Section 5.3 that generative models assess the claim-cited-text relationship therefore requires a length-balanced test set or an explicit regression control for claim length and provenance.
  2. [Section 5.3 / Table 3] The headline accuracies are reported as a single run, as indicated by 'Acc.(1 time)' in the table header. There are no confidence intervals or significance tests. The difference between Llama3 70B C-T Explain-Predict few-shot (0.624) and the same model's zero-shot result (0.580), or between Llama3 70B and GPT-4o (0.538), may be due to sampling noise. The authors should report multiple runs with variance, and ideally a paired statistical test, especially for the comparison to the human benchmark.
  3. [Section 5.1] The human benchmark is one author labeling 20 examples. This is too small and too unrepresentative to support the statement in Section 5.3 that Llama3 70B shows performance 'surpassing human benchmarks.' Moreover, the human test is described as having the word-count/label correlation eliminated, while the model test set is not length-balanced, so the human-model comparison is not on equal footing. A larger and more diverse human evaluation, or at least an explicit acknowledgment that the human figure is anecdotal, is needed before claiming human-level performance.
  4. [Section 2 / Figure 2] The dataset labels rely on the assumption that an amended claim in a granted patent is novel with respect to the originally cited paragraphs. Grant decisions can result from examiner arguments, amendments to other claims, or changes in the cited art, rather than from the amended claim being absent from those specific paragraphs. The paper does not quantify or discuss this label noise. At minimum, the authors should state this assumption as a limitation and validate a sample of labels against prosecution histories to estimate the noise rate.
  5. [Section 5.3 / Table 4] The observed Novel bias is interpreted as evidence that generative models 'assess relationship of Claim-Cited texts inputs rather than relying solely on claim texts.' A bias toward the label associated with longer, amended claims does not by itself demonstrate use of the cited texts; it is equally consistent with reliance on claim-internal cues such as length, vocabulary, or drafting style. The claim-only results being higher than the C-T results further weaken this inference. This claim needs additional support, such as an analysis showing that model predictions change when the cited texts are replaced with unrelated texts.
minor comments (6)
  1. [Abstract / Section 1] There are several wording issues: 'following the process similar to that of patent examiners done' is ungrammatical, and 'LLM models' is redundant. Please revise for clarity.
  2. [Section 4.2] The phrase 'which were fame from the training data' should read 'which were taken from the training data.' Also, Table 5's caption contains 'Expain-Predict' instead of 'Explain-Predict.'
  3. [Table 4] The ACC column appears to report per-class accuracy rather than an overall accuracy; please clarify what is being averaged and, if appropriate, report the overall accuracy to avoid confusion.
  4. [Section 6] The paper claims to be the first novelty-evaluation dataset using whole cited texts, but the discussion of PatentMatch (Risch et al., 2020) is brief. Please add an explicit comparison of task formulation, label source, and input granularity to make the novelty of the contribution precise.
  5. [Appendix / Reproducibility] The paper does not state whether the dataset will be released or how to access it. Given the central role of the dataset, a data availability statement with licensing details is important for reproducibility.
  6. [Table A2] 'Reputation Penalty' should be 'Repetition Penalty.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper evaluates LLMs on an externally derived USPTO examination dataset; no fitted input is renamed as a prediction and no load-bearing self-citation appears.

full rationale

The paper's central claim is an empirical evaluation: LLMs classify patent claims as Novel or Non-Novel given cited prior-art texts, with labels taken from actual USPTO examination outcomes (original claims receiving a first Non-Final Rejection vs. amended claims in granted patents). No model parameter is fitted to the test labels and then reported as a prediction; results are held-out accuracy on a 398-instance test set. The only apparent self-reference is dataset construction: the cited paragraphs come from the same Non-Final Rejection documents that define the label. That is task design with an external ground truth, not a derivation of the result from the model's own outputs. There are no equations defining one quantity in terms of another, no fitted parameter renamed as a prediction, and no load-bearing self-citations or imported uniqueness theorems. The known label/claim-length and provenance correlations are acknowledged (Section 3.2) and partially filtered for train/eval, and any residual confound is a validity concern about benchmark design, not circular reasoning. Under the stated criteria, this paper shows no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on treating examination outcomes as ground truth and on pairing both labels with the same first-rejection cited texts. No fitted parameters are required beyond standard model hyperparameters reported in Appendix B. No theoretical entities are introduced.

assumptions (5)
  • domain assumption Anticipation requires that each and every claim element be found, expressly or inherently, in a single prior art reference.
    Used to define the task and prompts; quoted from MPEP 2131 in Section 2 and Appendix C.
  • domain assumption USPTO non-final rejection and grant outcomes are valid novelty labels.
    Section 2 and Appendix A derive labels from examination outcomes rather than independent annotation.
  • domain assumption Paragraph numbers cited in Non-Final Rejection identify all relevant prior art disclosure for the claim.
    Appendix A step 4 extracts cited paragraphs by regex and discards cited figures and other content.
  • ad hoc to paper A granted amended claim is novel with respect to the originally cited prior art paragraphs.
    Section 2 pairs the final amended claim with the first-rejection cited texts and labels it Novel; allowance may occur for other reasons.
  • ad hoc to paper One author's labeling of 20 examples is a representative human benchmark.
    Section 5.1 reports n=1 and 20 examples; the authors note limited sample size.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art." pith.science (2026). https://pith.science/paper/KSMESKG6

@misc{pith2026250206316,
  author       = {Pith},
  title        = {Pith review of: Can AI Examine Novelty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KSMESKG6}},
  note         = {Machine review of arXiv:2502.06316}
}
read the original abstract

Assessing the novelty of patent claims is a critical yet challenging task traditionally performed by patent examiners. While advancements in NLP have enabled progress in various patent-related tasks, novelty assessment remains unexplored. This paper introduces a novel challenge by evaluating the ability of large language models (LLMs) to assess patent novelty by comparing claims with cited prior art documents, following the process similar to that of patent examiners done. We present the first dataset specifically designed for novelty evaluation, derived from real patent examination cases, and analyze the capabilities of LLMs to address this task. Our study reveals that while classification models struggle to effectively assess novelty, generative models make predictions with a reasonable level of accuracy, and their explanations are accurate enough to understand the relationship between the target patent and prior art. These findings demonstrate the potential of LLMs to assist in patent evaluation, reducing the workload for both examiners and applicants. Our contributions highlight the limitations of current models and provide a foundation for improving AI-driven patent analysis through advanced models and refined datasets.

Figures

Figures reproduced from arXiv: 2502.06316 by the authors.

Figure 1
Figure 1. An Overview of Patent Examination Process: [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Entire patent examination process and patent documents from which input data is extracted: the Input [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claims

    cs.CL 2025-05 conditional novelty 7.0 of 10

    PEDANTIC provides the first public dataset of 14k patent claims labeled with examiner-cited reasons for indefiniteness, along with baselines showing LLMs still lag logistic regression on binary prediction.

Reference graph

Works this paper leans on

25 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Peters, and Arman Cohan

    Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150

    arXiv 2020

  4. [4]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36

    2024

  5. [5]

    Xiaochen Gao, Zhaoyi Hou, Yifei Ning, Kewen Zhao, Beilei He, Jingbo Shang, and Vish Krishnan. 2022. Towards comprehensive patent approval predictions: Beyond traditional document classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    work page 2022

  6. [6]

    Google patents public datasets

    Google. Google patents public datasets. https://console.cloud.google.com/marketplace/product/google_patents_public_datasets/google-patents-public-data, Retrieved [Jun. 17, 2024]

    work page 2024

  7. [7]

    Chikara Hashimoto, Gautam Kumar, Shuichiro Hashimoto, and Jun Suzuki. 2023. Hunt for buried treasures: Extracting unclaimed embodiments from patent specifications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 25--36

    work page 2023

  8. [8]

    Shiyuan Huang, Siddarth Mamidanna, Shreedhar Jangam, Yilun Zhou, and Leilani H Gilpin. 2023. Can large language models explain themselves? a study of llm-generated self-explanations. arXiv preprint arXiv:2310.11207

    arXiv 2023

Show all 25 references
  1. [9]

    HuggingFace. Llama2. https://huggingface.co/docs/transformers/main/en/model_doc/llama2, Retrieved [Jun. 17, 2024]

    2024

  2. [10]

    Divyansh Kaushik and Zachary C. Lipton. 2018. https://doi.org/10.18653/v1/D18-1546 How much reading does reading comprehension require? a critical investigation of popular benchmarks . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p...

    2018 doi

  3. [11]

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199--22213

    2022

  4. [12]

    Yuta Koreeda and Christopher D Manning. 2021. Contractnli: A dataset for document-level natural language inference for contracts. arXiv preprint arXiv:2110.01799

    2021 arXiv

  5. [13]

    Ralf Krestel, Renukswamy Chikkamath, Christoph Hewel, and Julian Risch. 2021. A survey on deep learning for patent analysis. World Patent Information, 65:102035

    2021

  6. [14]

    Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. 2022. Can language models learn from explanations in context? arXiv preprint arXiv:2204.02329

    2022 arXiv

  7. [15]

    Jieh-Sheng Lee and Jieh Hsiang. 2019. Patentbert: Patent classification with fine-tuning a pre-trained bert model. arXiv preprint arXiv:1906.02124

    2019 arXiv

  8. [16]

    Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li, Fu-lee Wang, Qing Li, and Xiaoqin Zhong. 2023. Label supervised llama finetuning. arXiv preprint arXiv:2310.01208

    2023 arXiv

  9. [17]

    Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/, Retrieved [Jun. 17, 2024]

    2024

  10. [18]

    Oecd stat

    OECD. Oecd stat. https://stats.oecd.org/index.aspx?queryid=22014#,Retrieved [Jun. 27, 2024]

    2024

  11. [19]

    openAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/

    2024

  12. [20]

    Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. https://doi.org/10.18653/v1/S18-2023 Hypothesis only baselines in natural language inference . In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics,...

    2018 doi

  13. [21]

    Julian Risch, Nicolas Alder, Christoph Hewel, and Ralf Krestel. 2020. Patentmatch: a dataset for matching patent claims & prior art. arXiv preprint arXiv:2012.13919

    2020 arXiv

  14. [22]

    Valentin J Schmitt, Lothar Walter, and Frank C Schnittker. 2023. Assessment of patentability by means of semantic patent analysis--a mathematical-logical approach. World Patent Information, 73:102182

    2023

  15. [23]

    Mirac Suzgun, Luke Melas-Kyriazi, Suproteem Sarkar, Scott D Kominers, and Stuart Shieber. 2024. The harvard uspto patent dataset: A large-scale, well-structured, and multi-purpose corpus of patent applications. Advances in Neural Information Processing Systems, 36

    2024

  16. [24]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  17. [25]

    USPTO. 2014. Manual of PATENT EXAMINING PROCEDURE, ninth edition. USPTO. The latest revise in Feb. 2023

    2014

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.