Pith. sign in

REVIEW 4 major objections 4 minor 58 references

Can Large Language Models Understand As Well As Apply Patent Regulations to Pass a Hands-On Patent Attorney Test?

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read No current large language model could have passed the European Qualifying Examination for patent attorneys, the paper argues; the best model, o1, reached 0.82 accuracy against a 0.90 professional threshold.

desk verdict The benchmark is real and reusable, but the pass/fail conclusion rests on an uncited accuracy threshold and an incomplete exam. read the letter →

arxiv 2507.10576 v2 pith:5MPCCGLG submitted 2025-07-11 cs.CY cs.AIcs.CLcs.ET

Bhakti Khera , Rezvan Alamian , Pascal A. Scherz , Stephan M. Goetz This is my paper
classification cs.CYcs.AIcs.CLcs.ET
keywords largelanguagemodelsEuropeanQualifyingExaminationpatentlawlegalreasoningtemperaturesensitivitypromptengineeringmultimodalPDFprocessingnoveltyvsinventivestep
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to determine whether large language models can do more than recite patent law—whether they can actually apply European patent regulations well enough to pass the European Qualifying Examination (EQE) for patent attorneys. It evaluates fourteen open and proprietary models on the pre-examination's legal True/False questions from 2012 through 2024, with a random-guessing baseline of 0.5 and a professional pass bar it states at 0.90 average accuracy. No model reached that bar: the best model, o1, scored 0.82, while the two Llama 3.1 8B deployments scored 0.50 and 0.55, within the paper's random-guessing range. Blind human patent experts found systematic weaknesses in legal reasoning—especially confusion between novelty and inventive step—and the paper concludes that a virtual patent attorney remains out of reach for current AI.

What carries the argument

The load-bearing instrument is the EQE pre-examination itself: past official exam papers (2012–2024, with 2020 cancelled) whose True/False questions have published gold answers, so accuracy and F1 can be scored mechanically against a random baseline of 0.5. The argument works by comparing every model's accuracy and F1 to the paper's stated professional bar of 0.90 average accuracy, then supplementing that number with blind human expert scoring on completeness, clarity, terminology, and technical correctness. A secondary mechanism is an embedding-similarity analysis showing that models such as Gemma 2 and the Llama 3.1 family place high cosine similarity between the terms 'novel', 'inventive', 'nonobvious', and 'obvious,' which the paper identifies as a likely source of legal-concept confusion. A third mechanism is an output-reliability statistic, computed by bootstrap resampling at different temperatures, which shows that mid-range temperatures around 0.4–0.6 produce unstable answers and that a temperature near 0.3 balances accuracy and determinism.

What would settle it

Look up the official passing standard for the European Qualifying Examination: if the required average accuracy is below 0.82, or if the exam is graded on a curve, then o1's 0.82 clears the bar and the paper's central conclusion fails. A second check would rerun the temperature and prompt sweeps across all 480 questions instead of one randomly chosen four-statement question; if the large accuracy swings disappear, the claimed sensitivity is an artifact of a single trial.

Watch

Extended reading notes

Core claim

The central claim is that none of the evaluated LLMs could have passed the EQE fully, because no model's accuracy exceeded the 0.90 average that the paper treats as the professional standard. o1 came closest with 0.82 accuracy; AWS-deployed Llama 3.1 8B scored 0.50 and a Python-deployed Llama 3.1 8B scored 0.55, both within the chance range of the two-answer forced-choice design. Performance depended strongly on deployment details: the same 8B checkpoint behaved differently on AWS than in a local Python build, and modest temperature changes or the addition of a one-line justification prompt shifted answers substantially, especially for small models. Human patent experts, blinded to model identity, awarded higher scores to explanations that were clear and legally reasoned, not merely correct, and they found that most models conflated novelty with obviousness/inventive step, cited articles incorrectly, miscalculated deadlines, and mishandled dates and priorities. The paper therefore claims that current models, despite respectable structured-question accuracy, lack the consistent interpretive reasoning required for professional patent work.

Load-bearing premise

The central conclusion assumes that the exam requires an average accuracy of 0.90 to pass, a threshold the paper states without citing a source; if the real pass mark is lower, the best model's 0.82 could be a passing score.

Editorial extensions

If this is right

  • No evaluated model could be deployed as an autonomous patent attorney: the best score, 0.82, falls below the paper's 0.90 professional threshold, so qualified human review remains legally necessary.
  • Small open models under 100B parameters performed at or near random guessing on the legal True/False task, so they cannot yet substitute for larger proprietary systems without additional prompt or fine-tuning work.
  • Explicit justification prompts and a temperature near 0.3 improved accuracy and stability in small models, so prompt engineering should be treated as a first-class hyperparameter before scaling up model size.
  • Human expert evaluation diverged from automatic metrics, with clarity and legal rationale valued over raw correctness; future evaluations of legal LLMs should therefore combine both forms of assessment.
  • Multimodal processing of complete exam PDFs remains uneven: GPT-4o held around 0.70 accuracy while Claude 3 Opus dropped to about 0.67 with formatting losses, so full-exam automation is not yet reliable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the exam's actual pass mark is lower than 0.90, the paper's headline 'no model passed' may be true relative to its own threshold but false about the real exam; a single authoritative number would settle whether o1's 0.82 is a pass.
  • The embedding-similarity result suggests an untested remedy: increasing the semantic distance between 'novel' and 'inventive' through legal-domain pretraining or embedding adjustment should improve accuracy on novelty/inventive-step questions, which a future study could verify.
  • The paper excluded Questions 11–20, which contain extensive claim sets and diagrams, to keep models comparable; its 'no full pass' conclusion therefore covers the legal component plus a limited multimodal subset, not every section of the unmodified exam.
  • The prompt gains for small models imply a possible economic substitution: pairing a small open model with an explicit reasoning prompt may approach mid-size model accuracy at lower cost, but this needs validation across all exam years rather than the single random question tested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper evaluates 14 large language models on the legal component of the European Qualifying Examination (EQE) pre-exam for years 2012-2024, using True/False statements with written justifications, and reports accuracy, F1, and human expert ratings. It also studies the effects of temperature, prompting, context length, deployment platform, and multimodal input. The headline result is that OpenAI o1 achieves the highest accuracy (0.82), while the smaller Llama and Gemma models perform near or below random guessing, and the authors conclude that no evaluated model could have passed the examination fully because accuracy never exceeded a threshold of 0.90.

Significance. If the pass/fail framing were properly supported, this would be a useful independent benchmark for LLM competence in a specialized legal domain. The study has notable strengths: it uses an external benchmark with published answer keys, the human evaluation is blind and conducted by patent professionals, the class-imbalance and guessing baselines are explicitly reported, and the empirical measurements of accuracy across many models are informative on their own. However, the central claim is undercut by an uncited pass threshold and by the exclusion of the claims/diagrams portion for most models, so the paper's main conclusion currently outruns the evidence.

major comments (4)
  1. [Abstract; §5] The central claim that no model could have passed the examination fully depends on an average accuracy threshold of 0.90 that is stated in the abstract and §5 but never cited or derived from the EPO's official scoring regulations. Section 3.3 defines accuracy and F1 but does not map either metric to any official pass mark. If the actual EQE pass mark is below 0.82, o1 would have passed and the headline conclusion would be wrong. Please cite the official pass standard or replace all pass/fail wording with a narrower statement such as 'no model reached 0.90 accuracy'.
  2. [§3.2, §4.5, §5] The conclusion that no model could have passed 'the examination fully' is not supported by the evaluated subset. For most models, only Questions 1-10 (the legal part) were evaluated; Questions 11-20 (claim sets and diagrams) were excluded for all models except GPT-4o and Opus, and those two models saw full PDFs only for the years 2012-2022. A failure on the legal component alone does not imply failure of the complete examination. Please restrict the conclusion to the legal component or report complete-exam results for all models.
  3. [§4.1, §4.2] The temperature 'sweet spot' at T=0.3 and the prompt-gain figures (+12 and +10 percentage points for Llama 3.1 8B and Mistral 7B) are based on a single randomly chosen question with four True/False statements per model. A single question cannot support general claims about decoding parameters or prompting effects. These results should be presented as an illustrative case study or replicated across a much larger sample of questions.
  4. [§4.3, §4.4] The platform comparison confounds deployment environment with context length: the AWS-hosted Llama 3.1 8B received one 1,700-token prompt, while the Python-deployed Llama 3.1 8B received ten 420-token prompts. The accuracy difference (0.50 vs. 0.55) could be caused by the different input format rather than by the platform itself. A controlled experiment that uses identical prompts on both platforms is needed to support the claim of platform-specific KV-cache effects.
minor comments (4)
  1. [Abstract; §4] The abstract reports o1's F1 score as 0.81, while §4's opening paragraph reports 0.84; please reconcile these numbers.
  2. [Appendix A] Figure 17 is captioned '2018' but follows the 2021 figure in a yearly sequence, and no 2022 figure appears; please correct the captions or plots.
  3. [§4.7.2] The subsection heading 'Synopsys' appears to be a typo for 'Synopsis'.
  4. [§3.2] The phrase 'one question was choosen randomly' contains a spelling error; it should be 'chosen'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is self-contained against the external EQE benchmark, and the uncited 0.90 pass threshold is a validity concern rather than a circular step.

full rationale

The paper's central results are obtained by running LLMs on past EQE pre-examinations and scoring their True/False answers against the EPO's published ground-truth answers (Section 3.3: 'the ground-truth answers were available'). Accuracy and F1 are computed from this external benchmark, so no predicted quantity is defined in terms of a fitted parameter or of the models' own outputs. The self-citations in the literature review (Jiang and Goetz 2025; Jiang et al. 2024, 2025) merely motivate the gap and are not used to establish the experimental findings. The new 'output reliability' metric is an explicitly described bootstrap mode-agreement procedure, not a renamed known result, and it is ancillary to the main accuracy comparison. The main caveat is that the abstract and conclusion assert that a 0.90 average accuracy is 'required for professional-level standards' without citing the EPO's actual pass mark; if the operative threshold were at or below o1's 0.82, the pass/fail headline would be wrong. That is an unsupported empirical premise and a correctness risk, not a circularity, because the benchmark data themselves come from an external source. Similarly, the exclusion of Questions 11-20 for most models makes 'the examination fully' an overstatement, but again this is a scope/validity issue, not a derivation that reduces to its own inputs. No circular step requiring a flag was found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on an uncited passing threshold, a potentially non-representative question subset, and small-sample temperature/prompt experiments. No fitted parameter is used, so no free parameters are listed.

assumptions (3)
  • ad hoc to paper The EQE requires an average accuracy of 0.90 for a full pass at professional level.
    Stated in abstract and conclusion without citation; if the real pass mark is lower, the central claim fails.
  • ad hoc to paper Excluding Questions 11-20 (claims and diagrams) leaves the test sufficiently representative of the hands-on attorney examination.
    The paper excludes these questions for most models while titling the work 'Hands-On Patent Attorney Test'.
  • ad hoc to paper A single randomly chosen question (four True/False statements) is sufficient to characterize temperature and prompting effects.
    Experiments 3 and 4 rely on one question per model; no cross-validation or error bars are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Large Language Models Understand As Well As Apply Patent Regulations to Pass a Hands-On Patent Attorney Test?." pith.science (2026). https://pith.science/paper/5MPCCGLG

@misc{pith2026250710576,
  author       = {Pith},
  title        = {Pith review of: Can Large Language Models Understand As Well As Apply Patent Regulations to Pass a Hands-On Patent Attorney Test?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5MPCCGLG}},
  note         = {Machine review of arXiv:2507.10576}
}
read the original abstract

The legal field already uses various large language models (LLMs) in actual applications, but their quantitative performance and reasons for it are underexplored. We evaluated several open-source and proprietary LLMs -- including GPT-series, Anthropic, Deepseek and Llama-3, variants -- on parts of the European Qualifying Examination (EQE) for future European Patent Attorneys. OpenAI o1 led with 0.82 accuracy and 0.81 F1 score, whereas (Amazon Web Services) AWS Llama 3.1 8B lagged at 0.50 accuracy, and a Python-deployed Llama 3.1 8B scored 0.55. The latter two are within the range of mere guessing for the two-answer forced-choice design. None of the evaluated models could have passed the examination fully, as accuracy never exceeded the average threshold of 0.90 required for professional-level standards -- also not models that are regularly promoted for their assumed beyond-PhD- and bar-admitted-lawyer-level performance. GPT-4o excelled at integrating text and graphics, while Claude 3 Opus often lost formatting coherence. Human patent experts evaluated the textual justifications and uncovered various critical shortcomings of each model. They valued clarity and legal rationale over the raw correctness of the answers, which revealed misalignment between automatic metrics and expert judgment. Model outputs were sensitive to modest temperature changes and prompt wording, which underscores the remaining necessity of expert oversight. Future work should target logical consistency, robust multimodality, and adaptive prompting to approach human-level patent proficiency. In summary, despite the outstanding performance of recent large models, the general public might overestimate their performance. The field has a long way to go to develop a virtual patent attorney. This paper wants to point out several specific limitations that need solutions.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 35 canonical work pages

  1. [1]

    ://arxiv.org/abs/2412.19260, 2412.19260

    Abacha AB, wai Yim W, Fu Y, et al (2025) MEDEC : A benchmark for medical error detection and correction in clinical notes. ://arxiv.org/abs/2412.19260, 2412.19260

    arXiv 2025

  2. [2]

    World Patent Information 37:3--13

    Abbas A, Zhang L, Khan SU (2014) A literature review on the state-of-the-art in patent analysis. World Patent Information 37:3--13. doi:10.1016/j.wpi.2013.12.006

    work page doi:10.1016/j.wpi.2013.12.006 2014

  3. [3]

    Claude-3 Model Card

    Anthropic A (2024) The Claude 3 model family: Opus, Sonnet, Haiku . Claude-3 Model Card

    work page 2024

  4. [4]

    Bender EM, Gebru T, McMillan-Major A, et al (2021) On the dangers of stochastic parrots: Can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp 610--623

    work page 2021

  5. [5]

    In: Advances in Neural Information Processing Systems, pp 1877--1901

    Brown TB, Mann B, Ryder N, et al (2020) Language models are few-shot learners. In: Advances in Neural Information Processing Systems, pp 1877--1901

    work page 2020

  6. [6]

    World Patent Information 80:102341

    Bui LV (2025) Advancing patent law with generative ai: Human-in-the-loop systems for ai-assisted drafting, prior art search, and multimodal ip protection. World Patent Information 80:102341

    work page 2025

  7. [7]

    In: IET Conference Proceedings CP907, IET, pp 135--138

    Chen FC, Pan CL, Lin YC, et al (2024) Dependency-aware patent claim generator. In: IET Conference Proceedings CP907, IET, pp 135--138

    work page 2024

  8. [8]

    World Patent Information 80:102339

    Chen FC, Pan CL, Team AD (2025) Evaluating application of large language models to biomedical patent claim generation. World Patent Information 80:102339

    work page 2025

Show all 58 references
  1. [9]

    arXiv doi:10.48550/arxiv.2107.03374

    Chen M, Tworek J, et al (2021) Evaluating large language models trained on code. arXiv doi:10.48550/arxiv.2107.03374

    2021 doi arXiv

  2. [10]

    arXiv doi:10.48550/arxiv.2204.02311

    Chowdhery A, Narang S, Devlin J, et al (2022) PaLM: Scaling Language Modeling with Pathways . arXiv doi:10.48550/arxiv.2204.02311

    2022 doi arXiv

  3. [11]

    arXiv doi:10.48550/arxiv.2110.14168

    Cobbe K, Kosaraju V, Bavarian M, et al (2021) Training verifiers to solve math word problems. arXiv doi:10.48550/arxiv.2110.14168

    2021 doi arXiv

  4. [12]

    In: Proceedings of NAACL-HLT, pp 4171--4186

    Devlin J, Chang MW, Lee K, et al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp 4171--4186

    2019

  5. [13]

    In: Kotz S, Johnson NL (eds) Breakthroughs in Statistics: Methodology and Distribution

    Efron B (1992) Bootstrap methods: Another look at the jackknife. In: Kotz S, Johnson NL (eds) Breakthroughs in Statistics: Methodology and Distribution. Springer New York, New York, NY, p 569--593, doi:10.1007/978-1-4612-4380-9_41

    1992 doi

  6. [14]

    ://arxiv.org/abs/2401.14196, 2401.14196

    Guo D, Zhu Q, Yang D, et al (2024) Deepseek-coder: When the large language model meets programming -- the rise of code intelligence. ://arxiv.org/abs/2401.14196, 2401.14196

    2024 arXiv

  7. [15]

    ://arxiv.org/abs/1904.09751

    Holtzman A, Buys J, Du L, et al (2019) The curious case of neural text degeneration . ://arxiv.org/abs/1904.09751

    2019 arXiv

  8. [16]

    ://arxiv.org/abs/2410.01805

    Huang Y, Yuan B, Han X, et al (2024) Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices . ://arxiv.org/abs/2410.01805

    2024 arXiv

  9. [17]

    ://arxiv.org/abs/2310.06825, 2310.06825

    Jiang AQ, Sablayrolles A, Mensch A, et al (2023) Mistral 7b. ://arxiv.org/abs/2310.06825, 2310.06825

    2023 arXiv

  10. [18]

    Artificial Intelligence Review 58(7)

    Jiang L, Goetz SM (2025) Natural language processing in the patent domain: a survey. Artificial Intelligence Review 58(7). doi:10.1007/s10462-025-11168-z

    2025 doi

  11. [19]

    arXiv:241202549

    Jiang L, Scherz PA, Goetz S (2024) Patent-cr: A dataset for patent claim revision. arXiv:241202549

    2024

  12. [20]

    arXiv:250512568

    Jiang L, Li C, Goetz S (2025 a ) Enriching patent claim generation with european patent dataset. arXiv:250512568

    2025

  13. [21]

    arXiv:250511095

    Jiang L, Scherz PA, Goetz S (2025 b ) Towards better evaluation for generated patent claims. arXiv:250511095

    2025

  14. [22]

    Jiang L, Zhang C, Scherz PA, et al (2025 c ) Can large language models generate high-quality patent claims? ://arxiv.org/abs/2406.19465, 2406.19465

    2025 arXiv

  15. [23]

    Technovation 129:102883

    Just J (2024) Natural language processing for innovation search – reviewing an emerging non-human innovation intermediary. Technovation 129:102883. doi:10.1016/j.technovation.2023.102883

    2024

  16. [24]

    arXiv:241007009

    Knappich V, Razniewski S, H \"a tty A, et al (2024) Pap2pat: Towards automated paper-to-patent drafting using chunk-based outline-guided generation. arXiv:241007009

    2024

  17. [25]

    World Patent Information 65:102035

    Krestel R, Chikkamath R, Hewel C, et al (2021) A survey on deep learning for patent analysis. World Patent Information 65:102035. doi:10.1016/j.wpi.2021.102035

    2021

  18. [26]

    In: Proceedings of the 7th JURIX Doctoral Consortium (DC JURIX 2019), CEUR Workshop Proceedings, vol 2598

    Lee JS (2020) Patent transformer: a framework for personalized patent claim generation. In: Proceedings of the 7th JURIX Doctoral Consortium (DC JURIX 2019), CEUR Workshop Proceedings, vol 2598. CEUR-WS, conference held 11–12 Dec 2019

    2020

  19. [27]

    Artificial Intelligence and Law pp 1--44

    Lee JS (2024) Instructpatentgpt: training patent language models to follow instructions with human feedback. Artificial Intelligence and Law pp 1--44

    2024

  20. [28]

    World Patent Information 62:101983

    Lee JS, Hsiang J (2020) Patent claim generation by fine-tuning openai gpt-2. World Patent Information 62:101983. doi:10.1016/j.wpi.2020.101983

    2020

  21. [29]

    In: Proceedings of ACL , pp 7871--7880

    Lewis M, Liu Y, Goyal N, et al (2020) BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension . In: Proceedings of ACL , pp 7871--7880

    2020

  22. [30]

    In: Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp 74--81

    Lin CY (2004) Rouge: A package for automatic evaluation of summaries. In: Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pp 74--81

    2004

  23. [31]

    arXiv doi:10.48550/arxiv.2412.19437

    Liu A, Feng B, Xue B, et al (2024) DeepSeek-V3 Technical Report . arXiv doi:10.48550/arxiv.2412.19437

    2024 doi arXiv

  24. [32]

    ://arxiv.org/abs/1907.11692, 1907.11692

    Liu Y, Ott M, Goyal N, et al (2019) Roberta: A robustly optimized bert pretraining approach. ://arxiv.org/abs/1907.11692, 1907.11692

    2019 arXiv

  25. [33]

    Journal of the American Statistical Association 44(247):335--341

    Metropolis N, Ulam S (1949) The Monte Carlo method . Journal of the American Statistical Association 44(247):335--341. doi:10.1080/01621459.1949.10483310

    1949

  26. [34]

    arXiv doi:10.48550/arxiv.1301.3781

    Mikolov T, Chen K, Corrado GS, et al (2013) Efficient estimation of word representations in vector space. arXiv doi:10.48550/arxiv.1301.3781

    2013 doi arXiv

  27. [35]

    In: Human Language Technology: Proceedings of a Workshop, Plainsboro (New Jersey), March 8-11, 1994

    Miller S, Schwartz R, Bobrow R, et al (1994) Statistical language processing using hidden understanding models. In: Human Language Technology: Proceedings of a Workshop, Plainsboro (New Jersey), March 8-11, 1994

    1994

  28. [36]

    ://arxiv.org/abs/2108.08877, 2108.08877

    Ni J, Ábrego GH, Constant N, et al (2021) Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. ://arxiv.org/abs/2108.08877, 2108.08877

    2021 arXiv

  29. [37]

    arXiv doi:10.48550/arxiv.2303.08774

    OpenAI (2023) GPT-4 Technical Report . arXiv doi:10.48550/arxiv.2303.08774

    2023 doi arXiv

  30. [38]

    In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp 311--318

    Papineni K, Roukos S, Ward T, et al (2002) Bleu: A method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp 311--318

    2002

  31. [39]

    In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 1532--1543

    Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 1532--1543

    2014

  32. [40]

    In: Proceedings of the 6th Workshop on Patent and Scientific Literature Translation

    Pouliquen B (2015) Full-text patent translation at WIPO; scalability, quality and usability . In: Proceedings of the 6th Workshop on Patent and Scientific Literature Translation

    2015

  33. [41]

    OpenAI blog 1(8):9

    Radford A, Wu J, Child R, et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9. ://api.semanticscholar.org/CorpusID:160025533

    2019

  34. [42]

    Journal of Machine Learning Research 21(140):1--67

    Raffel C, Shazeer N, Roberts A, et al (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21(140):1--67. ://jmlr.org/papers/volume21/20-074/20-074.pdf

    2020

  35. [43]

    Knowledge and Information Systems 61(2):631--660

    Shalaby W, Zadrozny W (2019) Patent retrieval: a literature review. Knowledge and Information Systems 61(2):631--660. doi:10.1007/s10115-018-1322-7

    2019 doi

  36. [44]

    ://arxiv.org/abs/1906.03741, 1906.03741

    Sharma E, Li C, Wang L (2019) Bigpatent: A large-scale dataset for abstractive and coherent summarization. ://arxiv.org/abs/1906.03741, 1906.03741

    2019 arXiv

  37. [45]

    arXiv:250115074

    Shukla S, Sharma N, Gupta M, et al (2025) Patentlmm: Large multimodal model for generating descriptions for patent figures. arXiv:250115074

    2025

  38. [46]

    IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-1(2):164--172

    Suen CY (1979) n-gram statistics for natural language understanding and text processing. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI-1(2):164--172. doi:10.1109/TPAMI.1979.4766902

    1979

  39. [47]

    arXiv doi:10.48550/arxiv.2201.08239

    Thoppilan R, De Freitas D, Hall J, et al (2022) Lamda: Language models for dialog applications. arXiv doi:10.48550/arxiv.2201.08239

    2022 doi arXiv

  40. [48]

    ://arxiv.org/abs/2302.13971, 2302.13971

    Touvron H, Lavril T, Izacard G, et al (2023) Llama: Open and efficient foundation language models. ://arxiv.org/abs/2302.13971, 2302.13971

    2023 arXiv

  41. [49]

    In: International TRIZ Future Conference, Springer, pp 3--19

    Trapp S, Warschat J (2024) Llm-based extraction of contradictions from patents. In: International TRIZ Future Conference, Springer, pp 3--19

    2024

  42. [50]

    Tex Tech L Rev 54:255

    Tu SS, Cyphert A, Perl S (2022) Limits of using artificial intelligence and GPT-3 in patent prosecution. Tex Tech L Rev 54:255. doi:10.2139/ssrn.3899388, wVU College of Law Research Paper No.\ 2021-014

    2022 doi

  43. [51]

    In: Advances in Neural Information Processing Systems, pp 5998--6008

    Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Advances in Neural Information Processing Systems, pp 5998--6008

    2017

  44. [52]

    arXiv:241209796

    Wang Q, Ni S, Liu H, et al (2024 a ) Autopatent: A multi-agent framework for automatic patent generation. arXiv:241209796

    2024

  45. [53]

    arXiv:241218100

    Wang S, Yin X, Wang M, et al (2024 b ) Evopat: A multi-llm-based patents summarization and analysis agent. arXiv:241218100

    2024

  46. [54]

    arXiv:241021312

    Wang X, Zhang Y, Zhang X, et al (2024 c ) Patentagent: Intelligent agent for automated pharmaceutical patent analysis. arXiv:241021312

    2024

  47. [55]

    arXiv doi:10.48550/arxiv.2112.04359

    Weidinger L, Mellor J, Rauh M, et al (2021) Ethical and social risks of harm from language models. arXiv doi:10.48550/arxiv.2112.04359

    2021 doi arXiv

  48. [56]

    Pattern Recognition 23(5):509--528

    Yannakoudakis E, Tsomokos I, Hutton P (1990) n-grams and their implication to natural language understanding. Pattern Recognition 23(5):509--528. doi:10.1016/0031-3203(90)90072-S

    1990 doi

  49. [57]

    arXiv:250519345

    Yoo Y, Xu Q, Cao L (2025) Patentscore: Multi-dimensional evaluation of llm-generated patent claims. arXiv:250519345

    2025

  50. [58]

    ://arxiv.org/abs/1904.09675, 1904.09675

    Zhang T, Kishore V, Wu F, et al (2020) Bertscore: Evaluating text generation with bert. ://arxiv.org/abs/1904.09675, 1904.09675

    2020 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.