REVIEW 4 major objections 4 minor 26 references
AI4Contracts: LLM & RAG-Powered Encoding of Financial Derivative Contracts
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read CDMizer demonstrates that template-driven, component-wise generation converts OTC derivative contracts into schema-compliant CDM representations with 100% syntactic correctness and schema adherence, plus higher semantic coverage than…
desk verdict A practical template-driven CDM extraction system whose novelty is real but whose evaluation is closed-loop; worth revising and re-reviewing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the pruned CDM template produced by Algorithm 1. The algorithm flattens every example CDM instance into dot-separated keys, then recursively traverses the CDM JSON schema resolving $ref references and retaining only property paths that appear in at least one example. Each retained field receives a schema description and a type-appropriate placeholder, yielding a minimal but complete skeleton for a contract type. CDMizer then walks this tree with a depth threshold d (set to 4 in experiments), and for each sufficiently shallow subtree builds a prompt containing the object structure, its schema definition, the traversal path, and RAG-retrieved example chunks; the LLM fills the subtree, a validator enforces exact structural match, and empty fields are cleaned at the end. This machinery converts an intractable full-schema generation problem (over four million lines of JSON) into many small, locally constrained filling tasks.
What would settle it
Take a genuine OTC contract containing a provision whose CDM field path is absent from the 858 example instances used to build the template, run CDMizer on it, and check whether the output contains that field; if it is missing and the semantic evaluation marks it uncaptured, the example-derived coverage limit is demonstrated.
Extended reading notes
Core claim
The central claim is that template-driven, hierarchical generation outperforms direct full-document LLM generation for contract-to-CDM conversion, and that this superiority is structural rather than incidental. Because the template is derived from the CDM schema and example instances, every generated object is forced to conform to the schema, so syntactic correctness and schema adherence are guaranteed by construction; semantic accuracy then depends on how well the LLM, guided by RAG-retrieved example chunks, populates each field. The paper reports mean semantic coverage of 89.40% without RAG and 91.40% with RAG for the model with the highest reported score, Llama-3.1-8B-Instruct, versus noticeably lower and more variable coverage for baseline direct generation. The authors state this establishes a scalable foundation for automated contract understanding and validation.
Load-bearing premise
The template retains only fields that appear in at least one available CDM example, so any contract term outside that example set is structurally omitted no matter how well the LLM performs.
Editorial extensions
If this is right
- On the paper's tests, CDMizer guarantees 100% syntactic correctness and schema adherence for every contract type, so downstream systems can assume the CDM output is well-formed.
- Component-wise generation removes the token-limit failure mode that truncates direct generation of long contracts, making the approach scalable to full-length agreements.
- Retrieval augmentation generally raises semantic coverage for both baseline and CDMizer, so the RAG knowledge base of chunked CDM examples is a reusable asset for future contract types.
- The template-creation algorithm makes the full CDM schema tractable by pruning away every field not exercised by example instances, so the generation task stays small.
- The LLM-powered evaluation framework offers a repeatable protocol for measuring semantic coverage, with a weighted coverage score distinguishing captured, uncaptured, and extraneous information.
Reading between the lines
- Because templates retain only fields that appear in example instances, any contract term outside that set is structurally absent, so real-world semantic coverage may fall even though syntactic correctness stays at 100%.
- Because the test contracts were synthetic and generated from the same CDM examples used to build the templates, the reported semantic coverage likely overstates performance on genuine contracts with novel wording; running CDMizer on real term sheets would quantify the gap.
- The coverage-score weights (uncaptured at 0.3, extraneous at 0.1) are choices, and method rankings could shift if those weights changed, so the metric deserves sensitivity analysis before high-stakes comparisons.
- If CDM output is later compiled into executable smart contracts, the template's structural guarantee would suppress hallucinated-field bugs that plague direct LLM code generation, but the paper does not test this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CDMizer, a template-driven LLM and RAG pipeline that converts unstructured OTC derivative contract descriptions into FINOS CDM JSON. It creates per-contract-type templates by pruning the CDM schema to field paths appearing in 858 CDM example files (Algorithm 1), then populates these templates recursively with a depth threshold and retrieval-augmented prompts, validating each populated object. The authors compare CDMizer with a direct generation baseline (with and without RAG) on 30 synthetic contracts spanning six contract types. They report 100% syntactic correctness and schema adherence for CDMizer, semantic coverage scores around 89-91% with Llama-3.1-8B-Instruct, and propose an LLM-based evaluation framework with a weighted coverage score (Eq. 1).
Significance. If the claims were fully supported, CDMizer would be a practical contribution to structured document understanding in finance: its template decomposition addresses the token-limit problem of long, nested contracts, and Algorithm 1 is a concrete, reproducible deterministic procedure. The paper also makes a useful start on LLM-based semantic-coverage evaluation and honestly acknowledges in Section 7.2 that real contracts contain nuanced language not captured by synthetic data. However, the current empirical design is largely closed-world: both the templates and the test contracts derive from the same 858 CDM examples, so the reported coverage scores measure slot-filling within known fields rather than the capacity to handle terms absent from the corpus. The central architectural idea is defensible, but the evaluation needs substantial strengthening before the paper's stronger generalizability claims can be accepted.
major comments (4)
- [§4, §6.1, Algorithm 1] Algorithm 1 restricts the template vocabulary to keys found in the CDM examples: Step 2 (lines 14-16) prunes every schema path not present in the flattened key set extracted in lines 3-6. Section 4 generates the synthetic contract descriptions from those same 858 CDM examples, and Section 7 samples the 30 test contracts from that synthetic set. Consequently, every field required by a test contract is inside the template by construction, and the reported 89-91% semantic coverage does not exercise any contract term absent from the FINOS example corpus. The limitation is acknowledged in Section 7.2, but it is never quantified; the paper should evaluate on out-of-corpus contracts or manually inject novel clauses to test whether the templates and the pipeline can represent terms not seen in the example set.
- [§7.2, Eq. (1)] The semantic coverage hyperparameters mu=0.3 and epsilon=0.1 are reported as chosen 'after testing different configurations and comparing the resulting scores to human evaluations' on the same contracts used for the final results. This means the reported coverage scores are partly fitted to the evaluation set, not neutral measurements. The paper should either tune these weights on a separated validation set or provide a sensitivity analysis over a range of mu and epsilon. In addition, Figure 3 and Table 2 report mean coverage without error bars, confidence intervals, or significance tests; with only 30 synthetic contracts, the observed improvements over the baseline may not be statistically robust.
- [§6.2, §7.1, Abstract] The claim that the template-driven approach 'guarantees' 100% syntactic correctness and schema adherence is definitional rather than empirical: templates are constructed from schema keys, outputs are validated to have 'the exact same structure' as the input object, and empty fields are removed before finalization. As such, the 100% scores in Table 1 are not a meaningful experimental comparison for CDMizer. The paper should explicitly separate architectural invariants (e.g., 'the only outputs accepted are those conforming to the template structure') from empirical findings, and specify whether validation checks only JSON key structure or also CDM business rules and data-type constraints.
- [§1, §7] The contribution list in Section 1 describes the baseline as an 'LLM and RAG pipeline', but Section 7 excludes fine-tuning from the baseline configurations due to token limitations and rigid behavior. The experiments therefore compare CDMizer not to the originally specified baseline but to a degraded few-shot-style variant. This makes the claimed superiority less informative. The authors should either evaluate the fine-tuned baseline on a subset of shorter contracts where it produces complete outputs, or explicitly redefine the baseline as a non-fine-tuned direct-generation method throughout the paper.
minor comments (4)
- [§6.1, Algorithm 1] Line 17 contains a typographical error ('preferences another schema' should be 'references another schema'), and the pseudocode would be clearer if it distinguished object-valued properties from array-valued properties when updating the path and resolving $ref.
- [§6.2, Figure 2] The depth threshold d is defined in prose as limiting processing to substructures with depth ≤ d, but the notion of depth is ambiguous: it is not specified whether depth counts JSON nesting levels, schema-reference levels, or the number of nodes along the longest path to a leaf. A formal definition is needed for reproducibility.
- [§7.1] The 'Syntactical Correctness' metric, defined as the proportion of generated keys that exist in the CDM schema, is redundant for CDMizer because its templates are built from schema keys; the paper should state explicitly how this metric applies to the baseline and whether missing keys are counted only when a key is present in the output but not in the schema.
- [Abstract, §7.2] The abstract claims 'improved scalability', but no latency, token-usage, or throughput measurements are reported anywhere in the evaluation. Either provide such measurements for the depth-thresholded traversal or temper the scalability claim to 'enables processing long documents within token limits'.
Circularity Check
Evaluation is partially circular: templates and test contracts are both derived from the same 858 FINOS CDM examples, so the reported semantic coverage does not test out-of-corpus contract terms; the coverage weights are also tuned on the evaluation set.
-
self definitional
[Section 4 (Data Collection and Synthesis), Section 6.1 / Algorithm 1, Section 7 (Experimental Evaluation)]
"To address this gap, we leveraged an LLM to generate synthetic contract descriptions from existing CDM representations... For experimentation purposes and convenience, we categorized the collected and generated data into six contract types... Only fields traversed in at least one example are retained in the template."
The template's field vocabulary is exactly the set of dot-separated keys extracted from the 858 FINOS CDM examples (Algorithm 1, lines 3-6 and 14-16). The test contracts, however, are synthetic descriptions generated from those same CDM examples (Section 4), and the 30 evaluation contracts are sampled from that synthetic set (Section 7). Consequently, every field a test contract can require is already present in the template by construction. The reported 89-91% semantic coverage therefore measures how well the LLM fills known slots, not whether the framework can encode contract terms absent from the example corpus. No out-of-corpus contract is used, so the claimed 'comprehensive' templates and the coverage numbers do not provide independent evidence of generalization to real OTC contracts.
-
fitted input called prediction
[Section 7.1, Equation (1); Section 7.2 (Results and Discussion)]
"CoverageScore = C×100 / (C+µ×U+ϵ×E) ... For generating the results, we set the depth threshold to d=4, with weighting factors µ=0.3 and ϵ=0.1 ... These values were chosen after testing different configurations and comparing the resulting scores to human evaluations."
The semantic coverage score is a weighted function of captured, uncaptured, and extraneous elements, with weights µ and ϵ chosen after testing configurations and calibrating against human evaluations. The same contracts used for tuning are then scored with the tuned metric. The reported absolute coverage percentages and the 'outperformed' claim are therefore not fully independent measurements; they are partly fitted to the evaluation set. This does not necessarily change the rank ordering between methods, but it weakens the precision of the numerical coverage claims.
1 more flagged steps
-
self definitional
[Section 6 (CDMizer), first paragraph; Section 7.2 (Results and Discussion)]
"Templates enforce 100% schema adherence and correctness by providing a fixed structure ... As the approach suggests, CDMizer (both with or without RAG versions) guarantees a score of 100%."
The 100% schema adherence result is a logical consequence of the template mechanism, not an empirical finding. The template is created by pruning the CDM schema to paths that appear in examples, the LLM is asked to populate exactly that structure, and the validation step checks that the populated object has the same structure as the input template. Thus the measured '100% schema adherence' is definitionally guaranteed by the method's design; it does not test whether the schema is correctly inferred or whether the generated content is semantically valid.
full rationale
The central circularity is in the evaluation design: templates are built from the same 858 FINOS CDM examples that are used to synthesize the test contracts. Algorithm 1 retains only keys found in those examples, and Section 4 generates contract descriptions from the same example set. Therefore the semantic coverage metric measures slot-filling within a known field vocabulary, not the framework's ability to handle unseen contract terms. The weighting factors in Eq. (1) are also tuned on the evaluation set, further reducing the independence of the reported scores. The 100% schema adherence is definitional, though the paper explicitly acknowledges it is expected. I do not find load-bearing self-citation: the cited works by the authors appear only as related-work examples and do not justify the central claims. The framework itself has independent components (depth-based traversal, RAG, hierarchical generation), and a real out-of-corpus evaluation could provide genuine evidence, but the current paper does not provide it. Score 6 reflects that one of the central empirical claims is partially forced by construction rather than fully demonstrated.
Assumptions & free parameters
free parameters (3)
- Depth threshold d =
4
- Weighting factor mu =
0.3
- Weighting factor epsilon =
0.1
assumptions (4)
- domain assumption Synthetic contracts generated from CDM examples are representative of real OTC derivative contracts.
- domain assumption The set of JSON keys present in the available CDM examples is complete for each contract type.
- domain assumption An LLM with structured step-by-step prompts can accurately count captured, uncaptured, and extraneous information.
- domain assumption The FINOS CDM schema and examples are correct and authoritative.
Cite this review
Pith. "Pith review of AI4Contracts: LLM & RAG-Powered Encoding of Financial Derivative Contracts." pith.science (2026). https://pith.science/paper/WVNTCQYO
@misc{pith2026250601063,
author = {Pith},
title = {Pith review of: AI4Contracts: LLM & RAG-Powered Encoding of Financial Derivative Contracts},
year = {2026},
howpublished = {\url{https://pith.science/paper/WVNTCQYO}},
note = {Machine review of arXiv:2506.01063}
}
read the original abstract
Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) are reshaping how AI systems extract and organize information from unstructured text. A key challenge is designing AI methods that can incrementally extract, structure, and validate information while preserving hierarchical and contextual relationships. We introduce CDMizer, a template-driven, LLM, and RAG-based framework for structured text transformation. By leveraging depth-based retrieval and hierarchical generation, CDMizer ensures a controlled, modular process that aligns generated outputs with predefined schema. Its template-driven approach guarantees syntactic correctness, schema adherence, and improved scalability, addressing key limitations of direct generation methods. Additionally, we propose an LLM-powered evaluation framework to assess the completeness and accuracy of structured representations. Demonstrated in the transformation of Over-the-Counter (OTC) financial derivative contracts into the Common Domain Model (CDM), CDMizer establishes a scalable foundation for AI-driven document understanding, structured synthesis, and automated validation in broader contexts.
Figures
Reference graph
Works this paper leans on
-
[1]
[Agreement, 2002] ISDA Master Agreement. 2002 isda master agreement. https://www.isda.org/book/2002-isda-master-agr eement-mylibrary/,
work page 2002
-
[7]
Accessed: 2025-05-09. [FINOS, 2024b] FINOS. Overview of the finos cdm. https://cdm. finos.org/docs/cdm-overview/#purpose,
work page 2025
-
[8]
[FINOS, 2025] FINOS. common-domain-model. https://cdm.finos .org/,
work page 2025
-
[11]
Key trends in the size and composition of otc derivatives markets in the first half of
[ISDA, 2023] ISDA. Key trends in the size and composition of otc derivatives markets in the first half of
work page 2023
-
[13]
[J.P. Morgan, 2024] J.P. Morgan. Usd/inr irs disclosure. https://ww w.jpmorgan.com/content/dam/jpm/global/disclosures/IN/usd-inr -irs.pdf,
work page 2024
-
[14]
[Kanget al., 2024 ] Inwon Kang, William Van Woensel, and Oshani Seneviratne
Accessed: 2025-05-09. [Kanget al., 2024 ] Inwon Kang, William Van Woensel, and Oshani Seneviratne. Using large language models for generating smart contracts for health insurance from textual policies. InAI for Health Equity and Fairness: Leveraging AI to Address Social Determinants of Health, pages 129–146. Springer,
work page 2025
-
[15]
Solmover: Feasibility of using llms for translating smart contracts
[Karanjaiet al., 2024 ] Rabimba Karanjai, Lei Xudagger, and Wei- dong Shi. Solmover: Feasibility of using llms for translating smart contracts. In2024 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), pages 1–3. IEEE,
work page 2024
-
[16]
Competition-level code generation with alphacode.Science, 378(6624):1092–1097,
[Liet al., 2022 ] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R ´emi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode.Science, 378(6624):1092–1097,
work page 2022
Show all 26 references
-
[17]
Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems, 36,
[Liuet al., 2024 ] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation.Advances in Neural Information Processing Systems, 36,
2024
-
[18]
Automatic generation of ethereum-based smart contracts for agri-food traceability sys- tem.Ieee Access, 10:50363–50383,
[Marchesiet al., 2022 ] Lodovica Marchesi, Katiuscia Mannaro, Michele Marchesi, and Roberto Tonelli. Automatic generation of ethereum-based smart contracts for agri-food traceability sys- tem.Ieee Access, 10:50363–50383,
2022
-
[19]
Smart derivatives contracting: Automating in- terest rate swaps in the over-the-counter (otc) market with the daml.Available at SSRN 3750089,
[Oluwajebeet al., 2020 ] Olusegun Oluwajebe, Mary Duah, and Polina Golnikova. Smart derivatives contracting: Automating in- terest rate swaps in the over-the-counter (otc) market with the daml.Available at SSRN 3750089,
2020
-
[21]
Structured rates presentation or document
[RBC Capital Markets, 2024] RBC Capital Markets. Structured rates presentation or document. https://www.rbccm.com/stru cturedrates/file-566984.pdf,
2024
-
[22]
[Singhet al., 2024 ] Srisht Fateh Singh, Panagiotis Michalopoulos, and Andreas Veneris
Accessed: 2025-05-09. [Singhet al., 2024 ] Srisht Fateh Singh, Panagiotis Michalopoulos, and Andreas Veneris. Option contracts in the defi ecosys- tem: Motivation, solutions, & technical challenges. In2024 IEEE International Conference on Blockchain and Cryptocur- rency (ICBC)...
2025
-
[23]
(in) correct smart contract spec- ifications
[Sorensen, 2024] Derek Sorensen. (in) correct smart contract spec- ifications. In2024 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), pages 567–575. IEEE,
2024
-
[24]
Automatic smart contract generation us- ing controlled natural language and template.IBM Journal of Research and Development, 63(2/3):6–1,
[Tateishiet al., 2019 ] Takaaki Tateishi, Sachiko Yoshihama, Naoto Sato, and Shin Saito. Automatic smart contract generation us- ing controlled natural language and template.IBM Journal of Research and Development, 63(2/3):6–1,
2019
-
[26]
Translating clinical decision logic within knowledge graphs to smart contracts
[Van Woenselet al., 2023] William Van Woensel, Manan Shukla, and Oshani Seneviratne. Translating clinical decision logic within knowledge graphs to smart contracts. InSeWeBMeDA@ ESWC, 2023
2023
-
[2002]
Unsupervised evaluation of code llms with round-trip correctness.arXiv preprint arXiv:2402.08699,
[Allamaniset al., 2024 ] Miltiadis Allamanis, Sheena Panthap- lackel, and Pengcheng Yin. Unsupervised evaluation of code llms with round-trip correctness.arXiv preprint arXiv:2402.08699,
2024 arXiv
-
[2016]
Smart derivative contracts (detaching transactions from counterparty credit risk: Specification, parametrisation, val- uation).Available at SSRN 3163074,
[Fries and Kohl-Landgraf, 2018] Christian P Fries and Peter Kohl- Landgraf. Smart derivative contracts (detaching transactions from counterparty credit risk: Specification, parametrisation, val- uation).Available at SSRN 3163074,
2018
-
[2018]
Temporal aspects of smart contracts for financial derivatives
[Clack and Vanca, 2018] Christopher D Clack and Gabriel Vanca. Temporal aspects of smart contracts for financial derivatives. In Leveraging Applications of Formal Methods, Verification and Validation. Industrial Practice: 8th International Symposium, ISoLA 2018, Limassol, Cypr...
2018
-
[2019]
Expectation vs
[Vaithilingamet al., 2022 ] Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. InChi conference on human factors in computing sys- tems extended abstracts, pages 1–7,
2022
-
[2020]
Re- trieval augmented code generation and summarization.arXiv preprint arXiv:2108.11601,
[Parvezet al., 2021 ] Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Re- trieval augmented code generation and summarization.arXiv preprint arXiv:2108.11601,
2021 arXiv
-
[2021]
Trust, confidence, and au- tomation: The isda master agreement as a smart contract.Busi- ness Law Review, 43(2),
[Armitage, 2022] Matthew Armitage. Trust, confidence, and au- tomation: The isda master agreement as a smart contract.Busi- ness Law Review, 43(2),
2022
-
[2022]
Auto-generation of smart contracts from domain-specific ontologies and semantic rules
[Choudhuryet al., 2018 ] Olivia Choudhury, Nolan Rudolph, Issa Sylla, Noor Fairoza, and Amar Das. Auto-generation of smart contracts from domain-specific ontologies and semantic rules. In2018 IEEE International Conference on Internet of Things (iThings) and IEEE Green Computin...
2018
-
[2023]
https://www.isda.o rg/2023/12/07/key-trends-in-the-size-and-composition-of-otc-d erivatives-markets-in-the-first-half-of-2023,
2023
-
[2024]
Automatic smart contract generation for internet of media things.ICT Express, 7(3):274– 277,
[Alloucheet al., 2021 ] Mohamed Allouche, Mihai Mitrea, Alexan- dre Moreaux, and Sang-Kyun Kim. Automatic smart contract generation for internet of media things.ICT Express, 7(3):274– 277,
2021
-
[2025]
From institutions to code: Towards automated generation of smart contracts
[Frantz and Nowostawski, 2016] Christopher K Frantz and Mariusz Nowostawski. From institutions to code: Towards automated generation of smart contracts. In2016 IEEE 1st International Workshops on Foundations and Applications of Self* Systems (FAS* W), pages 210–215. IEEE,
2016
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.