{"id":"4948a999-ef10-4190-9c53-a02a3ce95808","arxiv_id":"2506.01063","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CDMizer uses example-derived templates, depth-based retrieval, and hierarchical LLM generation to convert OTC derivative contract text into CDM JSON, achieving guaranteed schema adherence and improved semantic coverage over direct generation.","lead":"CDMizer is a template-driven AI system that converts unstructured financial derivative contracts into a standardized machine-readable format called CDM, using LLMs and retrieval-augmented generation. It reports 100% schema compliance by design and higher content coverage than direct LLM generation, but was tested only on synthetic contracts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Template fields are frozen to the 858 CDM examples, and the synthetic test set is generated from those same examples, so the reported semantic-coverage gains do not test out-of-corpus contract terms.","rationale":"The reader's weakest assumption is the same one: the template retains only keys observed in examples. My concern sharpens it by noting that the evaluation set was generated from those same examples, so the empirical claim 'outperformed baseline in semantic coverage' is an in-distribution statement only. The paper does provide independent support for the framework's built-in structural guarantee: the template is derived from the schema and validated locally, so the 100% syntactic correctness and schema-adherence scores are true by construction under the paper's own key-matching definition. That part of the claim is secure. The weakness is semantic coverage and scalability to real contracts, which is exactly where CDMizer's fixed template imposes a ceiling. This does not make the framework worthless; it makes the current evidence conditional. The appropriate remedy is not rejection but a test on held-out or real data, plus reporting of unmappable terms. Since the reader already reached CONDITIONAL, my read does not change the verdict; it identifies the precise condition that must be met.","tokens_in":10703,"tokens_out":7148,"duration_ms":80707,"concrete_test":"Rebuild the templates from an 80% training split of the FINOS CDM examples per contract type; run CDMizer on the held-out 20% of synthetic contracts (or on the two real RBC/JPMorgan term sheets). For each held-out CDM, compute the fraction of schema paths required by the test object that are absent from the training-built template. If any material fraction is absent, or if semantic coverage drops substantially from the reported 89-91%, the original scores are an artifact of template/test overlap. Additionally, manually count essential terms in the real term sheets that map to pruned CDM paths; one unmapped essential term falsifies the 'comprehensive' wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 makes the template's field vocabulary a closed set: Step 1 (lines 3-6) flattens only keys present in the example files, and Step 2 (lines 14-16) prunes every schema path not in that set. Any contract term that appears in no FINOS CDM example is therefore structurally absent from the generated output, regardless of LLM performance. This would be a minor limitation if the evaluation used out-of-corpus contracts, but it does not. Section 4 synthesizes contract descriptions from the 858 CDM examples, and Section 7 samples the 30 test contracts from that synthetic set. The template for each contract type is built from the same example pool, so every field required by a test contract is inside the template by construction. The reported 89-91% semantic coverage thus measures how well the LLM fills known slots, not the framework's capacity to represent terms absent from the examples. For real OTC contracts, whose clauses and elections need not appear in the FINOS corpus, the template is not 'comprehensive'; the paper's Section 7.2 limitation paragraph admits this, but the empirical design never quantifies the coverage ceiling.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CDMizer, a template-driven LLM and RAG pipeline that converts unstructured OTC derivative contract descriptions into FINOS CDM JSON. It creates per-contract-type templates by pruning the CDM schema to field paths appearing in 858 CDM example files (Algorithm 1), then populates these templates recursively with a depth threshold and retrieval-augmented prompts, validating each populated object. The authors compare CDMizer with a direct generation baseline (with and without RAG) on 30 synthetic contracts spanning six contract types. They report 100% syntactic correctness and schema adherence for CDMizer, semantic coverage scores around 89-91% with Llama-3.1-8B-Instruct, and propose an LLM-based evaluation framework with a weighted coverage score (Eq. 1).","tokens_in":10907,"tokens_out":4337,"duration_ms":46468,"significance":"If the claims were fully supported, CDMizer would be a practical contribution to structured document understanding in finance: its template decomposition addresses the token-limit problem of long, nested contracts, and Algorithm 1 is a concrete, reproducible deterministic procedure. The paper also makes a useful start on LLM-based semantic-coverage evaluation and honestly acknowledges in Section 7.2 that real contracts contain nuanced language not captured by synthetic data. However, the current empirical design is largely closed-world: both the templates and the test contracts derive from the same 858 CDM examples, so the reported coverage scores measure slot-filling within known fields rather than the capacity to handle terms absent from the corpus. The central architectural idea is defensible, but the evaluation needs substantial strengthening before the paper's stronger generalizability claims can be accepted.","major_comments":[{"comment":"Algorithm 1 restricts the template vocabulary to keys found in the CDM examples: Step 2 (lines 14-16) prunes every schema path not present in the flattened key set extracted in lines 3-6. Section 4 generates the synthetic contract descriptions from those same 858 CDM examples, and Section 7 samples the 30 test contracts from that synthetic set. Consequently, every field required by a test contract is inside the template by construction, and the reported 89-91% semantic coverage does not exercise any contract term absent from the FINOS example corpus. The limitation is acknowledged in Section 7.2, but it is never quantified; the paper should evaluate on out-of-corpus contracts or manually inject novel clauses to test whether the templates and the pipeline can represent terms not seen in the example set.","section":"§4, §6.1, Algorithm 1"},{"comment":"The semantic coverage hyperparameters mu=0.3 and epsilon=0.1 are reported as chosen 'after testing different configurations and comparing the resulting scores to human evaluations' on the same contracts used for the final results. This means the reported coverage scores are partly fitted to the evaluation set, not neutral measurements. The paper should either tune these weights on a separated validation set or provide a sensitivity analysis over a range of mu and epsilon. In addition, Figure 3 and Table 2 report mean coverage without error bars, confidence intervals, or significance tests; with only 30 synthetic contracts, the observed improvements over the baseline may not be statistically robust.","section":"§7.2, Eq. (1)"},{"comment":"The claim that the template-driven approach 'guarantees' 100% syntactic correctness and schema adherence is definitional rather than empirical: templates are constructed from schema keys, outputs are validated to have 'the exact same structure' as the input object, and empty fields are removed before finalization. As such, the 100% scores in Table 1 are not a meaningful experimental comparison for CDMizer. The paper should explicitly separate architectural invariants (e.g., 'the only outputs accepted are those conforming to the template structure') from empirical findings, and specify whether validation checks only JSON key structure or also CDM business rules and data-type constraints.","section":"§6.2, §7.1, Abstract"},{"comment":"The contribution list in Section 1 describes the baseline as an 'LLM and RAG pipeline', but Section 7 excludes fine-tuning from the baseline configurations due to token limitations and rigid behavior. The experiments therefore compare CDMizer not to the originally specified baseline but to a degraded few-shot-style variant. This makes the claimed superiority less informative. The authors should either evaluate the fine-tuned baseline on a subset of shorter contracts where it produces complete outputs, or explicitly redefine the baseline as a non-fine-tuned direct-generation method throughout the paper.","section":"§1, §7"}],"minor_comments":[{"comment":"Line 17 contains a typographical error ('preferences another schema' should be 'references another schema'), and the pseudocode would be clearer if it distinguished object-valued properties from array-valued properties when updating the path and resolving $ref.","section":"§6.1, Algorithm 1"},{"comment":"The depth threshold d is defined in prose as limiting processing to substructures with depth ≤ d, but the notion of depth is ambiguous: it is not specified whether depth counts JSON nesting levels, schema-reference levels, or the number of nodes along the longest path to a leaf. A formal definition is needed for reproducibility.","section":"§6.2, Figure 2"},{"comment":"The 'Syntactical Correctness' metric, defined as the proportion of generated keys that exist in the CDM schema, is redundant for CDMizer because its templates are built from schema keys; the paper should state explicitly how this metric applies to the baseline and whether missing keys are counted only when a key is present in the output but not in the schema.","section":"§7.1"},{"comment":"The abstract claims 'improved scalability', but no latency, token-usage, or throughput measurements are reported anywhere in the evaluation. Either provide such measurements for the depth-thresholded traversal or temper the scalability claim to 'enables processing long documents within token limits'.","section":"Abstract, §7.2"}],"recommendation":"major_revision","confidential_remarks":"The paper falls within the scope of cs.IR/application-oriented NLP, though its contribution is more an engineering framework than a novel algorithmic result. The main concern is the closed-world evaluation: synthetic contracts generated from the same examples that define the templates cannot validate generalizability. If the authors add an out-of-corpus evaluation and address the tuning/significance issues, the paper would be a reasonable contribution. I would not recommend rejection, because the core template-driven idea is sound and the limitations are at least partially acknowledged."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a sensible template-driven system for converting OTC derivative contract text into CDM JSON, and the paper is honest about what it proves and what it doesn't. The genuinely new piece is Algorithm 1: build a minimal template by flattening keys from example CDM files and pruning the schema to those paths, then populate it recursively via depth-bounded prompts. That is a practical idea, and the description is detailed enough to reproduce. The depth-thresholded hierarchical population is a clean answer to the token-limit problem. Credit is also due for stating plainly that 100% schema adherence is a consequence of template structure, not an empirical result.\n\nThe soft spots are real, and the stress-test note lands. The synthetic test contracts are generated from the same 858 CDM examples that define the template's field vocabulary, so every field a test contract needs is in the template by construction. The 89–91% semantic coverage therefore measures how well the LLM fills known slots, not how well the framework handles terms absent from the FINOS corpus. Section 7.2 acknowledges this and defers real-contract testing to future work, but the paper never quantifies the coverage ceiling. The LLM-judge metric is also fitted: mu and epsilon were selected after comparing configurations to human evaluation on the same data, and no error bars or confidence intervals are reported for coverage. The 30-contract sample is small and synthetic. None of this falsifies the central claim—the framework does seem to produce schema-compliant output on its test distribution—but the paper should be read as a feasibility demonstration, not a validated production tool.\n\nIt would help if the authors released the templates, the synthetic dataset, and the evaluation prompts; without artifacts the 100% adherence claim is plausible but uncheckable.\n\nBottom line: worth a serious referee. The audience is people building structured extraction for legal and financial documents, and the template-pruning idea is a contribution they should see. My recommendation is major revision: add out-of-corpus evaluation (even a handful of real term sheets), report variance in coverage, and release artifacts. If those land, this becomes a useful reference point for the field.","headline":"A practical template-driven CDM extraction system whose novelty is real but whose evaluation is closed-loop; worth revising and re-reviewing.","tokens_in":11452,"tokens_out":2638,"would_cite":true,"duration_ms":27809,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CDMizer demonstrates that template-driven, component-wise generation converts OTC derivative contracts into schema-compliant CDM representations with 100% syntactic correctness and schema adherence, plus higher semantic coverage than…","keywords":["CDMizer","Common Domain Model","OTC derivatives","retrieval-augmented generation","template-driven generation","contract encoding","large language models","semantic coverage"],"falsifier":"Take a genuine OTC contract containing a provision whose CDM field path is absent from the 858 example instances used to build the template, run CDMizer on it, and check whether the output contains that field; if it is missing and the semantic evaluation marks it uncaptured, the example-derived coverage limit is demonstrated.","tokens_in":10494,"feed_emoji":"📑","tokens_out":9041,"duration_ms":76221,"temperature":0.7,"pith_summary":"This paper claims that converting unstructured over-the-counter (OTC) derivative contracts into the machine-readable Common Domain Model (CDM) becomes reliable when generation is constrained by a deterministic template rather than attempted in one pass. The proposed framework, CDMizer, builds a minimal CDM template per contract type by pruning the full schema to fields that appear in example CDM instances, then populates that template component by component using an LLM with retrieval-augmented generation (RAG). In tests on 30 synthetic contracts across six product types, CDMizer achieved 100% syntactic correctness and schema adherence, and higher mean semantic coverage than direct LLM generation, with retrieval generally helping. The paper also contributes an LLM-powered evaluation method that scores semantic coverage by classifying extracted contract details as captured, uncaptured, or extraneous. A sympathetic reader would care because OTC contracts are currently processed largely by hand, and a scalable, schema-compliant conversion would open the door to automated compliance, trade processing, and lifecycle management.","feed_headline":"Template-driven CDMizer hits 100% schema adherence on OTC contracts","feed_subtitle":"CDMizer fills pruned CDM templates piece by piece with LLM and RAG, beating direct generation.","key_machinery":"The load-bearing object is the pruned CDM template produced by Algorithm 1. The algorithm flattens every example CDM instance into dot-separated keys, then recursively traverses the CDM JSON schema resolving $ref references and retaining only property paths that appear in at least one example. Each retained field receives a schema description and a type-appropriate placeholder, yielding a minimal but complete skeleton for a contract type. CDMizer then walks this tree with a depth threshold d (set to 4 in experiments), and for each sufficiently shallow subtree builds a prompt containing the object structure, its schema definition, the traversal path, and RAG-retrieved example chunks; the LLM fills the subtree, a validator enforces exact structural match, and empty fields are cleaned at the end. This machinery converts an intractable full-schema generation problem (over four million lines of JSON) into many small, locally constrained filling tasks.","core_discovery":"The central claim is that template-driven, hierarchical generation outperforms direct full-document LLM generation for contract-to-CDM conversion, and that this superiority is structural rather than incidental. Because the template is derived from the CDM schema and example instances, every generated object is forced to conform to the schema, so syntactic correctness and schema adherence are guaranteed by construction; semantic accuracy then depends on how well the LLM, guided by RAG-retrieved example chunks, populates each field. The paper reports mean semantic coverage of 89.40% without RAG and 91.40% with RAG for the model with the highest reported score, Llama-3.1-8B-Instruct, versus noticeably lower and more variable coverage for baseline direct generation. The authors state this establishes a scalable foundation for automated contract understanding and validation.","pith_inferences":["Because templates retain only fields that appear in example instances, any contract term outside that set is structurally absent, so real-world semantic coverage may fall even though syntactic correctness stays at 100%.","Because the test contracts were synthetic and generated from the same CDM examples used to build the templates, the reported semantic coverage likely overstates performance on genuine contracts with novel wording; running CDMizer on real term sheets would quantify the gap.","The coverage-score weights (uncaptured at 0.3, extraneous at 0.1) are choices, and method rankings could shift if those weights changed, so the metric deserves sensitivity analysis before high-stakes comparisons.","If CDM output is later compiled into executable smart contracts, the template's structural guarantee would suppress hallucinated-field bugs that plague direct LLM code generation, but the paper does not test this."],"forward_implications":["On the paper's tests, CDMizer guarantees 100% syntactic correctness and schema adherence for every contract type, so downstream systems can assume the CDM output is well-formed.","Component-wise generation removes the token-limit failure mode that truncates direct generation of long contracts, making the approach scalable to full-length agreements.","Retrieval augmentation generally raises semantic coverage for both baseline and CDMizer, so the RAG knowledge base of chunked CDM examples is a reusable asset for future contract types.","The template-creation algorithm makes the full CDM schema tractable by pruning away every field not exercised by example instances, so the generation task stays small.","The LLM-powered evaluation framework offers a repeatable protocol for measuring semantic coverage, with a weighted coverage score distinguishing captured, uncaptured, and extraneous information."],"supporting_citations":[{"why":"It supplies the 858 CDM example instances and schema definitions used to build the templates and the RAG knowledge base.","marker":"[FINOS, 2024a]"},{"why":"It defines CDM as the standardized machine-readable model that the generated output must conform to.","marker":"[FINOS, 2025]"},{"why":"It provides the design principles of CDM (normalization, composability, modularization) that justify the template structure.","marker":"[FINOS, 2024b]"},{"why":"It is one of the two real natural-language contracts used as a reference input for generating and evaluating the synthetic contracts.","marker":"[RBC Capital Markets, 2024]"},{"why":"It is the other real contract example used to ground the synthetic data generation and as test material.","marker":"[J.P. Morgan, 2024]"},{"why":"It is cited to support the claim that smaller, well-defined subtasks improve LLM effectiveness, motivating the depth-thresholded component-wise population.","marker":"[Karanjai et al., 2024]"},{"why":"It is cited as evidence that retrieval-augmented generation improves coherence and correctness, motivating the RAG integration in both baseline and CDMizer.","marker":"[Parvez et al., 2021]"}],"fun_headline_variants":["Template-driven CDMizer guarantees schema fit for OTC contracts","RAG-boosted CDMizer beats direct LLM on contract encoding","CDMizer: structural schema adherence over direct generation","LLM+RAG template approach hits 91.4% semantic coverage on CDM","Hierarchical templates outperform direct generation for CDM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The template retains only fields that appear in at least one available CDM example, so any contract term outside that example set is structurally omitted no matter how well the LLM performs.","fun_headline_variants_meta":{"raw":{"variants":["Template-driven CDMizer guarantees schema fit for OTC contracts","RAG-boosted CDMizer beats direct LLM on contract encoding","CDMizer: structural schema adherence over direct generation","LLM+RAG template approach hits 91.4% semantic coverage on CDM","Hierarchical templates outperform direct generation for CDM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2811,"prompt_tokens":883,"completion_tokens":1928,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":499,"completion_tokens_details":{"reasoning_tokens":1839}},"tokens_in":499,"tokens_out":1928,"duration_ms":14252,"temperature":1.0,"reasoning_tokens":1839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:51:14.382616+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a genuine OTC contract containing a provision whose CDM field path is absent from the 858 example instances used to build the template, run CDMizer on it, and check whether the output contains that field; if it is missing and the semantic evaluation marks it uncaptured, the example-derived coverage limit is demonstrated.","supporting_citations":[{"cited_title":"common-domain-model","cited_arxiv_id":null,"evidence_quote":"It defines CDM as the standardized machine-readable model that the generated output must conform to."},{"cited_title":"Structured rates presentation or document","cited_arxiv_id":null,"evidence_quote":"It is one of the two real natural-language contracts used as a reference input for generating and evaluating the synthetic contracts."},{"cited_title":"Solmover: Feasibility of using llms for translating smart contracts","cited_arxiv_id":null,"evidence_quote":"It is cited to support the claim that smaller, well-defined subtasks improve LLM effectiveness, motivating the depth-thresholded component-wise population."}],"review_version":1}