{"id":"70953d0e-60a9-4796-9512-8755c3971444","arxiv_id":"2507.21056","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A framework that fine-tunes LLaMA-2 with LoRA to generate JSON Schema data contracts, claiming 92% field accuracy on a 500-table test set.","lead":"This paper proposes using fine-tuned large language models to generate data contracts from sample data. The authors report high accuracy and a 70% workload reduction, but provide no code, data, or detailed experimental artifacts to verify these claims.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section VI's 92% field-accuracy result cannot be separated from possible training/test overlap: both the 2,000-example training corpus and the 500-table test set are drawn from public datasets, and no deduplication or held-out source is described.","rationale":"The reader's weakest assumption is that the evaluation is representative and free of leakage; my concern sharpens that into a specific contamination risk: both training and test sets are drawn from public datasets, and the paper does not describe any deduplication or source-separation. This is the most load-bearing issue because the paper's entire contribution is the empirical claim that fine-tuning produces accurate, syntactically valid data contracts. If the 92% figure collapses on a clean split, the title-level contribution is unsupported. The paper does include plausible architectural ideas and acknowledged limitations (small/medium tables, hallucination, human-in-the-loop), and I do not see an internal mathematical inconsistency in the methodology. However, the missing artifacts plus the training/test overlap risk justify the reader's REJECT verdict; the correct scientific disposition is to withhold acceptance until the experimental evidence is verifiable. I mark agreement as 'partial' because the reader framed the problem more broadly around representativeness and labels, while I emphasize the specific leakage/memorization channel and the post-processing ambiguity in the syntax metric.","tokens_in":8125,"tokens_out":3941,"duration_ms":45166,"concrete_test":"Ask the authors to release the exact list of the 2,000 training JSON schemas and the 500 test tables, or reconstruct them from the named public sources. Canonicalize table names, column-name sets, and column-type vectors, and compute overlap between the training and test sets. Then rerun the fine-tuned model on a held-out set of schemas from a source absent from both corpora (e.g., enterprise metadata or newly generated tables) and recompute field accuracy and raw, pre-fix syntax validity. If the numbers drop materially on the disjoint split, the 92% claim does not support the generalization claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the Section VI numbers: the fine-tuned LLaMA-2-7B model 'correctly captured an average of 92% of fields with correct types' and achieved '99%' syntax validity. For these numbers to support the contribution, the model must generalize to unseen data assets, not recall schemas encountered during training. The paper's own description does not rule out memorization. The training corpus is '2,000 public JSON schema definitions and their corresponding data examples' (Methodology), while the test set is '500 data tables with known schemas. These included public datasets (from Kaggle and UCI) and artificially generated tables' (Experiments). With public Kaggle/UCI sources on both sides, any overlap in table names, column-name sets, or type patterns can inflate the 92% figure even if the model never learned to infer types from data samples. The unfine-tuned baseline's 58% is not an adequate control because a base LLaMA model may fail to follow JSON formatting instructions, so the gap could reflect format compliance as much as domain learning. The syntax-validity metric is also ambiguous: the paper says 'few minor formatting issues automatically fixed,' meaning the 99% is not a measure of raw model output. No dataset, prompt templates, split definition, or evaluation code is provided, so the claimed superiority cannot be independently checked. This is not an internal inconsistency, but it is an unvalidated empirical assertion at the exact point where the paper's contribution lies.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an AI-driven framework for automatically generating data contracts (JSON Schema, Avro, SQL DDL) from table metadata and sample rows, using parameter-efficient fine-tuning (LoRA, prefix tuning) of an LLM such as LLaMA-2-7B. The authors describe a system architecture integrated with Databricks and Snowflake, report a case study of generating Great Expectations rules, and present experiments claiming 92% field accuracy and 99% syntax validity for a LoRA-fine-tuned model versus 58% and 60% for an unfine-tuned baseline. The abstract also claims a reduction of manual workload by over 70%.","tokens_in":8450,"tokens_out":4340,"duration_ms":43747,"significance":"The task of automating data contract generation is topical and practically relevant for data governance, and the idea of applying PEFT methods to structured output generation is sensible. If the reported numbers were backed by a rigorous, reproducible evaluation, the work would be of interest to the data-engineering community. However, as submitted, the central empirical claim is not verifiable because the evaluation protocol, dataset, prompt templates, and code are not provided, and the paper contains multiple structural and integrity problems. The contribution is therefore currently not established.","major_comments":[{"comment":"The central empirical result (92% field accuracy, 99% syntax validity) is not reproducible from the information given. The test set is 500 tables from public datasets (Kaggle, UCI) plus artificial tables, while the training corpus is 2,000 public JSON schema definitions, but no deduplication or leakage-prevention step is described. Because public sources appear on both sides, the 92% figure could reflect memorization of schemas rather than generalization. The authors should provide the dataset, prompt templates, exact train/test split, and evaluation code, or otherwise the result cannot be independently checked.","section":"Section VI, 'Data and Setup'"},{"comment":"The abstract claims the system reduces manual workload by over 70%, but no experiment in Section VI measures workload reduction. The reported metrics are field accuracy, syntax validity, and a human completeness score. If the workload-reduction claim is retained, it must be supported by an explicit measurement (e.g., time or effort saved in a controlled comparison).","section":"Abstract and Section VI, 'Results'"},{"comment":"The syntax-validity metric is not defined on raw model output. The sentence 'few minor formatting issues automatically fixed' indicates that post-processing was applied before validity was computed, so the 99% figure is not a faithful measure of the model's ability to emit well-formed JSON. The authors should report the validity rate before any automatic fixes, and also state exactly what fixes were applied.","section":"Section VI, 'Results'"},{"comment":"The baseline comparison against an unfine-tuned LLaMA model is not a clean control for domain learning. A base model may fail to follow JSON-formatting instructions, so part of the 92% vs 58% gap could be due to format compliance rather than learned knowledge of schemas and data types. A more informative control would include a few-shot prompt or constrained decoding for the base model, or an ablation with the same fine-tuning but shuffled labels.","section":"Section VI, 'Results'"},{"comment":"The manuscript references figures and a table that do not appear: 'Figure 1', 'Figure 2', 'Figure 3', and 'Table 1' are mentioned, but no figures or table are present in the text. The system architecture is a stated contribution, and the table comparing full fine-tuning, LoRA, and prefix tuning is essential to the efficiency claims. The duplicate 'Methodology' heading and raw LaTeX artifacts (e.g., 'XiXi', 'W[2032?]', 'd ×dd ×d') further indicate the manuscript is incomplete.","section":"Section IV and 'Methodology' (duplicate section)"},{"comment":"The reference list contains entries that appear unverifiable or incorrect: reference [22] is given as 'J. Doe et al.', which is a placeholder name, and reference [12] attributes the LoRA paper to 'Edward J. Wang et al.' with the wrong arXiv identifier (the LoRA paper is by Hu et al., arXiv:2106.09685). In a journal submission, the related-work claims must be grounded in real, correctly cited sources; as written, the reliability of the literature basis is in question.","section":"References"}],"minor_comments":[{"comment":"The text contains many typos and spacing errors, e.g., 'labour-intensive' vs 'labor-intensive', 'accuracyingenerating', and missing spaces between words. A thorough copy-edit is needed.","section":"Throughout"},{"comment":"The description of the training corpus is vague: '2,000 public JSON schema definitions and their corresponding data examples' should be accompanied by a precise listing of sources and preprocessing steps, and the 80/20 train/val split mentioned in Section VI should be described consistently with this section.","section":"Section III, 'Fine-Tuning Data and Tasks'"},{"comment":"No error bars, confidence intervals, or statistical tests are provided for any of the reported accuracies; given the small test set (500 tables), the differences between models should be accompanied by uncertainty estimates.","section":"Section VI, 'Data and Setup'"},{"comment":"The Databricks and Snowflake case studies are described qualitatively, without code, configuration, or measured outcomes. It is unclear what 'prototype' means in this context and whether any real deployment occurred.","section":"Section V, 'Case Study'"},{"comment":"Several references are to blog posts or vendor documentation rather than peer-reviewed work; for a journal, the authors should strengthen the academic grounding where possible (e.g., citing published work on schema inference and LLM-based table understanding).","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript appears not to have been prepared to journal standards: it contains duplicated sections, missing figures and tables, raw LaTeX artifacts, and a reference list with placeholder entries (e.g., 'J. Doe et al.'). The central empirical claim is not reproducible from the submitted materials, and there is no external artifact to check. In my view this is a desk-reject-level concern rather than a question that can be resolved in a normal revision cycle, because the evaluation would need to be redone from scratch with a published dataset and code."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is an applied paper saying you can fine-tune LLaMA-2 with LoRA to generate JSON Schema data contracts from table samples, with 92% field accuracy and 99% syntax validity. The idea is plausible and the system description is straightforward, but the evidence is too thin to trust the numbers, and the citation list contains what look like fabricated references.\n\nWhat's new: the task framing—LLM-generated data contracts—is a legitimate extension of schema inference and structured output work. The paper describes a sensible pipeline: metadata extraction, LoRA fine-tuning, validation and post-processing, and integration with Databricks and Snowflake. That system sketch is genuinely useful for practitioners thinking about automating contract authoring. The LoRA vs prefix-tuning comparison (92% vs 88%) is the kind of detail that helps people.\n\nSoft spots, in proportion. The central claim is the 92%/99% result in Section VI, but the evaluation is not reproducible: no code, no dataset, no prompt templates, no error bars, and no clear split between the 2,000 training examples and the 500 test tables. Since both draw on public Kaggle/UCI sources, train/test overlap is a real risk—the model could be memorizing schemas rather than generalizing. The unfine-tuned baseline (58%) is also a weak control: base LLaMA may just be bad at following JSON format instructions, so the gap could be format compliance, not domain learning. The abstract's 'reduce manual workload by over 70%' never appears in the experiments. The syntax-validity metric is inflated by 'minor formatting issues automatically fixed'—that's not raw model output.\n\nBigger red flag: references [22]–[25] look invented. 'J. Doe et al.' in [22] and 'H. Smith and L. Nguyen' in [23] are placeholder names, and the venues don't check out. That alone makes me trust the surrounding scholarship less.\n\nWho it's for: practitioners evaluating whether LLMs can draft contracts might get useful context from the architecture and the LoRA/prefix-tuning comparison. As a research contribution, it's not there yet.\n\nRecommendation: if this came to my desk, I'd desk-reject it until the authors either make the data and code available or substantially deepen the evaluation. The idea has merit, but the current evidence doesn't justify referee time.","headline":"Plausible engineering story, but the evaluation is not reproducible and the reference list has integrity problems.","tokens_in":8903,"tokens_out":2951,"would_cite":false,"duration_ms":29642,"reading_group":"maybe","serious_thinker":"unclear","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A LoRA-fine-tuned LLM can generate data contracts that capture 92% of fields with correct types, versus 58% without fine-tuning.","keywords":["data contracts","large language models","LoRA fine-tuning","PEFT","JSON Schema","schema inference","data governance","structured output generation"],"falsifier":"A concrete check: release the training set, the 500 test tables, the prompt templates, and the evaluation script, then re-run the fine-tuned and baseline models on a test split that was never shown during training; if the fine-tuned model's field accuracy falls to the baseline's level or its syntax validity drops far below 99% once leakage and label errors are removed, the central claim is refuted.","tokens_in":7952,"feed_emoji":"📋","tokens_out":7660,"duration_ms":68127,"temperature":0.7,"pith_summary":"The paper claims that fine-tuning a large language model with parameter-efficient methods (LoRA and PEFT) turns it into a reliable data-contract generator: given column names and sample rows, it outputs a JSON Schema or Avro contract with the right fields and types. The reported experiments put the fine-tuned LLaMA-2 model at 92% field-and-type accuracy and 99% syntax validity, against 58% and 60% for the same model without fine-tuning, with human reviewers scoring drafts 4.7/5. These numbers, if they hold, mean data teams could automate most contract authoring and focus their edits on descriptions and quality constraints rather than writing contracts from scratch. The paper also sketches an architecture for embedding such a contract engine in modern data platforms so generated contracts are validated and enforced before producers' schema changes break downstream consumers.","feed_headline":"Fine-tuned LLMs write data contracts with 92% field accuracy","feed_subtitle":"Same model without tuning scores 58%; the tuned version hits 99% syntax validity.","key_machinery":"The load-bearing mechanism is the LoRA fine-tuning procedure applied to a base LLM, combined with prompt formatting and post-hoc validation. LoRA freezes the transformer weights and learns low-rank update matrices $A$ and $B$ added to the attention weight matrices, so only a few million parameters are trained; the paper uses rank $r=8$ on a 7B LLaMA model. Around this, the pipeline uses instruction-style prompts (given a table description and sample rows, output a JSON schema), validation during training that penalizes invalid JSON, and a post-generation validator that parses and repairs output, with fallback to a generic safe contract. The machinery carries the argument by tying the accuracy gains to the fine-tuning method rather than to prompt engineering alone.","core_discovery":"On the paper's own terms, the central discovery is that a 7-billion-parameter LLaMA model fine-tuned with LoRA on roughly 2,000 paired examples of data assets and their JSON-schema contracts becomes a competent automatic author of data contracts. It maps an input prompt containing table name, column names, and sample rows to a structured contract, and the experiments report that it captures 92% of fields with correct types (baseline: 58%), produces syntactically valid JSON 99% of the time (baseline: 60%), and earns 4.7/5 from human evaluators. The paper further reports that LoRA training took about 4 hours on a single GPU and about 5x less time than full fine-tuning, with full fine-tuning reaching only slightly higher accuracy, and that prefix tuning reached 88% field accuracy. These results are presented as evidence that parameter-efficient fine-tuning makes practical, scalable, AI-driven contract generation viable for enterprise data governance.","pith_inferences":["I infer that the same LoRA-tuned pipeline would transfer to other structured metadata outputs, such as Avro schemas, GraphQL types, or data-quality expectation suites, because the paper already treats contract languages as pluggable and the underlying task is the same formatted-text generation.","I infer that the reported 92% is an in-distribution number: the test tables come from the same public and synthetic sources as the 2,000 training examples, so out-of-distribution schemas would likely score lower, and the paper's own discussion of hallucinated field types on totally new schemas supports this.","A targeted ablation could isolate whether the validation-during-training penalty or the prompt template is responsible for the 99% syntax validity; the paper does not report that separation, so the mechanism behind the headline number is an open question.","The paper's reported 2% gain from chain-of-thought prompting suggests a testable extension: on wide schemas with hundreds of columns, where long prompts are a noted limitation, first generating a field list and then feeding it back to produce the full contract may yield larger gains."],"forward_implications":["If the accuracy figures hold, a data platform can generate first-draft contracts for new tables automatically, leaving engineers to review rather than author from scratch.","The 99% syntax validity means generated contracts can be parsed and enforced by existing validators without heavy repair logic, making the automation plug into current tooling.","Because LoRA achieves near full-fine-tune accuracy at roughly a fifth of the training time, organizations can re-tune contract models for new domains on modest hardware and update them as standards evolve.","The reported 4.7/5 human score suggests that human-in-the-loop review, not manual writing, becomes the bottleneck procedure for contract quality.","A contract engine integrated with a metadata catalog and validator can close the loop: schema changes produce contract mismatches that alert downstream consumers before breakage."],"supporting_citations":[{"why":"Supplies the LoRA method of freezing base weights and training low-rank adapters, which is the central adaptation technique in the experiments.","marker":"[12]"},{"why":"Provides the PEFT survey basis for claiming parameter-efficient fine-tuning reaches near full-fine-tune performance with far fewer trainable parameters.","marker":"[11]"},{"why":"Defines the LLaMA-2 7B base model used for both the fine-tuned and baseline experimental runs.","marker":"[17]"},{"why":"Establishes that LLMs can infer column types and hierarchies for tabular data, the prior result this work extends to full data contracts.","marker":"[8]"},{"why":"Supplies the data-quality tooling context used in the case study where the model generates expectation rules for a dataset.","marker":"[4]"},{"why":"Motivates the benefit of fine-tuning on domain-specific data and informs the case-study implementation in a modern data platform.","marker":"[6]"},{"why":"Justifies the secure deployment pattern of running the model near the data storage, used in the Snowflake case study.","marker":"[7]"},{"why":"Informs the structured-output prompting techniques, such as templates and constrained decoding, that keep generated contracts valid JSON.","marker":"[19]"}],"fun_headline_variants":["LLMs auto-write data contracts: 92% field accuracy, 70% less work","AI-generated data contracts: 99% valid syntax, 4.7/5 human rating","LoRA-tuned 7B LLM writes contracts 5x faster to train","AI drafts data contracts, humans approve at 4.7/5","LLM fine-tuned on 2k examples cuts contract effort 70%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation rests on the assumption that the 500 test tables and their ground-truth schemas are representative and correctly labeled, and that the reported accuracy and syntax numbers were computed on a clean split with no leakage; because the paper gives no dataset, prompt templates, or evaluation code, this assumption cannot be checked.","fun_headline_variants_meta":{"raw":{"variants":["LLMs auto-write data contracts: 92% field accuracy, 70% less work","AI-generated data contracts: 99% valid syntax, 4.7/5 human rating","LoRA-tuned 7B LLM writes contracts 5x faster to train","AI drafts data contracts, humans approve at 4.7/5","LLM fine-tuned on 2k examples cuts contract effort 70%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2602,"prompt_tokens":923,"completion_tokens":1679,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1568}},"tokens_in":539,"tokens_out":1679,"duration_ms":11966,"temperature":1.0,"reasoning_tokens":1568,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:01:21.980645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: release the training set, the 500 test tables, the prompt templates, and the evaluation script, then re-run the fine-tuned and baseline models on a test split that was never shown during training; if the fine-tuned model's field accuracy falls to the baseline's level or its syntax validity drops far below 99% once leakage and label errors are removed, the central claim is refuted.","supporting_citations":[{"cited_title":"Decoding the drive-bath interplay: A guideline to enhance superconductivity","cited_arxiv_id":"2306.02861","evidence_quote":"Supplies the LoRA method of freezing base weights and training low-rank adapters, which is the central adaptation technique in the experiments."},{"cited_title":"Taxonomy Inference for Tabular Data Using Large Language Models","cited_arxiv_id":"2503.21810","evidence_quote":"Establishes that LLMs can infer column types and hierarchies for tabular data, the prior result this work extends to full data contracts."},{"cited_title":"The Power of Fine- Tuning on Your Data: Quick Fix via Never-Ending Learning","cited_arxiv_id":null,"evidence_quote":"Motivates the benefit of fine-tuning on domain-specific data and informs the case-study implementation in a modern data platform."},{"cited_title":"Building a Data-Centric Platform for Generative AI and LLMs at Snowflake","cited_arxiv_id":null,"evidence_quote":"Justifies the secure deployment pattern of running the model near the data storage, used in the Snowflake case study."},{"cited_title":"Generating Struc- tured Output from LLMs","cited_arxiv_id":null,"evidence_quote":"Informs the structured-output prompting techniques, such as templates and constrained decoding, that keep generated contracts valid JSON."}],"review_version":1}