REVIEW 6 major objections 5 minor 23 references
AI-Driven Generation of Data Contracts in Modern Data Engineering Systems
T0 review · 6 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A LoRA-fine-tuned LLM can generate data contracts that capture 92% of fields with correct types, versus 58% without fine-tuning.
desk verdict Plausible engineering story, but the evaluation is not reproducible and the reference list has integrity problems. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the LoRA fine-tuning procedure applied to a base LLM, combined with prompt formatting and post-hoc validation. LoRA freezes the transformer weights and learns low-rank update matrices $A$ and $B$ added to the attention weight matrices, so only a few million parameters are trained; the paper uses rank $r=8$ on a 7B LLaMA model. Around this, the pipeline uses instruction-style prompts (given a table description and sample rows, output a JSON schema), validation during training that penalizes invalid JSON, and a post-generation validator that parses and repairs output, with fallback to a generic safe contract. The machinery carries the argument by tying the accuracy gains to the fine-tuning method rather than to prompt engineering alone.
What would settle it
A concrete check: release the training set, the 500 test tables, the prompt templates, and the evaluation script, then re-run the fine-tuned and baseline models on a test split that was never shown during training; if the fine-tuned model's field accuracy falls to the baseline's level or its syntax validity drops far below 99% once leakage and label errors are removed, the central claim is refuted.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a 7-billion-parameter LLaMA model fine-tuned with LoRA on roughly 2,000 paired examples of data assets and their JSON-schema contracts becomes a competent automatic author of data contracts. It maps an input prompt containing table name, column names, and sample rows to a structured contract, and the experiments report that it captures 92% of fields with correct types (baseline: 58%), produces syntactically valid JSON 99% of the time (baseline: 60%), and earns 4.7/5 from human evaluators. The paper further reports that LoRA training took about 4 hours on a single GPU and about 5x less time than full fine-tuning, with full fine-tuning reaching only slightly higher accuracy, and that prefix tuning reached 88% field accuracy. These results are presented as evidence that parameter-efficient fine-tuning makes practical, scalable, AI-driven contract generation viable for enterprise data governance.
Load-bearing premise
The evaluation rests on the assumption that the 500 test tables and their ground-truth schemas are representative and correctly labeled, and that the reported accuracy and syntax numbers were computed on a clean split with no leakage; because the paper gives no dataset, prompt templates, or evaluation code, this assumption cannot be checked.
Editorial extensions
If this is right
- If the accuracy figures hold, a data platform can generate first-draft contracts for new tables automatically, leaving engineers to review rather than author from scratch.
- The 99% syntax validity means generated contracts can be parsed and enforced by existing validators without heavy repair logic, making the automation plug into current tooling.
- Because LoRA achieves near full-fine-tune accuracy at roughly a fifth of the training time, organizations can re-tune contract models for new domains on modest hardware and update them as standards evolve.
- The reported 4.7/5 human score suggests that human-in-the-loop review, not manual writing, becomes the bottleneck procedure for contract quality.
- A contract engine integrated with a metadata catalog and validator can close the loop: schema changes produce contract mismatches that alert downstream consumers before breakage.
Reading between the lines
- I infer that the same LoRA-tuned pipeline would transfer to other structured metadata outputs, such as Avro schemas, GraphQL types, or data-quality expectation suites, because the paper already treats contract languages as pluggable and the underlying task is the same formatted-text generation.
- I infer that the reported 92% is an in-distribution number: the test tables come from the same public and synthetic sources as the 2,000 training examples, so out-of-distribution schemas would likely score lower, and the paper's own discussion of hallucinated field types on totally new schemas supports this.
- A targeted ablation could isolate whether the validation-during-training penalty or the prompt template is responsible for the 99% syntax validity; the paper does not report that separation, so the mechanism behind the headline number is an open question.
- The paper's reported 2% gain from chain-of-thought prompting suggests a testable extension: on wide schemas with hundreds of columns, where long prompts are a noted limitation, first generating a field list and then feeding it back to produce the full contract may yield larger gains.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an AI-driven framework for automatically generating data contracts (JSON Schema, Avro, SQL DDL) from table metadata and sample rows, using parameter-efficient fine-tuning (LoRA, prefix tuning) of an LLM such as LLaMA-2-7B. The authors describe a system architecture integrated with Databricks and Snowflake, report a case study of generating Great Expectations rules, and present experiments claiming 92% field accuracy and 99% syntax validity for a LoRA-fine-tuned model versus 58% and 60% for an unfine-tuned baseline. The abstract also claims a reduction of manual workload by over 70%.
Significance. The task of automating data contract generation is topical and practically relevant for data governance, and the idea of applying PEFT methods to structured output generation is sensible. If the reported numbers were backed by a rigorous, reproducible evaluation, the work would be of interest to the data-engineering community. However, as submitted, the central empirical claim is not verifiable because the evaluation protocol, dataset, prompt templates, and code are not provided, and the paper contains multiple structural and integrity problems. The contribution is therefore currently not established.
major comments (6)
- [Section VI, 'Data and Setup'] The central empirical result (92% field accuracy, 99% syntax validity) is not reproducible from the information given. The test set is 500 tables from public datasets (Kaggle, UCI) plus artificial tables, while the training corpus is 2,000 public JSON schema definitions, but no deduplication or leakage-prevention step is described. Because public sources appear on both sides, the 92% figure could reflect memorization of schemas rather than generalization. The authors should provide the dataset, prompt templates, exact train/test split, and evaluation code, or otherwise the result cannot be independently checked.
- [Abstract and Section VI, 'Results'] The abstract claims the system reduces manual workload by over 70%, but no experiment in Section VI measures workload reduction. The reported metrics are field accuracy, syntax validity, and a human completeness score. If the workload-reduction claim is retained, it must be supported by an explicit measurement (e.g., time or effort saved in a controlled comparison).
- [Section VI, 'Results'] The syntax-validity metric is not defined on raw model output. The sentence 'few minor formatting issues automatically fixed' indicates that post-processing was applied before validity was computed, so the 99% figure is not a faithful measure of the model's ability to emit well-formed JSON. The authors should report the validity rate before any automatic fixes, and also state exactly what fixes were applied.
- [Section VI, 'Results'] The baseline comparison against an unfine-tuned LLaMA model is not a clean control for domain learning. A base model may fail to follow JSON-formatting instructions, so part of the 92% vs 58% gap could be due to format compliance rather than learned knowledge of schemas and data types. A more informative control would include a few-shot prompt or constrained decoding for the base model, or an ablation with the same fine-tuning but shuffled labels.
- [Section IV and 'Methodology' (duplicate section)] The manuscript references figures and a table that do not appear: 'Figure 1', 'Figure 2', 'Figure 3', and 'Table 1' are mentioned, but no figures or table are present in the text. The system architecture is a stated contribution, and the table comparing full fine-tuning, LoRA, and prefix tuning is essential to the efficiency claims. The duplicate 'Methodology' heading and raw LaTeX artifacts (e.g., 'XiXi', 'W[2032?]', 'd ×dd ×d') further indicate the manuscript is incomplete.
- [References] The reference list contains entries that appear unverifiable or incorrect: reference [22] is given as 'J. Doe et al.', which is a placeholder name, and reference [12] attributes the LoRA paper to 'Edward J. Wang et al.' with the wrong arXiv identifier (the LoRA paper is by Hu et al., arXiv:2106.09685). In a journal submission, the related-work claims must be grounded in real, correctly cited sources; as written, the reliability of the literature basis is in question.
minor comments (5)
- [Throughout] The text contains many typos and spacing errors, e.g., 'labour-intensive' vs 'labor-intensive', 'accuracyingenerating', and missing spaces between words. A thorough copy-edit is needed.
- [Section III, 'Fine-Tuning Data and Tasks'] The description of the training corpus is vague: '2,000 public JSON schema definitions and their corresponding data examples' should be accompanied by a precise listing of sources and preprocessing steps, and the 80/20 train/val split mentioned in Section VI should be described consistently with this section.
- [Section VI, 'Data and Setup'] No error bars, confidence intervals, or statistical tests are provided for any of the reported accuracies; given the small test set (500 tables), the differences between models should be accompanied by uncertainty estimates.
- [Section V, 'Case Study'] The Databricks and Snowflake case studies are described qualitatively, without code, configuration, or measured outcomes. It is unclear what 'prototype' means in this context and whether any real deployment occurred.
- [References] Several references are to blog posts or vendor documentation rather than peer-reviewed work; for a journal, the authors should strengthen the academic grounding where possible (e.g., citing published work on schema inference and LLM-based table understanding).
Circularity Check
No significant circularity: the central results are empirically measured against an unfine-tuned baseline, and no load-bearing self-citation or by-construction equivalence was found.
full rationale
This paper makes an empirical rather than a formal-derivation claim: the contribution is that a LoRA-fine-tuned LLaMA-2-7B model generates data contracts with measured field accuracy, syntax validity, and human-evaluation scores. The reported 92% field accuracy and 4.7/5 human scores are compared against an unfine-tuned baseline and are not derived from the training data by construction. The LoRA/PEFT methodology is supported by independent standard references and is not used to define the evaluation metrics. Potential circularity candidates were checked. First, the training corpus and test set both draw on public datasets, which creates a possible train/test-overlap or memorization risk, but the paper does not state that the same tables are reused, and this is a reproducibility/leakage concern rather than a circular definition. Second, the syntax-validity figure is qualified by 'few minor formatting issues automatically fixed' and the methodology describes auto-fixing invalid outputs; this weakens the syntax metric but does not force the central field-accuracy claim, which remains an externally measured outcome. Third, no load-bearing self-citations were found; the cited prior work is independent or standard methodology. Therefore no circularity score above 0 is warranted; the significant caveats belong to empirical transparency and reproducibility rather than circular reasoning.
Assumptions & free parameters
free parameters (3)
- LoRA rank =
8
- Number of training epochs =
3
- Prefix-tuning token count =
100
assumptions (4)
- domain assumption LLMs can learn to emit structured outputs (such as JSON Schema) from prompt and fine-tuning examples.
- domain assumption LoRA and PEFT fine-tuning preserve most of the performance of full fine-tuning.
- domain assumption The 500-table test set is representative of real-world data contract generation scenarios.
- domain assumption JSON Schema and Avro are adequate representations for data contracts.
Cite this review
Pith. "Pith review of AI-Driven Generation of Data Contracts in Modern Data Engineering Systems." pith.science (2026). https://pith.science/paper/DHEM7HAC
@misc{pith2026250721056,
author = {Pith},
title = {Pith review of: AI-Driven Generation of Data Contracts in Modern Data Engineering Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/DHEM7HAC}},
note = {Machine review of arXiv:2507.21056}
}
read the original abstract
Data contracts formalize agreements between data producers and consumers regarding schema, semantics, and quality expectations. As data pipelines grow in complexity, manual authoring and maintenance of contracts becomes error-prone and labor-intensive. We present an AI-driven framework for automatic data contract generation using large language models (LLMs). Our system leverages parameter-efficient fine-tuning methods, including LoRA and PEFT, to adapt LLMs to structured data domains. The models take sample data or schema descriptions and output validated contract definitions in formats such as JSON Schema and Avro. We integrate this framework into modern data platforms (e.g., Databricks, Snowflake) to automate contract enforcement at scale. Experimental results on synthetic and real-world datasets demonstrate that the fine-tuned LLMs achieve high accuracy in generating valid contracts and reduce manual workload by over 70%. We also discuss key challenges such as hallucination, version control, and the need for continuous learning. This work demonstrates that generative AI can enable scalable, agile data governance by bridging the gap between intent and implementation in enterprise data management.
Figures
Reference graph
Works this paper leans on
-
[22]
DataOps and Generative AI: A Survey
J. Doe et al., “DataOps and Generative AI: A Survey.” IEEE Transactions on Knowledge and Data Engineering, 2024[22]
work page 2024
-
[25]
Schema Inference in Modern Data Lakes
M. Abadi et al., “Schema Inference in Modern Data Lakes.” Proc. IEEE BigData, 2023[25]. 7
work page 2023
-
[23]
Data Contracts: A Foundation for Data Products
H. Smith and L. Nguyen, “Data Contracts: A Foundation for Data Products.” Proc. SIGMOD Record, 2022[23]
work page 2022
-
[12]
Decoding the drive-bath interplay: A guideline to enhance superconductivity
Edward J. Wang et al., “LoRA: Low- Rank Adaptation of Large Language Models.” arXiv:2306.02861, 2023[12]
work page Pith review arXiv 2023
-
[1]
Data Contracts 101: What They Are, Why They Matter & How to Implement Them. Atlan, 2024[18]
work page 2024
-
[2]
John Kutay, “A Guide to Data Contracts.” Striim Blog, 2023[2]
work page 2023
-
[3]
Data Contracts for Schema Reg- istry
Confluent Inc., “Data Contracts for Schema Reg- istry.” Confluent Documentation, 2023[3]. [4]VinothKandA.James, “ASurveyonDataQual- ity Dimensions and Tools for Machine Learning.” arXiv preprint arXiv:2406.19614, 2024[4]
arXiv 2023
-
[5]
Use Gemini to ingest and understand data
“Use Gemini to ingest and understand data.” Google Cloud Blog, 2023[5]
work page 2023
Show all 23 references
-
[6]
The Power of Fine- Tuning on Your Data: Quick Fix via Never-Ending Learning
Samantha Banchik et al., “The Power of Fine- Tuning on Your Data: Quick Fix via Never-Ending Learning.” Databricks Mosaic AI Research Blog, April 2025[6]
2025
-
[7]
Building a Data-Centric Platform for Generative AI and LLMs at Snowflake
Yao Zhang, “Building a Data-Centric Platform for Generative AI and LLMs at Snowflake.” Snowflake Blog, 2023[7]
2023
-
[8]
Tax- onomy Inference for Tabular Data Using Large Lan- guage Models
Zhenyu Wu, Jiaoyan Chen, Norman Paton, “Tax- onomy Inference for Tabular Data Using Large Lan- guage Models.” Proc. ArXiv 2503.21810, 2025[8]
2025 arXiv
-
[9]
Generative AI
Stefan Feuerriegel, Jochen Hartmann, et al., “Generative AI.” Business & Information Systems En- gineering, Sept. 2023[9]
2023
-
[10]
Attention Is All You Need
A. Vaswani et al., “Attention Is All You Need.” Proc. NeurIPS, 2017[10]
2017
-
[11]
Parameter-Efficient Fine- Tuning for Large Models: A Comprehensive Survey
Zeyu Han et al., “Parameter-Efficient Fine- Tuning for Large Models: A Comprehensive Survey.” arXiv:2403.14608, 2024[11]
2024 arXiv
-
[13]
BERT: Pre-training of Deep Bidirectional Transformers for Language Under- standing
Jacob Devlin et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Under- standing.” NAACL, 2019[13]
2019
-
[14]
Improving Language Un- derstanding by Generative Pre-Training
Alec Radford et al., “Improving Language Un- derstanding by Generative Pre-Training.” OpenAI Tech Report, 2018[14]
2018
-
[15]
Language Models are Few-Shot Learners
Tom B. Brown et al., “Language Models are Few-Shot Learners.” NeurIPS, 2020[15]
2020
-
[16]
Exploring the Limits of Transfer Learning with a Text-to-Text Transformer (T5)
Colin Raffel et al., “Exploring the Limits of Transfer Learning with a Text-to-Text Transformer (T5).” J. of Machine Learning Research, 2020[16]
2020
-
[17]
LLaMA: Open and Effi- cient Foundation Language Models
Hugo Touvron et al., “LLaMA: Open and Effi- cient Foundation Language Models.” arXiv:2302.13971, 2023[17]
2023 arXiv
-
[18]
A Survey on the Structure of Data Contracts
Kirill Shokhin, “A Survey on the Structure of Data Contracts.” Atlan Blog, Dec 2024[18]
2024
-
[19]
Generating Struc- tured Output from LLMs
Christopher Brooks et al., “Generating Struc- tured Output from LLMs.” Quasilinear Musings Blog, 2024[19]
2024
-
[20]
InformationExtractionwith LLMs
OpenAI GPT-4 Technical Report, 2023[20]. [21]MikeMicheletal., “InformationExtractionwith LLMs.” arXiv:2402.01234, 2024[21]
2023 arXiv
-
[24]
An Experimental Evalu- ation of LLMs for Data Pipeline Automation
P. Carpenter et al., “An Experimental Evalu- ation of LLMs for Data Pipeline Automation.” VLDB Conf. Workshop, 2023[24]
2023
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.