REVIEW 3 major objections 7 minor 19 references
Property-Driven Synthetic Data Engineering for Data-Scarce Software Systems: Reflections from the Breast Cancer Domain
T0 review · 3 major / 7 minor · reviewed 2026-07-08 · glm-5.2
Pith's one-line read Synthetic data shifts the problem, not solves it
desk verdict Agenda-setting reflection on synthetic data as an SE problem; the central novelty claim is undercut by its own cited prior work. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the property set P — stakeholder-specific validity properties that synthetic data must satisfy — and the engineering lifecycle around it: elicitation, formalization, checking, trade-off analysis, and evolution. The paper also introduces a stakeholder-to-property mapping (Table 1) that connects different roles to their desired properties, example checks, and automation opportunities.
What would settle it
If synthetic data generators could be shown to automatically satisfy all stakeholder-relevant properties without explicit elicitation or trade-off — for instance, through sufficiently advanced generation techniques that preserve clinical constraints, statistical distributions, and privacy simultaneously — then the property-driven engineering process would reduce to generator selection, undermining the paper's central claim.
Extended reading notes
Core claim
The paper's central claim is that synthetic data generation in data-scarce domains is fundamentally a software engineering problem, not a data preprocessing step. The authors formalize this by defining synthetic data generation as producing a dataset D' that, when processed by an analysis A, yields results R' that must satisfy a set of properties P. These properties are stakeholder-dependent and potentially conflicting: a clinician needs clinical plausibility, a data scientist needs statistical fidelity, a privacy officer needs disclosure protection, and a software engineer needs test adequacy. The paper demonstrates through its IORT case study that different generators (CTGAN, TVAE, TabDDPM
Load-bearing premise
The paper assumes that framing synthetic data validation as a property-driven software engineering problem constitutes a new research paradigm, rather than an application of existing requirements engineering and data validation practices to a new artifact type. If property-driven validation of synthetic data turns out to be routine requirements engineering applied to synthetic datasets, the paper's contribution narrows from a new paradigm to a domain-specific case study.
Editorial extensions
If this is right
- If the paper is right, then selecting a synthetic data generator is not an ML model-selection problem but a requirements engineering problem: the choice depends on which stakeholder properties matter for the target task, not on aggregate similarity metrics.
- If validity is inherently multi-property and properties conflict, then synthetic data pipelines need explicit trade-off mechanisms — analogous to multi-objective optimization — rather than single-metric evaluation.
- If data cleaning decisions determine which properties can later be synthesized and validated, then data cleaning becomes part of the system specification, not a disposable preprocessing step.
- If synthetic data pipelines must evolve as populations, treatments, and regulations change, then continuous monitoring and drift detection of validity properties become a maintenance concern comparable to software regression testing.
- If the property-driven framing generalizes beyond medicine, then any data-scarce, privacy-constrained domain (e.g., safety-critical systems, regulated industries) faces the same shift from data-driven to property-driven validation.
Reading between the lines
- The paper's framing implies that existing requirements engineering techniques (e.g., goal-oriented modeling, conflict detection) could be adapted to elicit and resolve property conflicts in synthetic data pipelines, though the paper does not make this connection explicit.
- The stakeholder-to-property mapping in Table 1 suggests a formalization where each property could be encoded as a constraint or specification, enabling automated verification — but the paper stops short of proposing a concrete specification language.
- The observation that statistical similarity does not imply clinical validity hints at a deeper separation between distributional fidelity and semantic fidelity that could apply to any domain where synthetic data must satisfy domain-specific constraints.
- The evolution challenge implies a need for versioned property specifications tied to data versions, analogous to how software specifications are versioned alongside code — a direction the paper gestures toward but does not develop.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes framing synthetic data generation (SDG) in data-scarce, privacy-constrained software systems as a property-driven engineering problem. Using a breast cancer intraoperative radiotherapy (IORT) case study, the authors argue that SDG shifts rather than solves the central engineering challenge: the key difficulty becomes eliciting, formalizing, checking, and evolving stakeholder-specific validity properties (e.g., statistical fidelity, clinical plausibility, privacy protection, task utility) rather than merely generating statistically similar data. The paper reports preliminary experiments with five tabular SDG techniques on a 709-patient dataset, identifies four lessons (L1–L4), and outlines a research agenda for automated software engineering support. The paper is positioned as a reflection and agenda-setting piece, not as a method validation or empirical benchmark.
Significance. The paper addresses a timely and practically important problem at the intersection of software engineering and medical informatics. Its central observation—that synthetic data validity is inherently multi-property, stakeholder-dependent, and potentially conflicting—is well-motivated and illustrated with a real clinical collaboration. The mapping in Table 1 connecting stakeholders to properties, checks, and automation opportunities is a useful concrete artifact that could guide future tool development. The preliminary experiments, while not claiming clinical validity, effectively demonstrate the gap between statistical plausibility and clinical validity. However, the paper's novelty claim rests on the assertion that this constitutes a distinct SE research agenda, and this claim is not yet sufficiently differentiated from existing work in the medical informatics and synthetic data validation literature. The paper does not ship reproducible code or machine-checked artifacts, which is expected for a reflection piece but limits verifiability.
major comments (3)
- §5 (Related Work) and §6 (Conclusion): The paper's central novelty claim is that SDG introduces an SE problem not adequately addressed by current methodologies, specifically that stakeholders must define, trade off, and evolve validity properties. However, the paper itself cites Zazzetti et al. [18], which proposes the SAFE framework for synthetic breast cancer data that already assesses fidelity, privacy, and clinical utility — the exact multi-property validation space the paper claims is unaddressed. The paper does not specify what concrete technical capability is missing from SAFE and similar frameworks (e.g., automated property conflict detection, cross-stakeholder trade-off resolution, pipeline evolution monitoring) that would require new SE methods. Without this gap analysis, the claim that there is a distinct SE research agenda beyond what medical informatics already provides is a
- §4, Table 1: The mapping of stakeholders to properties, checks, and automation opportunities is the most concrete contribution, but the automation opportunities listed (e.g., 'constraint mining,' 'rule checking,' 'property-based test generation,' 'drift detection') are presented at a very high level without connecting to the state of the art in each sub-area. For the agenda to be actionable, the paper should identify which of these automation opportunities are genuinely open problems versus which could be addressed by adapting existing SE techniques (e.g., property-based testing, invariant mining, requirements monitoring). As it stands, it is unclear whether the research agenda requires fundamentally new SE methods or routine application of existing ones to a new domain.
- §3 (Preliminary Experimentation): The experimental results are used to motivate the research agenda but lack quantitative detail. The paper states that TVAE achieved better results than other approaches based on correlation matrix similarity (Figure 2) and Kaplan-Meier curve comparison, but no quantitative metrics are reported (e.g., correlation distance, KS test statistics, log-rank test p-values). Without at least summary metrics, the reader cannot assess whether the claimed differences between generators are meaningful or whether the observed statistical plausibility vs. clinical validity gap is substantive. This weakens the empirical grounding for the lessons L2 and L3.
minor comments (7)
- §2, Figure 1: The notation in the figure (D, A(D), R, D', A(D'), R', f, P) is introduced in the text but the figure itself lacks labels or a caption explaining the symbols. Adding a brief caption would improve readability.
- §3, Dataset Cleaning: The threshold of 80% missing values for column removal is stated without justification. A brief note on why this threshold was chosen (or whether it was validated with oncologists) would strengthen the discussion.
- §3: The paper mentions that 'some differences appeared at the end of the curves, where the number of patients was low' but does not report the sample sizes at the tail of the Kaplan-Meier curves. Including this information would help readers assess the reliability of the comparison.
- §4, L1: The lesson title 'Data cleaning is engineering, not pre-processing' is somewhat overstated relative to the supporting argument, which is that cleaning decisions affect downstream property satisfiability. Consider softening to 'Data cleaning is a specification activity, not mere pre-processing.'
- §5: The related work section is brief and could benefit from citing and positioning against requirements engineering approaches for data-intensive systems and data quality frameworks, which are directly relevant to the property-driven framing.
- §1: 'a software system that that supports' — duplicate 'that.'
- §6: 'current methodologies do not adequately support or address' is a strong claim that should either be softened or supported with specific examples of what existing methodologies fail to address.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive review. The referee's three major comments are all well-taken, and we will address each in a revised manuscript. Below we respond point by point.
read point-by-point responses
-
Referee: §5/§6: The paper cites Zazzetti et al. [18] (SAFE), which already assesses fidelity, privacy, and clinical utility — the exact multi-property validation space the paper claims is unaddressed. The paper does not specify what concrete technical capability is missing from SAFE and similar frameworks that would require new SE methods.
Authors: The referee is correct that our current draft does not adequately differentiate our research agenda from SAFE and similar validation frameworks. We will revise §5 to provide a concrete gap analysis. To preview the argument: SAFE and comparable frameworks (e.g., the benchmarking approach of Yan et al. [17]) provide post-hoc validation pipelines — they evaluate generated data against a fixed set of metrics (fidelity, privacy, utility) after generation. What they do not provide are SE capabilities for: (1) automated elicitation and formalization of stakeholder-specific validity properties from informal clinical requirements, (2) detection and resolution of conflicts between properties across stakeholders (e.g., when privacy constraints conflict with clinical plausibility requirements), (3) property-driven generator selection and configuration — i.e., reasoning about which generator satisfies which properties rather than evaluating all generators post-hoc, and (4) continuous monitoring and evolution of property satisfaction as real data, clinical guidelines, and stakeholder needs change over time. SAFE treats validation as a fixed checklist; our agenda treats it as an evolving, stakeholder-driven engineering process. We will make this distinction explicit in the revised §5 and adjust the novelty claim in §6 accordingly, acknowledging that validation frameworks like SAFE exist while arguing that the SE challenges of property elicitation, conflict resolution, and pipeline evolution are not addressed by them. revision: yes
-
Referee: §4, Table 1: The automation opportunities are presented at a very high level without connecting to the state of the art. The paper should identify which are genuinely open problems versus which could be addressed by adapting existing SE techniques.
Authors: This is a fair criticism. In the revision, we will add a paragraph (or an expanded table) that maps each automation opportunity to the relevant SE sub-field and assesses the gap. Concretely: (a) 'Constraint mining' and 'rule checking' connect to invariant mining and specification mining (e.g., Daikon-style dynamic invariant detection, association rule mining); these are partially adaptable but have not been applied to clinical plausibility constraints for synthetic data, where constraints are domain-specific and may involve temporal/sequential plausibility — a genuine gap. (b) 'Property-based test generation' connects to existing PBT frameworks (e.g., QuickCheck, Hypothesis); adapting these to synthetic data validation is largely an engineering effort, though defining meaningful property oracles for clinical validity remains open. (c) 'Drift detection' and 'impact analysis' connect to requirements monitoring and runtime verification (e.g., RV techniques); existing methods can detect distributional drift but do not reason about whether drift invalidates specific stakeholder properties — an open problem. (d) 'Privacy risk analysis' is an active area in the security/privacy community (membership inference, distance-based disclosure metrics) but is typically treated in isolation from clinical and utility properties, making cross-property trade-off analysis an open SE challenge. We will incorporate this analysis to make the agenda actionable and to honestly distinguish novel SE challenges from routine application of existing techniques. revision: yes
-
Referee: §3: Experimental results lack quantitative detail. No correlation distance, KS test statistics, log-rank test p-values, or other summary metrics are reported.
Authors: The referee is right that the current draft relies on visual comparison (Figure 2, correlation matrix inspection) without reporting quantitative metrics. We will add a summary table reporting, for each generator: (1) Frobenius norm distance between the correlation matrices of real and synthetic data, (2) column-wise KS test statistics (median and range across variables), (3) log-rank test p-values for the Kaplan-Meier curve comparisons (global and stratified), and (4) concordance index (C-index) for the univariate and multivariate Cox models fitted on real vs. synthetic data. We agree that without these metrics, the reader cannot assess whether the differences between generators are meaningful or whether the statistical-plausibility-vs-clinical-validity gap is substantive. Adding these metrics will also strengthen the empirical grounding for L2 and L3. We note that, as stated in the paper, we do not claim clinical validity for any generator; the quantitative metrics will support the more limited claim that generators can appear statistically plausible while requiring domain-specific validation. revision: yes
Circularity Check
No circularity: reflection/agenda paper with no derivation chain or fitted predictions to reduce to inputs
full rationale
This paper is a reflection and agenda-setting piece, not a derivation or empirical validation. It contains no mathematical derivation chain, no fitted parameters presented as predictions, and no self-citation load-bearing argument. The formalization in Figure 1 (D → A(D) → R, D' → A(D') → R', R' must satisfy P) is a definitional framing of the problem, not a claim that one quantity derives from another. The lessons L1–L4 are observations from preliminary experimentation, not predictions. The related work citation to Zazzetti et al. [18] is by a different author set (Zazzetti et al. vs. Zanenga et al.) and is used to position the paper's contribution, not to justify a load-bearing premise via self-citation. The paper's central claim—that SDG 'shifts rather than solves' the engineering problem—is an argument supported by the case study experience, not a result derived from its own inputs. While one may question whether the paper's novelty claim is sufficiently distinguished from existing frameworks like SAFE, that is a correctness/novelty concern, not circularity. The paper does not exhibit any of the enumerated circularity patterns: no self-definitional equivalence, no fitted input called prediction, no self-citation chain, no ansatz smuggled through citation, and no renaming of a known result as a derivation.
Assumptions & free parameters
assumptions (3)
- domain assumption Data scarcity fundamentally breaks standard SE assumptions, requiring a shift from data-driven validation to property-driven synthesis.
- domain assumption Stakeholder-specific validity properties are stakeholder-dependent and potentially conflicting.
- domain assumption Statistical similarity does not imply clinical validity.
Cite this review
Pith. "Pith review of Property-Driven Synthetic Data Engineering for Data-Scarce Software Systems: Reflections from the Breast Cancer Domain." pith.science (2026). https://pith.science/paper/UR73JUF5
@misc{pith2026260706133,
author = {Pith},
title = {Pith review of: Property-Driven Synthetic Data Engineering for Data-Scarce Software Systems: Reflections from the Breast Cancer Domain},
year = {2026},
howpublished = {\url{https://pith.science/paper/UR73JUF5}},
note = {Machine review of arXiv:2607.06133}
}
read the original abstract
Modern software systems increasingly depend on data for analysis, prediction, testing, and decision-making. Yet many important domains, including medicine, safety-critical systems, and regulated industries, lack abundant, shareable, or representative data. Synthetic data generation is often proposed as a remedy, but our experience engineering software for intraoperative radiotherapy (IORT) in breast cancer treatment suggests that synthetic data shifts rather than solves the central engineering problem. The key challenge becomes deciding which properties synthetic data must preserve, how these properties should be elicited from stakeholders, how they can be validated under privacy constraints, and how they evolve. We call this problem property-driven synthetic data engineering. Drawing on a collaboration with oncologists and preliminary experiments with a sensitive IORT dataset, we identify challenges in requirements, validation, privacy, and pipeline evolution. We argue that automated software engineering research should develop methods and tools for eliciting, formalizing, checking, and evolving validity properties for synthetic data in data-scarce software systems.
Figures
Reference graph
Works this paper leans on
-
[18]
Chao Yan, Yao Yan, Zhiyu Wan, Ziqi Zhang, Larsson Omberg, Justin Guinney, Sean D. Mooney, and Bradley A. Malin. 2022. A Multifaceted benchmarking of synthetic electronic health record generation models.Nature Communications 13, 1 (Dec. 2022). doi:10.1038/s41467-022-35295-1
-
[1]
Nepomuceno, Belén Vega-Márquez, and Is- abel A
Halal Abdulrahman Ahmed, Juan A. Nepomuceno, Belén Vega-Márquez, and Is- abel A. Nepomuceno-Chamorro. 2025. Synthetic Data Generation for Healthcare: Exploring Generative Adversarial Networks Variants for Medical Tabular Data. International Journal of Data Science and Analytics20, 6 (May 2025), 5739–5754. doi:10.1007/s41060-025-00816-w
-
[2]
Laith Alzubaidi, Jinshuai Bai, Aiman Al-Sabaawi, José I. Santamaría, A. Albahri, B. S. Al-dabbagh, M. Fadhel, M. Manoufali, Jinglan Zhang, Ali H. Al-timemy, Ye Duan, Amjed Abdullah, Laith Farhan, Yi Lu, Ashish Gupta, Felix Albu, Amin Abbosh, and Yuantong Gu. 2023. A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutio...
-
[3]
Mohammad Hossein Amini and Shiva Nejati. 2024. Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems. In Proceedings of the 39th IEEE/ACM International Conference on Automated Soft- ware Engineering(Sacramento, CA, USA)(ASE ’24). Association for Computing Machinery, New York, NY, USA, 732–744. doi:10.1145/3691620.3695067
-
[4]
J Martin Bland and Douglas G Altman. 1998. Survival probabilities (the Kaplan- Meier method).BMJ317, 7172 (Dec. 1998), 1572–1580. doi:10.1136/bmj.317.7172. 1572
-
[5]
Maria-Joao Cardoso, Philip Poortmans, Elżbieta Senkus, Oreste D Gentilini, and Nehmat Houssami. 2024. Breast cancer highlights from 2023: Knowledge to guide practice and future research.The Breast74 (2024), 103674
work page 2024
-
[6]
Giacomo Fantino, Marco Rondina, Antonio Vetrò, and Juan Carlos De Martin
-
[7]
InFundamental Approaches to Software Engineering, Elvira Albert and Corina Pasareanu (Eds.)
Quantifying Privacy Risks in Synthetic Data: A Study on Black-Box Mem- bership Inference. InFundamental Approaches to Software Engineering, Elvira Albert and Corina Pasareanu (Eds.). Springer Nature Switzerland, Cham, 86–106
Show all 19 references
-
[8]
Mandeep Goyal and Q. Mahmoud. 2024. A Systematic Review of Synthetic Data Generation Techniques Using Generative AI.Electronics(2024). doi:10.3390/ electronics13173509
2024
-
[9]
A Kiran and S Saravana Kumar. 2023. A Comparative Analysis of GAN and VAE based Synthetic Data Generators for High Dimensional, Imbalanced Tabular data. In2023 2nd International Conference for Innovation in Technology (INOCON). 1–6. doi:10.1109/INOCON57975.2023.10101315
2023 doi
-
[10]
Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. 2023. TabDDPM: Modelling Tabular Data with Diffusion Models. InProceedings of the 40th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 202), Andreas Krause, Emma B...
2023
-
[11]
Zheng Li, Yue Zhao, and Jialin Fu. 2020. SynC: A Copula based Framework for Generating Synthetic Data from Aggregated Sources. In2020 International Con- ference on Data Mining Workshops (ICDMW). 571–578. doi:10.1109/ICDMW51313. 2020.00082
2020 doi
-
[12]
Marko Miletic and Murat Sariyar. 2024. Challenges of Using Synthetic Data Generation Methods for Tabular Microdata.Applied Sciences14, 14 (2024). doi:10. 3390/app14145975
2024
-
[13]
Vittoriano Muttillo, Claudio Di Sipio, Riccardo Rubei, Luca Berardinelli, and MohammadHadi Dehghani. 2024. Towards Synthetic Trace Generation of Mod- eling Operations using In-Context Learning Approach. InProceedings of the 39th IEEE/ACM International Conference on Automated S...
2024 doi
-
[14]
World Health Organization and World Health Organization. 2024. WHO Global Breast Cancer Initiative: breast cancer awareness month.World Health Organization. https://www. who. int/news-room/events/detail/2024/10/01/default- calendar/who-global-breast-cancer-initiative–breast-ca...
2024
-
[15]
RichMD, J
Jason T. RichMD, J. Gail NeelyMD, Randal C. PanielloMD, Courtney C. J. Voelk- erMD DPhil, Brian NussenbaumMD, and Eric W. WangMD. 2010. A prac- tical guide to understanding Kaplan-Meier curves.Otolaryngology–Head and Neck Surgery143, 3 (2010), 331–336. doi:10.1016/j.otohns.201...
2010 doi
-
[16]
R. Shi, Y. Wang, M. Du, X. Shen, Y. Chang, and X. Wang. 2025. A Comprehensive Survey of Synthetic Tabular Data Generation.arXiv preprint arXiv:2504.16506 (2025)
2025 arXiv
-
[17]
Ghanem Soltana, Mehrdad Sabetzadeh, and Lionel C. Briand. 2017. Synthetic data generation for statistical testing. In2017 32nd IEEE/ACM International Con- ference on Automated Software Engineering (ASE). 872–882. doi:10.1109/ASE.2017. 8115698
2017 doi
-
[19]
Elena Zazzetti, Saverio D’Amico, Flavia Jacobs, Rita De Sanctis, Lorenzo Chi- udinelli, Mariangela Gaudio, Gianluca Asti, Mattia Delleani, Elisabetta Sauta, Mirco Quintavalla, et al. 2025. Longitudinal synthetic data generation by artificial intelligence to accelerate clinical...
2025
Reviewed July 8, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.