{"id":"f7e530d4-f431-4be4-9ca1-fbd1a81c03d2","arxiv_id":"2507.02922","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"CMML uses entity-relationship diagrams to guide data preparation, and its two case studies report improved machine learning performance in foster-care length-of-stay and medication-detection tasks.","lead":"This paper introduces CMML, five rules for using conceptual diagrams of a business domain to prepare data before training machine learning models. Two foster-care studies and a focus group suggest the rules can improve accuracy and traceability, though the reported evidence is incomplete.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central performance claim rests on comparing models trained on different prediction units and target distributions; re-evaluating on a common unit is required before CMML's benefit can be concluded.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: DS0 and the transformed TDSn datasets do not pose the same prediction task, so the reported performance improvements cannot be attributed to the CMML method. This is the most fundamental threat to the central claim because every quantitative performance conclusion in the paper flows through these comparisons. The Case 1 comparison changes row granularity and target distribution simultaneously; the Case 2 comparison removes a subset of the original data before evaluating recall. Both changes are confounded with the method being tested, and neither is addressed by the normalization discussion or the Wilcoxon tests. The Wilcoxon results in Table 5 reinforce the concern: paired tests require paired units, but DS0 and TDS1 have different row counts and different row meanings, and the reported sample size of 12,438 for the DS0 versus TDS1 comparison is not derived from the stated dataset sizes. The transparency claim is also weaker, resting on focus-group self-reports rather than an objective measure, but the performance comparison is the load-bearing support for the paper's main contribution. The CMML guidelines may be reasonable and the focus-group feedback is suggestive, but the empirical evaluation as presented does not establish that CMML improves ML performance. Therefore, the reader's REJECT verdict should be maintained; no verdict change is needed based on this stress-test pass.","tokens_in":21772,"tokens_out":3324,"duration_ms":42756,"concrete_test":"","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is the validity of comparing model performance across DS0 and TDSn datasets in Cases 1 and 2. In Case 1, DS0 has 25,462 placement rows while TDS1 has 4,437 episode rows, and the target mean/SD changes from 704.78/414.89 to 440.47/340.70 (Tables 3-4). The claimed RMSE/r2 gains are therefore entangled with a change in unit of analysis and target distribution. A model can appear better simply because episode-level data has lower target variance, or because placement-level repeated rows are harder to predict. The Wilcoxon 'paired' comparison in Table 5 is not a valid pairing because the rows are different units; the reported sample size of 12,438 for DS0 versus TDS1 is also unexplained. TDS2 and TDS3 are age subsets of DS0, so comparing their combined performance with DS0 tests subpopulation splitting, not the CMML data-preparation method. In Case 2, TDS1-Single is evaluated after dropping TDS2-Multiple (multi-child homes), so the recall gain from 59.28 to 84 may reflect removal of harder multi-child notes rather than CMML. Thus the central claim that applying CMML improves ML model performance is unsupported by the evaluation as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Conceptual Modeling for Machine Learning (CMML), a five-guideline method that uses Extended Entity-Relationship (EER) model constructs to guide data preparation for supervised machine learning. The method is evaluated in two real-world foster care cases: predicting episode length (regression) and detecting psychotropic medication prescription from case notes (classification). Models trained on the original dataset (DS0) are compared with models trained on transformed datasets (TDSn) using RMSE, r2, recall, precision, and F-measure, with statistical tests reported. The paper also reports a focus-group study with 15 data scientists to assess applicability and transparency. The central claim is that applying CMML improves both ML model performance and process transparency.","tokens_in":22015,"tokens_out":5084,"duration_ms":58703,"significance":"If the empirical claims held, the paper would make a useful contribution to data-centric ML by providing an operational, guideline-based way to inject domain knowledge from conceptual models into data preparation, and it would strengthen the link between conceptual modeling and ML practice. The five guidelines are clearly described, the application to foster care is socially relevant, the focus-group protocol is detailed, and the authors explicitly acknowledge limitations of the method. However, the reported comparisons are confounded by changes in unit of analysis, target distribution, and deliberate data subsetting, so the current evidence does not support the central performance claim as stated.","major_comments":[{"comment":"The DS0 versus TDS1 comparison changes the unit of analysis and the target distribution: DS0 contains 25,462 placement rows with the episode-length target repeated per placement, while TDS1 contains 4,437 episode rows, and the target mean/SD changes from 704.78/414.89 to 440.47/340.70. RMSE and r2 are therefore not directly comparable across these datasets, because the prediction targets are defined on different units and different target variances. The claim that TDS1 'consistently outperformed' DS0 is not supported unless both datasets are re-evaluated on a common prediction unit, for example by aggregating placement-level features to the episode level and comparing episode-level predictions.","section":"Sec. 4.1.1, Tables 3 and 4"},{"comment":"The Wilcoxon signed-rank comparison 'DS0 vs TDS1' lists a sample size of 12,438, which matches neither DS0 (25,462 rows) nor TDS1 (4,437 rows). More importantly, the pairing of prediction errors across rows that represent different units (placements versus episodes) is not described, and it is unclear how a paired test can be defined when the two datasets have different row sets. Without a valid paired design or an explanation of how the 12,438 pairs were formed, the statistical conclusion in the sentence following Table 5 ('we conclude that applying the CMML method can improve the performance of ML models') does not follow from the reported evidence.","section":"Sec. 4.1.1, Table 5"},{"comment":"TDS2 and TDS3 are age-based subsets of DS0, not datasets produced by applying Guidelines 2 and 4, so comparing their combined performance with DS0 tests only Guideline 5 (specialization) on subpopulations. The text states that combining TDS2 and TDS3 yielded equivalent performance to DS0 for four of the five models, yet Table 5 reports a significant Wilcoxon result for the same combined comparison; this contradiction needs to be resolved. In addition, the heading 'Guideline 7' in Table 4 for the TDS2/TDS3 rows appears to be a typo for Guideline 5. The conclusion should not attribute any combined gain to the full CMML method when only one guideline is being tested and the aggregate gain appears limited to the AutoML model.","section":"Sec. 4.1.1, Tables 4 and 5"},{"comment":"In Case 2, the dataset TDS2-Multiple (multi-child homes) is dropped before the performance comparison, and DS0 is at the level of home-visit notes (1,545 rows) while TDS1-Single is at the level of individual children (852 rows). The reported recall improvement from 59.28 to 84 may therefore reflect the removal of harder multi-child notes or the change in unit of analysis rather than the effect of the CMML guidelines. A child-level evaluation that includes all notes, or an explicit analysis of the dropped TDS2-Multiple subset, is required before concluding that the guidelines improved classification performance.","section":"Sec. 4.1.2, Tables 6 and 7"},{"comment":"The claim that CMML improves process transparency is supported only by a focus-group study in which 15 data scientists recruited for the study were asked whether the guidelines are useful. This is an applicability and acceptance check, not a measurement of transparency, and it is partly self-referential because the same team that developed the guidelines also conducted the assessment. The conclusion in Section 5 that the results show 'consistent and robust evidence for the benefits of using CMML to improve process transparency' overstates what a qualitative applicability check with a small, non-independent sample can establish.","section":"Sec. 4.2"}],"minor_comments":[{"comment":"The sentence 'Table 3 shows these results' appears to refer to Table 4, since Table 3 is a dataset description and Table 4 contains the RMSE and r2 results.","section":"Sec. 4.1.1"},{"comment":"The heading 'Guideline 7' in the TDS2-Younger and TDS3-Older rows should be 'Guideline 5' to match the dataset description in Table 3.","section":"Sec. 4.1.1, Table 4"},{"comment":"The phrase 'the F-measure difference was 5%, indicating that the application of our guideline improved performance, and the improvement was statistically significant [2]' cites reference [2] (Adomavicius et al.) for statistical significance, which is not a standard statistics reference; a statistics or methodology citation would be more appropriate.","section":"Sec. 4.1.2"},{"comment":"The text cites 'Ohno-Machado et al., 1998' but this reference is not included in the reference list; either add the full citation or remove the citation.","section":"Sec. 3.4.2"},{"comment":"There are several typographical errors, including 'no doubtly' in Section 2.1, 'iterativeley' in Section 2, and 'in iterativeley' in the same paragraph; a careful proofreading pass is needed.","section":"Throughout"},{"comment":"The discussion would benefit from a clearer separation between the evidence for the performance claim (which is currently confounded) and the evidence for the applicability/transparency claim (which is qualitative), so that readers can weigh the two contributions independently.","section":"Sec. 5"}],"recommendation":"major_revision","confidential_remarks":"The full text opens with a citation to 'Storey V.C., Parsons, J. Castellanos A., Tremblay M., Lukyanenko R., Maass, W., Castillo, A. (2025). Conceptual Modeling for Machine Learning: Using Domain Knowledge to Improve Performance and Process Transparency. Data & Knowledge Engineering, pp 1-41.' This suggests that the manuscript may be a version of, or very close to, a paper already published under a different title. The editors may wish to verify prior publication and duplicate-submission status before proceeding. Additionally, the empirical evaluation as reported does not support the central performance claim; the authors should be asked to re-analyze the data on a common unit of analysis (episode level in Case 1, child level in Case 2) before the paper can be considered further."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the five guidelines are a genuine, useful packaging of data-prep knowledge grounded in EER constructs, and the two real-world foster-care cases are substantively motivated. But the central claim—that CMML improves predictive performance—is not supported by the evaluation as reported. The comparisons change the unit of analysis: Case 1 compares models on 25,462 placement rows against 4,437 episode rows, with target mean shifting from 704.78 to 440.47. Any RMSE/r2 gain is entangled with granularity and target distribution. The Wilcoxon 'paired' test on 12,438 rows doesn't fix the mismatch; it's not a paired comparison on the same units. Case 2 drops multi-child homes before comparing recall, so the recall gain from 59 to 84 may be an artifact of removing harder cases. The transparency claim rests on focus-group self-reports, which is evidence of perceived usefulness, not measured transparency.\n\nWhere the paper does well: the guidelines are clearly derived from EER constructs (entity labeling, derived attributes, domain-aware imputation, entity summarization, subtype splitting), and they're stated concretely enough to apply. The discussion of how a conceptual model can prevent naive imputation of optional attributes is a real point. The authors are also honest that the method's boundary conditions need work and that they didn't evaluate the method as a whole.\n\nIs this fixable? Yes, in principle. Rerun Case 1 on a consistent unit—for instance, aggregate DS0 to episode level as a baseline, or at minimum report results on matching subsets—and rerun Case 2 on the full sample with multi-child notes handled explicitly. Provide code and data. Then the performance claim might be supportable.\n\nFor a peer reviewer: the paper deserves referee time because the underlying idea is important for data-centric ML and the guidelines are likely to be adopted. But as it stands, the load-bearing empirical claim doesn't hold. I'd recommend major revision with a controlled comparison.","headline":"CMML's guidelines are a sensible packaging of known data-prep practices, but the empirical case is compromised by unit-of-analysis changes, so the performance claim needs a redo on a common task.","tokens_in":22562,"tokens_out":1698,"would_cite":false,"duration_ms":19123,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A five-guideline method grounded in entity-relationship modeling improves machine learning performance and traceability in two foster-care applications.","keywords":["conceptual modeling","machine learning","data preparation","entity-relationship model","CMML","process transparency","feature engineering","foster care"],"falsifier":"Rerun Case 1 holding the unit of analysis fixed: keep the number of placements per episode as a feature while predicting episode length at the episode level, and compare against TDS1; if the advantage disappears, the reported gain is an artifact of changing granularity. Also check whether models trained on DS0 after applying only a statistical aggregation baseline (e.g., group-by means without entity labeling) match TDS1 performance.","tokens_in":117,"feed_emoji":"📊","tokens_out":3632,"duration_ms":49919,"temperature":0.7,"pith_summary":"The paper claims that the domain knowledge encoded in a conceptual model can be used to prepare training data, and that doing so yields machine learning models that are both more accurate and more transparent. It introduces CMML, a method with five guidelines built on the constructs of extended entity-relationship models (entities, attributes, relationships, cardinalities, and specialization). Applied to two real foster-care problems—predicting length of stay from structured placement records and detecting psychotropic-medication use from casework notes—the method reportedly improves RMSE, explained variance, and recall, while focus groups of data scientists judged the guidelines useful and traceable. If correct, the paper shows that a systematic, semantically grounded data-preparation step can improve ML outcomes without requiring more data.","feed_headline":"Five rules turn conceptual models into better ML training data","feed_subtitle":"A method grounded in entity-relationship semantics improves accuracy and recall in two real-world foster-care applications.","key_machinery":"The carrying mechanism is CMML itself: an iterative data-preparation method whose five guidelines map EER constructs onto dataset transformations. Guideline 1 labels each feature with its entity of origin; Guideline 2 derives features from derived attributes; Guideline 3 restricts imputation to values that are applicable but unknown; Guideline 4 summarizes the many side of a one-to-many relationship into counts and numeric summaries to remove duplicate target rows; Guideline 5 builds separate training datasets for specialized subtypes. The argument is that preserving entity semantics during preparation prevents unit-of-analysis mismatches, spurious imputation, and duplication, while keeping lineage of every feature.","core_discovery":"The central claim is that applying CMML to a raw tabular dataset (DS0) produces one or more training datasets (TDSn) that support better-performing and more transparent ML models. The paper reports that in the structured case, models trained on entity-summarized data (TDS1) outperformed those on the original placement-level data across five algorithms, with an average 23.8% increase in explained variance and a 7.4% RMSE reduction, and that splitting by child-age subtypes improved AutoML results. In the unstructured case, aggregating home-visit notes to the child level and dropping multi-child homes improved recall significantly at no significant precision cost. The paper also claims the method improves process transparency, primarily supported by focus-group agreement.","pith_inferences":["If the method generalizes, its main quantitative gains may come from aligning the unit of analysis with the target attribute rather than from the full conceptual-model semantics; a head-to-head test against a purely statistical group-by aggregation would separate these effects.","The transparency claim currently rests on focus-group testimony; an objective metric such as documentation completeness, feature-importance stability, or error traceability could put it on firmer ground.","Organizations without formal conceptual models could still benefit by automatically extracting entity-relationship structure from databases or from text using large language models, a direction the paper notes only as future work."],"forward_implications":["Structured tabular tasks where a target-bearing entity sits on the one side of a one-to-many relationship can be improved by entity summarization, without adding more data.","Missing-value handling should be driven by whether an attribute is optional versus unknown, which requires entity and subtype knowledge.","AutoML pipelines could take a conceptual model as an additional input to constrain preprocessing, reducing the usual tradeoff between performance and transparency.","In domains with meaningful subclasses, training a separate model per subtype can outperform a single combined model, at least in some settings.","Entity-labeled feature names provide a lineage trail that supports reproducibility, auditing, and regulatory review."],"supporting_citations":[{"why":"Defines the entity-relationship model, the underlying representation that CMML exploits.","marker":"[21]"},{"why":"Provides the extended entity-relationship (EER) constructs, including cardinalities and generalization, which the five guidelines map onto data transformations.","marker":"[78]"},{"why":"Documents data cascades and motivates data-centric preparation, the problem CMML addresses.","marker":"[67]"},{"why":"Supplies the feature-engineering context and terminology that Guidelines 1 and 2 are designed to systematize.","marker":"[25]"},{"why":"Provides the running customer-order example used throughout the paper to illustrate the guidelines.","marker":"[42]"},{"why":"Positions conceptual modeling as a support for machine learning phases, the theoretical basis CMML operationalizes.","marker":"[51]"},{"why":"Establishes the foster-care length-of-stay prediction context used in the Case 1 evaluation.","marker":"[7]"}],"fun_headline_variants":["Conceptual modeling boosts ML accuracy by 24%","CMML: Better ML via entity-based data prep","Domain knowledge improves ML training data","Entity semantics lift ML performance and clarity"],"cache_read_input_tokens":24704,"weakest_assumption_plain":"The comparison assumes that the original DS0 and each transformed TDSn are solving the same prediction task, so their error metrics can be directly compared; in Case 1 the target mean and standard deviation change between datasets, and in Case 2 a subset is dropped, which could confound any observed improvement.","fun_headline_variants_meta":{"raw":{"variants":["Conceptual modeling boosts ML accuracy by 24%","CMML: Better ML via entity-based data prep","Domain knowledge improves ML training data","Entity semantics lift ML performance and clarity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":1110,"prompt_tokens":842,"completion_tokens":268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":211}},"tokens_in":458,"tokens_out":268,"duration_ms":3336,"temperature":1.0,"reasoning_tokens":211,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:45:53.547099+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun Case 1 holding the unit of analysis fixed: keep the number of placements per episode as a feature while predicting episode length at the episode level, and compare against TDS1; if the advantage disappears, the reported gain is an artifact of changing granularity. Also check whether models trained on DS0 after applying only a statistical aggregation baseline (e.g., group-by means without entity labeling) match TDS1 performance.","supporting_citations":[{"cited_title":"ACM transactions on database systems (TODS), 1976","cited_arxiv_id":null,"evidence_quote":"Defines the entity-relationship model, the underlying representation that CMML exploits."},{"cited_title":"Yang, and J.P","cited_arxiv_id":null,"evidence_quote":"Provides the extended entity-relationship (EER) constructs, including cardinalities and generalization, which the five guidelines map onto data transformations."},{"cited_title":"Everyone wants to do the model work, not the data work","cited_arxiv_id":null,"evidence_quote":"Documents data cascades and motivates data-centric preparation, the problem CMML addresses."},{"cited_title":"2020: Cambridge University Press","cited_arxiv_id":null,"evidence_quote":"Supplies the feature-engineering context and terminology that Guidelines 1 and 2 are designed to systematize."},{"cited_title":"Information Systems Research, 2006","cited_arxiv_id":null,"evidence_quote":"Provides the running customer-order example used throughout the paper to illustrate the guidelines."},{"cited_title":"Using conceptual modeling to support machine learning","cited_arxiv_id":null,"evidence_quote":"Positions conceptual modeling as a support for machine learning phases, the theoretical basis CMML operationalizes."},{"cited_title":"Jordan, and R","cited_arxiv_id":null,"evidence_quote":"Establishes the foster-care length-of-stay prediction context used in the Case 1 evaluation."}],"review_version":1}