{"id":"77effa14-212b-41c6-990a-535aca0c0468","arxiv_id":"2501.10240","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper provides a categorized guide of over forty EHR data extraction and preparation challenges with actionable recommendations for dynamic prediction modeling.","lead":"A practical tutorial catalogues more than forty challenges in extracting and preparing electronic health record data for dynamic prediction models, with mitigation recommendations. It offers teams a structured checklist to catch data quality problems before they bias clinical predictions.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Main risk: the 'comprehensive list' claim rests on a selective review and one group's experience; without external coverage checks, common challenges may be missing, though the paper's explicit caveats weaken this concern.","rationale":"The reader's weakest_assumption identifies the representativeness of the selective evidence base as the main load-bearing premise, and I agree. I considered whether the paper contains internal inconsistencies, unsupported factual claims, or recommendations that conflict with each other. The advice on timestamps, for example, is internally coherent: it distinguishes registration time for features used at prediction time from clinically relevant time for outcome definitions. The paper also consistently advises against manual corrections that cannot be reproduced at implementation, and it repeatedly qualifies recommendations as context-dependent. The acknowledged limitations in Section 5, including the selective review, the single-site structured-data scope, and the lack of impact assessment, are stated openly rather than hidden. Because the paper's purpose is a practical guide rather than a systematic review or an empirical evaluation, these limitations do not undermine the ACCEPT verdict. The proposed concrete test would nonetheless be valuable: a structured coverage check against independently obtained challenges would settle whether the word 'comprehensive' is justified or whether the title and abstract should be softened. Until such a check is run, the concern remains plausible but not sufficient to change the verdict.","tokens_in":25179,"tokens_out":4187,"duration_ms":46261,"concrete_test":"Independently code the challenges reported in 20-30 recent single-site structured EHR dynamic prediction studies, drawn from a systematic search rather than from the paper's reference list, against the paper's tables. If more than 20% of encountered challenges are not covered by Tables 1-11, or if a majority of independent EHR data engineers rate the recommendations as not implementable in their own platforms, the comprehensiveness and actionability claims would need to be downgraded. If coverage is high and recommendations are widely rated as actionable, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the paper provides a comprehensive list of challenges and actionable recommendations that improve data quality and real-world applicability of dynamic prediction models. The load-bearing premise is representativeness: the list must cover the common single-site structured EHR extraction and preparation challenges across hospitals, EHR vendors, and extraction platforms. This premise rests on a 'selective literature review' and the authors' own experience (Section 2), and Section 5 concedes the list is not exhaustive and that 'every project will face use case specific challenges.' Because no systematic search or external validation is reported, the list could omit common challenges, for instance issues tied to specific EHR vendor configurations, data-sharing governance, or longitudinal identifier changes across systems, making the 'comprehensive' claim an overstatement even if every individual recommendation is reasonable. A related weakness is that recommendations are not empirically validated: Section 5 explicitly says the paper does not assess the impact of each challenge on final model performance, so the claim that following the recommendations 'can enhance data quality' is supported chiefly by plausibility and the authors' practical experience. However, the paper repeatedly frames itself as a pragmatic guide, advises case-by-case assessment, and explicitly disclaims exhaustiveness; those caveats substantially reduce the risk. The concern is therefore real but not disqualifying.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper is a tutorial-style practical guide to the data extraction and preparation stages that precede dynamic prediction modeling with single-hospital structured EHR data. The authors describe the data flow from collection through extraction (ETL) and preparation to model building and clinical implementation, and then present a categorized list of over 40 challenges in cohort definition, outcome definition, feature engineering, and data cleaning. Each challenge is accompanied by a description, a recommendation, and a mapping to the Weiskopf–Weng and METRIC data quality frameworks. The central contribution is the checklist itself, with particular emphasis on implementation-time reproducibility: correct interpretation of timestamps, avoidance of temporal leaks, distinction between clinically relevant and recording timestamps, and reproducible programmatic data cleaning. The paper explicitly declares its scope (single-site structured data, excluding multi-center and unstructured text) and its limitations (selective literature review, no empirical assessment of the impact of each challenge on model performance).","tokens_in":25341,"tokens_out":7370,"duration_ms":75761,"significance":"If judged as a consolidation of practical expertise rather than as an empirical evaluation, the paper is valuable. It provides a well-organized inventory that maps to established data-quality frameworks and draws attention to implementation-time reproducibility issues, such as timestamp semantics (Table 8), temporal leaks (Table 9), and aggregate features (Table 7), that are often underemphasized in machine-learning guides. The authors are transparent about their methods and limitations: Section 2 states that the review was selective, and Section 5 explicitly disclaims exhaustiveness and notes that the impact of individual challenges on model performance is not assessed. The recommendations are appropriately hedged as context-dependent, and the fitness-for-use principle is invoked throughout. The paper does not overclaim empirical support; its value lies in providing a structured, expert-derived checklist for practitioners. The main limitations, which are disclosed, mean the work should be read as a practical guide rather than a systematic review or a quantitative study.","major_comments":[],"minor_comments":[{"comment":"The phrase 'comprehensive list' is stronger than the method and limitations support. Section 5 explicitly states that the review was selective and cannot be exhaustive. I recommend replacing 'comprehensive' with 'structured' or 'broad' in the abstract and in Section 2, and adjusting the final sentence of Section 5, so that the claim matches the disclosed limitations.","section":"Abstract and Section 2"},{"comment":"In the third paragraph, 'OMOP CMD' should be 'OMOP CDM'; the acronym is spelled correctly elsewhere, e.g., in Section 1 and in Table 6.","section":"Section 2"},{"comment":"The recommendation for 'Missing values that can be fully recovered from other fields' suggests using a powerful missing-data imputation algorithm. To be consistent with the paper's earlier advice in Section 3(c) to separate train and test sets before preprocessing, the recommendation should explicitly state that the imputation model must be fitted on the training data only, otherwise readers may inadvertently introduce test-train leakage.","section":"Table 11"},{"comment":"Several challenges that are fundamentally about timeliness, such as 'Label leakage due to temporal leakage' and 'Temporal leaks due to extraction errors', are mapped to METRIC/WW = NA. If the mapping is intended to be complete, it would help to explain why these entries have no mapping, or to add the relevant Timeliness/Currency dimensions.","section":"Tables 4 and 9"},{"comment":"The tables are dense and would benefit from a short 'how to use this list' paragraph stating whether the order of challenges implies priority. The current text advises case-by-case assessment but does not suggest an order of operations or a triage strategy, which would increase the practical actionability of the guide.","section":"Section 4 (Tables 1-11)"}],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a genuinely useful practical tutorial, not a research result. If you work with single-site structured EHR data for prediction, it is the kind of document you wish someone had handed you at the start of the project. It synthesizes over forty challenges into four categories—cohort definition, outcome definition, feature engineering, and data cleaning—and pairs each with concrete, actionable recommendations. The dynamic-prediction focus is what sets it apart from generic data-cleaning checklists: the authors consistently ask whether a correction made at training time can be reproduced when the model runs live in the EHR. That question runs through the entries on timestamps, nursing documentation, coded procedures, and aggregate features, and it is the most valuable thread in the paper. The advice is also refreshingly pragmatic; leaving things as-is is treated as a legitimate choice, not a failure.\n\nWhat the paper does well is organize existing lessons-learned literature into a usable reference and map it to the Weiskopf–Weng and METRIC quality frameworks. The writing is clear, the tables are usable, and the authors are transparent that their review was selective and their scope single-site structured data. They also state plainly that they do not assess the impact of each challenge on final model performance, so the recommendations are expert opinion supported by plausibility and experience, not measured effect sizes. For a tutorial, that is acceptable; the paper should not be read as evidence that following the advice improves model performance.\n\nThe soft spots are real but mostly self-acknowledged. The abstract's “comprehensive list” overstates what a selective review plus one group's experience can support. There is no systematic search, no coverage check across vendors or extraction platforms, and the authors note in Section 5 that other groups will likely identify additional issues. That caveat substantially weakens the concern. A second weakness is that the framework mappings feel somewhat decorative; they do not drive the entries, and some mappings are clear while others read as assignments after the fact. Minor.\n\nCitations are fine. The authors draw on the prior lessons-learned papers you would expect, and their self-citations are relevant examples of competing-risk and auxiliary-outcome modeling, not circular justification. The central content is an independent synthesis.\n\nVerdict: send it to peer review. A serious referee will probably ask for a more carefully qualified scope statement or a small empirical check of coverage, but this paper deserves referee time. I would not bring it to a reading group for debate; it is a reference, not a provocation. I would cite it in my own work when doing EHR prediction.","headline":"A genuinely useful, well-organized practical guide to EHR data extraction; the 'comprehensive' claim is stronger than the selective review can support, but the paper's own caveats keep that from being disqualifying.","tokens_in":25947,"tokens_out":4614,"would_cite":true,"duration_ms":45698,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper compiles more than forty named challenges in extracting and preparing electronic health record data for dynamic prediction modeling, pairs each with actionable recommendations, and argues that following them improves data…","keywords":["electronic health records","dynamic prediction models","data extraction","data preparation","data quality","temporal leakage","feature engineering","cohort definition"],"falsifier":"A project-level audit could settle the generality claim: take the documented data-preparation incident logs from several completed EHR prediction projects at hospitals that did not contribute to this paper, and check whether every incident falls into one of the four categories and is addressed by a listed recommendation; if more than a small fraction, say 20 percent, of incidents fall outside the taxonomy or lack a usable recommendation, the claim that the list is comprehensive for single-hospital structured extractions would be disproved.","tokens_in":24954,"feed_emoji":"🏥","tokens_out":7635,"duration_ms":67362,"temperature":0.7,"pith_summary":"This paper argues that the quality and clinical usefulness of dynamic prediction models built from electronic health records are decided before any model is trained, in the data extraction and preparation stages. It assembles more than forty concrete problems observed in those stages and organizes them into four categories—cohort definition, outcome definition, feature engineering, and data cleaning—each paired with practical recommendations. The intended contribution is a hands-on checklist that data engineers and researchers can consult during a project, so that data quality issues are anticipated and handled in a way that can be reproduced at the moment the model runs in the hospital. A sympathetic reader would care because model performance reported in training can be misleading if extraction and preparation introduce temporal leaks, selection bias, or cleaning steps that cannot be rerun at prediction time.","feed_headline":"Over 40 EHR data pitfalls mapped to practical fixes","feed_subtitle":"A four-category checklist for cohort, outcome, features, and cleaning keeps dynamic clinical models honest in live use.","key_machinery":"The carrying object is the four-category challenge-and-recommendation taxonomy—cohort definition, outcome definition, feature engineering, and data cleaning—with each item mapped to established data-quality dimensions. The mechanism that connects the taxonomy to model performance is the staged data flow from collection to extraction to preparation to implementation, together with a point-in-time discipline: feature construction must use the time a predictor actually becomes available in the system, while outcome labels should use the clinically relevant time of the event; mismatches between these two timestamps are the main generator of temporal leaks.","core_discovery":"The paper's central claim is that the recurring failures of EHR-based dynamic prediction models can be traced to a finite, nameable set of problems in the stages before modelling, and that these problems can be managed with preparation-stage practices rather than by changing EHR software or clinical workflows. It claims that cohort definitions must be checked against what is available at the first prediction trigger; that outcomes should prefer timestamped clinical or surveillance definitions over billing codes; that feature engineering must respect the difference between the clinically relevant time of an event and the time the data item becomes available in the system; and that data cleaning should be programmatic, reproducible, and 'leave-as-is' when a correction cannot be rerun at implementation. If the paper is right, the practical payoff is a shorter path from raw hospital databases to models whose evaluation reflects what the model will encounter in live use.","pith_inferences":["Editorial inference: the four-category taxonomy could be turned into a standardised pre-modelling audit questionnaire, letting hospitals report which pitfalls they actually encountered and generating evidence about which items matter most for model performance.","Editorial inference: the paper's timestamp principle suggests a concrete stress test for any dynamic model: replay each admission incrementally up to each prediction trigger and compare the features with the batch-computed version; the size of any difference quantifies the temporal-leak risk the paper warns about.","Editorial inference: the reproducibility principle implies that post-deployment monitoring should watch for changes in data recording and extraction logic, not only drift in model inputs, because the paper notes that EHR software updates introduce new bugs over time."],"forward_implications":["Teams can use the four-category checklist during project planning to spot likely data problems before extraction, for example whether key features have usable timestamps or enough historical coverage.","Dynamic prediction models should be built and evaluated under the same extraction and preparation logic that will run at implementation, so that training-time performance is not inflated by temporal leaks.","Acute-event outcomes should be derived from timestamped clinical or surveillance definitions rather than billing codes, which are typically assigned after discharge and are not timestamped.","Data cleaning should be programmatic and reproducible at prediction time; manual one-off corrections to training data widen the gap between development and live data.","Extracting a broader patient context than the strict inclusion criteria, such as full admissions and a larger set of clinical items, gives preparation and imputation more room and reduces selection bias."],"supporting_citations":[{"why":"It supplies the data-quality dimensions used throughout the paper to map every listed challenge.","marker":"Weiskopf and Weng, 2013"},{"why":"It supplies the second data-quality framework whose clusters and dimensions are used alongside the first in the mapping.","marker":"Schwabe et al., 2024"},{"why":"It documents common EHR data science pitfalls and avoidance strategies that several cohort and feature recommendations build on.","marker":"Sauer et al., 2022"},{"why":"It provides the cleaning examples and preparation framework that the data-cleaning recommendations extend.","marker":"Miao et al., 2023"},{"why":"It grounds the look-back period and outcome-suitability recommendations by assessing general-practice EHR records for prediction-model development.","marker":"Thuraisingam et al., 2021"},{"why":"It supplies the continuous-prediction workflow practices, including timestamp conventions and outlier capping, that the time-sensitive recommendations adapt.","marker":"Tomašev et al., 2021"},{"why":"It supports the documentation recommendations by defining the steps that turn a clinical database into a research dataset.","marker":"de Kok et al., 2024"},{"why":"It documents that mitigation strategies for EHR data quality remain largely undocumented, which motivates the paper's recommendation list.","marker":"Honeyford et al., 2022"},{"why":"It contributes lessons and tips on prediction-task definition, diagnosis coding, and clinician-assessed versus patient-reported fields.","marker":"Arbet et al., 2021"},{"why":"It supplies concrete examples of identifier linking, bed identifiers, date shifting, and sensor artifacts that several challenges reference.","marker":"Maletzky et al., 2022"}],"fun_headline_variants":["EHR data prep: 40+ challenges, 4 categories, 1 guide","Nameable EHR pitfalls, practical fixes for models","Don't rebuild EHR systems—fix data prep instead","The EHR data preparation checklist for dynamic models","Why dynamic EHR models fail: data prep fixes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The list's coverage rests on the assumption that the authors' own extraction projects and the selectively reviewed literature are representative enough that the named challenges and recommendations generalize to other hospitals, EHR vendors, and extraction platforms, a limitation the paper itself acknowledges.","fun_headline_variants_meta":{"raw":{"variants":["EHR data prep: 40+ challenges, 4 categories, 1 guide","Nameable EHR pitfalls, practical fixes for models","Don't rebuild EHR systems—fix data prep instead","The EHR data preparation checklist for dynamic models","Why dynamic EHR models fail: data prep fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000285,"raw_usage":{"total_tokens":1620,"prompt_tokens":830,"completion_tokens":790,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":446,"completion_tokens_details":{"reasoning_tokens":711}},"tokens_in":446,"tokens_out":790,"duration_ms":7935,"temperature":1.0,"reasoning_tokens":711,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:17:37.036836+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A project-level audit could settle the generality claim: take the documented data-preparation incident logs from several completed EHR prediction projects at hospitals that did not contribute to this paper, and check whether every incident falls into one of the four categories and is addressed by a listed recommendation; if more than a small fraction, say 20 percent, of incidents fall outside the taxonomy or lack a usable recommendation, the claim that the list is comprehensive for single-hospital structured extractions would be disproved.","supporting_citations":[{"cited_title":"Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research","cited_arxiv_id":null,"evidence_quote":"It supplies the data-quality dimensions used throughout the paper to map every listed challenge."},{"cited_title":"The metric-framework for assessing data quality for trustworthy ai in medicine: a systematic review","cited_arxiv_id":null,"evidence_quote":"It supplies the second data-quality framework whose clusters and dimensions are used alongside the first in the mapping."},{"cited_title":"Leveraging electronic health records for data science: common pitfalls and how to avoid them","cited_arxiv_id":null,"evidence_quote":"It documents common EHR data science pitfalls and avoidance strategies that several cohort and feature recommendations build on."},{"cited_title":"A data preparation framework for cleaning electronic health records and assessing cleaning outcomes for secondary analysis","cited_arxiv_id":null,"evidence_quote":"It provides the cleaning examples and preparation framework that the data-cleaning recommendations extend."},{"cited_title":"Assessing the suitability of general practice electronic health records for clinical prediction model development: a data quality assessment","cited_arxiv_id":null,"evidence_quote":"It grounds the look-back period and outcome-suitability recommendations by assessing general-practice EHR records for prediction-model development."},{"cited_title":"Table 0; documenting the steps to go from clinical database to research dataset","cited_arxiv_id":null,"evidence_quote":"It supports the documentation recommendations by defining the steps that turn a clinical database into a research dataset."},{"cited_title":"Challenges and recommendations for high quality research using electronic health records","cited_arxiv_id":null,"evidence_quote":"It documents that mitigation strategies for EHR data quality remain largely undocumented, which motivates the paper's recommendation list."},{"cited_title":"Lessons and tips for designing a machine learning study using ehr data","cited_arxiv_id":null,"evidence_quote":"It contributes lessons and tips on prediction-task definition, diagnosis coding, and clinician-assessed versus patient-reported fields."},{"cited_title":"Lifting hospital electronic health record data treasures: challenges and opportunities","cited_arxiv_id":null,"evidence_quote":"It supplies concrete examples of identifier linking, bed identifiers, date shifting, and sensor artifacts that several challenges reference."}],"review_version":1}