Pith. sign in

REVIEW 4 major objections 5 minor 139 references

A nine-field event log could give medical AI the systematic monitoring that syslog gives computer networks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:31 UTC pith:CK2YXPWC

load-bearing objection Solid protocol proposal whose abstract overclaims deployment evidence the body doesn't provide. the 4 major comments →

arxiv 2510.04033 v2 pith:CK2YXPWC submitted 2025-10-05 cs.AI

A global log for medical AI

Ayush Noori , Aaron E. Boussina , Hai Ho Bich , James Anibal , Julia Maslinski , Manuel Burger , Martin Faltys , Adam Rodman
show 43 more authors
Alan Karthikesalingam Alessandro Blasimme Annelia Itwaru Ben Kaplan Bilal A. Mateen Christopher A. Longhurst Daniel Yang Dave deBronkart Effy Vayena Fedor Sergeev Gauden Galea Ha Thi Hai Duong Harold F. Wolf III Jacob Waxman Joerg C. Schefold Joshua C. Mandel Juliana Rotich Kenneth D. Mandl Lily Poursoltan Maryam Mustafa Melissa Miles Nigam H. Shah Noa Dagan Pavan Bodanki Peter Lee Philipp Koralus Prathamesh Parchure Prem Timsina Ran D. Balicer Robert Korom Scott Mahoney Seth Hain Tien Yin Wong Trevor Mundel Vivek Natarajan Ankit Sakhuja Benjamin Glicksberg C. Louise Thwaites Gunnar R\"atsch Karandeep Singh David A. Clifton Isaac S. Kohane Marinka Zitnik
This is my paper
classification cs.AI
keywords medical AI loggingevent-level protocolclinical AI monitoringdataset driftAI safety auditnine-field recordalgorithmic surveillancehealthcare AI deployment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that medicine has no counterpart to syslog—no standard way to record how, when, by whom, and for whom AI models are used—and that this gap blocks real-world evaluation, safety monitoring, and bias detection. It introduces MedLog, a protocol that writes one record per model invocation with nine core fields, and claims that assembling these records from immutable messages handles both simple and multi-agent workflows. The authors assert that MedLog was applied across four deployments and can capture downstream effects such as performance degradation during severe weather and increased laboratory testing after ICU alerts. A case study of a hospitalization-risk model shows that continuous monitoring of input features caught a subtle dataset shift caused by a change in a laboratory test kit. If the protocol is adopted, the authors argue, it would create the missing data layer for surveillance, auditing, and improvement of clinical AI.

Core claim

The central claim is that a standardized, event-level log—MedLog—can provide the observational foundation that clinical AI currently lacks. Each time a model is invoked, a record is created with nine fields: header, model instance, user identity, target identity, inputs, internal artifacts, patient- or clinician-facing outputs, outcomes, and user feedback. Records are built incrementally from immutable messages linked by an event identifier, allowing the same schema to represent a single prompt or a long-running agentic workflow. The paper further claims that this structure, combined with sampling and retention policies, makes continuous monitoring feasible even in low-resource settings, and

What carries the argument

The nine-field MedLog record schema is the load-bearing object. It is designed so that any AI event—human-AI, algorithm-AI, or automated workflow—maps onto the same fields, and it is assembled over time: an initial message establishes an event_id and optional run_id, and later messages append artifacts, outputs, outcomes, and feedback. This incremental immutable-message design is what lets one protocol cover both single-shot calls and multi-stage agentic traces, and the optional linkage fields make it transport- and orchestration-agnostic.

Load-bearing premise

The paper's load-bearing premise is that MedLog was actually used across four real deployments; the full text does not describe those deployments or show any MedLog records from them, so the empirical validation rests on an unshown application.

What would settle it

Examine the public MedLog repository and documentation for any real event records from the four named deployments (ICU deterioration, tetanus monitoring, sepsis reporting, patient attendance); if none exist, the claim that MedLog was applied across those deployments is unsubstantiated. Alternatively, emit MedLog records from a live ICU deterioration alert system for one month and check whether the outcomes and feedback fields can be populated from existing EHR data; if they cannot, the nine-field schema is not sufficient for the claimed monitoring.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Health systems could measure real-world AI performance and link each recommendation to downstream actions and outcomes.
  • Regulators and safety teams could detect adverse events and near misses in near-real time, and auditing frameworks would have a standard artifact to analyze.
  • Dataset shift and subgroup bias become continuously monitorable rather than discovered after the fact.
  • De-identified records aggregated across sites would enable international benchmarking of clinical AI and reveal geoeconomic disparities.
  • Logged uncertainty and feedback would support active learning and iterative model improvement from real-world failures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract claims four deployments, but the full text contains no deployment sections or data for those systems; the only concrete case study is an input-feature drift monitoring system, not a MedLog deployment. If the deployment claim is not backed by actually emitted MedLog records, the paper is best read as a protocol proposal plus an illustrative drift-detection case study.
  • If MedLog were adopted, the incremental immutable-message design could plausibly extend beyond clinical AI to other high-stakes AI domains that need auditable interaction logs.
  • A testable extension would be to instrument existing EHR-integrated AI calls with the nine-field schema and measure whether outcome linkage can be automated from EHR audit logs and provider attestations; the paper does not show this linkage in practice.
  • The protocol's success depends on coordination among EHR vendors, AI vendors, and health systems; without regulatory or payment incentives, adoption may repeat the uneven health information exchange pattern the authors themselves cite.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MedLog, a proposed protocol for event-level logging of clinical AI interactions, inspired by syslog. Each model invocation would produce a record with nine fields (header, model instance, user identity, target identity, inputs, internal artifacts, outputs, outcomes, and feedback), assembled incrementally from immutable messages. The authors discuss implementation considerations (privacy, storage, low-resource deployment, governance) and describe potential applications in surveillance, bias monitoring, drift detection, and global benchmarking. They also present a case study of input-feature drift detection at Clalit Health Services and provide a link to a proof-of-concept implementation. The abstract claims that MedLog was applied across four deployments in the US, Switzerland, and Vietnam with specific empirical findings, but the full text contains no such deployment descriptions or results.

Significance. If the protocol were established, it could fill a real gap in post-deployment monitoring of medical AI, complementing model cards and datasheets. The proposal is timely and includes useful connections to existing standards (syslog, PROV, OpenTelemetry, FHIR). The open-source proof-of-concept and the explicit discussion of low-resource settings are strengths. However, the paper's value is currently that of a well-argued proposal with an illustrative case study, not a validated system. The disconnect between the abstract's empirical claims and the body's stated future work is a serious issue that must be resolved before publication.

major comments (4)
  1. [Abstract; Code availability] The abstract asserts: 'We apply MedLog across four deployments in the US, Switzerland, and Vietnam: ICU deterioration prediction, tetanus progression monitoring from wearable signals, automated sepsis quality reporting, and patient attendance prediction' and reports specific findings such as 'AI performance degradation during severe weather events' and 'increased laboratory testing after ICU deterioration alerts.' The full text contains no deployment sections, no data, no records from these systems, and no description of how MedLog was used in them. This is contradicted by the Code availability section, which states that 'Future work and community consensus are critically needed to develop interoperability standards and test MedLog systems in real-world healthcare settings.' This discrepancy is load-bearing because the abstract's empirical validation claim is unsupported. The authors mus
  2. [Case study: AI monitoring detects real-life data drift] The sole case study does not demonstrate MedLog. The Clalit example describes monitoring of a single input feature (LDH) in a gradient-boosting hospitalization-risk model. It does not use the nine-field MedLog record, immutable message assembly, outcome linkage, or any MedLog system component. It therefore cannot validate the claim that MedLog enables detection of data drift or that MedLog was applied in practice. The case study is motivation for standardized monitoring, but it is not evidence for the protocol proposed.
  3. [Table 1; Building a MedLog system] Table 1 estimates MedLog storage as '≳GB-TB day−1 per hospital.' This figure is used to motivate infrastructure needs and lifecycle-aware retention, but no derivation or reference is provided. Storage volume is a central feasibility concern, especially for the emphasized low-resource settings. The authors should provide a transparent estimate (e.g., bytes per record, expected record volume per model, number of models) or label the range as an assumption.
  4. [Outcomes field (field 8)] The protocol lists Outcome as one of nine core fields, but the text acknowledges that outcome linkage is often indirect, delayed, and incomplete. It would strengthen the proposal to explicitly mark Outcome (and Target identity) as conditionally populated fields rather than implying they are always required; otherwise, conformance and interoperability expectations are ambiguous. This is a design clarity issue for a protocol whose value depends on consistent schema interpretation.
minor comments (5)
  1. [Abstract] Several missing spaces appear in the abstract in the PDF text ('computersystems', 'onsyslog', 'arecord', 'schemacompliant'). These are typographical issues that should be fixed in the final version.
  2. [Section 2 (MedLog records)] The nine fields are listed and described, but the section would benefit from a single normative schema table or JSON/example with all fields together; Figure 1a shows four separate examples but no consolidated schema. This would improve usability for implementers.
  3. [Case study: AI monitoring detects real-life data drift] The simulated impact results are reported as 'absolute risk scores shifted by >0.1%' and 'by >1%.' Please clarify whether these are percentage points or relative percentages, as the current phrasing is ambiguous.
  4. [Table 1] The table uses 'Outcome' while the text and Figure 1 use 'Outcomes' (plural). This inconsistency should be reconciled.
  5. [References] Reference [25] has a typo in the author list ('BagleyStevenC.,P.M.'). Several other references contain duplicated or incorrect formatting from the TeX source; a full proofread is recommended.

Circularity Check

0 steps flagged

No circularity: MedLog is a protocol proposal; the central schema definition does not reduce to its own outputs, and the admitted lack of real-world testing is a correctness gap, not a derivation loop.

full rationale

MedLog is a proposed logging schema, not a derived result. The nine-field record is defined by stipulation, and the paper's claims about what such records could enable are forward-looking applications of that definition, not predictions fitted from MedLog data. The Clalit case study describes a separate input-feature monitoring system (LDH distribution shift), and the paper explicitly states that the model was monitored via its input features rather than via MedLog records. No equation or fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing prior result by the same authors is invoked to force the design. The self-citations to OPTICA, assurance laboratories, and syslog are contextual background and governance parallels, not evidence that the MedLog schema is entailed by them. One substantive concern is present but it is a correctness risk, not circularity: the abstract claims MedLog was 'applied across four deployments' and reports specific findings (severe-weather degradation, increased lab testing after ICU alerts), yet the full text contains no deployment descriptions or MedLog records from those systems, and the Code availability section states that 'Future work and community consensus are critically needed to ... test MedLog systems in real-world healthcare settings.' That contradiction means the empirical validation claim is unsupported, but it does not make the protocol's definition circular: the schema is not defined in terms of those findings, and the findings are not derived from the schema by construction. Under the stated rules, an unsupported empirical assertion is a correctness/evidence problem, not a circularity finding.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

There are no fitted parameters or mathematical axioms. The load-bearing assumptions are empirical and design-based: every invocation can be logged, the syslog model transfers to medical AI, outcomes can be linked, and the abstract's four-deployment claim implies that MedLog was actually used. The latter is unsupported by the body text.

axioms (4)
  • domain assumption All AI model invocations can be intercepted at a collector, API gateway, or sidecar.
    MedLog's incremental record assembly depends on an observable event stream; the paper acknowledges proxies/sidecars but does not prove coverage for embedded or offline models (Section 'MedLog records are assembled incrementally').
  • domain assumption Syslog-style 'rough consensus and running code' can produce a global standard for medical AI logging.
    The syslog analogy is the paper's motivating premise, but no evidence is given that health systems will agree on nomenclature or share logs (Introduction; 'A call to action').
  • domain assumption Downstream clinical outcomes can be linked to log records via provider attestations, temporal proximity, or trial emulation.
    The Outcomes field is load-bearing for surveillance claims; the paper lists constraints but assumes partial linkage is sufficient (Section 'MedLog record', field 8).
  • ad hoc to paper Input-feature monitoring (Clalit LDH case) is evidence for MedLog's value.
    The case study demonstrates distribution-shift detection by an unspecified monitoring system, not a MedLog deployment; using it as a proof of MedLog requires an extra assumption not stated in the case study.
invented entities (1)
  • MedLog protocol independent evidence
    purpose: To define event-level logging of clinical AI across heterogeneous systems
    A concrete specification and proof-of-concept prototype on GitHub provide a falsifiable implementation target; however no production validation is reported.

pith-pipeline@v1.3.0-alltime-deepseek · 20452 in / 11860 out tokens · 92938 ms · 2026-08-04T11:31:28.941816+00:00 · methodology

0 comments
read the original abstract

Modern computer systems rely on syslog, a universal protocol that records critical events across heterogeneous infrastructure. Medicine's rapidly growing AI stack has no equivalent. As medicine deploys AI tools at scale, there is no standard way to record how, when, by whom, and for whom these models are used. Without such records, it is difficult to measure real-world performance and outcomes, detect adverse events, or identify bias and dataset drift. Here we introduce MedLog, a protocol for event-level logging of medical AI. Each time an AI model interacts with a human, another algorithm, or an automated workflow, MedLog creates a record. Each record contains nine core fields: header, model, user, target, inputs, artifacts, outputs, outcomes, and feedback. We apply MedLog across four deployments in the US, Switzerland, and Vietnam: ICU deterioration prediction, tetanus progression monitoring from wearable signals, automated sepsis quality reporting, and patient attendance prediction. MedLog records capture model behavior, workflow interactions, and downstream outcomes, including AI performance degradation during severe weather events in patient attendance prediction and increased laboratory testing after ICU deterioration alerts. MedLog limits the data footprint through risk-based sampling, lifecycle-aware retention policies, and write-behind caching, enabling deployment in low-resource settings. It also supports detailed traces for complex, agentic, or multi-stage workflows, creating a foundation for continuous monitoring, auditing, and improvement of medical AI.

Figures

Figures reproduced from arXiv: 2510.04033 by Aaron E. Boussina, Adam Rodman, Alan Karthikesalingam, Alessandro Blasimme, Ankit Sakhuja, Annelia Itwaru, Ayush Noori, Benjamin Glicksberg, Ben Kaplan, Bilal A. Mateen, Christopher A. Longhurst, C. Louise Thwaites, Daniel Yang, Dave deBronkart, David A. Clifton, Effy Vayena, Fedor Sergeev, Gauden Galea, Gunnar R\"atsch, Hai Ho Bich, Harold F. Wolf III, Ha Thi Hai Duong, Isaac S. Kohane, Jacob Waxman, James Anibal, Joerg C. Schefold, Joshua C. Mandel, Julia Maslinski, Juliana Rotich, Karandeep Singh, Kenneth D. Mandl, Lily Poursoltan, Manuel Burger, Marinka Zitnik, Martin Faltys, Maryam Mustafa, Melissa Miles, Nigam H. Shah, Noa Dagan, Pavan Bodanki, Peter Lee, Philipp Koralus, Prathamesh Parchure, Prem Timsina, Ran D. Balicer, Robert Korom, Scott Mahoney, Seth Hain, Tien Yin Wong, Trevor Mundel, Vivek Natarajan.

Figure 1
Figure 1. Figure 1: (a) Examples of clinical AI interactions that will be logged under the MedLog protocol, as well as the MedLog records they would create. (b) Timeline demonstrating that MedLog records are progressively built from a stream of messages. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: (a) Example patterns of model invocation and corresponding record creation in a MedLog implementation. De-identified MedLog records can be aggregated across healthcare systems to support downstream applications. (b) MedLog will transform medicine by enabling evaluation, auditing, and improvement of medical AI. alyze MedLog records to detect both positive [92] and negative [93] performance changes with clin… view at source ↗
Figure 3
Figure 3. Figure 3: Density plots show the distribution of the “Lactate Dehydrogenase Last Value (LDH)” feature during the training period (January 2018), immediately after the test kit change (March 2023), and in subsequent quarterly snapshots through September 2024. The introduction of the new test kit caused a gradual shift in the distribution of LDH values, which was automatically detected by the AI monitoring system. ove… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

139 extracted references · 29 canonical work pages

  1. [1]

    & Parker, J

    Palmer, K., Ross, C. & Parker, J. E.A guide to the health systems and companies driving adoption2023

  2. [2]

    doi:10.1056/AIcs2300191(2024)

    Umeton, R.et al.GPT-4 in a Cancer Center — Institute-Wide Deployment Challenges and Lessons Learned.NEJM AI1,AIcs2300191. doi:10.1056/AIcs2300191(2024)

    work page doi:10.1056/aics2300191(2024 2024

  3. [3]

    Epic introduces Launchpad to spur generative AI adoption.Fierce Healthcare (2025)

    Landi, H. Epic introduces Launchpad to spur generative AI adoption.Fierce Healthcare (2025)

    2025

  4. [4]

    American Medical Association.Physician sentiments around the use of AI in heath care: motivations, opportunities, risks, and use casestech. rep. (American Medical Association, Chicago, Illinois, 2025)

    2025

  5. [5]

    Elsevier.Clinician of the Future 2025tech. rep. (Elsevier, Amsterdam, Netherlands, 2025)

    2025

  6. [6]

    Low-Cost

    Zeng, D., Qin, Y., Sheng, B. & Wong, T. Y. DeepSeek’s “Low-Cost” Adoption Across China’s Hospital Systems: Too Fast, Too Soon?JAMA333,1866–1869. doi:10.1001/ jama.2025.6571(2025)

    arXiv 2025

  7. [7]

    Kanjee,Z.,Crowe,B.&Rodman,A.AccuracyofaGenerativeArtificialIntelligenceModel inaComplexDiagnosticChallenge.JAMA330,78–80.doi:10.1001/jama.2023.8288 (2023)

    arXiv 2023

  8. [8]

    doi:10.1001/jamanetworkopen

    Goh, E.et al.Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial.JAMA Network Open7,e2440969. doi:10.1001/jamanetworkopen. 2024.40969(2024)

    arXiv 2024

  9. [9]

    2024.0295(2024)

    Cabral, S.et al.Clinical Reasoning of a Generative Artificial Intelligence Model Compared WithPhysicians.JAMAInternalMedicine184,581–583.doi:10.1001/jamainternmed. 2024.0295(2024)

    arXiv 2024

  10. [11]

    APreliminary Studyof o1 inMedicine: AreWe Closer toan AIDoctor?2024

    Xie, Y.et al. APreliminary Studyof o1 inMedicine: AreWe Closer toan AIDoctor?2024. doi:10.48550/arXiv.2409.15277

    Pith/arXiv arXiv doi:10.48550/arxiv.2409.15277 2024

  11. [12]

    doi:10.1056/AIdbp2300192(2024)

    Katz, U.et al.GPT versus Resident Physicians — A Benchmark Based on Official Board Scores.NEJM AI1,AIdbp2300192. doi:10.1056/AIdbp2300192(2024)

    work page doi:10.1056/aidbp2300192(2024 2024

  12. [13]

    doi:10.1038/s41586-023-06291-2(2023)

    Singhal, K.et al.Large language models encode clinical knowledge.Nature620,172–180. doi:10.1038/s41586-023-06291-2(2023)

    work page doi:10.1038/s41586-023-06291-2(2023 2023

  13. [14]

    doi:10.1056/ AIoa2300138(2024)

    Tu, T.et al.Towards Generalist Biomedical AI.NEJM AI1,AIoa2300138. doi:10.1056/ AIoa2300138(2024)

    2024

  14. [15]

    W.et al.Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.JAMA Internal Medicine183, 589–596

    Ayers, J. W.et al.Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum.JAMA Internal Medicine183, 589–596. doi:10.1001/jamainternmed.2023.1838(2023). 21

    arXiv 2023

  15. [16]

    Tu,T.etal.TowardsConversationalDiagnosticAI2024.doi:10.48550/arXiv.2401. 05654

    doi:10.48550/arxiv.2401

  16. [17]

    Brodeur, P. G.et al. Superhuman performance of a large language model on the reasoning tasks of a physician2024. doi:10.48550/arXiv.2412.10849

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.10849

  17. [18]

    A., Yaramis, I

    Ahmad, M. A., Yaramis, I. & Roy, T. D.Creating Trustworthy LLMs: Dealing with Hallu- cinations in Healthcare AI2023. doi:10.48550/arXiv.2311.01463

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.01463

  18. [19]

    Kim,Y.etal.MedicalHallucinationsinFoundationModelsandTheirImpactonHealthcare

  19. [20]

    Mittermaier, M., Raza, M. M. & Kvedar, J. C. Bias in AI-based models for medical appli- cations: challenges and mitigation strategies.npj Digital Medicine6,113. doi:10.1038/ s41746-023-00858-z(2023)

    2023

  20. [21]

    L., Choma, M

    Cross, J. L., Choma, M. A. & Onofrey, J. A. Bias in medical AI: Implications for clinical decision-making.PLOS Digital Health3,e0000651. doi:10.1371/journal.pdig. 0000651(2024)

    doi:10.1371/journal.pdig 2024

  21. [22]

    Wornow, M.et al.The shaky foundations of large language models and foundation models forelectronichealthrecords.npjDigitalMedicine6,1–10.doi:10.1038/s41746-023- 00879-8(2023)

    doi:10.1038/s41746-023- 2023

  22. [23]

    Arora, R. K.et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health2025. doi:10.48550/arXiv.2505.08775

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.08775

  23. [24]

    Bedi,S.etal.MedHELM:HolisticEvaluationofLargeLanguageModelsforMedicalTasks

  24. [25]

    doi:10.1001/jama.2019.10306(2019)

    Shah,N.H.,Milstein,A.&BagleyStevenC.,P.MakingMachineLearningModelsClinically Useful.JAMA322,1351–1352. doi:10.1001/jama.2019.10306(2019)

    arXiv 2019

  25. [26]

    doi:10.48550/arXiv.2505.23802

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.23802

  26. [27]

    Retrospective Evaluation of a Generative AI-Enabled Electronic Medical Record System in Primary Health Care Facilities in Kenya2025

    Agweyu, A.et al. Retrospective Evaluation of a Generative AI-Enabled Electronic Medical Record System in Primary Health Care Facilities in Kenya2025. doi:10.1101/2025. 09.05.25335163

    doi:10.1101/2025 2025

  27. [28]

    doi:10.1038/s41591-023-02540-z(2023)

    Youssef,A.,Pencina,M.,Thakur,A.,Zhu,T.,Clifton,D.&Shah,N.H.Externalvalidation of AI models in health should be replaced with recurring local validation.Nature Medicine 29,2686–2687. doi:10.1038/s41591-023-02540-z(2023)

    work page doi:10.1038/s41591-023-02540-z(2023 2023

  28. [29]

    Collins,G.S.etal.TRIPOD+AIstatement:updatedguidanceforreportingclinicalprediction modelsthatuseregressionormachinelearningmethods.BMJ385,e078378.doi:10.1136/ bmj-2023-078378(2024)

    2023

  29. [30]

    doi:10.1016/S0140-6736(19)30037-6(2019)

    Collins,G.S.&Moons,K.G.M.Reportingofartificialintelligencepredictionmodels.The Lancet393,1577–1579. doi:10.1016/S0140-6736(19)30037-6(2019)

    work page doi:10.1016/s0140-6736(19)30037-6(2019 2019

  30. [31]

    1038/s41591-021-01229-5(2021)

    Vasey, B.et al.DECIDE-AI: new reporting guidelines to bridge the development-to- implementationgapinclinicalartificialintelligence.NatureMedicine27,186–187.doi:10. 1038/s41591-021-01229-5(2021). 22

    2021

  31. [32]

    doi:10.1038/s41591-020-0941-1(2020)

    Sounderajah, V.et al.Developing specific reporting guidelines for diagnostic accuracy studies assessing AI interventions: The STARD-AI Steering Group.Nature Medicine26, 807–808. doi:10.1038/s41591-020-0941-1(2020)

    work page doi:10.1038/s41591-020-0941-1(2020 2020

  32. [33]

    Reporting Guideline for Chatbot Health Advice Studies: The CHARTStatement.JAMANetworkOpen8,e2530220.doi:10.1001/jamanetworkopen

    The CHART Collaborative. Reporting Guideline for Chatbot Health Advice Studies: The CHARTStatement.JAMANetworkOpen8,e2530220.doi:10.1001/jamanetworkopen. 2025.30220(2025)

    arXiv 2025

  33. [34]

    Cruz Rivera, S., Liu, X., Chan, A.-W., Denniston, A. K. & Calvert, M. J. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension.Nature Medicine26,1351–1363. doi:10 . 1038 / s41591 - 020 - 1037 - 7 (2020)

    2020

  34. [35]

    Callahan, A.et al.Standing on FURM Ground: A Framework for Evaluating Fair, Useful, and Reliable AI Models in Health Care Systems.NEJM Catalyst5,CAT.24.0131. doi:10. 1056/CAT.24.0131(2024)

    2024

  35. [36]

    doi:10.1056/AIcs2300269(2024)

    Dagan, N.et al.Evaluation of AI Solutions in Health Care Organizations — The OPTICA Tool.NEJM AI1,AIcs2300269. doi:10.1056/AIcs2300269(2024)

    work page doi:10.1056/aics2300269(2024 2024

  36. [37]

    Ong, J. C. L.et al.International partnership for governing generative artificial intelligence models in medicine.Nature Medicine,1–4. doi:10.1038/s41591- 025- 03787- 4 (2025)

    doi:10.1038/s41591- 2025

  37. [38]

    Lekadir, K.et al.FUTURE-AI: international consensus guideline for trustworthy and de- ployable artificial intelligence in healthcare.BMJ388,e081554. doi:10 . 1136 / bmj - 2024-081554(2025)

    2024

  38. [39]

    Coalition for Health AI.Assurance Standards Guide2024

  39. [40]

    Epic Open Source.epic-open-source/seismometerVerona, WI, 2025

    2025

  40. [41]

    Pencina,M.J.,McCall,J.&Economou-Zavlanos,N.J.AFederatedRegistrationSystemfor ArtificialIntelligenceinHealth.JAMA332,789–790.doi:10.1001/jama.2024.14026 (2024)

    arXiv 2024

  41. [42]

    D.et al.A framework for the oversight and local deployment of safe and high- quality prediction models.Journal of the American Medical Informatics Association29, 1631–1636

    Bedoya, A. D.et al.A framework for the oversight and local deployment of safe and high- quality prediction models.Journal of the American Medical Informatics Association29, 1631–1636. doi:10.1093/jamia/ocac078(2022)

    work page doi:10.1093/jamia/ocac078(2022 2022

  42. [43]

    J., Tazbaz, T

    Warraich, H. J., Tazbaz, T. & Califf, R. M. FDA Perspective on the Regulation of Artificial Intelligence in Health Care and Biomedicine.JAMA333,241–247 (2025)

    2025

  43. [44]

    H.et al.A Nationwide Network of Health AI Assurance Laboratories.JAMA331, 245–249

    Shah, N. H.et al.A Nationwide Network of Health AI Assurance Laboratories.JAMA331, 245–249. doi:10.1001/jama.2023.26930(2024)

    arXiv 2023

  44. [45]

    doi:10.17487/RFC5424

    Gerhards, R.The Syslog ProtocolRFC 5424 (Internet Engineering Task Force, 2009). doi:10.17487/RFC5424

    work page doi:10.17487/rfc5424 2009

  45. [46]

    M.The BSD Syslog ProtocolRFC 3164 (Internet Engineering Task Force, 2001)

    Lonvick, C. M.The BSD Syslog ProtocolRFC 3164 (Internet Engineering Task Force, 2001). doi:10.17487/RFC3164

    work page doi:10.17487/rfc3164 2001

  46. [47]

    ForresterResearch,Inc.TheTotalEconomicImpact™OfMicrosoftSentinelForresterTotal Economic Impact™Study (Forrester Research, Inc., Cambridge, Massachusetts, 2024). 23

    2024

  47. [48]

    1109/TKDE.2022.3222417(2023)

    Zhang,T.,Qiu,H.,Castellano,G.,Rifai,M.,Chen,C.S.&Pianese,F.SystemLogParsing: ASurvey.IEEETransactionsonKnowledgeandDataEngineering35,8596–8614.doi:10. 1109/TKDE.2022.3222417(2023)

    arXiv 2022

  48. [49]

    OpenAI.GPT-4o System Card2024

  49. [50]

    Model Cards for Model ReportinginProceedings of the Conference on Fairness, Accountability, and Transparency(2019), 220–229

    Mitchell, M.et al. Model Cards for Model ReportinginProceedings of the Conference on Fairness, Accountability, and Transparency(2019), 220–229. doi:10.1145/3287560. 3287596

    doi:10.1145/3287560 2019

  50. [51]

    Datasheets for Datasets2021

    Gebru, T.et al. Datasheets for Datasets2021. doi:10.48550/arXiv.1803.09010

    Pith/arXiv arXiv doi:10.48550/arxiv.1803.09010

  51. [52]

    Meta.Llama 3.2 Model Card2024

  52. [53]

    S.et al.Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions.npj Digital Medicine5,1–10

    Hinson, J. S.et al.Multisite implementation of a workflow-integrated machine learning system to optimize COVID-19 hospital admission decisions.npj Digital Medicine5,1–10. doi:10.1038/s41746-022-00646-1(2022)

    work page doi:10.1038/s41746-022-00646-1(2022 2022

  53. [54]

    Williams, C. Y. K.et al.Use of a Large Language Model to Assess Clinical Acuity of Adults in the Emergency Department.JAMA Network Open7,e248895. doi:10.1001/ jamanetworkopen.2024.8895(2024)

    arXiv 2024

  54. [55]

    Advances in Neural Information Processing Systems35,24824–24837 (2022)

    Wei, J.et al.Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems35,24824–24837 (2022)

    2022

  55. [56]

    doi:10.1016/j.cell.2024.09.022(2024)

    Gao, S.et al.Empowering biomedical discovery with AI agents.Cell187,6125–6151. doi:10.1016/j.cell.2024.09.022(2024)

    work page doi:10.1016/j.cell.2024.09.022(2024 2024

  56. [57]

    doi:10.1609/aaai.v38i16.29720(2024)

    Besta, M.et al.Graph of Thoughts: Solving Elaborate Problems with Large Language Models.Proceedings of the AAAI Conference on Artificial Intelligence38,17682–17690. doi:10.1609/aaai.v38i16.29720(2024)

    work page doi:10.1609/aaai.v38i16.29720(2024 2024

  57. [58]

    Advances in Neural Information Processing Systems36,11809–11822 (2023)

    Yao, S.et al.Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Advances in Neural Information Processing Systems36,11809–11822 (2023)

    2023

  58. [59]

    doi:10.48550/arXiv.2309.13007

    Chen,J.C.-Y.,Saha,S.&Bansal,M.ReConcile:Round-TableConferenceImprovesReason- ing via Consensus among Diverse LLMs2024. doi:10.48550/arXiv.2309.13007

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.13007

  59. [60]

    Ng, K. K. Y., Matsuba, I. & Zhang, P. C. RAG in Health Care: A Novel Framework for Improving Communication and Decision-Making by Addressing LLM Limitations.NEJM AI0,AIra2400380. doi:10.1056/AIra2400380(2024)

    work page doi:10.1056/aira2400380(2024 2024

  60. [61]

    doi:10.1109/ACCESS.2022.3197671(2022)

    Saraswat, D.et al.Explainable AI for Healthcare 5.0: Opportunities and Challenges.IEEE Access10,84486–84517. doi:10.1109/ACCESS.2022.3197671(2022)

    arXiv 2022

  61. [62]

    J., Janizek, J

    DeGrave, A. J., Janizek, J. D. & Lee, S.-I. AI for radiographic COVID-19 detection selects shortcuts over signal.Nature Machine Intelligence3,610–619. doi:10.1038/s42256- 021-00338-7(2021)

    doi:10.1038/s42256- 2021

  62. [63]

    MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models2025

    Li, Z.et al. MemOS: An Operating System for Memory-Augmented Generation (MAG) in Large Language Models2025. doi:10.48550/arXiv.2505.22101

    Pith/arXiv arXiv doi:10.48550/arxiv.2505.22101

  63. [64]

    A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence

    Gao, H.-a.et al. A Survey of Self-Evolving Agents: On Path to Artificial Super Intelligence

  64. [65]

    doi:10.48550/arXiv.2507.21046

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.21046

  65. [66]

    & Adler-Milstein, J

    Kannampallil, T. & Adler-Milstein, J. Using electronic health record audit log data for re- search:insightsfromearlyefforts.JournaloftheAmericanMedicalInformaticsAssociation : JAMIA30,167–171. doi:10.1093/jamia/ocac173(2022)

    work page doi:10.1093/jamia/ocac173(2022 2022

  66. [67]

    doi:10.48550/arXiv.2501.07278

    Zheng,J.etal.LifelongLearningofLargeLanguageModelbasedAgents:ARoadmap2025. doi:10.48550/arXiv.2501.07278

    doi:10.48550/arxiv.2501.07278

  67. [68]

    Guo,Y.,Cheng,Z.,Tang,X.,Tu,Z.&Lin,T.DynamicMixtureofExperts:AnAuto-Tuning ApproachforEfficientTransformerModels2025.doi:10.48550/arXiv.2405.14297. 24

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.14297

  68. [69]

    doi:10.1038/s41467-024- 52618-6(2024)

    Yang,J.etal.GeneralizabilityassessmentofAImodelsacrosshospitalsinalow-middleand high income country.Nature Communications15,8270. doi:10.1038/s41467-024- 52618-6(2024)

    doi:10.1038/s41467-024- 2024

  69. [70]

    ISO/TC 215.ISO 27799:2016: Health informatics — Information security management in health using ISO/IEC 270022016

    2016

  70. [71]

    ISO/IEC JTC 1/SC 27.ISO/IEC 27001:2022: Information security, cybersecurity and pri- vacy protection — Information security management systems — Requirements2022

    2022

  71. [72]

    doi:10.1109/ACCESS.2017.2730843(2017)

    Xia,Q.,Sifah,E.B.,Asamoah,K.O.,Gao,J.,Du,X.&Guizani,M.MeDShare:Trust-Less Medical Data Sharing Among Cloud Service Providers via Blockchain.IEEE Access5, 14757–14767. doi:10.1109/ACCESS.2017.2730843(2017)

    arXiv 2017

  72. [73]

    Embi, P. J. Algorithmovigilance—Advancing Methods to Analyze and Monitor Artificial Intelligence–Driven Health Care for Effectiveness and Equity.JAMA Network Open4, e214622. doi:10.1001/jamanetworkopen.2021.4622(2021)

    arXiv 2021

  73. [74]

    doi:10.1038/s41746-020-00323-1(2020)

    Rieke, N.et al.The future of digital health with federated learning.npj Digital Medicine3, 1–7. doi:10.1038/s41746-020-00323-1(2020)

    work page doi:10.1038/s41746-020-00323-1(2020 2020

  74. [75]

    1038/s41591-018-0316-z(2019)

    Esteva,A.etal.Aguidetodeeplearninginhealthcare.NatureMedicine25,24–29.doi:10. 1038/s41591-018-0316-z(2019)

    2019

  75. [76]

    Banks,M.A.Sizingupbigdata.NatureMedicine26,5–6.doi:10.1038/s41591-019- 0703-0(2020)

    work page doi:10.1038/s41591-019- 2020

  76. [77]

    & Jiang, X

    Telenti, A. & Jiang, X. Treating medical data as a durable asset.Nature Genetics52,1005–

  77. [78]

    W.et al.Cooking Up An Open Source EMR For Developing Countries: Open- MRS–ARecipeForSuccessfulCollaboration.AMIAAnnualSymposiumProceedings2006, 529–533 (2006)

    Mamlin, B. W.et al.Cooking Up An Open Source EMR For Developing Countries: Open- MRS–ARecipeForSuccessfulCollaboration.AMIAAnnualSymposiumProceedings2006, 529–533 (2006)

    2006

  78. [79]

    Dehnavieh, R.et al.The District Health Information System (DHIS2): A literature review and meta-synthesis of its strengths and operational challenges based on the experiences of 11 countries.Health Information Management Journal48,62–75. doi:10 . 1177 / 1833358318777713(2019)

    2019

  79. [80]

    PROV-O: The PROV OntologyW3C Recommendation (World Wide Web Consortium (W3C), Cambridge, MA, 2013)

    Khalid Belhajjameet al. PROV-O: The PROV OntologyW3C Recommendation (World Wide Web Consortium (W3C), Cambridge, MA, 2013)

    2013

  80. [81]

    Office of the National Coordinator for Health Information Technology.Hospital Selection of Public Health Measures in Medicare EHR Incentive Programtech. rep. 16 (Washington D.C., 2016). 25

    2016

Showing first 80 references.