Pith. sign in

REVIEW 3 major objections 5 minor 7 references

Medical Image De-Identification Resources: Synthetic DICOM Data and Tools for Validation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that de-identification of medical imaging data can be objectively benchmarked: it releases a DICOM dataset with 53,581 images injected with synthetic patient identifiers, plus an answer key and validation script that…

desk verdict Solid resource paper with a real dataset and validation tool; the answer-key circularity risk is disclosed and addressable, so don't desk-reject. read the letter →

arxiv 2508.01889 v1 pith:MPUQWOLU submitted 2025-08-03 cs.CV

Michael W. Rutherford , Tracy Nolan , Linmin Pei , Ulrike Wagner , Qinyan Pan , Phillip Farmer , Kirk Smith , Benjamin Kopchick
show 5 more authors
Laura Opsahl-Ong Granger Sutton David Clunie Keyvan Farahani Fred Prior
This is my paper · ORCID
classification cs.CV
keywords SyntheticDataDICOMDe-identificationValidationMedicalImagingPHIPIIHIPAA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that de-identification of medical images can be measured rather than merely hoped for. To that end it releases the MIDI dataset: 53,581 DICOM instances built from publicly available de-identified images, into which synthetic patient identifiers were deliberately injected in structured metadata, free-text fields, and burned-in pixels. A matching answer key records the exact transformation that would reverse each injected identifier, and a validation script compares any de-identified output against that key, producing pass/fail and percentage scores per action. If the framework works as claimed, developers and institutions can benchmark de-identification tools against a known ground truth aligned with HIPAA Safe Harbor and DICOM PS3.15, instead of relying on spot checks and expert opinion.

What carries the argument

The load-bearing object is the answer key: a per-record table of expected transformations keyed by SOP Instance UID, scope (study, series, or instance), tag path and name, synthetic file value, expected action, and action text. Actions include text removed, text retained, date shifted, uid changed, uid consistent, patid consistent, pixels hidden, and tag retained, each tagged with a source category such as HIPAA, DICOM, or TCIA. The validation script evaluates these actions token-by-token for text, OCR-checks specified pixel rectangles, verifies UID and Patient ID mapping consistency, and logs per-action results to SQLite, with an additional DICOM-conformance check applied to confirm de-identification does not worsen standard conformance. This combination converts de-identification from a binary 'looks clean' judgment into a deterministic, auditable score.

What would settle it

Feed the MIDI validation script a dataset that has had zero de-identification performed, so every synthetic identifier remains; the script should flag failures across every action type, and any category that still reports a 100% pass reveals a gap in the answer key's coverage. A complementary check: a tool that aggressively erases all DICOM elements and blanks all pixel images should score poorly on text-retained and tag-retained actions even though privacy is trivially achieved, showing the benchmark also measures utility preservation.

Watch

Extended reading notes

Core claim

The central discovery is that a public, standards-aligned ground truth for image de-identification can be built by injecting synthetic PHI/PII into real DICOM structures and logging every injection as a reversible action. The resource covers 538 subjects, 605 studies, 708 series, and 53,581 instances across multiple modalities and vendors, with synthetic identifiers placed in structured data elements, plain text elements, and pixel regions. The accompanying validation script walks every DICOM instance, compares header elements and specified pixel rectangles against the answer key, checks UID and Patient ID consistency via mapping files, and reports discrepancies at action, category, and series level. In the example run, the reference curation pipeline scored 99.45% pass on validation and 99.25% on test, with most failures attributed to conservative text retention and pre-existing data quirks.

Load-bearing premise

The whole evaluation collapses if the answer key is not an independent, correct statement of what de-identification must do: the paper reports that the key was incrementally revised after test runs with one specific de-identification product, so the ground truth may encode that tool's behavior rather than standing apart from all tools being benchmarked.

Editorial extensions

If this is right

  • A de-identification tool that passes MIDI at 100% on both subsets has demonstrably removed or transformed every synthetic identifier the dataset carries, giving regulators and data custodians a concrete, reproducible basis for confidence.
  • New tools can be regression-tested against the public answer key: a change in output that drops the pass rate is immediately visible at action and category level, not buried in a manual review.
  • Because the answer key separates HIPAA, DICOM, and TCIA categories, score reports show not just whether privacy risks were removed but which regulatory standard drove each action.
  • The framework can be reused for quality assurance beyond privacy, since it also catches missing tags, UID mismatches, formatting errors, and DICOM-conformance drift introduced by the de-identification process.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same answer-key mechanism could be applied prospectively: an institution creates its own synthetic dataset with injected identifiers before deploying a de-identification pipeline, effectively turning the benchmark into a continuous integration test for privacy.
  • If the answer key is extended with realistic multimodal cases such as vendor-specific private tags, structured reports, and inter-instance date coherence, the benchmark could cover failure modes the authors themselves flag as currently untested, including whether shifted dates stay consistent across related objects.
  • A natural next experiment is to compare tools head-to-head on the same MIDI subsets and report per-action scores; those scores would quantify the practical trade-off between privacy (text removed) and utility (text retained) that the current aggregate pass rate only hints at.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents the Medical Image De-identification (MIDI) resource: a synthetic DICOM dataset of 538 subjects and 53,581 instances built from TCIA data, with synthetic PHI/PII inserted into structured metadata, free-text fields, and pixel data; an answer key recording expected de-identification actions; mapping files for Patient IDs and UIDs; and a Python validation script that scores a de-identified dataset against the answer key with per-action, token-level checks. The authors report a demonstration in which the TCIA curation pipeline processed the dataset and the validation script produced 99.45% and 99.25% pass rates on the validation and test subsets. The paper claims that the framework provides objective, standards-aligned (HIPAA Safe Harbor, DICOM PS3.15) benchmarking of de-identification workflows.

Significance. If the resources work as described, this is a valuable contribution: it addresses a real gap in reproducible de-identification evaluation, ships a sizeable public dataset and open-source validation script, and its action taxonomy (date shifted, text removed, pixels hidden, uid consistent, etc.) is useful for reporting and auditing. The paper is also transparent about limitations, including curation-judgment dependence, inherited data inconsistencies, and the need for mapping files in pseudonymization workflows. The main caveat is that the answer key was refined after testing a specific commercial de-identification product, so the objectivity of the benchmark hinges on demonstrating that the refinements were principled and independent of that tool. The TCIA-curated demonstration is a useful sanity check but does not by itself resolve that concern.

major comments (3)
  1. [Section 6.1; Section 3.1] The final answer key was produced after running the Google Cloud de-identification product: Section 6.1 states that issues with the product, the dataset, and the answer key 'were addressed incrementally to refine the dataset and generate the final version of the answer key,' while Section 3.1 states that actions for some elements are 'consensus value judgments' that 'can change at the discretion of the curator.' This creates a circularity risk: if answer-key entries were changed to align with what a specific tool outputs, subsequent benchmark scores will reward tool-like behavior rather than measure independent PHI/PII removal quality. This is load-bearing for the paper's central 'objective, standards-driven evaluation' claim. Please provide a versioned changelog of the answer-key revisions, with each change justified by DICOM PS3.15, HIPAA Safe Harbor, or the TCIA private-tag knowledgebase rather than by GCP output, and report an evaluation using a de-identification tool that did not participate in answer-key refinement (or a pre-registered frozen answer key).
  2. [Section 6.2, Tables 19-20] The TCIA-curated demonstration supports that the script runs end-to-end, but the reported pass rates do not by themselves establish the answer key as an independent ground truth. TCIA and the MIDI resource share institutional context (UAMS/TCIA; Sections 1 and 5.1.2), so high agreement may reflect shared curation conventions. Moreover, only a minority of failures are accounted for by the 'Pre' and 'Mis' labels: in the test subset, 1,335 actions failed, of which 157 are 'Pre' and 220 are 'Mis', leaving 958 failures (including 789 'text retained' and 246 'text removed') that are attributed qualitatively to conservative handling. Please quantify how many failures reflect judgment-dependent answer-key choices versus genuine tool deficiencies, and discuss how different reasonable curator decisions would change the scores.
  3. [Section 5.3.3] The pixel evaluation checks OCR in fixed rectangular regions defined by answer-key coordinates. The paper does not state how the script behaves if a de-identification pipeline resamples, crops, lossily compresses, or otherwise changes image geometry, which would invalidate the coordinate-based check. Since burned-in pixel PHI/PII is a core part of the claimed evaluation coverage, please state the geometric-invariance assumptions and either handle or explicitly exclude such transformations.
minor comments (5)
  1. [Section 5.1.2] The sentence 'These files were removed to correct the anomaly and ensure DICOM compliance' repeats 'removal of files' from the preceding sentence; please rephrase for readability.
  2. [Table 2] The row totals in Table 2 (587 patients, 661 studies, 709 series) differ from Table 1 (538/605/708); the text explains the reason, but a footnote in Table 2 would make the discrepancy less confusing for readers.
  3. [Section 5.3.3] The script computes a continuous per-action score but final scoring uses only binary pass/fail; please justify this choice or clarify why the continuous score is reported if it does not influence the outcome.
  4. [Section 6.1] Since an earlier version of the answer key was refined, please specify the versioning scheme for the answer key and dataset so users can reproduce the exact reported numbers.
  5. [Section 5.3.5 and Table 18] The dciodvfy report is described as validating DICOM conformance, but the paper should state that dciodvfy warnings are not necessarily de-identification failures; consider separating conformance issues from PHI/PII evaluation in the summary reports.

Circularity Check

1 steps flagged · score 4.0 of 10

Final answer key was refined using GCP de-identification output, making benchmark scores partly self-referential; otherwise the resource is self-contained.

  1. other [Section 6.1 (Validation, Description of Approaches Ensuring the Quality of Data)]
    "The results of these tests were aggregated and reviewed by an informal group of NCI staff, contractors, and advisors, who identified issues with the GCP de-identification product, the synthetic dataset, and the answer key. These issues were addressed incrementally to refine the dataset and generate the final version of the answer key."

    The final answer key is the ground truth against which every de-identification tool is scored by the validation script. Because that key was revised after observing the outputs of one specific tool (Google Cloud's de-identification product), subsequent evaluation scores are not fully independent of GCP's behavior: if any answer-key entry was changed to accept a transformation GCP performs, then a tool producing that transformation will pass by construction, not because it is better at PHI/PII removal. Section 3.1 reinforces this concern by stating that actions for some elements are 'consensus value judgments' and 'can change at the discretion of the curator,' so the refinement in Section 6.1 is not guaranteed to have been limited to objective bug fixes.

full rationale

The core resource is largely self-contained: synthetic PHI/PII were inserted by rule-based templates, each insertion was logged at the time of insertion to form the answer key, and the validation script compares tool output against that logged truth. That design is not circular in itself. The one significant feedback loop is in Section 6.1: the final answer key was refined after test runs of the Google Cloud de-identification product, so the ground truth used for all subsequent scoring is not fully independent of that specific tool. Section 3.1 reinforces that many answer-key actions are curator judgment calls, so the refinement cannot be assumed to have been confined to objective corrections. This weakens the 'objective benchmarking' claim for tools whose behavior resembles GCP, though it does not invalidate the resource: the majority of answer-key entries are mechanical (date shifts, UID and patient-ID consistency), the TCIA-curated validation (99.45%/99.25% pass) demonstrates consistency with an independent curation pipeline, and the insertion log would still support an objective ground truth if the post-hoc GCP adjustments were isolated and documented as bug fixes. Score 4 reflects this partial, limited circularity rather than a fully self-referential derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no new physical entities; the free parameters are design choices in the synthetic data generator. The assumptions are the standards and curation practices used to define ground truth, plus the representativeness of the rule-based insertion templates.

free parameters (3)
  • Patient-specific date shift offsets
    Dates were shifted by per-patient offsets to preserve temporal relationships (Section 5.1.2). The offsets are hand-chosen parameters in the synthetic data generation, not derived from data.
  • Series modification percentages
    Templates applied changes to 'a defined percentage of series' (Section 5.2.2), a hand-selected knob that controls how often synthetic values are inserted.
  • Pixel burn-in font size and bounding boxes
    The JSON object in Figure 2 specifies text content, bounding box coordinates, and font size for burned-in pixels; these are manually chosen per image.
assumptions (4)
  • domain assumption DICOM PS3.15 Confidentiality Profiles define the correct de-identification actions.
    The answer key actions are modeled after the Basic Application Confidentiality Profile and its options (Section 5.1.2).
  • domain assumption HIPAA Privacy Rule Safe Harbor identifier list is an appropriate ground truth for what must be removed.
    The framework aligns with HIPAA Safe Harbor (Section 3), and answer key categories reference HIPAA codes.
  • domain assumption TCIA curation practices and the private tag knowledgebase represent correct handling of private data elements.
    Answer key entries are categorized as TCIA-PTKB or TCIA-REV, based on years of TCIA curation experience (Section 3.1).
  • ad hoc to paper The rule-based synthetic insertion templates adequately model real-world PHI/PII leak locations.
    The paper relies on hand-crafted templates to place synthetic identifiers, and Section 3.1 admits these may not capture the full complexity of real-world imaging environments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Medical Image De-Identification Resources: Synthetic DICOM Data and Tools for Validation." pith.science (2026). https://pith.science/paper/MPUQWOLU

@misc{pith2026250801889,
  author       = {Pith},
  title        = {Pith review of: Medical Image De-Identification Resources: Synthetic DICOM Data and Tools for Validation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPUQWOLU}},
  note         = {Machine review of arXiv:2508.01889}
}
read the original abstract

Medical imaging research increasingly depends on large-scale data sharing to promote reproducibility and train Artificial Intelligence (AI) models. Ensuring patient privacy remains a significant challenge for open-access data sharing. Digital Imaging and Communications in Medicine (DICOM), the global standard data format for medical imaging, encodes both essential clinical metadata and extensive protected health information (PHI) and personally identifiable information (PII). Effective de-identification must remove identifiers, preserve scientific utility, and maintain DICOM validity. Tools exist to perform de-identification, but few assess its effectiveness, and most rely on subjective reviews, limiting reproducibility and regulatory confidence. To address this gap, we developed an openly accessible DICOM dataset infused with synthetic PHI/PII and an evaluation framework for benchmarking image de-identification workflows. The Medical Image de-identification (MIDI) dataset was built using publicly available de-identified data from The Cancer Imaging Archive (TCIA). It includes 538 subjects (216 for validation, 322 for testing), 605 studies, 708 series, and 53,581 DICOM image instances. These span multiple vendors, imaging modalities, and cancer types. Synthetic PHI and PII were embedded into structured data elements, plain text data elements, and pixel data to simulate real-world identity leaks encountered by TCIA curation teams. Accompanying evaluation tools include a Python script, answer keys (known truth), and mapping files that enable automated comparison of curated data against expected transformations. The framework is aligned with the HIPAA Privacy Rule "Safe Harbor" method, DICOM PS3.15 Confidentiality Profiles, and TCIA best practices. It supports objective, standards-driven evaluation of de-identification workflows, promoting safer and more consistent medical image sharing.

Figures

Figures reproduced from arXiv: 2508.01889 by the authors.

Figure 1
Figure 1. Processing steps for the creation of the datasets. In addition to the TCIA audit log analysis, we utilized the DICOM Standard and the HIPAA Privacy Rule’s 18 Safe Harbor elements as guidance to identify standard DI￾COM data elements suitable for synthetic value insertion and address the omission of common fields in the log anal￾ysis. This approach ensured comprehensive insertion of synthetic values, including patien… view at source ↗
Figure 2
Figure 2. An example of burned-in pixels and the JSON data used to represent it. example image with embedded text and the corresponding JSON object, which defines the text content, bounding box coordinates, and font size. 5.3 Software Stack The validation software resource was built using Python 3.8 and tested to work with Python 3.9 and 3.10. The resource can be run on Windows, Mac, and Linux and depends on multiple existing… view at source ↗
Figure 4
Figure 4. Validation Script Process Flow. success in medical imaging datasets. As described in the Methods section, the tool compares de-identified DICOM files against a structured answer key, verifying that pre￾scribed transformations were correctly applied to metadata and pixel data. Each evaluation run produces a SQLite database of comparison outcomes and generates summary reports in both machine-readable and human-readabl… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 5 canonical work pages

  1. [7]

    URL http://www.nature.com/articles/ s41597-021-00967-y . U.S. Dept. of Health and Human Services. Guidance Regarding Methods for De-identification of Pro- tected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule, 2012. URL https://www. hhs.gov/hipaa/for-professionals/privacy/ special-topics/...

    work page 2012

  2. [1323]

    URL http://pubs.rsna.org/doi/10.1148/ rg.2015140244. NEMA. Digital Imaging and Communications in Medicine (DICOM) Standard, a. URL http://www. dicomstandard.org/. NEMA. Digital Imaging and Communications in Medicine (DICOM) Standard PS3.15: Security and System Man- agement Profiles, b. URL http://dicom.nema.org/. Vasileios C. Pezoulas, Dimitrios I. Zaridi...

    work page

  3. [1696]

    Michael Rutherford, Seong K

    URL http://www.nature.com/articles/ nbt.3696. Michael Rutherford, Seong K. Mun, Betty Levine, William Bennett, Kirk Smith, Phil Farmer, Quasar Jarosz, Ulrike 15 Michael W. Rutherford and Fred Prior, 2025 Wagner, John Freyman, Geri Blake, Lawrence Tarbox, Keyvan Farahani, and Fred Prior. A DICOM dataset for evaluation of medical image de-identification. Sc...

    work page 2025

  4. [2023]

    URL https://dx.plos

    ISSN 2767-3170. . URL https://dx.plos. org/10.1371/journal.pdig.0000082. Google Cloud. Cloud Healthcare API documentation. URL https://cloud.google.com/healthcare-api/ docs. Haridimos Kondylakis, Rocio Catalan, Sara Martinez Al- abart, Caroline Barelle, Paschalis Bizopoulos, Ma- ciej Bobowicz, Jonathan Bona, Dimitrios I. Fotiadis, Teresa Garcia, Ignacio G...

    doi:10.1371/journal.pdig.0000082

  5. [4463]

    Oliver Diaz, Kaisar Kushibar, Richard Osuala, Akis Linar- dos, Lidia Garrucho, Laura Igual, Petia Radeva, Fred Prior, Polyxeni Gkontra, and Karim Lekadir

    URL https://www.nature.com/articles/ s41597-023-02256-2 . Oliver Diaz, Kaisar Kushibar, Richard Osuala, Akis Linar- dos, Lidia Garrucho, Laura Igual, Petia Radeva, Fred Prior, Polyxeni Gkontra, and Karim Lekadir. Data preparation for artificial intelligence in medical imaging: A comprehensive guide to open-access platforms and tools. Physica Medica , 83:2...

    work page 2021

  6. [6352]

    URL https://www.nature.com/articles/ s41746-021-00507-3 . John B. Freymann, Justin S. Kirby, John H. Perry, David A. Clunie, and C. Carl Jaffe. Image Data Shar- ing for Biomedical Research—Meeting HIPAA Require- ments for De-identification. Journal of Digital Imaging , 25(1):14–24, February 2012. ISSN 0897-1889, 1618- 727X. . URL http://link.springer.com/...

    work page 2012

  7. [7079]

    Quentin Langlois, Nicolas Szelagowski, Jean Vander- donckt, and S ´ebastien Jodogne

    URL https://www.ncbi.nlm.nih.gov/pmc/ articles/PMC6502465/. Quentin Langlois, Nicolas Szelagowski, Jean Vander- donckt, and S ´ebastien Jodogne. Open Platform for the De-identification of Burned-in Texts in Medical Images using Deep Learning:. In Proceedings of the 17th International Joint Conference on Biomedical Engineering Systems and Technologies , pa...

    arXiv 2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.