Pith. sign in

REVIEW 4 major objections 4 minor 19 references

A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Pith's one-line read A hybrid rule-plus-AI pipeline de-identifies DICOM files to 99.91% accuracy on the MIDI-B benchmark.

desk verdict Solid, honest challenge write-up with open code and a strong benchmark number; just don't mistake the per-tag score for a PHI-safety rate, and the discrepancy stats need a second pass. read the letter →

arxiv 2509.00437 v1 pith:5DHG4ANS submitted 2025-08-30 cs.CR

classification cs.CR
keywords DICOMde-identificationPHI/PIIremovalhybridrule-basedAIRoBERTaNLPPaddleOCRprivatetagsMIDI-Bbenchmarkmedicalimagingprivacy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that reliable DICOM de-identification comes from a division of labor: rule-based logic handles structured metadata and private vendor tags, while a transformer NLP model handles only free text and OCR-extracted image text, with a DICOM validator closing gaps. The paper shows that this hybrid, refined over several rounds, reaches 99.91% accuracy on the MIDI-B test dataset, up from 84.36% for the initial design that applied AI broadly to all fields. The result isolates a practical architecture decision—keeping AI away from structured fields—rather than proposing a single new model, and it quantifies where remaining failures concentrate: missed PHI detection and private-tag mishandling.

What carries the argument

The central mechanism is a division of labor enforced across four components: DCMTCIADeidentifier, a rule-based de-identifier following TCIA's Safe Harbor method with 40 custom rules; PrivateTagsDeidentifier, which maps each private tag to a processing rule using an 8,788-entry vendor dictionary; DCMImageDeidentifier, which uses PaddleOCR to find burned-in text and masks detected PHI boxes; and DcmValidator, which runs dciodvfy and fills missing attributes. The RoBERTa PHI detector is deliberately applied only to free-text fields and OCR output, not to structured tags.

What would settle it

Manually audit the 8,676 test-set mismatches and count how many of the 0.0-check-score tags contain genuine PHI (real patient names, dates, or IDs) rather than benign metadata differences; if a sizable fraction are real identifiers, the near-perfect benchmark score overstates privacy safety.

Watch

Extended reading notes

Core claim

The paper's central claim is that a de-identification framework combining a DICOM PS 3.15/TCIA-Safe-Harbor rule-based core, a RoBERTa-based PHI detector restricted to free text, PaddleOCR for burned-in image text, a curated private-tag dictionary, and the dciodvfy validator achieves near-perfect de-identification on the MIDI-B test set (99.91%). The iterative validation trajectory is the evidence: swapping the initial PS 3.15 rules for TCIA Safe Harbor rules raised accuracy from 84.36% to 94.71%; restricting AI to free text and adding 40 custom rules, then private-tag handling, then validation, pushed the score to 99.98% on validation and 99.91% on the unseen test set. The paper also reports

Load-bearing premise

The 99.91% score assumes the MIDI-B automated report really measures privacy safety, and that an NLP model trained on clinic notes works on DICOM text and scanned image text without retraining.

Editorial extensions

If this is right

  • Using the same rule-based core for structured data and reserving AI for free text outperforms applying AI to all fields, so future DICOM de-identification systems can start from this split rather than a monolithic model.
  • Private tags are a major source of error: adding the vendor dictionary was one of the largest single accuracy gains, so any deployment must inventory private tags before claiming safety.
  • Post-de-identification validation is not optional: the dciodvfy step caught missing attributes and contributed the final accuracy jump, and it keeps files usable after PHI removal.
  • On the test set, most remaining errors are missed PHI and private-tag rule gaps; improving those two areas should be the next target rather than tuning the NLP model's architecture.
  • The reported 99.91% is specific to the MIDI-B scoring definition; on this benchmark it implies near-total removal of expected identifiers, but the paper itself cautions against assuming the same rate on other institutional data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension outside the paper: run the same pipeline on a multi-institution corpus after stripping the private-tag dictionary; if accuracy drops sharply, the private-tag coverage is vendor-specific and the headline score overstates generality.
  • The check-score analysis suggests many mismatches are partial text overlaps rather than complete leaks, so a manual audit of the 0.0-score tags would tell whether the remaining errors are genuine PHI or benign formatting differences.
  • Because the PHI detector was trained on clinical narratives, fine-tuning it on DICOM free text and OCR output—flagged as future work in the paper—is likely to reduce the 59.3% PHI-detection failure share more than any rule change.
  • If benchmark accuracy is used as a procurement criterion, institutions should weight private-tag coverage and a manual spot-check, since the automated score rewards exact expected output rather than privacy semantics alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper describes a hybrid DICOM de-identification framework developed for the MIDI-B challenge. The system combines a rule-based DICOM metadata de-identifier (TCIA Safe Harbor / DICOM PS 3.15 based) with PaddleOCR for burned-in image text and a RoBERTa model fine-tuned on i2b2 2014 for PHI detection in free text. The authors report iterative improvements on the validation set and a final test accuracy of 99.91% on MIDI-B. They also describe a private-tag de-identifier, a DICOM validator integration, a 150-term imaging whitelist to reduce false positives, and a discrepancy analysis of test failures. The code is publicly available on GitHub.

Significance. If the reported 99.91% MIDI-B test score is taken at face value, the work demonstrates a competitive, openly available hybrid de-identification pipeline for DICOM files, with useful engineering details such as TCIA custom rules, private-tag handling, and validator-based repair. The main strengths are the external challenge evaluation, the public implementation, and the explicit breakdown of failure categories. However, the paper's central interpretation of the score as 'near-perfect de-identification' is not supported without a precise definition of the evaluation metric and a separate PHI-level recall measure. The authors' own failure analysis reports 5,145 mismatches attributed to missed PHI detection, which indicates that a high per-tag score can coexist with many potentially sensitive values remaining in the output. The paper is therefore useful as a systems description but needs substantial revision to make its claims about de-identification safety commensurate with the evidence.

major comments (4)
  1. [Section 4, Table 2 and Figure 3] The headline 99.91% is called 'accuracy', but the underlying MIDI-B metric (check_score) is never defined. The reader cannot tell whether the score is computed per DICOM tag, per file, or per value, or how partial credit is awarded. The presence of partial scores in Figure 3 (e.g., 0.7–0.8) shows that the metric is not a simple binary PHI detection rate. The Discussion (Section 5) equates this score with 'near-perfect de-identification', which overstates what the metric establishes. Please define the metric explicitly, report confidence intervals or at least totals, and provide entity-level precision/recall for PHI, not just an aggregate tag score.
  2. [Section 4, discrepancy analysis / Figure 3] The text states that there are 8,676 mismatches from the test data, but the check-score distribution in Figure 3 sums to 5,144 (2,528 + 582 + 204 + 305 + 1,130 + 395). The percentage 49.1% is computed as 2,528/5,144, yet the text presents this as a fraction of the mismatched tags, implying 2,528/8,676 ≈ 29.1%. This internal inconsistency makes the failure analysis uninterpretable. If Figure 3 covers only a subset of mismatches, state that explicitly; if it is meant to cover all mismatches, correct the counts.
  3. [Section 4, Figure 2] The failure category 'limitations in PHI detection' accounts for 5,145 cases, described as 'missed identification of sensitive information'. If these correspond to actual PHI values that remain in the de-identified output, then the framework has leaked thousands of protected values even though the benchmark score is 99.91%. This directly contradicts the 'near-perfect de-identification' claim. The paper should either provide a separate PHI-level analysis showing how many of those 5,145 mismatches were true PHI, or substantially weaken the claim in Section 5. As written, the paper's own failure breakdown undermines the central safety claim.
  4. [Section 3.3, whitelist] The 150-term imaging whitelist excludes detections overlapping curated anatomy/case terms to reduce RoBERTa false positives. No analysis is provided to show that this whitelist never removes a true patient identifier (e.g., a surname that coincides with an anatomy term or a laterality abbreviation). Since this mechanism directly affects de-identification safety, a false-negative analysis or a more conservative, context-aware filtering strategy should be described and justified.
minor comments (4)
  1. [Table 1] Typo in the second row: 'DCMTICADeIdentifier' should read 'DCMTCIADeIdentifier'. The footnote about 'self-derived method' is unclear and should be rewritten; it is not apparent how a validation score can be derived from the difference from a previous score.
  2. [Table 3] The two-column layout with the repeated header 'Tag Name Failed' is confusing. It is unclear whether the second column is a continuation or a separate set of tags. Please reformat the table to make the pairs explicit.
  3. [Section 3.2.2 / References] The text cites 'obi/deid_roberta_i2b2', while the reference list entry is titled 'obi/deid_bert_i2b2'. Please align the model name and reference.
  4. [Figure 3] The term 'check_score' is used without a formal definition in the text. Define the score and its range before presenting the distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical score on an external benchmark, not a derivation from the paper's own inputs.

full rationale

The paper makes no mathematical derivation chain; its central claim (99.91% accuracy on the MIDI-B test dataset, Section 4, Table 2) is an empirical evaluation against an externally defined challenge benchmark. Iterative refinement on the validation set and subsequent testing on a held-out test set is standard machine-learning practice and does not constitute fitting the test result. The components (RoBERTa trained on i2b2, PaddleOCR, TCIA rules, dciodvfy validator) are all external resources, and the paper contains no load-bearing self-citations or imported uniqueness theorems. The Table 1 footnote states that the final validation-phase score of 99.98% was 'computed using a self-derived method,' but the test-phase score in Table 2 is reported from the challenge's validation script. The Discussion explicitly concedes that the score 'may not generalize,' and the paper's own discrepancy analysis shows 5,145 missed PHI detections, but these are correctness/overfitting concerns about the chosen metric, not circular reasoning. The evaluation is self-contained against an external benchmark, so the circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on external benchmark definitions, external rule sets and models, and hand-tuned lists. The paper provides open code but no full enumeration of the whitelist or custom rules, so several load-bearing components are not independently inspectable from the text.

free parameters (3)
  • imaging_term_whitelist = 150 terms (CT, MR, T1, abdomen, ...)
    Hand-curated from a RadioPaedia article (Section 3.3) to suppress RoBERTa false positives. Changing this list changes the reported accuracy.
  • custom_rules_v2 = 40 rules
    Hand-written modifications to TCIA rules (remove allergies, occupation, comments; retain station name, device UID) tuned during validation (Section 3.2.5).
  • date_shift_days = 120 (default)
    Configurable number of days used to shift date tags (Section 3.2.3). Chosen by hand; not central to the headline accuracy but affects date consistency.
assumptions (5)
  • domain assumption The MIDI-B evaluation metric and discrepancy report are a valid measure of de-identification completeness.
    The headline 99.91% is only as meaningful as the benchmark's ground-truth definition. Section 4 reports accuracy without specifying how mismatches are weighted or whether all PHI categories matter equally.
  • domain assumption The TCIA rule table and the 8,788-entry private tag dictionary comprehensively cover required DICOM de-identification behavior.
    Sections 3.2.4 to 3.2.6 rely on these external resources without independently auditing their completeness; private-tag handling accounted for 36.8% of test mismatches.
  • domain assumption The i2b2-trained RoBERTa model transfers to DICOM free text and OCR output without DICOM-specific fine-tuning.
    Section 3.2.2 uses obi/deid_roberta_i2b2 directly. The Discussion concedes this may not generalize and suggests DICOM-specific training as future work.
  • domain assumption PaddleOCR detects all burned-in PHI text in medical images; text not detected by OCR is treated as absent.
    Section 3.2.7 obscures only bounding boxes returned by PaddleOCR. No OCR recall analysis is provided.
  • ad hoc to paper The 150-term whitelist correctly separates imaging anatomy terms from PHI without filtering real patient identifiers.
    The whitelist is introduced in Section 3.3 specifically for this challenge and is not independently validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge." pith.science (2026). https://pith.science/paper/5DHG4ANS

@misc{pith2026250900437,
  author       = {Pith},
  title        = {Pith review of: A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5DHG4ANS}},
  note         = {Machine review of arXiv:2509.00437}
}
read the original abstract

Ensuring the de-identification of medical imaging data is a critical step in enabling safe data sharing. This paper presents a hybrid de-identification framework designed to process Digital Imaging and Communications in Medicine (DICOM) files. Our framework adopts a modified, pre-built rule-based component, updated with The Cancer Imaging Archive (TCIA)'s best practices guidelines, as outlined in DICOM PS 3.15, for improved performance. It incorporates PaddleOCR, a robust Optical Character Recognition (OCR) system for extracting text from images, and RoBERTa, a fine-tuned transformer-based model for identifying and removing Personally Identifiable Information (PII) and Protected Health Information (PHI). Initially, the transformer-based model and the rule-based component were integrated to process for both structured data and free text. However, this coarse-grained approach did not yield optimal results. To improve performance, we refined our approach by applying the transformer model exclusively to free text, while structured data was handled only by rule-based methods. In this framework the DICOM validator dciodvfy was leveraged to ensure the integrity of DICOM files after the deID process. Through iterative refinement, including the incorporation of custom rules and private tag handling, the framework achieved a de-identification accuracy of 99.91% on the MIDI-B test dataset. The results demonstrate the effectiveness of combining rule-based compliance with AI-enabled adaptability in addressing the complex challenges of DICOM de-identification.

Figures

Figures reproduced from arXiv: 2509.00437 by the authors.

Figure 1
Figure 1. Core components of De-identification System. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Check Score Distribution to structured metadata fields common in DICOM files. These cases often require complementary rule-based logic to ensure accurate de-identification. The iterative refine￾ments applied throughout the challenge led to progressively improved results, underscoring the importance of a hybrid, adaptive approach. Our final model achieved an excep￾tionally high score of 99.91%, indicating near-perfec… view at source ↗
Figure 2
Figure 2. De-identification Failures by Category 5. Discussion The development of a robust de-identification framework for DICOM files presents unique challenges, particularly in balancing patient privacy with data utility. Our approach to the MIDI-B challenge highlights both the strengths and limitations of relying on transformer-based models for PHI detection. While AI-based methods such as RoBERTa demonstrate strong perfor… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

19 extracted references · 17 canonical work pages

  1. [1]

    URL https://huggingface.co/obi/deid_bert_i2b2

    obi/deid \_ bert \_ i2b2, 2019. URL https://huggingface.co/obi/deid_bert_i2b2. Accessed on 15 October, 2024

  2. [2]

    Murphy, Willie Boag, Wei - Hung Weng, Di Jin, Tristan Naumann, and Matthew B

    Emily Alsentzer, John R. Murphy, Willie Boag, Wei - Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. Publicly available clinical BERT embeddings. CoRR, abs/1904.03323, 2019. URL http://arxiv.org/abs/1904.03323

  3. [3]

    Free dicom de-identification tools in clinical research: functioning and safety of patient privacy, 2015

    KYE Aryanto, M Oudkerk, and PMA Van Ooijen. Free dicom de-identification tools in clinical research: functioning and safety of patient privacy, 2015

  4. [4]

    David A. Clunie. Dicom validator- dciodvfy, 2021. URL https://dclunie.com/dicom3tools/dciodvfy.html. Accessed on 15 October, 2024

  5. [5]

    The role of artificial intelligence and the internet of things in the development of medical radiology (an experimental study on magnetic resonance imaging)

    Mohammad Khalaf Daoud and Mohammed Otair. The role of artificial intelligence and the internet of things in the development of medical radiology (an experimental study on magnetic resonance imaging). In 2020 International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI), pages 17--20, 2020. doi:10.1109/ICHCI51889.2020.00011

  6. [6]

    Meddisc: A medical de-identification, segmentation & curation tool, 2024

    Apostolos Dimoulakis, Evangelos Politis, Michail Nikolaidis, Paschalis Bizopoulos, and Konstantinos Votis. Meddisc: A medical de-identification, segmentation & curation tool, 2024

  7. [7]

    Labeled imaging anatomy cases

    Craig Hacking, Campos A, and Knipe H et al. Labeled imaging anatomy cases. Labeled imaging anatomy cases radiopaedia, 2018. URL https://radiopaedia.org/articles/62414

  8. [8]

    Kitwaremedical/dicom-anonymizer: Tool to anonymize dicom files according to the dicom standard, 2023

    Edern Haumont. Kitwaremedical/dicom-anonymizer: Tool to anonymize dicom files according to the dicom standard, 2023. URL https://github.com/KitwareMedical/dicom-anonymizer. Accessed on 15 October, 2024

Show all 19 references
  1. [9]

    Anonymizing radiographs using an object detection deep learning algorithm, 2023

    Bardia Khosravi, John P Mickley, Pouria Rouzrokh, Michael J Taunton, A Noelle Larson, Bradley J Erickson, and Cody C Wyles. Anonymizing radiographs using an object detection deep learning algorithm, 2023

  2. [10]

    De-identification of clinical free text using natural language processing: A systematic review of current approaches, 2024

    Aleksandar Kova c evi \'c , Bojana Ba s aragin, Nikola Milo s evi \'c , and Goran Nenadi \'c . De-identification of clinical free text using natural language processing: A systematic review of current approaches, 2024

  3. [11]

    RoBERTa : A Robustly Optimized BERT Pretraining Approach , 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa : A Robustly Optimized BERT Pretraining Approach , 2019. URL https://arxiv.org/abs/1907.11692. \_eprint: 1907.11692

  4. [12]

    Nci workshop addresses current challenges in de-identifying medical images, 2023

    NCI. Nci workshop addresses current challenges in de-identifying medical images, 2023. URL https://datascience.cancer.gov/news-events/blog/nci-workshop-addresses-current-challenges-de-identifying-medical-images. Accessed on 15 October, 2024

  5. [13]

    E attribute confidentiality profiles, 2023

    NEMA. E attribute confidentiality profiles, 2023. URL https://dicom.nema.org/dicom/2013/output/chtml/part15/chapter_E.html#table_E.1-1. Accessed on 15 October, 2024

  6. [14]

    Paddleocr/docs/index.en.md at main · paddlepaddle, 2023

    PaddleOCR. Paddleocr/docs/index.en.md at main · paddlepaddle, 2023. URL https://github.com/PaddlePaddle/PaddleOCR/blob/main/docs/index.en.md. Accessed on 15 October, 2024

  7. [15]

    A dicom dataset for evaluation of medical image de-identification

    Michael Rutherford, Seong K Mun, Betty Levine, William Bennett, Kirk Smith, Phil Farmer, Quasar Jarosz, Ulrike Wagner, John Freyman, Geri Blake, et al. A dicom dataset for evaluation of medical image de-identification. Scientific Data, 8 0 (1): 0 183, 2021

  8. [16]

    De-identification in medical imaging, 2022

    Florent Saint-Clair. De-identification in medical imaging, 2022. URL https://dcmsys.com/project/de-identification-in-medical-imaging/. Accessed on 15 October, 2024

  9. [17]

    A two-stage de-identification process for privacy-preserving medical image analysis, 2022

    Arsalan Shahid, Mehran H Bazargani, Paul Banahan, Brian Mac Namee, Tahar Kechadi, Ceara Treacy, Gilbert Regan, and Peter MacMahon. A two-stage de-identification process for privacy-preserving medical image analysis, 2022

  10. [18]

    Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus

    Amber Stubbs and \"O zlem Uzuner. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus. Journal of biomedical informatics, 58: 0 S20--S29, 2015

  11. [19]

    Submission and de-identification overview- cancer imaging archive wiki, 2024

    TCIA. Submission and de-identification overview- cancer imaging archive wiki, 2024. URL https://wiki.cancerimagingarchive.net/display/Public/Submission+and+De-identification+Overview. Accessed on 15 October, 2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.