REVIEW 4 major objections 4 minor 19 references
A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge
T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A hybrid rule-plus-AI pipeline de-identifies DICOM files to 99.91% accuracy on the MIDI-B benchmark.
desk verdict Solid, honest challenge write-up with open code and a strong benchmark number; just don't mistake the per-tag score for a PHI-safety rate, and the discrepancy stats need a second pass. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a division of labor enforced across four components: DCMTCIADeidentifier, a rule-based de-identifier following TCIA's Safe Harbor method with 40 custom rules; PrivateTagsDeidentifier, which maps each private tag to a processing rule using an 8,788-entry vendor dictionary; DCMImageDeidentifier, which uses PaddleOCR to find burned-in text and masks detected PHI boxes; and DcmValidator, which runs dciodvfy and fills missing attributes. The RoBERTa PHI detector is deliberately applied only to free-text fields and OCR output, not to structured tags.
What would settle it
Manually audit the 8,676 test-set mismatches and count how many of the 0.0-check-score tags contain genuine PHI (real patient names, dates, or IDs) rather than benign metadata differences; if a sizable fraction are real identifiers, the near-perfect benchmark score overstates privacy safety.
Extended reading notes
Core claim
The paper's central claim is that a de-identification framework combining a DICOM PS 3.15/TCIA-Safe-Harbor rule-based core, a RoBERTa-based PHI detector restricted to free text, PaddleOCR for burned-in image text, a curated private-tag dictionary, and the dciodvfy validator achieves near-perfect de-identification on the MIDI-B test set (99.91%). The iterative validation trajectory is the evidence: swapping the initial PS 3.15 rules for TCIA Safe Harbor rules raised accuracy from 84.36% to 94.71%; restricting AI to free text and adding 40 custom rules, then private-tag handling, then validation, pushed the score to 99.98% on validation and 99.91% on the unseen test set. The paper also reports
Load-bearing premise
The 99.91% score assumes the MIDI-B automated report really measures privacy safety, and that an NLP model trained on clinic notes works on DICOM text and scanned image text without retraining.
Editorial extensions
If this is right
- Using the same rule-based core for structured data and reserving AI for free text outperforms applying AI to all fields, so future DICOM de-identification systems can start from this split rather than a monolithic model.
- Private tags are a major source of error: adding the vendor dictionary was one of the largest single accuracy gains, so any deployment must inventory private tags before claiming safety.
- Post-de-identification validation is not optional: the dciodvfy step caught missing attributes and contributed the final accuracy jump, and it keeps files usable after PHI removal.
- On the test set, most remaining errors are missed PHI and private-tag rule gaps; improving those two areas should be the next target rather than tuning the NLP model's architecture.
- The reported 99.91% is specific to the MIDI-B scoring definition; on this benchmark it implies near-total removal of expected identifiers, but the paper itself cautions against assuming the same rate on other institutional data.
Reading between the lines
- A testable extension outside the paper: run the same pipeline on a multi-institution corpus after stripping the private-tag dictionary; if accuracy drops sharply, the private-tag coverage is vendor-specific and the headline score overstates generality.
- The check-score analysis suggests many mismatches are partial text overlaps rather than complete leaks, so a manual audit of the 0.0-score tags would tell whether the remaining errors are genuine PHI or benign formatting differences.
- Because the PHI detector was trained on clinical narratives, fine-tuning it on DICOM free text and OCR output—flagged as future work in the paper—is likely to reduce the 59.3% PHI-detection failure share more than any rule change.
- If benchmark accuracy is used as a procurement criterion, institutions should weight private-tag coverage and a manual spot-check, since the automated score rewards exact expected output rather than privacy semantics alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes a hybrid DICOM de-identification framework developed for the MIDI-B challenge. The system combines a rule-based DICOM metadata de-identifier (TCIA Safe Harbor / DICOM PS 3.15 based) with PaddleOCR for burned-in image text and a RoBERTa model fine-tuned on i2b2 2014 for PHI detection in free text. The authors report iterative improvements on the validation set and a final test accuracy of 99.91% on MIDI-B. They also describe a private-tag de-identifier, a DICOM validator integration, a 150-term imaging whitelist to reduce false positives, and a discrepancy analysis of test failures. The code is publicly available on GitHub.
Significance. If the reported 99.91% MIDI-B test score is taken at face value, the work demonstrates a competitive, openly available hybrid de-identification pipeline for DICOM files, with useful engineering details such as TCIA custom rules, private-tag handling, and validator-based repair. The main strengths are the external challenge evaluation, the public implementation, and the explicit breakdown of failure categories. However, the paper's central interpretation of the score as 'near-perfect de-identification' is not supported without a precise definition of the evaluation metric and a separate PHI-level recall measure. The authors' own failure analysis reports 5,145 mismatches attributed to missed PHI detection, which indicates that a high per-tag score can coexist with many potentially sensitive values remaining in the output. The paper is therefore useful as a systems description but needs substantial revision to make its claims about de-identification safety commensurate with the evidence.
major comments (4)
- [Section 4, Table 2 and Figure 3] The headline 99.91% is called 'accuracy', but the underlying MIDI-B metric (check_score) is never defined. The reader cannot tell whether the score is computed per DICOM tag, per file, or per value, or how partial credit is awarded. The presence of partial scores in Figure 3 (e.g., 0.7–0.8) shows that the metric is not a simple binary PHI detection rate. The Discussion (Section 5) equates this score with 'near-perfect de-identification', which overstates what the metric establishes. Please define the metric explicitly, report confidence intervals or at least totals, and provide entity-level precision/recall for PHI, not just an aggregate tag score.
- [Section 4, discrepancy analysis / Figure 3] The text states that there are 8,676 mismatches from the test data, but the check-score distribution in Figure 3 sums to 5,144 (2,528 + 582 + 204 + 305 + 1,130 + 395). The percentage 49.1% is computed as 2,528/5,144, yet the text presents this as a fraction of the mismatched tags, implying 2,528/8,676 ≈ 29.1%. This internal inconsistency makes the failure analysis uninterpretable. If Figure 3 covers only a subset of mismatches, state that explicitly; if it is meant to cover all mismatches, correct the counts.
- [Section 4, Figure 2] The failure category 'limitations in PHI detection' accounts for 5,145 cases, described as 'missed identification of sensitive information'. If these correspond to actual PHI values that remain in the de-identified output, then the framework has leaked thousands of protected values even though the benchmark score is 99.91%. This directly contradicts the 'near-perfect de-identification' claim. The paper should either provide a separate PHI-level analysis showing how many of those 5,145 mismatches were true PHI, or substantially weaken the claim in Section 5. As written, the paper's own failure breakdown undermines the central safety claim.
- [Section 3.3, whitelist] The 150-term imaging whitelist excludes detections overlapping curated anatomy/case terms to reduce RoBERTa false positives. No analysis is provided to show that this whitelist never removes a true patient identifier (e.g., a surname that coincides with an anatomy term or a laterality abbreviation). Since this mechanism directly affects de-identification safety, a false-negative analysis or a more conservative, context-aware filtering strategy should be described and justified.
minor comments (4)
- [Table 1] Typo in the second row: 'DCMTICADeIdentifier' should read 'DCMTCIADeIdentifier'. The footnote about 'self-derived method' is unclear and should be rewritten; it is not apparent how a validation score can be derived from the difference from a previous score.
- [Table 3] The two-column layout with the repeated header 'Tag Name Failed' is confusing. It is unclear whether the second column is a continuation or a separate set of tags. Please reformat the table to make the pairs explicit.
- [Section 3.2.2 / References] The text cites 'obi/deid_roberta_i2b2', while the reference list entry is titled 'obi/deid_bert_i2b2'. Please align the model name and reference.
- [Figure 3] The term 'check_score' is used without a formal definition in the text. Define the score and its range before presenting the distribution.
Circularity Check
No significant circularity: the central claim is an empirical score on an external benchmark, not a derivation from the paper's own inputs.
full rationale
The paper makes no mathematical derivation chain; its central claim (99.91% accuracy on the MIDI-B test dataset, Section 4, Table 2) is an empirical evaluation against an externally defined challenge benchmark. Iterative refinement on the validation set and subsequent testing on a held-out test set is standard machine-learning practice and does not constitute fitting the test result. The components (RoBERTa trained on i2b2, PaddleOCR, TCIA rules, dciodvfy validator) are all external resources, and the paper contains no load-bearing self-citations or imported uniqueness theorems. The Table 1 footnote states that the final validation-phase score of 99.98% was 'computed using a self-derived method,' but the test-phase score in Table 2 is reported from the challenge's validation script. The Discussion explicitly concedes that the score 'may not generalize,' and the paper's own discrepancy analysis shows 5,145 missed PHI detections, but these are correctness/overfitting concerns about the chosen metric, not circular reasoning. The evaluation is self-contained against an external benchmark, so the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- imaging_term_whitelist =
150 terms (CT, MR, T1, abdomen, ...)
- custom_rules_v2 =
40 rules
- date_shift_days =
120 (default)
assumptions (5)
- domain assumption The MIDI-B evaluation metric and discrepancy report are a valid measure of de-identification completeness.
- domain assumption The TCIA rule table and the 8,788-entry private tag dictionary comprehensively cover required DICOM de-identification behavior.
- domain assumption The i2b2-trained RoBERTa model transfers to DICOM free text and OCR output without DICOM-specific fine-tuning.
- domain assumption PaddleOCR detects all burned-in PHI text in medical images; text not detected by OCR is treated as absent.
- ad hoc to paper The 150-term whitelist correctly separates imaging anatomy terms from PHI without filtering real patient identifiers.
Cite this review
Pith. "Pith review of A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge." pith.science (2026). https://pith.science/paper/5DHG4ANS
@misc{pith2026250900437,
author = {Pith},
title = {Pith review of: A Hybrid AI-based and Rule-based Approach to DICOM De-identification: A Solution for the MIDI-B Challenge},
year = {2026},
howpublished = {\url{https://pith.science/paper/5DHG4ANS}},
note = {Machine review of arXiv:2509.00437}
}
read the original abstract
Ensuring the de-identification of medical imaging data is a critical step in enabling safe data sharing. This paper presents a hybrid de-identification framework designed to process Digital Imaging and Communications in Medicine (DICOM) files. Our framework adopts a modified, pre-built rule-based component, updated with The Cancer Imaging Archive (TCIA)'s best practices guidelines, as outlined in DICOM PS 3.15, for improved performance. It incorporates PaddleOCR, a robust Optical Character Recognition (OCR) system for extracting text from images, and RoBERTa, a fine-tuned transformer-based model for identifying and removing Personally Identifiable Information (PII) and Protected Health Information (PHI). Initially, the transformer-based model and the rule-based component were integrated to process for both structured data and free text. However, this coarse-grained approach did not yield optimal results. To improve performance, we refined our approach by applying the transformer model exclusively to free text, while structured data was handled only by rule-based methods. In this framework the DICOM validator dciodvfy was leveraged to ensure the integrity of DICOM files after the deID process. Through iterative refinement, including the incorporation of custom rules and private tag handling, the framework achieved a de-identification accuracy of 99.91% on the MIDI-B test dataset. The results demonstrate the effectiveness of combining rule-based compliance with AI-enabled adaptability in addressing the complex challenges of DICOM de-identification.
Figures
Reference graph
Works this paper leans on
-
[1]
URL https://huggingface.co/obi/deid_bert_i2b2
obi/deid \_ bert \_ i2b2, 2019. URL https://huggingface.co/obi/deid_bert_i2b2. Accessed on 15 October, 2024
work page 2019
-
[2]
Murphy, Willie Boag, Wei - Hung Weng, Di Jin, Tristan Naumann, and Matthew B
Emily Alsentzer, John R. Murphy, Willie Boag, Wei - Hung Weng, Di Jin, Tristan Naumann, and Matthew B. A. McDermott. Publicly available clinical BERT embeddings. CoRR, abs/1904.03323, 2019. URL http://arxiv.org/abs/1904.03323
arXiv 1904
-
[3]
KYE Aryanto, M Oudkerk, and PMA Van Ooijen. Free dicom de-identification tools in clinical research: functioning and safety of patient privacy, 2015
work page 2015
-
[4]
David A. Clunie. Dicom validator- dciodvfy, 2021. URL https://dclunie.com/dicom3tools/dciodvfy.html. Accessed on 15 October, 2024
work page 2021
-
[5]
Mohammad Khalaf Daoud and Mohammed Otair. The role of artificial intelligence and the internet of things in the development of medical radiology (an experimental study on magnetic resonance imaging). In 2020 International Conference on Intelligent Computing and Human-Computer Interaction (ICHCI), pages 17--20, 2020. doi:10.1109/ICHCI51889.2020.00011
-
[6]
Meddisc: A medical de-identification, segmentation & curation tool, 2024
Apostolos Dimoulakis, Evangelos Politis, Michail Nikolaidis, Paschalis Bizopoulos, and Konstantinos Votis. Meddisc: A medical de-identification, segmentation & curation tool, 2024
work page 2024
-
[7]
Craig Hacking, Campos A, and Knipe H et al. Labeled imaging anatomy cases. Labeled imaging anatomy cases radiopaedia, 2018. URL https://radiopaedia.org/articles/62414
work page 2018
-
[8]
Kitwaremedical/dicom-anonymizer: Tool to anonymize dicom files according to the dicom standard, 2023
Edern Haumont. Kitwaremedical/dicom-anonymizer: Tool to anonymize dicom files according to the dicom standard, 2023. URL https://github.com/KitwareMedical/dicom-anonymizer. Accessed on 15 October, 2024
work page 2023
Show all 19 references
-
[9]
Anonymizing radiographs using an object detection deep learning algorithm, 2023
Bardia Khosravi, John P Mickley, Pouria Rouzrokh, Michael J Taunton, A Noelle Larson, Bradley J Erickson, and Cody C Wyles. Anonymizing radiographs using an object detection deep learning algorithm, 2023
2023
-
[10]
De-identification of clinical free text using natural language processing: A systematic review of current approaches, 2024
Aleksandar Kova c evi \'c , Bojana Ba s aragin, Nikola Milo s evi \'c , and Goran Nenadi \'c . De-identification of clinical free text using natural language processing: A systematic review of current approaches, 2024
2024
-
[11]
RoBERTa : A Robustly Optimized BERT Pretraining Approach , 2019
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa : A Robustly Optimized BERT Pretraining Approach , 2019. URL https://arxiv.org/abs/1907.11692. \_eprint: 1907.11692
2019 arXiv
-
[12]
Nci workshop addresses current challenges in de-identifying medical images, 2023
NCI. Nci workshop addresses current challenges in de-identifying medical images, 2023. URL https://datascience.cancer.gov/news-events/blog/nci-workshop-addresses-current-challenges-de-identifying-medical-images. Accessed on 15 October, 2024
2023
-
[13]
E attribute confidentiality profiles, 2023
NEMA. E attribute confidentiality profiles, 2023. URL https://dicom.nema.org/dicom/2013/output/chtml/part15/chapter_E.html#table_E.1-1. Accessed on 15 October, 2024
2023
-
[14]
Paddleocr/docs/index.en.md at main · paddlepaddle, 2023
PaddleOCR. Paddleocr/docs/index.en.md at main · paddlepaddle, 2023. URL https://github.com/PaddlePaddle/PaddleOCR/blob/main/docs/index.en.md. Accessed on 15 October, 2024
2023
-
[15]
A dicom dataset for evaluation of medical image de-identification
Michael Rutherford, Seong K Mun, Betty Levine, William Bennett, Kirk Smith, Phil Farmer, Quasar Jarosz, Ulrike Wagner, John Freyman, Geri Blake, et al. A dicom dataset for evaluation of medical image de-identification. Scientific Data, 8 0 (1): 0 183, 2021
2021
-
[16]
De-identification in medical imaging, 2022
Florent Saint-Clair. De-identification in medical imaging, 2022. URL https://dcmsys.com/project/de-identification-in-medical-imaging/. Accessed on 15 October, 2024
2022
-
[17]
A two-stage de-identification process for privacy-preserving medical image analysis, 2022
Arsalan Shahid, Mehran H Bazargani, Paul Banahan, Brian Mac Namee, Tahar Kechadi, Ceara Treacy, Gilbert Regan, and Peter MacMahon. A two-stage de-identification process for privacy-preserving medical image analysis, 2022
2022
-
[18]
Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus
Amber Stubbs and \"O zlem Uzuner. Annotating longitudinal clinical narratives for de-identification: The 2014 i2b2/uthealth corpus. Journal of biomedical informatics, 58: 0 S20--S29, 2015
2014
-
[19]
Submission and de-identification overview- cancer imaging archive wiki, 2024
TCIA. Submission and de-identification overview- cancer imaging archive wiki, 2024. URL https://wiki.cancerimagingarchive.net/display/Public/Submission+and+De-identification+Overview. Accessed on 15 October, 2024
2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.