REVIEW 3 major objections 3 minor 32 references
RadHarmony claims that a single harmonizer layer can standardize 24 public radiology datasets into one tabular format, making multi-dataset deep-learning training as simple as single-dataset training.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 04:49 UTC pith:BBQE5YEI
load-bearing objection Useful data-layer library with an honest case study, but the 24-dataset support claim is only demonstrated for three and the rest rest on unverified agent-generated code. the 3 major comments →
RadHarmony: Radiological Data Handling in the Era of Agentic AI
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On its own terms, the paper's central discovery is that the heterogeneity of radiological datasets — different file formats, directory layouts, label schemas, and annotation types — can be absorbed by a harmonizer layer that outputs a fixed tabular schema (patient id, study id, image path, plus optional labels, masks, boxes, reports). The case study shows that three datasets spanning JPEG, DICOM, and PNG formats can be loaded, harmonized, and concatenated into a 473,719-image pretraining set through three constructor calls and a concatenation; linear-probe evaluation on a held-out dataset shows the multi-dataset model matches a CheXpert-only model at full label budget, which the authors pres
What carries the argument
The harmonizer layer: dataset-specific code that reads each raw metadata CSV and produces a standardized table with patient and study identifiers, a resolvable image path, and optional columns for view position, segmentation mask path, bounding boxes, and binary finding labels. This table is the interface between raw data and the deep-learning data loader; it is what makes the three-constructor multi-dataset training and the AI-agent integration workflow possible.
Load-bearing premise
The library's utility rests on each of the 24 harmonizers mapping its dataset's labels, image paths, and annotations correctly to the unified schema, while the case study only verifies three of them.
What would settle it
Run the library's built-in integrity checks (verify every image path resolves and every sample decodes) on all 24 datasets and compare the harmonized output against the official dataset statistics — for example, the number of images and the prevalence of each finding label should match published numbers. A single dataset whose harmonized labels diverge from the official semantics would invalidate the 24-dataset claim.
If this is right
- Multi-dataset pretraining and cross-dataset evaluation become a matter of listing dataset constructors rather than writing per-dataset glue code.
- An AI-agent skill guided by a declarative JSON configuration can integrate a new dataset end-to-end, as shown by the RSNA Pneumonia dataset integrated by a physician with no prior framework familiarity.
- The same unified interface exposes classification labels, segmentation masks, bounding boxes, and report text, so one dataset class can serve multiple task types.
- Patient-level splitting and k-fold cross-validation are built in, reducing leakage risk in multi-dataset experiments.
- A held-out linear-probe benchmark shows that adding two extra datasets to CheXpert yields comparable performance, establishing a cheap way to run rigorous null experiments.
Where Pith is reading between the lines
- If the harmonizers are correct, the library effectively turns 24 public datasets into a single reusable benchmark substrate, which could accelerate reproducibility and cross-dataset comparisons across labs.
- The harmonized schema could become a de facto interchange format; a natural test is whether third-party datasets can be integrated purely from a JSON config without touching library source.
- A testable extension: verify all 24 harmonizers by checking that harmonized label distributions and image counts match published dataset statistics; the paper only exercises three datasets in the case study.
- The AI-agent workflow's reliance on human review means its quality ceiling depends on reviewer vigilance; a possible extension is automated differential checks against known dataset statistics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces RadHarmony, an open-source Python library for harmonizing radiological datasets. It claims to standardize metadata from 24 public datasets into a single tabular schema, provide a unified PyTorch/MONAI dataset interface with patient-level splits and caching, and include an AI-agent skill for semi-automated dataset integration. The authors demonstrate the framework by pretraining RadHarmony-ViT on CheXpert, MIMIC-CXR, and ChestX-ray14 via a single constructor API, plus a CheXpert-only ablation and a 512-resolution variant, evaluating linear probes on the external VinDr-CXR dataset.
Significance. If the 24-dataset harmonization is correct, RadHarmony would fill a real gap in medical imaging tooling and make multi-dataset experiments substantially easier. The paper's strengths include public code and model weights, a transparently framed capability demonstration rather than a performance claim, per-fold standard deviations and paired t-tests, honest reporting of the null diversity result, and an external held-out benchmark. However, the breadth claim is disproportionate to the evidence: only three datasets are substantively validated, and the resolution-scaling result is confounded. With re-scoping and added semantic validation, the tool could be a useful community contribution.
major comments (3)
- [Section 3.6, Table 3] The central claim of 24-dataset harmonization is not supported by the evidence. Only three datasets (CheXpert, MIMIC-CXR, ChestX-ray14) are exercised in the case study; VinDr-CXR is used only for evaluation. Section 3.6 states that the remaining 21 datasets were integrated through human-agent collaboration, with verification limited to verify_images() (file resolution) and check_dataset() (decode/shape checks). These checks cannot detect silent semantic errors such as incorrect label-column mapping, wrong join keys, inverted label conventions, or misparsing of MD.ai JSON exports (Appendix A). A harmonizer with such an error passes CI while producing wrong metadata. The abstract and Table 3 present 24-dataset support as current; please add per-dataset semantic validation or re-scope the claim.
- [Section 3.8, Section 5] The resolution-scaling result is confounded. Full(512) is obtained by continuing pretraining of Full(256) for 5 additional epochs, so the gains in Table 7 mix increased resolution with additional optimization steps (~9k steps at batch 256). A clean resolution ablation would continue at 256 for the same steps or train from scratch at 512. As reported, the 'resolution effect' is not identifiable.
- [Section 3.6] The AI-agent skill is a headline contribution, but the only usability evidence is a single integration of RSNA Pneumonia by one physician user. No metrics such as success rate, number of agent iterations, or human corrections are reported. The Limitations appropriately call the workflow assistive, but the abstract's claim that the skill 'guides the full integration workflow' is not quantitatively supported. Please either provide an evaluation or temper the claim.
minor comments (3)
- [Section 3.2 / Code 2] The VinDr-CXR evaluation concatenates the official train and test splits into an 18k pool for 5-fold CV. The paper should state this explicitly and justify using the official test labels for training in some folds.
- [Table 7] The paired t-tests with only 5 folds have very low power; the p-values should be interpreted cautiously, and multiple testing across metrics and label budgets should be acknowledged.
- [Tables 5, 6] Macro SDs are computed across only 5 folds; consider reporting per-fold results or bootstrap confidence intervals for the macro averages.
Circularity Check
No significant circularity; the case study is an external-benchmark demonstration, with an unverified 24-dataset coverage gap that is a correctness risk rather than a circular step.
full rationale
RadHarmony makes implementation claims (a harmonizer layer, dataset/transform layers, an AI-agent skill) and supports them with a case study. I walked the claimed derivation chain and found no step that reduces to its own inputs. (i) No self-definitional relation: the harmonized schema in Table 2 is an output contract; each dataset's harmonizer encodes external domain knowledge, and no quantity is defined in terms of the outcome it is used to demonstrate. (ii) No fitted-input-called-prediction: the VinDr-CXR linear-probe results are external evaluations, and the paper explicitly calls the CheXpert-only comparison a 'capability demonstration rather than a performance claim.' (iii) The only author-overlapping citation is [9], used for 'DICOM VOI correction' in the beta preprocessor layer; the 24-dataset claim and the RadHarmony-ViT results do not rest on it, so it is not load-bearing. (iv) LeJEPA and NaFlex are externally cited methods and VinDr-CXR is an external held-out benchmark; no ansatz is smuggled in via a self-citation and no known result is merely renamed. The real weakness is evidentiary, not circular: Section 3.6 says 21 of 24 datasets were integrated 'through human-agent collaboration' and verified by verify_images() and check_dataset(), which confirm path resolution and decode/shape consistency, not semantic label correctness; the Limitations section concedes 'end-to-end correctness relies on the reviewing user, and broader qualitative evaluation ... remains future work.' That is an unverified-coverage/correctness risk, not a reduction of the central claim to its inputs. Score 1 reflects the mild self-referential character of demonstrating one's own workflow with one's own implementation; no circular step was exhibited.
Axiom & Free-Parameter Ledger
axioms (4)
- domain assumption Dataset metadata and label semantics are correctly preserved by the harmonizers
- domain assumption The AI coding agent generates correct code with human verification; one successful physician integration is representative
- domain assumption MONAI, LeJEPA, NaFlex, and Gradio function as documented
- domain assumption VinDr-CXR radiologist labels provide a valid held-out benchmark for representation quality
read the original abstract
Training deep learning models on radiological images requires integrating heterogeneous datasets across different sources, file formats, directory layouts, label schemas, and annotation types. We present RadHarmony, an open-source Python library that provides a unified API for loading, harmonizing, and augmenting radiological datasets, with a primary focus on chest radiographs and early support for computed tomography (CT) and magnetic resonance imaging (MRI). RadHarmony standardizes metadata from 24 public datasets into a single tabular format, wraps MONAI's map-style datasets for deep-learning-ready sample delivery with optional on-disk caching, and supports classification labels, segmentation masks, bounding boxes, and radiology report text through a single interface, with an interactive visualization tool for dataset exploration and verification. To lower the barrier for integrating new datasets, RadHarmony introduces an AI-agent skill that guides the full integration workflow from raw data inspection through code generation and testing. We demonstrate the library's utility by pretraining RadHarmony-ViT, a reference vision transformer baseline that combines three heterogeneous chest radiograph datasets with no dataset-specific code. The code and pretrained model weights are available at https://github.com/f10409/RadHarmony.
Reference graph
Works this paper leans on
-
[1]
Proceedings of the AAAI Conference on Artificial Intelligence (2019)
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., Seekins, J., Mong, D.A., Halabi, S.S., Sandberg, J.K., Jones, R., Larson, D.B., Langlotz, C.P., Patel, B.N., Lungren, M.P., Ng, A.Y.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedin...
2019
-
[2]
Scientific Data6 (2019) https://doi.org/10.1038/s41597-019-0322-0
Johnson, A.E.W., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.-y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data6 (2019) https://doi.org/10.1038/s41597-019-0322-0
-
[3]
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx- ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) https://doi.org/10.1109/CVPR.2017.369
-
[4]
Nature Biomedical Engineering (2025) https://doi.org/10
Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N., Doga, I., Durugol, O.F., Dai, W., Xu, M., Dasdelen, M.F., Wittmann, B., Ami- ranashvili, T., Simsar, E., Simsar, M., Erdemir, E.B., Alanbay, A., Sekuboyina, A., Lafci, B., Bluethgen, C., Batmanghelich, K., Ozdemir, M.K., Menze, B.: Developing generalist foundation models from a multi...
2025
-
[5]
Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murrey, B., Myronenko, A., Zhao, C., Yang, D., Nath, V., He, Y., Xu, Z., Hatamizadeh, A., Zhu, W., Liu, Y., Zheng, M., Tang, Y., Yang, I., Zephyr, M., Hashemian, B., Alle, S., Darestani, M.Z., Budd, C., Modat, M., Vercauteren, T., Wang, G., Li, Y., Hu, Y., Fu, Y., Gorman, B., Johnson, H., Ge...
2022
-
[6]
Computer Methods and Programs in Biomedicine208 (2021) https://doi.org/10.1016/j.cmpb.2021.106236
P´ erez-Garc ´ ıa, F., Sparks, R., Ourselin, S.: Torchio: A python library for effi- cient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning. Computer Methods and Programs in Biomedicine208 (2021) https://doi.org/10.1016/j.cmpb.2021.106236
arXiv 2021
-
[7]
Scientific Data10(2023) https://doi.org/10.1038/s41597-022-01721-8
Yang, J., Shi, R., Wei, D., Liu, Z., Zhao, L., Ke, B., Pfister, H., Ni, B.: Medmnist v2 – a large-scale lightweight benchmark for 2d and 3d biomedical image classi- fication. Scientific Data10(2023) https://doi.org/10.1038/s41597-022-01721-8
-
[8]
Proceedings of Machine Learning Research172, 231–249 (2022)
Cohen, J.P., Viviano, J.D., Bertin, P., Morrison, P., Torabian, P., Guarrera, M., Lungren, M.P., Chaudhari, A., Brooks, R., Hashir, M., Bertrand, H.: Torchxrayvi- sion: A library of chest x-ray datasets and models. Proceedings of Machine Learning Research172, 231–249 (2022)
2022
-
[9]
Journal of Imaging Informatics in Medicine38, 3040–3048 (2025) https://doi
Dapamede, T., Li, F., Khosravi, B., Purkayastha, S., Trivedi, H., Gichoya, J.: Dicom lut is a key step in medical image preprocessing towards ai generalizability. Journal of Imaging Informatics in Medicine38, 3040–3048 (2025) https://doi. org/10.1007/s10278-025-01418-5
-
[10]
ICML 2019 Workshop on Human in the Loop Learning (2019)
Abid, A., Abdalla, A., Abid, A., Khan, D., Alfozan, A., Zou, J.: Gradio: Hassle- free sharing and testing of ml models in the wild. ICML 2019 Workshop on Human in the Loop Learning (2019)
2019
-
[11]
Radiology: Artificial Intelligence1(2019) https://doi.org/10.1148/ryai.2019180041
Shih, G., Wu, C.C., Halabi, S.S., Kohli, M.D., Prevedello, L.M., Cook, T.S., Sharma, A., Amorosa, J.K., Arteaga, V., Galperin-Aizenberg, M., Gill, R.R., Godoy, M.C.B., Hobbs, S., Jeudy, J., Laroia, A., Shah, P.N., Vummidi, D., Yaddanapudi, K., Stein, A.: Augmenting the national institutes of health chest radiograph dataset with expert annotations of possi...
-
[12]
arXiv preprint arXiv:2511.08544 (2025)
Balestriero, R., LeCun, Y.: Lejepa: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544 (2025)
Pith/arXiv arXiv 2025
-
[13]
arXiv preprint arXiv:2502.14786 (2025)
Tschannen, M., Gritsenko, A., Wang, X., Naeem, M.F., Alabdulmohsin, I., Parthasarathy, N., Evans, T., Beyer, L., Xia, Y., Mustafa, B., H´ enaff, O., Harm- sen, J., Steiner, A., Zhai, X.: Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786 (2025)
Pith/arXiv arXiv 2025
-
[14]
Scientific Data9 20 (2022) https://doi.org/10.1038/s41597-022-01498-w
Nguyen, H.Q., Lam, K., Le, L.T., Pham, H.H., Tran, D.Q., Nguyen, D.B., Le, D.D., Pham, C.M., Tong, H.T.T., Dinh, D.H., Do, C.D., Doan, L.T., Nguyen, C.N., Nguyen, B.T., Nguyen, Q.V., Hoang, A.D., Phan, H.N., Nguyen, A.T., Ho, P.H., Ngo, D.T., Nguyen, N.T., Nguyen, N.T., Dao, M., Vu, V.: Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotat...
-
[15]
Nature Machine Intelligence7, 119–130 (2025) https://doi.org/10.1038/s42256-024-00965-w
P´ erez-Garc ´ ıa, F., Sharma, H., Bond-Taylor, S., Bouzid, K., Salvatelli, V., Ilse, M., Bannur, S., Castro, D.C., Schwaighofer, A., Lungren, M.P., Wetscherek, M.T., Codella, N., Hyland, S.L., Alvarez-Valle, J., Oktay, O.: Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence7, 119–130 (2025) https://doi.org/10.10...
-
[16]
arXiv preprint arXiv:2405.19538 (2024)
Chambon, P., Delbrouck, J.-B., Sounack, T., Huang, S.-C., Chen, Z., Varma, M., Truong, S.Q.H., Chuong, C.T., Langlotz, C.P.: Chexpert plus: Augmenting a large chest x-ray dataset with text radiology reports, patient demographics and additional image formats. arXiv preprint arXiv:2405.19538 (2024)
Pith/arXiv arXiv 2024
-
[17]
arXiv preprint arXiv:1901.07042 (2019)
Johnson, A.E.W., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., Deng, C.- y., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., Horng, S.: Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019)
Pith/arXiv arXiv 1901
-
[18]
Medical Image Analysis66, 101797 (2020) https://doi.org/10.1016/j.media.2020.101797
Bustos, A., Pertusa, A., Salinas, J.-M., Iglesia-Vaya, M.D.L.: Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical Image Analysis66, 101797 (2020) https://doi.org/10.1016/j.media.2020.101797
arXiv 2020
-
[19]
Scientific Data9(1), 487 (2022) https:// doi.org/10.1038/s41597-022-01608-8
Reis, E.P., Paiva, J.P.D., Silva, M.C.D., Ribeiro, G.A., Paiva, V.F., Bulgarelli, L., Lee, H.M., Santos, P.V., Brito, V.M., Amaral, L.T., Beraldo, G.L.: Brax, Brazilian labeled chest x-ray dataset. Scientific Data9(1), 487 (2022) https:// doi.org/10.1038/s41597-022-01608-8
-
[20]
arXiv preprint arXiv:2505.00228 (2025)
Zhang, X., Acosta, J.N., Miller, J., Huang, O., Rajpurkar, P.: ReXGradient-160K: A large-scale publicly available dataset of chest radiographs with free-text reports. arXiv preprint arXiv:2505.00228 (2025)
Pith/arXiv arXiv 2025
-
[21]
Kaggle (2019)
Society for Imaging Informatics in Medicine: SIIM-ACR Pneu- mothorax Segmentation. Kaggle (2019). https://www.kaggle.com/c/ siim-acr-pneumothorax-segmentation
2019
-
[22]
Kaggle (2021)
Kemp, A., Zawacki, A., Carr, C., Shih, G., Mongan, J., Elliott, J., Kaiwen, Lakhani, P., Culliton, P.: SIIM-FISABIO-RSNA COVID-19 Detection. Kaggle (2021)
2021
-
[23]
Kaggle (2020)
Seah, J., Jen, Maggie, Law, M., Culliton, P., Dowd, S.: RANZCR CLiP – Catheter and Line Position Challenge. Kaggle (2020)
2020
-
[24]
Scientific Data13(1), 632 (2026) https://doi.org/10.1038/s41597-026-07271-7 21
Truhn, D., Geiger, D., Siepmann, R., St¨ uck, M.S., Bressem, K.K., Kather, J.N., Kuhl, C., M¨ uller-Franzes, G., Nebelung, S.: A comprehensive bedside chest radio- graphy dataset with structured, itemized and graded radiologic reports. Scientific Data13(1), 632 (2026) https://doi.org/10.1038/s41597-026-07271-7 21
-
[25]
Demner-Fushman, D., Kohli, M.D., Rosenman, M.B., Shooshan, S.E., Rodriguez, L., Antani, S., Thoma, G.R., McDonald, C.J.: Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association23(2), 304–310 (2016) https://doi.org/10.1093/jamia/ ocv080
doi:10.1093/jamia/ 2016
-
[26]
Quantitative Imaging in Medicine and Surgery4(6), 475–477 (2014) https://doi
Jaeger, S., Candemir, S., Antani, S., W´ ang, Y.-X.J., Lu, P.-X., Thoma, G.: Two public chest X-ray datasets for computer-aided screening of pulmonary diseases. Quantitative Imaging in Medicine and Surgery4(6), 475–477 (2014) https://doi. org/10.3978/j.issn.2223-4292.2014.11.20
arXiv 2014
-
[27]
Radiology290(2), 498–503 (2019) https://doi.org/10.1148/radiol.2018180736
Halabi, S.S., Prevedello, L.M., Kalpathy-Cramer, J., Mamonov, A.B., Bilbily, A., Cicero, M., Pan, I., Pereira, L.A., Sousa, R.T., Abdala, N., Kitamura, F.C., Thodberg, H.H., Chen, L., Shih, G., Andriole, K., Kohli, M.D., Erickson, B.J., Flanders, A.E.: The RSNA pediatric bone age machine learning challenge. Radiology290(2), 498–503 (2019) https://doi.org/...
-
[28]
Medical Image Analysis67(2021) https://doi.org/10.1016/j.media.2020.101857
Draelos, R.L., Dov, D., Mazurowski, M.A., Lo, J.Y., Henao, R., Rubin, G.D., Carin, L.: Machine-learning-based multiple abnormality prediction with large- scale chest computed tomography volumes. Medical Image Analysis67(2021) https://doi.org/10.1016/j.media.2020.101857
arXiv 2021
-
[29]
Radiology: Artificial Intelligence3(2), 200254 (2021) https://doi.org/10.1148/ryai.2021200254
Colak, E., Kitamura, F.C., Hobbs, S.B., Wu, C.C., Lungren, M.P., Prevedello, L.M., Kalpathy-Cramer, J., Ball, R.L., Shih, G., Stein, A., Halabi, S.S.: The RSNA pulmonary embolism CT dataset. Radiology: Artificial Intelligence3(2), 200254 (2021) https://doi.org/10.1148/ryai.2021200254
-
[30]
Kaggle (2022)
Flanders, A., Carr, C., Colak, E., Kitamura, F., Lin, H.M., Rudie, J., Mongan, J., Andriole, K., Prevedello, L., Riopel, M., Ball, R., Dane, S.: RSNA 2022 Cervical Spine Fracture Detection. Kaggle (2022)
2022
-
[31]
Radiology: Artificial Intelligence6(6), 240101 (2024) https://doi.org/10.1148/ryai.240101
Rudie, J.D., Lin, H.M., Ball, R.L., Jalal, S., Prevedello, L.M., Nicolaou, S., Marinelli, B.S., Flanders, A.E., Magudia, K., Shih, G., Davis, M.A.: The RSNA abdominal traumatic injury CT (RATIC) dataset. Radiology: Artificial Intelligence6(6), 240101 (2024) https://doi.org/10.1148/ryai.240101
-
[32]
Kaggle (2024) 22
Richards, T., Talbott, J., Ball, R., Colak, E., Flanders, A., Kitamura, F., Mon- gan, J., Prevedello, L., Vazirabad, M.: RSNA 2024 Lumbar Spine Degenerative Classification. Kaggle (2024) 22
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.