Pith. sign in

REVIEW 5 major objections 5 minor 20 references

PARROT: An Open Multilingual Radiology Reports Dataset

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PARROT assembles 2,658 fictional radiology reports in 13 languages and shows that even radiologists struggle to tell them from AI-generated reports.

desk verdict A genuinely useful open multilingual radiology text resource, but the 'largest' and representativeness claims are softer than the paper lets on. read the letter →

arxiv 2507.22939 v2 pith:ULG4XTAH submitted 2025-07-25 cs.CL cs.AI

Bastien Le Guellec , Kokou Adambounou , Lisa C Adams , Thibault Agripnidis , Sung Soo Ahn , Radhia Ait Chalal , Tugba Akinci D Antonoli , Philippe Amouyel
show 80 more authors
Henrik Andersson Raphael Bentegeac Claudio Benzoni Antonino Andrea Blandino Felix Busch Elif Can Riccardo Cau Armando Ugo Cavallo Christelle Chavihot Erwin Chiquete Renato Cuocolo Eugen Divjak Gordana Ivanac Barbara Dziadkowiec Macek Armel Elogne Salvatore Claudio Fanni Carlos Ferrarotti Claudia Fossataro Federica Fossataro Katarzyna Fulek Michal Fulek Pawel Gac Martyna Gachowska Ignacio Garcia Juarez Marco Gatti Natalia Gorelik Alexia Maria Goulianou Aghiles Hamroun Nicolas Herinirina Krzysztof Kraik Dominik Krupka Quentin Holay Felipe Kitamura Michail E Klontzas Anna Kompanowska Rafal Kompanowski Alexandre Lefevre Tristan Lemke Maximilian Lindholz Lukas Muller Piotr Macek Marcus Makowski Luigi Mannacio Aymen Meddeb Antonio Natale Beatrice Nguema Edzang Adriana Ojeda Yae Won Park Federica Piccione Andrea Ponsiglione Malgorzata Poreba Rafal Poreba Philipp Prucker Jean Pierre Pruvo Rosa Alba Pugliesi Feno Hasina Rabemanorintsoa Vasileios Rafailidis Katarzyna Resler Jan Rotkegel Luca Saba Ezann Siebert Arnaldo Stanzione Ali Fuat Tekin Liz Toapanta Yanchapaxi Matthaios Triantafyllou Ekaterini Tsaoulia Evangelia Vassalou Federica Vernuccio Johan Wasselius Weilang Wang Szymon Urban Adrian Wlodarczak Szymon Wlodarczak Andrzej Wysocki Lina Xu Tomasz Zatonski Shuhang Zhang Sebastian Ziegelmayer Gregory Kuchcinski Keno K Bressem
This is my paper · ORCID
classification cs.CLcs.AI
keywords radiologyreportsmultilingualdatasetnaturallanguageprocessinglargemodelsfictionalmedicaldatahuman-AIdifferentiationICD-10codingopenbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PARROT is an attempt to break the English-only bottleneck in radiology natural-language processing by building the largest openly available multilingual collection of radiology reports: 2,658 fictional but expert-written reports from 76 radiologists in 21 countries and 13 languages, released without patient-data restrictions. The paper argues that because the reports are written by working radiologists following their usual local formats and terminology, the dataset captures authentic cross-country variation in reporting style that synthetic English corpora and privacy-limited real records cannot. To show the reports are not trivially replaceable by AI output, the authors ran a discrimination study in which 154 participants—radiologists, other healthcare staff, and laypeople—could distinguish human PARROT reports from frontier-language-model-generated ones only 53.9% of the time overall, with radiologists at 56.9%. If the central claim holds, PARROT gives NLP researchers a legal, reusable benchmark for training and evaluating tools that must work across languages and healthcare systems.

What carries the argument

The carrying object is the PARROT corpus itself: a version-controlled, JSONL-formatted collection of fictional radiology reports contributed by 76 radiologists, each writing in the format, terminology, and structure they would use in their own clinical practice, with metadata and ICD-10 codes attached. The human-versus-AI differentiation study acts as the proof-of-concept: using a frontier large language model to generate comparison reports and asking 154 participants to classify them tests whether PARROT's expert-written text is distinguishable from AI text, supporting the paper's positioning of the dataset as a middle ground between synthetic corpora and privacy-restricted real records.

What would settle it

Compare distributions of report structure, terminology, and length between PARROT reports and a matched sample of real reports from the same institutions and languages; if real and fictional reports differ systematically, or if an NLP model trained on PARROT fails to match a model trained on real reports on a held-out real-language report understanding task, the central utility claim is contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that fictional reports, when authored by practising radiologists following their own local reporting conventions, can serve as a proxy for real clinical text for natural-language-processing benchmarking. The dataset itself is the discovery: 2,658 reports with metadata on modality, anatomy, clinical context, ICD-10 codes, and English translations for non-English entries, spanning CT, MRI, radiography, and ultrasound. The validation experiment shows that humans, including radiologists, have difficulty telling these reports from frontier-LLM output; this is presented as evidence that the PARROT texts carry authentic stylistic and reasoning patterns that pure synthetic generation misses, while remaining free of privacy constraints.

Load-bearing premise

The dataset presupposes that fictional reports written on request by volunteer radiologists, who were told to follow their usual style and to include plausible findings, are representative enough of real radiology reporting in those countries and languages for NLP models benchmarked on them to transfer to clinical practice.

Editorial extensions

If this is right

  • Researchers can develop and compare radiology NLP tools on one legal, open dataset spanning 13 languages, without negotiating data-sharing agreements or de-identification protocols.
  • Workflows such as report summarization, structured extraction, and ICD-10 coding can be tested on expert-written input in languages that previously had no public benchmark.
  • Because French and Spanish each come from multiple countries, PARROT enables studying whether reporting-style variation within a language changes model performance.
  • The near-chance human-versus-AI result implies that quality control for AI-generated radiology text should not rely on human detection alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If PARROT reports are representative, the differentiation result hints at a practical detector: radiologists outperform others by about seven points, so extracting the features they use could yield a machine classifier, although the paper does not attempt this.
  • A stronger test of the dataset's premise would be to collect real de-identified reports from the same contributing radiologists and measure the style gap; the present release does not include that comparison and could not detect it.
  • The dataset's open license and English translations make it a plausible starting point for cross-lingual training, but users should validate transfer to their own local clinical text before deployment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces PARROT, a dataset of 2,658 fictional radiology reports written by 76 radiologists from 21 countries in 13 languages, with metadata including imaging modality, anatomical region, clinical context, English translations for non-English reports, and ICD-10 code assignments. The authors also report a human-versus-AI discrimination study in which 154 participants (radiologists, other healthcare professionals, and non-healthcare professionals) judged whether individual reports were human-authored or generated by GPT-o1, achieving an overall accuracy of 53.9% with radiologists performing better (56.9%). The paper claims that PARROT is the largest openly available multilingual radiology-report dataset and that it enables development and validation of NLP applications across linguistic, geographic, and clinical boundaries without privacy constraints.

Significance. If the claims hold, PARROT would be a useful resource for multilingual NLP in radiology, addressing the English-centric bias of widely used datasets such as MIMIC-CXR. The fictional nature of the reports and the decision to release the data openly (under CC-BY-NC-SA) are practical strengths that avoid privacy barriers, and the human/AI discrimination study addresses a timely question about the detectability of AI-generated medical text. However, the manuscript overreaches in several respects: the 'largest' superlative is not backed by a systematic comparison with other open multilingual radiology-corpus resources, the descriptive statistics contain internal inconsistencies, the discrimination study is under-specified, and the central assumption that fictional reports are representative of real clinical reporting is not validated. The resource itself has potential, but the current presentation needs revision before the claims can be accepted.

major comments (5)
  1. [Abstract and §4.2] The claim that PARROT 'represents the largest openly available multilingual radiology-report dataset' is not supported by any systematic comparison with existing resources. Section 4 contrasts only MIMIC-IV and MIMIC-CXR, which are English-only, and does not enumerate or compare counts with other open multilingual radiology-report corpora. Without a comparative table listing existing datasets, their languages, report counts, and licensing, the superlative is unsubstantiated. Please either provide that comparison or temper the claim (e.g., 'largest to our knowledge') with supporting evidence.
  2. [§3.2 and Table 1] Table 1 lists body-area counts that sum to 3,397, whereas the dataset contains 2,658 reports. The text and abstract report chest (19.9%), abdomen (18.6%), head (17.3%), and pelvis (14.1%) as if these were proportions of reports, but these percentages are consistent with 677/3397, 631/3397, 588/3397, and 480/3397, respectively, indicating the percentages are calculated over body-area labels rather than reports. If a report can be assigned multiple body areas, this must be stated explicitly and report-level versus label-level statistics should be presented separately; if each report has exactly one body area, the table sums are erroneous. As written, this is an internal inconsistency in the core descriptive results.
  3. [§3.1] The sentence 'Nations from the Global South—Argentina (71; 2.7 %), China (100; 3.8 %), and Mexico (75; 2.8 %)—together represent roughly 19 % of the collection' is arithmetically incorrect: the three named countries contribute 246 reports, which is 9.3% of 2,658, not 19%. If the intended 19% figure includes additional countries beyond the three named, those countries should be listed; otherwise the percentage should be corrected. Given that geographic diversity is a central selling point, this overstatement must be fixed.
  4. [§2.4, §2.5, §3.3] The human-versus-AI differentiation study is not described in sufficient detail to be reproducible or interpretable. Section 2.4 does not specify the prompt, model version, or generation settings used for GPT-o1; the number of AI-generated reports; how the five human reports and five AI reports were selected and matched for each participant (e.g., by language, modality, and anatomical region); or whether participants saw reports with metadata and translations. Section 2.5 states that differences were assessed with a χ2 test, but Section 3.3 reports a logistic regression with participant-level estimates; the unit of analysis (participant vs. report), the handling of repeated observations (each participant judged ten reports), and the clustering of observations are not described. These gaps make the headline 53.9% accuracy and the radiologist-superiority finding difficult to evaluate.
  5. [§2.2 and §4.1] The central utility claim that PARROT 'serves as a valuable benchmark for creating and testing AI applications capable of functioning across diverse healthcare systems and languages' (§4.2) presupposes that the fictional reports are representative of real clinical reporting in each language and region. The paper does not provide direct evidence for this. The discrimination study (§3.3) shows only that PARROT reports are hard to distinguish from GPT-o1 output; it does not show that either text resembles real clinical reports. The instructions in §2.2 to create 'plausible but non-specific clinical scenarios' with 'typical incidental findings' and 'realistic frequencies' could systematically exclude the abbreviations, telegraphic phrasing, and inconsistencies common in real-world reports. The Limitations section (§4.1) acknowledges the lack of linkage to real imaging and outcomes and selection bias, but it does not test whether the corpus's textual distribution matches real reports. Please either add a validation study comparing lexical and structural statistics of PARROT reports against de-identified real reports from the same institutions, or explicitly state that the dataset is not a proxy for real clinical text and adjust the claims accordingly.
minor comments (5)
  1. [§2.4] The discrimination study used reports in English, German, Italian, French, Greek, and Polish, while the dataset contains 13 languages; the paper should clarify why these six were chosen and whether the reported 53.9% accuracy is averaged across all six languages or weighted by participant counts.
  2. [Table 3] The row 'NA 0.0 U09' is confusing because U09 (post-COVID-19 condition) is a valid ICD-10 code; if it is classified as 'NA' this needs explanation, and the chapter percentages should sum to 100% with no unexplained 'NA' category.
  3. [Figures 2 and 3] Both Figure 2 and Figure 3 appear to include a panel C titled 'Effect sizes from logistic regression model'; please check whether this is a duplication and ensure each figure presents a unique set of panels.
  4. [§2.2] The description of ICD-10 coding says codes were assigned 'by the contributor or by B. Le Guellec with assistance from the o3-mini-high language model (OpenAI)'; for reproducibility, please specify the model version and date and describe how disagreements were resolved beyond 'dialogue with contributors.'
  5. [§1] The statement that 'evaluation metrics developed for one region often fail to generalize across reports from different countries' cites a paper about metric inconsistencies; please ensure the citation supports the claim about regional generalization.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the dataset construction and the human/AI discrimination experiment are self-contained, and no claimed derivation reduces to its own inputs.

full rationale

This paper is a dataset resource paper, not a derivation. It reports the construction of 2,658 fictional radiology reports and a human/AI discrimination experiment, neither of which involves fitting a parameter and then predicting a closely related quantity. The discrimination study compares PARROT reports against GPT-o1 output and measures participant accuracy (53.9% overall), which is an independent empirical result. The claim that PARROT enables NLP development across languages is a utility assertion supported by the dataset's composition; the concern that fictional reports may not be representative of real clinical practice is a validity limitation acknowledged in the Limitations section, but this is an evidential weakness, not circularity. The only self-referential elements are that some co-authors contributed reports and that coding was assisted by an author and an LLM, but these do not make any measured result equivalent to an input by construction. The 'largest' superlative is not supported by a systematic comparison with other open multilingual radiology datasets, but that is a correctness/evidence issue, not a circularity issue. No specific step in the paper reduces to its own inputs by definition or by self-citation, so the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

This paper adds no fitted parameters or invented entities; its claims rest on dataset-construction assumptions about authenticity, annotation, translation, and the generalizability of the human/AI discrimination test.

assumptions (5)
  • domain assumption Volunteer-authored fictional radiology reports capture authentic national reporting styles and terminology.
    Section 2.2 instructs contributors to follow typical practice; Section 4.1 acknowledges self-selection and lack of validation against real reports.
  • domain assumption All submissions are genuinely fictional, so open sharing carries no privacy risk.
    Section 2.1 relies on each contributor's attestation; no independent de-identification or verification is described.
  • domain assumption English translations provided by contributors are clinically accurate and preserve meaning.
    Section 2.2 states translations were provided; no translation QA or back-translation check is reported.
  • domain assumption ICD-10 codes assigned by contributors or by o3-mini with author review are correct enough for benchmarking.
    Section 2.2 describes LLM-assisted coding; no inter-rater reliability or manual audit is reported.
  • domain assumption The human/AI discrimination study is a valid measure of report authenticity.
    Section 2.4 describes the setup; participants were recruited via LinkedIn and professional networks, and AI-report generation details are not fully specified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PARROT: An Open Multilingual Radiology Reports Dataset." pith.science (2026). https://pith.science/paper/ULG4XTAH

@misc{pith2026250722939,
  author       = {Pith},
  title        = {Pith review of: PARROT: An Open Multilingual Radiology Reports Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULG4XTAH}},
  note         = {Machine review of arXiv:2507.22939}
}
read the original abstract

Rationale and Objectives: To develop and validate PARROT (Polyglottal Annotated Radiology Reports for Open Testing), a large, multicentric, open-access dataset of fictional radiology reports spanning multiple languages for testing natural language processing applications in radiology. Materials and Methods: From May to September 2024, radiologists were invited to contribute fictional radiology reports following their standard reporting practices. Contributors provided at least 20 reports with associated metadata including anatomical region, imaging modality, clinical context, and for non-English reports, English translations. All reports were assigned ICD-10 codes. A human vs. AI report differentiation study was conducted with 154 participants (radiologists, healthcare professionals, and non-healthcare professionals) assessing whether reports were human-authored or AI-generated. Results: The dataset comprises 2,658 radiology reports from 76 authors across 21 countries and 13 languages. Reports cover multiple imaging modalities (CT: 36.1%, MRI: 22.8%, radiography: 19.0%, ultrasound: 16.8%) and anatomical regions, with chest (19.9%), abdomen (18.6%), head (17.3%), and pelvis (14.1%) being most prevalent. In the differentiation study, participants achieved 53.9% accuracy (95% CI: 50.7%-57.1%) in distinguishing between human and AI-generated reports, with radiologists performing significantly better (56.9%, 95% CI: 53.3%-60.6%, p<0.05) than other groups. Conclusion: PARROT represents the largest open multilingual radiology report dataset, enabling development and validation of natural language processing applications across linguistic, geographic, and clinical boundaries without privacy constraints.

Figures

Figures reproduced from arXiv: 2507.22939 by the authors.

Figure 1
Figure 1. Geographic distribution and characteristics of the PARROT dataset. A) World [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Results of the human vs. AI report differentiation study A) Distribution of [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Analysis of human vs. AI report differentiation performance A) Accuracy heat [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Treemap of ICD-10 Codes in the PARROT dataset, grouped by the chapter [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

  1. [1]

    M. P. Hartung, I. C. Bickle, F. Gaillard, and J. P. Kanne. How to create a great radiology report. RadioGraphics, 40(6):1658–1670, 2020

    work page 2020

  2. [2]

    C. M. Hawkins, S. Hall, B. Zhang, and A. J. Towbin. Creation and implementation of department-widestructured reports: An analysis of the impact on errorrate in radiology reports. Journal of Digital Imaging, 27(5):581–587, 2014

    work page 2014

  3. [3]

    J. M. Nobel, K. Van Geel, and S. G. F. Robben. Structured reporting in radiology: a systematic review to explore its potential.European Radiology, 32(4):2837–2854, 2022

    work page 2022

  4. [4]

    L. H. Schwartz, D. M. Panicek, A. R. Berk, Y. Li, and H. Hricak. Improving com- munication of diagnostic radiology findings through structured reporting. Radiology, 260(1):174–181, 2011

    work page 2011

  5. [5]

    Keshavarz, S

    P. Keshavarz, S. Bagherieh, S. A. Nabipoorashrafi, H. Chalian, A. A. Rahsepar, G. H. J. Kim, et al. Chatgpt in radiology: A systematic review of performance, pitfalls, and future perspectives. Diagnostic and Interventional Imaging, 105(7):251–265, 2024

    work page 2024

  6. [6]

    R. J. Gertz, T. Dratsch, A. C. Bunck, S. Lennartz, A.-I. Iuga, M. G. Hellmich, et al. Potential of gpt-4 for detecting errors in radiology reports: Implications for reporting accuracy. Radiology, 311:e232714, 2024. 16

    work page 2024

  7. [7]

    Le Guellec, C

    B. Le Guellec, C. Bruge, N. Chalhoub, V. Chaton, E. De Sousa, Y. Gaillandre, et al. Comparison between multimodal foundation models and radiologists for the diagnosis of challenging neuroradiology cases with text and images.Diagnostic and Interventional Imaging, 2025

    work page 2025

  8. [8]

    L. C. Adams, D. Truhn, F. Busch, A. Kader, S. M. Niehues, M. R. Makowski, et al. Leveraging gpt-4 for post hoc transformation of free-text radiology reports into struc- tured reporting: A multilingual feasibility study.Radiology, page e230725, 2023

    work page 2023

Show all 20 references
  1. [9]

    Le Guellec, A

    B. Le Guellec, A. Lefèvre, C. Geay, L. Shorten, C. Bruge, L. Hacein-Bey, et al. Perfor- mance of an open-source large language model in extracting information from free-text radiology reports. Radiology: Artificial Intelligence, page e230364, 2024

    2024

  2. [10]

    K. S. Amin, M. A. Davis, R. Doshi, A. H. Haims, P. Khosla, and H. P. Forman. Accuracy of chatgpt, google bard, and microsoft bing for simplifying radiology reports.Radiology, 309:e232561, 2023

    2023

  3. [11]

    A. E. W. Johnson, L. Bulgarelli, L. Shen, A. Gayles, A. Shammout, S. Horng, et al. Mimic-iv, a freely accessible electronic health record dataset.Scientific Data, 10:1, 2023

    2023

  4. [12]

    A. E. W. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. Deng, et al. Mimic-cxr, a de-identified publicly available database of chest radio- graphs with free-text reports.Scientific Data, 6:317, 2019

    2019

  5. [13]

    J. Wu, X. Liu, M. Li, W. Li, Z. Su, S. Lin, et al. Clinical text datasets for medical artificial intelligence and large language models — a systematic review.NEJM AI, 1, 2024

    2024

  6. [14]

    Banerjee, A

    O. Banerjee, A. Saenz, K. Wu, W. Clements, A. Zia, D. Buensalido, et al. Rexamine- global: A framework for uncovering inconsistencies in radiology report generation met- rics. In Biocomputing 2025, pages 185–198. World Scientific, 2024

    2025

  7. [15]

    Al-Duwais, H

    M. Al-Duwais, H. Al-Khalifa, and A. Al-Salman. A benchmark evaluation of multilin- gual large language models for arabic cross-lingual named-entity recognition.Electron- ics, 13(17):3574, 2024

    2024

  8. [16]

    Campos, V

    L. Campos, V. Pedro, and F. Couto. Impact of translation on named-entity recognition in radiology texts.Database, 2017, 2017

    2017

  9. [17]

    Busch, L

    F. Busch, L. Hoffmann, L. Xu, L. Zhang, B. Hu, I. García-Juárez, et al. Multinational attitudes towards ai in healthcare and diagnostics among hospital patients.medRxiv, 2024

    2024

  10. [18]

    Bias in large language models across clinical applications: A systematic review.arXiv, 2025

    Thanathip Suenghataiphorn, Narisara Tribuddharat, Pojsakorn Danpanichkul, and Narathorn Kulthamrongsri. Bias in large language models across clinical applications: A systematic review.arXiv, 2025

    2025

  11. [19]

    Openai o1 system card

    OpenAI. Openai o1 system card. https://openai.com/index/ openai-o1-system-card/, 2024. Accessed December 11, 2024. 17

    2024

  12. [20]

    J. Wang, H. Deng, B. Liu, A. Hu, J. Liang, L. Fan, et al. Systematic evaluation of research progress on natural language processing in medicine over the past 20 years: Bibliometric study on pubmed. Journal of Medical Internet Research, 22(1):e16816, 2020. 4.3 Supplementary fig...

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.