Pith. sign in

REVIEW 3 major objections 5 minor 43 references

Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Six clinical NLP systems disagree sharply on what pediatric chest X-ray reports say, so none can be trusted for automated labeling without local validation.

desk verdict Solid independent benchmark showing real variability among commercial and open-source CXR NLP systems, but the headline accuracy numbers are consensus-agreement scores against a self-included majority vote, not clinical accuracy. read the letter →

arxiv 2505.23030 v1 pith:JONGCCIJ submitted 2025-05-29 cs.CL

Shruti Hegde , Mabon Manoj Ninan , Jonathan R. Dillman , Shireen Hayatghaibi , Lynn Babcock , Elanchezhian Somasundaram This is my paper
classification cs.CL
keywords clinicalnaturallanguageprocessingchestradiographreportspediatricradiologyassertiondetectionnamedentityrecognitionCheXpertlabelscommercialNLPsystemsevaluationstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether off-the-shelf clinical NLP systems can reliably turn pediatric chest radiograph reports into structured disease labels. It compares four commercial general-purpose systems and two dedicated open-source labelers on 95,008 pediatric CXR reports, and finds wide disagreement in which entities are extracted and whether each is called positive, negative, or uncertain. Against a consensus ground truth created by majority voting across all six systems, overall assertion accuracy is 62 percent, with the best system at 76 percent and the worst at 50 percent; the two open-source labelers both sit at 56 percent. The authors conclude that no single system is uniformly reliable and that task-specific, institution-specific validation with manual review is required before deployment.

What carries the argument

The load-bearing object is the evaluation pipeline that renders six incompatible output schemas comparable. A regular-expression mapper standardizes each vendor's extracted entities onto the 13 CheXpert labels; each system's native assertion output is mapped into a common three-way schema of positive, negative, and uncertain; and a consensus pseudo-ground truth is formed by majority voting across the six systems, with ties assigned to uncertain. Fleiss' kappa is then computed under two conditions, all exams and exams excluding unanimous 'absent' predictions. This pipeline defines every accuracy number in the paper, so the paper's conclusions stand or fall with the consensus definition it embeds.

What would settle it

Take a stratified random sample of roughly 500 to 1,000 of these pediatric CXR impressions, have two or more radiologists independently assign the 13 CheXpert labels with adjudication, and recompute each system's assertion accuracy against that manual gold standard. If the system ranking against radiologist labels differs materially from the ranking against the majority-vote consensus, the paper's relative accuracy claims fail; if the rankings match, the consensus approach is validated for this dataset.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that six NLP systems do not form a stable consensus about pediatric CXR reports. Across the 13 CheXpert disease categories, inter-model agreement looks substantial only because the systems frequently agree that a disease is absent; once unanimously absent cases are removed, mean Fleiss kappa drops from 0.68 to 0.35. When each system's assertions are compared with the majority-vote pseudo-ground truth, mean accuracy is 62 ± 9 percent: SparkNLP reaches 76 percent, Amazon Comprehend Medical 50 percent, and CheXpert and CheXbert each 56 percent. Category-level accuracy ranges from 14 percent for consolidation to 72 percent for pleural effusion. The paper's intended message is that these systems capture real signal but operationalize negation, uncertainty, and disease presence in incompatible ways, and the authors explicitly caution that metrics against the pseudo-ground truth may differ from metrics against radiologist-assigned labels.

Load-bearing premise

The accuracy and ranking figures assume that a majority vote across the six NLP systems is a good stand-in for what a radiologist would say; if the majority is wrong, the reported numbers measure inter-system agreement, not clinical correctness.

Editorial extensions

If this is right

  • No single commercial or open-source NLP system should be assumed reliable for pediatric CXR labeling without a local validation step on the institution's own reports.
  • Labels produced by different NLP systems are not interchangeable in clinical research; apparent disease counts and uncertainty rates depend on which system generated them.
  • Aggregate accuracy hides category-level failures, so systems should be selected per disease category rather than globally.
  • Because agreement collapses when unanimously absent cases are removed, the hard problem is assigning positive, negative, or uncertain status when a disease is mentioned, not detecting it at all.
  • Downstream models trained on one system's labels inherit that system's assertion errors; the paper points to ensembling multiple systems as a natural next step for improving robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit is that procurement decisions for clinical NLP should budget for a reference-standard validation pass on local data, because vendor benchmarks and adult datasets will not transfer to pediatric chest radiography.
  • The convention of treating a missing label as 'absent' inflates both inter-model agreement and apparent accuracy; an evaluation restricted to reports where at least one system mentioned the disease would likely show lower numbers and a different ranking.
  • A testable extension of the paper's ensembling suggestion: treat the six systems as a voting committee and compare the majority-vote label's accuracy against a small radiologist-annotated gold set; if the committee outperforms every member, ensemble pipelines are worth building.
  • Another testable extension: quantify how much each system's assertion errors propagate into a downstream task, such as pneumonia outcome prediction, to turn the observed disagreement into a clinical impact estimate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper compares four commercial clinical NLP systems (AWS Comprehend Medical, Google Healthcare NLP, Azure Clinical NLP, and John Snow Labs SparkNLP) and two open-source CXR labelers (CheXpert, CheXbert) on 95,008 pediatric chest radiograph reports. The authors standardize entity types and assertion categories, map extracted entities to CheXpert disease labels using a custom regex algorithm, and evaluate inter-system agreement with Fleiss' kappa and per-system accuracy against a consensus pseudo-ground truth built by majority voting across all six systems. They report significant variability in entity counts, assertion distributions, and CheXpert-label agreement, with overall assertion accuracy ranging from 50% (AWS) to 76% (SP), and CheXpert/CheXbert at 56%. The paper concludes that no single system is reliable without local validation.

Significance. The central qualitative finding—large cross-system variability in entity extraction and assertion detection—is important and actionable for healthcare institutions considering commercial NLP for radiology report labeling. The study's strengths include its large independent pediatric dataset, transparent reporting of raw entity counts and assertion distributions, use of chi-square tests and Fleiss' kappa (including an absent-excluded sensitivity analysis), and concrete examples of discrepant interpretations. These observational results likely hold regardless of the pseudo-ground-truth construction. However, the headline per-system accuracy numbers in Table 5 and the Abstract are agreement scores against a consensus that includes each system itself, not measures of clinical correctness; they should be interpreted and presented with that caveat. The finding of variability is robust, but the quantitative accuracy ranking is not.

major comments (3)
  1. [Methods, Evaluation metrics; Table 5; Abstract; Conclusions] The pseudo-ground-truth accuracy scores are partially self-referential. For each disease label, the consensus is formed by majority vote over all six systems, and each system's own output is then scored against that consensus. A system that agrees with the majority more often is rewarded not because it is clinically correct but because it is typical. The Discussion (Limitations and future directions) acknowledges that performance metrics may differ from radiologist labels, but the Abstract and Conclusions present the numerical accuracy values (SP 76%, AWS 50%, CheXpert/CheXbert 56%) without this caveat. This is load-bearing because those numbers are the quantitative headline of the paper. Please recompute the accuracy scores using a leave-one-out majority vote (excluding the system being scored) or, preferably, validate on a manually annotated subset, and report how the ranking changes. At minimum, relabel all such values as 'agreement with consensus' rather than 'accuracy' throughout the Abstract and Results.
  2. [Methods, Evaluation metrics; Methods, CheXpert disease labels and models] The assignment of 'absent' whenever a system does not emit a label conflates extraction failure with a confident negative prediction. In the Fleiss' kappa calculation and in the accuracy evaluation, a system that fails to detect a finding is treated as having predicted 'absent', so it is scored as correct whenever the majority also predicts 'absent'. This artificially inflates agreement and accuracy for rare or absent-heavy categories, and it penalizes systems that are simply less sensitive. The paper should either treat non-detection as a separate category (e.g., 'not mentioned' versus 'absent') or perform a sensitivity analysis on exams where at least one system detected the disease, as is already done for the kappa analysis. This issue directly affects the Table 5 accuracy values and the overall ranking.
  3. [Methods, Table 2 (Standardization of assertion status)] The AWS assertion mapping relies on arbitrary negation confidence thresholds: CS < 0.25 is positive, CS > 0.75 is negative, and the interval in between is uncertain. These thresholds are free parameters chosen by the authors, and the paper reports no sensitivity analysis. Since AWS's accuracy in Table 5 is the lowest (50%), and its assertion standardization is unique among the six systems, the ranking could change with different thresholds. Please justify the choice of thresholds or report results over a range of thresholds (e.g., 0.1/0.9, 0.2/0.8, 0.4/0.6) to show that the main conclusions are not threshold-dependent.
minor comments (5)
  1. [Results, Section a] The manuscript contains an unresolved reference artifact: 'Error! Reference source not found.' appears after Table 3 in the Results section. This should be replaced with the actual table or figure reference.
  2. [Methods, section ordering] The Methods section uses repeated and out-of-order subsection labels: 'd) Mapping clinical entities...' follows an earlier 'd) Standardization of assertion status', and 'e) Evaluation metrics' appears twice. Please renumber the subsections sequentially.
  3. [Table 5, Overall Mean row] The 'Overall Mean' row in Table 5 reports AWS as 50 ± 0 and GC as 63 ± 0, but a simple mean of the per-disease percentages in the AWS column is approximately 42%, not 50%. Please clarify how the overall mean and its standard deviation are computed; if the overall mean is weighted by the number of exams per category, state that explicitly and report the weights or the formula.
  4. [Abstract and Results] The text alternates between calling the pseudo-ground-truth comparison 'accuracy' and 'agreement' (e.g., 'assertion accuracy' in the Abstract vs. 'agreement' in the Discussion). For consistency and to avoid overclaiming, use 'agreement with consensus' wherever the pseudo-ground-truth scores are reported, and reserve 'accuracy' for any future comparison against manual annotations.
  5. [Results, Table 4 and Figure 4] The paper reports percentages with counts but does not provide confidence intervals for the assertion distributions or the kappa values. Given the large sample size, even small differences will be statistically significant; reporting effect sizes or confidence intervals would help readers assess the magnitude of variability.

Circularity Check

2 steps flagged · score 6.0 of 10

Pseudo-ground-truth circularity: Table 5 accuracy scores each system against a majority vote that includes that system, and treats non-detection as 'absent', so the reported ranking is partly self-referential; the variability findings remain independent.

  1. self definitional [Methods, Evaluation metrics (pseudo-ground truth and assertion accuracy); Table 5]
    "To estimate model-specific assertion performance, a pseudo–ground truth was established using a majority voting strategy across outputs from all six NLP systems. For each disease entity, the consensus assertion label was determined by majority vote. If no majority was reached, the entity was assigned to the uncertain category. Assertion accuracy for each model was computed by comparing its predicted assertion category to the consensus label for each disease."

    The reference standard for scoring each system is a majority vote over the outputs of the six evaluated systems, and the system being scored contributes one of those six votes. Each system's reported 'accuracy' therefore measures how often its assertion matches a consensus that it helped construct, not agreement with an independent clinical reference. The headline values in Table 5 and the Abstract (SP 76%, AWS 50%, CheXpert/CheXbert 56%) are, by construction, partly measures of self-consensus; the ranking is not an independent measure of clinical correctness.

  2. self definitional [Methods, Evaluation metrics (assignment of absent category)]
    "The assertion category absent was assigned when an NLP system did not detect a given label."

    This scoring convention converts extraction failure into the assertion category 'absent'. Because the consensus pseudo-ground truth is built by majority voting over these categories, a system that simply fails to emit a disease label is credited as correct whenever the majority is also absent, and the system itself contributes to that majority. The accuracy metric thus rewards non-detection as a confident negative, further coupling the reported scores to the systems' own output behavior rather than to verified radiological absence.

full rationale

The paper's central qualitative finding—large variability across systems in entity counts and assertion distributions—is supported by raw counts, paired t-tests, chi-square tests, and Fleiss' Kappa, and is not circular. However, the quantitative 'assertion accuracy' results (Table 5) are evaluated against a pseudo-ground truth formed by majority vote across the six evaluated systems, with each system included in its own reference standard. A system's accuracy therefore reduces, in part, to agreement with a consensus it helps create. This is compounded by assigning 'absent' to any label the system failed to emit, so non-detection is scored as a correct negative whenever the majority is absent. The authors disclose in the Limitations that manual annotation was not performed and that pseudo-ground-truth metrics may differ from radiologist labels, which is honest and prevents a higher score, but the Abstract and Results still present the consensus-relative numbers as 'accuracy'. No load-bearing self-citation or imported uniqueness theorem was found. Score 6 reflects partial circularity: the variability claim is independent, while the accuracy ranking is self-referential by construction.

Assumptions & free parameters 1 free parameters · 2 assumptions · 0 invented entities

This is an empirical evaluation, not a theory paper. It introduces no new particles, forces, or conserved quantities. The only hand-tuned component is the AWS confidence threshold, and the only notable assumptions are about the validity of the majority-vote pseudo-ground truth and the treatment of missing labels as absences.

free parameters (1)
  • AWS negation confidence thresholds = 0.25 and 0.75
    AWS outputs only a numerical confidence score for negation, and the authors chose CS < 0.25 as positive, CS > 0.75 as negative, and the middle range as uncertain. This hand-picked mapping directly affects AWS assertion distributions and accuracy.
assumptions (2)
  • domain assumption Majority vote across the six NLP systems is a usable proxy for true radiologist interpretation.
    The consensus pseudo-ground truth is the reference for all assertion accuracy calculations (Methods, Evaluation metrics). The authors acknowledge in Limitations that this is not a radiologist-annotated standard.
  • domain assumption A missing label by a system means the disease is absent.
    In the CheXpert label comparison, the assertion category "absent" is assigned when an NLP system does not detect a given label, which conflates extraction failure with a confident negative (Methods, Evaluation metrics).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset." pith.science (2026). https://pith.science/paper/JONGCCIJ

@misc{pith2026250523030,
  author       = {Pith},
  title        = {Pith review of: Can Modern NLP Systems Reliably Annotate Chest Radiography Exams? A Pre-Purchase Evaluation and Comparative Study of Solutions from AWS, Google, Azure, John Snow Labs, and Open-Source Models on an Independent Pediatric Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JONGCCIJ}},
  note         = {Machine review of arXiv:2505.23030}
}
read the original abstract

General-purpose clinical natural language processing (NLP) tools are increasingly used for the automatic labeling of clinical reports. However, independent evaluations for specific tasks, such as pediatric chest radiograph (CXR) report labeling, are limited. This study compares four commercial clinical NLP systems - Amazon Comprehend Medical (AWS), Google Healthcare NLP (GC), Azure Clinical NLP (AZ), and SparkNLP (SP) - for entity extraction and assertion detection in pediatric CXR reports. Additionally, CheXpert and CheXbert, two dedicated chest radiograph report labelers, were evaluated on the same task using CheXpert-defined labels. We analyzed 95,008 pediatric CXR reports from a large academic pediatric hospital. Entities and assertion statuses (positive, negative, uncertain) from the findings and impression sections were extracted by the NLP systems, with impression section entities mapped to 12 disease categories and a No Findings category. CheXpert and CheXbert extracted the same 13 categories. Outputs were compared using Fleiss Kappa and accuracy against a consensus pseudo-ground truth. Significant differences were found in the number of extracted entities and assertion distributions across NLP systems. SP extracted 49,688 unique entities, GC 16,477, AZ 31,543, and AWS 27,216. Assertion accuracy across models averaged around 62%, with SP highest (76%) and AWS lowest (50%). CheXpert and CheXbert achieved 56% accuracy. Considerable variability in performance highlights the need for careful validation and review before deploying NLP tools for clinical report labeling.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 42 canonical work pages

  1. [1]

    No appreciable pneumothorax on the left

    Pectus bars in place. 2. No appreciable pneumothorax on the left. SP CheXpert, CheXbert, AWS, GC AZ 2 pneumonia Findings consistent with viral or reactive airways disease without focal pneumonia. AZ SP CheXpert, CheXbert, AWS, GC 3 cardiomegaly No acute cardiopulmonary abnormality with stable cardiomegaly and fracture of one of the pacemakers leads. CheXp...

    work page

  2. [2]

    22, Mar 3

    Soumya Upadhyay, H.-f.H., A Qualitative Analysis of the Impact of Electronic Health Records (EHR) on Healthcare Quality and Safety: Clinicians’ Lived Experiences. 22, Mar 3

    work page

  3. [3]

    The European Journal of Public Health, 2016

    Campanella, P., et al., The impact of electronic health records on healthcare quality: a systematic review and meta-analysis. The European Journal of Public Health, 2016. 26(1): p. 60-64

    work page 2016

  4. [4]

    Journal of biomedical informatics, 2018

    Wang, Y., et al., Clinical information extraction applications: a literature review. Journal of biomedical informatics, 2018. 77: p. 34-49

    work page 2018

  5. [5]

    Hang Dong, M.F., William Whiteley, Beatrice Alex, Joshua Matterson, Shaoxiong Ji, Jiaoyan Chen, Honghan Wu, Automated clinical coding: what, why, and where we are? npj Digital Medicine, 2022

    work page 2022

  6. [6]

    J Intell Connect Emerg Technol, 2023

    Ahmed, U., et al., Natural language processing for clinical decision support systems: a review of recent advances in healthcare. J Intell Connect Emerg Technol, 2023. 8(2): p. 1-17

    work page 2023

  7. [7]

    2020, Oct 14

    Emma L Barber, R.G., Christianne Persenaire, Melissa Simon Natural Language Processing with Machine Learning to Predict Outcomes after Ovarian Cancer Surgery. 2020, Oct 14

    work page 2020

  8. [8]

    Journal of biomedical informatics,

    Velupillai, S., et al., Using clinical natural language processing for health outcomes research: overview and actionable suggestions for future advances. Journal of biomedical informatics,

    work page

Show all 43 references
  1. [9]

    Selden, and R

    Jerfy, A., O. Selden, and R. Balkrishnan, The Growing Impact of Natural Language Processing in Healthcare and Public Health. INQUIRY: The Journal of Health Care Organization, Provision, and Financing, 2024. 61: p. 00469580241290095

    2024

  2. [10]

    RSNA, 2016

    Ewoud Pons, L.M.M.B., M G Myriam Hunink, Jan A Kors Natural Language Processing in Radiology: A Systematic Review. RSNA, 2016

    2016

  3. [11]

    American Journal of Infection Control, 2024

    Mahmud Omar, D.B., Benjamin Glicksberg, Eyal Klang, Utilizing natural language processing and large language models in the diagnosis and prediction of infectious diseases: A systematic review. American Journal of Infection Control, 2024. 52(9)

    2024

  4. [12]

    CTS, 2020

    Kevin B Johnson, W.Q.W., Dilhan Weeraratne, Mark E Frisse, Karl Misulis, Kyu Rhee, Juan Zhao, Jane L Snowdon, Precision Medicine, AI, and the Future of Personalized Health Care. CTS, 2020

    2020

  5. [13]

    Health and Technology, 2022

    Papadopoulos, P., et al., A systematic review of technologies and standards used in the development of rule-based clinical decision support systems. Health and Technology, 2022. 12(4): p. 713-727

    2022

  6. [14]

    Slawomir Kierner, J.K., Zofia Kierner, Taxonomy of hybrid architectures involving rule-based reasoning and machine learning in clinical decision systems: A scoping review. 2023

    2023

  7. [15]

    JMIR Med Inform, 2019

    Aryan Arbabi, D.R.A., Sanja Fidler, Michael Brudno Identifying Clinical Terms in Medical Text Using Ontology-Guided Machine Learning. JMIR Med Inform, 2019

    2019

  8. [16]

    Benyamin Ghojogh, A.G., Recurrent Neural Networks and Long Short-Term Memory Networks: Tutorial and Survey. 2023

    2023

  9. [17]

    Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need

    Ashish Vaswani, N.S., Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin, Attention Is All You Need. 2017

    2017

  10. [18]

    Radford, A., et al., Improving language understanding by generative pre-training. 2018

    2018

  11. [19]

    Jacob Devlin, M.-W.C., Kenton Lee, Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2018

    2018

  12. [20]

    American Journal of Roentgenology,

    Makhnevich, A., et al., The clinical utility of chest radiography for identifying pneumonia: accounting for diagnostic uncertainty in radiology reports. American Journal of Roentgenology,

  13. [21]

    Kim, J. and K.H. Kim, Role of chest radiographs in early lung cancer detection. Translational lung cancer research, 2020. 9(3): p. 522

    2020

  14. [22]

    Paparo, and G

    Piccazzo, R., F. Paparo, and G. Garlaschi, Diagnostic accuracy of chest radiography for the diagnosis of tuberculosis (TB) and its role in the detection of latent TB infection: a systematic review. The Journal of Rheumatology Supplement, 2014. 91: p. 32-40

    2014

  15. [23]

    79(930): p

    Gatt, M., et al., Chest radiographs in the emergency department: is the radiologist really necessary? Postgraduate medical journal, 2003. 79(930): p. 214-217

    2003

  16. [24]

    World journal of radiology, 2014

    Cardinale, L., et al., Effectiveness of chest radiography, lung ultrasound and thoracic computed tomography in the diagnosis of congestive heart failure. World journal of radiology, 2014. 6(6): p. 230

    2014

  17. [25]

    IEEE Access, 2020

    Anis, S., et al., An overview of deep learning approaches in chest radiograph. IEEE Access, 2020. 8: p. 182347-182354

    2020

  18. [26]

    Johnson, J. and J.A. Kline, Intraobserver and interobserver agreement of the interpretation of pediatric chest radiographs. Emergency radiology, 2010. 17: p. 285-290

    2010

  19. [27]

    Mong, Safwan S

    Jeremy Irvin, P.R., Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Ma...

    2019

  20. [28]

    Nature biomedical engineering, 2022

    Tiu, E., et al., Expert-level detection of pathologies from unannotated chest X-ray images via self- supervised learning. Nature biomedical engineering, 2022. 6(12): p. 1399-1406

    2022

  21. [29]

    arXiv preprint arXiv:2211.12737, 2022

    Chambon, P., et al., Roentgen: vision-language foundation model for chest x-ray generation. arXiv preprint arXiv:2211.12737, 2022

    2022 arXiv

  22. [30]

    Alistair E. W. Johnson, T.J.P., Seth J. Berkowitz, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih-ying Deng, Roger G. Mark & Steven Horng MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. 2019

    2019

  23. [31]

    Chexpert++: Approximating the chexpert labeler for speed, differentiability, and probabilistic output

    McDermott, M.B., et al. Chexpert++: Approximating the chexpert labeler for speed, differentiability, and probabilistic output. in Machine Learning for Healthcare Conference. 2020. PMLR

    2020

  24. [32]

    arXiv preprint arXiv:2311.01349, 2023

    Weber, T., et al., Post-hoc Orthogonalization for Mitigation of Protected Feature Bias in CXR Embeddings. arXiv preprint arXiv:2311.01349, 2023

    2023 arXiv

  25. [33]

    Journal of the American Medical Informatics Association, 2010

    Savova, G.K., et al., Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications. Journal of the American Medical Informatics Association, 2010. 17(5): p. 507-513

    2010

  26. [34]

    Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program

    Aronson, A.R. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. in Proceedings of the AMIA Symposium. 2001

    2001

  27. [35]

    https://aws.amazon.com/comprehend/medical/

    AWS. https://aws.amazon.com/comprehend/medical/

  28. [36]

    Clinical entity extraction: comparison between MetaMap, cTAKES, CLAMP and Amazon Comprehend Medical

    Bai, L., et al. Clinical entity extraction: comparison between MetaMap, cTAKES, CLAMP and Amazon Comprehend Medical. in 2021 32nd Irish Signals and Systems Conference (ISSC). 2021. IEEE

    2021

  29. [37]

    Microsoft Azure

    AZ. Microsoft Azure. Available from: https://azure.microsoft.com/en-us/

  30. [38]

    Google Cloud

    GC. Google Cloud. Available from: https://cloud.google.com/healthcare-api/

  31. [39]

    Ng, Matthew P

    Akshay Smit, S.J., Pranav Rajpurkar, Anuj Pareek, Andrew Y. Ng, Matthew P. Lungren, CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERT. 2020

    2020

  32. [40]

    John Snow Labs

    JSL. John Snow Labs. Available from: https://www.johnsnowlabs.com/

  33. [41]

    Diego Pinheiro da Silva, W.d.R.F., Blanda Helena de Mello, Renata Vieira, Sandro José Rigo, Exploring named entity recognition and relation extraction for ontology and medical records integration. 2023

    2023

  34. [42]

    Sleeman, Jatinder Palta, Rishabh Kapoor, Preetam Ghosh, A Survey on Recent Named Entity Recognition and Relationship Extraction Techniques on Clinical Texts

    Priyankar Bose, S.S., William C. Sleeman, Jatinder Palta, Rishabh Kapoor, Preetam Ghosh, A Survey on Recent Named Entity Recognition and Relationship Extraction Techniques on Clinical Texts. 2021

    2021

  35. [2019]

    1207-1212

    213(6): p. 1207-1212

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.