Pith. sign in

REVIEW 4 major objections 5 minor 22 references

RelCAT: Advancing Extraction of Clinical Inter-Entity Relationships from Unstructured Electronic Health Records

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that RelCAT, a BERT-based relation classifier integrated with MedCAT, achieves macro F1 of 0.977 on the n2c2 clinical relation extraction benchmark — above prior state-of-the-art — and transfers to two real-world NHS…

desk verdict RelCAT is a useful MedCAT integration with two new NHS datasets, but the headline 0.977 n2c2 macro-F1 is contradicted by the paper's own Table 9, which averages to 0.960, so the SOTA claim does not hold as stated. read the letter →

arxiv 2501.16077 v1 pith:KYHNALRG submitted 2025-01-27 cs.CL

Shubham Agarwal , Vlad Dinu , Thomas Searle , Mart Ratas , Anthony Shek , Dan F. Stein , James Teo , Richard Dobson This is my paper · ORCID
classification cs.CL
keywords relationextractionclinicalNLPelectronichealthrecordsBERTSNOMEDCTn2c2datasettransferlearningannotationtoolkit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RelCAT is a workflow for classifying relationships between medical concepts already extracted from clinical notes. The paper's central claim is that a BERT-large classifier that marks entity spans, pools their hidden states, and is fine-tuned with all layers unfrozen achieves a macro F1 of 0.977 on the n2c2 benchmark, surpassing the previous best of 0.961. The same approach transfers to two small NHS datasets — spatial relations and physiotherapy mobility — where fine-tuning from the n2c2 model gives F1 scores of 0.918 and 0.938. The paper also introduces an interactive annotation tool and an automatic way to generate non-relation examples using SNOMED concept-type pairs. If the results hold, clinical information extraction pipelines could reliably pull structured relationships out of free-text records for analytics and decision support.

What carries the argument

The load-bearing mechanism is the entity-marker representation: adding [s1], [e1], [s2], and [e2] tokens around the two concepts, taking the hidden states of the entity spans, max-pooling them, and concatenating with BERT's pooled output before a classification head. This lets the model attend to the lexical and positional context between entities despite the many intervening words that separate clinical concepts. Class weights and stratified batching are used to keep minority relation classes from being crushed by the dominant classes in n2c2.

What would settle it

Two checks would settle the central claims: (1) recompute the macro F1 for the unfrozen BERT model from the per-class F1 values in Tables 9 and 14, which average to about 0.960, not 0.977, and reproduce 0.977 on the official n2c2 test splits; (2) on the NHS datasets, have clinicians manually annotate a held-out set of true non-relations from the same notes and compare the model's F1 on that set against the F1 computed on the automatically generated 'Other' class. A drop of more than a few points on the manually annotated negatives would indicate the auto-generated negatives inflate the reported performance.

Watch

Extended reading notes

Core claim

RelCAT's contribution is framing inter-entity relation classification as a span-pair classification task on transformer models: text is tokenized, the two entities are marked with special tokens or index positions, the model's hidden states for each entity (max-pooled) plus the pooled sequence representation are stacked, and a fully connected head predicts the relation type. On the n2c2 gold standard, the paper reports the unfrozen BERT-large configuration reaching a macro F1 of 0.977, with ADE-Drug and Duration-Drug minority classes at 0.866 and 0.933. On the NHS Spatial and Physiotherapy-Mobility datasets, the best models reach 0.933 and 0.938 F1. The paper further shows that zero-shot and few-shot prompting with Llama and Mistral lags far behind fine-tuning, with best zero/few-shot F1 around 0.49 and 0.46 respectively.

Load-bearing premise

The NHS non-relation ('Other') class is generated automatically by pairing entities whose SNOMED concept types do not match the target relation, capped at 70 per project, rather than being annotated by clinicians; if these generated negatives are simpler or less ambiguous than real non-relations, the reported 0.90-0.93 F1 overstates how the model would perform on genuinely unrelated entity pairs.

Editorial extensions

If this is right

  • If the 0.977 F1 is reproducible, clinical relation extraction from discharge summaries is near-solved for the eight n2c2 relation types, enabling downstream pharmacovigilance and medication-safety analytics.
  • Transferring a model trained on n2c2 to small specialist tasks yields a 15-16% gain over training from scratch, meaning shared general clinical relation models can bootstrap new relation types with modest annotation effort.
  • The automatic generation of non-relations from SNOMED type pairs allows creation of balanced binary datasets without manual negative annotation, a pattern other annotation pipelines could adopt.
  • Llama with frozen layers performed competitively on the small NHS datasets (0.933 and 0.835 F1), suggesting large frozen models are a viable alternative when compute for fine-tuning is limited.
  • The integrated annotation tool and training pipeline give a complete path from raw text to a trained relation model, lowering the barrier for hospitals to create task-specific relation extractors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline macro F1 of 0.977 does not match the macro average of the per-class F1 values in Table 9 (about 0.960), nor Table 14 (about 0.961); the number likely comes from a different run or aggregation, and it should be reconciled before being taken at face value.
  • The automatic non-relation generation likely makes the NHS evaluation easier than real clinical text with arbitrary entity co-occurrences; a blinded test with manually labeled negatives would give a truer estimate of deployed performance.
  • Because general BERT-large outperformed BioBERT and BioClinicalBERT when unfrozen, the paper suggests that general pre-training plus task fine-tuning can beat domain-specific pre-training for relation extraction, which runs counter to common practice and could change model selection in clinical NLP.
  • The toolkit's design — annotation tool, auto-negatives, training pipeline — may be the more durable contribution than any single model, since the same workflow could be applied to other domains where entity pairs and ontologies exist.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents RelCAT, a toolkit and workflow for classifying relations between clinical entities previously extracted by MedCAT. The authors integrate relation annotation into MedCATTrainer, train BERT-family and Llama models using entity markers and context windows, and evaluate on the n2c2 2018 medication-relation dataset and on two small NHS datasets (Spatial and Physiotherapy-Mobility). The headline results are a claimed macro F1 of 0.977 on n2c2, which the authors state surpasses a prior state-of-the-art macro F1 of 0.9610, and F1 scores of 0.90–0.94 on the two NHS datasets.

Significance. The toolkit itself, with its open-source integration into MedCAT, the comparative evaluation of multiple transformer and LLM approaches, and the use of real-world NHS data, is a potentially useful systems contribution to clinical NLP. If the empirical claims were correct, the n2c2 result would be a notable advance. However, the central SOTA claim is contradicted by the paper's own per-class table, and the construction of the NHS negative class raises validity concerns. The paper's value currently rests on the systems contribution rather than on the reported benchmark advantage, and the claims as stated are not reliable.

major comments (4)
  1. [Section III-1 / Table 9 / Abstract] The headline 'macro F1-score of 0.977' reported in the abstract, Section III-1, Table 4, and Section VI is not the macro average of the per-class F1 scores in Table 9 for the BERT (layers unfrozen) configuration. The eight per-class F1 values (0.993, 0.990, 0.964, 0.975, 0.933, 0.977, 0.979, 0.866) have an arithmetic mean of 0.960. The value 0.977 equals the micro-averaged F1 (which coincides with accuracy in this multi-class setting) and is close to the support-weighted average of the per-class scores. Since the cited prior SOTA of 0.9610 is a macro F1, the claimed improvement does not hold: 0.960 < 0.9610. This internal inconsistency undermines the central claim of the paper and must be corrected by reporting the true macro F1 or by explicitly reframing the comparison as micro/weighted F1.
  2. [Section II-C3 / Tables 2 and 3] The description of non-relation generation is internally inconsistent. The text states 'the number of non-relations were limited to a maximum 70 per project,' but Table 2 reports 332 'Other' instances in the Spatial dataset and Table 3 reports 301 in the Physiotherapy-Mobility dataset. If the cap applies per some other unit (e.g., per document, per TUI pair, or per annotator), that unit must be specified; as written, the negative-sampling protocol cannot be reproduced and the NHS results cannot be interpreted.
  3. [Section II-C3 / Tables 7 and 8] The automatically generated non-relation ('Other') class appears systematically different from the positive class in terms of entity-type (TUI) pairs. For example, in the Spatial dataset the top 'Other' pair is qualifier value–qualifier value (76 instances), whereas the top 'Spatial' pairs are body structure–morphologic abnormality and body structure–disorder. This raises a concrete risk that the classifier learns to predict the relation from the ontological types of the two entities rather than from textual evidence. The paper should provide an analysis showing that the model performs well when the TUI-pair distribution of the negative set is matched to the positive set, or should otherwise moderate the NHS performance claims.
  4. [Section III] The n2c2 evaluation does not specify how the 505 discharge summaries were divided into training, development, and test sets, nor whether the test set coincides with the official n2c2 2018 shared-task split. Without this information, the comparison with previously published macro F1 scores is not well-defined. The paper should also report multiple runs or confidence intervals, since the corrected macro F1 difference to the prior SOTA is tiny (0.960 vs. 0.9610).
minor comments (5)
  1. [Appendix] The LLM prompts contain typos: 'adminstered' and 'CLassify' should be corrected, and the capitalization of category descriptions should be made consistent.
  2. [References] References [19] and [20] are the same n2c2 paper (Henry et al., JAMIA 2020) and should be consolidated into a single citation.
  3. [Section V / Conclusion] The code link in Section VI points to the general MedCAT repository (https://github.com/CogStack/MedCAT) rather than to a specific RelCAT module, tag, or release; the paper states the repository 'contains all implementations necessary to reproduce the results' but gives no version identifier, so the current link does not support reproducibility.
  4. [Tables 13 and 14] Table 13 is captioned 'last layer unfrozen' and Table 14 'last layer frozen,' but Table 14 reports overall F1 and accuracy of 0.977, identical to the all-layers-unfrozen model in Table 9, while Table 13 reports lower values (e.g., ADE-Drug F1 0.77). The relationship among these configurations should be clarified; as printed, the reader cannot tell which configuration produced the headline result.
  5. [Section IV-A] The text says 'With layers frozen, the Llama model achieves the highest overall performance among all models, with an F1-score of 0.929,' but Table 4 lists Llama (last layer unfrozen) at 0.949 and Table 12 reports frozen Llama F1 0.929; the narrative should be reconciled with the tables.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: RelCAT's relation-extraction claims are empirical, benchmarked on the external n2c2 dataset and against prior published systems; the reported macro-F1 discrepancy is a metric-labeling defect, not a derivation that reduces to its own inputs.

full rationale

I walked the paper's claimed derivation chain. There is no formal derivation: the contributions are an annotation workflow, a BERT/Llama relation-classification pipeline, and empirical evaluations on the gold-standard n2c2 dataset and two internally annotated NHS datasets. The classification modelling (entity markers, hidden-state pooling, class weights, stratified batching) is a standard supervised setup in which no parameter is fitted to the reported test metric and no 'prediction' is taken from the same fitted quantity. The n2c2 benchmark is external, and the state-of-the-art comparisons cite Henry et al. and Yang et al., not the authors' own work. The MedCAT and CogStack citations are prior software artifacts that support entity extraction and annotation; they do not force the relation-classification result, and the headline n2c2 numbers are independent of them. The auto-generated NHS 'Other' class is an internal data-construction choice that could make the NHS tasks easier, but it is not circular reasoning and it does not affect the n2c2 evaluation. I did find an internal inconsistency: the abstract and Table 4 report a 'macro F1-score of 0.977', whereas the per-class F1 values in Table 9 (0.993, 0.990, 0.964, 0.975, 0.933, 0.977, 0.979, 0.866) average to about 0.960; the 0.977 figure matches the support-weighted average and accuracy rather than the unweighted macro F1. This is a correctness and reporting concern about the central numerical claim, not a circularity, because it does not involve a fitted parameter renamed as a prediction or a definitional equivalence. No load-bearing self-citation chain, imported uniqueness theorem, or ansatz smuggled in via citation was found. The honest finding is therefore no significant circularity, score 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No physical or theoretical entities are introduced. The main ledger items are hand-set data-generation choices, especially the non-relation cap and TUI filters, plus the assumption that auto-generated negatives are representative.

free parameters (4)
  • Non-relation cap per NHS project = up to 70 per project
    Hand-set limit on generated 'Other' samples; the authors say it was crucial after extensive testing and it directly shapes the NHS F1 scores.
  • Class weights for n2c2 minority classes = not reported
    Used to reweight the loss for ADE-Drug and Duration-Drug; exact weights are not given.
  • Layer freezing regime = all frozen, last layer unfrozen, all unfrozen
    Selected per model after experimentation; it changes reported F1 by several points.
  • Context selection window = not reported
    The model only sees tokens near the entity markers; the window size and maximum inter-entity distance are not specified.
assumptions (4)
  • domain assumption n2c2 gold-standard annotations and its official split are a valid external benchmark.
    The SOTA comparison relies on this; the paper does not fully document the preprocessing that produced the sample counts in Table 1.
  • ad hoc to paper TUI-based automatic generation creates valid non-relation labels.
    Section II-C3 generates 'Other' class by pairing entities whose concept types are not in the target relation and capping at 70 per project, rather than manual annotation of true negatives.
  • domain assumption Manual annotations on the NHS datasets are accurate ground truth.
    No inter-annotator agreement, adjudication, or annotation counts per annotator are reported in Section II-C2.
  • domain assumption MedCAT entity linking supplies correct concept identifiers for the relation pairs.
    The pipeline in Section II-A depends on MedCAT NER+L; errors in entity linking propagate to relation classification, though entities are manually validated in the NHS annotation flow.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RelCAT: Advancing Extraction of Clinical Inter-Entity Relationships from Unstructured Electronic Health Records." pith.science (2026). https://pith.science/paper/KYHNALRG

@misc{pith2026250116077,
  author       = {Pith},
  title        = {Pith review of: RelCAT: Advancing Extraction of Clinical Inter-Entity Relationships from Unstructured Electronic Health Records},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KYHNALRG}},
  note         = {Machine review of arXiv:2501.16077}
}
read the original abstract

This study introduces RelCAT (Relation Concept Annotation Toolkit), an interactive tool, library, and workflow designed to classify relations between entities extracted from clinical narratives. Building upon the CogStack MedCAT framework, RelCAT addresses the challenge of capturing complete clinical relations dispersed within text. The toolkit implements state-of-the-art machine learning models such as BERT and Llama along with proven evaluation and training methods. We demonstrate a dataset annotation tool (built within MedCATTrainer), model training, and evaluate our methodology on both openly available gold-standard and real-world UK National Health Service (NHS) hospital clinical datasets. We perform extensive experimentation and a comparative analysis of the various publicly available models with varied approaches selected for model fine-tuning. Finally, we achieve macro F1-scores of 0.977 on the gold-standard n2c2, surpassing the previous state-of-the-art performance, and achieve performance of >=0.93 F1 on our NHS gathered datasets.

Figures

Figures reproduced from arXiv: 2501.16077 by the authors.

Figure 1
Figure 1. CogStack pipeline • Experimental results comparing LLM-based ap￾proaches for relation extraction including in-context learning. II. METHODOLOGY RelCAT, part of a complete NLP pipeline, uses Med￾CAT, a robust entity identification and linking solution to perform the pre-processing step[3]. The pipeline tasks are: entity extraction using MedCAT (NER+L coupled with meta classification) and relation classification with … view at source ↗
Figure 2
Figure 2. MedCAT NLP framework classes. Stratified batching improves performance by ensuring that each batch contains a balanced representation of all classes, unlike random batch￾ing, which can overrepresent majority classes in imbalanced datasets [12] [13]. B. Large Language Models for in-context learning Large Language Models (LLMs) are built on the trans￾former architecture [9], and have set new benchmarks in natural lang… view at source ↗
Figure 3
Figure 3. RelCAT processing approach output. Using MedCAT’s NER capabilities, RelCAT can perform the entire process from entity recogni￾tion to relation extraction in one seamless operation. • Standalone Relation Classification In this mode, RelCAT functions as an independent module fo￾cused solely on classifying relationships. It requires text with preidentified entities as input, making it suitable for scenarios where entit… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: MedCAT Trainer interface for annotating relations [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 15 canonical work pages

  1. [1]

    Snomed international

    SNOMED, “Snomed international.” Accessed on August 27, 2024

    work page 2024

  2. [2]

    Cogstack-experiences of deploying integrated information retrieval and extraction services in a large national health service foundation trust hospital,

    R. Jackson, I. Kartoglu, C. Stringer, G. Gorrell, A. Roberts, X. Song, H. Wu, A. Agrawal, K. Lui, T. Groza, et al., “Cogstack-experiences of deploying integrated information retrieval and extraction services in a large national health service foundation trust hospital,” BMC medical informatics and decision making , vol. 18, pp. 1–13, 2018

    work page 2018

  3. [3]

    Multi-domain clinical natural language processing with medcat: the medical concept annotation toolkit,

    Z. Kraljevic, T. Searle, A. Shek, L. Roguski, K. Noor, D. Bean, A. Mascio, L. Zhu, A. A. Folarin, A. Roberts, et al. , “Multi-domain clinical natural language processing with medcat: the medical concept annotation toolkit,” Artificial intelligence in medicine , vol. 117, p. 102083, 2021

    work page 2021

  4. [4]

    Named entity recognition and relation detection for biomedical information extraction,

    N. Perera, M. Dehmer, and F. Emmert-Streib, “Named entity recognition and relation detection for biomedical information extraction,” Frontiers in cell and developmental biology , vol. 8, p. 673, 2020

    work page 2020

  5. [5]

    Understanding spatial language in radiology: Representation framework, annotation, and spatial relation extraction from chest x-ray reports using deep learning,

    S. Datta, Y . Si, L. Rodriguez, S. E. Shooshan, D. Demner-Fushman, and K. Roberts, “Understanding spatial language in radiology: Representation framework, annotation, and spatial relation extraction from chest x-ray reports using deep learning,” Journal of biomedical informatics, vol. 108, p. 103473, 2020

    work page 2020

  6. [6]

    The unified medical language system (umls): integrating biomedical terminology.,

    O. Bodenreider, “The unified medical language system (umls): integrating biomedical terminology.,” Nucleic Acids Res. , vol. 32, no. Database-Issue, pp. 267–270, 2004

    work page 2004

  7. [7]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pp. 4171–4186, Association for Computatio...

    work page 2019

  8. [8]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971 , 2023

    arXiv 2023

Show all 22 references
  1. [9]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems , vol. 30, 2017

    2017

  2. [10]

    Addressing the class imbalance problem in medical datasets,

    M. M. Rahman and D. N. Davis, “Addressing the class imbalance problem in medical datasets,” International Journal of Machine Learning and Computing, vol. 3, no. 2, p. 224, 2013

    2013

  3. [11]

    What is the effect of importance weighting in deep learning?,

    J. Byrd and Z. Lipton, “What is the effect of importance weighting in deep learning?,” in International conference on machine learning , pp. 872–881, PMLR, 2019

    2019

  4. [12]

    Imbalanced learning: foundations, algorithms, and applications,

    H. He and Y . Ma, “Imbalanced learning: foundations, algorithms, and applications,” 2013

    2013

  5. [13]

    Linear and stratified sampling-based deep learning models for improving the river streamflow forecasting to mitigate flooding disaster,

    H. A. Afan, A. Yafouz, A. H. Birima, A. N. Ahmed, O. Kisi, B. Chaplot, and A. El-Shafie, “Linear and stratified sampling-based deep learning models for improving the river streamflow forecasting to mitigate flooding disaster,” Natural Hazards, vol. 112, no. 2, pp. 1527– 1545, 2022

    2022

  6. [14]

    MEGA VERSE: Benchmarking large language models across languages, modalities, models and tasks,

    S. Ahuja, D. Aggarwal, V . Gumma, I. Watts, A. Sathe, M. Ochieng, R. Hada, P. Jain, M. Ahmed, K. Bali, and S. Sitaram, “MEGA VERSE: Benchmarking large language models across languages, modalities, models and tasks,” in Proceedings of the 2024 Conference of the North American C...

    2024

  7. [15]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. l. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023

    2023 arXiv

  8. [16]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  9. [17]

    Generalizing from a few examples: A survey on few-shot learning,

    Y . Wang, Q. Yao, J. T. Kwok, and L. M. Ni, “Generalizing from a few examples: A survey on few-shot learning,” ACM computing surveys (csur), vol. 53, no. 3, pp. 1–34, 2020

    2020

  10. [18]

    Making pre-trained language models better few-shot learners,

    T. Gao, A. Fisch, and D. Chen, “Making pre-trained language models better few-shot learners,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long P...

    2021

  11. [19]

    2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records,

    S. Henry, K. Buchan, M. Filannino, A. Stubbs, and O. Uzuner, “2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records,” Journal of the American Medical Informatics Association , vol. 27, no. 1, pp. 3–12, 2020

    2018

  12. [20]

    2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records,

    S. Henry, K. Buchan, M. Filannino, A. Stubbs, and O. Uzuner, “2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records,” Journal of the American Medical Informatics Association , vol. 27, pp. 3–12, 10 2019

    2018

  13. [21]

    Clinical relation extraction using transformer-based models,

    X. Yang, Z. Yu, Y . Guo, J. Bian, and Y . Wu, “Clinical relation extraction using transformer-based models,”arXiv preprint arXiv:2107.08957, 2021

    2021 arXiv

  14. [22]

    Bertnesia: Investigating the capture and forgetting of knowledge in BERT,

    J. Wallat, J. Singh, and A. Anand, “Bertnesia: Investigating the capture and forgetting of knowledge in BERT,” CoRR, vol. abs/2106.02902, 2021

    2021 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.