{"id":"f7e45711-dcae-4742-a292-881d523cf877","arxiv_id":"2501.16077","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"RelCAT adds relation classification to MedCAT and reports F1 scores of 0.977 on n2c2 and above 0.9 on two NHS datasets, though the headline metric appears to be a weighted average mislabeled as macro.","lead":"The paper introduces RelCAT, a toolkit that classifies relationships between medical concepts already identified in clinical notes, and reports high scores on a public benchmark and two hospital datasets. A wide audience might care because extracting relations such as drug dosage or tumor location from free-text records could improve reuse of electronic health records for research and care.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed macro F1 of 0.977 on n2c2 is actually a support-weighted average; per-class F1s in Table 9 average to 0.960, below the cited SOTA.","rationale":"The paper's most load-bearing quantitative assertion is the 0.977 macro F1 that 'surpasses' prior work. This assertion is contradicted by the paper's own Table 9, where the unweighted mean of the per-class F1 scores is approximately 0.960. The 0.977 figure appears to be a support-weighted average or overall accuracy, but it is labeled 'macro F1'. The reader's REJECT verdict is justified; we find no need to change it. We do not take the NHS negative-sampling issue as the primary concern because the n2c2 claim is the abstract's headline and is numerically inconsistent, whereas the NHS claim is more hedged and the negative-sampling procedure is a methodological limitation, not an internal contradiction. The toolkit may still be useful, but the central SOTA claim as stated is not supported by the data presented.","tokens_in":14272,"tokens_out":4558,"duration_ms":35830,"concrete_test":"Recompute the macro F1 from Table 9's per-class F1 values as the unweighted mean over the eight classes. If it yields ~0.960 rather than 0.977, the reported 'macro F1' is mislabeled. Then recompute using class-size weights from Table 1; if the weighted value matches 0.977, the abstract conflates the two metrics. Additionally, verify whether the SOTA comparator in [21] reports macro F1; if so, RelCAT's macro F1 (0.960) does not exceed it.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-1 and Table 9 (Layers unfrozen) report per-class F1 scores for the eight n2c2 relation types. The arithmetic mean of these values is 0.960 (0.993, 0.990, 0.964, 0.975, 0.933, 0.977, 0.979, 0.866). Yet the abstract and Table 4 report a 'macro F1' of 0.977. The 0.977 figure matches the support-weighted average (about 0.976) and the overall accuracy, not the unweighted macro F1. Since the cited prior SOTA of 0.9610 is a macro F1, the claimed improvement does not hold: 0.960 < 0.9610. The headline result therefore rests on a mislabeled metric. This discrepancy is internal and reproducible from the paper's own numbers; it is not a matter of evaluation protocol or dataset split. It undermines the central claim of surpassing SOTA, even if the toolkit itself is functional.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RelCAT, a toolkit and workflow for classifying relations between clinical entities previously extracted by MedCAT. The authors integrate relation annotation into MedCATTrainer, train BERT-family and Llama models using entity markers and context windows, and evaluate on the n2c2 2018 medication-relation dataset and on two small NHS datasets (Spatial and Physiotherapy-Mobility). The headline results are a claimed macro F1 of 0.977 on n2c2, which the authors state surpasses a prior state-of-the-art macro F1 of 0.9610, and F1 scores of 0.90–0.94 on the two NHS datasets.","tokens_in":14451,"tokens_out":10067,"duration_ms":84749,"significance":"The toolkit itself, with its open-source integration into MedCAT, the comparative evaluation of multiple transformer and LLM approaches, and the use of real-world NHS data, is a potentially useful systems contribution to clinical NLP. If the empirical claims were correct, the n2c2 result would be a notable advance. However, the central SOTA claim is contradicted by the paper's own per-class table, and the construction of the NHS negative class raises validity concerns. The paper's value currently rests on the systems contribution rather than on the reported benchmark advantage, and the claims as stated are not reliable.","major_comments":[{"comment":"The headline 'macro F1-score of 0.977' reported in the abstract, Section III-1, Table 4, and Section VI is not the macro average of the per-class F1 scores in Table 9 for the BERT (layers unfrozen) configuration. The eight per-class F1 values (0.993, 0.990, 0.964, 0.975, 0.933, 0.977, 0.979, 0.866) have an arithmetic mean of 0.960. The value 0.977 equals the micro-averaged F1 (which coincides with accuracy in this multi-class setting) and is close to the support-weighted average of the per-class scores. Since the cited prior SOTA of 0.9610 is a macro F1, the claimed improvement does not hold: 0.960 < 0.9610. This internal inconsistency undermines the central claim of the paper and must be corrected by reporting the true macro F1 or by explicitly reframing the comparison as micro/weighted F1.","section":"Section III-1 / Table 9 / Abstract"},{"comment":"The description of non-relation generation is internally inconsistent. The text states 'the number of non-relations were limited to a maximum 70 per project,' but Table 2 reports 332 'Other' instances in the Spatial dataset and Table 3 reports 301 in the Physiotherapy-Mobility dataset. If the cap applies per some other unit (e.g., per document, per TUI pair, or per annotator), that unit must be specified; as written, the negative-sampling protocol cannot be reproduced and the NHS results cannot be interpreted.","section":"Section II-C3 / Tables 2 and 3"},{"comment":"The automatically generated non-relation ('Other') class appears systematically different from the positive class in terms of entity-type (TUI) pairs. For example, in the Spatial dataset the top 'Other' pair is qualifier value–qualifier value (76 instances), whereas the top 'Spatial' pairs are body structure–morphologic abnormality and body structure–disorder. This raises a concrete risk that the classifier learns to predict the relation from the ontological types of the two entities rather than from textual evidence. The paper should provide an analysis showing that the model performs well when the TUI-pair distribution of the negative set is matched to the positive set, or should otherwise moderate the NHS performance claims.","section":"Section II-C3 / Tables 7 and 8"},{"comment":"The n2c2 evaluation does not specify how the 505 discharge summaries were divided into training, development, and test sets, nor whether the test set coincides with the official n2c2 2018 shared-task split. Without this information, the comparison with previously published macro F1 scores is not well-defined. The paper should also report multiple runs or confidence intervals, since the corrected macro F1 difference to the prior SOTA is tiny (0.960 vs. 0.9610).","section":"Section III"}],"minor_comments":[{"comment":"The LLM prompts contain typos: 'adminstered' and 'CLassify' should be corrected, and the capitalization of category descriptions should be made consistent.","section":"Appendix"},{"comment":"References [19] and [20] are the same n2c2 paper (Henry et al., JAMIA 2020) and should be consolidated into a single citation.","section":"References"},{"comment":"The code link in Section VI points to the general MedCAT repository (https://github.com/CogStack/MedCAT) rather than to a specific RelCAT module, tag, or release; the paper states the repository 'contains all implementations necessary to reproduce the results' but gives no version identifier, so the current link does not support reproducibility.","section":"Section V / Conclusion"},{"comment":"Table 13 is captioned 'last layer unfrozen' and Table 14 'last layer frozen,' but Table 14 reports overall F1 and accuracy of 0.977, identical to the all-layers-unfrozen model in Table 9, while Table 13 reports lower values (e.g., ADE-Drug F1 0.77). The relationship among these configurations should be clarified; as printed, the reader cannot tell which configuration produced the headline result.","section":"Tables 13 and 14"},{"comment":"The text says 'With layers frozen, the Llama model achieves the highest overall performance among all models, with an F1-score of 0.929,' but Table 4 lists Llama (last layer unfrozen) at 0.949 and Table 12 reports frozen Llama F1 0.929; the narrative should be reconciled with the tables.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The metric mislabeling is severe and must be fixed before any resubmission. If the corrected macro F1 is indeed below the prior SOTA, the authors should remove the SOTA claim and reposition the paper as a systems contribution. Given the NHS negative-construction issue, I would also require a validation of the automatic non-relation generation. The toolkit itself appears functional, so revision is feasible, but the current claims are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper ships a functional toolkit and two new private NHS relation datasets, but its central headline number is wrong. The abstract claims a macro F1 of 0.977 on n2c2, surpassing a prior SOTA of 0.9610. Their own Table 9 (layers unfrozen) gives per-class F1s of 0.993, 0.990, 0.964, 0.975, 0.933, 0.977, 0.979, 0.866, which average to 0.960. The 0.977 figure matches the overall accuracy / support-weighted average, not a macro F1. So the \"surpassing SOTA\" claim collapses: 0.960 < 0.9610. This isn't a subtle evaluation-protocol issue; it's an internal inconsistency in the paper.\n\nWhat's genuinely new: the integration of relation annotation into MedCATTrainer and the RelCAT workflow built on MedCAT is a legitimate engineering contribution, and the two NHS datasets (spatial and physiotherapy-mobility) are new empirical resources. The special-tokens-plus-entity-hidden-states recipe is standard relation extraction, but the packaging for clinical users is real value. The comparison across BERT, BioBERT, BioClinicalBERT, Llama, and in-context learning is reasonably thorough.\n\nSoft spots beyond the metric: the NHS non-relation class is auto-generated by pairing entities whose type identifiers are not in the target relation, capped at 70 per project. That makes the \"Other\" class easier than real clinical text, so the 0.90-0.93 F1 numbers likely overstate deployable performance. The NHS datasets are also tiny (119 and 486 documents, ~600 relations each), and there are no confidence intervals or significance tests anywhere. The code pointer points to the general MedCAT repo rather than a specific RelCAT release, so exact reproduction is harder than claimed. Minor: the labels on Tables 13 and 14 appear swapped or at least confusing relative to the text.\n\nWho should read this: people building on MedCAT who want relation extraction without reinventing the annotation pipeline, and people who want to see how the n2c2 relation task looks when solved by a straightforward fine-tuned BERT. The paper deserves a serious referee because the toolkit and datasets are useful and the claims are concrete enough to correct, but the authors must fix the metric reporting and add uncertainty quantification before the results are usable.\n\nRecommendation: accept for peer review with heavy revision required. Don't desk reject, but don't let the 0.977 figure stand.","headline":"RelCAT is a useful MedCAT integration with two new NHS datasets, but the headline 0.977 n2c2 macro-F1 is contradicted by the paper's own Table 9, which averages to 0.960, so the SOTA claim does not hold as stated.","tokens_in":15026,"tokens_out":2327,"would_cite":false,"duration_ms":20869,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that RelCAT, a BERT-based relation classifier integrated with MedCAT, achieves macro F1 of 0.977 on the n2c2 clinical relation extraction benchmark — above prior state-of-the-art — and transfers to two real-world NHS…","keywords":["relation extraction","clinical NLP","electronic health records","BERT","SNOMED CT","n2c2 dataset","transfer learning","annotation toolkit"],"falsifier":"Two checks would settle the central claims: (1) recompute the macro F1 for the unfrozen BERT model from the per-class F1 values in Tables 9 and 14, which average to about 0.960, not 0.977, and reproduce 0.977 on the official n2c2 test splits; (2) on the NHS datasets, have clinicians manually annotate a held-out set of true non-relations from the same notes and compare the model's F1 on that set against the F1 computed on the automatically generated 'Other' class. A drop of more than a few points on the manually annotated negatives would indicate the auto-generated negatives inflate the reported performance.","tokens_in":14034,"feed_emoji":"🩺","tokens_out":6067,"duration_ms":51793,"temperature":0.7,"pith_summary":"RelCAT is a workflow for classifying relationships between medical concepts already extracted from clinical notes. The paper's central claim is that a BERT-large classifier that marks entity spans, pools their hidden states, and is fine-tuned with all layers unfrozen achieves a macro F1 of 0.977 on the n2c2 benchmark, surpassing the previous best of 0.961. The same approach transfers to two small NHS datasets — spatial relations and physiotherapy mobility — where fine-tuning from the n2c2 model gives F1 scores of 0.918 and 0.938. The paper also introduces an interactive annotation tool and an automatic way to generate non-relation examples using SNOMED concept-type pairs. If the results hold, clinical information extraction pipelines could reliably pull structured relationships out of free-text records for analytics and decision support.","feed_headline":"RelCAT reports 0.977 F1 on clinical relation extraction","feed_subtitle":"BERT-based toolkit also transfers to NHS notes, reaching 0.93+ F1 on two specialist tasks.","key_machinery":"The load-bearing mechanism is the entity-marker representation: adding [s1], [e1], [s2], and [e2] tokens around the two concepts, taking the hidden states of the entity spans, max-pooling them, and concatenating with BERT's pooled output before a classification head. This lets the model attend to the lexical and positional context between entities despite the many intervening words that separate clinical concepts. Class weights and stratified batching are used to keep minority relation classes from being crushed by the dominant classes in n2c2.","core_discovery":"RelCAT's contribution is framing inter-entity relation classification as a span-pair classification task on transformer models: text is tokenized, the two entities are marked with special tokens or index positions, the model's hidden states for each entity (max-pooled) plus the pooled sequence representation are stacked, and a fully connected head predicts the relation type. On the n2c2 gold standard, the paper reports the unfrozen BERT-large configuration reaching a macro F1 of 0.977, with ADE-Drug and Duration-Drug minority classes at 0.866 and 0.933. On the NHS Spatial and Physiotherapy-Mobility datasets, the best models reach 0.933 and 0.938 F1. The paper further shows that zero-shot and few-shot prompting with Llama and Mistral lags far behind fine-tuning, with best zero/few-shot F1 around 0.49 and 0.46 respectively.","pith_inferences":["The headline macro F1 of 0.977 does not match the macro average of the per-class F1 values in Table 9 (about 0.960), nor Table 14 (about 0.961); the number likely comes from a different run or aggregation, and it should be reconciled before being taken at face value.","The automatic non-relation generation likely makes the NHS evaluation easier than real clinical text with arbitrary entity co-occurrences; a blinded test with manually labeled negatives would give a truer estimate of deployed performance.","Because general BERT-large outperformed BioBERT and BioClinicalBERT when unfrozen, the paper suggests that general pre-training plus task fine-tuning can beat domain-specific pre-training for relation extraction, which runs counter to common practice and could change model selection in clinical NLP.","The toolkit's design — annotation tool, auto-negatives, training pipeline — may be the more durable contribution than any single model, since the same workflow could be applied to other domains where entity pairs and ontologies exist."],"forward_implications":["If the 0.977 F1 is reproducible, clinical relation extraction from discharge summaries is near-solved for the eight n2c2 relation types, enabling downstream pharmacovigilance and medication-safety analytics.","Transferring a model trained on n2c2 to small specialist tasks yields a 15-16% gain over training from scratch, meaning shared general clinical relation models can bootstrap new relation types with modest annotation effort.","The automatic generation of non-relations from SNOMED type pairs allows creation of balanced binary datasets without manual negative annotation, a pattern other annotation pipelines could adopt.","Llama with frozen layers performed competitively on the small NHS datasets (0.933 and 0.835 F1), suggesting large frozen models are a viable alternative when compute for fine-tuning is limited.","The integrated annotation tool and training pipeline give a complete path from raw text to a trained relation model, lowering the barrier for hospitals to create task-specific relation extractors."],"supporting_citations":[{"why":"Supplies the MedCAT NER+L pipeline and annotation infrastructure that RelCAT extends.","marker":"[3]"},{"why":"Provides the BERT-large base model used for the highest-performing relation classifier.","marker":"[7]"},{"why":"Provides the n2c2 gold-standard dataset and the original state-of-the-art baseline (0.956 F1) that the paper aims to surpass.","marker":"[19]"},{"why":"The transformer-based previous state-of-the-art (0.961 F1) that the paper claims to beat.","marker":"[21]"},{"why":"Defines the Llama family used in frozen-layer and in-context learning experiments.","marker":"[8]"}],"fun_headline_variants":["RelCAT hits 0.977 F1 on clinical relation extraction","RelCAT's BERT tops n2c2, transfers to NHS with 0.93+ F1","RelCAT: span-pair classification reaches 0.977 F1","RelCAT surpasses prior SOTA on clinical relation extraction","RelCAT reports 0.977 F1 on n2c2, 0.93 on NHS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The NHS non-relation ('Other') class is generated automatically by pairing entities whose SNOMED concept types do not match the target relation, capped at 70 per project, rather than being annotated by clinicians; if these generated negatives are simpler or less ambiguous than real non-relations, the reported 0.90-0.93 F1 overstates how the model would perform on genuinely unrelated entity pairs.","fun_headline_variants_meta":{"raw":{"variants":["RelCAT hits 0.977 F1 on clinical relation extraction","RelCAT's BERT tops n2c2, transfers to NHS with 0.93+ F1","RelCAT: span-pair classification reaches 0.977 F1","RelCAT surpasses prior SOTA on clinical relation extraction","RelCAT reports 0.977 F1 on n2c2, 0.93 on NHS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00078,"raw_usage":{"total_tokens":3424,"prompt_tokens":903,"completion_tokens":2521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":2414}},"tokens_in":519,"tokens_out":2521,"duration_ms":17466,"temperature":1.0,"reasoning_tokens":2414,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:44:37.798065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Two checks would settle the central claims: (1) recompute the macro F1 for the unfrozen BERT model from the per-class F1 values in Tables 9 and 14, which average to about 0.960, not 0.977, and reproduce 0.977 on the official n2c2 test splits; (2) on the NHS datasets, have clinicians manually annotate a held-out set of true non-relations from the same notes and compare the model's F1 on that set against the F1 computed on the automatically generated 'Other' class. A drop of more than a few points on the manually annotated negatives would indicate the auto-generated negatives inflate the reported performance.","supporting_citations":[{"cited_title":"Multi-domain clinical natural language processing with medcat: the medical concept annotation toolkit,","cited_arxiv_id":null,"evidence_quote":"Supplies the MedCAT NER+L pipeline and annotation infrastructure that RelCAT extends."},{"cited_title":"BERT: Pre-training of deep bidirectional transformers for language understanding,","cited_arxiv_id":null,"evidence_quote":"Provides the BERT-large base model used for the highest-performing relation classifier."},{"cited_title":"2018 n2c2 shared task on adverse drug events and medication extraction in electronic health records,","cited_arxiv_id":null,"evidence_quote":"Provides the n2c2 gold-standard dataset and the original state-of-the-art baseline (0.956 F1) that the paper aims to surpass."}],"review_version":1}