Pith. sign in

REVIEW 4 major objections 5 minor 104 references

EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces EvidenceOutcomes, an open corpus of 640 randomized-trial abstracts with span-level annotations of clinically meaningful outcomes, and shows a PubMedBERT baseline reaching 0.69 entity-level F1.

desk verdict Useful new outcome-annotation corpus, but the reported inter-annotator agreement predates the final annotation pass and doesn't characterize the released labels. read the letter →

arxiv 2506.05380 v1 pith:YGQUPACY submitted 2025-06-03 cs.CL

Yiliang Zhou , Abigail M. Newbury , Gongbo Zhang , Betina Ross Idnay , Hao Liu , Chunhua Weng , Yifan Peng This is my paper
classification cs.CL
keywords PICOextractionoutcomeentityannotationclinicaltrialabstractsnamedrecognitionEvidenceOutcomesrandomizedcontrolledtrialsguidelinePubMedBERT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to fix the weakest link in PICO extraction from clinical-trial literature: the Outcome element, which existing corpora either annotate poorly or drop. It introduces EvidenceOutcomes, an open corpus of 640 randomized-trial abstracts whose Results and Conclusions sections are manually marked up with clinically meaningful outcome entities under a detailed 15-rule guideline. The guideline anchors the definition of an outcome to the trial's planned endpoints, so that only pre-specified measurable results are annotated. Independent annotators reach substantial agreement (average Cohen's kappa 0.748 on the PubMed half, 0.738 on the EBM-NLP half), and a PubMedBERT model fine-tuned on the corpus reaches 0.69 entity-level F1 on a 140-abstract test set. If the corpus holds, it offers a shared benchmark and training resource for outcome extraction that existing PICO resources lack.

What carries the argument

The load-bearing object is the EvidenceOutcomes annotation guideline, a 15-rule protocol for marking outcome entities as contiguous spans that refer to pre-specified, measurable study endpoints and can stand alone outside the abstract. Its distinctive mechanism is cross-referencing: annotators first infer outcomes from the abstract and consult the matching trial-registry record when uncertain, thereby tying ground truth to planned endpoints rather than to any observed result. On top of that, a fine-tuned PubMedBERT model with a partial-parameter-update strategy is the baseline, and the annotations are released in the standard standoff format used by the online annotation tool.

What would settle it

Re-annotate a random sample of abstracts whose registry entries were amended or withdrawn after publication, with annotators blinded to the amendments, and measure agreement with the published labels; a drop in agreement on those trials would show that the reported kappa depends on registry stability.

Watch

Extended reading notes

Core claim

EvidenceOutcomes is a manually annotated corpus of 640 RCT abstracts—500 sampled from the bibliographic database plus 140 drawn from an existing outcome-annotated corpus—in which Outcome entities are labeled only in Results and Conclusions, require that the expression refer to a measurable result specified in the study design, and exclude modifiers, p-values, observed values, and non-standalone phrases. The paper reports substantial inter-annotator agreement (0.748 and 0.738 average Cohen's kappa on the two halves) and shows that a PubMedBERT model fine-tuned on the 500 abstracts reaches 0.69 F1 at the entity level and 0.76 at the token level on the 140-abstract test set. It further reports that combining the older corpus and EvidenceOutcomes training data improves entity-level F1 over training on either corpus alone, indicating the new annotations carry signal not present in the older corpus.

Load-bearing premise

The corpus's quality rests on the premise that a trial's planned endpoints can be recovered from the abstract plus its registry record; when the registry is incomplete, outdated, or worded differently from the abstract, the gold-standard labels become underdetermined and the reported agreement can overstate reliability.

Editorial extensions

If this is right

  • The 640-abstract corpus and its 15-rule guideline give the community a reusable standard for deciding what counts as an outcome in an RCT abstract.
  • A PubMedBERT model fine-tuned on the corpus reaches 0.69 entity-level F1 on the EvidenceOutcomes test labels, so downstream systems have a concrete number to beat.
  • Adding EvidenceOutcomes training data to the older corpus improves entity-level F1 over training on either corpus alone, suggesting the new labels add signal rather than duplicating the old ones.
  • Because only Results and Conclusions sections are labeled, models trained on EvidenceOutcomes learn to ignore Background and Methods text, which should reduce false outcome spans.
  • The annotated entities are stand-alone, pre-specified endpoints; that makes them directly usable for grouping trials that measure the same outcome and for checking whether reported results match core outcome sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's registry-anchored definition means EvidenceOutcomes is really a benchmark for extracting pre-specified endpoints, not a benchmark for recognizing every measured result the abstract reports; a reader wanting the latter should treat the labels as a lower bound on the outcomes discussed.
  • A natural test the authors do not run is to apply the trained model to full-text articles or to abstracts without a registry record; if performance drops there, the corpus would be shown to depend on registry-aligned reporting.
  • Because modifiers such as 'higher' and 'reduced' are deliberately excluded from spans, tools that want to aggregate trials by outcome direction will need a second layer of annotation to recover those modifiers.
  • The improved performance when combining corpora suggests that a merged annotation scheme separating 'planned endpoint' from 'reported result' could push extraction further; such a scheme would go beyond what either guideline currently specifies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces EvidenceOutcomes, a manually annotated corpus of 640 randomized controlled trial abstracts (500 from PubMed and 140 from EBM-NLP) with span-level Outcome annotations. The authors develop an annotation guideline that defines outcomes as metrics specified in the study design, with ClinicalTrials.gov outcome measures as a reference, and release the guideline, the annotations, and fine-tuning code. They report Cohen's kappa around 0.75 and a PubMedBERT baseline with entity-level F1 of 0.69 on the 140 EBM-NLP abstracts re-annotated with EvidenceOutcomes labels.

Significance. EvidenceOutcomes fills a real gap: existing PICO corpora under-annotate Outcome elements, and the paper's guideline is unusually explicit about boundary decisions (modifiers, lists, temporal components, coordination ellipsis). The corpus is open, includes a detailed annotation manual, and is externally anchored to ClinicalTrials.gov, which is a strength over purely text-derived annotation. The baseline experiments report 95% confidence intervals and are reproducible from the released code. If the reliability of the final annotations is established, the corpus would be a useful benchmark for outcome extraction. The main gaps are in the inter-annotator-agreement reporting and in the interpretation of the benchmark comparison.

major comments (4)
  1. [§2.3, §4.1, Table 1; abstract] The abstract claims 'high-quality annotations of an inter-rater agreement of 0.76,' but Table 1 reports averages of 0.748 (360 PubMed abstracts, guideline version 1) and 0.738 (140 EBM-NLP abstracts, version 2). The single number 0.76 matches only the 0.760 cell for one secondary annotator on EBM-NLP and conflates two different annotation subsets and two guideline versions. More importantly, the PubMed kappa was measured on the version-1 annotations of 360 abstracts before the version-2 revision described in §2.3, which changed annotation decisions ('from baseline' inclusion, 'or' as a mathematical symbol, flexibility for 'increase'/'improvement'). The final released PubMed corpus is the 500 version-2 abstracts, but no agreement statistic is reported for those labels, and the paper does not state whether the version-2 revision was performed by the primary annotator alone or adjudicated. Please report agreement on the final labels (at least on a representative sample), specify the annotation unit and matching criterion for the kappa computation, and revise the abstract to state the subset and guideline version to which the kappa applies.
  2. [§4.1] The kappa computation is underspecified: the paper does not state whether agreement was computed at token level or entity level, whether spans were required to match exactly or partially, whether only the Results and Conclusions sections were considered in the agreement calculation, or how entity categories (only 'Outcome' is used) enter the kappa formula. Without this information, the reported kappa values are not interpretable or comparable to prior PICO annotation studies (e.g., EBM-NLP). Please provide the exact calculation procedure, including the definition of a positive/negative unit.
  3. [§5.1, §5.2, Tables 3-6] The paper interprets the higher F1 on EvidenceOutcomes labels (0.69 entity F1) versus EBM-NLP labels (0.26) as evidence that 'the more comprehensive and detailed outcomes annotation guidelines' are better. This is an over-interpretation: the model was fine-tuned on EvidenceOutcomes labels, so the evaluation on EvidenceOutcomes labels measures how well the model learns that particular annotation scheme, not the clinical quality of the scheme. The comparison with EBM-NLP labels is confounded by the fact that the model has seen many EvidenceOutcomes examples and no EBM-NLP examples during fine-tuning. Reframe the benchmark as a demonstration that the corpus is learnable, or provide a controlled comparison (e.g., train on EBM-NLP labels and test on both schemes, or use an external human judgment task) before claiming guideline superiority.
  4. [§2.1 and §3 of the supplementary guideline; §5.3] The operational definition of a 'clinically meaningful outcome' relies on the premise that the study design is recoverable from the abstract plus ClinicalTrials.gov Primary and Secondary Outcome Measures. The guideline correctly instructs annotators to consult the registry only when the abstract is unclear, but the paper does not discuss what happens when registry entries are incomplete, outdated, or worded differently from the abstract. This dependency is load-bearing for the corpus-quality claim, yet it is absent from the Limitations section (§5.3), which only mentions distribution normalization and EBM-NLP scheme differences. Please add a discussion of this dependency and, if feasible, a sensitivity analysis (e.g., how often annotators consulted the registry and whether kappa differs on abstracts with complete versus incomplete registry entries).
minor comments (5)
  1. [Abstract] The phrase 'fine-tuned PubMedBERT model, applied to these 500 PubMed abstracts' should read 'trained on the 500 PubMed abstracts and evaluated on the 140 EBM-NLP abstracts'; as written it states the model was applied to the training set.
  2. [§4.2, Figure 6] Define what 'overlapping outcomes' means (exact string match, normalized text, or entity-level agreement between the two annotation schemes); the current description leaves the overlap computation unclear.
  3. [§3] There is a typo: 'MIT licencse' should be 'MIT license.'
  4. [§2.3] The sentence 'An additional 140 abstracts from the EBM-NLP corpus were also randomly selected and the outcome elements were reannotated' is slightly ambiguous; specify that these abstracts were newly annotated under version 2 rather than only 'reannotated.'
  5. [§5.1] Please report the variance of the reported F1 scores over different random seeds; the paper specifies 10 epochs and the lowest-validation-loss checkpoint, but not seed variability, which would help assess the stability of the confidence intervals.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: EvidenceOutcomes is a self-contained annotated corpus whose quality is measured by held-out inter-annotator agreement and a held-out NER benchmark.

full rationale

The paper's central product is a manually annotated corpus. Outcomes are operationalized by study design and cross-checked against ClinicalTrials.gov registry entries, which is an external reference rather than a tautology: the guideline is not derived from the labels it is later used to produce, and the corpus's existence does not depend on the model. The inter-annotator agreement (average kappa 0.748 on PubMed v1 and 0.738 on EBM-NLP v2) is computed on independent annotators' pre-consensus labels, and the PubMedBERT baseline is evaluated on a held-out 140-abstract subset while trained on a disjoint 500-abstract set, so the reported F1 scores are genuine held-out measurements rather than refits or renamed training performance. The visible concerns in the text—that the released PubMed annotations were revised under guideline v2 after the reported kappa was measured, and that comparing F1 across label schemes is confounded by training/test scheme match—are validity and interpretation issues, not circular derivations; they do not reduce any claimed result to its own inputs. Self-citations to prior work by the same group are used only for disease-domain categorization and for baseline model-selection motivation, neither of which is load-bearing for the corpus quality claim.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central contribution is a corpus rather than a derivation; no fitted physical parameters or invented entities are introduced. The key assumption is that registry-anchored outcome measures, read together with Results and Conclusions text, define the ground truth for clinically meaningful outcomes.

free parameters (1)
  • PubMedBERT fine-tuning hyperparameters (learning rate 3e-5, batch size 8, 10 epochs)
    Chosen by hand for the baseline benchmark; they affect the reported F1 scores but not the construction of the corpus itself.
assumptions (3)
  • domain assumption Outcome entities are fully identifiable from Results and Conclusions sections of abstracts together with ClinicalTrials.gov Primary and Secondary Outcome Measures.
    Annotation guideline Section 3 instructs annotators to infer outcomes from abstract semantics and consult the registry file only when unclear; if registry entries are incomplete, the ground truth is underdetermined.
  • standard math Cohen's kappa computed on annotator pairs over non-overlapping abstract subsets is a valid measure of corpus-wide inter-annotator agreement.
    Section 4.1 reports kappa from pairwise comparisons on 360 PubMed abstracts and 140 EBM-NLP abstracts, each secondary annotator covering a subset; no full triple-annotation reliability is provided.
  • ad hoc to paper The version 2 guideline rules, including the 'from baseline' restriction and 'or as mathematical symbol' exception, capture clinically meaningful outcome boundaries.
    These rules were introduced to resolve disagreements during annotation and define the corpus's notion of outcome; they are dataset-specific design choices.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes." pith.science (2026). https://pith.science/paper/YGQUPACY

@misc{pith2026250605380,
  author       = {Pith},
  title        = {Pith review of: EvidenceOutcomes: a Dataset of Clinical Trial Publications with Clinically Meaningful Outcomes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGQUPACY}},
  note         = {Machine review of arXiv:2506.05380}
}
read the original abstract

The fundamental process of evidence extraction and synthesis in evidence-based medicine involves extracting PICO (Population, Intervention, Comparison, and Outcome) elements from biomedical literature. However, Outcomes, being the most complex elements, are often neglected or oversimplified in existing benchmarks. To address this issue, we present EvidenceOutcomes, a novel, large, annotated corpus of clinically meaningful outcomes extracted from biomedical literature. We first developed a robust annotation guideline for extracting clinically meaningful outcomes from text through iteration and discussion with clinicians and Natural Language Processing experts. Then, three independent annotators annotated the Results and Conclusions sections of a randomly selected sample of 500 PubMed abstracts and 140 PubMed abstracts from the existing EBM-NLP corpus. This resulted in EvidenceOutcomes with high-quality annotations of an inter-rater agreement of 0.76. Additionally, our fine-tuned PubMedBERT model, applied to these 500 PubMed abstracts, achieved an F1-score of 0.69 at the entity level and 0.76 at the token level on the subset of 140 PubMed abstracts from the EBM-NLP corpus. EvidenceOutcomes can serve as a shared benchmark to develop and test future machine learning algorithms to extract clinically meaningful outcomes from biomedical abstracts.

Figures

Figures reproduced from arXiv: 2506.05380 by the authors.

Figure 2
Figure 2. Data collection procedure released under MIT licencse. 2.2 Data collection strategy A total number of 500 RCT abstracts were collected from PubMed and 140 abstracts from the EBM￾NLP corpus [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Workflow of the annotation process annotators compared, discussed the results, and refined the guidelines further. Once the 132 abstracts were annotated, the guideline was finalized and dubbed "version 1". To ensure all annotators fully understood this version, an additional eight abstracts were annotated. Subsequently, the remaining 360 abstracts were annotated in four separate batches using the established “versio… view at source ↗
Figure 4
Figure 4. Statistics of EvidenceOutcomes. (a) Sentences per abstract. (b) Words per sentence. (c) Outcomes per sentence Cardiology/Vascular Diseases Dental and Oral Health Dermatology Endocrinology Gastroenterology Genetic Disease Healthy Hematology Hepatology Immunology Infections and Infectious Diseases Metabolism and Nutrition Miscellanea Neonatology Nephrology Neurology Obstetrics/Gynecology Oncology Ophthalmology Orthope… view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Disease domains represented in EvidenceOutcomes. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Statistics of outcomes per abstract in EvidenceOutcomes and EBM-NLP. Outcomes, and 695 unique to EBM-NLP. This difference highlighted that EvidenceOutcomes employs a distinct set of annotation guidelines compared to EBM-NLP. For example, in the abstract of PMID 1939750…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

104 extracted references · 75 canonical work pages

  1. [1]

    The well-built clinical question: a key to evidence-based decisions.ACP J

    W S Richardson, M C Wilson, J Nishikawa, and R S Hayward. The well-built clinical question: a key to evidence-based decisions.ACP J. Club, 123(3):A12–3, 1995. ISSN 1056-8751

    1995

  2. [2]

    Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements.J

    Tove Faber Frandsen, Michael Friberg Bruun Nielsen, Christina Louise Lindhardt, and Mette Brandt Eriksen. Using the full PICO model as a search tool for systematic reviews resulted in lower recall for some PICO elements.J. Clin. Epidemiol., 127:69–75, November 2020. ISSN 0895-4356, 1878-5921. doi: 10.1016/j.jclinepi.2020.07.005

    work page doi:10.1016/j.jclinepi.2020.07.005 2020

  3. [3]

    Improving reference priori- tisation with PICO recognition.BMC Med

    Austin J Brockmeier, Meizhi Ju, Piotr Przybyła, and Sophia Ananiadou. Improving reference priori- tisation with PICO recognition.BMC Med. Inform. Decis. Mak., 19(1):256, December 2019. ISSN 1472-6947. doi: 10.1186/s12911-019-0992-8

    work page doi:10.1186/s12911-019-0992-8 2019

  4. [4]

    PICO entity extraction for preclinical animal literature.Syst

    Qianying Wang, Jing Liao, Mirella Lapata, and Malcolm Macleod. PICO entity extraction for preclinical animal literature.Syst. Rev., 11(1):209, September 2022. ISSN 2046-4053. doi: 10.1186/s13643-022-02074-4

    work page doi:10.1186/s13643-022-02074-4 2022

  5. [5]

    Pretraining to recognize pico elements from randomized controlled trial literature.Studies in health technology and informatics, 264:188, 2019

    Tian Kang, Shirui Zou, and Chunhua Weng. Pretraining to recognize pico elements from randomized controlled trial literature.Studies in health technology and informatics, 264:188, 2019

    2019

  6. [6]

    A span-based model for extracting overlapping PICO entities from randomized controlled trial publications.J

    Gongbo Zhang, Yiliang Zhou, Yan Hu, Hua Xu, Chunhua Weng, and Yifan Peng. A span-based model for extracting overlapping PICO entities from randomized controlled trial publications.J. Am. Medical Informatics Assoc., 31(5):1163–1171, 2024. doi: 10.1093/JAMIA/OCAE065. URL https://doi.org/10.1093/jamia/ocae065

    work page doi:10.1093/jamia/ocae065 2024

  7. [7]

    Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain J Marshall, Ani Nenkova, and Byron C Wallace. A corpus with Multi-Level annotations of patients, interventions and outcomes to support language processing for medical literature.Proc Conf Assoc Comput Linguist Meet, 2018:197–207, July 2018. ISSN 0736-587X. 10

    2018

  8. [8]

    Medical entity corpus with PICO elements and sentiment analysis

    Markus Zlabinger, Linda Andersson, Allan Hanbury, Michael Andersson, Vanessa Quasnik, and Jon Brassey. Medical entity corpus with PICO elements and sentiment analysis. InProceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 292– 296, Miyazaki, Japan, May 2018. European Language Resources Association (ELRA)

    2018

Show all 104 references
  1. [9]

    Recommendations for kidney disease guideline updating: a report by the KDIGO methods committee

    Katrin Uhlig, Jeffrey S Berns, Serena Carville, Wiley Chan, Michael Cheung, Gordon H Guyatt, Allyson Hart, Sandra Zelman Lewis, Marcello Tonelli, Angela C Webster, Timothy J Wilt, and Bertram L Kasiske. Recommendations for kidney disease guideline updating: a report by the KDI...

    2016 doi

  2. [10]

    Patient-Centered outcomes research institute (PCORI)

    PCORI. Patient-Centered outcomes research institute (PCORI). https://www.pcori.org/ about/about-pcori, October 2021. Accessed: 2023-8-2

    2021

  3. [11]

    Core outcome measures in effectiveness trials

    COMET Initiative. Core outcome measures in effectiveness trials. https://www. comet-initiative.org/, 2023. Accessed: 2023-8-2

    2023

  4. [12]

    Knowledge extraction for clinical question answering: Prelimi- nary results

    Dina Demner-Fushman and Jimmy Lin. Knowledge extraction for clinical question answering: Prelimi- nary results. InProceedings of the AAAI-05 Workshop on Question Answering in Restricted Domains, pages 9–13. AAAI Press (American Association for Artificial Intelligence) Pittsbur...

    2005

  5. [13]

    Combin- ing classifiers for robust PICO element detection.BMC Med

    Florian Boudin, Jian-Yun Nie, Joan C Bartlett, Roland Grad, Pierre Pluye, and Martin Dawes. Combin- ing classifiers for robust PICO element detection.BMC Med. Inform. Decis. Mak., 10:29, May 2010. ISSN 1472-6947. doi: 10.1186/1472-6947-10-29

    2010 doi

  6. [14]

    Efficacy and safety of evolocumab in individuals with type 2 diabetes mellitus: primary results of the randomised con- trolled BANTING study.Diabetologia, 62(6):948–958, June 2019

    Robert S Rosenson, Martha L Daviglus, Yehuda Handelsman, Paolo Pozzilli, Harold Bays, Maria Laura Monsalvo, Mary Elliott-Davey, Ransi Somaratne, and Peter Reaven. Efficacy and safety of evolocumab in individuals with type 2 diabetes mellitus: primary results of the randomised ...

    2019 doi

  7. [15]

    An annotated corpus of clinical trial publications supporting schema-based relational information extraction.J

    Olivia Sanchez-Graillet, Christian Witte, Frank Grimm, and Philipp Cimiano. An annotated corpus of clinical trial publications supporting schema-based relational information extraction.J. Biomed. Semantics, 13(1):14, May 2022. ISSN 2041-1480. doi: 10.1186/s13326-022-00271-7

    2022 doi

  8. [16]

    A general introduction to the e-utilities

    Eric Sayers. A general introduction to the e-utilities. https://www.ncbi.nlm.nih.gov/ books/NBK25497/, November 2022. Accessed: 2023-5-11

    2022

  9. [17]

    Interrater reliability: the kappa statistic.Biochem

    Mary L McHugh. Interrater reliability: the kappa statistic.Biochem. Med., 22(3):276–282, 2012. ISSN 1330-0962. doi: 10.1016/j.jocd.2012.03.005

    2012 doi

  10. [18]

    Evaluating discourse and dialogue coding schemes.Comput

    Richard Craggs and Mary Mcgee Wood. Evaluating discourse and dialogue coding schemes.Comput. Linguist. Assoc. Comput. Linguist., 31(3):289–296, September 2005. ISSN 0891-2017, 1530-9312. doi: 10.1162/089120105774321109

    2005 doi

  11. [19]

    Ontology- based categorization of clinical studies by their conditions.J

    Hao Liu, Simona Carini, Zhehuan Chen, Spencer Phillips Hey, Ida Sim, and Chunhua Weng. Ontology- based categorization of clinical studies by their conditions.J. Biomed. Inform., 135:104235, November

  12. [20]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...

    2019

  13. [21]

    Transfer learning in biomedical natural language process- ing: An evaluation of BERT and ELMo on ten benchmarking datasets

    Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language process- ing: An evaluation of BERT and ELMo on ten benchmarking datasets. InProceedings of the 18th BioNLP Workshop and Shared Task, pages 58–65, Stroudsburg, PA, USA, August 2019. Assoc...

    2019 doi

  14. [22]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining

    Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020

    2020

  15. [23]

    Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets

    Yifan Peng, Shankai Yan, and Zhiyong Lu. Transfer learning in biomedical natural language processing: An evaluation of bert and elmo on ten benchmarking datasets. InProceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pages 58–65, 2019

    2019

  16. [24]

    Domain-Specific language model pretraining for biomedical natural language processing.ACM Trans

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-Specific language model pretraining for biomedical natural language processing.ACM Trans. Comput. Healthcare, 3(1):1–23, October 2021. ISSN 2691-1...

    2021 doi

  17. [25]

    Bioelectra: pretrained biomedical text encoder using discriminators

    Kamal Raj Kanakarajan, Bhuvana Kundumani, and Malaikannan Sankarasubbu. Bioelectra: pretrained biomedical text encoder using discriminators. InProceedings of the 20th workshop on biomedical language processing, pages 143–154, 2021

    2021

  18. [26]

    Raise a child in large language model: Towards effective and generalizable fine-tuning

    Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. Raise a child in large language model: Towards effective and generalizable fine-tuning. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages...

    2021 doi

  19. [27]

    What metrics are used to test the study hypothesis?

    seqeval. seqeval: A python framework for sequence labeling evaluation(named-entity recognition, pos tagging, etc.).https://github.com/chakki-works/seqeval, 2023. Accessed: 2023-8-2. 12 Outcome Entity Extraction Annotation Guideline Abigail M. Newbury1, Betina Ross Idnay1, Hao ...

    2023

  20. [28]

    Change From Baseline in Basophils, Eosinophils, Lymphocytes, Monocytes, Total Neutrophils, Platelet and White Blood Cell (NBC) Count

  21. [29]

    Change From Baseline in Hemoglobin Level

  22. [30]

    The temporal component is explicitly stated by the study design. This means that the temporal component is either, i) stated as part of the defined outcome in sections prior to the results and conclusions sections in the abstract or ii) stated as part of the defined outcome in...

  23. [31]

    Typically, adjectival use of temporal components indicates that the inclusion of the temporal component adds clinical significance and meaning to the outcome

    The temporal component is used as an adjective preceding the rest of the outcome entity. Typically, adjectival use of temporal components indicates that the inclusion of the temporal component adds clinical significance and meaning to the outcome. An example of this is as foll...

  24. [32]

    Mean Change From Baseline in Plasma Human Immunodeficiency Virus Type 1 (HIV-1) Ribonucleic Acid (RNA) at Day 8 Secondary Outcome Measures:

  25. [33]

    Absolute Values in Plasma HIV-1 RNA Over Time

  26. [34]

    Mean Change From Baseline in Plasma HIV-1 RNA Over Time

  27. [35]

    Number of Participants With Plasma HIV-1 RNA <50 c/mL Over Time

  28. [36]

    Number of Participants With Plasma HIV-1 RNA <400 c/mL Over Time

  29. [37]

    Absolute Values in Cluster of Differentiation 4+ (CD4+) Cell Counts Over Time

  30. [38]

    Median Change From Baseline in CD4+ Cell Counts Over Time

  31. [39]

    Absolute Values in Cluster of Differentiation 8+ (CD8+) Cell Counts Over Time

  32. [40]

    Median Change From Baseline in CD8+ Cell Counts Over Time

  33. [41]

    Number of Participants With the Indicated Type of HIV-1 Disease Progression (Acquired Immunodeficiency Syndrome [AIDS] or Death [DM]

  34. [42]

    Number of Participants With Any Adverse Event (Serious and Non-serious) of the Indicated Grade

  35. [43]

    Number of Participants With the Maximum Post-Baseline-emergent Clinical Chemistry Toxicities of the Indicated Grade

  36. [44]

    Number of Participants With the Maximum Post-Baseline-emergent Hematology Toxicities of the Indicated Grade

  37. [45]

    Plasma DTG Pre-dose Concentration (CO) at Day 8, Day 28, and Week 24; and Average DTG CO (CO An) at Week 24

  38. [46]

    Number of Participants With the Indicated Treatment-emergent Integrase (IN) Mutations Detected at the Time of Defined Virologic Failure (PDVF), as a Measure of Genotypic Resistance

  39. [47]

    PDVF FC/Baseline FC Ratio) at the Time of PDVF, as a Measure of Phenotypic Resistance

    Number of Participants With the Indicated Fold Increase in Fold Change (FC) in the 50% Inhibitory Concentration Relative to Wild-type Virus for DTG (i.e. PDVF FC/Baseline FC Ratio) at the Time of PDVF, as a Measure of Phenotypic Resistance

  40. [48]

    Number of Participants Who Discontinued Study Treatment Due to AEs

  41. [49]

    Number of Participants With Clinically Significant Electrocardiogram (ECG) Findings

  42. [50]

    Change From Baseline in Systolic Blood Pressure (SBP) and Diastolic Blood Pressure (DBP)

  43. [51]

    Change From Baseline in Heart Rate

  44. [52]

    Change From Baseline in Albumin Level

  45. [53]

    Change From Baseline in Alkaline Phosphatase (ALP), Alanine Aminotransferase (ALT), Aspartate Aminotransferase (AST) and Creatine Kinase

  46. [54]

    11,4) and Creatinine Levels

    Change From Baseline in Total Bilirubin (T. 11,4) and Creatinine Levels

  47. [55]

    Change From Baseline in Cholesterol, Chloride, Carbon Dioxide (CO2)/Bicarbonate (HCO3), Glucose, High Density Lipoprotein Cholesterol, Potassium, Low Density Lipoprotein (LDL) Cholesterol, Sodium, Phosphorus, Triglycerides and Urea/Blood Urea Nitrogen

  48. [56]

    Change From Baseline in Creatinine Clearance

  49. [57]

    Change From Baseline in Lipase Levels

  50. [60]

    Change From Baseline in Hematocrit Level

  51. [61]

    Change From Baseline in Mean Corpuscle V olume

  52. [62]

    Line 12 consists of the outcome entity ‘adjusted mean change in HIV-1 RNA at day 8’

    Change From Baseline in Red Blood Cell Count 5/9 Lines 10 and 11 provide baseline characteristics about the study population and thus nothing in these sentences is highlighted as an outcome entity. Line 12 consists of the outcome entity ‘adjusted mean change in HIV-1 RNA at da...

  53. [63]

    Absolute Change From Baseline in Hemoglobin (Hgb) to End of Treatment (Week 6) Secondary Outcome Measures:

  54. [64]

    Change From Baseline in HO at Week 1, Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  55. [65]

    Change From Baseline in Hematocrit (HCT) at Week 1, Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  56. [66]

    Change From Baseline in Red Blood Cell (RBC) Count at Week 1, Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  57. [67]

    Change From Baseline in Absolute Reticulocyte Count at Week 1, Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  58. [68]

    Change From Baseline in Reticulocyte Hgb Content at Week 6

  59. [69]

    Maximum Change From Baseline in 110

  60. [70]

    Maximum Change From Baseline in HCT

  61. [71]

    Maximum Change From Baseline in RBC Count

  62. [72]

    Maximum Change in Absolute Reticulocyte Count From Baseline

  63. [73]

    Number of Participants With Absolute Change From Baseline in Hgb a 0.4, 0.6, 0.8, and 1.0 g/dL at the End of Dosing Period

  64. [74]

    Number of Participants With Change From Baseline in Ho a5.0, 7.5, and 10.0% by the End of Dosing Period

  65. [75]

    Number of Participants With Change From Baseline in HCT a5.0, 7.5, and 10.0% by the End of Dosing Period

  66. [76]

    Number of Participants With Change From Baseline in RBC Count a5.0, 7.5, and 10.0% by the End of Dosing Period

  67. [77]

    Number of Participants With Change From Baseline in Reticulocyte Count a6000, 12000, and 18000 Cells/uL by the End of Dosing Period

  68. [78]

    Change From Baseline in Total Iron at Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  69. [79]

    Change From Baseline in Unsaturated Iron Binding Capacity at Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  70. [80]

    Change From Baseline in Iron Saturation at Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  71. [81]

    Change From Baseline in Total Iron Binding Capacity (TIBC) at Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  72. [82]

    Change From Baseline in Ferritin at Week 2, Week 4, Week 6, and Follow-up Visit (up to Week 8)

  73. [83]

    Change From Baseline in Erythropoietin at Week 2, Week 6, and Follow-up Visit (up to Week 8)

  74. [84]

    Change From Baseline in tieRciain at Week 6

  75. [85]

    Mean Plasma Vadadustat Concentrations on Week 2 and Week 4

  76. [86]

    Mean Plasma Vadadustat Acyl-Glucuronide Concentrations on Week 2 and Week 4

  77. [87]

    Number of Participants Treatment-emergent Adverse Events (TEAEs) and Serious Adverse Events (SAEs)

  78. [88]

    Number of Participants With Clinically Significant Changes From Baseline in Vital Signs Parameter

  79. [89]

    Number of Participants With Clinically Significant Abnormal 12-Electrocardiogram (ECG) Findings

  80. [90]

    Mean Change From Baseline in PR Interval, QT Interval, QRS Interval, and QT Corrected (OTC) Interval

  81. [91]

    In sentence 14, ‘Hb after 6 weeks’ is highlighted as the outcome entity

    Number of Participants With Clinically Significant Changes From Baseline in Laboratory Parameter Values Sentence 13 specifies details of the trial procedure and thus nothing is highlighted as an outcome. In sentence 14, ‘Hb after 6 weeks’ is highlighted as the outcome entity. ...

    2019

  82. [92]

    Time to Improvement in Coronavirus Disease 2019 (COVID-19) Clinical Status

    2019

  83. [93]

    Number of Participants Who Experience an Adverse Event (AE) Secondary Outcome Measures:

  84. [94]

    Percentage of Participants Who Died or Had Respiratory Failure (RF)

  85. [95]

    Time to Disease Progression in Clinical Status of COVID-19

  86. [96]

    Number of Participants Who Died Due to Any Cause

  87. [97]

    Rate of Clinical Relapse

  88. [98]

    Conversion Rate of COVID-19 Clinical Status

  89. [99]

    Time to Hospital Discharge

  90. [100]

    Duration of Pressors

  91. [101]

    Duration of High Flow Oxygen Therapy

  92. [102]

    Length of Hospital Stay

  93. [103]

    Change From Baseline in Absolute Lymphocyte Count

  94. [104]

    In line 18, ‘28-day clinical improvement rate’ and ‘median time to clinical improvement’ are highlighted as outcomes, as indicated by line 10

    Change From Baseline in D—Dimer Concentration Lines 16 and 17 describe the trial design. In line 18, ‘28-day clinical improvement rate’ and ‘median time to clinical improvement’ are highlighted as outcomes, as indicated by line 10. Note here that ‘28-day’ is an adjective to ‘c...

  95. [2019]

    doi: 10.18653/v1/N19-1423

    Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. 11

    doi

  96. [2022]

    doi: 10.1016/j.jbi.2022.104235

    ISSN 1532-0464, 1532-0480. doi: 10.1016/j.jbi.2022.104235

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.