Pith. sign in

REVIEW 2 major objections 4 minor 53 references

Consensus Measures for Unstructured Biomedical Text Annotations

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For open-ended biomedical annotations, inter-rater reliability can be recovered by replacing exact string matches with a semantic-equivalence estimate; NLI mutual entailment is the best practical compromise.

desk verdict A solid empirical comparison of soft IRR measures with a convincing NLI recommendation, but the real-world ground truth is a proxy that deserves scrutiny before the ranking is treated as settled. read the letter →

arxiv 2608.03529 v1 pith:XNPBSQXD submitted 2026-08-04 cs.CL

Pascal Wullschleger , Christian Kreis , Martin A. Walter , Marc Pouly , Jennifer Foster This is my paper · ORCID
classification cs.CL
keywords softinter-raterreliabilitysemanticequivalencenaturallanguageinferencebiomedicaltextannotationembeddingsimilarityLLM-as-a-judgeCohen'skappafree-textannotations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles a blind spot in biomedical literature mining: when annotators label text with open-ended descriptions instead of predefined codes, standard inter-rater reliability (IRR) collapses because exact string matches are rare. It claims that IRR can be recovered by replacing exact label agreement with a continuous semantic-equivalence estimate, and systematically benchmarks equivalence measures—token overlap, edit distance, sentence embeddings, natural language inference (NLI), and LLM-as-a-judge—against ground-truth agreement computed on standardised terminology labels. In synthetic synonym-replacement experiments and two real-world datasets (Derm1M captions, REFLACX radiology reports), the best deep-learning measures estimate Cohen's kappa to within 0.005-0.065 mean absolute error. The paper's key finding is that the choice of measure changes both the estimate and its failure mode: embeddings are fast but treat similar-but-distinct concepts as equivalent, LLMs are most accurate but too slow for estimating chance agreement, and NLI mutual entailment is the recommended compromise.

What carries the argument

The load-bearing identity is soft Cohen's kappa: kappa_s = (p_s_o - p_s_e)/(1 - p_s_e), where p_s_o is the mean pairwise semantic equivalence and p_s_e is the expected equivalence under a permuted distribution of the second annotator's labels. Equivalence itself is estimated by mutual entailment, P(a≡b) = sqrt(P(a⇒b) * P(b⇒a)), computed by an NLI model or an LLM judge, with embeddings and edit distances serving as alternative estimators. The set-valued extension solves a weighted bipartite assignment with the Hungarian algorithm before averaging the matched equivalences.

What would settle it

Take a corpus of free-text biomedical annotation pairs and have human experts judge semantic equivalence directly; compute soft IRR with the paper's NLI-based measure against human-derived agreement. If NLI-based estimates no longer match human agreement better than embedding or edit-distance baselines—or if the error is dominated by cases where the terminology mapping disagrees with human equivalence judgements—the central recommendation fails.

Watch

Extended reading notes

Core claim

The paper establishes soft IRR for unstructured text annotations: define the observed agreement as the mean semantic equivalence between paired annotations, estimate chance agreement from the expected equivalence between randomly permuted annotation pairs, and plug both into Cohen's kappa. Equivalence is estimated as the probability of mutual entailment—the geometric mean of NLI entailment probabilities in both directions—for NLI and LLM variants, as cosine similarity for embeddings, and as string-based scores for edit and set-overlap measures. For set-valued annotations, pairwise equivalences are first matched with the Hungarian algorithm. Evaluated against exact-match IRR on ICD-11, MeSH a

Load-bearing premise

The evaluation assumes that the standardised labels used as ground truth—exact concept identities in synthetic data, terminology mappings in Derm1M, and the closed anomaly set in REFLACX—capture exactly the semantic agreement between the free-text annotations.

Editorial extensions

If this is right

  • Soft IRR with semantic equivalence recovers ground-truth agreement on standardised labels to within 0.005-0.065 MAE on real-world datasets, accurate enough for routine use in annotation studies.
  • Embedding-based equivalence measures should be avoided for biomedical annotation agreement because they conflate similarity with equivalence and fail on close, non-equivalent concepts.
  • NLI-based measures (PubMedBERT-NLI, mDeBERTa-v3) are the recommended compromise: decent scalability and stable performance across random, equality, generalisation and lexical negative types.
  • LLM-based measures are the most accurate but their computational cost makes Monte Carlo estimation of chance agreement impractical at scale.
  • All soft measures degrade less steeply than exact matching as true agreement rises, so exact matching systematically underestimates reliability when annotators use synonymous wording.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same equivalence-based IRR machinery could be applied to evaluating human-versus-LLM extraction agreement in systematic-review mining, where the goal is matching underlying information rather than exact codes.
  • A testable extension: replace terminology-mapping ground truths with direct human judgements of pairwise semantic equivalence of free-text annotations; if NLI-based estimates still track human agreement best, the recommendation holds regardless of mapping noise.
  • Because embeddings fail on near-synonym negatives, their use in any equivalence-based metric should be gated behind calibration or thresholding before being fed into agreement statistics.
  • The equivalence estimates are measure-agnostic, so the same soft IRR framework transfers to other agreement statistics such as Krippendorff's alpha, which the paper notes in passing; the ranking of measures would likely transfer as well.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes and evaluates soft inter-rater reliability (IRR) measures for unstructured biomedical text annotations. It replaces exact label identity in Cohen's κ, Fleiss' κ, and Scott's π with continuous semantic-equivalence estimates derived from set overlap, edit distance, sentence embeddings, mutual-entailment NLI, and LLM-as-a-judge (both log-prob and verbalised variants). The methods are tested on synthetic terminology data (ICD-11, MeSH, MedDRA) with controlled synonym replacement and negative sampling, and on two real-world datasets (Derm1M and REFLACX) where free-text annotations are compared against IRR computed from standardized labels. The paper reports MAE against ground-truth IRR and concludes that the choice of equivalence measure has a significant impact, that embedding-based measures fail on semantically close negatives, and that NLI-based measures are the best compromise for biomedical annotation tasks.

Significance. The question is timely for exploratory biomedical annotation and literature mining, where open-ended text annotations are common but standard IRR tools assume discrete categories. The paper is one of the first to systematically compare NLI-based and LLM-based equivalence estimators for soft IRR, and the synthetic setup with known synonyms and controlled negative types is a useful contribution. The use of open-weights models, the computational cost table, and paired permutation tests strengthen reproducibility. If the empirical ranking is valid, the recommendation to prefer NLI-based measures and avoid embedding-based measures would be practically important. However, the set-level evaluation contains a technical mismatch between the estimator and the ground truth, and the real-world ground-truth assumptions are not validated; both issues directly affect the paper's central empirical claims.

major comments (2)
  1. [§4.2, Eq. (13); §5.1 set-level generation] Eq. (13) defines set-level equivalence as 2/(|a|+|b|) times the sum over the optimal matching, i.e. the Dice coefficient of the matched sets. The set-level synthetic ground truth, however, is the Jaccard coefficient of the concept sets before synonym replacement (Section 5.1: 'The ground truth is the Jaccard coefficient between set-pairs before synonym replacement'). These are not interchangeable: for two sets of size 2 with one shared concept, Eq. (13) returns 0.5 while Jaccard returns 1/3. Because Jaccard is a nonlinear function of the Dice coefficient, the per-item aggregation and the chance-agreement term in κ are also affected, not just a constant rescaling. Thus, even a perfect semantic-equivalence oracle would incur a large systematic MAE in the set experiments. This contaminates the set-level MAE values and rankings in Fig. 3(b), Fig. 4 (label-set panels), and the corresponding a
  2. [§5.2, Table 1] The real-world evaluation treats standardized terminology mappings and closed-set anomaly labels as ground truth for semantic equivalence of free-text annotations, but this assumption is not validated. For Derm1M, captions from different web sources are treated as independent annotators, and the mapping to the dermatology terminology is used without any check of completeness or correctness; if the mapping is noisy or derived from the captions themselves, the reported MAE measures how well each estimator reproduces that mapping rather than true semantic agreement. For REFLACX, free-text findings may be semantically equivalent to closed-set anomaly labels the annotator did not select, or may describe concepts outside the closed set. Because the conclusion that NLI measures are robust and embedding measures should be avoided is read directly from Table 1, this is load-bearing. The authors s
minor comments (4)
  1. [§4.1, Eq. (12)] The expected chance agreement p_s_e is estimated with only D=3 permutations in the single-label experiments and D=2 in the set experiments. This is a very small Monte Carlo sample, especially with high label uniqueness. No variance or sensitivity analysis for D is reported; please add a stability check or report confidence intervals for p_s_e.
  2. [§7] The statement that this is 'the first time that IRR has been quantified in this way for unstructured text annotations' is stronger than the related-work discussion supports, since Braylan et al. (2022) already compute agreement for free-text using distance functions. The novelty is better stated as the first systematic comparison of multiple semantic-equivalence measures against ground truth.
  3. [Table 1] The significance information conveyed by bold type is not visible in the plain-text version of the table. Please ensure the typeset version clearly distinguishes entries that are not significantly different from the best measure.
  4. [Fig. 5 and Fig. 8] The text does not explicitly state whether the random-negative baseline and the three hard-negative conditions use the same population sizes, D values, and equivalence-measure settings. Please clarify this in the experimental setup.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: soft-IRR estimators are evaluated against external, independently defined ground truths; no fitted parameter is renamed as a prediction.

full rationale

The paper's central comparison is an empirical evaluation of pre-trained or fixed equivalence measures against ground-truth IRR values computed from standardized labels (exact concept matches in synthetic data, ontology mappings in Derm1M, closed-set anomaly selections in REFLACX). None of the equivalence measures (edit distances, set overlaps, embeddings, NLI models, LLM judges) has parameters fitted to the target IRR values or to the free-text annotations used in evaluation. The soft IRR definition replaces the indicator in Eqs. (10)-(12) with a semantic equivalence estimate, and the use of the same equivalence function for observed and chance agreement is explicitly acknowledged as Cohen's weighted kappa with model-derived weights (footnote 3), i.e., a standard definition rather than a hidden reuse of the target. The real-world ground truths are independent of the soft-measure inputs: the standardized labels are not the same representation as the free-text captions, so the MAE measures genuine cross-representation transfer. The only self-citation identified (Gröger et al. 2026, including co-author Pouly) supports an empirical property of Derm1M (duplicate captions) used for dataset construction; it is not the basis of the estimator comparison and could be independently verified, so it is not load-bearing for the central claim. The stated limitations about domain transfer and pretraining overlap are appropriate caveats, not evidence that any result reduces to its inputs by construction.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new physical or modeling entities. The only free hyperparameter is the small Monte Carlo permutation count, which affects chance-agreement estimation. The key domain assumptions are that semantic equivalence is a valid probability-like quantity and that the ground-truth labels are faithful to the free-text semantics.

free parameters (1)
  • number of permutations D for Monte Carlo chance agreement = 3 (single-label), 2 (set-level)
    Used to estimate ps_e in Eq. 12. Chosen ad hoc rather than justified; small values increase estimator variance, and the paper does not analyze this sensitivity.
assumptions (3)
  • domain assumption Semantic equivalence between two free-text annotations can be represented as a probability in [0,1] and substituted into kappa-style IRR formulas.
    This is the foundational assumption of soft IRR, introduced in Section 2.2 and used throughout. It is a modeling choice without a rigorous derivation.
  • domain assumption The ground-truth labels (exact concept matches in synthetic data, terminology mappings in Derm1M, closed-set labels in REFLACX) correctly capture the intended semantics of the free-text annotations.
    The evaluation compares soft IRR on free text to hard IRR on standardized labels. If the mappings are wrong, incomplete, or the filtered subsets are biased, the MAE is not a valid quality measure. Assumed in Sections 5.1 and 5.2.
  • standard math The empirical label distributions of annotators are stationary, and the permutation-based Monte Carlo estimate of chance agreement is an unbiased estimator of the expected equivalence.
    Used to define ps_e in Eqs. 11-12. This is a standard assumption for permutation tests, but with very few permutations (D=2 or 3) the estimator has high variance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Consensus Measures for Unstructured Biomedical Text Annotations." pith.science (2026). https://pith.science/paper/XNPBSQXD

@misc{pith2026260803529,
  author       = {Pith},
  title        = {Pith review of: Consensus Measures for Unstructured Biomedical Text Annotations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XNPBSQXD}},
  note         = {Machine review of arXiv:2608.03529}
}
read the original abstract

Biomedical literature is increasingly mined for knowledge beyond the questions it was written to answer. Because the target concepts are not known in advance, annotators prefer open-ended labels, whose agreement is hard to quantify. We study soft inter-rater reliability for annotators providing unstructured texts for biomedical annotation tasks. Synthetic experiments show that soft reliability can be quantified using a variety of semantic equivalence measures, and that the choice of measure affects failure modes of the estimation. Embeddings are scalable, but limited when differentiating similar but distinct concepts. Large language models are promising, but limited by scalability for estimating agreement by chance. Finally, we suggest measures based on natural language inference as a sensible compromise.

Figures

Figures reproduced from arXiv: 2608.03529 by the authors.

Figure 1
Figure 1. Toy example highlighting the difference be [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Toy example of suboptimal greedy assignment [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Violin plots showing the distribution of the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Plots using MeSH synthetic experiment data showing the MAE on κ as a function of the ground-truth and calibration plots for the different equivalence estimates in both the single-label and set configuration. The expected calibration error (ECE) is shown in the legend a…
Figure 5
Figure 5. Figure 5: ∆ MAE against the random-negative baseline for Cohen’s κ for different negative sampling conditions and equivalence measures. Asterisks (∗ ) indicate a significant difference to the baseline according to a permutation test with 10k samples. changing the way we sample n…
Figure 6
Figure 6. Figure 6: Violin plots showing the distribution of the [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Violin plots showing the distribution of the [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: ∆ MAE against the random-negative baseline for Cohen’s κ for different negative sampling conditions and equivalence measures on the set-valued annotations. Asterisks (∗ ) indicate a significant difference to the baseline according to a permutation test with 10k samples…
Figure 9
Figure 9. Figure 9: Violin plots showing distributions for the [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Violin plots showing distributions for the [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Violin plots showing distributions for the [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Plots showing the MAE on κ as a function of the ground-truth for the different equivalence estimates in both the single-label and set configuration for all synthetic terminologies separately. Exact matching and the best model per group (set based, edit-based, emb., NL…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 33 canonical work pages

  1. [1]

    , author =

    THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE. , author =. New Phytologist , volume = 11, number = 2, pages =. doi:https://doi.org/10.1111/j.1469-8137.1912.tb05611.x , url =

    arXiv 1912

  2. [2]

    Measuring Agreement on Set-valued Items (

    Passonneau, Rebecca , year = 2006, month = may, booktitle =. Measuring Agreement on Set-valued Items (

    work page 2006

  3. [3]

    Soviet physics

    Binary codes capable of correcting deletions, insertions, and reversals , author =. Soviet physics. Doklady , volume = 10, pages =

    work page

  4. [4]

    The String-to-String Correction Problem , author =. J. ACM , publisher =. doi:10.1145/321796.321811 , issn =

    work page arXiv

  5. [5]

    Naval Research Logistics Quarterly , volume = 2, number =

    The Hungarian method for the assignment problem , author =. Naval Research Logistics Quarterly , volume = 2, number =. doi:https://doi.org/10.1002/nav.3800020109 , url =. https://onlinelibrary.wiley.com/doi/pdf/10.1002/nav.3800020109 , abstract =

    doi:10.1002/nav.3800020109

  6. [6]

    Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , publisher =

    From Alignment to Assignment: Frustratingly Simple Unsupervised Entity Alignment , author =. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , publisher =. doi:10.18653/v1/2021.emnlp-main.226 , url =

    work page doi:10.18653/v1/2021.emnlp-main.226 2021

  7. [7]

    Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop , publisher =

    A large-scale computational study of content preservation measures for text style transfer and paraphrase generation , author =. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop , publisher =. doi:10.18653/v1/2022.acl-srw.23 , url =

    doi:10.18653/v1/2022.acl-srw.23 2022

  8. [8]

    Improving Paraphrase Detection with the Adversarial Paraphrasing Task , author =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , publisher =. doi:10.18653/v1/2021.acl-long.552 , url =

    doi:10.18653/v1/2021.acl-long.552 2021

Show all 53 references
  1. [9]

    Educational and Psychological Measurement , volume = 20, number = 1, pages =

    A Coefficient of Agreement for Nominal Scales , author =. Educational and Psychological Measurement , volume = 20, number = 1, pages =. doi:10.1177/001316446002000104 , url =

    doi

  2. [10]

    International Classification of Diseases 11th Revision , author =

  3. [11]

    Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024 , publisher =

    PASSION for Dermatology: Bridging the Diversity Gap with Pigmented Skin Images from Sub-Saharan Africa , author =. Medical Image Computing and Computer Assisted Intervention -- MICCAI 2024 , publisher =

    2024

  4. [12]

    and Chan, Jessica and Duong, Phuong-Anh T

    Bigolin Lanfredi, Ricardo and Zhang, Mingyuan and Auffermann, William F. and Chan, Jessica and Duong, Phuong-Anh T. and Srikumar, Vivek and Drew, Trafton and Schroeder, Joyce D. and Tasdizen, Tolga , year = 2022, journal =

    2022

  5. [13]

    Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages =

    Derm1M: A Million-scale Vision-Language Dataset Aligned with Clinical Ontology Knowledge for Dermatology , author =. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages =

  6. [14]

    Computational Linguistics , volume = 34, number = 4, pages =

    Survey Article: Inter-Coder Agreement for Computational Linguistics , author =. Computational Linguistics , volume = 34, number = 4, pages =. doi:10.1162/coli.07-034-R2 , url =

    doi

  7. [15]

    American Sociological Review , publisher =

    The Open and Closed Question , author =. American Sociological Review , publisher =

  8. [16]

    Text Annotation via Inductive Coding: Comparing Human Experts to

    Parfenova, Angelina and Marfurt, Andreas and Pfeffer, J. Text Annotation via Inductive Coding: Comparing Human Experts to. Findings of the Association for Computational Linguistics: NAACL 2025 , publisher =. doi:10.18653/v1/2025.findings-naacl.361 , isbn =

    2025 doi

  9. [17]

    doi:10.18653/v1/2023.findings-emnlp.669 , url =

    Dai, Shih-Chieh and Xiong, Aiping and Ku, Lun-Wei , year = 2023, month = dec, booktitle =. doi:10.18653/v1/2023.findings-emnlp.669 , url =

    2023 doi

  10. [18]

    Public Opinion Quarterly , volume = 19, number = 3, pages =

    Reliability of Content Analysis:The Case of Nominal Scale Coding , author =. Public Opinion Quarterly , volume = 19, number = 3, pages =. doi:10.1086/266577 , issn =

    doi

  11. [19]

    Journal of Biomedical Informatics , volume = 47, pages =

    NCBI disease corpus: A resource for disease name recognition and concept normalization , author =. Journal of Biomedical Informatics , volume = 47, pages =. doi:https://doi.org/10.1016/j.jbi.2013.12.006 , issn =

    2013 doi

  12. [20]

    Hruschka and Deborah Schwartz and Daphne Cobb St.John and Erin Picone-Decaro and Richard A

    Daniel J. Hruschka and Deborah Schwartz and Daphne Cobb St.John and Erin Picone-Decaro and Richard A. Jenkins and James W. Carey , year = 2004, journal =. Reliability in Coding Open-Ended Data: Lessons Learned from. doi:10.1177/1525822X04266540 , url =. https://doi.org/10.1177...

    2004 doi

  13. [21]

    doi:10.18653/v1/2020.acl-main.704 , url =

    Sellam, Thibault and Das, Dipanjan and Parikh, Ankur , year = 2020, month = jul, booktitle =. doi:10.18653/v1/2020.acl-main.704 , url =

    2020 doi

  14. [22]

    Proceedings of the ACM Web Conference 2022 , location =

    Measuring Annotator Agreement Generally across Complex Structured, Multi-object, and Free-text Annotation Tasks , author =. Proceedings of the ACM Web Conference 2022 , location =. doi:10.1145/3485447.3512242 , isbn = 9781450390965, url =

    2022

  15. [23]

    The First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP (AnnoNLP 2019) , url =

    Distance-based Consensus Modeling for Complex Annotations , author =. The First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP (AnnoNLP 2019) , url =

    2019

  16. [24]

    , author =

    Measuring nominal scale agreement among many raters. , author =. Psychological bulletin , publisher =

  17. [25]

    Advances in Data Analysis and Classification , volume = 4, number = 4, pages =

    Inequalities between multi-rater kappas , author =. Advances in Data Analysis and Classification , volume = 4, number = 4, pages =. doi:10.1007/s11634-010-0073-4 , issn =

    doi

  18. [26]

    Psychological bulletin , volume = 70, number = 4, pages =

    Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit , author =. Psychological bulletin , volume = 70, number = 4, pages =. doi:10.1037/h0026256 , issn =

    doi

  19. [27]

    Educational and Psychological Measurement , volume = 30, number = 1, pages =

    Estimating the Reliability, Systematic Error and Random Error of Interval Data , author =. Educational and Psychological Measurement , volume = 30, number = 1, pages =. doi:10.1177/001316447003000105 , url =

    doi

  20. [28]

    Proceedings of

    Manual Corpus Annotation: Giving Meaning to the Evaluation Metrics , author =. Proceedings of

  21. [29]

    2603.06865 , archiveprefix =

    Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation , author =. 2603.06865 , archiveprefix =

  22. [30]

    doi:10.18653/v1/2023.eacl-main.148 , url =

    Muennighoff, Niklas and Tazi, Nouamane and Magne, Loic and Reimers, Nils , year = 2023, month = may, booktitle =. doi:10.18653/v1/2023.eacl-main.148 , url =

    2023 doi

  23. [31]

    doi:10.18653/v1/2021.emnlp-main.552 , url =

    Gao, Tianyu and Yao, Xingcheng and Chen, Danqi , year = 2021, month = nov, booktitle =. doi:10.18653/v1/2021.emnlp-main.552 , url =

    2021 doi

  24. [32]

    Sentence-

    Reimers, Nils and Gurevych, Iryna , year = 2019, month = nov, booktitle =. Sentence-. doi:10.18653/v1/D19-1410 , url =

    2019 doi

  25. [33]

    BMC Medical Informatics and Decision Making , volume = 21, number = 6, pages = 206, doi =

    ICD-11: an international classification of diseases for the twenty-first century , author =. BMC Medical Informatics and Decision Making , volume = 21, number = 6, pages = 206, doi =

  26. [34]

    Research Synthesis Methods , volume = 15, number = 4, pages =

    Can large language models replace humans in systematic reviews? Evaluating GPT-4's efficacy in screening and extracting data from peer-reviewed and grey literature in multiple languages , author =. Research Synthesis Methods , volume = 15, number = 4, pages =. doi:https://doi....

    doi

  27. [35]

    Mezzetti, David , year = 2023, note =

    2023

  28. [36]

    The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks , publisher =

    Multiple Evidence Combination for Fact-Checking of Health-Related Information , author =. The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks , publisher =

  29. [37]

    doi:10.13026/C2RS98 , url =

    Shivade, Chaitanya , year = 2019, month = oct, journal =. doi:10.13026/C2RS98 , url =

    2019 doi

  30. [38]

    MedGemma 1.5 Technical Report , author =

  31. [39]

    Laurer, Moritz and Atteveldt, Wouter van and Casas, Andreu Salleras and Welbers, Kasper , year = 2022, month = jun, journal =. Less

    2022

  32. [40]

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models , author =

  33. [41]

    International Conference on Learning Representations , url =

    BERTScore: Evaluating Text Generation with BERT , author =. International Conference on Learning Representations , url =

  34. [42]

    doi:10.3115/1073083.1073135 , url =

    Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing , year = 2002, month = jul, booktitle =. doi:10.3115/1073083.1073135 , url =

    2002

  35. [43]

    CoRR , volume =

    Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation , author =. CoRR , volume =. 1609.08144 , timestamp =

    arXiv

  36. [44]

    2601.00840 , archiveprefix =

    A Global Atlas of Digital Dermatology to Map Innovation and Disparities , author =. 2601.00840 , archiveprefix =

  37. [45]

    Medical Subject Headings (

  38. [46]

    Medical Dictionary for Regulatory Activities (

  39. [47]

    Journal of medical Internet research , volume = 23, number = 12, pages =

    The Utility of Different Data Standards to Document Adverse Drug Event Symptoms and Diagnoses: Mixed Methods Study , author =. Journal of medical Internet research , volume = 23, number = 12, pages =. doi:10.2196/27188 , issn =

    doi

  40. [48]

    BMJ Open , publisher =

    Inter-reviewer reliability of human literature reviewing and implications for the introduction of machine-assisted systematic reviews: a mixed-methods review , author =. BMJ Open , publisher =. doi:10.1136/bmjopen-2023-076912 , issn =

    2023 doi

  41. [49]

    Nature Communications , volume = 14, number = 1, pages = 7329, doi =

    The Imageable Genome , author =. Nature Communications , volume = 14, number = 1, pages = 7329, doi =

  42. [50]

    Nature Communications , volume = 15, number = 1, pages = 10904, doi =

    The Theranostic Genome , author =. Nature Communications , volume = 15, number = 1, pages = 10904, doi =

  43. [51]

    Communications Medicine , doi =

    Repurposing molecular imaging to map drug targets in vivo , author =. Communications Medicine , doi =

  44. [52]

    BMJ , publisher =

    Effects of antithyroid drugs on radioiodine treatment: systematic review and meta-analysis of randomised controlled trials , author =. BMJ , publisher =. doi:10.1136/bmj.39114.670150.BE , issn =

  45. [53]

    Computer Vision -- ECCV 2020 , publisher =

    End-to-End Object Detection with Transformers , author =. Computer Vision -- ECCV 2020 , publisher =

    2020

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.