Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models

James Mork; Victor H. Cid

REVIEW 4 major objections 4 minor 46 references

Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned Transformer models can outperform the legacy MTI system on MEDLINE publication-type tagging.

desk verdict Useful feasibility study undermined by an apples-to-oranges MTI comparison; the internal results are plausible but the headline claim doesn't hold as written. read the letter →

arxiv 2506.03321 v1 pith:OSNAP2AM submitted 2025-06-03 cs.DL cs.LG

Victor H. Cid , James Mork This is my paper

classification cs.DLcs.LG

keywords MEDLINEMeSHPublicationTypesPre-trainedFoundationModelsNaturalLanguageProcessingMachineLearningBERTDistilmulti-labelclassification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that modern Transformer models—BERT and its lighter cousin DistilBERT—can predict MeSH Publication Types (PTs) for MEDLINE citations more accurately and efficiently than the legacy MTI automated indexing system. The authors build two kinds of classifiers: a single multi-label BERT that predicts several PT tags in one pass, and an ensemble of binary DistilBERT classifiers, one per tag. Using only the citation's journal ID, title, and abstract, the best configurations reach F1 scores above 0.9 for common tags like 'Randomized Controlled Trial' and 'Systematic Review,' while the MTI baseline's macro F1 is 0.64. The study also shows the monolithic model is roughly 70 percent faster at inference than the ensemble, which matters for MEDLINE's daily influx of up to 20,000 citations.

What carries the argument

The machinery that carries the argument is a fine-tuned Transformer classifier fed a compact tokenized string of three metadata fields—journal ID, title, and abstract—with special separators, so the model can infer PTs from the journal's publishing habits and the text's vocabulary. Two architectures use this input: a monolithic multi-label BERT that outputs several PT probabilities in one forward pass, and an ensemble of binary DistilBERT classifiers each trained on a balanced 50/50 sample. A 'tag compiler' post-processing module filters low-confidence predictions, enforces NLM co-occurrence rules, and limits the number of tags per citation. Stratified sampling of the extreme multi-label data ensures rare PTs appear in training and evaluation sets.

What would settle it

Run MTI and the best fine-tuned BERT/DistilBERT models on the exact same held-out MEDLINE test set and compare macro-F1; if MTI's macro-F1 equals or exceeds the Transformers' macro-F1, the paper's central claim of improved tagging accuracy is falsified.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that transfer learning with pre-trained Transformers is a feasible and advantageous replacement for the rule-and-NLP-based MTI pipeline in assigning PT tags. On a corpus of over five million manually indexed 2016–2023 MEDLINE citations, the authors fine-tune BERT and DistilBERT on the structured input 'Journal ID<1>Title<2>Abstract' and find that both a monolithic multi-label classifier and a binary ensemble produce recall and F1 that generally exceed the MTI numbers reproduced in Table 1 (macro precision 0.84, recall 0.53, F1 0.64). The ensemble of 52 binary DistilBERT classifiers reaches per-tag F1 scores above 0.9 for most tags, while the monolithic BERT, restricted to the top 10 tags, achieves a micro-averaged F1 of 0.78; the monolithic model's performance drops as more tags are added (to 0.34 macro F1 at 52 tags).

Load-bearing premise

The comparison assumes the MTI baseline numbers from a 40K-citation sample are directly comparable to the Transformer models' scores on a different 5% test split of the 2016–2023 corpus, so if those test sets differ in composition, the claimed improvement may not hold.

Editorial extensions

If this is right

A monolithic BERT model can process roughly 20,000 citations in about one hour on modest hardware, versus about 3.5 hours for an 11–15 member ensemble, making it a realistic candidate for daily MEDLINE ingestion.
The ensemble architecture gives near-0.9 to 0.99 F1 for most of the 52 most frequent PT tags, so it can serve as a precision-oriented module for rare or high-value tags.
Accuracy of the monolithic model degrades as the number of PT classes grows (macro F1 falls from 0.78 on top-10 tags to 0.34 on top-52 tags), implying that production systems should either cap the tag set or hybridize with per-tag classifiers.
Stratified sampling alone improves recall for minority tags such as Meta-Analysis by about 20 percentage points compared with unstratified training, so data balancing is a prerequisite for fair PT prediction.
Prioritizing recall over precision, as the tag compiler does with dynamic thresholds, aligns with MEDLINE's goal of comprehensive literature retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The headline 'improvement over MTI' rests on a comparison between MTI scores from a 40K-citation sample and Transformer scores from a different 5% test split; a fair head-to-head on an identical test set could narrow or reverse the gap. This is an editorial caution, not a paper claim.
Editorial note: the text attributes DistilBERT's 'preserves over 97% of BERT's performance' to reference [43], but the reference list ends at [37], so that attribution is unverifiable from the manuscript, though the measured F1 scores in the results stand on the authors' own experiments.
The same two-architecture recipe (monolithic multi-label plus binary ensemble with a tag compiler) could transfer to other controlled-vocabulary indexing tasks, such as MeSH main headings, chemical tags, or funding-category labels, provided the input includes informative metadata fields.
Because the study deliberately excludes citations that were indexed by automated systems, the models are trained only on human-indexed data; a live deployment would need periodic retraining to keep pace with the shift to fully automated indexing after April 2023.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Quick read of arXiv:2506.03321. The paper fine-tunes BERT/DistilBERT for MeSH Publication Type tagging on a large MEDLINE corpus and compares monolithic multi-label classifiers against ensembles of binary classifiers. That is a legitimate extension of prior work that only targeted single PTs. The engineering and evaluation choices — stratified sampling, tag compiler thresholds, held-out test set, inference speed comparisons — are sensible and clearly described. The per-class F1 scores for the ensemble (many above 0.9) are interesting and suggest the approach is worth pursuing operationally.

But the central claim, "Transformers significantly improve PT tagging accuracy over MTI," is not established. The MTI baseline in Table 1 comes from an unspecified 40K citation sample supplied by the MTI team, while all transformer results come from a 5% holdout of the 2016–2023 corpus. No attempt is made to run MTI on the same holdout, control for label set size, normalization (e.g., removing 'Journal Article' from multi-tagged records), or tag-compiler thresholding. Any observed gap between their macro/micro F1 and MTI's 0.64 could simply reflect test-set differences. Given the authors are at NLM, a matched baseline should have been straightforward; its absence is a serious, fixable flaw.

There are also reporting inconsistencies. The conclusion mentions monolithic macro and micro F1 of 0.75–0.85, but Table 5 shows the top-52 monolithic model at 0.34 macro (or micro, the table note is confusing) with per-class F1s that don't align with that range. The paper also silently drops to 52 of 66 PT classes, normalizes labels, and applies thresholds, all of which shape the numbers. These need to be stated clearly and consistently.

On the plus side, the internal comparisons between architectures are on the same data and are credible. The ensemble vs monolithic trade-off, the resource constraints, and the emphasis on recall over precision for retrieval are all handled honestly. The paper is not circular; evaluation uses held-out data.

For whom is this? Someone building a next-gen automated indexing pipeline at NLM or elsewhere will get a useful feasibility template. A reader searching for evidence that Transformers beat MTI will be misled unless the baseline issue is addressed. It deserves a serious referee — not because the current claim is proven, but because the work is substantive and the flaws are reparable. With a matched MTI baseline and corrected reporting, I could see this being a solid applied paper.

Recommendation: send to peer review with the expectation of major revision; encourage the authors to run MTI on their exact test set.

Best.

Referee Report

4 major / 4 minor

Summary. The paper reports on fine-tuning BERT and DistilBERT models to assign MeSH Publication Types (PTs) from MEDLINE citation metadata. Two architectures are explored: a monolithic multi-label classifier and an ensemble of binary classifiers, both followed by a 'tag compiler' post-processing module. The authors present per-class precision, recall, and F1 results for a monolithic BERT model (Table 4), a 52-classifier DistilBERT ensemble (Table 6), ensemble configurations with different tag limits and thresholds (Table 7), and inference-time comparisons (Table 8). The central claim, stated in the abstract and conclusion, is that Transformer-based models can significantly improve PT tagging accuracy over the existing automated MTI indexing system.

Significance. If the claimed improvement over MTI were supported, the results would be practically relevant for automated biomedical indexing, given the scale of MEDLINE and the need for efficient, accurate PT assignment. The paper's strengths include the use of a large real-world corpus (over five million citations), explicit attention to class imbalance via stratified sampling, and a concrete discussion of inference throughput and resource constraints. These aspects make the work a useful feasibility study. However, the headline claim of superiority over MTI is not established because the baseline and the proposed models are evaluated on different datasets with different preprocessing, label sets, and evaluation protocols. The paper would need a matched baseline on the same holdout set, or a reframed claim, to support its central conclusion. No code, data, or MTI outputs are provided, which limits reproducibility of the comparison.

major comments (4)

[Section 2.2, Table 1; Section 4.1.4-4.1.5; Section 5] The MTI baseline in Table 1 is computed on a different 40K citation sample whose date range, label set, and preprocessing are not specified, whereas the Transformer results in Tables 4-7 are measured on a 5% holdout of the 2016-2023 corpus after normalization that removes 'Journal Article' from multi-tagged records and after tag-compiler thresholding. Because the test sets are not matched, the observed F1 differences (e.g., MTI macro F1 0.64 in Table 1 versus ensemble micro F1 up to 0.83 in Table 7) cannot be attributed to model superiority. To support the abstract's claim, the authors must run MTI on the same holdout under the same label set and normalization, or remove the comparative claim and present the work as a feasibility study.
[Section 7.1, Conclusion] The conclusion states that the monolithic model achieved 'macro and micro F1-scores of 0.75-0.85', but Table 5 reports micro F1 of 0.78 (Top-10), 0.69 (Top-15), and 0.34 (Top-52). The source of the 0.75-0.85 range is not identified and appears inconsistent with the reported numbers. This overstates the monolithic model's performance and should be corrected with exact figures and a clear statement of which configuration supports the claim.
[Section 5.2, Table 7] The ensemble's headline result is presented for only four configurations after tag filtering and truncation, while Table 6 reports per-class scores for binary classifiers evaluated individually. It is not shown how per-class scores are combined into the micro-averaged F1, what the P=0.9 threshold does per class, or whether the same tag-compiler settings were applied to the monolithic model. Without a common evaluation protocol, the architecture comparison in Section 6.2 is not quantitatively grounded.
[Section 5, Tables 4-7] All reported metrics are point estimates without confidence intervals, significance tests, or error bars. Given the extreme class imbalance, threshold tuning in the tag compiler, and the different sample sizes across classes, the differences between configurations and between models could be within noise. At minimum, the authors should report bootstrap confidence intervals or standard errors for the main F1 values.

minor comments (4)

[Table 5] The column heading 'Avg. Macro-Average F1-score' contradicts the table caption, which says the numbers are micro-averages. Please clarify which averaging was actually used.
[Throughout] There are several typographical errors: 'MELDINE' in Section 3, 'MDELINE' in the Figure 1 caption, 'Merilles' in Section 4.1.4 versus 'Merrillees' in reference [33], and 'RT 1080 Ti' in Section 4.4.3, which should likely be 'GTX 1080 Ti'.
[Section 5.3, Table 8] The inference-time comparison reports times rounded to the closest half hour but does not specify whether batch sizes, sequence lengths, and hardware were identical across the monolithic and ensemble runs. Please provide these details or note that the comparison is approximate.
[Section 4] The paper does not include a data availability or code availability statement, nor does it describe hyperparameter values (e.g., learning rate, batch size, number of epochs) or random seeds. Adding these would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the Transformer evaluations are held-out and self-contained; the MTI comparison is a study-design concern, not a circular derivation.

full rationale

The paper's central claim is a feasibility result: fine-tuned BERT and DistilBERT models predict MeSH Publication Types on a held-out test set. The models are trained on 90% of the 2016-2023 MEDLINE corpus, tuned on a 5% validation split, and evaluated on a separate 5% test split (Section 4.1.4), with thresholds tuned on validation data (Section 4.2.5). The reported Transformer F1 scores in Tables 4-7 are therefore empirical measurements on unseen data, not quantities forced by construction from the training labels. The only potentially load-bearing external comparison is the MTI baseline in Table 1, which was 'kindly supplied by the NLM MTI team' from a 40K citation sample. If the MTI sample and the Transformer test set are not matched, the headline improvement over MTI is unestablished; however, that is a validity or comparability problem, not circularity, because the Transformer numbers are not derived from the MTI numbers nor defined in terms of them. The paper's self-citations to the MTI literature (references [6], [7], [10], co-authored by James Mork) are background descriptions of the baseline system and are not used to derive the Transformer results. No equation or definition reduces the predicted PT tags to the evaluation labels, the MTI baseline, or any self-cited theorem. Accordingly, no circular step satisfies the requirement of exhibiting a specific reduction, and the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The work depends on standard assumptions of supervised learning with a manually annotated corpus. No new theoretical entities are introduced. The free parameters are the tuning knobs of the tag compiler and class weighting, none of which are reported with concrete values.

free parameters (3)

Tag compiler thresholds = not reported
Per-PT thresholds for inclusion were optimized on validation data to balance precision and recall (Section 4.3); values are not listed.
Class weights for binary classifiers = not reported
Section 4.1.4 states class weights were used to penalize misclassifications, but the weights are not reported.
Fixed threshold (0.5) for tag inclusion = 0.5
Mentioned as an example of a fixed threshold in Section 4.3, though it may not be the exact value used.

assumptions (3)

domain assumption Human manual indexing of MEDLINE during 2016-2023 is an accurate ground truth for PT labels.
The paper uses these manual labels as training and test targets without assessing inter-indexer variability or errors (Section 4.1.1).
domain assumption The selected metadata fields (title, abstract, journal ID) are sufficient for PT prediction.
The authors exclude full-text and other metadata due to token limits, assuming the retained fields carry the discriminative signal (Section 4.1.3).
standard math The stratification algorithm of Merrillees and Du preserves the relevant multi-label distribution.
The algorithm is cited but not proven suitable for this data; the authors rely on it to maintain proportional representation (Section 4.1.4).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models." pith.science (2026). https://pith.science/paper/OSNAP2AM

@misc{pith2026250603321,
  author       = {Pith},
  title        = {Pith review of: Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OSNAP2AM}},
  note         = {Machine review of arXiv:2506.03321}
}

read the original abstract

We investigated the feasibility of predicting Medical Subject Headings (MeSH) Publication Types (PTs) from MEDLINE citation metadata using pre-trained Transformer-based models BERT and DistilBERT. This study addresses limitations in the current automated indexing process, which relies on legacy NLP algorithms. We evaluated monolithic multi-label classifiers and binary classifier ensembles to enhance the retrieval of biomedical literature. Results demonstrate the potential of Transformer models to significantly improve PT tagging accuracy, paving the way for scalable, efficient biomedical indexing.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 45 canonical work pages

[1]

Its signiﬁcance lies not only in its vast scope but also in its ability to organize and provide eﬃcient access to this wealth of information

Introduction The MEDLINE indexed subset of the National Library of Medicine’s (NLM’s) PubMed service is a cornerstone of biomedical knowledge, housing millions of citations from journals worldwide. Its signiﬁcance lies not only in its vast scope but also in its ability to organize and provide eﬃcient access to this wealth of information. MEDLINE's indexin...

work page
[2]

Clinical Trials

Previous work 2.1. MEDLINE and PT Tags MEDLINE organizes biomedical citations with MeSH PTs [1] [2], facilitating targeted searches. PTs categorize documents by research methodologies and publication formats, such as "Clinical Trials" or "Review," rather than content. Previous work has shown the utility of MEDLINE Publication Type tags to perform a variet...

work page 2024
[3]

Journal Article

Challenges in PT Prediction Accurate prediction of MeSH Publication Types (PTs) from MEDLINE citation metadata presents several signiﬁcant challenges, which can be broadly categorized into three groups: Data-Related Issues: A signiﬁcant challenge lies in the characteristics of the data itself. Our corpus consists of over 5 million MELDINE citations manual...

work page 2016
[4]

Preprint

Methodology We used two distinct model architectures: monolithic multilabel classiﬁers and an ensemble of binary classiﬁers. Both required similar data preprocessing steps, with some diﬀerences as described below. 4.1. Data Sources and Preparation 4.1.1. Data Sources The dataset for this research was derived from MEDLINE, an indexed subset of PubMed, whic...

work page 2016
[5]

Journal Article

Results The experiments provided valuable insights into the performance of selective predictive models and the eﬀectiveness of various strategies for addressing challenges in PT prediction. Below, we elaborate on the ﬁndings for the major types of models used and the role of stratiﬁed sampling in enhancing performance. 5.1. Monolithic Multi-Label Classiﬁe...

work page
[6]

Journal Article

Discussion 6.1. Interpretation of Results The experimental results demonstrate that modern Transformer-based models can signiﬁcantly enhance the accuracy and eﬃciency of PT tagging for MEDLINE citations. Both the ensemble of 19 binary classiﬁers and the monolithic multi-label classiﬁer showed notable improvements in recall compared to legacy systems, ensu...

work page
[7]

Journal Article

Conclusion This study demonstrates the feasibility and advantages of using Transformer-based machine learning models to predict MeSH Publication Types (PTs) for MEDLINE citations. The results highlight the substantial opportunities that these technologies oﬀer to improve the automated indexing of MEDLINE citations. 7.1. Summary of Findings The ensemble of...

work page
[8]

Acknowledgments This work was supported by the Lister Hill National Center for Biomedical Communications of the National Library of Medicine (NLM), National Institutes of Health

work page

Show all 46 references

[9]

Publication Characteristics (Publication Types) with Scope Notes,

NLM, "Publication Characteristics (Publication Types) with Scope Notes," 22 December
[10]

The road from manual to automatic semantic indexing of biomedical literature: a 10 years journey.,

A. Krithara, J. Mork, A. Nentidis and G. Paliouras, "The road from manual to automatic semantic indexing of biomedical literature: a 10 years journey.," Front Res Metr Anal, vol. 8, p. 1250930, 29 September 2023

2023
[11]

MEDLINE/PubMed Data Element (Field) Descriptions.,

NLM, "MEDLINE/PubMed Data Element (Field) Descriptions.," [Online]. Available: https://www.nlm.nih.gov/bsd/mms/medlineelements.html. [Accessed 3 June 2024]

2024
[12]

Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews,

J. Schneider, L. Hoang, K. Y . and C. A.M., "Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews," JAMIA Open, vol. 5, no. 1, p. ooac015, 30 March 2022

2022
[13]

Testing a ﬁltering strategy for systematic reviews: evaluating work savings and recall.,

R. Proescholdt, T. K. Hsiao, J. Schneider, A. M. Cohen, M. S. McDonagh and N. R. Smalheiser, "Testing a ﬁltering strategy for systematic reviews: evaluating work savings and recall.," AMIA Annual Symposium Proceedings, pp. 406-413, 2022

2022
[14]

Articles on Complementary Medicine in the Mainstream Medical Literature: An Investigation of MEDLINE, 1966 through 1996.,

J. Barnes, N. C. Abbot, H. E. F. and E. Ernst, "Articles on Complementary Medicine in the Mainstream Medical Literature: An Investigation of MEDLINE, 1966 through 1996.," Arch Intern Med, vol. 159, no. 15, pp. 1721-1725, 1999

1966
[15]

The NLM Indexing Initiative's Medical Text Indexer,

A. Aronson, J. Mork, C. Gay, S. Humphrey and W. Rogers, "The NLM Indexing Initiative's Medical Text Indexer," Studies in health technology and informatics, vol. 107, no. 1, pp. 268-272, 2004

2004
[16]

The NLM Medical Text Indexer System for Indexing Biomedical Literature,

J. Mork, A. Jimeno-Yepes and A. Aronson, "The NLM Medical Text Indexer System for Indexing Biomedical Literature," BioASQ@CLEF, 2013

2013
[17]

NLM Technical Bulletin,

NLM, "NLM Technical Bulletin," 1 December 2021. [Online]. Available: https://www.nlm.nih.gov/pubs/techbull/nd21/nd21_medline_2022.html. [Accessed 3 June 2024]

2021
[18]

Automated Indexing using MRT's Medical Text Indexer (MTI) compared to human indexing in Medline: a pilot study.,

E. Chen, J. Bullard and D. Giustini, "Automated Indexing using MRT's Medical Text Indexer (MTI) compared to human indexing in Medline: a pilot study.," Journal of the Medical Library Association, vol. 111, no. 3, pp. 684-694, 2023

2023
[19]

Performance Comparison of Transformer- Based Models on Twitter Health Mention Classiﬁcation,

P . I. Khan, I. Razzak, A. Dengel and S. Ahmed, "Performance Comparison of Transformer- Based Models on Twitter Health Mention Classiﬁcation," IEEE Transactions on Computational Social Systems, vol. 10, no. 3, pp. 1140-1149, 2023

2023
[20]

Manual Versus machine: How accurately Does the Medical Text Indexer (MTI) Classify Diﬀerent Document Type into Disease Areas?,

D. Moore, Y . O. and B. Sampat, "Manual Versus machine: How accurately Does the Medical Text Indexer (MTI) Classify Diﬀerent Document Type into Disease Areas?," PLoS One, vol. 19, no. 3, p. e0297526, 2024

2024
[21]

Bat4RCT: A suite of benchmark data and baseline methods for text classiﬁcation of randomized controlled trials,

J. Kim, J. Kim, A. Lee and J. Kim, "Bat4RCT: A suite of benchmark data and baseline methods for text classiﬁcation of randomized controlled trials," PloS one, vol. 18, no. 3, p. e0283342, 2023. 24

2023
[22]

Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach.,

B. C. Wallace, A. Noel-Storr, I. J. Marshall, A. M. Cohen, N. R. Smalheiser and J. Thomas, "Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach.," Journal of the American Medical Informatics Association : JAMIA, v...

2017
[23]

Extending PubMed searches to ClinicalTrials.gov through a machine learning approach for systematic reviews,

C. Lanera, C. Minto, A. Sharma, D. Gregori, P . Berchialla and I. Baldi, "Extending PubMed searches to ClinicalTrials.gov through a machine learning approach for systematic reviews," J Clin Epidemiol, vol. 103, no. Nov, pp. 22-30, 2018

2018
[24]

Automated conﬁdence ranked classiﬁcation of randomized controlled trial articles: an aid to evidence-based medicine,

A. M. Cohen, N. R. Smalheiser, M. S. McDonagh, C. Yu, C. E. Adams, J. M. Davis and P . S. Yu, "Automated conﬁdence ranked classiﬁcation of randomized controlled trial articles: an aid to evidence-based medicine," Journal of the American Medical Informatics Association : JAMIA,...

2015
[25]

srBERT: automatic article classiﬁcation model for systematic review using BERT,

S. Aum and S. Choe, "srBERT: automatic article classiﬁcation model for systematic review using BERT," Syst Rev, vol. 10, no. 1, p. 285, 2021

2021
[26]

Transformers: State-of-the-Art Natural Language Processing,

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P . Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P . von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. Le Scao and e. al., "Transformers: State-of-the-Art Natural Language Processing," in Procee...

2020
[27]

Balancing Methods for Multi- label Text Classiﬁcation with Long-Tailed Class Distribution,

Y . Huang, B. Giledereli, A. Köksal, A. Özgür and E. Ozkirimli, "Balancing Methods for Multi- label Text Classiﬁcation with Long-Tailed Class Distribution," in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic, 2021

2021
[28]

Comparison of Pretraining Models and Strategies for Health-Related Social Media Text Classiﬁcation.,

Y . Guo, Y . Ge, Y . C. Yang, M. A. Al-Garadi and A. Sarker, "Comparison of Pretraining Models and Strategies for Health-Related Social Media Text Classiﬁcation.," Healthcare, vol. 10, no. 8, p. 1478, 2022

2022
[29]

The inﬂuence of preprocessing on text classiﬁcation using a bag-of-words representation,

Y . HaCohen-Kerner, D. Miller and Y . Yigal, "The inﬂuence of preprocessing on text classiﬁcation using a bag-of-words representation," PloS one, vol. 15, no. 5, p. e0232525
[30]

A scoping review of preprocessing methods for unstructured text data to assess data quality,

M. Nesca, A. Katz, C. Leung and L. M. Lix, "A scoping review of preprocessing methods for unstructured text data to assess data quality," International journal of population data science, vol. 7, no. 1, p. 1757, 2022. 25

2022
[31]

A systematic map of medical data preprocessing in knowledge discovery,

A. Idri, H. Benhar, J. L. Fernández-Alemán and I. Kadi, "A systematic map of medical data preprocessing in knowledge discovery," Computer methods and programs in biomedicine, vol. 162, pp. 69-85, 2018

2018
[32]

Towards Transfer Learning Techniques-BERT, DistilBERT, BERTimbau, and DistilBERTimbau for Automatic Text Classiﬁcation from Diﬀerent Languages: A Case Study.,

R. Silva Barbon and A. A. T., "Towards Transfer Learning Techniques-BERT, DistilBERT, BERTimbau, and DistilBERTimbau for Automatic Text Classiﬁcation from Diﬀerent Languages: A Case Study.," Sensors (Basel), vol. 22, no. 21, p. 8184, 26 October 2022

2022
[33]

Natural language processing analysis applied to COVID-19 open-text opinions using a distilBERT model for sentiment categorization.,

M. Jojoa, P . Eftekhar, B. Nowrouzi-Kia and B. Garcia-Zapirain, "Natural language processing analysis applied to COVID-19 open-text opinions using a distilBERT model for sentiment categorization.," Ai & Society, vol. 39, pp. 883-890, 2022

2022
[34]

Authorship identiﬁcation using ensemble learning.,

A. Abbasi, A. R. Javed, F. Iqbal, Z. Jalil, T. R. Gadekallu and N. Kryvinska, "Authorship identiﬁcation using ensemble learning.," Scientiﬁc Reports, vol. 12, no. 1, p. 9537, 2022

2022
[35]

"Identifying Potential Lyme Disease Cases Using Self-Reported Worldwide Tweets: Deep Learning Modeling Approach Enhanced With Sentimental Words Through Emojis.,

E. K. E. Laison, M. Hamza Ibrahim, S. Boligarla, J. Li, R. Mahadevan, A. Ng, V. Muthuramalingam, W. Y . Lee, Y . Yin and B. R. Nasri, " "Identifying Potential Lyme Disease Cases Using Self-Reported Worldwide Tweets: Deep Learning Modeling Approach Enhanced With Sentimental Wor...

2023
[36]

When BERT meets Bilbo: a learning curve analysis of pretrained language model on disease classiﬁcation.,

X. Li, W. Yuan, D. Peng, Q. Mei and Y . Wang, "When BERT meets Bilbo: a learning curve analysis of pretrained language model on disease classiﬁcation.," BMC Medical Informatics and Decision Making, vol. 21, no. Suppl 9, p. 377, 2022

2022
[37]

Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation.,

H. Wu, P . Judd, X. Zhang, M. Isaev and P . Micikevicius, "Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation.," ArXiv, vol. abs/2004.09602, 2020

2004 arXiv
[38]

Indexing Manual of the National Library of Medicine, Chapter 17.6, Publication Type Combinations.,

NLM, "Indexing Manual of the National Library of Medicine, Chapter 17.6, Publication Type Combinations.," [Online]. Available: https://wiki.nlm.nih.gov/conﬂuence/display/IM/Chapter+17%3A+Publication+Types. [Accessed 18 June 2024]

2024
[39]

Automated Indexing FAQs,

NLM, "Automated Indexing FAQs," [Online]. Available: https://support.nlm.nih.gov/kbArticle/?pn=KA-05326. [Accessed 9 December 2024]

2024
[40]

On the Stratiﬁcation of Multi-label Data.,

K. Sechidis, G. Tsoumakas and I. Vlahavas, "On the Stratiﬁcation of Multi-label Data.," in Machine Learning and Knowledge Discovery in Databases, Berlin, 2011. 26

2011
[41]

A Network Perspective on Stratiﬁcation of Multi-Label Data.,

Szymański, P . and ,. Kajdanowicz, "A Network Perspective on Stratiﬁcation of Multi-Label Data.," in LIDTA@PKDD/ECML, 2017

2017
[42]

Stratiﬁed Sampling for Extreme Multi-Label Data.,

M. Merrillees and L. Du, "Stratiﬁed Sampling for Extreme Multi-Label Data.," in Advances in Knowledge Discovery and Data Mining, Cham, 2021

2021
[43]

Learning from imbalanced data.,

H. He and E. A. Garcia, "Learning from imbalanced data.," IEEE Trans. Knowl. Data Eng., vol. 21, no. 9, pp. 1263-1284, 2009

2009
[44]

The precision-recall plot is more informative than the ROC plot when evaluating binary classiﬁers on imbalanced datasets.,

T. Saito and M. Rehmsmeier, "The precision-recall plot is more informative than the ROC plot when evaluating binary classiﬁers on imbalanced datasets.," PloS one, vol. 10, no. 3, p. e0118432, 2015

2015
[45]

NIH HPC Systems,

NLM, "NIH HPC Systems," [Online]. Available: https://hpc.nih.gov/systems/. [Accessed 24 Feb 2025]

2025
[2023]

Available: https://www.nlm.nih.gov/mesh/pubtypes.html

[Online]. Available: https://www.nlm.nih.gov/mesh/pubtypes.html. [Accessed 3 June 2024]. 23

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Its signiﬁcance lies not only in its vast scope but also in its ability to organize and provide eﬃcient access to this wealth of information

Introduction The MEDLINE indexed subset of the National Library of Medicine’s (NLM’s) PubMed service is a cornerstone of biomedical knowledge, housing millions of citations from journals worldwide. Its signiﬁcance lies not only in its vast scope but also in its ability to organize and provide eﬃcient access to this wealth of information. MEDLINE's indexin...

work page

[2] [2]

Clinical Trials

Previous work 2.1. MEDLINE and PT Tags MEDLINE organizes biomedical citations with MeSH PTs [1] [2], facilitating targeted searches. PTs categorize documents by research methodologies and publication formats, such as "Clinical Trials" or "Review," rather than content. Previous work has shown the utility of MEDLINE Publication Type tags to perform a variet...

work page 2024

[3] [3]

Journal Article

Challenges in PT Prediction Accurate prediction of MeSH Publication Types (PTs) from MEDLINE citation metadata presents several signiﬁcant challenges, which can be broadly categorized into three groups: Data-Related Issues: A signiﬁcant challenge lies in the characteristics of the data itself. Our corpus consists of over 5 million MELDINE citations manual...

work page 2016

[4] [4]

Preprint

Methodology We used two distinct model architectures: monolithic multilabel classiﬁers and an ensemble of binary classiﬁers. Both required similar data preprocessing steps, with some diﬀerences as described below. 4.1. Data Sources and Preparation 4.1.1. Data Sources The dataset for this research was derived from MEDLINE, an indexed subset of PubMed, whic...

work page 2016

[5] [5]

Journal Article

Results The experiments provided valuable insights into the performance of selective predictive models and the eﬀectiveness of various strategies for addressing challenges in PT prediction. Below, we elaborate on the ﬁndings for the major types of models used and the role of stratiﬁed sampling in enhancing performance. 5.1. Monolithic Multi-Label Classiﬁe...

work page

[6] [6]

Journal Article

Discussion 6.1. Interpretation of Results The experimental results demonstrate that modern Transformer-based models can signiﬁcantly enhance the accuracy and eﬃciency of PT tagging for MEDLINE citations. Both the ensemble of 19 binary classiﬁers and the monolithic multi-label classiﬁer showed notable improvements in recall compared to legacy systems, ensu...

work page

[7] [7]

Journal Article

Conclusion This study demonstrates the feasibility and advantages of using Transformer-based machine learning models to predict MeSH Publication Types (PTs) for MEDLINE citations. The results highlight the substantial opportunities that these technologies oﬀer to improve the automated indexing of MEDLINE citations. 7.1. Summary of Findings The ensemble of...

work page

[8] [8]

Acknowledgments This work was supported by the Lister Hill National Center for Biomedical Communications of the National Library of Medicine (NLM), National Institutes of Health

work page