Pith. sign in

REVIEW 4 major objections 4 minor 33 references

Advancing Scientific Text Classification: Fine-Tuned Models with Dataset Expansion and Hard-Voting

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding query-retrieved articles labeled by model agreement improves scientific text classification, with SciBERT reaching micro F1 0.8924 on the expanded WoS-53949 set.

desk verdict Pseudo-labeled test data make the reported gains untrustworthy, but the paper's reproducible artifacts and clean write-up are worth acknowledging. read the letter →

arxiv 2504.19021 v2 pith:35CGQBSO submitted 2025-04-26 cs.CL cs.AI

Zhyar Rzgar K Rostam , Gábor Kertész This is my paper · ORCID
classification cs.CLcs.AI
keywords scientifictextclassificationpre-trainedlanguagemodelsdatasetaugmentationhard-votingWebofSciencefine-tuningdomain-specificearlystopping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that scientific text classification can be improved without new human annotation by pulling additional articles from the Web of Science database and labeling them with a hard-voting ensemble of four fine-tuned language models. On the expanded WoS-53949 dataset, the best model, SciBERT, reports micro F1 0.8924, beating BERT and earlier hierarchical deep-learning baselines such as HDLTex. The result matters because curated labels are the main bottleneck in scientific document organization; if pseudo-labeled retrieval works, larger and more current training sets become cheap.

What carries the argument

The mechanism is a three-stage loop. First, seven targeted queries retrieve 1,000 candidate articles per WoS category. Second, four fine-tuned pre-trained language models each predict a label, and a hard-voting rule, which assigns the category receiving the most model votes, keeps only high-agreement articles. Third, the newly labeled articles are merged with the original WoS datasets, and the models are fine-tuned on the expanded sets using AdamW, a dynamic learning-rate search over four values, linear warmup, and early stopping on validation F1.

What would settle it

Have human annotators label a random sample of the 6,964 added articles, compare their labels with the hard-voted labels, and if agreement is low, retrain on human-corrected labels; the central claim survives only if the F1 gains persist.

Watch

Extended reading notes

Core claim

Fine-tuning pre-trained language models on WoS-46985 plus 6,964 query-retrieved articles, whose labels come from agreement among BERT, SciBERT, BioBERT, and BlueBERT, raises classification performance over the original data alone. The paper reports SciBERT at micro F1 0.8924 on WoS-53949, BioBERT at 0.9247 on WoS-18932, and BERT at 0.9782 on WoS-8716, with domain-specific models generally ahead of general-purpose BERT. These results are presented as outperforming the HDLTex hierarchical deep-learning baseline and the earlier fine-tuning study on the original datasets.

Load-bearing premise

The load-bearing premise is that the hard-voted pseudo-labels assigned to the roughly 7,000 added articles are correct enough to serve as ground truth in both the training and test splits.

Editorial extensions

If this is right

  • On the largest expanded set, WoS-53949, SciBERT reaches micro F1 0.8924, above BERT's 0.8764 and above the 76.58 percent accuracy reported for HDLTex on the original WoS-46985.
  • Domain-specific models (SciBERT, BioBERT) outperform general-purpose BERT on most expanded datasets, supporting the paper's claim that scientific pre-training plus broader category coverage carries the gain.
  • The expansion recipe works across all three WoS sizes, with the smallest set WoS-8716 reaching micro F1 above 0.97 for three of the four models.
  • Using pseudo-labeled retrieved articles as additional training data, combined with dynamic learning rates and early stopping, is presented as a scalable way to keep classifiers current without new manual annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Since the hard-voted labels come from the same models that are later evaluated on the expanded test split, the reported F1 gains may partly reflect agreement with the models' own biases; a human-annotated sample of the added articles would show how much of the gain is real category accuracy.
  • The same query-retrieve-vote-then-fine-tune loop could transfer to other taxonomies, such as arXiv subject classes or PubMed topics, where the bottleneck is also curated labels.
  • The paper's finding that keyword-only and abstract-only predictions behave differently across domains suggests a testable variant: fuse the two input views before voting, not after.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes expanding three Web of Science classification datasets (WoS-5736, WoS-11967, WoS-46985) by querying the WoS database for 1,000 articles per target category, labeling the retrieved unlabeled articles via hard voting over the predictions of four PLMs (BERT, SciBERT, BioBERT, BlueBERT) that were fine-tuned in the authors' prior work, and then fine-tuning the same four PLMs on the expanded datasets. The authors report that the expanded datasets improve classification performance, with the best result being SciBERT at micro F1 0.8924 on WoS-53949 (Table VII), and that their models outperform HDLTex and other baselines (Table VIII). They attribute the gains to dataset augmentation, hard-voting, dynamic learning rates, and early stopping.

Significance. If the central claim were sound, the paper would offer a practical recipe for cheap dataset expansion: using model-generated labels plus hard voting to improve scientific text classification without manual annotation, with evidence that domain-specific PLMs benefit. The authors also provide a GitHub repository, which is helpful for reproducibility. However, the experimental design does not support the claim as it stands: the pseudo-labels are used as ground truth for both training and test, and the headline comparison against HDLTex mixes changes in label source with changes in test-set composition. The practical significance is therefore not established by the reported experiments.

major comments (4)
  1. [III.B, IV.B, V.D (Tables IV and VII)] The evaluation is circular. The 6,964 articles added to WoS-46985 to form WoS-53949 are labeled by hard voting over the predictions of the fine-tuned PLMs from [20] (Sections III.B and IV.B). Table IV then splits the combined WoS-53949 into 80/10/10, so the test set of 10,790 documents contains on average about 1,393 model-labeled articles. Fine-tuning on this expanded dataset and testing on the same split therefore measures agreement with the earlier models' predictions rather than accuracy with respect to true categories. The reported micro-F1 values in Table VII cannot be read as classification accuracy on genuinely labeled data.
  2. [Table VIII and Section V.D] The comparison against baselines is confounded. Table VIII compares the expanded-dataset results (WoS-53949, WoS-18932, WoS-8716) with HDLTex and other methods evaluated on the original WoS-46985, WoS-11967, and WoS-5736 datasets, and with the authors' prior fine-tuning results [20] on the original datasets. Because the test sets differ in composition (original vs. expanded) and in label source (human labels vs. model-generated hard-voted labels), any difference in accuracy or F1 could be caused by label distribution or test-set shift rather than by dataset expansion. A controlled comparison would fine-tune the same models on the original dataset with the identical 80/10/10 split and report both training-only and expanded-training results on the same held-out test partition.
  3. [IV.B and Table VI] No evidence is provided that the hard-voted pseudo-labels are accurate. Table VI compares model predictions with 'query classes,' but query classes are not ground-truth labels: a query for a category returns articles that match the query terms, not necessarily articles belonging exclusively to that category. Agreement with query classes is therefore a weak and potentially biased proxy. Without human evaluation or a held-out gold-standard subset, the hard-voting step (Figure 1) may be selecting confident errors, and the label noise in the expanded dataset is unquantified.
  4. [VI and Table VII] The paper claims in Section VI that the hard-voting strategy improved classification accuracy, but no ablation is reported. The reported results compare models fine-tuned on the expanded dataset with models fine-tuned on the original dataset, and the expanded dataset differs from the original in both size and label source; the effect of hard voting per se is not isolated. A comparison between training on labels from a single model and training on hard-voted labels, with the same added articles, would be needed to support this contribution.
minor comments (4)
  1. [III.B] The text says the expanded datasets contain 8,715, 18,932, and 53,949 unique documents, but Tables I--III and Table IV use 8,716, 18,932, and 53,949; the first number should be corrected.
  2. [V.F] The text says BERT achieved the highest micro F1 (0.9782), but the following sentences report SciBERT and BioBERT at 0.9816, which are higher; the sentences are internally inconsistent.
  3. [Table VII] Many rows report identical micro F1, micro recall, and micro precision; given class imbalance and potential label noise, this should be explained or the numbers should be reported to more decimal places.
  4. [Throughout] The terms 'LLM' and 'PLM' are used interchangeably, but BERT-class models are encoder-only pre-trained language models; the terminology should be made consistent and precise.

Circularity Check

2 steps flagged · score 7.0 of 10

Expansion labels are model predictions that end up in the test set, so the reported F1 partly measures self-consistency rather than true classification accuracy.

  1. fitted input called prediction [Section III.B (Constructed Dataset), Section IV.B (Hard-voting and Models Agreement), Section IV.D (Data Preparation), Section V.D (Experimental Results: WoS-53949)]
    "Multi-model inference and hard-voting were used to assign labels based on model agreement. ... The dataset was divided into training (80%), testing (20%), and validation (50% of the testing set, which is equal to 10% of the dataset) as presented in Table IV. ... The experimental results on the WoS-53949 dataset demonstrated strong performance across all four PLMs. SciBERT emerged as the top-performing model, achieving micro F1 0.8924."

    The hard-voted labels are model outputs, not human annotations. The retrieved articles are labeled by the same PLM families that are later fine-tuned and evaluated, and these pseudo-labeled articles are merged with the original WoS data before the 80/10/10 split (Table IV). Roughly 13% of the WoS-53949 test set (about 1,393 of 10,790 items) therefore has test labels that are hard-voted predictions of the very models under test. The reported micro-F1, including SciBERT's 0.8924, consequently measures, in part, how well the fine-tuned models reproduce the labeler's outputs that were used as training labels rather than agreement with true categories for those items. The claimed improvement over the original WoS-46985 results is thus confounded with the presence of self-generated test labels.

  2. self citation load bearing [Section IV.A (Models Inference)]
    "We utilized the fine-tuned models (BERT, SciBERT, BioBERT, and BlueBERT) obtained from the baseline study by Rostam and Kertész [20] to perform document classification and category prediction for retrieved unseen datasets."

    The label-generation stage depends entirely on the authors' own earlier fine-tuned models, cited as [20]. No independent ground truth (manual annotation, external classifier, or held-out human labels) is used to verify the hard-voted labels before they enter the training and test sets. The later fine-tuned models are then tested on these same self-generated labels. The self-citation is load-bearing because the validity of the expanded dataset's labels, and hence of the central comparison, rests on the prior model outputs from the same research group rather than on an externally verified label source.

full rationale

The central derivation chain is: retrieve articles with targeted queries, label them by hard-voting the predictions of the authors' own fine-tuned PLMs [20], merge these pseudo-labeled articles with WoS-46985, fine-tune the same PLM families on the expanded set, and report F1 on a random 20% test split of the expanded set. Because the test split includes pseudo-labeled articles whose labels are model outputs, the headline F1 values (e.g., 0.8924 for SciBERT on WoS-53949) measure, in part, the models' ability to reproduce the hard-voted labels they were trained on. This is partial circularity: the 'prediction' target is derived from the same model family that is being evaluated. The comparison to HDLTex and to the prior WoS-46985 results in Table VIII is additionally confounded by the different train/test composition, so the claimed superiority over baselines is not established. The paper is not wholly circular because the majority of test items still carry original human labels, but the expansion-specific component of the result is not independently validated. No other circular steps (e.g., ansatz-smuggling or uniqueness-importing) were found.

Assumptions & free parameters 2 free parameters · 2 assumptions · 0 invented entities

The approach assumes that the models used to label new data are already reliable and that WoS query output is topically pure. No free parameter is fitted to external ground truth, but the design choices (retrieval count, committee composition) are unanalyzed and the label source is circular.

free parameters (2)
  • Retrieved articles per category = 1000
    The number of articles pulled per WoS query is set to 1,000 without sensitivity analysis; it directly controls the amount of pseudo-labeled data added.
  • Hard-voting committee size = 4
    Four fixed PLMs vote; no analysis shows how the result depends on the number or choice of voters.
assumptions (2)
  • domain assumption Query keywords in WoS retrieval yield articles whose true category matches the query class
    Used in Section V.A-C as the reference for inference 'accuracy'; keyword retrieval does not guarantee topical purity.
  • domain assumption Pseudo-labels produced by hard voting are correct enough to serve as ground truth for training and testing
    The entire expansion procedure in Sections III.B and IV.B depends on this; without independent validation, errors in pseudo-labels propagate into the reported F1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Advancing Scientific Text Classification: Fine-Tuned Models with Dataset Expansion and Hard-Voting." pith.science (2026). https://pith.science/paper/35CGQBSO

@misc{pith2026250419021,
  author       = {Pith},
  title        = {Pith review of: Advancing Scientific Text Classification: Fine-Tuned Models with Dataset Expansion and Hard-Voting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/35CGQBSO}},
  note         = {Machine review of arXiv:2504.19021}
}
read the original abstract

Efficient text classification is essential for handling the increasing volume of academic publications. This study explores the use of pre-trained language models (PLMs), including BERT, SciBERT, BioBERT, and BlueBERT, fine-tuned on the Web of Science (WoS-46985) dataset for scientific text classification. To enhance performance, we augment the dataset by executing seven targeted queries in the WoS database, retrieving 1,000 articles per category aligned with WoS-46985's main classes. PLMs predict labels for this unlabeled data, and a hard-voting strategy combines predictions for improved accuracy and confidence. Fine-tuning on the expanded dataset with dynamic learning rates and early stopping significantly boosts classification accuracy, especially in specialized domains. Domain-specific models like SciBERT and BioBERT consistently outperform general-purpose models such as BERT. These findings underscore the efficacy of dataset augmentation, inference-driven label prediction, hard-voting, and fine-tuning techniques in creating robust and scalable solutions for automated academic text classification.

Figures

Figures reproduced from arXiv: 2504.19021 by the authors.

Figure 1
Figure 1. PLMs agreement results across both scenarios (abstract and keywords) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 4 canonical work pages

  1. [20]

    Fine-tuning large language models for scientific text classification: A comparative study,

    Z. R. K. Rostam and G. Kert ´esz, “Fine-tuning large language models for scientific text classification: A comparative study,” inProceedings of the 6th IEEE International Symposium on Logistics and Industrial Informatics (LINDI), (Karaganda, Kazakhstan), pp. 233–238, IEEE, October 23–25 2024

    work page 2024

  2. [1]

    A survey of text classification with transformers: How wide? how large? how long? how accurate? how expensive? how safe?,

    J. Fields, K. Chovanec, and P. Madiraju, “A survey of text classification with transformers: How wide? how large? how long? how accurate? how expensive? how safe?,”IEEE Access, 2024

    2024

  3. [2]

    Improving imbalanced scientific text classification using sampling strategies and dictionaries,

    L. Borrajo, R. Romero, E. L. Iglesias, and C. R. Marey, “Improving imbalanced scientific text classification using sampling strategies and dictionaries,”Journal of integrative bioinformatics, vol. 8, no. 3, pp. 90– 104, 2011

    work page 2011

  4. [3]

    Exploring small language models with prompt-learning paradigm for efficient domain-specific text classifica- tion,

    H. Luo, P. Liu, and S. Esping, “Exploring small language models with prompt-learning paradigm for efficient domain-specific text classifica- tion,”arXiv preprint arXiv:2309.14779, 2023

    arXiv 2023

  5. [4]

    Multilabel text classification with label-dependent representation,

    R. Alfaro, H. Allende-Cid, and H. Allende, “Multilabel text classification with label-dependent representation,”Applied Sciences, vol. 13, no. 6, 2023

    2023

  6. [5]

    Novel deep learning approach for sci- entific literature classification,

    M. M. Ahanger and M. A. Wani, “Novel deep learning approach for sci- entific literature classification,” in2022 9th International Conference on Computing for Sustainable Global Development (INDIACom), pp. 249– 254, IEEE, 2022

    2022

  7. [6]

    Benchmarking for biomedical natural language processing tasks with a domain specific albert,

    U. Naseem, A. G. Dunn, M. Khushi, and J. Kim, “Benchmarking for biomedical natural language processing tasks with a domain specific albert,”BMC bioinformatics, vol. 23, no. 1, p. 144, 2022

    2022

  8. [7]

    A brief survey of text classification methods,

    Q. Jiao, “A brief survey of text classification methods,” in2023 IEEE 3rd International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA), vol. 3, pp. 1384–1389, IEEE, 2023

    2023

Show all 33 references
  1. [8]

    Deep learning–based text classification: a comprehensive review,

    S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep learning–based text classification: a comprehensive review,”ACM computing surveys (CSUR), vol. 54, no. 3, pp. 1–40, 2021

    2021

  2. [9]

    Survey of pre-trained models for natural language processing,

    J. Peng and K. Han, “Survey of pre-trained models for natural language processing,” in2021 International Conference on Electronic Communi- cations, Internet of Things and Big Data (ICEIB), pp. 277–280, IEEE, 2021

    2021

  3. [10]

    Text clas- sification via large language models,

    X. Sun, X. Li, J. Li, F. Wu, S. Guo, T. Zhang, and G. Wang, “Text clas- sification via large language models,”arXiv preprint arXiv:2305.08377, 2023

    2023 arXiv

  4. [11]

    Cross-domain limi- tations of neural models on biomedical relation classification,

    I. Alimova, E. Tutubalina, and S. I. Nikolenko, “Cross-domain limi- tations of neural models on biomedical relation classification,”IEEE Access, vol. 10, pp. 1432–1439, 2021

    2021

  5. [12]

    Finbert: Financial sentiment analysis with pre-trained lan- guage models,

    D. Araci, “Finbert: Financial sentiment analysis with pre-trained lan- guage models,”arXiv preprint arXiv:1908.10063, 2019

    1908 arXiv

  6. [13]

    Sentiment analysis with neural models for hungarian,

    L. J. Laki and Z. G. Yang, “Sentiment analysis with neural models for hungarian,”Acta Polytechnica Hungarica, vol. 20, no. 5, 2023

    2023

  7. [14]

    Topic modeling: a comprehensive review,

    P. Kherwa and P. Bansal, “Topic modeling: a comprehensive review,” EAI Endorsed transactions on scalable information systems, vol. 7, no. 24, 2019

    2019

  8. [15]

    Inte- grating text classification into topic discovery using semantic embedding models,

    A. L. Lezama-S ´anchez, M. Tovar Vidal, and J. A. Reyes-Ortiz, “Inte- grating text classification into topic discovery using semantic embedding models,”Applied Sciences, vol. 13, no. 17, p. 9857, 2023

    2023

  9. [16]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong,et al., “A survey of large language models,”arXiv preprint arXiv:2303.18223, 2023

    2023 arXiv

  10. [17]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  11. [18]

    Scibert: A pretrained language model for scientific text,

    I. Beltagy, K. Lo, and A. Cohan, “Scibert: A pretrained language model for scientific text,”arXiv preprint arXiv:1903.10676, 2019

    1903 arXiv

  12. [19]

    Biobert: a pre-trained biomedical language representation model for biomedical text mining,

    J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, “Biobert: a pre-trained biomedical language representation model for biomedical text mining,”Bioinformatics, vol. 36, no. 4, pp. 1234–1240, 2020

    2020

  13. [21]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  14. [22]

    Finbert: A pre-trained financial language representation model for financial text mining,

    Z. Liu, D. Huang, K. Huang, Z. Li, and J. Zhao, “Finbert: A pre-trained financial language representation model for financial text mining,” in Proceedings of the twenty-ninth international conference on interna- tional joint conferences on artificial intelligence, pp. 4513–4519, 2021

    2021

  15. [23]

    Structured information extraction from complex scientific text with fine-tuned large language models,

    A. Dunn, J. Dagdelen, N. Walker, S. Lee, A. S. Rosen, G. Ceder, K. Pers- son, and A. Jain, “Structured information extraction from complex scientific text with fine-tuned large language models,”arXiv preprint arXiv:2212.05238, 2022

    2022 arXiv

  16. [24]

    Matscibert: A materials domain language model for text mining and information extraction,

    T. Gupta, M. Zaki, N. A. Krishnan, and Mausam, “Matscibert: A materials domain language model for text mining and information extraction,”npj Computational Materials, vol. 8, no. 1, p. 102, 2022

    2022

  17. [25]

    Transfer learning in biomedical natural lan- guage processing: An evaluation of bert and elmo on ten benchmarking datasets,

    Y . Peng, S. Yan, and Z. Lu, “Transfer learning in biomedical natural lan- guage processing: An evaluation of bert and elmo on ten benchmarking datasets,” inProceedings of the 2019 Workshop on Biomedical Natural Language Processing (BioNLP 2019), pp. 58–65, 2019

    2019

  18. [26]

    Medibiodeberta: Biomedical lan- guage model with continuous learning and intermediate fine-tuning,

    E. Kim, Y . Jeong, and M.-s. Choi, “Medibiodeberta: Biomedical lan- guage model with continuous learning and intermediate fine-tuning,” IEEE Access, 2023

    2023

  19. [27]

    The classification of short scientific texts using pretrained bert model,

    G. Danilov, T. Ishankulov, K. Kotik, Y . Orlov, M. Shifrin, and A. Potapov, “The classification of short scientific texts using pretrained bert model,” inPublic Health and Informatics, pp. 83–87, IOS press, 2021

    2021

  20. [28]

    Ensemble learning with pre-trained transformers for crash severity classification: A deep nlp approach,

    S. Jaradat, R. Nayak, A. Paz, and M. Elhenawy, “Ensemble learning with pre-trained transformers for crash severity classification: A deep nlp approach,”Algorithms, vol. 17, no. 7, p. 284, 2024

    2024

  21. [29]

    Eda: Easy data augmentation techniques for boosting performance on text classification tasks,

    J. Wei and K. Zou, “Eda: Easy data augmentation techniques for boosting performance on text classification tasks,”arXiv preprint arXiv:1901.11196, 2019

    1901 arXiv

  22. [30]

    Hdltex: Hierarchical deep learning for text classification,

    K. Kowsari, D. E. Brown, M. Heidarysafa, K. Jafari Meimandi, , M. S. Gerber, and L. E. Barnes, “Hdltex: Hierarchical deep learning for text classification,” inMachine Learning and Applications (ICMLA), 2017 16th IEEE International Conference on, IEEE, 2017

    2017

  23. [31]

    A survey of active learning for text classi- fication using deep neural networks,

    C. Schr ¨oder and A. Niekler, “A survey of active learning for text classi- fication using deep neural networks,”arXiv preprint arXiv:2008.07267, 2020

    2008 arXiv

  24. [32]

    Recurrent convolutional neural networks for text classification,

    S. Lai, L. Xu, K. Liu, and J. Zhao, “Recurrent convolutional neural networks for text classification,” inProceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2267–2273, 2015

    2015

  25. [33]

    Sequential short-text classification with recurrent and convolutional neural networks,

    J. Y . Lee and F. Dernoncourt, “Sequential short-text classification with recurrent and convolutional neural networks,”arXiv preprint arXiv:1603.03827, 2016

    2016 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.