REVIEW 3 major objections 5 minor 44 references
On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts
T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A few-shot-prompted Llama 3.1 sorts 59,344 scientific texts into five research areas with 0.82 accuracy, beating the reported BERT baseline of 0.74 without any fine-tuning.
desk verdict A useful empirical case study of LLMs for coarse research-area tagging, but the head-to-head accuracy claim against BERT is not fully supported as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a few-shot prompt that hands the model the ORKG five-domain taxonomy as a hierarchy and one fully worked example, plus the temperature parameter set to 0.8. The prompt turns the LLM's pretrained world knowledge into a zero-training classifier, while temperature tunes the randomness of the generated label. The evaluation harness compares models on the same 59,344 title-abstract records, using accuracy against the human-assigned FORC labels as the yardstick. The baseline machinery is a standard BERT classifier and a BiLSTM, whose reported accuracies of 0.74 and 0.66 are the numbers the 0.82 must beat.
What would settle it
Reproduce the experiment with BERT and BiLSTM fine-tuned on a train/test split of the same 59,344 title-abstract pairs and the same five-class accuracy definition; if a properly tuned BERT reaches or exceeds 0.82, the claimed LLM advantage is an artifact of an unequal baseline.
Extended reading notes
Core claim
The paper claims that off-the-shelf open-weights LLMs can classify scientific texts into the five top-level research areas of the ORKG taxonomy without any fine-tuning. On the FORC dataset of 59,344 title-abstract pairs, Llama 3.1 (70B) with a few-shot prompt at temperature 0.8 reaches accuracy 0.82, 0.08 above the reported BERT accuracy of 0.74 and 0.16 above BiLSTM's 0.66. Two prompt types (zero-shot and few-shot) and temperatures from 0.2 to 1.0 were tested across four models, with few-shot prompting and the 0.8 temperature yielding the best results and temperature 1.0 decreasing quality. The authors frame the result as evidence that modern LLMs can replace supervised classifiers for coarse-grained scholarly tagging.
Load-bearing premise
The reported advantage of 0.82 over 0.74 assumes the BERT and BiLSTM baselines were trained and measured under the same conditions as the LLMs, but the paper gives no fine-tuning protocol, train/test split, or metric definition for them.
Editorial extensions
If this is right
- Institutions running institutional repositories can deploy an off-the-shelf LLM as a first-pass subject tagger, reserving human effort for ambiguous cases.
- Few-shot prompting with a taxonomy excerpt emerges as a cheap alternative to fine-tuning for coarse-grained classification, at least within the five top-level domains tested.
- The temperature result suggests that too little output randomness undercuts performance and too much randomness at temperature 1.0 also hurts, so parameter selection matters for this task.
- The approach is claimed to extend to the lower two levels of the ORKG taxonomy and to other schemes such as ACM or Dewey Decimal Classification, though these extensions were not run in the paper.
Reading between the lines
- If the result replicates under a controlled baseline comparison, the practical import is that coarse research-area tagging becomes a prompt-engineering task rather than a model-training task, lowering the barrier for smaller institutions.
- Because the evaluation uses only title and abstract, accuracy likely varies with abstract length and completeness; a stratified check would show whether the 0.82 is carried by well-formed abstracts.
- The published comparison may overstate the gap: with fine-tuning details for BERT unspecified, a properly tuned BERT on the same five-class split could close or reverse the 0.08 difference.
- A class-balanced evaluation is missing; with five highly imbalanced top-level domains, macro-F1 could be much lower than accuracy, so the LLM's practical advantage may concentrate in the majority classes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates four open-source LLMs (Llama 3.1 70B, Gemma 2 27B, Mistral Nemo 12B, and Phi 3.5 3.8B) on the FORC dataset, using zero-shot and few-shot prompts at five temperatures to classify 59,344 scientific titles and abstracts into five ORKG top-level research areas. It reports that Llama 3.1 with a few-shot prompt at temperature 0.8 achieves 0.82 accuracy, exceeding the reported BERT accuracy of 0.74 and BiLSTM accuracy of 0.66, and concludes that off-the-shelf LLMs can match or surpass fine-tuned transformer classifiers on this task. The paper also includes a small manual error analysis and releases code and data.
Significance. If the comparison were properly controlled, the result would be practically useful: it would show that a parameterized prompt and temperature sweep on a large open model can replace fine-tuned encoders for coarse scholarly tagging, with implications for libraries and research information systems. The paper's strengths are its use of a realistic public dataset, its coverage of several open-weight models, and its explicit release of data and code. However, the central quantitative claim currently rests on an unreported baseline evaluation protocol and a post-hoc temperature selection on the same data used for the headline accuracy. These are fixable methodological gaps rather than fatal flaws, but they are load-bearing for the paper's main comparison.
major comments (3)
- [§5.3 and Table 4] The baseline evaluation protocol is not specified: there is no train/test split, no fine-tuning procedure, no hyperparameter settings, no epoch count, and no definition of the accuracy metric. The statement that BERT and BiLSTM were 'implemented using the models available on this platform' is insufficient to establish that the 0.74 and 0.66 numbers in Table 4 were obtained under conditions comparable to the LLM evaluation. If the baselines were evaluated on the same 59,344 texts used to tune the LLM temperature, or if they were not fine-tuned on FORC training data, the headline 0.82-versus-0.74 gap is an artifact. Please report the exact split, training details, and metric, and ideally provide per-class accuracy and standard errors.
- [§6 and Tables 5–6] Temperature selection is performed on the same data as the headline evaluation. Section 6 states that all models were evaluated 'with all nearly 60.000 titles and abstracts' and that the temperature that 'seems to work best' was then reported. This makes the 0.82 figure the maximum over five evaluated temperatures rather than an independent test result, which inflates the expected out-of-sample accuracy. Use a held-out validation split for model selection and report the test-set accuracy for the chosen configuration, with confidence intervals or significance tests against the BERT baseline.
- [Table 1] The label-matching rule is undefined. The zero-shot example shows Expected Output 'Social and Behavioral Sciences' while the model Output is 'Social Science'; if exact string matching is used, semantically correct labels are counted as errors, while if fuzzy or synonym matching is used, the rule must be disclosed. Please specify the normalization and matching procedure and report how many LLM outputs required normalization, since this directly affects every accuracy value in Tables 5 and 6.
minor comments (5)
- [Table 2] Table 2 states that Gemma has '27 million parameters' while the surrounding text gives 27.2B parameters; the correct value is 27.2 billion.
- [Concluding remarks] The concluding section refers to 'two baseline BERT models', but the baselines are BERT and BiLSTM, not two BERT variants.
- [Table 1] The few-shot prompt references a 'taxonomy of research field extracted from ORKG' but does not include the taxonomy hierarchy, making the few-shot condition non-reproducible from the paper alone; please either include the hierarchy or refer to a stable appendix.
- [Throughout] There are several minor inconsistencies and typos: 'LLaMa' versus 'Llama', 'temperature values from 0.2 up to 1.0' versus 'temperature 0.8', and a missing space in 'factors influencing' in Table 1.
- [References] The in-text citation 'Enamoto et al.' does not match the bibliography entry 'Enamoto'; please harmonize the spelling.
Circularity Check
Temperature is selected on the full test corpus, so the reported 0.82 is a post-hoc best-of-five rather than an independent prediction; no load-bearing self-citation or derivation-level circularity was found.
-
fitted input called prediction
[Section 6 (Results) and Section 5.1.1 (Dataset, Data Cleaning and Preprocessing)]
"We evaluated the baseline models and the selected LLMs according to the two explained prompts and temperature values, from 0.2 up to 1.0 with all nearly 60.000 titles and abstracts from the FORC dataset."
The headline accuracy of 0.82 is reported for temperature 0.8, which the paper identifies as the best among the five tested values on the same full corpus: 'after reaching the best performance at 0.8, quality starts decreasing'. Section 5.1.1 states that title and abstract from all 59,344 texts were used and accuracy was derived on this full set; no held-out split is described. Selecting the temperature on the evaluation set and then reporting its accuracy as the central result is a post-hoc maximum, so the 0.82-versus-0.74 comparison is not an independent prediction of a fixed configuration but a best-of-five selection on the test data. This is the fitted-input-called-prediction pattern, although it affects the headline number rather than the entire empirical comparison.
full rationale
This paper is an empirical evaluation rather than a mathematical derivation, so most circularity patterns do not apply. The only circularity-adjacent step is the temperature selection: the paper evaluates all temperature values on the full FORC corpus and then reports the best observed accuracy as the main result, with no held-out validation or fixed-protocol pre-registration. This makes the exact 0.82 figure statistically inflated and not an unbiased prediction, but it does not make the broader findings (few-shot outperforms zero-shot, larger models perform better, all LLMs beat the reported baselines) equivalent to the inputs by construction. The BERT and BiLSTM baseline accuracies are reported without a fine-tuning protocol or train/test split, which is a serious evaluation-comparability risk but not circularity. The self-citations to the authors' prior work are not load-bearing for the central claim. Overall, the paper has partial circularity in the headline number due to test-set temperature selection, but the core empirical content is not forced by definition or by a self-citation chain.
Assumptions & free parameters
free parameters (1)
- temperature =
0.8 (best of 0.2, 0.4, 0.6, 0.8, 1.0)
assumptions (4)
- domain assumption FORC labels are correct enough to serve as ground truth for accuracy measurement.
- domain assumption The five ORKG top-level domains are mutually exclusive and each text has one correct top-level research area.
- ad hoc to paper Baseline models were trained and evaluated in a way comparable to the LLM evaluation.
- domain assumption LLM outputs can be unambiguously matched to one of the five labels.
Cite this review
Pith. "Pith review of On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts." pith.science (2026). https://pith.science/paper/CKT4IFVE
@misc{pith2026250215745,
author = {Pith},
title = {Pith review of: On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts},
year = {2026},
howpublished = {\url{https://pith.science/paper/CKT4IFVE}},
note = {Machine review of arXiv:2502.15745}
}
read the original abstract
The rapid advancement of Large Language Models (LLMs) has led to a multitude of application opportunities. One traditional task for Information Retrieval systems is the summarization and classification of texts, both of which are important for supporting humans in navigating large literature bodies as they e.g. exist with scientific publications. Due to this rapidly growing body of scientific knowledge, recent research has been aiming at building research information systems that not only offer traditional keyword search capabilities, but also novel features such as the automatic detection of research areas that are present at knowledge intensive organizations in academia and industry. To facilitate this idea, we present the results obtained from evaluating a variety of LLMs in their ability to sort scientific publications into hierarchical classifications systems. Using the FORC dataset as ground truth data, we have found that recent LLMs (such as Meta Llama 3.1) are able to reach an accuracy of up to 0.82, which is up to 0.08 better than traditional BERT models.
Figures
Reference graph
Works this paper leans on
-
[1]
write newline
" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
-
[2]
Abburi, H., Suesserman, M., Pudota, N., Veeramani, B., Bowen, E., and Bhattacharya, S. (2023). Generative ai text classification using ensemble llm approaches. arXiv preprint arXiv:2309.07755
arXiv 2023
-
[3]
Abu Ahmad, R., Borisova, E., and Rehm, G. (2024). Forc@ nslp2024: Overview and insights from the field of research classification shared task. In International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs , pages 189--204. Springer
work page 2024
-
[4]
Al Nazi, Z., Hossain, M. R., and Al Mamun, F. (2025). Evaluation of open and closed-source llms for low-resource language with zero-shot, few-shot, and chain-of-thought prompting. Natural Language Processing Journal , page 100124
work page 2025
-
[5]
Auer, S. and Mann, S. (2019). Towards an open research knowledge graph. The Serials Librarian , 76(1-4):35--41
work page 2019
-
[6]
Bird, S., Dale, R., Dorr, B. J., Gibson, B. R., Joseph, M. T., Kan, M.-Y., Lee, D., Powley, B., Radev, D. R., Tan, Y. F., et al. (2008). The acl anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In LREC
work page 2008
-
[7]
Bornmann, L., Haunschild, R., and Mutz, R. (2021). Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanities and Social Sciences Communications , 8(1):1--15
work page 2021
-
[8]
Desale, S. K. and Kumbhar, R. M. (2014). Research on automatic classification of documents in library environment: a literature review. KO KNOWLEDGE ORGANIZATION , 40(5):295--304
work page 2014
Show all 44 references
-
[9]
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
2018 arXiv
-
[10]
R., Maia, R., Weigang, L., and Filho, G
Enamoto, L., Santos, A. R., Maia, R., Weigang, L., and Filho, G. P. R. (2022). Multi-label legal text classification with bilstm and attention. International Journal of Computer Applications in Technology , 68(4):369--378
2022
-
[11]
Feuerriegel, S., Hartmann, J., Janiesch, C., and Zschech, P. (2024). Generative ai. Business & Information Systems Engineering , 66(1):111--126
2024
-
[12]
and Paynter, G
Frank, E. and Paynter, G. W. (2004). Predicting library of congress classifications from library of congress subject headings. Journal of the American Society for Information Science and Technology , 55(3):214--227
2004
-
[13]
Gao, A. (2023). Prompt engineering for large language models. Available at SSRN 4504303
2023
-
[14]
B., D'Souza, J., and Auer, S
Giglou, H. B., D'Souza, J., and Auer, S. (2024). Llms4synthesis: Leveraging large language models for scientific synthesis. arXiv preprint arXiv:2409.18812
2024 arXiv
-
[15]
a ck, J., and Ard \
Golub, K., Hagelb \"a ck, J., and Ard \"o , A. (2020). Automatic classification of swedish metadata using dewey decimal classification: a comparison of approaches. Journal of Data and Information Science , 5(1):18--38
2020
-
[16]
Hacker, P., Engel, A., and Mauer, M. (2023). Regulating chatgpt and other large generative ai models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages 1112--1123
2023
-
[17]
and Knoth, P
Herrmannova, D. and Knoth, P. (2016). An analysis of the microsoft academic graph. D-lib Magazine , 22(9/10):37
2016
-
[18]
Huang, Z., Xu, W., and Yu, K. (2015). Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991
2015 arXiv
-
[19]
Jiang, M., D’Souza, J., Auer, S., and Downie, J. S. (2020). Improving scholarly knowledge representation: Evaluating bert-based models for scientific relation classification. In Digital Libraries at Times of Massive Societal Transition: 22nd International Conference on Asia-Pa...
2020
-
[20]
Kalyan, K. S. (2023). A survey of gpt-3 family large language models including chatgpt and gpt-4. Natural Language Processing Journal , page 100048
2023
-
[21]
Kinney, R., Anastasiades, C., Authur, R., Beltagy, I., Bragg, J., Buraczynski, A., Cachola, I., Candra, S., Chandrasekhar, Y., Cohan, A., et al. (2023). The semantic scholar open data platform. arXiv preprint arXiv:2301.10140
2023 arXiv
-
[22]
Liu, S., Yu, S., Lin, Z., Pathak, D., and Ramanan, D. (2024). Language models as black-box optimizers for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12687--12697
2024
-
[23]
Mahapatra, R., Gayan, M., Jamatia, B., et al. (2024). Artificial intelligence tools to enhance scholarly communication: An exploration based on a systematic review
2024
-
[24]
and Chiang, M
Morgan, J. and Chiang, M. (2024). Ollama . https://ollama.com. Online; accessed 6 August 2024
2024
-
[25]
Mosca, E., Abdalla, M. H. I., Basso, P., Musumeci, M., and Groh, G. (2023). Distinguishing fact from fiction: A benchmark dataset for identifying machine-generated scientific papers in the llm era. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (...
2023
-
[26]
Murphy, K. P. (2022). Probabilistic machine learning: an introduction . MIT press
2022
-
[27]
Nah, F., Cai, J., Zheng, R., and Pang, N. (2023). An activity system-based perspective of generative ai: Challenges and research directions. AIS Transactions on Human-Computer Interaction , 15(3):247--267
2023
-
[28]
A., and Chakraborty, C
Pal, S., Bhattacharya, M., Islam, M. A., and Chakraborty, C. (2024). Ai-enabled chatgpt or llm: a new algorithm is required for plagiarism-free scientific writing. International Journal of Surgery , 110(2):1329--1330
2024
-
[29]
and Po \'s wiata, R
Pere kiewicz, M. and Po \'s wiata, R. (2024). A review of the challenges with massive web-mined corpora used in large language models pre-training. arXiv preprint arXiv:2407.07630
2024 arXiv
-
[30]
Pertsas, V., Kasapaki, M., and Constantopoulos, P. (2024). An annotated dataset for transformer-based scholarly information extraction and linguistic linked data generation. In Proceedings of the 9th Workshop on Linked Data in Linguistics@ LREC-COLING 2024 , pages 84--93
2024
-
[31]
Rabby, G., Auer, S., D'Souza, J., and Oelen, A. (2024). Fine-tuning and prompt engineering with cognitive knowledge graphs for scholarly knowledge organization. arXiv preprint arXiv:2409.06433
2024 arXiv
-
[32]
Rous, B. (2012). Major update to acm's computing classification system. Communications of the ACM , 55(11):12--12
2012
-
[33]
Scott, M. L. (1998). Dewey decimal classification. Libraries Unlimited
1998
-
[34]
Shahi, G. K. and Hummel, O. (2024). Enhancing research information systems with identification of domain experts. In Proceedings of the Bibliometric-enhanced Information Retrieval Workshop (BIR) at the European Conference on Information Retrieval (ECIR 2024) , CEUR Workshop Pr...
2024
-
[35]
Shahi, G. K. and Nandini, D. (2020). Fake C ovid -- a multilingual cross-domain fact check news dataset for covid-19. In Proceedings of the 14th International AAAI C onference on W eb and S ocial M edia
2020
-
[36]
S., Love, J., et al
Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi \`e re, M., Kale, M. S., Love, J., et al. (2024). Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295
2024 arXiv
-
[37]
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971
2023 arXiv
-
[38]
Wang, J. (2009). An extensive study on automated dewey decimal classification. Journal of the American Society for Information Science and Technology , 60(11):2269--2286
2009
-
[39]
Wang, K., Shen, Z., Huang, C., Wu, C.-H., Dong, Y., and Kanakia, A. (2020). Microsoft academic graph: When experts are not enough. Quantitative Science Studies , 1(1):396--413
2020
-
[40]
Wang, S., Hu, T., Xiao, H., Li, Y., Zhang, C., Ning, H., Zhu, R., Li, Z., and Ye, X. (2024). Gpt, large language models (llms) and generative artificial intelligence (gai) models in geospatial science: a systematic review. International Journal of Digital Earth , 17(1):2353122
2024
-
[41]
Wiggins, B. J. (2009). Acquisitions and bibliographic access directorate library of congress report of fiscal year 2009 (fiscal year ended september 30, 2009)
2009
-
[42]
Young, J. S. and Lammert, M. (2024). Chatgpt for classification: Evaluation of an automated course mapping method in academic libraries
2024
-
[43]
Zhang, C., Tian, L., and Chu, H. (2023). Usage frequency and application variety of research methods in library and information science: Continuous investigation from 1991 to 2021. Information Processing & Management , 60(6):103507
2023
-
[44]
X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023). A survey of large language models. arXiv preprint arXiv:2303.18223
2023 arXiv
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.