Pith. sign in

REVIEW 3 major objections 6 minor 51 references

A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper's central claim is that per-category binary fine-tuned transformer classifiers, trained on 3,000 majority-vote human-annotated sentences, can label the 1.6-million-sentence corpus of grant peer-review reports with an average…

desk verdict Solid, reproducible applied NLP pipeline for grant review text with an evaluation-split flaw that likely inflates the headline F1 scores, but the central qualitative findings survive. read the letter →

arxiv 2411.16662 v2 pith:F6KD7WUR submitted 2024-11-25 econ.EM

Gabriel Okasa , Alberto de León , Michaela Strinzel , Anne Jorstad , Katrin Milzow , Matthias Egger , Stefan Müller This is my paper · ORCID
classification econ.EM
keywords grantpeerreviewtextclassificationfine-tunedtransformerssentence-levelannotationcontentanalysisfundingevaluationinter-coderagreementclassimbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to establish that the content of grant peer-review reports can be made machine-readable at scale. Its recipe is to define twelve categories that a funding agency cares about—evaluation criteria, what the reviewer focuses on, and whether statements are positive, negative, justified, or suggest improvements—then have instructed human annotators label 3,000 sentences, aggregate their judgments by majority vote, and fine-tune a separate transformer classifier for each category. The headline result is an average macro F1 of 0.85 (a balanced precision–recall average) across the twelve binary classifiers, with the hardest category (Rationale) at 0.71 and the easiest (Applicant: Quantity) at 0.93. If this performance holds on new reports, the pipeline makes the full corpus of roughly 1.6 million sentences available for structured audit, letting funders measure how reviewers allocate attention, how often they give reasons, and how often they offer suggestions. The paper also argues for a design principle: per-category binary models beat shared multi-label and multi-task architectures, and the quality of the human-annotated training data matters more than which pre-trained transformer backbone is selected.

What carries the argument

The machinery is a set of twelve fine-tuned binary transformer classifiers, one per category, built on a pre-trained scientific-text encoder (SPECTER2, a BERT-style model pre-trained on scientific papers with citation-graph signals). For each sentence the encoder produces a CLS-token embedding, which a linear output layer with sigmoid activation maps to a class probability, and the loss is binary cross-entropy minimized by AdamW over three epochs. The training signal comes from the annotation procedure: an eleven-round codebook refinement, four instructed annotators, three labels per sentence, and majority-vote aggregation into the final 3,000-sentence gold set. The method comparison is part of the machinery: it contrasts the twelve binary heads with one multi-label head and one shared-encoder multi-task setup, establishing which design generalizes.

What would settle it

Re-run the classifier evaluation with a report-stratified split—all sentences from any given review report placed entirely in training or entirely in test—and compare the average macro F1 to the reported 0.85; a large drop would show that the headline accuracy depends on within-report leakage.

Watch

Extended reading notes

Core claim

The paper claims that a supervised pipeline—iterated codebook development, multi-annotator consensus labeling, and per-category fine-tuned transformer classifiers—can identify twelve content categories in individual sentences of grant peer-review reports. The strongest evidence is the performance of twelve separately fine-tuned binary classifiers: macro F1 ranges from 0.71 for Rationale to 0.93 for Applicant: Quantity, averaging 0.85, and five-fold cross-validation confirms the same ordering. The comparison with multi-label (0.73) and multi-task (0.62) classifiers shows that separate binary models are the decisive design choice. The comparison across three pre-trained transformer backbones, all reaching the same 0.85 average, indicates that the annotated data rather than the model family drives accuracy. The paper also reports that a few-shot large-language-model baseline reaches only 0.70 and fails on rare categories, so the pipeline retains human annotation plus fine-tuned transformers.

Load-bearing premise

The evaluation randomly splits the 3,000 annotated sentences into training and test sets without grouping by source review report, so sentences from the same report can appear in both sets and the model can learn reviewer-specific wording that would not generalize to a new report.

Editorial extensions

If this is right

  • If the central claim holds, the funder's full corpus of about 1.6 million review sentences can be labeled automatically at an average macro F1 of 0.85, making category-level monitoring feasible in practice.
  • Binary, category-specific models are the recommended architecture: they outperform multi-label (0.73) and multi-task (0.62) alternatives and, unlike the shared models, their F1 scores are not correlated with how common the category is.
  • Because three different pre-trained transformer backbones all average 0.85, the human-annotation data is what carries performance; further gains should come from better labels, not bigger models.
  • Few-shot prompting of an 8-billion-parameter language model achieves only 0.70 average macro F1 and performs especially poorly on rare categories, so LLM-generated annotations cannot replace the human-consensus gold set.
  • Sentence-level classification is the right unit: adding surrounding context for the hardest category, Rationale, leaves F1 at 0.71, so the pipeline can stay sentence-based and cheap to scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A report-stratified split, not tested in the paper, would reveal how much of the 0.85 F1 comes from content versus reviewer-specific phrasing; if the drop on held-out reviews is large, the models would need calibration before use on a new review cohort.
  • The strong negative correlation between annotators' perceived difficulty and F1 (-0.73) suggests that refining the codebook for the hardest category (Rationale) could buy more accuracy than annotating additional sentences for easy categories; this is directly testable in another annotation round.
  • The pipeline's recipe—codebook, majority-vote annotation, per-category binary fine-tuning—is portable to other text genres with non-exclusive and imbalanced categories, such as journal peer review or internal evaluation memos, though prevalence numbers like the low suggestion rate are corpus-specific and should not be generalized.
  • Since model choice had essentially no effect, other funders adopting this approach could reasonably start with any modern pre-trained transformer and concentrate their effort on the codebook and annotation protocol.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The manuscript develops and validates a supervised machine learning pipeline for classifying sentences in grant peer review reports from the Swiss National Science Foundation into twelve content categories. The authors describe an iterative human annotation process with majority aggregation, compare binary, multi-label, and multi-task fine-tuned transformer models (SPECTER2, BERT, RoBERTa), and report that binary classifiers achieve an average macro F1 of 0.85 on a held-out test set (Section 3.2.1). They then apply the pipeline to 1.6 million sentences and present descriptive prevalence results, along with robustness checks including 5-fold cross-validation, training-set size ablation, longer-context evaluation, and few-shot LLM comparison.

Significance. If the reported performance estimates are reliable, the paper provides a reusable, open-source toolkit for analyzing grant peer review content, with a carefully documented annotation codebook and publicly available fine-tuned models. The strongest contributions are the transparency of the annotation process, the comparison of multiple classification architectures, and the practical focus on a funder's operational needs. However, the central performance claim is currently supported by an evaluation protocol that does not account for the nested structure of the data (sentences within reports), so the reported F1 scores may not reflect performance on genuinely new review reports.

major comments (3)
  1. [Section 2.6 and Section 3.2.1] The evaluation uses a sentence-level random split of the 3,000 annotated sentences into training and test sets, but the natural sampling unit is the review report: the corpus contains 47,522 reports and 1,612,405 sentences (Section 2.1), so on average about 34 sentences come from each report. When sentences from the same report appear in both training and test sets, a transformer can exploit report-specific phrasing and style, inflating the estimated macro F1 of 0.85. Because the pipeline is intended for classifying the full corpus and future incoming reviews, the relevant performance is on reports unseen during training. I request a report-blocked evaluation: split the data by report ID (e.g., ensure all sentences of a report are in either training or test), or at least use clustered cross-validation by report, and report the resulting F1 scores. If the results change little, this should be stated; if they change substantially, the conclusions need to be revised.
  2. [Section 3.4, Figure 7] The reported correlation of r=0.99 between predicted prevalence in the full corpus and prevalence in the annotated sample is not a validation of classifier accuracy. The classifier is trained on the annotated sample, and the full-corpus predictions are produced by that same classifier; the comparison is largely tautological and would be high even if the classifier learned a constant shift or merely reproduced training-set proportions. A meaningful validation would compare predictions against an independently annotated holdout sample (e.g., a new annotation round not used in training) or test whether the predicted full-corpus prevalence is within the sampling variability expected from the annotated-sample prevalence. Please either add such a comparison or temper the interpretation of Figure 7.
  3. [Section 3.2.1, Table 2 and Table A4] The test set consists of 500 sentences, which yields very small numbers of positive examples for rare categories. For instance, Applicant: Quantity has a 1.6% prevalence, corresponding to roughly 8 positive test sentences; a single-sentence change alters the F1 by several points. The reported F1 of 0.93 for this category is therefore unstable. The average macro F1 of 0.85 is also a simple mean over categories with widely differing prevalence, so it can be driven by high-F1 rare categories. I ask for confidence intervals (e.g., bootstrap) for the test-set metrics and, if feasible, a prevalence-weighted overall F1 as a complementary summary.
minor comments (6)
  1. [Section 2.6] The description of 'stratifying by the outcome category' is unclear for a multi-label setting with twelve non-exclusive categories; please specify how stratification was implemented (e.g., iterative stratification for multi-label data).
  2. [Section 3.2.1] The 'Average F1 Score Across all Categories' is an unweighted mean; please note that it does not represent performance on a random sentence.
  3. [Table 4] The placeholder 'unk' is the most predictive term for Applicant and Track Record; please discuss whether the classifier may be relying on the anonymization token rather than substantive content, and whether this could limit transfer to review corpora with different anonymization procedures.
  4. [Section 3.6.2] The longer-context evaluation is only performed for Rationale; this is a reasonable limited test, but the one-sentence limitation should be acknowledged more explicitly in the discussion.
  5. [Figures 3 and 4] The category labels on the y-axis are rotated and difficult to read; consider using horizontal labels or a table format.
  6. [Section 4] The caveat that classifiers should not be used without human oversight is important; consider moving it to the abstract or a prominent limitations paragraph.

Circularity Check

1 steps flagged · score 2.0 of 10

Mild circularity: the r=0.99 full-corpus prevalence check compares model predictions to the very annotated sample used for fine-tuning; the core F1 evaluation is not circular.

  1. fitted input called prediction [Section 3.4, Figure 7; training setup in Sections 2.4 and 2.6]
    "Further, we compared the prevalence of categories in the annotated sample of 3,000 sentences with the predicted prevalence in Project Funding review sentences from 2016 to 2024. A reliable classification should show correspondence between these aggregated percentages (Müller and Fujimura, 2024). Figure 7 confirms that aggregated predictions for 1,612,405 review sentences align with the proportions in the annotated sample of 3,000 sentences (r = 0.99)."

    The 'predicted prevalence' is obtained by applying classifiers that were fine-tuned on the same annotated sample of 3,000 sentences whose prevalence is the comparison target. On an exchangeable corpus, a model that has learned the training label distribution will tend to produce aggregate predictions close to that distribution even if its sentence-level errors are large; hence the high correlation is partly built in and does not independently confirm classification quality. This is a calibration sanity check rather than an out-of-sample prediction, so calling it a confirmation of reliability is mildly circular. The main F1 results remain genuine held-out estimates.

full rationale

The paper's central claim is an out-of-sample F1 evaluation: 3,000 human-annotated sentences are randomly split into 2,500 training and 500 test sentences, models are fine-tuned only on the training portion, and macro F1 is computed on the untouched test set (Section 2.6, Table 2). That procedure is standard supervised learning and not circular. The one mild circularity is the Section 3.4 validation comparing full-corpus predicted prevalence to the annotated-sample prevalence used for training; this is a weak, partly self-referential sanity check, not a forced identity. Self-citations (Severin et al. 2023; Okasa and Jorstad 2024; Müller and Fujimura 2024) are not load-bearing for the main result, and the SPECTER2 model choice is independently checked against BERT and RoBERTa. The sentence-level random split may cause report-level leakage, but that is a benchmark-validity concern, not circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result is an empirical engineering contribution, not a law-like derivation. The main design choices are the annotation threshold and model hyperparameters; the main assumptions are the validity of human labels, the independence of sentences, and the representativeness of the English-only corpus. No new physical or conceptual entities are introduced.

free parameters (3)
  • Fine-tuning hyperparameters (learning rate, weight decay, epochs, batch size) = 2e-5, 0.01, 3, 10
    Hand-specified in Table A3 following Devlin et al. (2018). Not tuned on this dataset, but they influence the F1 scores and are part of the method as implemented.
  • Majority agreement threshold for gold-standard labels = At least 2 of 3 annotators agree
    This definitional choice determines the training and test labels. A stricter (unanimity) threshold would change prevalence and F1, as explored in Section 3.6.3.
  • Classification threshold for positive label = 0.5
    The sigmoid output is thresholded at 0.5 in Section 2.4. Changing it would alter precision-recall tradeoffs for imbalanced categories.
assumptions (4)
  • domain assumption Human majority annotation is a valid gold standard for the twelve content categories
    The paper treats a sentence as belonging to a category if at least two of three annotators agree (Section 2.2). Where full agreement is low, e.g., Proposal at 64%, the labels are noisy and bound the achievable classifier performance.
  • domain assumption Sentences are independent units, so a random sentence-level split gives an unbiased estimate of generalization to new review reports
    Section 2.6 splits the 3,000 annotated sentences randomly without grouping by source review. If multiple sentences come from the same report, the model may learn reviewer-specific phrasing and inflate test performance.
  • domain assumption English-language reviews are representative of the grant review reports
    Section 2.1 excludes 10.2% of reviews written in German or French, all from humanities and social sciences, so the descriptive prevalence results may not generalize to those disciplines.
  • domain assumption The twelve categories are the relevant dimensions for funders
    The codebook was developed by the authors in consultation with SNSF criteria (Section 2.2). There is no external benchmark validating that these categories capture all content funders care about.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports." pith.science (2026). https://pith.science/paper/F6KD7WUR

@misc{pith2026241116662,
  author       = {Pith},
  title        = {Pith review of: A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F6KD7WUR}},
  note         = {Machine review of arXiv:2411.16662}
}
read the original abstract

Peer review in grant evaluation informs funding decisions, but the contents of peer review reports are rarely analyzed. In this work, we develop a thoroughly tested pipeline to analyze the texts of grant peer review reports using methods from applied Natural Language Processing (NLP) and machine learning. We start by developing twelve categories reflecting content of grant peer review reports that are of interest to research funders. This is followed by multiple human annotators' iterative annotation of these categories in a novel text corpus of grant peer review reports submitted to the Swiss National Science Foundation. After validating the human annotation, we use the annotated texts to fine-tune pre-trained transformer models to classify these categories at scale, while conducting several robustness and validation checks. Our results show that many categories can be reliably identified by human annotators and machine learning approaches. However, the choice of text classification approach considerably influences the classification performance. We also find a high correspondence between out-of-sample classification performance and human annotators' perceived difficulty in identifying categories. Our results and publicly available fine-tuned transformer models will allow researchers and research funders and anybody interested in peer review to examine and report on the contents of these reports in a structured manner. Ultimately, we hope our approach can contribute to ensuring the quality and trustworthiness of grant peer review.

Figures

Figures reproduced from arXiv: 2411.16662 by the authors.

Figure 1
Figure 1. Pipeline of identifying, classifying, and validating textual characteristics in grant [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Prevalence of 12 categories in an annotated set of 3,000 sentences. [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Macro F1 scores based on 5-fold cross-validation for all three classification [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Cross-validation macro F1 scores for binary classifiers fine-tuned based on the [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: The relationship between the prevalence of categories and macro F1 scores for [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: The relationship between macro F1 scores for binary classifiers (left panel), [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Correspondence between prevalence in human-annotated set of sentences and [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Distribution of the predicted prevalences across all sentences in each review, for [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]
Figure 9
Figure 9. Figure 9: F1 scores (macro-average) depending on the size of the training set [PITH_FULL_IMAGE:figures/full_fig_p025_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 40 canonical work pages

  1. [1]

    A., Nunoo-Mensah, H., and Chen, W

    Acheampong, F. A., Nunoo-Mensah, H., and Chen, W. (2021). Transformer models for text-based emotion detection: a review of BERT -based approaches. Artificial Intelligence Review , 54(8):5789--5829

    work page 2021

  2. [2]

    Llama 3 model card

    AI@Meta (2024). Llama 3 model card. https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct

    work page 2024

  3. [3]

    Beltagy, I., Lo, K., and Cohan, A. (2019). Scibert: A pretrained language model for scientific text. In EMNLP . Association for Computational Linguistics

    work page 2019

  4. [4]

    E., Laver, M., and Mikhaylov, S

    Benoit, K., Conway, D., Lauderdale, B. E., Laver, M., and Mikhaylov, S. (2016). Crowd-sourced text analysis: Reproducible and agile production of political data. American Political Science Review , 110(2):278--295

    2016

  5. [5]

    Benoit, K., Watanabe, K., Wang, H., Nulty, P., Obeng, A., M \"u ller, S., and Matsuo, A. (2018). quanteda. The Journal of Open Source Software , 3(30):774

    work page 2018

  6. [6]

    Bladek, M. (2014). DORA : San F rancisco D eclaration on R esearch A ssessment ( M ay 2013). College & Research Libraries News , 75(4):191--196

    work page 2014

  7. [7]

    Brown, T. B. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  8. [8]

    Bucher, M. J. J. and Martini, M. (2024). Fine-tuned `small' LLM s (still) significantly outperform zero-shot generative AI models in text classification. arXiv preprint arXiv:2406.08660

    arXiv 2024

Show all 51 references
  1. [9]

    Cohan, A., Feldman, S., Beltagy, I., Downey, D., and Weld, D. S. (2020). Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180

    2020 arXiv

  2. [10]

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT : Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

    2018 arXiv

  3. [11]

    A., Grant, S., Chen, M.-C., Lindner, M

    Erosheva, E. A., Grant, S., Chen, M.-C., Lindner, M. D., Nakamura, R. K., and Lee, C. J. (2020). NIH peer review: Criterion scores completely account for racial disparities in overall impact scores. Science Advances , 6(23):eaaz4868

    2020

  4. [12]

    Forster, M., Schulz, C., Nokku, P., Mirsafian, M., Kasundra, J., and Skylaki, S. (2024). The right model for the job: An evaluation of legal multi-label classification baselines. arXiv preprint arXiv:2401.11852

    2024 arXiv

  5. [13]

    Fromm, M., Faerman, E., Berrendorf, M., Bhargava, S., Qi, R., Zhang, Y., Dennert, L., Selle, S., Mao, Y., and Seidl, T. (2021). Argument mining driven analysis of peer-reviews. Proceedings of the AAAI Conference on Artificial Intelligence , 35(6):4758–4766

    2021

  6. [14]

    K., and Ekbal, A

    Ghosal, T., Kumar, S., Bharti, P. K., and Ekbal, A. (2022). Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. PLoS One , 17(1):e0259238

    2022

  7. [15]

    Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., and Clavel, C. (2023). Automatic analysis of substantiation in scientific peer reviews. arXiv preprint arXiv:2311.11967

    2023 arXiv

  8. [16]

    Gupta, A., Norberg, J., Schnidman, E., Viswanathan, S., Zhang, K., and Shi, H. (2024). From W est to the rest: Growing dispersion of AI jobs in A merica. SSRN Electronic Journal

    2024

  9. [17]

    C., and Diesner, J

    Han, K., Rezapour, R., Nakamura, K., Devkota, D., Miller, D. C., and Diesner, J. (2023). An expert-in-the-loop method for domain-specific document categorization based on small training data. Journal of the Association for Information Science and Technology , 74(6):669--684

    2023

  10. [18]

    Hastie, T., Tibshirani, R., and Friedman, J. H. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction , volume 2. Springer

    2009

  11. [19]

    Heyard, R., Ott, M., Salanti, G., and Egger, M. (2022). Rethinking the funding line at the swiss national science foundation: Bayesian ranking and lottery. Statistics and Public Policy , 9(1):110--121

    2022

  12. [20]

    G., Norman, C

    Hren, D., Pina, D. G., Norman, C. R., and Maru s i \'c , A. (2022). What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports. Journal of Informetrics , 16(2):101289

    2022

  13. [21]

    Hua, X., Nikolov, M., Badugu, N., and Wang, L. (2019). Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North . Association for Computational Linguistics

    2019

  14. [22]

    Hug, S. E. (2024). How do referees integrate evaluation criteria into their overall judgment? evidence from grant peer review. Scientometrics , 129(3):1231--1253

    2024

  15. [23]

    Hug, S. E. and Aeschbach, M. (2020). Critera for assessing grant applications: A systematic review. Palgrave Communications , 6(1):1--15

    2020

  16. [24]

    Johnson, J. M. and Khoshgoftaar, T. M. (2019). Survey on deep learning with class imbalance. Journal of Big Data , 6(1):1--54

    2019

  17. [25]

    M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J

    Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., Lu, S., Mausam, Mieskes, M., Névéol, A., Pruthi, D., Qu, L., Schwartz, R., Smith, N. A., Solorio, T., Wang, J., Zhu, X., Rogers, A., Shah,...

    2024 arXiv

  18. [26]

    Langfeldt, L., Reymert, I., and Svartefoss, S. M. (2024). Distrust in grant peer review — reasons and remedies. Science and Public Policy , 51(1):28--41

    2024

  19. [27]

    and Reed, C

    Lawrence, J. and Reed, C. (2020). Argument mining: A survey. Computational Linguistics , 45(4):765–818

    2020

  20. [28]

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa : A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

    2019 arXiv

  21. [29]

    and Hutter, F

    Loshchilov, I. and Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint arXiv: 1711.05101

    2017 arXiv

  22. [30]

    Luo, J., Feliciani, T., Reinhart, M., Hartstein, J., Das, V., Alabi, O., and Shankar, K. (2021). Analyzing sentiments in peer review reports: Evidence from two science funding agencies. Quantitative Science Studies , 2(4):1271--1295

    2021

  23. [31]

    and Schutze, H

    Manning, C. and Schutze, H. (1999). Foundations of Statistical Natural Language Processing . MIT Press, Cambridge, Massachusetts

    1999

  24. [32]

    Minaee, S., Kalchbrenner, N., Cambria, E., Nikzad, N., Chenaghlu, M., and Gao, J. (2021). Deep learning-based text classification: A comprehensive review. ACM Computing Surveys (CSUR) , 54(3):1--40

    2021

  25. [33]

    Molnar, C. (2020). Interpretable Machine Learning . Lulu.com

    2020

  26. [34]

    and Fujimura, N

    M \"u ller, S. and Fujimura, N. (2024). Campaign communication and legislative leadership. Political Science Research and Methods , published ahead of print

    2024

  27. [35]

    and Jorstad, A

    Okasa, G. and Jorstad, A. (2024). The value of pre-training for scientific text similarity: Evidence from matching grant proposals to reviewers. In Corsin, C., Mark, C., Albert, W., Claudiu, M., Elisabeth, M., and Lucas, Z., editors, Proceedings of the 9th edition of the Swiss...

    2024

  28. [36]

    Pelaez, S., Verma, G., Ribeiro, B., and Shapira, P. (2023). Large-scale text analysis using generative language models: A case study in discovering public value expressions in AI patents. arXiv preprint arXiv:2305.10383

    2023 arXiv

  29. [37]

    Rainio, O., Teuho, J., and Kl \'e n, R. (2024). Evaluation metrics and statistical tests for machine learning. Scientific Reports , 14(1):6086

    2024

  30. [38]

    V., and M \"u ller, S

    Severin, A., Strinzel, M., Egger, M., Barros, T., Sokolov, A., Mouatt, J. V., and M \"u ller, S. (2023). Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews. PLoS Biology , 21(8):e3002238

    2023

  31. [39]

    Singh, A., D'Arcy, M., Cohan, A., Downey, D., and Feldman, S. (2022). Scirepeval: A multi-format benchmark for scientific document representations. arXiv preprint arXiv:2211.13308

    2022 arXiv

  32. [40]

    P., and Álvaro Rocha (2019)

    Sizo, A., Lino, A., Reis, L. P., and Álvaro Rocha (2019). An overview of assessing the quality of peer review reports of scientific articles. International Journal of Information Management , 46:286--293

    2019

  33. [41]

    Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. Y. (2008). Cheap and fast -- but is it good? E valuating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing , pages 254--263

    2008

  34. [42]

    Squazzoni, F., Bravo, G., Farjam, M., Marusic, A., Mehmani, B., Willis, M., Birukou, A., Dondio, P., and Grimaldo, F. (2021). Peer review and gender bias: A study on 145 scholarly journals. Science Advances , 7(2):eabd0299

    2021

  35. [43]

    Sun, C., Qiu, X., Xu, Y., and Huang, X. (2019). How to fine-tune BERT for text classification? In Chinese Computational Linguistics: 18th China National conference, CCL 2019, Kunming, China, October 18--20, 2019, proceedings 18 , pages 194--206. Springer

    2019

  36. [44]

    Sun, X., Li, X., Li, J., Wu, F., Guo, S., Zhang, T., and Wang, G. (2023). Text classification via large language models. arXiv preprint arXiv:2305.08377

    2023 arXiv

  37. [45]

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023). Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  38. [46]

    Tunstall, L., Von Werra, L., and Wolf, T. (2022). Natural Language Processing with Transformers: Building Language Applications with Hugging Face . O'Reilly, Beijing

    2022

  39. [47]

    N., Kaiser, ., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing systems , 30

    2017

  40. [48]

    Williams, K., Michalska, S., Cohen, E., Szomszor, M., and Grant, J. (2023). Exploring the application of machine learning to expert evaluation of research impact. PloS One , 18(8):e0288469

    2023

  41. [49]

    Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process...

    2020

  42. [50]

    Wolff, B., Seidlmayer, E., and F \"o rstner, K. U. (2024). Enriched BERT embeddings for scholarly publication classification. In International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs , pages 234--243. Springer

    2024

  43. [51]

    Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE International Conference on Computer Vision , ...

    2015

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.