{"id":"eb4a8a12-d75e-4ada-b6bc-442beaff0e8c","arxiv_id":"2411.16662","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Fine-tuned transformer models classify Swiss National Science Foundation grant peer review sentences into twelve content categories with an average macro F1 of 0.85, enabling large-scale analysis of review reports.","lead":"Researchers built and validated a machine learning pipeline that classifies grant peer review report sentences into twelve content categories, and they release the fine-tuned models. The tool lets funding agencies analyze thousands of confidential reviews in a structured way, which could improve transparency and quality of grant evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Report-level leakage likely inflates benchmark: sentence-level random split lets models memorize reviewer-specific wording; a report-blocked split is needed before F1 claims are accepted.","rationale":"The reader's weakest_assumption identifies precisely the load-bearing issue: Section 2.6 randomly splits sentences without grouping by source review, while the inference target is classification of whole review reports. If this concern lands, the headline F1 numbers in Table 2 and Figure 3 overstate performance on genuinely new reports, and the corpus-level prevalence estimates in Section 3.5 inherit the same optimism. The concern is not merely theoretical: reports average about 34 sentences, so random sampling of 3,000 sentences creates many same-report pairs across the train/test boundary; transformer classifiers are sensitive to such textual style leakage, and the keyness analysis confirms that surface-level stylistic tokens are highly predictive. I do not see an internal inconsistency in the formal setup, and the paper is otherwise careful: code and models are public, model comparisons and several robustness checks such as LLM few-shot, longer context, and full-agreement training are valuable. But the evaluation split is the weakest link in the central claim. Because the reader already returned CONDITIONAL and this concern is exactly what the condition should require, the verdict should remain unchanged: a report-blocked evaluation should be added or the performance claims qualified. A secondary metric caveat is worth noting: the reported macro F1 averages positive- and negative-class F1, so the average of 0.85 partly reflects high F1 on the majority 'absent' class; the appendix does report label=1 F1, for example 0.52 for Rationale, but the headline could mislead if read as category-detection accuracy. The report-level split test is the single check that would settle the main risk.","tokens_in":30229,"tokens_out":4563,"duration_ms":44463,"concrete_test":"Re-run the binary SPECTER2 pipeline with group-based splitting: assign all sentences from the same review report to the same fold (e.g., GroupKFold by report ID, or a 2,500/500 train/test split in which no report contributes sentences to both sets), preserving stratification as far as possible. Compare the resulting macro F1 and label=1 F1 for all 12 categories against Table 2 and Figure 3. If the average macro F1 falls materially below 0.85, or if rare categories such as Rationale and Suggestion degrade by more than a few points, the headline performance is not transportable to new reports and should be qualified accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that binary classifiers reach an average macro F1 of 0.85 (Section 3.2.1, Table 2). The evaluation protocol in Section 2.6 randomly divides 3,000 annotated sentences into 2,500 training and 500 test sentences, and the five-fold cross-validation repeats the same sentence-level randomization. The sampling unit in the underlying corpus is the review report, not the sentence: 47,522 reports contain 1,612,405 sentences, so multiple sampled sentences come from the same report. With a sentence-level split, sentences from one report can appear in both training and test sets; a transformer can then exploit report-specific phrasing, evaluative style, and repeated criteria language, yielding optimistic F1 scores. The keyness analysis in Table 4 shows that style markers such as placeholder 'unk', 'expertise', and 'excellent' are highly predictive, so leakage can directly inflate apparent performance. The paper's data-leakage mitigation, offline training in Section 2, addresses internet leakage, not this split-level dependence. Because the stated goal is to classify the full corpus and future incoming reviews, the relevant performance is on reports not seen during training; the current random-sentence split does not estimate that quantity.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript develops and validates a supervised machine learning pipeline for classifying sentences in grant peer review reports from the Swiss National Science Foundation into twelve content categories. The authors describe an iterative human annotation process with majority aggregation, compare binary, multi-label, and multi-task fine-tuned transformer models (SPECTER2, BERT, RoBERTa), and report that binary classifiers achieve an average macro F1 of 0.85 on a held-out test set (Section 3.2.1). They then apply the pipeline to 1.6 million sentences and present descriptive prevalence results, along with robustness checks including 5-fold cross-validation, training-set size ablation, longer-context evaluation, and few-shot LLM comparison.","tokens_in":30480,"tokens_out":5869,"duration_ms":52019,"significance":"If the reported performance estimates are reliable, the paper provides a reusable, open-source toolkit for analyzing grant peer review content, with a carefully documented annotation codebook and publicly available fine-tuned models. The strongest contributions are the transparency of the annotation process, the comparison of multiple classification architectures, and the practical focus on a funder's operational needs. However, the central performance claim is currently supported by an evaluation protocol that does not account for the nested structure of the data (sentences within reports), so the reported F1 scores may not reflect performance on genuinely new review reports.","major_comments":[{"comment":"The evaluation uses a sentence-level random split of the 3,000 annotated sentences into training and test sets, but the natural sampling unit is the review report: the corpus contains 47,522 reports and 1,612,405 sentences (Section 2.1), so on average about 34 sentences come from each report. When sentences from the same report appear in both training and test sets, a transformer can exploit report-specific phrasing and style, inflating the estimated macro F1 of 0.85. Because the pipeline is intended for classifying the full corpus and future incoming reviews, the relevant performance is on reports unseen during training. I request a report-blocked evaluation: split the data by report ID (e.g., ensure all sentences of a report are in either training or test), or at least use clustered cross-validation by report, and report the resulting F1 scores. If the results change little, this should be stated; if they change substantially, the conclusions need to be revised.","section":"Section 2.6 and Section 3.2.1"},{"comment":"The reported correlation of r=0.99 between predicted prevalence in the full corpus and prevalence in the annotated sample is not a validation of classifier accuracy. The classifier is trained on the annotated sample, and the full-corpus predictions are produced by that same classifier; the comparison is largely tautological and would be high even if the classifier learned a constant shift or merely reproduced training-set proportions. A meaningful validation would compare predictions against an independently annotated holdout sample (e.g., a new annotation round not used in training) or test whether the predicted full-corpus prevalence is within the sampling variability expected from the annotated-sample prevalence. Please either add such a comparison or temper the interpretation of Figure 7.","section":"Section 3.4, Figure 7"},{"comment":"The test set consists of 500 sentences, which yields very small numbers of positive examples for rare categories. For instance, Applicant: Quantity has a 1.6% prevalence, corresponding to roughly 8 positive test sentences; a single-sentence change alters the F1 by several points. The reported F1 of 0.93 for this category is therefore unstable. The average macro F1 of 0.85 is also a simple mean over categories with widely differing prevalence, so it can be driven by high-F1 rare categories. I ask for confidence intervals (e.g., bootstrap) for the test-set metrics and, if feasible, a prevalence-weighted overall F1 as a complementary summary.","section":"Section 3.2.1, Table 2 and Table A4"}],"minor_comments":[{"comment":"The description of 'stratifying by the outcome category' is unclear for a multi-label setting with twelve non-exclusive categories; please specify how stratification was implemented (e.g., iterative stratification for multi-label data).","section":"Section 2.6"},{"comment":"The 'Average F1 Score Across all Categories' is an unweighted mean; please note that it does not represent performance on a random sentence.","section":"Section 3.2.1"},{"comment":"The placeholder 'unk' is the most predictive term for Applicant and Track Record; please discuss whether the classifier may be relying on the anonymization token rather than substantive content, and whether this could limit transfer to review corpora with different anonymization procedures.","section":"Table 4"},{"comment":"The longer-context evaluation is only performed for Rationale; this is a reasonable limited test, but the one-sentence limitation should be acknowledged more explicitly in the discussion.","section":"Section 3.6.2"},{"comment":"The category labels on the y-axis are rotated and difficult to read; consider using horizontal labels or a table format.","section":"Figures 3 and 4"},{"comment":"The caveat that classifiers should not be used without human oversight is important; consider moving it to the abstract or a prominent limitations paragraph.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-organized and the authors are transparent about their data and code. The main concern is the evaluation split; this is a methodological issue that is fixable. The paper might be a better fit for a specialized NLP/infometrics venue, but it should be publishable in an empirical-methods journal if the report-level analysis is added."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid, well-documented applied NLP pipeline for classifying grant peer review sentences. The main quantitative claims are plausible, but the reported F1 scores (average macro 0.85) are likely somewhat inflated because the train/test split is by sentence, not by report. This is fixable and doesn't undermine the central qualitative findings.\n\nWhat's genuinely new: a 3,000-sentence annotated corpus from SNSF grant reviews, a twelve-category codebook, and fine-tuned transformer models (SPECTER2, BERT, RoBERTa) that are publicly released. The comparison of binary, multi-label, and multi-task approaches is real, and the robustness work—training-set-size ablation, context inclusion, full-agreement training, LLM few-shot baseline—is above average for this literature. The code and models on GitHub and Hugging Face make this easy to build on.\n\nThe biggest soft spot is the evaluation protocol. The 3,000 annotated sentences are randomly split into 2,500 training and 500 test sentences, and the five-fold cross-validation repeats the same sentence-level randomization. The sampling unit in the underlying corpus is the review report (47,522 reports, 1.6M sentences), so multiple sentences from the same report appear on both sides of the split. A transformer can pick up report-specific phrasing and style—and the keyness analysis itself shows that markers like 'unk' (the applicant-name placeholder) and words like 'excellent' are highly predictive. That likely inflates F1, especially for Applicant, Track Record, and Positive/Negative. A report-blocked split, or at least a clustered evaluation, is needed to estimate performance on genuinely new reports. I don't think this is fatal; the pipeline's purpose is descriptive analysis of large corpora, and even a few points lower F1 probably leaves the main prevalence patterns intact. But Tables 2 and 3 should not be read as out-of-domain performance.\n\nA smaller issue: no confidence intervals around the F1 scores. For rare categories like Applicant: Quantity (1.6% prevalence) or Suggestion (4.5%), the 500-sentence test set contains only a handful of positives, so the 0.93 F1 for Applicant: Quantity is based on very few observations. Bootstrap intervals would help. The English-only corpus (10.2% of reports excluded, mostly German/French humanities/social science) is acknowledged and limits external validity, but that's an explicit scope restriction, not a hidden flaw.\n\nWhat the paper does well: the annotation process is careful—multiple rounds, codebook refinement, majority-agreement aggregation—and the public release of code, models, and codebook is genuinely useful. The LLM few-shot comparison is honest; they show it underperforms, which is a useful negative result.\n\nThis deserves a serious referee. I'd send it out with a request for a report-blocked cross-validation and uncertainty quantification. The central argument holds up; it's a contribution that needs revision, not rejection.","headline":"Solid, reproducible applied NLP pipeline for grant review text with an evaluation-split flaw that likely inflates the headline F1 scores, but the central qualitative findings survive.","tokens_in":31003,"tokens_out":2624,"would_cite":true,"duration_ms":26247,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that per-category binary fine-tuned transformer classifiers, trained on 3,000 majority-vote human-annotated sentences, can label the 1.6-million-sentence corpus of grant peer-review reports with an average…","keywords":["grant peer review","text classification","fine-tuned transformers","sentence-level annotation","content analysis","funding evaluation","inter-coder agreement","class imbalance"],"falsifier":"Re-run the classifier evaluation with a report-stratified split—all sentences from any given review report placed entirely in training or entirely in test—and compare the average macro F1 to the reported 0.85; a large drop would show that the headline accuracy depends on within-report leakage.","tokens_in":30056,"feed_emoji":"📄","tokens_out":12105,"duration_ms":104762,"temperature":0.7,"pith_summary":"This paper sets out to establish that the content of grant peer-review reports can be made machine-readable at scale. Its recipe is to define twelve categories that a funding agency cares about—evaluation criteria, what the reviewer focuses on, and whether statements are positive, negative, justified, or suggest improvements—then have instructed human annotators label 3,000 sentences, aggregate their judgments by majority vote, and fine-tune a separate transformer classifier for each category. The headline result is an average macro F1 of 0.85 (a balanced precision–recall average) across the twelve binary classifiers, with the hardest category (Rationale) at 0.71 and the easiest (Applicant: Quantity) at 0.93. If this performance holds on new reports, the pipeline makes the full corpus of roughly 1.6 million sentences available for structured audit, letting funders measure how reviewers allocate attention, how often they give reasons, and how often they offer suggestions. The paper also argues for a design principle: per-category binary models beat shared multi-label and multi-task architectures, and the quality of the human-annotated training data matters more than which pre-trained transformer backbone is selected.","feed_headline":"Twelve binary models read grant reviews; average F1 hits 0.85","feed_subtitle":"A 3,000-sentence human-annotated corpus lets per-category models audit 1.6 million review sentences.","key_machinery":"The machinery is a set of twelve fine-tuned binary transformer classifiers, one per category, built on a pre-trained scientific-text encoder (SPECTER2, a BERT-style model pre-trained on scientific papers with citation-graph signals). For each sentence the encoder produces a CLS-token embedding, which a linear output layer with sigmoid activation maps to a class probability, and the loss is binary cross-entropy minimized by AdamW over three epochs. The training signal comes from the annotation procedure: an eleven-round codebook refinement, four instructed annotators, three labels per sentence, and majority-vote aggregation into the final 3,000-sentence gold set. The method comparison is part of the machinery: it contrasts the twelve binary heads with one multi-label head and one shared-encoder multi-task setup, establishing which design generalizes.","core_discovery":"The paper claims that a supervised pipeline—iterated codebook development, multi-annotator consensus labeling, and per-category fine-tuned transformer classifiers—can identify twelve content categories in individual sentences of grant peer-review reports. The strongest evidence is the performance of twelve separately fine-tuned binary classifiers: macro F1 ranges from 0.71 for Rationale to 0.93 for Applicant: Quantity, averaging 0.85, and five-fold cross-validation confirms the same ordering. The comparison with multi-label (0.73) and multi-task (0.62) classifiers shows that separate binary models are the decisive design choice. The comparison across three pre-trained transformer backbones, all reaching the same 0.85 average, indicates that the annotated data rather than the model family drives accuracy. The paper also reports that a few-shot large-language-model baseline reaches only 0.70 and fails on rare categories, so the pipeline retains human annotation plus fine-tuned transformers.","pith_inferences":["A report-stratified split, not tested in the paper, would reveal how much of the 0.85 F1 comes from content versus reviewer-specific phrasing; if the drop on held-out reviews is large, the models would need calibration before use on a new review cohort.","The strong negative correlation between annotators' perceived difficulty and F1 (-0.73) suggests that refining the codebook for the hardest category (Rationale) could buy more accuracy than annotating additional sentences for easy categories; this is directly testable in another annotation round.","The pipeline's recipe—codebook, majority-vote annotation, per-category binary fine-tuning—is portable to other text genres with non-exclusive and imbalanced categories, such as journal peer review or internal evaluation memos, though prevalence numbers like the low suggestion rate are corpus-specific and should not be generalized.","Since model choice had essentially no effect, other funders adopting this approach could reasonably start with any modern pre-trained transformer and concentrate their effort on the codebook and annotation protocol."],"forward_implications":["If the central claim holds, the funder's full corpus of about 1.6 million review sentences can be labeled automatically at an average macro F1 of 0.85, making category-level monitoring feasible in practice.","Binary, category-specific models are the recommended architecture: they outperform multi-label (0.73) and multi-task (0.62) alternatives and, unlike the shared models, their F1 scores are not correlated with how common the category is.","Because three different pre-trained transformer backbones all average 0.85, the human-annotation data is what carries performance; further gains should come from better labels, not bigger models.","Few-shot prompting of an 8-billion-parameter language model achieves only 0.70 average macro F1 and performs especially poorly on rare categories, so LLM-generated annotations cannot replace the human-consensus gold set.","Sentence-level classification is the right unit: adding surrounding context for the hardest category, Rationale, leaves F1 at 0.71, so the pipeline can stay sentence-based and cheap to scale."],"supporting_citations":[{"why":"Supplies the BERT architecture and the fine-tuning hyperparameters (learning rate 2e-5, 3 epochs) that the classifiers are trained with.","marker":"Devlin et al., 2018"},{"why":"Introduces SPECTER, the citation-informed scientific encoder whose updated version is the main model fine-tuned here.","marker":"Cohan et al., 2020"},{"why":"Provides SPECTER2, the specific pre-trained transformer backbone used for the main results.","marker":"Singh et al., 2022"},{"why":"Pioneered the binary per-category peer-review classification and keyness-analysis design that this pipeline adopts and extends.","marker":"Severin et al., 2023"},{"why":"Supplies a comparable peer-review classification benchmark whose reported F1 scores the paper's results are measured against.","marker":"Ghosal et al., 2022"},{"why":"Justifies the single training-test split workflow used for fine-tuning and evaluation.","marker":"Tunstall et al., 2022"},{"why":"Supports the majority-agreement aggregation of multiple annotator judgments used to create the training labels.","marker":"Benoit et al., 2016"},{"why":"Recommends the five-fold cross-validation protocol used to check that results do not depend on one split.","marker":"Hastie et al., 2009"}],"fun_headline_variants":["Binary models hit F1 0.85 on grant review sentences","Per-category classifiers win over multi-label for grant reviews","0.85 F1: Twelve separate models decode grant review text","Grant review NLP: binary models beat LLM, reach 0.85 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation randomly splits the 3,000 annotated sentences into training and test sets without grouping by source review report, so sentences from the same report can appear in both sets and the model can learn reviewer-specific wording that would not generalize to a new report.","fun_headline_variants_meta":{"raw":{"variants":["Binary models hit F1 0.85 on grant review sentences","Per-category classifiers win over multi-label for grant reviews","0.85 F1: Twelve separate models decode grant review text","Grant review NLP: binary models beat LLM, reach 0.85 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1610,"prompt_tokens":971,"completion_tokens":639,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":587,"tokens_out":639,"duration_ms":5970,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:50:28.653812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the classifier evaluation with a report-stratified split—all sentences from any given review report placed entirely in training or entirely in test—and compare the average macro F1 to the reported 0.85; a large drop would show that the headline accuracy depends on within-report leakage.","supporting_citations":[{"cited_title":"V., and M \\\"u ller, S","cited_arxiv_id":null,"evidence_quote":"Pioneered the binary per-category peer-review classification and keyness-analysis design that this pipeline adopts and extends."},{"cited_title":"K., and Ekbal, A","cited_arxiv_id":null,"evidence_quote":"Supplies a comparable peer-review classification benchmark whose reported F1 scores the paper's results are measured against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the single training-test split workflow used for fine-tuning and evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Recommends the five-fold cross-validation protocol used to check that results do not depend on one split."}],"review_version":1}