{"id":"547666fc-8e5e-4525-88d2-684da3923809","arxiv_id":"2412.00098","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A benchmark comparison of fine-tuned BERT, SciBERT, BioBERT, and BlueBERT on Web of Science scientific text classification, claiming SciBERT performs best.","lead":"The authors fine-tuned four transformer models (BERT, SciBERT, BioBERT, BlueBERT) on three Web of Science text classification datasets, using abstracts and keywords separately. They report that the science-specific SciBERT usually scores highest, but the paper's own tables disagree on several accuracy numbers and the data-split arithmetic is inconsistent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table IV's split sizes do not sum to the dataset totals, indicating a potentially leaky train/validation/test partition that the paper never reconciles, so the reported SciBERT superiority rests on unverified data handling.","rationale":"The central claim is that SciBERT, as a domain-specific LLM, consistently beats both the general-purpose BERT and other domain-specific models (BioBERT, BlueBERT) on the three WoS datasets. The strongest evidence is Table VI, where SciBERT is first or tied in all six settings, but the margins are small (1–4 points). The most concrete and independent threat is the arithmetic inconsistency in Table IV: for every dataset, train + test + validation exceeds the total by exactly the validation size. Given that validation is described as '20% of the test set,' the only plausible reconciliation is that validation samples were taken from the test set but not removed from it, so the test set is not an independent hold-out. This is a reproducibility and validity problem that does not rely on assumptions about seed variance or statistical testing; it is visible directly from the numbers in the paper. The lack of repeated runs or error bars is a secondary concern: even with a correct split, a single 1-point difference between models may be noise, but that concern cannot be resolved until the split is fixed. Because the reader already identified the split issue as the weakest assumption, and because this concern must be resolved before any of the quantitative claims can be accepted, the verdict remains CONDITIONAL. I therefore recommend no change to the reader's verdict.","tokens_in":10347,"tokens_out":7714,"duration_ms":65572,"concrete_test":"Inspect the linked GitHub repository's data-splitting script and verify that train, validation, and test sets are disjoint and that their sizes equal 46,985 / 11,967 / 5,736 respectively. If the script creates validation as a subset of the test set without removing those samples from test, recompute all test accuracies on the remaining disjoint test samples only; if any of SciBERT's advantages over BERT (or among the other models) reverses or changes by more than 1 accuracy point, the paper's central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table IV reports splits whose sums exceed the dataset sizes: WoS-46985 gives 37,588+9,397+1,880=48,865 vs 46,985; WoS-11967 gives 12,446 vs 11,967; WoS-5736 gives 5,966 vs 5,736. The text says validation is '20% of the test set,' which implies validation is carved out of the test set without reducing the reported test count. If so, the test set used for the accuracies in Tables VI and VII overlaps with validation samples used for model monitoring, creating a direct leakage path that can inflate absolute performance and bias the relative comparison if model selection exploits validation. This undermines the central claim that SciBERT consistently outperforms BERT, BioBERT, and BlueBERT, because the observed 1–2 point margins could be artifacts of the contaminated split. The paper provides no code-level verification of the split, despite linking a repository, and the arithmetic inconsistency is never addressed in the text.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper fine-tunes four transformer-based language models (BERT, SciBERT, BioBERT, and BlueBERT) on three Web of Science derived datasets (WoS-46985, WoS-11967, WoS-5736), using either paper abstracts or keyword lists as input, and reports accuracy and F1 metrics for scientific text classification. The authors conclude that domain-specific models, especially SciBERT, consistently outperform general-purpose BERT and the biomedical models, and they compare their results with classical deep learning baselines from the literature.","tokens_in":10665,"tokens_out":3871,"duration_ms":46839,"significance":"If the conclusions are supported, the paper would provide a useful empirical benchmark on the value of domain-adaptive pretraining for scientific text classification, with a clear experimental design across two input representations and three dataset sizes. The authors provide a public repository link, use a standardized fine-tuning protocol, and explicitly acknowledge limitations such as fixed hyperparameters and restricted dataset scope. These are real strengths. However, the central claim of consistent SciBERT superiority is currently undermined by unresolved arithmetic inconsistencies in the data split (Table IV), conflicting accuracy numbers between Tables VI and VII, and single-run results without variance or significance testing.","major_comments":[{"comment":"The train/test/validation split counts do not add up to the stated dataset totals: for WoS-46985, 37,588 + 9,397 + 1,880 = 48,865, which is 1,880 more than the 46,985 total; for WoS-11967, 9,573 + 2,394 + 479 = 12,446, exceeding 11,967 by 479; and for WoS-5736, 4,588 + 1,148 + 230 = 5,966, exceeding 5,736 by 230. The text states that validation is \"20% of the test set,\" which implies that validation examples are carved out of the test set without reducing the reported test count. As a result, the test set used for Tables VI and VII may overlap with the validation set used for model monitoring, creating a potential leakage path that could inflate absolute accuracies and bias the relative comparison. The paper never reconciles this arithmetic inconsistency or specifies the exact split procedure, and the repository is not referenced with a commit or code excerpt that would allow verification. Please provide the exact partition code/seed and confirm that training, validation, and test sets are disjoint, or recompute all results on a properly documented split.","section":"§IV.A, Table IV"},{"comment":"The accuracy values for the same experimental conditions differ between Table VI and Table VII. For WoS-5736 abstracts, Table VI reports BERT 97%, SciBERT 98%, BioBERT 98%, and BlueBERT 96%, while Table VII reports BERT 96%, SciBERT 97%, BioBERT 98%, and BlueBERT 97%. The text in §V.E agrees with Table VI, not Table VII. This is not a formatting-only discrepancy: under Table VII, BioBERT ties or exceeds SciBERT on WoS-5736 abstracts, which directly contradicts the abstract's claim that \"SciBERT demonstrated superior performance... consistently surpassing other models such as BERT, BioBERT, and BlueBERT.\" The authors must reconcile these numbers and state which table reports the final test-set results.","section":"Tables VI and VII, §V.E"},{"comment":"All results are reported as single-point estimates from one run, with no error bars, multiple seeds, or statistical significance tests. The differences that drive the central claim are often 1–2 percentage points (e.g., WoS-46985 abstracts: SciBERT 87% vs. BioBERT/BlueBERT 86%; WoS-46985 keywords: SciBERT 80% vs. BioBERT 79%), and in at least one case (WoS-5736 abstracts) BioBERT's micro F1 of 0.9773 exceeds SciBERT's 0.9756 even though both accuracies are 98%. Without variance estimates or repeated runs, the evidence does not support the strong claim that SciBERT \"consistently outperforms\" or \"consistently surpasses\" the other models. Please report means and standard deviations over multiple seeds, or apply an appropriate statistical test for classifier comparison.","section":"§V, Tables VI and VII"}],"minor_comments":[{"comment":"The warmup steps value is written as \"1×10−4 steps,\" which is not an integer step count; this should be clarified (e.g., as a fraction of total steps or as an actual number of warmup steps).","section":"Table V"},{"comment":"The BlueBERT row lists Micro Precision as 0.7879 while Micro F1 and Micro Recall are both 0.7987; this is internally inconsistent and likely a typo that should be corrected.","section":"Table VI, WoS-46985 Keywords, BlueBERT"},{"comment":"The text states that \"each model achieved its peak performance prior to the 10th epoch,\" but the preceding sentences describe BERT reaching 0.98 \"by the final epoch\" and BioBERT reaching 0.99 as a \"final micro F1 score\"; this is contradictory and should be clarified.","section":"§V.E"},{"comment":"The phrase \"the dataset randomized\" should be \"the datasets were randomized,\" and the sentence \"Each dataset... underwent... to extract four primary attributes\" should be reworded for grammatical correctness.","section":"§IV.A"},{"comment":"The phrase \"domain-adaptive pertaining\" should read \"domain-adaptive pretraining.\"","section":"§VII"},{"comment":"Reference [27] contains a stray comma in the author list (\"K. Jafari Meimandi, , M. S. Gerber\"), and the formatting of some author names and accents is inconsistent; these should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core experimental question is appropriate for the venue, and the paper is readable, but the central claim currently rests on an unverified and arithmetically inconsistent data split and on conflicting accuracy tables. These are fixable with additional analysis and reporting, so I recommend major revision rather than rejection. The authors should also be asked to verify the repository contents and provide exact split code, because the repository link is the only external evidence for the data-handling procedure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things: the paper does a clean, standard fine-tuning comparison of four BERT-family models on three WoS subsets, and it has an accounting problem in its data split that makes every reported accuracy suspect. The abstract-versus-keyword comparison is the only genuinely useful addition.\n\nThe experimental setup is straightforward and reproducible in principle: BERT, SciBERT, BioBERT, and BlueBERT are fine-tuned with the same hyperparameters on WoS-46985, WoS-11967, and WoS-5736, using abstracts and separately keywords. They report macro/micro/weighted precision, recall, F1, and accuracy, and they link a repository. The observation that SciBERT tends to edge out BERT is consistent with prior work, and the keyword-only condition is a genuinely useful data point.\n\nThe soft spots are substantial. Table IV gives train/test/validation counts that sum to 48,865 / 12,446 / 5,966 for datasets that contain 46,985 / 11,967 / 5,736 documents. The method text says validation is \"20% of the test set,\" which implies validation is cut out of the test set, so the reported test sets overlap with validation by construction. That is a potential leakage path for model selection. The paper never reconciles this. On top of that, Table VI and Table VII disagree on the WoS-5736 abstract accuracy (SciBERT 98% vs 97%, BlueBERT 96% vs 97%), and all numbers come from a single run with no error bars. The fixed hyperparameters are a stated limitation, fine, but they mean the 1-2 point margins could be noise.\n\nThe qualitative conclusion—that domain-adapted SciBERT is a reasonable default for this WoS-style task—may survive a re-run, but the paper as written does not support the specific quantitative claims.\n\nThis is a workshop-level benchmark note for practitioners who want a quick default model for WoS-style classification. It is not a research advance. If the authors fix the partition, reconcile the tables, and add repeated runs, it could be a citable data point. As it stands, I would desk reject it, with an invitation to resubmit after fixing the split and the inconsistencies. It does not deserve a serious referee until the numbers are verifiable.","headline":"A straightforward BERT-family benchmark on WoS whose data-split accounting and inconsistent tables make the reported accuracies unverifiable as written.","tokens_in":11060,"tokens_out":3805,"would_cite":false,"duration_ms":109349,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A model pretrained on science text classifies scientific papers better than general or biomedical models.","keywords":["domain-specific text classification","fine-tuning LLMs","transformer-based language models","text representation","LLM performance evaluation","SciBERT","scientific text classification","Web of Science dataset"],"falsifier":"A decisive check is to re-run all four models on the three WoS subsets with a single clean random split whose train, test, and validation sizes sum exactly to the dataset total and are disjoint; if SciBERT does not match or beat the other models on all six conditions, the claim of consistent superiority fails.","tokens_in":10149,"feed_emoji":"🔬","tokens_out":11148,"duration_ms":85180,"temperature":0.7,"pith_summary":"The paper sets out to show that the pretraining corpus, not just the architecture, decides how well a transformer classifies scientific texts. It fine-tunes four publicly available models—BERT, SciBERT, BioBERT, and BlueBERT—on three Web of Science subsets, using either paper abstracts or keyword lists as input. Across the six experimental conditions, the science-pretrained SciBERT achieves the highest or tied-highest accuracy and F1, while the general-purpose BERT and the biomedical-pretrained models lag by one to a few points. If the finding holds, it gives practitioners a concrete default choice for scientific literature classification and confirms that domain-matched pretraining is worth the extra effort.","feed_headline":"SciBERT tops general and biomedical models on WoS paper sets","feed_subtitle":"The science-pretrained model matches or beats general and biomedical baselines on all three WoS datasets.","key_machinery":"The machinery is a controlled fine-tuning comparison across four pretrained transformer encoders: general English BERT, science-pretrained SciBERT, and biomedical-pretrained BioBERT and BlueBERT. All four share the same classification head and training protocol—AdamW with learning rate $2\\times10^{-5}$, linear warmup, 20 epochs—so the only deliberate differences are the pretraining corpus and the input representation (abstract text vs. keyword lists). The comparison is carried out on three nested subsets of the WoS-46985 benchmark, which share the same seven parent scientific domains but differ in category granularity and size. This setup is meant to isolate what domain-matched pretraining adds on top of a fixed architecture and to measure how much classification signal each input type carries.","core_discovery":"The central claim is that domain-specific pretraining is the decisive factor in scientific text classification. On the paper's own numbers, SciBERT—the BERT variant pretrained on a large scientific corpus—reaches 87% accuracy on WoS-46985 abstracts versus 85% for BERT and 86% for both biomedical models, and 80% on keywords versus 79% for BERT (tied with BlueBERT). The pattern repeats on the smaller subsets: 92% versus 91% on WoS-11967 abstracts, 87% versus 84–86% on its keyword version, and 98% with abstracts (tied with BioBERT) and 94% with keywords on WoS-5736. The paper reads these results as showing that matching the pretraining corpus to the target domain matters more than the choice among transformer architectures, and that abstract-based inputs consistently carry more classification signal than keyword-only inputs. It also positions the fine-tuned LLMs above the earlier deep learning baselines and the hierarchical HDLTex model reported in the literature.","pith_inferences":["A direct extension the paper does not run is concatenating abstracts and keywords: given the consistent 4–7 point gap, it is worth testing whether combined input adds signal or just noise.","The split arithmetic in Table IV is inconsistent (for example, WoS-46985: 37,588 + 9,397 + 1,880 = 48,865, not 46,985), so the ranking should be re-run on a clean, disjoint split before the accuracy numbers are taken as final.","The domain-match principle predicts a testable specialization ladder: a model pretrained on a tightly matched subcorpus, such as materials-science text, should beat SciBERT on that subdomain even if it loses on general science.","With only single runs and fixed hyperparameters, margins of one to three points could plausibly flip under per-model tuning or across seeds; the paper's qualitative ranking is robust only if it survives those variations."],"forward_implications":["For scientific paper classification, the safe default is a science-pretrained encoder such as SciBERT: it matches or beats the general and biomedical models on every condition the paper tests.","Abstract-only pipelines should be preferred over keyword-only pipelines: the gap is consistently 4–7 accuracy points in favor of abstracts across all models and datasets.","On these benchmarks, fine-tuned transformers displace the earlier deep learning state of the art: the best LLM accuracy on WoS-46985 abstracts (87%) exceeds the 76.58% reported for HDLTex.","Domain match to the label space matters more than biomedical specialization: BioBERT and BlueBERT do not consistently beat general BERT, while SciBERT's broader scientific pretraining transfers across all seven WoS parent domains.","Because the fine-tuning protocol is held fixed, the ranking differences are attributed to pretraining corpus, not to architecture or hyperparameter choice."],"supporting_citations":[{"why":"Supplies SciBERT, the science-pretrained model that the paper claims wins, and the argument that scientific pretraining helps downstream scientific tasks.","marker":"[20]"},{"why":"Supplies BERT, the general-purpose baseline and the shared transformer architecture for all fine-tuned models.","marker":"[17]"},{"why":"Supplies BioBERT, one of the two biomedical-pretrained comparison models the paper tests.","marker":"[21]"},{"why":"Supplies BlueBERT, the other biomedical-pretrained comparison model the paper tests.","marker":"[22]"},{"why":"Provides the WoS-46985 source data and the HDLTex deep-learning baseline results the paper compares against.","marker":"[27]"},{"why":"Supplies the transformer architecture underlying all four models, the foundation for the LLM approach.","marker":"[16]"}],"fun_headline_variants":["SciBERT beats BERT and biomedical models on WoS","Domain-specific pretraining wins in scientific text classification","Science-tuned LLMs top generalists on paper classification","SciBERT leads across all WoS classification datasets","Pretraining on science boosts LLM paper classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the train/test/validation split is clean and disjoint; the sizes in Table IV sum to more than the dataset totals (for example, 48,865 versus 46,985 for WoS-46985), and the paper never reconciles this.","fun_headline_variants_meta":{"raw":{"variants":["SciBERT beats BERT and biomedical models on WoS","Domain-specific pretraining wins in scientific text classification","Science-tuned LLMs top generalists on paper classification","SciBERT leads across all WoS classification datasets","Pretraining on science boosts LLM paper classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1866,"prompt_tokens":941,"completion_tokens":925,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":850}},"tokens_in":557,"tokens_out":925,"duration_ms":9021,"temperature":1.0,"reasoning_tokens":850,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:59:39.901895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to re-run all four models on the three WoS subsets with a single clean random split whose train, test, and validation sizes sum exactly to the dataset total and are disjoint; if SciBERT does not match or beat the other models on all six conditions, the claim of consistent superiority fails.","supporting_citations":[{"cited_title":"Hdltex: Hierarchical deep learning for text classification,","cited_arxiv_id":null,"evidence_quote":"Provides the WoS-46985 source data and the HDLTex deep-learning baseline results the paper compares against."}],"review_version":1}