{"id":"2a4576dc-b4e3-4c04-984d-334c2658b9c6","arxiv_id":"2504.19021","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning BERT-family models on a Web of Science dataset expanded with model-generated pseudo-labels and hard voting yields higher self-reported F1 scores, but the evaluation lacks independent ground truth.","lead":"This paper tests whether adding thousands of self-labeled articles to a scientific-text dataset improves automated classification accuracy. It fine-tunes BERT-family models with a voting scheme and reports higher F1 scores, but the improvement is measured against the models' own labels.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation is circular: hard-voted model predictions are used as test ground truth in the expanded datasets, and the comparison to HDLTex is confounded by different train/test compositions.","rationale":"The reader's weakest assumption identifies exactly the circular use of hard-voted pseudo-labels as ground truth for testing. The paper's central evaluation (Section V.D, Tables VII) is built on a dataset whose test split includes model-generated labels, making the reported F1 scores inflated by construction. The comparison to HDLTex in Table VIII is additionally confounded because the baselines were evaluated on the original WoS-46985, not on the expanded dataset. These problems jointly undermine the central claim that dataset expansion with hard-voting improves classification accuracy. The fix is straightforward—evaluate on gold-labeled data only and use a matched baseline—but without it the results do not support the claims. I therefore agree with the reader's REJECT verdict and see no reason to adjust it.","tokens_in":9406,"tokens_out":2604,"duration_ms":28362,"concrete_test":"Compute micro-F1 for each fine-tuned model only on the subset of the WoS-53949 test split whose labels come from the original WoS-46985 gold labels, and compare those scores against HDLTex (and the prior fine-tuned models from [20]) evaluated on the same gold test set. If the gold-only F1 does not exceed the baselines, or drops materially relative to the mixed-label result, the dataset-expansion claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that dataset expansion with hard-voted pseudo-labels improves scientific text classification, with SciBERT reaching micro F1 0.8924 on WoS-53949 (Table VII). The load-bearing weakness is in Section III.B and IV.B: 6,964 added articles are labeled by the models themselves via hard voting, and these labels are treated as ground truth. Section V.D then uses the full WoS-53949 dataset—including these pseudo-labeled articles—with an 80/10/10 split (Table IV), so roughly one-seventh of the test set consists of articles whose labels were produced by the very models being evaluated. Reported micro-F1 therefore measures agreement with the models' own prior predictions, not true classification accuracy. This circularity is compounded by Table VIII, which compares the expanded dataset results against HDLTex and prior fine-tuning results on the original WoS-46985, so any improvement is confounded with the change in test-set composition. The claimed superiority over baselines is thus not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes expanding three Web of Science classification datasets (WoS-5736, WoS-11967, WoS-46985) by querying the WoS database for 1,000 articles per target category, labeling the retrieved unlabeled articles via hard voting over the predictions of four PLMs (BERT, SciBERT, BioBERT, BlueBERT) that were fine-tuned in the authors' prior work, and then fine-tuning the same four PLMs on the expanded datasets. The authors report that the expanded datasets improve classification performance, with the best result being SciBERT at micro F1 0.8924 on WoS-53949 (Table VII), and that their models outperform HDLTex and other baselines (Table VIII). They attribute the gains to dataset augmentation, hard-voting, dynamic learning rates, and early stopping.","tokens_in":9620,"tokens_out":6120,"duration_ms":56946,"significance":"If the central claim were sound, the paper would offer a practical recipe for cheap dataset expansion: using model-generated labels plus hard voting to improve scientific text classification without manual annotation, with evidence that domain-specific PLMs benefit. The authors also provide a GitHub repository, which is helpful for reproducibility. However, the experimental design does not support the claim as it stands: the pseudo-labels are used as ground truth for both training and test, and the headline comparison against HDLTex mixes changes in label source with changes in test-set composition. The practical significance is therefore not established by the reported experiments.","major_comments":[{"comment":"The evaluation is circular. The 6,964 articles added to WoS-46985 to form WoS-53949 are labeled by hard voting over the predictions of the fine-tuned PLMs from [20] (Sections III.B and IV.B). Table IV then splits the combined WoS-53949 into 80/10/10, so the test set of 10,790 documents contains on average about 1,393 model-labeled articles. Fine-tuning on this expanded dataset and testing on the same split therefore measures agreement with the earlier models' predictions rather than accuracy with respect to true categories. The reported micro-F1 values in Table VII cannot be read as classification accuracy on genuinely labeled data.","section":"III.B, IV.B, V.D (Tables IV and VII)"},{"comment":"The comparison against baselines is confounded. Table VIII compares the expanded-dataset results (WoS-53949, WoS-18932, WoS-8716) with HDLTex and other methods evaluated on the original WoS-46985, WoS-11967, and WoS-5736 datasets, and with the authors' prior fine-tuning results [20] on the original datasets. Because the test sets differ in composition (original vs. expanded) and in label source (human labels vs. model-generated hard-voted labels), any difference in accuracy or F1 could be caused by label distribution or test-set shift rather than by dataset expansion. A controlled comparison would fine-tune the same models on the original dataset with the identical 80/10/10 split and report both training-only and expanded-training results on the same held-out test partition.","section":"Table VIII and Section V.D"},{"comment":"No evidence is provided that the hard-voted pseudo-labels are accurate. Table VI compares model predictions with 'query classes,' but query classes are not ground-truth labels: a query for a category returns articles that match the query terms, not necessarily articles belonging exclusively to that category. Agreement with query classes is therefore a weak and potentially biased proxy. Without human evaluation or a held-out gold-standard subset, the hard-voting step (Figure 1) may be selecting confident errors, and the label noise in the expanded dataset is unquantified.","section":"IV.B and Table VI"},{"comment":"The paper claims in Section VI that the hard-voting strategy improved classification accuracy, but no ablation is reported. The reported results compare models fine-tuned on the expanded dataset with models fine-tuned on the original dataset, and the expanded dataset differs from the original in both size and label source; the effect of hard voting per se is not isolated. A comparison between training on labels from a single model and training on hard-voted labels, with the same added articles, would be needed to support this contribution.","section":"VI and Table VII"}],"minor_comments":[{"comment":"The text says the expanded datasets contain 8,715, 18,932, and 53,949 unique documents, but Tables I--III and Table IV use 8,716, 18,932, and 53,949; the first number should be corrected.","section":"III.B"},{"comment":"The text says BERT achieved the highest micro F1 (0.9782), but the following sentences report SciBERT and BioBERT at 0.9816, which are higher; the sentences are internally inconsistent.","section":"V.F"},{"comment":"Many rows report identical micro F1, micro recall, and micro precision; given class imbalance and potential label noise, this should be explained or the numbers should be reported to more decimal places.","section":"Table VII"},{"comment":"The terms 'LLM' and 'PLM' are used interchangeably, but BERT-class models are encoder-only pre-trained language models; the terminology should be made consistent and precise.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript is largely an incremental extension of the authors' own prior work [20], and the new ingredient, pseudo-labeling via hard voting, is exactly where the evaluation is circular. The central empirical claim cannot be assessed without a redesigned protocol that uses gold-standard labels for evaluation and compares baselines under identical train/test conditions. I would not encourage resubmission without that redesign."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the central claim—that query-based dataset expansion plus hard-voting improves scientific text classification—is not supported by the experiments as run. The added articles' labels come from the same models under test, and those labels appear in the test set. The comparison to HDLTex is also confounded. I agree with the reader's take on this.\n\nWhat's actually new: the specific numbers on three expanded WoS datasets (e.g., SciBERT at 0.8924 on WoS-53949) and the released code/data on GitHub. The method itself is not new—pseudo-labeling via self-training and hard-voting are established techniques—so the value is in the clean application and reproducible setup.\n\nWhat the paper does well: clear structure, honest limitations section, and a public repository with datasets and fine-tuning results. That is real evidence of reproducibility and should be credited.\n\nWhere it falls down: Section III.B describes generating labels for the 6,964 retrieved articles through multi-model inference and hard-voting. Section V.D then fine-tunes on the full expanded dataset and evaluates on a random 80/10/10 split. Since the added articles are roughly 13% of the total, about 1,400 of the 10,790 test instances carry labels produced by the very models being measured. Reported micro-F1 therefore measures consistency with the prior models' predictions, not agreement with true categories. There is no same-protocol baseline on the original WoS-46985, so the claimed improvement over HDLTex in Table VIII is confounded by a changed test set. That is a load-bearing flaw, not a minor shortcoming.\n\nIs the argument otherwise sound? The inference tables (Table VI) suggest the query-based categories are roughly aligned with the models' predictions, but that only shows the queries are coherent, not that the pseudo-labels are correct enough to serve as ground truth.\n\nWho would get value from this: someone looking for a worked example of a pseudo-labeling pipeline on WoS, with ready-made code. As a scientific claim about improving classification, the paper doesn't stand.\n\nMy recommendation: desk reject. The evaluation would need to be redone on a held-out human-labeled test set, or at least with the original WoS test split reported separately. That is a major revision and the authors should be told exactly why. A referee's time is better spent on papers whose central comparisons are clean.","headline":"Pseudo-labeled test data make the reported gains untrustworthy, but the paper's reproducible artifacts and clean write-up are worth acknowledging.","tokens_in":10096,"tokens_out":3471,"would_cite":false,"duration_ms":34852,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding query-retrieved articles labeled by model agreement improves scientific text classification, with SciBERT reaching micro F1 0.8924 on the expanded WoS-53949 set.","keywords":["scientific text classification","pre-trained language models","dataset augmentation","hard-voting","Web of Science","fine-tuning","domain-specific models","early stopping"],"falsifier":"Have human annotators label a random sample of the 6,964 added articles, compare their labels with the hard-voted labels, and if agreement is low, retrain on human-corrected labels; the central claim survives only if the F1 gains persist.","tokens_in":9247,"feed_emoji":"📄","tokens_out":5806,"duration_ms":48977,"temperature":0.7,"pith_summary":"This paper claims that scientific text classification can be improved without new human annotation by pulling additional articles from the Web of Science database and labeling them with a hard-voting ensemble of four fine-tuned language models. On the expanded WoS-53949 dataset, the best model, SciBERT, reports micro F1 0.8924, beating BERT and earlier hierarchical deep-learning baselines such as HDLTex. The result matters because curated labels are the main bottleneck in scientific document organization; if pseudo-labeled retrieval works, larger and more current training sets become cheap.","feed_headline":"SciBERT hits 0.8924 F1 after voted dataset expansion","feed_subtitle":"Adding 7,000 query-retrieved articles labeled by model agreement beats prior deep-learning baselines.","key_machinery":"The mechanism is a three-stage loop. First, seven targeted queries retrieve 1,000 candidate articles per WoS category. Second, four fine-tuned pre-trained language models each predict a label, and a hard-voting rule, which assigns the category receiving the most model votes, keeps only high-agreement articles. Third, the newly labeled articles are merged with the original WoS datasets, and the models are fine-tuned on the expanded sets using AdamW, a dynamic learning-rate search over four values, linear warmup, and early stopping on validation F1.","core_discovery":"Fine-tuning pre-trained language models on WoS-46985 plus 6,964 query-retrieved articles, whose labels come from agreement among BERT, SciBERT, BioBERT, and BlueBERT, raises classification performance over the original data alone. The paper reports SciBERT at micro F1 0.8924 on WoS-53949, BioBERT at 0.9247 on WoS-18932, and BERT at 0.9782 on WoS-8716, with domain-specific models generally ahead of general-purpose BERT. These results are presented as outperforming the HDLTex hierarchical deep-learning baseline and the earlier fine-tuning study on the original datasets.","pith_inferences":["Since the hard-voted labels come from the same models that are later evaluated on the expanded test split, the reported F1 gains may partly reflect agreement with the models' own biases; a human-annotated sample of the added articles would show how much of the gain is real category accuracy.","The same query-retrieve-vote-then-fine-tune loop could transfer to other taxonomies, such as arXiv subject classes or PubMed topics, where the bottleneck is also curated labels.","The paper's finding that keyword-only and abstract-only predictions behave differently across domains suggests a testable variant: fuse the two input views before voting, not after."],"forward_implications":["On the largest expanded set, WoS-53949, SciBERT reaches micro F1 0.8924, above BERT's 0.8764 and above the 76.58 percent accuracy reported for HDLTex on the original WoS-46985.","Domain-specific models (SciBERT, BioBERT) outperform general-purpose BERT on most expanded datasets, supporting the paper's claim that scientific pre-training plus broader category coverage carries the gain.","The expansion recipe works across all three WoS sizes, with the smallest set WoS-8716 reaching micro F1 above 0.97 for three of the four models.","Using pseudo-labeled retrieved articles as additional training data, combined with dynamic learning rates and early stopping, is presented as a scalable way to keep classifiers current without new manual annotation."],"supporting_citations":[{"why":"Supplies the original WoS-5736, WoS-11967, and WoS-46985 datasets plus the HDLTex baseline that the expanded-data results are compared against.","marker":"[30]"},{"why":"Provides the fine-tuned BERT, SciBERT, BioBERT, and BlueBERT models used for inference and hard-voting, and the prior fine-tuning performance that this study extends.","marker":"[20]"},{"why":"SciBERT is the best-performing model on WoS-53949; the comparison rests on its domain-specific pre-training.","marker":"[18]"},{"why":"BioBERT achieves the top micro F1 0.9247 on WoS-18932 and is a key domain-specific comparator.","marker":"[19]"},{"why":"BERT is the general-purpose baseline model whose lower performance supports the domain-specific advantage claim.","marker":"[21]"},{"why":"BlueBERT contributes the fourth vote in the hard-voting ensemble and a biomedical-domain comparison point.","marker":"[25]"}],"fun_headline_variants":["Hard-voting + query expansion lift SciBERT to 0.8924","Expanded data, hard voting: SciBERT tops 0.8924","Agreement labels from 4 models expand training set","Model consensus labels 7,000 new articles for fine-tuning","Voting on query articles boosts domain-specific PLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hard-voted pseudo-labels assigned to the roughly 7,000 added articles are correct enough to serve as ground truth in both the training and test splits.","fun_headline_variants_meta":{"raw":{"variants":["Hard-voting + query expansion lift SciBERT to 0.8924","Expanded data, hard voting: SciBERT tops 0.8924","Agreement labels from 4 models expand training set","Model consensus labels 7,000 new articles for fine-tuning","Voting on query articles boosts domain-specific PLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001126,"raw_usage":{"total_tokens":4638,"prompt_tokens":856,"completion_tokens":3782,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":3694}},"tokens_in":472,"tokens_out":3782,"duration_ms":25744,"temperature":1.0,"reasoning_tokens":3694,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:02:29.817810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators label a random sample of the 6,964 added articles, compare their labels with the hard-voted labels, and if agreement is low, retrain on human-corrected labels; the central claim survives only if the F1 gains persist.","supporting_citations":[{"cited_title":"Fine-tuning large language models for scientific text classification: A comparative study,","cited_arxiv_id":null,"evidence_quote":"Provides the fine-tuned BERT, SciBERT, BioBERT, and BlueBERT models used for inference and hard-voting, and the prior fine-tuning performance that this study extends."}],"review_version":1}