{"id":"07cc9235-b455-4857-92a2-bb0299ae29e9","arxiv_id":"2411.17614","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuned SciBERT on the ETD-CL dataset achieves the best chapter classification F1 (0.67), and multi-label top-3 prediction raises accuracy to 0.91.","lead":"This paper tests whether machine learning and large language models can automatically label chapters of theses and dissertations with academic subject categories. The authors find that fine-tuned BERT and SciBERT models outperform traditional classifiers, and that allowing multiple labels per chapter improves accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ETD-CL labels are department metadata mapped to ProQuest categories (Sec. III-E), not chapter-content labels, so the reported F1=0.6666 does not establish chapter-level classification; agreement with content-based labels must be tested.","rationale":"The reader's weakest-assumption analysis identifies precisely the concern that is most load-bearing: the ETD-CL ground truth comes from departmental metadata rather than from chapter content. This is not a peripheral issue; it determines what the reported numbers mean. The paper's own introduction argues that ETD chapters are often interdisciplinary and that document-level metadata is insufficient; yet the evaluation labels every chapter with the document's department-derived ProQuest category, directly contradicting that motivation. Consequently, the headline F1 values and the model comparisons are measured against a proxy that can be systematically wrong for exactly the cases the system is intended to handle. The ranking between models could survive even with noisy labels, but the central claim about chapter-level classification is not established without content validation. I considered a second concern, namely possible overlap between the FTD fine-tuning corpus and the ETD-CL evaluation set, since both come from the same large ETD collection and the paper does not state they are disjoint. That is a real risk to the fine-tuning improvement claim, but the label-proxy issue is directly evidenced in the text and affects the core construct being measured. The lack of released code/data and significance testing are secondary weaknesses, and the abstract's discoverability claim is tempered by the conclusion's statement that quantifying improvement is future work. The conditional verdict remains appropriate: the paper is a useful comparative study, but its chapter-level classification claim needs a content-validated evaluation before it can be accepted at face value.","tokens_in":12731,"tokens_out":6992,"duration_ms":61631,"concrete_test":"Construct a content-validated test set from ETD-SGT: take 300 chapters with manually identified boundaries, have two annotators independently assign ProQuest categories based only on the chapter text, and compute annotator agreement and agreement between content-based labels and the department-derived labels. Then re-run the Table III comparison (SVM, BERT, SciBERT, SciBERT+ETD) on this content-labeled set. If agreement with the department-derived labels is low (e.g., Cohen's kappa below 0.6), or if the ranking or F1 changes materially, the ETD-CL results should be reported as department-proxy agreement rather than chapter-classification accuracy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C says ETD-CL is built by selecting 200 documents from each of 47 departments, and Section III-E says labels are produced by mapping 'department information from the ETD-CL metadata to the corresponding ProQuest categories.' Table I nonetheless describes ETD-CL as 'Manually assigned labels.' As a result, every chapter (or document; the unit is not made explicit) inherits the department's ProQuest category. The introduction motivates the task with dissertations whose chapters cross fields, such as a statistics chapter inside an environmental-science dissertation; such chapters are mislabeled by construction. The reported F1=0.6666 for SciBERT+ETD (Table III) therefore measures how well the model recovers a department-derived proxy, not whether chapter content is correctly categorized. The comparative ranking of models may still be informative, but the central claim about achieving chapter-level classification is not directly supported. The paper provides no manual content validation of the ETD-CL labels, and the 'Manually assigned labels' wording in Table I conflicts with the mapping procedure in Section III-E.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses a real problem in digital libraries: ETDs lack chapter-level metadata, so the authors propose a pipeline that segments ETDs, extracts text, and classifies chapters into ProQuest subject categories. They compare random forests, SVM, BERT, SciBERT and their fine-tuned versions on two datasets (PQDT and the newly introduced ETD-CL), evaluate multi-label top-3 prediction, and experiment with Llama-2 and Llama-3 with zero-shot, few-shot, and instruction-tuned prompting. The main reported results are that language models outperform traditional classifiers, fine-tuning on an ETD corpus improves F1 (SciBERT+ETD reaches 0.6666 on ETD-CL), and top-3 multi-label accuracy reaches 0.91.","tokens_in":12946,"tokens_out":6646,"duration_ms":53318,"significance":"If the chapter-level claim were supported, this would be a useful contribution to computational archival science and ETD accessibility: the manual segmentation effort (ETD-SGT) and the systematic comparison across six classifiers plus two LLM families provide a useful reference point. The paper also deserves credit for including a concrete error analysis for LLM output (82 predicted classes versus 47 labels) and a reproducibility-conscious description of the experimental setup. However, the central evaluation is weakened by the construction of ETD-CL labels as department metadata mapped to ProQuest categories rather than as content-based chapter labels, and by several evaluation-design issues. The comparative ranking of models may be informative, but the headline chapter-level interpretation is not currently supported.","major_comments":[{"comment":"The ETD-CL labels are not chapter-content labels. Section III-C says ETD-CL was built by selecting 200 documents per department, and Section III-E says labels are obtained by mapping department metadata to ProQuest categories; however, Table I describes ETD-CL as \"Manually assigned labels.\" Because each chapter inherits the department's ProQuest category, the motivating example in the Introduction (an environmental-science dissertation with a statistics chapter) is mislabeled by construction. The F1 values in Table III therefore measure how well the classifier recovers a department-derived proxy, not whether chapter content is categorized correctly. The authors should either validate the mapping against human content-based annotations on a sample of chapters, or reframe the experiments explicitly as department-level classification. This is the central claim of the paper, so it must be addressed before the chapter-level results can be accepted.","section":"Sec. III-E and Table I"},{"comment":"The relationship between the manually segmented ETD-SGT corpus (244 documents, 11 departments) and the ETD-CL corpus (9,400 documents, 47 departments) is never made explicit. Tables II and III are reported as classification results, but it is unclear whether the input units are chapters or whole documents, and whether ETD-CL is segmented at all. If the ETD-CL experiments use whole documents, then the \"chapter-level classification\" claim is not tested. Please specify the unit of analysis for every table and, if necessary, rerun the comparison on the segmented chapters in ETD-SGT.","section":"Sec. III-A, III-C and Tables II-III"},{"comment":"The multi-label comparison is not apples-to-apples. Table VI compares multi-class top-1 accuracy (0.60–0.66) with sigmoid-based top-3 accuracy (0.85, 0.91), and \"accuracy\" is defined as whether the single ground-truth label appears among the top three predictions. Since the chance baseline differs between top-1 and top-3 (e.g., 1/47 versus 3/47 for random selection), these numbers are not directly comparable to the top-1 accuracies. To support the RQ3 conclusion that multi-label classification outperforms multi-class classification, the authors should report matched evaluation criteria (e.g., top-1 accuracy for both, or top-3 recall with the same number of allowed labels) and ideally precision/recall at the top-3 cutoff.","section":"Table VI and Sec. V-D1"},{"comment":"No statistical significance or variance is reported for the core classifier comparison. The gap between SciBERT (F1=0.6592) and SciBERT+ETD (F1=0.6666) in Table III is small, and the paper does not state how the train/test splits were performed or whether repeated runs were averaged. Without confidence intervals or significance tests, the claims that fine-tuning helps (RQ2) and that language models outperform SVM/RF rest on point estimates that may not be stable. Please add standard deviations or significance tests, and describe the splitting procedure.","section":"Sec. V-A and Tables II-III"}],"minor_comments":[{"comment":"The term \"Receiver Operative Characteristic\" should be \"Receiver Operating Characteristic.\"","section":"Sec. II"},{"comment":"The sentence \"Performance for instruction-tuned Llama-2 is compared with Llama-3 in Fig. IV\" references a nonexistent figure; it should refer to Table IV.","section":"Sec. V-C1"},{"comment":"The statement that setting the temperature to 0 would result in a division-by-zero error is not a general property of LLM sampling implementations; please clarify the specific framework behavior.","section":"Sec. V-C"},{"comment":"In the few-shot row, the standard deviation for F1 is reported as 0 while the standard deviation for recall is 0.0057; this inconsistency should be explained or corrected.","section":"Table V"},{"comment":"The paper says all three ProQuest category levels and the subject code are recorded, but it does not specify which level is used as the classification label in Tables II, III, and VI; please state this explicitly.","section":"Sec. III-E"},{"comment":"The choice of 0.6 as the cosine-similarity threshold for \"limited alignment\" is not justified; please provide a rationale or use a more standard agreement measure.","section":"Sec. V-D2"}],"recommendation":"major_revision","confidential_remarks":"The paper is an application-oriented contribution suited to a digital libraries / computational archival science venue. The label-validity issue in ETD-CL is substantial but fixable by adding a content-based validation study or by explicitly weakening the claims to document-level classification. I also note that no code or dataset availability statement is included, which limits reproducibility. The comparative model ranking is a useful baseline even if the chapter-level framing needs revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"One thing to know: the paper's headline numbers do not mean what they claim. The ETD-CL ground truth is built by mapping department metadata to ProQuest categories (Sec. III-E), so every chapter inherits the document's department. The F1=0.6666 for SciBERT+ETD measures how well the model recovers that department proxy, not chapter-level classification. The introduction motivates with interdisciplinary chapters (statistics inside an environmental-science dissertation) that are mislabeled by construction. Table I says 'Manually assigned labels' but the mapping procedure contradicts that; no manual content validation is reported.\n\nWhat is genuinely useful: the paper gives a systematic comparison of SVM/RF, BERT/SciBERT, fine-tuned variants, and LLMs, with a multi-label top-3 setting. Fine-tuning on the FTD corpus consistently helps, and the top-3 accuracy (0.91) is a sensible way to handle interdisciplinarity. The LLM section is honest about formatting failures, 82 predicted classes vs 47, and low similarity scores; the standard deviations for Llama runs are a nice touch. The ETD-CL dataset, despite its label problem, is a contribution the community can reuse or critique.\n\nThe soft spots are proportionate to the critique above. The label proxy is load-bearing: it undermines the central claim. Second, there are no significance tests; the gap between BERT and SciBERT might be noise. Third, the relationship between Tables II and III is unclear—Table II likely uses PQDT and Table III ETD-CL, but the text says both were used. Fourth, the abstract promises improved discoverability while the conclusion admits future work will quantify it. These are fixable, but they need attention.\n\nWho is this for? Digital library and archival informatics researchers. They will find the comparison useful as a benchmark, but they should treat the F1 scores as results on a department-label prediction task, not chapter classification. I would send this to a serious referee: the task is relevant, the flaw is identifiable, and a revision with content-based label validation would make it solid. My own verdict: skeptical of the central claim as is.","headline":"Solid classifier benchmark for ETD chapters, but the ground truth is department metadata, not chapter content, so the reported F1 numbers overstate the claim.","tokens_in":13466,"tokens_out":2700,"would_cite":false,"duration_ms":23707,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned language models can automatically classify individual thesis chapters by subject, outperforming traditional machine learning and generic pre-trained models.","keywords":["electronic theses and dissertations","chapter-level classification","BERT","SciBERT","large language models","multi-label classification","metadata","digital libraries"],"falsifier":"Have two annotators read a random sample of, say, 200 ETD-CL chapters and assign ProQuest categories from content alone, then measure agreement with the department-derived labels; if agreement is near chance, the reported F1 scores mostly reflect departmental signal, not chapter-level content.","tokens_in":12546,"feed_emoji":"🎓","tokens_out":7478,"duration_ms":63106,"temperature":0.7,"pith_summary":"Electronic theses and dissertations are usually described by document-level metadata, so a reader cannot search inside them by chapter. This paper proposes automatically assigning subject labels to individual chapters, using a pipeline that segments the document, extracts clean text, and classifies each chapter into ProQuest subject categories. On a 9,400-document benchmark built from departmental metadata, the authors find that language-model classifiers (BERT and SciBERT) clearly outperform traditional machine learning baselines (SVM and random forest), and that fine-tuning on an ETD corpus improves them further, with the best model reaching $F_1 = 0.6666$. They also find that allowing a chapter to receive its top three labels raises accuracy markedly, reflecting the interdisciplinary character of many ETD chapters, while open-ended LLMs produce plausible but unreliable labels.","feed_headline":"Fine-tuned SciBERT beats older models at labeling thesis chapters","feed_subtitle":"Chapter-level tags would let researchers search inside dissertations and surface interdisciplinary content.","key_machinery":"The load-bearing object is the classification benchmark ETD-CL: 9,400 documents from 47 departments, with each document's department metadata mapped to the three-level ProQuest subject taxonomy to form class labels. Around it sits a pipeline: manual chapter segmentation (ETD-SGT), text extraction that combines a commercial OCR service with an object-detection model to strip headers, footers, captions, figures, and equations, and supervised classifiers trained on the ProQuest label space. The decisive mechanism is fine-tuning: BERT and SciBERT are adapted on the FTD corpus of 8,200 born-digital ETDs, and this domain adaptation is what pushes the bidirectional models past both the traditional baselines and their own pre-trained versions. For multi-label output, the same models swap softmax for sigmoid and take the top three predictions.","core_discovery":"The central claim is that chapter-level classification of ETDs is not only feasible but best done by domain-adapted bidirectional language models. Fine-tuning BERT and SciBERT on born-digital ETD text improves both models on both test sets; SciBERT+ETD reaches $F_1 = 0.6666$ on the ETD-CL dataset, the best of any single-label classifier tested, versus $F_1 = 0.340$ for SVM and $F_1 = 0.228$ for random forest. When the task is relaxed to top-three multi-label prediction with a sigmoid output, the fine-tuned SciBERT model achieves 0.91 accuracy, and the paper reads this as evidence that multi-label prediction better captures the interdisciplinary nature of chapters. For generative LLMs, the paper reports best $F_1$ around 0.53, with instruction-tuned Llama-2 and few-shot Llama-3 producing labels that often fall outside the fixed 47-class set and require subject-matter expertise to map.","pith_inferences":["The reported top-three accuracy is a relaxed hit-rate, not a full multi-label metric; a stricter evaluation would require predicting every relevant label per chapter, which would need content-based annotations.","The ground-truth labels come from department metadata, so a content-based re-annotation of a random chapter sample would test whether the benchmark measures chapter content or departmental affiliation.","A natural follow-up is a retrieval experiment: index chapters by these labels and measure whether scholars find relevant sections faster than with document-level metadata; the paper lists quantifying this improvement as future work."],"forward_implications":["Once chapters are segmented, the classification step itself is fully automatic: chapter text plus a fixed label set is enough for the fine-tuned models to produce subject tags.","Fine-tuning on ETD text is a worthwhile cost: BERT+ETD and SciBERT+ETD beat their pre-trained counterparts on both the PQDT and ETD-CL datasets.","Multi-label top-three prediction is a better fit for interdisciplinary chapters than single-label classification, lifting accuracy from around 0.66 to 0.85–0.91 for the fine-tuned models.","Generative LLMs are not yet reliable classifiers for this task: Llama-2 and Llama-3 underperform fine-tuned SciBERT and produce output that does not stay within the label set."],"supporting_citations":[{"why":"Supplies the earlier SVM and random forest classification of ETD chapters into 28 ProQuest categories, the traditional baseline this paper extends and compares against.","marker":"[21]"},{"why":"Describes the multi-university collection of over half a million ETDs from which ETD-CL and FTD are drawn.","marker":"[35]"},{"why":"Prior exploratory analysis of the same ETD collection that motivates the classification study and provides context for the fine-tuning experiments.","marker":"[23]"},{"why":"Defines BERT, the bidirectional architecture used as one of the language-model classifiers and as the base for fine-tuned variants.","marker":"[9]"},{"why":"Defines SciBERT, the scientific-text pre-trained model whose ETD-fine-tuned version achieves the best results.","marker":"[10]"},{"why":"Provides the ProQuest subject category taxonomy used as the label space for all classification tasks.","marker":"[22]"},{"why":"Supplies the object-detection model used to filter page elements during chapter text extraction.","marker":"[37]"},{"why":"Provides the Sentence-BERT embeddings used to compute cosine similarity between LLM-generated subcategories and ground truth categories.","marker":"[42]"}],"fun_headline_variants":["SciBERT fine-tuned on ETDs outperforms SVM at chapter labeling","Chapter-level metadata via fine-tuned SciBERT for theses","Automated chapter classification for ETDs with SciBERT","Fine-tuned SciBERT bests classical ML for thesis chapters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark labels are derived from each document's department mapped to ProQuest categories, assuming that what a chapter says matches the department of the whole dissertation; the paper does not verify this against chapter content.","fun_headline_variants_meta":{"raw":{"variants":["SciBERT fine-tuned on ETDs outperforms SVM at chapter labeling","Chapter-level metadata via fine-tuned SciBERT for theses","Automated chapter classification for ETDs with SciBERT","Fine-tuned SciBERT bests classical ML for thesis chapters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1643,"prompt_tokens":1012,"completion_tokens":631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":560}},"tokens_in":628,"tokens_out":631,"duration_ms":6135,"temperature":1.0,"reasoning_tokens":560,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:54:38.541346+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two annotators read a random sample of, say, 200 ETD-CL chapters and assign ProQuest categories from content alone, then measure agreement with the department-derived labels; if agreement is near chance, the reported F1 scores mostly reflect departmental signal, not chapter-level content.","supporting_citations":[{"cited_title":"Increasing Accessibility of Electronic Theses and Disser- tations (ETDs) Through Chapter-level Classification,","cited_arxiv_id":null,"evidence_quote":"Supplies the earlier SVM and random forest classification of ETD chapters into 28 ProQuest categories, the traditional baseline this paper extends and compares against."},{"cited_title":"Building A large collection of multi-domain electronic theses and dissertations,","cited_arxiv_id":null,"evidence_quote":"Describes the multi-university collection of over half a million ETDs from which ETD-CL and FTD are drawn."},{"cited_title":"Applications of data analysis on scholarly long documents,","cited_arxiv_id":null,"evidence_quote":"Prior exploratory analysis of the same ETD collection that motivates the classification study and provides context for the fine-tuning experiments."},{"cited_title":"SciBERT: A pretrained language model for scientific text,","cited_arxiv_id":null,"evidence_quote":"Defines SciBERT, the scientific-text pre-trained model whose ETD-fine-tuned version achieves the best results."},{"cited_title":"Subject Categories 2019-2020 Academic Year,","cited_arxiv_id":null,"evidence_quote":"Provides the ProQuest subject category taxonomy used as the label space for all classification tasks."},{"cited_title":"Parsing Electronic Theses and Dissertations Using Object Detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the object-detection model used to filter page elements during chapter text extraction."}],"review_version":1}