{"id":"b6c493b1-2023-4597-9acd-3b6990bbe146","arxiv_id":"2501.00174","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":1.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey monograph on text classification pipeline stages, with proposed preprocessing acronyms and a reproduced word-embedding case study, but no new results.","lead":"This preprint is a survey of text classification, covering datasets, preprocessing, representations, classifiers, evaluation metrics, and GPT-based prompting. There are no new experiments or results; the practical value is a structured overview, though with notable editorial errors.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The monograph's central claim of evaluating each pipeline component is not self-contained: §3.3.17 announces a preprocessing evaluation that is actually in Siino et al. 2024c, and §4.2 imports an analysis from Siino et al.","rationale":"The reader's weakest assumption and my concern share a root: the claimed novel components are not derived in this monograph. I focus on a slightly more specific internal inconsistency: §3.3.17 says 'We then performed an evaluation' but immediately delegates to Siino et al. 2024c, and no such evaluation is present in the text. This is load-bearing because the abstract's central promise is to 'evaluate the impact of each component.' If the monograph is read as a standalone reference, the evaluation claim is unverifiable. At the same time, the survey portions—datasets, preprocessing techniques, classifiers, and metrics—are broadly consistent with the literature and have reference value; I would not reject. The fix is textual and scoping: either include the evaluation or change the claims. This is exactly why the reader's conditional verdict remains appropriate.","tokens_in":45613,"tokens_out":4236,"duration_ms":42961,"concrete_test":"Build an inventory of every table, figure, and quantitative comparison in the manuscript, tagging each as 'original to this text' or 'reported from a cited work.' Specifically check §3.3 for an experimental protocol and results table for the claimed four-dataset, three-technique evaluation. If no such evaluation appears, the monograph's only concrete pipeline evaluations are imported from Siino et al. 2022a/2024c/2024d, and the abstract and Chapter 3 should be revised to claim 'surveys and synthesizes' rather than 'evaluates' and 'extends beyond a typical survey.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract promises 'evaluating the impact of each component on the overall performance of text classification models' and says the work's contributions 'extend beyond a typical survey.' The monograph's only concrete pipeline evaluations, however, are not performed or presented in the manuscript. In §3.3.17 the text states: 'We then performed an evaluation and comparison of the three most common techniques in four datasets from various domains.' The next sentence immediately attributes the actual experiments to 'the study in Siino et al., 2024c'; no experimental setup, dataset splits, results table, or analysis appears in this chapter. Similarly, §4.2 says the case study is 'from a previous work of ours (Siino et al., 2022a)', and Figures 4.2 and 4.3 are imported from that work. Thus the claimed 'extension beyond a typical survey' is not verifiable from the monograph alone: a reader cannot check the central evaluation claim without consulting external prior papers. This is a self-containedness and correctness-risk issue, not a matter of disagreeing with a consensus. The survey portions have reference value, but the distinctive contribution advertised in the abstract is unsupported within the submitted text.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a monograph-length survey of text classification organized by pipeline stages: tasks and datasets (Ch. 2), preprocessing (Ch. 3), text representation (Ch. 4), classifiers from traditional ML to transformer-based LLMs/GPTs (Ch. 5), and evaluation metrics (Ch. 6), with conclusions in Ch. 7. The authors state in the abstract and Sec. 1.1 that the work not only surveys but also evaluates 'the impact of each component on the overall performance' and that its contributions 'extend beyond a typical survey.' The two components that would support this stronger claim are (i) a comparison of preprocessing techniques and (ii) a case-study analysis of a learned word-embedding space; both are explicitly drawn from the authors' earlier papers (Siino et al., 2024d/2024c and Siino et al., 2022a, respectively) and are not derived or reported in this manuscript. The survey portions cover a broad literature and include useful taxonomies, but the distinctive advertised contribution is not self-contained.","tokens_in":45845,"tokens_out":8713,"duration_ms":75374,"significance":"The manuscript has genuine reference value: it assembles a wide range of datasets, preprocessing operations with a systematic acronym-based nomenclature, static and contextual embedding models, traditional and deep classifiers, transformer architectures, prompt-engineering techniques, and evaluation metrics, with many pointers to recent work. The comparative organization around the full pipeline is a useful pedagogical framing. However, the claim that the monograph evaluates the impact of each pipeline component is not verifiable from the submitted text because the only concrete evaluations are imported from the authors' previous publications. There are also internal technical errors, including the Bayes formula in Sec. 5.1.2, Eq. (4.3), citation years, and figure-referencing inconsistencies, that a careful reader would notice in a work advertised as an accurate single-source reference. The weaknesses are localizable and fixable by adding the actual experimental material or by rewriting the claims, so the appropriate decision is major revision rather than rejection.","major_comments":[{"comment":"The abstract and §1.1 promise that the monograph evaluates 'the impact of each component on the overall performance of text classification models' and that the contributions 'extend beyond a typical survey.' The only experimental evaluation in the preprocessing chapter is not actually in the manuscript: §3.3.17 first says 'We then performed an evaluation and comparison of the three most common techniques in four datasets from various domains,' and then attributes the tests, results, and best/worst strategies to 'the study in Siino et al., 2024c' without presenting any experimental setup, dataset splits, results table, or analysis. As written, the central evaluation claim is unverifiable from the submitted text. The authors should either include the full evaluation in this monograph or revise the abstract and chapter to state unambiguously that this evaluation was reported in Siino et al., 2024c and is only summarized here.","section":"Abstract; §3.3.17"},{"comment":"The case study in §4.2 is presented as one of the monograph's contributions ('we present the results of a case study analyzing a word embedding trained from scratch'), but the section explicitly states that the analysis is 'from a previous work of ours (Siino et al., 2022a)' and the figures (labeled 4.5 and 4.6 in the captions, but referred to as Figure 4.2 in the text) are imported from that work. No training details, evaluation, or new analysis are provided in this manuscript. The same issue affects the claim in the Chapter 4 overview that 'we also show and analyse a word embedding space trained from scratch.' If this is intended as a contribution, the full analysis must be included or re-derived; otherwise the section should be labeled as a summary of previous work and the abstract adjusted accordingly.","section":"§4.2"},{"comment":"The Bayes theorem passage is garbled and internally duplicated: the sentence beginning 'Given the number of documents n to be classified into z classes' appears twice, and Eq. (5.1) is printed as P(x|y)=P(x)P(y|x)P(y), which is not Bayes theorem. The correct relation is P(x|y)=P(y|x)P(x)/P(y). Since this section is meant to define the Naive Bayes classifier in a reference work, this is a substantive correctness error that needs to be fixed, not merely a typographical one.","section":"§5.1.2, Eq. (5.1)"}],"minor_comments":[{"comment":"ELMo is cited as 'Peters et al.,1802'; the year should be 2018. Please correct this and scan all citations for similar malformed years or missing spaces, e.g., 'Garrido-Merchan et al.,2023'.","section":"§4.1.2"},{"comment":"Equation (4.3), 'tf→id f=tfij↑id fi', is unclear and appears to be a corrupted version of the TF-IDF product; it should be written as tfidf_{ij} = tf_{ij} × idf_i with standard notation.","section":"Eq. (4.3)"},{"comment":"The text refers to 'Figure 4.2', 'Figure 4.2a', and 'Figure 4.2b' for the embedding-space visualizations, but the captions in the manuscript number these figures as Figure 4.5 and Figure 4.6. The figure references need to be reconciled.","section":"§4.2 and Figures 4.5/4.6"},{"comment":"The string 'HaTMLCing' appears in Table 3.1 (e.g., 'Emoticon HaTMLCing', 'Negation HaTMLCing') and in §3.3.2 ('by haTMLCing these informal words'); this looks like a text-generation artifact and should be replaced with the intended word, e.g., 'handling'.","section":"Table 3.1; §3.3.2; §3.3.15"},{"comment":"There are small grammar errors such as 'feds the matrix' in §5.2.3 and 'text text classification' in §5.3.1; the manuscript would benefit from a careful proofreading pass.","section":"§5.2.3 and §5.3.1"},{"comment":"The title page includes both 'The Text Classification Pipeline: Starting Shallow, going Deeper' and a second subtitle 'From Foundations to GPT in Text Classification: A Comprehensive Survey on Current Approaches and Future Trends,' and the 'Suggested Citation' block lists volume/page placeholders ('Vol. xx, No. xx, pp 1–18'). Please clarify the intended title and complete the production metadata.","section":"Title page"}],"recommendation":"major_revision","confidential_remarks":"The manuscript relies heavily on the authors' own prior works (Siino et al., 2022a; 2024b; 2024c; 2024d) for the two components that distinguish it from a survey. Given that the submitted text does not reproduce those experiments, I would ask the editor to verify that these references are published and that the authors have permission to reuse figures from them. The production metadata (title duplication, volume/page placeholders) also needs attention before any acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a survey, not a research contribution. The abstract promises an evaluation of each pipeline component and claims the contributions 'extend beyond a typical survey,' but the two places where that could happen outsource the work: §3.3.17 promises a comparison of preprocessing techniques and then points to Siino et al. 2024c, and §4.2 presents a word-embedding case study from Siino et al. 2022a. Neither is reproduced or verified here. So the distinctive claim is not checkable from the manuscript.\n\nWhat the paper does well: as a survey, it covers the pipeline from datasets and preprocessing through representation, classifiers, and evaluation, with a number of useful tables (e.g., dataset statistics, preprocessing acronyms, related-work matrix). It is organized sensibly and reflects the common literature. For a reader new to text classification, it could serve as a roadmap.\n\nThe soft spots besides the outsourcing: mechanical errors and sloppy presentation. ELMo is cited as 'Peters et al., 1802' in §4.1.2. The Naïve Bayes derivation in §5.1.2 is garbled and partially duplicated, and Equation 4.3 ('tf→id f=tfij↑id fi') is notationally unclear. The figure references in §4.2 don't match the figures as numbered. These are fixable, but they lower confidence in the survey's reliability.\n\nThe citation pattern is heavy on the authors' own prior work, though the cited works are relevant; self-citation alone is not the problem. The problem is that the paper's original-sounding claims rest on those self-citations without including the evidence.\n\nIn short: the survey has reference value, and with careful revision (and deleting the 'beyond a survey' claim) it could be a useful teaching document. As it stands, it is not a serious research contribution. I would still send it to peer review—a good referee could separate the salvageable survey from the unsupported novelty claims and force the fixes. But it should not be accepted without substantial revision.","headline":"A broad but error-prone survey that mostly summarizes known material; the advertised extension beyond a survey consists of pointers to the authors' own prior work rather than results in this text.","tokens_in":46415,"tokens_out":3545,"would_cite":false,"duration_ms":34202,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Preprocessing choices can shift text-classification accuracy by more than 25 percent, and the monograph argues every pipeline stage deserves deliberate tuning.","keywords":["text classification pipeline","text preprocessing","word embeddings","deep learning classifiers","transformers","large language models","prompt engineering","evaluation metrics"],"falsifier":"Re-run the nine-model, four-dataset preprocessing grid behind Chapter 3 with fixed hyperparameters and a pre-registered preprocessing order. If no preprocessing strategy changes accuracy by more than a small margin, or if the reported 25 percent swing does not reproduce, the monograph's central practical claim fails.","tokens_in":45344,"feed_emoji":"📚","tokens_out":9404,"duration_ms":89659,"temperature":0.7,"pith_summary":"This monograph tries to establish that text classification is best understood as a single five-stage pipeline—datasets, preprocessing, representation, classification, and evaluation—whose stages have measurable and interdependent effects on final performance. It gathers the field from one-hot encoding, bag-of-words, and TF-IDF through Word2Vec, CNNs, RNNs, BERT and other LLMs, and GPT prompting techniques, and argues that each stage deserves deliberate, task-specific choices rather than default conventions. The practical claim is that preprocessing alone can change accuracy by more than 25 percent on the same model and dataset, and that a well-matched simple classifier can rival a Transformer. The monograph also presents a post-training analysis of a word-embedding space as a way to see whether classes have separated early in the network. A sympathetic reader would take the book's contribution to be a single source that connects every decision point in the pipeline to measured outcomes.","feed_headline":"Preprocessing alone can swing text-classification accuracy by 25 percent","feed_subtitle":"A new monograph walks the full pipeline and argues that preprocessing choices can gain or lose 25 percent accuracy.","key_machinery":"The carrying object is the pipeline itself, a five-stage flowchart from raw text to predicted labels. Within it, the argument runs on three named taxonomies: the sixteen preprocessing technique acronyms (from DON, meaning do nothing, to WSG, meaning word segmentation), the representation spectrum from one-hot encoding and TF-IDF to static and contextual embeddings, and the three classifier families: TMLCs, FDLMs, and Transformers. The pipeline framing is what lets the authors attribute performance differences to individual stages and give task-specific advice.","core_discovery":"The monograph's central claim is that no single stage of the text classification pipeline can be optimized in isolation: dataset choice sets the task, preprocessing can shift accuracy by more than 25 percent on the same model and dataset, representation determines which semantic information reaches the classifier, and evaluation metrics decide what counts as success. On this view, simple classifiers can match Transformer-based models when preprocessing is tailored to the model and dataset, and even a word-embedding layer trained from scratch can already separate classes for author-profiling tasks. The book's contribution is to present these stages and their interactions as one coherent pipeline, with the evidence drawn from the literature and from the authors' own earlier experiments.","pith_inferences":["The authors do not draw this conclusion, but if the preprocessing results generalize, leaderboards that omit preprocessing details may be comparing models under different implicit pipelines, and standardizing those details could change published rankings.","The embedding-space case study suggests a cheap diagnostic the monograph does not generalize: train a shallow model, project its embedding layer, and stop adding layers once the classes separate.","The sixteen-technique taxonomy invites a natural automated extension: treat the preprocessing sequence as a hyperparameter to be searched per dataset and model, rather than chosen by hand.","Because the cited evidence is mostly English-centric, a direct extension would be to run the same preprocessing grid on multilingual datasets to test whether the advice transfers."],"forward_implications":["Researchers who adopt the monograph's view will report preprocessing choices alongside model and dataset, since a default pipeline can cost more than 25 percent accuracy.","Benchmark comparisons across papers become fairer when preprocessing is documented with the proposed acronym-based nomenclature.","Shallow classifiers remain viable baselines; on smaller datasets with tuned preprocessing they can beat much larger Transformer models.","Embedding-space inspection after training gives a concrete signal for whether deeper layers are needed, so architecture complexity can be chosen from evidence.","Prompt engineering and retrieval-augmented generation enter the pipeline as legitimate classification stages alongside traditional feature engineering."],"supporting_citations":[{"why":"This is the comparative preprocessing study whose technique table and evaluation approach the monograph takes as the foundation for Chapter 3.","marker":"Symeonidis et al., 2018"},{"why":"This is the nine-model, four-dataset study behind the monograph's claim that preprocessing choices can change accuracy by more than 25 percent.","marker":"Siino et al., 2024c"},{"why":"This earlier study is the stated basis for Chapter 3's preprocessing taxonomy and acronym nomenclature.","marker":"Siino et al., 2024d"},{"why":"This earlier work supplies the trained CNN, the FNS dataset, and the word-embedding-space analysis presented as the representation case study.","marker":"Siino et al., 2022a"},{"why":"This prior model-focused survey is one of the references the monograph positions itself against by covering the full pipeline.","marker":"Li et al., 2020"},{"why":"This is the prior survey of preprocessing and feature extraction whose scope the monograph extends with deep learning and evaluation stages.","marker":"Kowsari et al., 2019"},{"why":"This deep-learning-focused survey provides the baseline of model architectures and quantitative results that the monograph complements with the other pipeline stages.","marker":"Minaee et al., 2021"}],"fun_headline_variants":["Preprocessing alone swings text accuracy by 25 percent","Simple classifiers can match Transformers with tuned preprocessing","Preprocessing choices can gain or lose 25% accuracy","The whole text pipeline matters, not just the model","Pipeline view: preprocessing swings text accuracy by 25%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The monograph's novel contributions—the preprocessing taxonomy and the embedding-space analysis—are inherited from its authors' earlier studies and assumed correct without being re-derived here.","fun_headline_variants_meta":{"raw":{"variants":["Preprocessing alone swings text accuracy by 25 percent","Simple classifiers can match Transformers with tuned preprocessing","Preprocessing choices can gain or lose 25% accuracy","The whole text pipeline matters, not just the model","Pipeline view: preprocessing swings text accuracy by 25%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001225,"raw_usage":{"total_tokens":4997,"prompt_tokens":869,"completion_tokens":4128,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":4051}},"tokens_in":485,"tokens_out":4128,"duration_ms":32562,"temperature":1.0,"reasoning_tokens":4051,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:57:41.315082+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the nine-model, four-dataset preprocessing grid behind Chapter 3 with fixed hyperparameters and a pre-registered preprocessing order. If no preprocessing strategy changes accuracy by more than a small margin, or if the reported 25 percent swing does not reproduce, the monograph's central practical claim fails.","supporting_citations":[],"review_version":1}