{"id":"d1489776-4e97-462a-87a7-6167c6b3a486","arxiv_id":"2504.13653","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On telecom review data, combining Word2Vec and FastText word vectors with the first PCA component improves classifier F1 over simple averaging, while BERT-PCA helps most in hard multiclass tasks.","lead":"This paper compares word embedding methods for classifying telecom customer reviews into star ratings, using a new scraped Trustpilot dataset. It reports that a PCA-based way of combining word vectors often beats simple averaging for Word2Vec and FastText, and that energy costs vary widely across methods.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'clear advantage' of per-document PCA over averaging is asserted from aggregate point estimates without significance testing or per-classifier consistency; the advantage may be driven by a few linear classifiers.","rationale":"The reader's leakage concern is real for BERT-PCA and Doc2Vec-PCA and should be fixed by fitting PCA inside each training fold. However, the proposed first-PC aggregation for Word2Vec and FastText is per-document and does not involve a global PCA fit, so leakage does not bear on the paper's primary claim. The primary claim requires the comparison to be statistically and practically robust, and that is where the argument is least secure: no error bars or significance tests are reported, and the paper's own results show many ties and some reversals, with the aggregate mean pulled upward by linear classifiers. The recommended condition is to report fold-level variance, paired significance tests, and classifier-stratified win/loss tables, plus a normalization control. If the advantage is confined to LR/SGD or disappears after normalization, the abstract and Section 3.3.2 should be revised. The reader's verdict of CONDITIONAL already captures the need for revision, so the verdict does not need to change, but the stated weakest assumption differs from the one identified here.","tokens_in":21827,"tokens_out":12682,"duration_ms":131362,"concrete_test":"Re-run the W2V-Average vs W2V-PCA and FT-Average vs FT-PCA comparisons with identical 5-fold splits and: (a) record per-fold macro-F1 for every classifier and dataset; (b) repeat with at least 10 random seeds; (c) apply a paired test on the per-fold differences (e.g., Wilcoxon signed-rank or paired bootstrap); and (d) tabulate wins/losses/ties separately for each classifier across the nine datasets. As a control, L2-normalize the average document vectors before classification. If the PCA advantage is not significant, appears only for LR/SGD, or disappears when average vectors are normalized, the central claim should be weakened or withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, stated in Section 3.3.2 and Figures 10-12, is that for Word2Vec and FastText the first-principal-component combination 'shows clear advantages' over averaging. For this to hold, the observed F1 differences must be real, stable, and general. The paper reports only macro-F1 point values averaged over seven classifiers and nine datasets: Section 2.3.4 says five-fold CV was used, but no fold-level standard deviations, confidence intervals, or significance tests are given. The paper's own evidence is internally mixed: in Figure 12 the median F1 is slightly lower with PCA for Word2Vec (0.89 vs 0.92) and essentially equal for FastText (0.91 vs 0.92); the higher means come from shortening the lower tail, which is dominated by Logistic Regression and SGD. For several classifiers the paper reports identical scores for average and PCA (GBC, DT, KNN), and for SVC and Random Forest on multiclass data the average is often better (e.g., Table 1 shows SVC with W2V-Average 0.92 vs W2V-PCA 0.68, and RF with FT-Average 0.95 vs FT-PCA 0.91). Thus the aggregate 'clear advantages' is not established; it may reflect classifier-specific scaling or regularization effects rather than a general property of the PCA combination.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an empirical comparison of word embedding techniques—Word2Vec, FastText, BERT, Doc2Vec, and TF-IDF—combined with seven classifiers for predicting star ratings from telecom customer reviews. The authors propose combining per-word vectors via the first principal component rather than averaging, and also apply PCA as dimensionality reduction for BERT and Doc2Vec. They evaluate on a newly scraped Trustpilot dataset with binary and multiclass tasks, sample sizes 1K, 5K, and 10K, and additionally measure feature-extraction time, classifier training time, and energy consumption. The headline claims are that the PCA-based combination is clearly superior to averaging for Word2Vec and FastText, and that BERT combined with PCA achieves the highest performance on the more challenging multiclass tasks.","tokens_in":22082,"tokens_out":5515,"duration_ms":45659,"significance":"If the PCA-combination claim held, it would provide a cheap, model-agnostic feature-engineering improvement for short-text classification with static embeddings, and the energy-consumption comparison is practically relevant. The paper's strengths include its large novel dataset, systematic coverage of 63 dataset-classifier-embedding combinations, and the inclusion of runtime and CodeCarbon-based energy measurements. However, the lack of error bars or significance tests, the potential PCA leakage for BERT and Doc2Vec, and the internal inconsistency between the aggregate means and medians weaken the reliability of the two headline claims. The claimed significance is therefore not yet established, although the underlying comparison is a useful contribution to the empirical literature on embedding-based text classification.","major_comments":[{"comment":"The claim that PCA-based combination \"shows clear advantages\" over averaging for Word2Vec and FastText is not supported by the reported evidence. In Figure 12 the median F1-score is lower with PCA for Word2Vec (0.89 vs 0.92) and essentially unchanged for FastText (0.91 vs 0.92); the higher means arise from a shorter lower tail, which Section 3.3.1 shows is dominated by Logistic Regression and SGD. Section 3.3.1 also reports multiple cases where averaging is better, including SVC with Word2Vec (F1 0.92 average vs 0.68 PCA) and Random Forest with FastText (0.95 vs 0.91) in Table 1. Since the evaluation in Section 2.3.4 reports only macro-F1 point values with no fold-level variance, confidence intervals, or paired significance tests, the aggregate \"clear advantage\" is not statistically established.","section":"§3.3.2, Figs. 10–12, Table 1"},{"comment":"For BERT-PCA and Doc2Vec-PCA, the PCA projection is computed on the full document-vector matrix before the data are split into the five cross-validation folds. This introduces a leakage path: the projection matrix is estimated with information from the test folds, so the reported gains of BERT-PCA over BERT-Average (Section 3.3.1) and the Doc2Vec-PCA results may be artificially inflated. PCA should be estimated within each training fold (e.g., via a scikit-learn Pipeline) so that the test folds remain unseen, and the experiments should be repeated under this corrected protocol.","section":"§2.2.2, §2.2.3, §2.3.4"},{"comment":"The abstract's statement that \"BERT combined with PCA stood out with the highest performance metrics\" for the more challenging tasks is not supported by the paper's own aggregate results. Section 3.3.3 reports that Gradient Boosting achieves F1 = 0.92 on all Multi-Class datasets for every word embedding, whereas BERT-PCA's best multiclass result is 0.90 with Random Forest, and Section 3.3.1 shows BERT-PCA often gives lower scores than BERT-Average for SVC. The superlative appears to be based on selected classifier-embedding pairs rather than a consistent overall comparison, and no multiple-testing correction is applied.","section":"Abstract; §3.3.1, §3.3.3"}],"minor_comments":[{"comment":"There is a typo: \"in additon\" should be \"in addition.\"","section":"§2.1"},{"comment":"There is a typo: \"invetigate\" should be \"investigate.\"","section":"§2.2.3"},{"comment":"The phrase \"to to the dataset\" contains a duplicated word and should be corrected.","section":"§2.3.1"},{"comment":"The Logistic Regression entry lists both solver='sag' and solver='lbfgs'; please specify the effective solver used in the experiments.","section":"Appendix A"},{"comment":"The BERT reference appears to be misattributed to Chang et al. (2019); the BERT architecture is due to Devlin et al. (2019), and the citation should be corrected.","section":"References"},{"comment":"There is no data or code availability statement. Since the datasets are described as original and scraped from public forums, making them available (with the scraping and cleaning scripts) would substantially improve reproducibility.","section":"Data availability"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical benchmark with a useful new dataset, but the central claim rests on aggregate point estimates and a potential leakage issue. In revision, the authors should be asked to (i) provide error bars or significance tests, (ii) fix the PCA-in-CV procedure so that PCA is fitted inside training folds, and (iii) temper or re-justify the abstract's superlative claims. These issues are addressable without new external data, so this is a major-revision rather than a rejection. Please also encourage the authors to release the dataset and code, since the current manuscript provides no reproducibility artifacts beyond the paper text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a useful applied benchmark, not a breakthrough. The genuinely new bit is the per-document PCA weighting for Word2Vec and FastText: take the first principal component of the document's word vectors and use that as a weighted average. It is a variation of known PCA sentence-embedding tricks, but someone had to test it on a real short-text task, and the paper does that carefully. The scraped telecom dataset is a legitimate addition, and the energy/CO2 measurements are a nice practical touch.\n\nWhat I like: the comparison is broad (9 representations, 7 classifiers, 9 datasets), and the reporting is transparent enough that you can see the anomalies. The authors note where PCA does not help (Doc2Vec) and where the median drops even when the mean rises. That suggests honest work.\n\nThe soft spots are in the claims, not the experiment. First, the abstract's 'clear advantages' for PCA over averaging is too strong. Looking at Figure 12, the mean F1 rises because the lower tail shortens, but the median is actually slightly lower for Word2Vec-PCA than for the average, and for FastText essentially tied. The gain is concentrated in Logistic Regression and SGD; for GBC, DT, and KNN the scores are identical, and SVC and Random Forest on multiclass often do better with the average. So the per-document PCA is a useful trick for some linear classifiers, not a general improvement.\n\nSecond, there is a real leakage problem in the BERT-PCA and Doc2Vec-PCA pipelines. The PCA projection is fitted on all documents before the 5-fold split, so test-fold information enters the training features. That can inflate BERT-PCA's reported advantage on multiclass tasks. The fix is easy: fit PCA inside each training fold.\n\nThird, no error bars or significance tests anywhere. For a paper whose headline is a difference of means, that is a necessary addition, not a luxury.\n\nThe citation pattern is okay, though the paper misses the first-PC sentence embedding literature, which is a minor omission.\n\nBottom line: worth a serious referee, but only with major revision. I'd ask for fold-level variances or a paired test, a leak-free pipeline, and a toned-down abstract. If the authors release data and code, this becomes a citable resource for applied NLP.\n\nRecommendation: send to peer review, conditional on the above.","headline":"A broad, honest applied benchmark that would become convincing with significance tests and a fix for the PCA leakage; the Word2Vec/FastText per-document PCA result is real but not as general as the abstract claims.","tokens_in":22647,"tokens_out":3019,"would_cite":false,"duration_ms":27160,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A PCA-based way of combining word vectors into a document vector outperforms the standard average for classifying telecom customer review star ratings.","keywords":["text classification","word embeddings","feature extraction","short texts","telecom data","principal component analysis","star ratings","BERT"],"falsifier":"Re-run BERT-PCA and Doc2Vec-PCA with the dimension-reduction fit performed separately inside each of the five cross-validation folds, and check whether BERT-PCA still beats BERT-Average on the multiclass 10,000-review dataset by the reported margin.","tokens_in":21599,"feed_emoji":"⭐","tokens_out":7450,"duration_ms":61396,"temperature":0.7,"pith_summary":"This paper tries to establish that how you combine word vectors into a document vector matters as much as which embedding you choose. On a new dataset of UK telecom customer reviews scored one to five stars, the authors compare Word2Vec, FastText, BERT and Doc2Vec across seven classifiers, and for Word2Vec and FastText they replace the usual arithmetic average of a document's word vectors with a weighted combination taken from the first principal component of those vectors. They report that this PCA-based representation consistently improves precision, recall and F1-score over averaging, and that BERT with PCA dimensionality reduction gives the highest scores on the harder multiclass tasks. The practical interest is that the proposed PCA combination is a cheap feature-engineering change that can improve short-text classifiers without retraining embeddings, and the study also quantifies the energy cost of each embedding choice.","feed_headline":"PCA weighting beats plain averaging for review star ratings","feed_subtitle":"A cheap PCA tweak lifts Word2Vec and FastText classifiers on short telecom reviews","key_machinery":"The central object is the first principal component of the matrix $A_d = [v_1, \\ldots, v_{n_d}] \\in \\mathbb{R}^{m \\times n_d}$ formed from the word vectors of document $d$: instead of averaging the columns equally, the document is represented as $w_1 v_1 + \\cdots + w_{n_d} v_{n_d}$, with weights taken from the first principal component, i.e. the direction of maximum variance among that document's word vectors. For BERT and Doc2Vec the same PCA idea is used differently, as global dimensionality reduction on the pooled document vectors (384 to 50 for BERT, 300 to 100 for Doc2Vec). The evaluation machinery is five-fold cross-validated macro-averaged precision, recall and F1, plus measured feature-extraction and training time and CodeCarbon energy and CO2 estimates.","core_discovery":"The paper's central claim is that the first principal component of the matrix of term vectors for a document is a better summary of that document than the equally weighted average, at least for the static embeddings Word2Vec and FastText. Across nine dataset configurations (three task types times three sample sizes) and seven classifiers, the average F1-score over classifiers was higher for the PCA version in almost every case, with relative gains of 3.5% to 12% for FastText and large individual jumps such as logistic regression reaching 0.93 instead of 0.60 on the radical-binary 1000-sample task. For BERT, reducing 384-dimensional pooled averages to 50 principal components gave roughly equal or better average F1-scores in six of nine datasets, and BERT-PCA produced the strongest results on the five-class multiclass datasets; for Doc2Vec, PCA reduction to 100 dimensions was not generally beneficial. The authors also find that TF-IDF performs worst for most classifiers but best for support vector machines, and that energy consumption varies widely, with TF-IDF and Doc2Vec cheapest and FastText and BERT costliest.","pith_inferences":["The per-document first-principal-component weighting is a generic way to summarise a bag of vectors, so the same trick could be tested on sentence embeddings, retrieval documents, or image patch features; the paper only evaluates it on English telecom reviews.","Because the BERT-PCA and Doc2Vec-PCA projections are fit on all documents before the five-fold split, the reported multiclass gains may include leakage; re-running with PCA fit inside each training fold is a direct way to see how much of the advantage is real.","The energy measurements suggest a practical deployment rule the paper does not state: use Doc2Vec or TF-IDF when the compute budget is small, and reserve BERT-PCA for hard multiclass tasks where its F1 lead is worth the extra cost."],"forward_implications":["For static embeddings like Word2Vec and FastText, replacing the average of word vectors with a PCA-weighted combination is a cheap way to improve short-text classifiers; the paper reports higher mean F1 over seven classifiers for nearly every dataset, with FastText gains of 3.5% to 12%.","On the five-class task, BERT-PCA is the strongest representation for several classifiers, for example an F1 of 0.90 for Random Forest at all sample sizes, so dimensionality-reduced BERT features can beat full BERT averages on harder problems.","The best embedding depends on the classifier: TF-IDF is the worst representation for most classifiers but the best for the Support Vector Classifier, so feature-choice recommendations should be paired with a specific classifier.","Energy use is not proportional to accuracy: TF-IDF and Doc2Vec are far cheaper than FastText and BERT, giving a concrete trade-off between cost and classification quality for real deployments."],"supporting_citations":[{"why":"Supplies the Word2Vec model whose average and PCA representations are compared.","marker":"(Mikolov et al., 2013)"},{"why":"Supplies the FastText bag-of-character-n-grams embedding used for the FastText comparisons.","marker":"(Lakmal et al., 2020)"},{"why":"Supplies the BERT embedding used to produce BERT-Average and BERT-PCA features.","marker":"(Chang et al., 2019)"},{"why":"Supplies the Doc2Vec document-vector method whose full and PCA-reduced vectors are compared.","marker":"(Shao et al., 2018)"},{"why":"Provides the PCA method used for both per-document weighted combinations and global dimensionality reduction.","marker":"(Anderson, 2003)"},{"why":"Prior result motivating PCA-based word-embedding representations that the study extends.","marker":"(Lebret and Collobert, 2014)"},{"why":"Prior evidence that PCA on text features improves classifiers, the comparison point for the proposed method.","marker":"(Eisa et al., 2018)"},{"why":"Defines the precision, recall and F1 measures used to compare all methods.","marker":"(Powers, 2011)"}],"fun_headline_variants":["First principal component beats averaging for star ratings","PCA vector summary outperforms averaged embeddings in review tasks","PCA-based doc vectors beat mean vectors for telecom review ratings","Using first PCA component improves review star rating classifiers","PCA weighting outperforms average vector for embedding classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison assumes that the dimension-reduction step was computed using only the training reviews, not all reviews including the ones held out for testing; if it used all reviews, the reported gains for the reduced versions could be exaggerated.","fun_headline_variants_meta":{"raw":{"variants":["First principal component beats averaging for star ratings","PCA vector summary outperforms averaged embeddings in review tasks","PCA-based doc vectors beat mean vectors for telecom review ratings","Using first PCA component improves review star rating classifiers","PCA weighting outperforms average vector for embedding classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001063,"raw_usage":{"total_tokens":4483,"prompt_tokens":999,"completion_tokens":3484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":3411}},"tokens_in":615,"tokens_out":3484,"duration_ms":21720,"temperature":1.0,"reasoning_tokens":3411,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:02:35.979344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run BERT-PCA and Doc2Vec-PCA with the dimension-reduction fit performed separately inside each of the five cross-validation folds, and check whether BERT-PCA still beats BERT-Average on the multiclass 10,000-review dataset by the reported margin.","supporting_citations":[],"review_version":1}