REVIEW 3 major objections 6 minor 35 references
Word Embedding Techniques for Classification of Star Ratings
T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read A PCA-based way of combining word vectors into a document vector outperforms the standard average for classifying telecom customer review star ratings.
desk verdict A broad, honest applied benchmark that would become convincing with significance tests and a fix for the PCA leakage; the Word2Vec/FastText per-document PCA result is real but not as general as the abstract claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the first principal component of the matrix $A_d = [v_1, \ldots, v_{n_d}] \in \mathbb{R}^{m \times n_d}$ formed from the word vectors of document $d$: instead of averaging the columns equally, the document is represented as $w_1 v_1 + \cdots + w_{n_d} v_{n_d}$, with weights taken from the first principal component, i.e. the direction of maximum variance among that document's word vectors. For BERT and Doc2Vec the same PCA idea is used differently, as global dimensionality reduction on the pooled document vectors (384 to 50 for BERT, 300 to 100 for Doc2Vec). The evaluation machinery is five-fold cross-validated macro-averaged precision, recall and F1, plus measured feature-extraction and training time and CodeCarbon energy and CO2 estimates.
What would settle it
Re-run BERT-PCA and Doc2Vec-PCA with the dimension-reduction fit performed separately inside each of the five cross-validation folds, and check whether BERT-PCA still beats BERT-Average on the multiclass 10,000-review dataset by the reported margin.
Extended reading notes
Core claim
The paper's central claim is that the first principal component of the matrix of term vectors for a document is a better summary of that document than the equally weighted average, at least for the static embeddings Word2Vec and FastText. Across nine dataset configurations (three task types times three sample sizes) and seven classifiers, the average F1-score over classifiers was higher for the PCA version in almost every case, with relative gains of 3.5% to 12% for FastText and large individual jumps such as logistic regression reaching 0.93 instead of 0.60 on the radical-binary 1000-sample task. For BERT, reducing 384-dimensional pooled averages to 50 principal components gave roughly equal or better average F1-scores in six of nine datasets, and BERT-PCA produced the strongest results on the five-class multiclass datasets; for Doc2Vec, PCA reduction to 100 dimensions was not generally beneficial. The authors also find that TF-IDF performs worst for most classifiers but best for support vector machines, and that energy consumption varies widely, with TF-IDF and Doc2Vec cheapest and FastText and BERT costliest.
Load-bearing premise
The headline comparison assumes that the dimension-reduction step was computed using only the training reviews, not all reviews including the ones held out for testing; if it used all reviews, the reported gains for the reduced versions could be exaggerated.
Editorial extensions
If this is right
- For static embeddings like Word2Vec and FastText, replacing the average of word vectors with a PCA-weighted combination is a cheap way to improve short-text classifiers; the paper reports higher mean F1 over seven classifiers for nearly every dataset, with FastText gains of 3.5% to 12%.
- On the five-class task, BERT-PCA is the strongest representation for several classifiers, for example an F1 of 0.90 for Random Forest at all sample sizes, so dimensionality-reduced BERT features can beat full BERT averages on harder problems.
- The best embedding depends on the classifier: TF-IDF is the worst representation for most classifiers but the best for the Support Vector Classifier, so feature-choice recommendations should be paired with a specific classifier.
- Energy use is not proportional to accuracy: TF-IDF and Doc2Vec are far cheaper than FastText and BERT, giving a concrete trade-off between cost and classification quality for real deployments.
Reading between the lines
- The per-document first-principal-component weighting is a generic way to summarise a bag of vectors, so the same trick could be tested on sentence embeddings, retrieval documents, or image patch features; the paper only evaluates it on English telecom reviews.
- Because the BERT-PCA and Doc2Vec-PCA projections are fit on all documents before the five-fold split, the reported multiclass gains may include leakage; re-running with PCA fit inside each training fold is a direct way to see how much of the advantage is real.
- The energy measurements suggest a practical deployment rule the paper does not state: use Doc2Vec or TF-IDF when the compute budget is small, and reserve BERT-PCA for hard multiclass tasks where its F1 lead is worth the extra cost.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents an empirical comparison of word embedding techniques—Word2Vec, FastText, BERT, Doc2Vec, and TF-IDF—combined with seven classifiers for predicting star ratings from telecom customer reviews. The authors propose combining per-word vectors via the first principal component rather than averaging, and also apply PCA as dimensionality reduction for BERT and Doc2Vec. They evaluate on a newly scraped Trustpilot dataset with binary and multiclass tasks, sample sizes 1K, 5K, and 10K, and additionally measure feature-extraction time, classifier training time, and energy consumption. The headline claims are that the PCA-based combination is clearly superior to averaging for Word2Vec and FastText, and that BERT combined with PCA achieves the highest performance on the more challenging multiclass tasks.
Significance. If the PCA-combination claim held, it would provide a cheap, model-agnostic feature-engineering improvement for short-text classification with static embeddings, and the energy-consumption comparison is practically relevant. The paper's strengths include its large novel dataset, systematic coverage of 63 dataset-classifier-embedding combinations, and the inclusion of runtime and CodeCarbon-based energy measurements. However, the lack of error bars or significance tests, the potential PCA leakage for BERT and Doc2Vec, and the internal inconsistency between the aggregate means and medians weaken the reliability of the two headline claims. The claimed significance is therefore not yet established, although the underlying comparison is a useful contribution to the empirical literature on embedding-based text classification.
major comments (3)
- [§3.3.2, Figs. 10–12, Table 1] The claim that PCA-based combination "shows clear advantages" over averaging for Word2Vec and FastText is not supported by the reported evidence. In Figure 12 the median F1-score is lower with PCA for Word2Vec (0.89 vs 0.92) and essentially unchanged for FastText (0.91 vs 0.92); the higher means arise from a shorter lower tail, which Section 3.3.1 shows is dominated by Logistic Regression and SGD. Section 3.3.1 also reports multiple cases where averaging is better, including SVC with Word2Vec (F1 0.92 average vs 0.68 PCA) and Random Forest with FastText (0.95 vs 0.91) in Table 1. Since the evaluation in Section 2.3.4 reports only macro-F1 point values with no fold-level variance, confidence intervals, or paired significance tests, the aggregate "clear advantage" is not statistically established.
- [§2.2.2, §2.2.3, §2.3.4] For BERT-PCA and Doc2Vec-PCA, the PCA projection is computed on the full document-vector matrix before the data are split into the five cross-validation folds. This introduces a leakage path: the projection matrix is estimated with information from the test folds, so the reported gains of BERT-PCA over BERT-Average (Section 3.3.1) and the Doc2Vec-PCA results may be artificially inflated. PCA should be estimated within each training fold (e.g., via a scikit-learn Pipeline) so that the test folds remain unseen, and the experiments should be repeated under this corrected protocol.
- [Abstract; §3.3.1, §3.3.3] The abstract's statement that "BERT combined with PCA stood out with the highest performance metrics" for the more challenging tasks is not supported by the paper's own aggregate results. Section 3.3.3 reports that Gradient Boosting achieves F1 = 0.92 on all Multi-Class datasets for every word embedding, whereas BERT-PCA's best multiclass result is 0.90 with Random Forest, and Section 3.3.1 shows BERT-PCA often gives lower scores than BERT-Average for SVC. The superlative appears to be based on selected classifier-embedding pairs rather than a consistent overall comparison, and no multiple-testing correction is applied.
minor comments (6)
- [§2.1] There is a typo: "in additon" should be "in addition."
- [§2.2.3] There is a typo: "invetigate" should be "investigate."
- [§2.3.1] The phrase "to to the dataset" contains a duplicated word and should be corrected.
- [Appendix A] The Logistic Regression entry lists both solver='sag' and solver='lbfgs'; please specify the effective solver used in the experiments.
- [References] The BERT reference appears to be misattributed to Chang et al. (2019); the BERT architecture is due to Devlin et al. (2019), and the citation should be corrected.
- [Data availability] There is no data or code availability statement. Since the datasets are described as original and scraped from public forums, making them available (with the scraping and cleaning scripts) would substantially improve reproducibility.
Circularity Check
No significant circularity: the paper reports empirical benchmark measurements, and the PCA-vs-average comparison is not equivalent to its inputs by construction.
full rationale
The paper's central claim, that first-principal-component weighting of Word2Vec/FastText vectors outperforms averaging, is an empirical observation from F1-score benchmarks, not a derivation. In Section 2.2.1 the PCA-based feature vector is defined as a weighted sum w1v1+...+wndvnd with weights from the first principal component of the per-document term-vector matrix, whereas the average vector is the equal-weight sum (v1+...+vnd)/nd; these are distinct constructions, so the claimed advantage does not reduce to the definition of either method. No parameter is fitted to the star-rating labels and then renamed as a prediction, so the fitted-input-called-prediction pattern does not apply. The paper contains no self-citations and invokes no uniqueness theorem, so self-citation and imported-uniqueness patterns are absent. The strongest remaining concern is methodological rather than circular: for BERT and Doc2Vec, PCA dimensionality reduction is described in Sections 2.2.2 and 2.2.3 as being applied before the five-fold cross-validation described in Section 2.3.4, so the PCA transform may incorporate information from all documents including test folds. That is a potential data-leakage/validity problem and could inflate the reported BERT-PCA advantage, but it is not a case of a result being equivalent to its inputs by construction. Likewise, asserting an advantage after inspecting the same 63-case grid that produced it is post-hoc selection, not circular reasoning. The paper is a self-contained empirical study; therefore the appropriate circularity score is 1, reflecting the minor validity concern rather than any circular derivation.
Assumptions & free parameters
free parameters (2)
- BERT-PCA n_components =
50
- Doc2Vec-PCA n_components =
100
assumptions (4)
- domain assumption The scraped Trustpilot star ratings are accurate ground-truth labels for the review text.
- domain assumption Fitting PCA on the full dataset before cross-validation does not inflate performance.
- domain assumption The word embedding models used (unnamed BERT variant, Word2Vec 100-dim, FastText 300-dim) are representative of each method.
- domain assumption CodeCarbon measurements correctly attribute energy and CO2 for Google Colab sessions.
Cite this review
Pith. "Pith review of Word Embedding Techniques for Classification of Star Ratings." pith.science (2026). https://pith.science/paper/GESLPVQD
@misc{pith2026250413653,
author = {Pith},
title = {Pith review of: Word Embedding Techniques for Classification of Star Ratings},
year = {2026},
howpublished = {\url{https://pith.science/paper/GESLPVQD}},
note = {Machine review of arXiv:2504.13653}
}
read the original abstract
Telecom services are at the core of today's societies' everyday needs. The availability of numerous online forums and discussion platforms enables telecom providers to improve their services by exploring the views of their customers to learn about common issues that the customers face. Natural Language Processing (NLP) tools can be used to process the free text collected. One way of working with such data is to represent text as numerical vectors using one of many word embedding models based on neural networks. This research uses a novel dataset of telecom customers' reviews to perform an extensive study showing how different word embedding algorithms can affect the text classification process. Several state-of-the-art word embedding techniques are considered, including BERT, Word2Vec and Doc2Vec, coupled with several classification algorithms. The important issue of feature engineering and dimensionality reduction is addressed and several PCA-based approaches are explored. Moreover, the energy consumption used by the different word embeddings is investigated. The findings show that some word embedding models can lead to consistently better text classifiers in terms of precision, recall and F1-Score. In particular, for the more challenging classification tasks, BERT combined with PCA stood out with the highest performance metrics. Moreover, our proposed PCA approach of combining word vectors using the first principal component shows clear advantages in performance over the traditional approach of taking the average.
Figures
Figures from the paper (14 more)
Reference graph
Works this paper leans on
-
[1]
T. W. Anderson. An Introduction to Multivariate Statistical Analysis. W iley, 3 edition, 2003
work page 2003
-
[2]
N. Bannour, S. Ghannay, A. N \'e v \'e ol, and A.-L. Ligozat. Evaluating the carbon footprint of nlp methods: a survey and analysis of existing tools. In Proceedings of the second workshop on simple and efficient natural language processing, pages 11--21, 2021
work page 2021
- [3]
-
[4]
T. Carneiro, R. V. M. Da N \'o brega, T. Nepomuceno, G.-B. Bian, V. H. C. De Albuquerque, and P. P. Reboucas Filho. Performance analysis of google colaboratory as a tool for accelerating deep learning applications. IEEE Access, 6: 0 61677--61685, 2018
work page 2018
-
[5]
Language Model Pre-training for Hierarchical Document Representations
M.-W. Chang, K. Toutanova, K. Lee, and J. Devlin. Language model pre-training for hierarchical document representations. ArXiv preprint: arXiv.1901.09128, 2019. doi:10.48550/arXiv.1901.09128. URL https://doi.org/10.48550/arXiv.1901.09128
work page Pith review arXiv doi:10.48550/arxiv.1901.09128 1901
- [6]
-
[7]
B. Courty, V. Schmidt, Goyal-Kamal, MarionCoutarel, B. Feld, J. Lecourt, LiamConnell, SabAmine, Inimaz, Supatomic, M. L\'eval, L. Blanche, A. Cruveiller, Ouminasara, F. Zhao, A. Joshi, A. Bogroff, A. Saboni, H. De Lavoreille, N. Laskaris, E. Abati, D. Blank, Z. Wang, A. Catovic, Alencon, M. St e ch y, C. Bauer, Lucas-Otavio, Jpw, and MinervaBooks. mlco2/c...
-
[8]
D. A. Eisa, A. I. Taloba, and S. S. Ismail. A comparative study on using P rinciple C omponent A nalysis with different text classifiers. International Journal of Computer Applications, 180 0 (31): 0 1--6, Apr 2018. ISSN 0975-8887. doi:10.5120/ijca2018916800. URL https://ijcaonline.org/archives/volume180/number31/29239-2018916800/
Show all 35 references
-
[9]
Freitag, M
C. Freitag, M. Berners-Lee, K. Widdicks, B. Knowles, G. S. Blair, and A. Friday. The real climate and transformative impact of ICT : A critique of estimates, trends, and regulations. Patterns, 2 0 (9), 2021
2021
-
[10]
Garc\' a-Mart\' n, C
E. Garc\' a-Mart\' n, C. F. Rodrigues, G. Riley, and H. Grahn. Estimation of energy consumption in machine learning. Journal of Parallel and Distributed Computing, 134: 0 75--88, 2019. ISSN 0743-7315. doi:https://doi.org/10.1016/j.jpdc.2019.07.007. URL https://www.sciencedirec...
2019 doi
-
[11]
Gupta, S
V. Gupta, S. Giesselbach, S. R \"u ping, and C. Bauckhage. Improving word embeddings using kernel PCA . In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 200--208, 2019
2019
-
[12]
Kottur, R
S. Kottur, R. Vedantam, J. M. F. Moura, and D. Parikh. Visual word2vec (vis-w2v): Learning visually grounded word embeddings using abstract scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016
2016
-
[13]
Lakmal, S
D. Lakmal, S. Ranathunga, S. Peramuna, and I. Herath. Word embedding evaluation for S inhala. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1874--1881, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-3...
2020
-
[14]
M. A. Laurenzano, A. Tiwari, A. Jundt, J. Peraza, W. A. Ward, R. Campbell, and L. Carrington. Characterizing the performance-energy tradeoff of small arm cores in hpc computation. In F. Silva, I. Dutra, and V. Santos Costa, editors, Euro-Par 2014 Parallel Processing, pages 124...
2014
-
[15]
Lebret and R
R. Lebret and R. Collobert. Word embeddings through hellinger PCA . In S. Wintner, S. Goldwater, and S. Riezler, editors, Proceedings of the 14th Conference of the E uropean Chapter of the Association for Computational Linguistics , pages 482--490, Gothenburg, Sweden, Apr. 201...
2014 doi
-
[16]
B. C. Lee and D. M. Brooks. Accurate and efficient regression modeling for microarchitectural performance and power prediction. In Proceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS XII, pages 185 -...
2006
-
[17]
Lottick, S
K. Lottick, S. Susai, S. A. Friedler, and J. P. Wilson. Energy usage reports: Environmental awareness as part of algorithmic accountability. Workshop on Tackling Climate Change with Machine Learning at NeurIPS 2019, 2019
2019
-
[18]
Mahajan, M
Y. Mahajan, M. Freestone, S. Aakur, and S. Karmaker. Revisiting word embeddings in the LLM era. arXiv preprint arXiv:2502.19607, 2025
2025 arXiv
-
[19]
Mazouz, D
A. Mazouz, D. C. Wong, D. Kuck, and W. Jalby. An incremental methodology for energy measurement and modeling. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering, ICPE '17, pages 15 -- 26, New York, NY, USA, 2017. Association for Computing...
2017
-
[20]
Mehlin, S
V. Mehlin, S. Schacht, and C. Lanquillon. Towards energy-efficient deep learning: An overview of energy-efficient approaches along the deep learning lifecycle. arXiv preprint arXiv:2303.01980, 2023
2023 arXiv
-
[21]
Mikolov, K
T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013
2013 arXiv
-
[22]
Monett and H
D. Monett and H. Stolte. Predicting star ratings based on annotated reviews of mobile apps. In 2016 Federated conference on computer science and information systems (FedCSIS), pages 421--428. IEEE, 2016
2016
-
[23]
Oscar Deho, A
B. Oscar Deho, A. William Agangiba, L. Felix Aryeh, and A. Jeffery Ansah. Sentiment analysis with word embedding. In 2018 IEEE 7th International Conference on Adaptive Science & Technology (ICAST), pages 1--4, 2018. doi:10.1109/ICASTECH.2018.8506717
2018
-
[24]
Pennington, R
J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543, 2014
2014
-
[25]
D. M. Powers. Evaluation: from precision, recall and F -measure to ROC , informedness, markedness and correlation. Journal of Machine Learning Technologies, pages 37--63, 2011
2011
-
[26]
Sabbeh and H
S. Sabbeh and H. Fasihuddin. A comparative analysis of word embedding and deep learning for arabic sentiment classification. electronics 2023, 12, 1425, 2023
2023
-
[27]
Sammut and G
C. Sammut and G. I. Webb, editors. TF--IDF, pages 986--987. Springer US, Boston, MA, 2010. ISBN 978-0-387-30164-8. doi:10.1007/978-0-387-30164-8_832. URL https://doi.org/10.1007/978-0-387-30164-8_832
2010 doi
-
[28]
Saumya, J
S. Saumya, J. P. Singh, A. M. Baabdullah, N. P. Rana, and Y. K. Dwivedi. Ranking online consumer reviews. Electronic Commerce Research and Applications, 29: 0 78--89, 2018
2018
-
[29]
Y. Shao, S. Taylor, N. Marshall, C. Morioka, and Q. Zeng-Treitler. Clinical text classification with word embedding features vs. bag-of-words features. In 2018 IEEE International Conference on Big Data (Big Data), pages 2874--2878, 2018. doi:10.1109/BigData.2018.8622345
2018
-
[30]
Terenius, P
P. Terenius, P. Garraghan, and R. Harper. A material social view on data center waste heat: Novel uses and metrics. Frontiers in Sustainability, 3: 0 1008583, 2023
2023
-
[31]
Upadhyay, J
D. Upadhyay, J. Manero, M. Zaman, and S. Sampalli. Gradient boosting feature selection with machine learning classifiers for intrusion detection on power grids. IEEE Transactions on Network and Service Management, 18 0 (1): 0 1104--1116, 2020
2020
-
[32]
C. Wang, P. Nulty, and D. Lillis. A comparative study on word embeddings in deep learning for text classification. In Proceedings of the 4th international conference on natural language processing and information retrieval, pages 37--46, 2020
2020
-
[33]
Q. Wang, P. Liu, Z. Zhu, H. Yin, Q. Zhang, and L. Zhang. A text abstraction summary model based on bert word embedding and reinforcement learning. Applied Sciences, 9 0 (21), 2019. ISSN 2076-3417. doi:10.3390/app9214701. URL https://www.mdpi.com/2076-3417/9/21/4701
2019 doi
-
[34]
Y. Wang, S. Liu, N. Afzal, M. Rastegar-Mojarad, L. Wang, F. Shen, P. Kingsbury, and H. Liu. A comparison of word embeddings for the biomedical natural language processing. Journal of Biomedical Informatics, 87: 0 12--20, 2018
2018
-
[35]
T. Yao, Z. Zhai, and B. Gao. Text classification model based on fasttext. In 2020 IEEE International Conference on Artificial Intelligence and Information Systems (ICAIIS), pages 154--157. IEEE, 2020
2020
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.