REVIEW 4 major objections 5 minor 30 references
EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that ChatGPT's responses to tweets about generative AI are 90% positive while human comments are 72% negative, reflecting LLMs' built-in tendency to stay constructive and positive.
desk verdict The 90%-positive ChatGPT claim is not derivable from the described K-means pipeline because the paper never explains how clusters were labeled positive or negative. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is an unsupervised sentiment-analysis pipeline: BERT word embeddings of 768 dimensions are computed for every token and averaged to sentence embeddings, then K-means clustering with k=2 is applied separately to human words, ChatGPT words, human sentences, and ChatGPT sentences. Cluster quality is measured with the Silhouette score, and t-SNE projects the high-dimensional clusters to two dimensions for visualization. The cleaning steps differ between the two sides: human comments get URLs, emojis, and @mentions removed, while ChatGPT responses additionally have frequent neutral words such as 'AI', 'language', and 'model' stripped out before the same clustering is run.
What would settle it
Recompute Table V with a documented cluster-labeling rule, for example assigning the cluster whose centroid has higher average overlap with positive sentiment-lexicon words as 'positive'; if the resulting percentages are close to 90% and 27% the claim stands, and if they differ sharply the polarity assignment was arbitrary. A second concrete check is to rerun the full pipeline with identical stopword and neutral-word removal for both human comments and ChatGPT responses, since the paper removes frequent neutral words only from ChatGPT responses.
Extended reading notes
Core claim
On its own terms, EmoXpt claims to show that human comments about generative AI on Twitter are predominantly negative (72%), while ChatGPT's responses to those same tweets are overwhelmingly positive (90%). The sentence-level K-means clusters for ChatGPT reach a Silhouette score of 0.58 versus 0.19 for human comments, which the paper reads as ChatGPT's responses being more focused, cohesive, and uniformly toned. The paper interprets the split as evidence that LLMs exhibit a positivity bias, responding constructively even to negative or fearful prompts, and concludes that AI-generated text is efficient and positive but does not capture the full emotional nuance of human interaction.
Load-bearing premise
The central load-bearing premise is that the two K-means clusters correspond to positive and negative sentiment, and that the reported 90% and 27% figures follow from that correspondence; the paper gives no rule for which cluster is which.
Editorial extensions
If this is right
- If the claim is right, ChatGPT's public-facing text is systematically skewed toward positive sentiment compared with human expression, so treating LLM output as representative human opinion would overestimate support for generative AI.
- The 90% versus 27% split suggests that alignment or safety training suppresses negative framing in ChatGPT even when the input tweet is critical or worried.
- The large gap in sentence-level Silhouette scores (0.58 for ChatGPT versus 0.19 for humans) implies that ChatGPT's emotional tone is more uniform and less varied than human tone.
- The framework gives a label-free way to compare emotional variance across language models and social-media populations, without needing manually annotated sentiment data.
Reading between the lines
- Going beyond the paper, the headline percentages depend on an unstated rule for assigning 'positive' and 'negative' to the two K-means clusters; labeling the same clusters using a sentiment lexicon could confirm or reverse the 90% versus 27% split.
- Because frequent neutral words were removed from ChatGPT responses but not from human comments, the same pipeline with symmetric cleaning on both sides is a direct test of whether the positivity gap is a property of the model or an artifact of preprocessing.
- If the claim holds, downstream analytics that use LLM-generated replies as proxies for user sentiment will systematically overestimate approval, which matters for product feedback, public-opinion tracking, and content moderation.
- The same two-cluster pipeline could be applied to other language models and to non-English posts to test whether near-universal positivity is model-specific or a general property of aligned conversational AI.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces EmoXpt, an unsupervised sentiment analysis framework that uses BERT embeddings and k-means clustering to compare the sentiment of human Twitter comments about generative AI with ChatGPT-generated responses to those tweets. The central quantitative claim, stated in the abstract, conclusion, and Table V, is that ChatGPT responses are overwhelmingly positive (90% positive, 10% negative) while human comments are mostly negative (27% positive, 72% negative). The paper also reports silhouette scores for word- and sentence-level clustering and interprets these as evidence that ChatGPT responses are more efficient, cohesive, and consistently positive than human comments. The manuscript includes data collection details over a two-month period in 2023, preprocessing steps, a description of the k-means pipeline, and a discussion of threats to validity.
Significance. If the headline result were properly supported, the paper would offer a useful, albeit modest, data point on the sentiment of LLM-generated responses relative to human commentary. The proposed framework itself, however, is a straightforward application of standard BERT embeddings plus k-means, and the paper's distinctive contribution is entirely the empirical comparison in Table V. Unfortitingly, the paper provides no reproducible pipeline, no code or data, and no validation of the claimed sentiment percentages. The central claim therefore rests on an unspecified methodological step, and the internal data inconsistencies further undermine confidence in the reported numbers. The strengths are the clear motivation to compare human and LLM sentiment and the explicit acknowledgment of some validity threats, but these do not compensate for the missing analysis.
major comments (4)
- [§IV-B, Table V] The central result of the paper, that ChatGPT responses are 90% positive and human comments are 27% positive, is not a determinate consequence of the described method. The only modeling step described is k-means with k=2 on BERT embeddings (Section III-D), which outputs unlabeled clusters. Neither the Silhouette Scores in Table IV nor the t-SNE plots in Figures 9–10 assign polarity to clusters. The paper does not specify any cluster-labeling rule, such as centroid inspection against a sentiment lexicon, a labeled validation set, or a bootstrapping procedure. Therefore the percentages in Table V cannot be derived from the described pipeline, and the abstract's and conclusion's claims that LLM-generated responses are 'notably more positive' are unsupported. The authors must specify exactly how clusters were mapped to positive/negative labels and how the percentages were computed.
- [§III-C] The preprocessing is asymmetric. Human comments are cleaned with steps shown in Figure 5, while ChatGPT responses additionally have 'frequently occurring neutral words' such as 'AI', 'language', and 'model' removed (Figure 6). This step is applied only to ChatGPT responses, not to human comments. Removing words like 'AI' from one corpus but not the other changes the token distributions and resulting BERT embeddings, which in turn can systematically alter the k-means clusters and any derived sentiment percentages. The paper provides no justification for why these words are 'neutral' in ChatGPT responses but not in human comments, and no sensitivity analysis is given to show that the headline 90% versus 27% comparison is robust to this asymmetric cleaning. This is a second load-bearing bias that could inflate ChatGPT's apparent positivity even if the cluster-labeling problem were resolved.
- [§III-A and §IV-B] Several data statistics in the paper are internally inconsistent. The paper states there are 512 human tweets, 429 user comments, and ChatGPT responses for each tweet (Section III-A). It also states that 75% of tweets have no comments and that the average number of comments is 28.4 with a maximum of 29; with 429 comments over 512 tweets the mean is approximately 0.84, and an average of 28.4 is arithmetically impossible. Additionally, in Section III-C the sentence embedding matrix is described as '(768, 429) where 768 is the number of data points and 429 is the features extracted for each data point,' which confuses the embedding dimension with the sample count; the correct dimensions would be (429, 768) for human comments and (512, 768) for ChatGPT responses. These inconsistencies indicate that the data handling and reporting are not reliable, which further erodes confidence in Table V.
- [§IV-B, RQ1 and RQ2] The interpretation of the silhouette scores is not supported by the metric. The paper states that ChatGPT words have a silhouette score of 0.053 and human words 0.13, and interprets the lower ChatGPT score as indicating 'a more focused, repetitive vocabulary' and 'consistent and direct' language. However, a silhouette score near zero actually indicates that clusters are not well separated and that data points are about as close to their own cluster as to the neighboring cluster; it does not measure vocabulary diversity or focus. The same misunderstanding applies to the sentence-level scores in Table IV. The paper's claim that ChatGPT responses are 'cohesive' based on these scores is therefore not justified.
minor comments (5)
- [Abstract / Conclusion] The claim that ChatGPT responses are 'more efficient, cohesive, and consistently positive' uses terms that are not defined or measured in the experimental section; the paper only measures clustering quality and sentiment polarity, not efficiency or cohesion in any operational sense.
- [§III-C heading] The subsection title 'Data Prepossessing' is a typo for 'Data Preprocessing'.
- [§III-D] The phrase 'which is usesful for identifying contrasting sentiments' contains a typo ('usesful' should be 'useful').
- [§V] In 'Threats to Conclusion Validity,' the sentence 'To reduce the conclusion validity of this work' should read 'To reduce the threat to conclusion validity' or similar.
- [§VI references] Several references are incomplete or formatted inconsistently; for example, reference [9] has the title 'Usecase' rather than a meaningful title, and references [15], [23], and [24] mix URL text with the actual resource names.
Circularity Check
No circular reduction is present in the derivation chain; the RQ3 positivity claim is underdetermined by the described K-means pipeline because cluster-to-polarity labeling is never specified, and the only self-citations ([26], [27]) are non-load-bearing.
full rationale
The paper's central claim, that ChatGPT responses are about 90% positive versus 27% for human comments (Table V, Section IV-B), is presented as the outcome of an unsupervised sentiment analysis pipeline (Section III-D) that trains K-means with k=2 on BERT embeddings. That pipeline outputs unlabeled partitions, and no equation or rule in the paper maps the two clusters to the polarity labels 'positive' and 'negative'; Section IV-B simply states 'Based on the percentages in Table V, we infer...', so the percentages are not shown to be a consequence of the described method. This is an underdetermination or completeness gap, not circularity: the conclusion is not an input to the method, no parameter was fitted to the 90%/27% target, and the claim does not reduce by construction to the K-means output or to the BERT embeddings. The paper's own Section V concedes that 'ChatGPT often generates positive sentiments, even using negative sentiment words in a positive way, making it challenging for an unsupervised model to predict', corroborating that the polarity assignment is an unreported extra step rather than a forced derivation. The asymmetric cleaning step (Section III-C) that removes frequent words such as 'AI', 'language', and 'model' from ChatGPT responses only is a plausible source of bias, but removing tokens does not by construction force the 90%/27% split, so it is a validity concern, not a circular one. The only self-citations are [26] and [27], both co-authored by T. S. Zaman, cited in Section V merely to frame the four standard threats-to-validity categories; the sentiment pipeline itself rests on independent references (BERT [6], K-means [14], the unsupervised sentiment pipeline [5] and [15], silhouette scores [16], and t-SNE [17]), none of which are self-citations. Accordingly, the derivation chain is not self-referential, and the score of 2 reflects the presence of minor non-load-bearing self-citations rather than any circularity.
Assumptions & free parameters
free parameters (3)
- Number of K-means clusters =
2
- ChatGPT response generation settings =
Not reported
- Removed 'neutral' word list =
e.g., 'AI', 'language', 'model'
assumptions (4)
- domain assumption BERT embeddings capture sentiment-relevant semantics
- domain assumption K-means clusters correspond to positive and negative sentiment
- domain assumption The cleaning procedures do not differentially bias sentiment
- domain assumption The collected tweets are representative
Cite this review
Pith. "Pith review of EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses." pith.science (2026). https://pith.science/paper/AUYF263F
@misc{pith2026250106597,
author = {Pith},
title = {Pith review of: EmoXpt: Analyzing Emotional Variances in Human Comments and LLM-Generated Responses},
year = {2026},
howpublished = {\url{https://pith.science/paper/AUYF263F}},
note = {Machine review of arXiv:2501.06597}
}
read the original abstract
The widespread adoption of generative AI has generated diverse opinions, with individuals expressing both support and criticism of its applications. This study investigates the emotional dynamics surrounding generative AI by analyzing human tweets referencing terms such as ChatGPT, OpenAI, Copilot, and LLMs. To further understand the emotional intelligence of ChatGPT, we examine its responses to selected tweets, highlighting differences in sentiment between human comments and LLM-generated responses. We introduce EmoXpt, a sentiment analysis framework designed to assess both human perspectives on generative AI and the sentiment embedded in ChatGPT's responses. Unlike prior studies that focus exclusively on human sentiment, EmoXpt uniquely evaluates the emotional expression of ChatGPT. Experimental results demonstrate that LLM-generated responses are notably more efficient, cohesive, and consistently positive than human responses.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
Available: https: //www.cloudflare.com/learning/ai/what-is-large-language-model/
What is a large language model (llm)? [Online]. Available: https: //www.cloudflare.com/learning/ai/what-is-large-language-model/
- [3]
-
[4]
Generative ai,
S. Feuerriegel, J. Hartmann, C. Janiesch, and P. Zschech, “Generative ai,” Business & Information Systems Engineering , vol. 66, no. 1, pp. 111–126, 2024
2024
-
[5]
Unsupervised sentiment analysis with emotional signals,
X. Hu, J. Tang, H. Gao, and H. Liu, “Unsupervised sentiment analysis with emotional signals,” in Proceedings of the 22nd International Conference on World Wide Web, ser. WWW ’13. New York, NY , USA: Association for Computing Machinery, 2013, p. 607–618. [Online]. Available: https://doi.org/10.1145/2488388.2488442
arXiv 2013
-
[6]
Homophobic and hate speech detection using multilingual-bert model on turkish social media,
H. Karayi ˘git, A. Akdagli, and C ¸ . ˙I. Aci, “Homophobic and hate speech detection using multilingual-bert model on turkish social media,” Information Technology and Control , vol. 51, no. 2, pp. 356–375, 2022
work page 2022
-
[7]
Comparison of bert models and machine learning methods for sentiment analysis on turkish tweets,
Z. A. Guven, “Comparison of bert models and machine learning methods for sentiment analysis on turkish tweets,” in 2021 6th International Conference on Computer Science and Engineering (UBMK) , 2021, pp. 98–101
work page 2021
-
[8]
Review of research on text sentiment analysis based on deep learning,
W. Li et al. , “Review of research on text sentiment analysis based on deep learning,” Open Access Library Journal , vol. 7, no. 03, p. 1, 2020
work page 2020
Show all 30 references
-
[9]
[Online]
Usecase. [Online]. Available: https://www.nytimes.com/2023/02/16/ technology/bing-chatbot-microsoft-chatgpt.html
2023
-
[10]
I think this is the most disruptive technology: Exploring sentiments of chatgpt early adopters using twitter data,
M. U. Haque, I. Dharmadasa, Z. T. Sworna, R. N. Rajapakse, and H. Ahmad, “I think this is the most disruptive technology: Exploring sentiments of chatgpt early adopters using twitter data,” 2022. [Online]. Available: https://arxiv.org/abs/2212.05856
2022 arXiv
-
[11]
Public perception of generative ai on twitter: an empirical study based on occupation and usage,
K. Miyazaki, T. Murayama, T. Uchiba, J. An, and H. Kwak, “Public perception of generative ai on twitter: an empirical study based on occupation and usage,” EPJ Data Science , vol. 13, no. 1, p. 2, 2024
2024
-
[12]
Analyzing the user’s sentiments of chatgpt using twitter data,
A. Korkmaz, C. Akturk, and T. Talan, “Analyzing the user’s sentiments of chatgpt using twitter data,” Iraqi Journal F or Computer Science and Mathematics, vol. 4, no. 2, p. 202–214, May 2023. [Online]. Available: https://journal.esj.edu.iq/index.php/IJCM/article/view/618
2023
-
[13]
[Online]
Nltk. [Online]. Available: https://www.nltk.org/https://huggingface.co/
-
[14]
An unsupervised fuzzy clustering method for twitter sentiment analysis,
H. Suresh and G. R. S., “An unsupervised fuzzy clustering method for twitter sentiment analysis,” in 2016 International Conference on Com- putation System and Information Technology for Sustainable Solutions (CSITSS), 2016, pp. 80–85
2016
-
[15]
Twitter, myspace, digg: Unsupervised sentiment analysis in social media,
G. Paltoglou and M. Thelwall, “Twitter, myspace, digg: Unsupervised sentiment analysis in social media,” ACM Trans. Intell. Syst. Technol., vol. 3, no. 4, Sep. 2012. [Online]. Available: https: //doi.org/10.1145/2337542.2337551
2012
-
[16]
Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,
P. J. Rousseeuw, “Silhouettes: A graphical aid to the interpretation and validation of cluster analysis,” Journal of Computational and Applied Mathematics , vol. 20, pp. 53–65, 1987. [Online]. Available: https://www.sciencedirect.com/science/article/pii/0377042787901257
1987
-
[17]
Visualizing data using t-sne
L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research , vol. 9, no. 11, 2008
2008
-
[18]
A survey of text representation and embedding techniques in nlp,
R. Patil, S. Boit, V . Gudivada, and J. Nandigam, “A survey of text representation and embedding techniques in nlp,” IEEE Access, vol. 11, pp. 36 120–36 146, 2023
2023
-
[19]
Euclidean distance mapping,
P.-E. Danielsson, “Euclidean distance mapping,” Computer Graphics and image processing , vol. 14, no. 3, pp. 227–248, 1980
1980
-
[20]
[Online]
Python 3.10. [Online]. Available: https://www.python.org/downloads/ release/python-3100/
-
[21]
[Online]
Pandas. [Online]. Available: https://pandas.pydata.org/
-
[22]
[Online]
Numpy. [Online]. Available: https://numpy.org/
-
[23]
[Online]
Hugging face. [Online]. Available: https://huggingface.co/https: //scikit-learn.org/
-
[24]
[Online]
Scikit-learn. [Online]. Available: https://scikit-learn.org/https:// matplotlib.org/
-
[25]
Matplotlib in python,
N. Ari and M. Ustazhanov, “Matplotlib in python,” in 2014 11th International Conference on Electronics, Computer and Computation (ICECCO). IEEE, 2014, pp. 1–6
2014
-
[26]
Extracting implicit programming rules: comparing static and dynamic approaches,
T. S. Zaman and T. Yu, “Extracting implicit programming rules: comparing static and dynamic approaches,” in Proceedings of the 7th International Workshop on Software Mining , ser. SoftwareMining 2018. New York, NY , USA: Association for Computing Machinery, 2018, p. 1–7. [Onli...
2018
-
[27]
Netminer: Identifying failure-inducing patterns in the logs generated by network monitoring software,
G. R. Sathi, L. Vedullapalli, M. H. Kishan, T. S. Zaman, M. T. Islam, and M. M. Badr, “Netminer: Identifying failure-inducing patterns in the logs generated by network monitoring software,” in 2023 14th International Conference on Computing Communication and Networking Technol...
2023
-
[28]
[Online]
Gemini. [Online]. Available: https://www.gemini.com/
-
[29]
[Online]
Claude. [Online]. Available: https://www.anthropic.com/claude
-
[30]
[Online]
Llama. [Online]. Available: https://www.llama.com/
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.