Assessing Sentiment of the Expressed Stance on Social Media

Abeer Aldayel; Walid Magdy

REVIEW 3 major objections 5 minor 24 references

Assessing Sentiment of the Expressed Stance on Social Media

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sentiment alone cannot tell support from opposition in tweets

desk verdict A modest, honest confirmation that sentiment polarity does not equal stance; the new dataset is the real asset but it is not released and the analysis lacks reliability metrics. read the letter →

arxiv 1908.03181 v1 pith:C6WFEQCA submitted 2019-08-08 cs.SI cs.CL

Abeer Aldayel , Walid Magdy This is my paper · ORCID

classification cs.SIcs.CL

keywords stancedetectionsentimentanalysispublicopinioneventsocialmediaTwitterstance-sentimentalignment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sentiment polarity—whether a tweet reads positive, negative, or neutral—is often treated as a shortcut for detecting which side of a controversial issue the writer supports. This paper tests that shortcut directly, building a new context-dependent dataset of 6,324 reply tweets on four topics with separate gold labels for sentiment and stance, and re-analysing the established SemEval stance benchmark. The result is that sentiment and stance are far from interchangeable: negative tweets make up the bulk of both favour and against stances, while tweets that are both positive and in favour are rare. The authors conclude that public-opinion readings built on sentiment polarity alone will systematically misrepresent true support.

What carries the argument

The load-bearing objects are two sentiment-and-stance annotated tweet collections: the SemEval-2016 Task 6 benchmark (4,063 tweets, five topics) and a new context-dependent (CD) dataset of 6,324 reply tweets on four topics, annotated by five workers per tweet with majority-vote gold labels and the parent tweet shown for context. The analysis itself rests on conditional distributions of sentiment given stance (and vice versa), which expose the misalignment, and on Jaccard similarity between the top TF-IDF-weighted vocabularies of sentiment labels and stance labels, which quantifies how different the two expression systems are. The example rows in Table 2—positive-sentiment tweets that oppose, negative-sentiment tweets that support—carry the qualitative force of the argument.

What would settle it

Compute inter-annotator agreement on the CD dataset and recompute the sentiment-stance distributions using only tweets all five annotators labelled identically; if the against-negative and favour-positive proportions move substantially toward alignment, the paper's mismatch is inflated by label noise.

Watch

Extended reading notes

Core claim

The paper's central claim is that stance and sentiment are different dimensions of expression, and that polarity cannot be used on its own to infer stance toward a topic. In both the SemEval dataset and the new CD dataset, negative sentiment is the dominant polarity for supporting and opposing stances alike: over 54% of favour tweets in both datasets carry negative sentiment. Only about 33.9% of SemEval tweets and 30.9% of CD tweets are the 'matching' combination of against plus negative, while favour plus positive tweets make up just 8.9% and 4.0% respectively. A Jaccard analysis of the most frequent words per label shows less than 20% overlap between the vocabulary of favour stance and the vocabulary of positive sentiment, while against stance shares more words with negative sentiment. The authors therefore state that simple sentiment polarity cannot substitute for stance detection.

Load-bearing premise

The correctness of the gold labels—the majority-vote annotations in the CD dataset and the SemEval labels—is the load-bearing premise; if annotation noise differs across sentiment and stance classes, the observed mismatch could be a labeling artifact rather than a property of how people express stance.

Editorial extensions

If this is right

Sentiment polarity should not be used as a stand-alone signal for stance detection or public-opinion measurement.
Studies that equated negative sentiment with opposition and positive sentiment with support should be revisited, since the favour-positive match is below 9% in both datasets.
Stance classifiers that combine sentiment with other features have the right shape; the paper supports using sentiment as a complement, not a substitute.
Negative sentiment is a partially informative cue for against-stances, but it is diluted by the large share of negative favour tweets, so any threshold-based polarity rule will overcount opposition.
Future annotation efforts should label stance and sentiment as separate dimensions, as the CD dataset does, rather than deriving one from the other.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the new dataset consists of reply tweets, the mismatch the paper finds may be especially pronounced in conversational contexts, where rebuttals, irony, and defensive agreement routinely separate the literal polarity of words from the position being defended; extending the same analysis to standalone posts could show a weaker or stronger gap.
A direct, testable extension would be to train a stance classifier on sentiment features only and compare its accuracy with the majority-class baseline; the distributions here imply it would barely beat the baseline, which would quantify the ceiling of polarity-based stance inference.
The fact that favour-positive agreement is so low (4–9%) suggests that support in public discourse is often expressed as criticism of the other side rather than praise of the target; if that is general, positive sentiment is a poor instrument for detecting approval in any contested domain.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know this paper is a small, careful empirical study that confirms something the field already largely accepted: sentiment polarity alone does not tell you a Twitter user's stance. The authors themselves cite Mohammad et al. (2016, 2017) to that effect, so the conceptual result is not new. What is new is an annotated context-dependent dataset of 6,324 reply tweets across four controversial topics, with the parent tweet shown to annotators so they could judge stance in context. That dataset is potentially valuable, but the paper does not release it, and it reports no inter-annotator agreement for the five-worker majority labels. That is the main soft spot.

The strengths: the descriptive analysis is clear and the examples in Table 2 are genuinely illustrative (e.g., "Life is our first and most basic human right" labeled positive sentiment and Against stance on abortion). Using the SemEval 2016 stance dataset as a second, independent validation is good practice, and the SemEval numbers alone support the conclusion: within Favor tweets, 56% are negative, and only 8.9% of all tweets are Favor-Positive. So even if the CD dataset vanished, the central claim holds.

The weaknesses are proportionate. There are no significance tests, confidence intervals, or sensitivity analysis to label noise. The "mixed" category in Figure 4 is coarse—it lumps everything that is not Against-Negative or Favor-Positive, including neutral sentiment and None stance, into one bucket, which inflates the apparent mismatch. The appendix Jaccard analysis is underdeveloped and has a typo ("Jacquard"). But these are fixable and do not undermine the qualitative conclusion.

For a reader, this is a useful cautionary citation and a reasonable dataset paper if the authors ever release the data. As a research contribution it is confirmatory rather than groundbreaking, and a desk editor could reasonably note that the novelty is modest. However, the work is coherent, honest, and relevant to anyone using sentiment as a proxy for public opinion on social media. I would send it out to a serious referee, primarily to push for IAA, a data-release plan, and sharper metrics, not because the paper is flawed beyond repair.

Referee Report

3 major / 5 minor

Summary. The paper examines whether sentiment polarity can serve as a proxy for stance in social-media posts. It analyzes the SemEval-2016 Task 6 stance benchmark (4,063 tweets, five topics) and a new 'context-dependent' dataset of 6,324 reply tweets across four controversial topics, each annotated by five workers via majority vote. The authors report sentiment-by-stance distributions, the overall proportion of tweets in which stance and sentiment agree (Favor-Positive and Against-Negative), and TF-IDF top-word Jaccard similarities. They conclude that sentiment and stance are not highly aligned and that sentiment polarity alone should not be used to infer stance.

Significance. The paper addresses a practically important question: public-opinion studies frequently use sentiment as a proxy for stance, and this work provides quantitative and qualitative evidence that the two constructs differ. The use of the established SemEval benchmark as an external validation is a clear strength, and the four examples in Table 2 convincingly illustrate the phenomenon. The contribution would be more complete with reliability statistics for the new dataset and statistical tests for the aggregate comparisons; as it stands, the strength of the quantitative claims is limited by the absence of such measures. No code or data release is provided, so the new dataset cannot be independently checked.

major comments (3)

[Section 3 (Data collection)] The CD dataset is central to the paper's empirical contribution, but no inter-annotator agreement is reported for the five-annotator majority-vote labels. Because the entire alignment analysis depends on the reliability of these sentiment and stance gold labels, the paper should report Fleiss' kappa (or equivalent) per topic and per label type, and should report the distribution of agreement. Without this, the CD-based percentages (e.g., 65.1% mixed and 4.0% Favor-Positive in Fig. 4b) cannot be distinguished from annotation noise. The dataset also does not appear to be released; please provide a public link or a clear availability statement.
[Sections 4.1 and Appendix A] The central claim that sentiment and stance are 'not highly aligned' is supported only by raw proportions and informal comparisons of Jaccard curves; no significance test, confidence interval, or association measure (e.g., chi-square, Cramér's V, mutual information) is reported. The apparent mismatch is large, but the reader cannot assess whether the observed pattern exceeds what would be expected by chance or by the marginal distributions of the labels. Please add statistical tests or resampling baselines for the contingency tables in Fig. 2 and the Jaccard curves in Fig. 3, and consider a sensitivity analysis that varies the SemEval gold labels.
[Sections 1 and 5 (RQ2)] RQ2 asks 'When does positive/negative sentiment indicate support/against stance?', but the analysis only establishes that sentiment is not a reliable indicator overall; it does not identify conditions under which sentiment and stance do align beyond illustrative examples in Table 2. The Discussion's statement that negative sentiment 'could help' with against stances is not operationalized. Either provide a concrete characterization (e.g., by topic, target, or textual pattern) or explicitly narrow the paper's contribution to answering RQ1.

minor comments (5)

[Appendix A] The surname is misspelled as 'Jacquard'; it should be 'Jaccard' throughout.
[Section 5] The phrase 'The words choice gap exists' is awkward; consider 'There is a vocabulary gap between in-favor stance and positive sentiment' or similar.
[Figures 1, 2, and 4] The captions do not state whether the displayed percentages are normalized per row or per column; please make this explicit so the distributions can be interpreted correctly.
[Section 3] Since the CD labels are claimed to follow SemEval annotation guidelines, including the actual annotation instruction sheet in an appendix or supplementary material would help readers judge label quality.
[Overall] The paper would benefit from a short limitations paragraph explicitly acknowledging the reliance on gold labels and the absence of significance testing, rather than leaving these points implicit.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical comparison of independently collected gold labels, with the SemEval benchmark providing external support.

full rationale

The paper's central claim is that sentiment polarity alone cannot reliably denote stance. This claim is supported by descriptive statistics over two datasets: the existing SemEval stance benchmark and a newly constructed CD dataset. The SemEval dataset is an external resource with independently assigned gold labels, and the CD dataset is newly annotated by five annotators per tweet using majority voting. No parameter is fitted to the outcome, no prediction is derived from the data and then validated against the same data in a way that is forced by construction, and no load-bearing step depends on a self-citation. The Jaccard similarity measure in Equation 1 is a standard definition used to quantify word overlap, not an inference that presupposes the conclusion. The paper's finding that sentiment and stance are misaligned is a direct empirical observation about label distributions, such as the fact that within Favor-labeled SemEval tweets only 35.0% are positive while 56.1% are negative. Even if the CD dataset's reliability were questioned because no inter-annotator agreement is reported, that would be a data-quality limitation, not circularity. Therefore, the derivation chain is self-contained and the conclusion rests on independent evidence.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. Its conclusions rely on the accuracy of human annotations and on the choice of TF-IDF and Jaccard as a vocabulary-overlap measure.

assumptions (3)

domain assumption Majority vote of five Figure-eight annotators produces reliable gold labels for sentiment and stance in the CD dataset.
No inter-annotator agreement or annotation quality metrics are reported; the analysis treats these labels as ground truth.
domain assumption The SemEval stance dataset labels are correct and comparable to the CD labels.
The paper relies on the SemEval benchmark as validation without accounting for potential differences in annotation guidelines or data collection.
domain assumption TF-IDF and Jaccard similarity over top-N words is a valid measure of overlap between the vocabularies of sentiment and stance classes.
The appendix uses this method to draw conclusions about word-choice gaps; the choice of N and the threshold are not statistically justified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Assessing Sentiment of the Expressed Stance on Social Media." pith.science (2026). https://pith.science/paper/C6WFEQCA

@misc{pith2026190803181,
  author       = {Pith},
  title        = {Pith review of: Assessing Sentiment of the Expressed Stance on Social Media},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C6WFEQCA}},
  note         = {Machine review of arXiv:1908.03181}
}

read the original abstract

Stance detection is the task of inferring viewpoint towards a given topic or entity either being supportive or opposing. One may express a viewpoint towards a topic by using positive or negative language. This paper examines how the stance is being expressed in social media according to the sentiment polarity. There has been a noticeable misconception of the similarity between the stance and sentiment when it comes to viewpoint discovery, where negative sentiment is assumed to mean against stance, and positive sentiment means in-favour stance. To analyze the relation between stance and sentiment, we construct a new dataset with four topics and examine how people express their viewpoint with regards these topics. We validate our results by carrying a further analysis of the popular stance benchmark SemEval stance dataset. Our analyses reveal that sentiment and stance are not highly aligned, and hence the simple sentiment polarity cannot be used solely to denote a stance toward a given topic.

Figures

Figures reproduced from arXiv: 1908.03181 by the authors.

**Figure 2.** Distribution of sentiment per a given stance. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Jaccard similarity of the top N-most frequent words between sentiment [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Tweets with matching and mixed stance and sentiment. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages

[1]

In: International Conference on data warehousing and knowledge discovery

Achananuparp, P., Hu, X., Shen, X.: The evaluation of sentence similarity mea- sures. In: International Conference on data warehousing and knowledge discovery. pp. 305–316. Springer (2008)

work page 2008
[2]

Journal of Information and Optimization Sciences 39(1), 303–317 (2018)

Agarwal, A., Singh, R., Toshniwal, D.: Geospatial sentiment analysis using twit- ter data for uk-eu referendum. Journal of Information and Optimization Sciences 39(1), 303–317 (2018)

work page 2018
[3]

An, J., Kwak, H., Posegga, O., Jungherr, A.: Political discussions in homogeneous and cross-cutting communication spaces (2019)

work page 2019
[4]

Discourse processes 11(1), 1–34 (1988)

Biber, D., Finegan, E.: Adverbial stance types in english. Discourse processes 11(1), 1–34 (1988)

work page 1988
[5]

In: ASONAM’17 (2017)

Darwish, K., Magdy, W., Zanouda, T.: Improved Stance Prediction in a User Sim- ilarity Feature Space. In: ASONAM’17 (2017)

work page 2017
[6]

In: COLING

Ebrahimi, J., Dou, D., Lowd, D.: A Joint Sentiment-Target-Stance Model for Stance Classiﬁcation in Tweets. In: COLING. pp. 2656–2665 (2016)

work page 2016
[7]

In: Proceedings of the 10th International Work- shop on Semantic Evaluation (SemEval-2016)

Elfardy, H., Diab, M.: Cu-gwu perspective at semeval-2016 task 6: Ideological stance detection in informal text. In: Proceedings of the 10th International Work- shop on Semantic Evaluation (SemEval-2016). pp. 434–439 (2016)

work page 2016
[8]

International Journal of Computer Applications 68(13), 13–18 (2013)

Gomaa, W.H., Fahmy, A.A.: A survey of text similarity approaches. International Journal of Computer Applications 68(13), 13–18 (2013)

work page 2013

Show all 24 references

[9]

In: Twenty-Third International Joint Conference on Artiﬁcial Intelligence (2013)

Hu, Y., Wang, F., Kambhampati, S.: Listening to the crowd: automated analysis of events via aggregated twitter sentiment. In: Twenty-Third International Joint Conference on Artiﬁcial Intelligence (2013)

2013
[10]

In: SemEval@ NAACL-HLT

Igarashi, Y., Komatsu, H., Kobayashi, S., Okazaki, N., Inui, K.: Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection. In: SemEval@ NAACL-HLT. pp. 401–407 (2016)

2016
[11]

arXiv preprint arXiv:1904.02000 (2019)

Kareem, D., Peter, S., Aupetit, M.J., Preslav, N.: Unsupervised user stance detec- tion on twitter. arXiv preprint arXiv:1904.02000 (2019)

2019 arXiv
[12]

In: SemEval@ NAACL-HLT

Krejzl, P., Steinberger, J.: UWB at SemEval-2016 Task 6: Stance Detection. In: SemEval@ NAACL-HLT. pp. 408–412 (2016)

2016
[13]

In: International Conference on Social Informatics

Lee, H.W.: Using twitter hashtags to gauge real-time changes in public opinion: An examination of the 2016 us presidential election. In: International Conference on Social Informatics. pp. 168–175. Springer (2018)

2018
[14]

Handbook of natural language processing 2(2010), 627–666 (2010)

Liu, B., et al.: Sentiment analysis and subjectivity. Handbook of natural language processing 2(2010), 627–666 (2010)

2010
[15]

In: SemEval@ NAACL-HLT

Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X.D., Cherry, C.: SemEval-2016 Task 6: Detecting Stance in Tweets. In: SemEval@ NAACL-HLT. pp. 31–41 (2016)

2016
[16]

ACM Transactions on Internet Technology (TOIT) 17(3), 26 (2017)

Mohammad, S.M., Sobhani, P., Kiritchenko, S.: Stance and sentiment in tweets. ACM Transactions on Internet Technology (TOIT) 17(3), 26 (2017)

2017
[17]

In: International Conference on Social Computing, Behavioral- Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation

Overbey, L.A., Batson, S.C., Lyle, J., Williams, C., Regal, R., Williams, L.: Linking twitter sentiment and event data to monitor public opinion of geopolitical develop- ments and trends. In: International Conference on Social Computing, Behavioral- Cultural Modeling and Predi...

2017
[18]

In: Pro- ceedings of the ACM 2011 conference on Computer supported cooperative work

Park, S., Ko, M., Kim, J., Liu, Y., Song, J.: The politics of comments: predicting political orientation of news stories with commenters’ sentiment patterns. In: Pro- ceedings of the ACM 2011 conference on Computer supported cooperative work. pp. 113–122. ACM (2011) 8 A. Alday...

2011
[19]

In: Pro- ceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017

Smith, K.S., McCreadie, R., Macdonald, C., Ounis, I.: Analyzing disproportion- ate reaction via comparative multilingual targeted sentiment in twitter. In: Pro- ceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017. pp. ...

2017
[20]

In: Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics

Sobhani, P., Mohammad, S., Kiritchenko, S.: Detecting stance in tweets and ana- lyzing its interaction with sentiment. In: Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics. pp. 159–169 (2016)

2016
[21]

In: Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text

Somasundaran, S., Wiebe, J.: Recognizing stances in ideological on-line debates. In: Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text. pp. 116–124. Association for Com- putational Linguistics (2010)

2010
[22]

In: Twelfth International AAAI Con- ference on Web and Social Media (2018)

Trabelsi, A., Zaiane, O.R.: Unsupervised model for topic viewpoint discovery in online debates leveraging author interactions. In: Twelfth International AAAI Con- ference on Web and Social Media (2018)

2018
[23]

In: International Conference on Application of Natural Language to Information Systems

Tsolmon, B., Kwon, A.R., Lee, K.S.: Extracting social events based on timeline and sentiment analysis in twitter corpus. In: International Conference on Application of Natural Language to Information Systems. pp. 265–270. Springer (2012)

2012
[24]

In: International Conference on Web Information Systems Engineering

Unankard, S., Li, X., Sharaf, M., Zhong, J., Li, X.: Predicting elections from social networks based on sub-event detection and sentiment analysis. In: International Conference on Web Information Systems Engineering. pp. 1–16. Springer (2014) A Analysis of the textual patterns...

2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

In: International Conference on data warehousing and knowledge discovery

Achananuparp, P., Hu, X., Shen, X.: The evaluation of sentence similarity mea- sures. In: International Conference on data warehousing and knowledge discovery. pp. 305–316. Springer (2008)

work page 2008

[2] [2]

Journal of Information and Optimization Sciences 39(1), 303–317 (2018)

Agarwal, A., Singh, R., Toshniwal, D.: Geospatial sentiment analysis using twit- ter data for uk-eu referendum. Journal of Information and Optimization Sciences 39(1), 303–317 (2018)

work page 2018

[3] [3]

An, J., Kwak, H., Posegga, O., Jungherr, A.: Political discussions in homogeneous and cross-cutting communication spaces (2019)

work page 2019

[4] [4]

Discourse processes 11(1), 1–34 (1988)

Biber, D., Finegan, E.: Adverbial stance types in english. Discourse processes 11(1), 1–34 (1988)

work page 1988

[5] [5]

In: ASONAM’17 (2017)

Darwish, K., Magdy, W., Zanouda, T.: Improved Stance Prediction in a User Sim- ilarity Feature Space. In: ASONAM’17 (2017)

work page 2017

[6] [6]

In: COLING

Ebrahimi, J., Dou, D., Lowd, D.: A Joint Sentiment-Target-Stance Model for Stance Classiﬁcation in Tweets. In: COLING. pp. 2656–2665 (2016)

work page 2016

[7] [7]

In: Proceedings of the 10th International Work- shop on Semantic Evaluation (SemEval-2016)

Elfardy, H., Diab, M.: Cu-gwu perspective at semeval-2016 task 6: Ideological stance detection in informal text. In: Proceedings of the 10th International Work- shop on Semantic Evaluation (SemEval-2016). pp. 434–439 (2016)

work page 2016

[8] [8]

International Journal of Computer Applications 68(13), 13–18 (2013)

Gomaa, W.H., Fahmy, A.A.: A survey of text similarity approaches. International Journal of Computer Applications 68(13), 13–18 (2013)

work page 2013