Pith. sign in

REVIEW 4 major objections 5 minor 24 references

On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform Setting

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Majority voting across sentiment tools rarely beats the best single tool in cross-platform settings.

desk verdict Useful negative result about majority-vote ensembles in cross-platform sentiment analysis, but the 'in most cases' claim rests on unseeded runs with random tie-breaking. read the letter →

arxiv 2502.06665 v1 pith:IRWO46CL submitted 2025-02-10 cs.SE

Martin Obaidi , Henrik Holm , Kurt Schneider , Jil Klünder This is my paper · ORCID
classification cs.SE
keywords sentimentanalysissoftwareengineeringmajorityvotingensemblelearningcross-platformpolarityclassificationappreviewsdevelopercommunication
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests a proposed remedy for a known failure: sentiment analysis tools trained on one software-engineering platform (such as GitHub) lose accuracy when applied to another (such as Stack Overflow), and prior work suggested combining tools by majority vote to spread domain expertise across platforms. The authors build a voting classifier from three tools and five polarity data sets, then compare the ensemble with its best member in within-platform and cross-platform experiments. The result is mixed: the vote gains a small 1–2% accuracy edge within a known domain, but in cross-platform runs it beats the best single tool in only 2 of 20 runs and usually ties or trails it. If correct, this means ensemble voting is not a reliable cure for cross-platform degradation, and practitioners facing an unknown domain should prefer the strongest individual tool.

What carries the argument

The carrying object is the voting classifier: a majority-vote ensemble in which Senti4SD (a support-vector-machine tool), RoBERTa (a transformer model), and SEnti-Analyzer (a meeting-oriented classifier) each label a statement's polarity, the majority label wins, and a random label is assigned when all three disagree. The comparison metrics are accuracy and macro-averaged F1, with Fleiss' $\kappa$ measuring how often the members agree. The $\kappa$ values drop from 0.68–0.84 within-platform to 0.14–0.42 cross-platform, quantifying exactly where the ensemble stops helping: out-of-domain members disagree more, disagreement rates rise to as much as 12% of sentences, and the vote then has no majority to impose.

What would settle it

Re-label the API and app-review test sets under the same emotion-model-to-polarity guideline used for the GitHub gold standard, then re-run the twelve first-experiment cross-platform combinations: if the voting classifier beats the best member in a majority of those runs, the paper's negative verdict would be an artifact of test-label noise, while if it still trails, the verdict survives label quality.

Watch

Extended reading notes

Core claim

The paper's finding is that a majority-vote ensemble of sentiment analysis tools is a good idea inside a known domain but not as a fix for the cross-platform problem. Trained and tested within the same platform, the voting classifier beats its best member by 1–2% on four of the five data sets; but across platforms it beats the best member in only one of the twelve first-experiment runs (62% vs. 61% on the app-review data) and is seldom top in the second experiment (highest or tied in three of eight runs). The decisive pattern is the spread among members: when the tools pre-trained in one domain disagree sharply with the others — as JIRA-trained members repeatedly did, scoring as low as 25% on app reviews while a GitHub-trained RoBERTa scored 77% — the majority vote inherits the weak members rather than being rescued by the strong one. The authors conclude that in an unknown domain the best individual tool is generally preferable, and that the greatest influence on a tool's cross-platform accuracy is the data set it was pre-trained on.

Load-bearing premise

The load-bearing premise is that the five data sets' labels are reliable and comparable enough that the accuracy differences reflect genuine tool and ensemble quality rather than annotation noise, even though the authors flag that the API and app-review sets were 'presumably labeled ad hoc' and the JIRA labels date from 2016.

Editorial extensions

If this is right

  • Within a known domain, a majority vote of well-matched tools is a simple way to buy a 1–2% accuracy gain over the best single member.
  • In an unknown domain, the vote fails exactly when the members' performances diverge, so a tool pre-trained on data far from the target domain drags the ensemble below the strongest member.
  • The pre-training data set shapes cross-platform accuracy more than the choice of tool architecture does: GitHub-trained members dominated, JIRA-trained members were worst, and no API- or JIRA-trained tool was ever the best.
  • Inter-tool agreement is a usable early-warning signal: when Fleiss' $\kappa$ drops to the 0.14–0.42 range, the ensemble loses to its best member, and the sentences where all three tools disagree (up to 12% of app-review data) receive random labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper leaves untested is member pruning or weighted voting: the data suggest an ensemble should admit only tools whose expected accuracies lie within a few points of each other, since the weak member is what sinks the vote.
  • The 'best individual tool' result is entangled with the 'best-labeled data set' result — RoBERTa's edge may be partly a GitHub-label-quality edge — so the ranking could shift on re-labeled versions of the ad hoc data sets.
  • The same failure mode should appear in other multi-classifier ensembles in software engineering (e.g., bug-versus-feature classification), where a majority vote inherits the worst member whenever member quality varies, so reliability-weighted fusion is the general lesson.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates a majority-voting ensemble of three sentiment analysis tools (Senti4SD, RoBERTa, SEnti-Analyzer) on five software-engineering datasets in within-domain and cross-platform settings. In within-domain experiments (RQ1), the voting classifier achieves accuracies roughly 1–2% above the best individual tool in four of five datasets. In cross-platform experiments (RQ2.1), the voting classifier beats or ties the best individual tool in only a minority of runs, and in RQ2.2, which uses RoBERTa variants trained on different domains, it is top or tied in a small number of runs. The paper concludes that while a voting classifier is useful within a known domain, it is not a reliable fix for cross-platform sentiment analysis and that the best individual tool is generally preferable.

Significance. The research question is relevant to software-engineering sentiment analysis, where cross-platform degradation of supervised tools is an established problem. If the reported comparison were accompanied by proper uncertainty quantification, the negative result would be a useful practical caveat against treating majority voting as a general remedy. The paper also has strengths: it uses several publicly available datasets, reports accuracies and Fleiss' kappa for every run, and includes a candid threats-to-validity section. However, the central quantitative claim currently rests on single unseeded runs with random tie-breaking, and several reported margins are within what could plausibly be noise.

major comments (4)
  1. [Section 3.2, Section 4.2, Tables 6 and 7] The central comparison between the voting classifier and the best individual tool depends on counts of wins, ties, and losses, but every run uses random tie-breaking when the three tools disagree, and no seed, repetition, variance estimate, or confidence interval is reported. Section 3.2 states that on complete disagreement the output label is set randomly, and Section 4.2 reports disagreement rates up to 12.0% (ID 10.2) and 11.7% (ID 8.2) on the APP test set. Several margins in Tables 6 and 7 are only 0.01–0.02 (e.g., IDs 7.2, 9.1, 14.1), and some are exact ties. A single unseeded draw cannot establish the claim that 'in most cases, the best individual tool in the ensemble is preferable'; the authors should report repeated runs with explicit seeds or provide statistical tests (e.g., McNemar or bootstrap confidence intervals) and show that the win/tie/loss counts are stable.
  2. [Section 4.1, Section 5.3] The within-domain claim that the voting classifier is a good choice rests on 1–2% accuracy gains over the best individual tool in IDs 1.1, 2.1, 4.1, and 5.1. The threats-to-validity section mentions that the evaluation was repeated multiple times, but the paper provides no numbers, seeds, or statistical tests for these repetitions. Without a significance test or variance estimate, these gains could be noise, which would weaken RQ1 and the contrast with RQ2. The authors should either report the repeated-evaluation evidence quantitatively or apply a paired test appropriate for the classification task.
  3. [Section 3.3, Section 5.2, Section 5.3] Label comparability is load-bearing for the cross-platform conclusion. The five datasets use different labeling processes: GitHub and Stack Overflow are emotion-model-based gold standards, JIRA is mapped from emotion labels, and API and APP are acknowledged in Section 5.3 as 'presumably labeled ad hoc'. The authors themselves suggest in Section 5.2 that subjectivity and labeling rigor may explain results. Since the central claim is that the voting classifier is not preferable in cross-platform domains, the analysis should include a robustness check, for example by excluding APP or JIRA, or by analyzing whether label noise, rather than tool/ensemble behavior, drives the observed accuracy differences.
  4. [Section 3.7, Table 4, Section 5.3] RQ2.2 is described as combining the best tools for different domains, but Table 4 shows that all three ensemble members are RoBERTa variants, differing only in training domain. This means RQ2.2 does not test complementary algorithmic diversity, only repeated training of the same architecture. The construct-validity threat in Section 5.3 acknowledges this, but the conclusion that RQ2.2 is preferable to RQ2.1 should be more strongly qualified, or a diversity metric among the models should be reported.
minor comments (5)
  1. [Section 2.2] The sentence 'To the best of your knowledge' should read 'To the best of our knowledge'.
  2. [Table 7] The row label '14.1' appears twice, once for the JIRA test set and once for APP; the APP row should be '14.2' to be consistent with the numbering scheme described in Section 3.7.
  3. [Section 3.3] The statement that 'Unlike the other data sets, the JIRA statements have emotions as a label' is confusing, because the GitHub and Stack Overflow datasets are also described as emotion-based in Section 2.3; the comparison to the other datasets should be clarified.
  4. [Table 1 and Table 7] The JIRA row in Table 7 reports 3976 test instances, while Table 1 lists 3974 documents; the discrepancy should be resolved or explained.
  5. [Section 4.2] For several runs the reported differences are at the level of rounding precision (e.g., 0.73 versus 0.73), making it unclear whether 'tie' means identical accuracy or a difference smaller than 0.005; reporting exact counts or additional decimals would improve interpretability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: central results are held-out empirical comparisons; self-citations are not load-bearing.

full rationale

The paper's central claims are derived from held-out test-set accuracies reported in Tables 5-7, not from the assumptions or fitted parameters that define the voting classifier. The voting classifier is a deterministic (except for random tie-breaking at complete disagreement) function of the three tools' predictions, and its accuracy is compared to the individual tools on the same test folds. For RQ2.2, the selection of the 'best tool for each domain' is based on within-domain cross-validation accuracies and F1 scores, not on the cross-platform test labels, so the later cross-platform comparison is not forced by construction. The self-citations ([6] Herrmann and Klünder, [18] Obaidi et al.) appear in the motivation for using majority voting and as the origin of one tool (SEnti-Analyzer); they do not constitute the evidence for the paper's conclusions. The random label assignment on complete disagreement (Section 3.2) without reported seeds or repetitions is a reproducibility and validity concern, as is the authors' own note that the RQ1 gains could be coincidental, but these are not circularity because the reported accuracies are still independent measurements rather than restatements of inputs. No load-bearing reduction of a prediction to a fit or to a self-citation chain was found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper's claims rest on the reliability and comparability of five external polarity datasets, on the complementarity of the three chosen tools, and on the neutrality of random tie-breaking for three-way disagreements. No new entities or fitted parameters are introduced. The label-comparability premise is the most fragile, and the authors themselves supply evidence against it in Sections 5.2 and 5.3.

assumptions (3)
  • domain assumption Polarity labels in the five datasets are reliable and comparable enough to measure cross-domain performance differences.
    Invoked throughout Sections 3.3 and 4. Sections 5.2 and 5.3 admit that API and APP labels were likely assigned ad hoc and JIRA labels are old, so this premise is load-bearing.
  • domain assumption The three selected tools represent complementary sentiment analysis approaches, so a majority vote is a meaningful ensemble.
    Section 3.2 states the tools use different ML approaches and thus likely complement each other. If they are not complementary, the negative result may be an artifact of the chosen ensemble composition.
  • ad hoc to paper Random label assignment on three-way disagreement is a neutral tie-break.
    Section 3.2 introduces this rule without analysis. In cross-platform runs disagreement reaches 12%, so this premise can materially affect voting classifier accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform Setting." pith.science (2026). https://pith.science/paper/IRWO46CL

@misc{pith2026250206665,
  author       = {Pith},
  title        = {Pith review of: On the Limitations of Combining Sentiment Analysis Tools in a Cross-Platform Setting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IRWO46CL}},
  note         = {Machine review of arXiv:2502.06665}
}
read the original abstract

A positive working climate is essential in modern software development. It enhances productivity since a satisfied developer tends to deliver better results. Sentiment analysis tools are a means to analyze and classify textual communication between developers according to the polarity of the statements. Most of these tools deliver promising results when used with test data from the domain they are developed for (e.g., GitHub). But the tools' outcomes lack reliability when used in a different domain (e.g., Stack Overflow). One possible way to mitigate this problem is to combine different tools trained in different domains. In this paper, we analyze a combination of three sentiment analysis tools in a voting classifier according to their reliability and performance. The tools are trained and evaluated using five already existing polarity data sets (e.g. from GitHub). The results indicate that this kind of combination of tools is a good choice in the within-platform setting. However, a majority vote does not necessarily lead to better results when applying in cross-platform domains. In most cases, the best individual tool in the ensemble is preferable. This is mainly due to the often large difference in performance of the individual tools, even on the same data set. However, this may also be due to the different annotated data sets.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 10 canonical work pages

  1. [1]

    Knowledge-B ased Systems 195, 105633 (2020)

    Cabrera-Diego, L.A., Bessis, N., Korkontzelos, I.: Clas sifying emotions in stack overflow and jira using a multi-label approach. Knowledge-B ased Systems 195, 105633 (2020). https://doi.org/10.1016/j.knosys.2020.105633

    work page arXiv 2020

  2. [2]

    Empirical Software Enginee ring 23(3), 1352–1382 (2018)

    Calefato, F., Lanubile, F., Maiorano, F., Novielli, N.: S entiment polarity detec- tion for software development. Empirical Software Enginee ring 23(3), 1352–1382 (2018). https://doi.org/10.1007/s10664-017-9546-9

    doi:10.1007/s10664-017-9546-9 2018

  3. [3]

    Psychological Bulletin 76(5), 378–382 (1971)

    Fleiss, J.L.: Measuring nominal scale agreement among ma ny raters. Psychological Bulletin 76(5), 378–382 (1971). https://doi.org/doi.org/10.1037/h0031619

    doi:10.1037/h0031619 1971

  4. [4]

    In: P roceedings of the 39th International Conference on Software Engineering: Ne w Ideas and Emerging Results Track

    Gachechiladze, D., Lanubile, F., Novielli, N., Serebren ik, A.: Anger and its direction in collaborative software development. In: P roceedings of the 39th International Conference on Software Engineering: Ne w Ideas and Emerging Results Track. p. 11–14. ICSE-NIER ’17, IEEE Press (2017). https://doi.org/10.1109/ICSE-NIER.2017.18

    work page doi:10.1109/icse-nier.2017.18 2017

  5. [5]

    Journal of Software: Evolution and Process 27(7), 467–487 (2015)

    Graziotin, D., Wang, X., Abrahamsson, P.: Do feelings mat ter? on the cor- relation of affects and the self-assessed productivity in so ftware engineer- ing. Journal of Software: Evolution and Process 27(7), 467–487 (2015). https://doi.org/10.1002/smr.1673

    doi:10.1002/smr.1673 2015

  6. [6]

    In : 2021 IEEE 29th Inter- national Requirements Engineering Conference Workshops ( REW)

    Herrmann, M., Klünder, J.: From textual to verbal communi cation: Towards ap- plying sentiment analysis to a software project meeting. In : 2021 IEEE 29th Inter- national Requirements Engineering Conference Workshops ( REW). pp. 371–376 (2021). https://doi.org/10.1109/REW53955.2021.00065

    arXiv 2021

  7. [7]

    https://doi.org/10.1016/j.jss.2022.111448

    Herrmann, M., Obaidi, M., Chazette, L., Klünder, J.: On th e subjectivity of emotions in software projects: How reliable are pre-labe led data sets for sentiment analysis? Journal of Systems and Software 193, 111448 (2022). https://doi.org/10.1016/j.jss.2022.111448

    work page arXiv 2022

  8. [8]

    Biometrics 33(1), 159–174 (1977)

    Landis, J.R., Koch, G.G.: The measurement of observer agr ee- ment for categorical data. Biometrics 33(1), 159–174 (1977). https://doi.org/https://doi.org/10.2307/2529310

    doi:10.2307/2529310 1977

Show all 24 references
  1. [9]

    ACM Trans

    Lin, B., Cassee, N., Serebrenik, A., Bavota, G., Novielli , N., Lanza, M.: Opin- ion mining for software development: A systematic literatu re review. ACM Trans. Softw. Eng. Methodol. 31(3) (mar 2022). https://doi.org/10.1145/3490388

    2022 doi

  2. [10]

    , Oliveto, R.: Sen- timent analysis for software engineering: How far can we go? In: Proceed- ings of the 40th International Conference on Software Engin eering

    Lin, B., Zampetti, F., Bavota, G., Di Penta, M., Lanza, M. , Oliveto, R.: Sen- timent analysis for software engineering: How far can we go? In: Proceed- ings of the 40th International Conference on Software Engin eering. p. 94–104. ICSE ’18, Association for Computing Machinery...

    2018

  3. [11]

    https://doi.org/10.48550/ARXIV.1907.11692

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Le vy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimiz ed bert pretraining approach (2019). https://doi.org/10.48550/ARXIV.1907.11692

    2019 doi arXiv

  4. [12]

    Cambridge Univ

    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to information retrieval. Cambridge Univ. Press, Cambridge, reprinted. edn. (2009) 16 Obaidi et al

    2009

  5. [13]

    Novielli, N., Calefato, F., Dongiovanni, D., Girardi, D ., Lanubile, F.: Can we use se-specific sentiment analysis tools in a cross-platform se tting? In: Proceedings of the 17th International Conference on Mining Software Repos itories. p. 158–168. MSR ’20, Association for Com...

    2020

  6. [14]

    https://doi.org/10.6084/m9.figshare.11604597.v1

    Novielli, N., Calefato, F., Dongiovanni, D., Girardi, D ., Lanubile, F.: A gold standard for polarity of emotions of software developers in github (3 2020). https://doi.org/10.6084/m9.figshare.11604597.v1

    2020 doi

  7. [16]

    Empirical Software Engineering 26(4), 77 (Jun 2021)

    Novielli, N., Calefato, F., Lanubile, F., Serebrenik, A .: Assessment of off-the-shelf se-specific sentiment analysis tools: An exte nded replica- tion study. Empirical Software Engineering 26(4), 77 (Jun 2021). https://doi.org/10.1007/s10664-021-09960-w

    2021 doi

  8. [17]

    In: Proceedin gs of the 15th In- ternational Conference on Mining Software Repositories

    Novielli, N., Girardi, D., Lanubile, F.: A benchmark stu dy on sentiment analysis for software engineering research. In: Proceedin gs of the 15th In- ternational Conference on Mining Software Repositories. p . 364–375. MSR ’18, Association for Computing Machinery, New York, NY...

    2018

  9. [18]

    Information and Software Technology 151, 107018 (2022)

    Obaidi, M., Nagel, L., Specht, A., Klünder, J.: Sentimen t anal- ysis tools in software engineering: A systematic mapping st udy. Information and Software Technology 151, 107018 (2022). https://doi.org/https://doi.org/10.1016/j.infsof.2022.107018

    2022

  10. [19]

    In: Proceedings of the 13th International Conference on Mining Software Repos itories

    Ortu, M., Murgia, A., Destefanis, G., Tourani, P., Tonel li, R., Marchesi, M., Adams, B.: The emotional side of software developers in jira . In: Proceedings of the 13th International Conference on Mining Software Repos itories. p. 480–483. MSR ’16, Association for Computing M...

    2016

  11. [20]

    psychology press (2001)

    Parrott, W.G.: Emotions in social psychology: Essentia l readings. psychology press (2001)

    2001

  12. [21]

    Journal of Systems and Software 143, 59–70 (2018)

    Schneider, K., Klünder, J., Kortum, F., Handke, L., Stra ube, J., Kauffeld, S.: Positive affect through interactions in meetings: The ro le of proactive and supportive statements. Journal of Systems and Software 143, 59–70 (2018). https://doi.org/10.1016/j.jss.2018.05.001

    2018 doi

  13. [22]

    ACM Trans

    Uddin, G., Guéhénuc, Y.G., Khomh, F., Roy, C.K.: An empir ical study of the effectiveness of an ensemble of stand-alone sentiment detec tion tools for soft- ware engineering datasets. ACM Trans. Softw. Eng. Methodol . 31(3) (apr 2022). https://doi.org/10.1145/3491211

    2022 doi

  14. [23]

    IEEE Transactions on Software Engineering 47(3), 522–559 (2021)

    Uddin, G., Khomh, F.: Automatic mining of opinions expre ssed about apis in stack overflow. IEEE Transactions on Software Engineering 47(3), 522–559 (2021). https://doi.org/10.1109/TSE.2019.2900245

    2021

  15. [24]

    Springe r, Berlin (2012)

    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnel l, B., Wess- lén, A.: Experimentation in software engineering. Springe r, Berlin (2012). https://doi.org/10.1007/978-3-642-29044-2

    2012 doi

  16. [25]

    Zhang, T., Xu, B., Thung, F., Haryono, S.A., Lo, D., Jiang , L.: Sentiment analysis for software engineering: How far can pre-trained transfor mer models go? In: 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME). pp. 70–80 (2020). https://doi.org...

    2020

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.