Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Towards Trustworthy Sentiment Analysis in Software Engineering: Dataset Characteristics and Tool Selection

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that the best sentiment-analysis tool for a software-engineering dataset can be selected from 13 linguistic features and 8 statistics, via a questionnaire that matches the dataset to the platform whose profile it most…

desk verdict Solid descriptive benchmark; the tool-selection questionnaire is the weak link — the paper's own numbers show the recommendation barely beats always picking SetFit. read the letter →

arxiv 2507.02137 v2 pith:33A5AAHT submitted 2025-07-02 cs.SE

Martin Obaidi , Marc Herrmann , Jil Klünder , Kurt Schneider This is my paper · ORCID
classification cs.SE
keywords sentimentanalysissoftwareengineeringtoolselectiondatasetcharacteristicsdevelopercommunicationquestionnairetransformermodelsplatform
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the right sentiment-analysis tool for a software-engineering text dataset can be chosen from the dataset's observable characteristics. It shows that ten labeled datasets from five platforms (app reviews, code reviews, GitHub, Jira, Stack Overflow) differ measurably in 13 linguistic features and 8 statistics, and that 14 tools' F1 scores vary with those platform-linked traits. On that basis it proposes a questionnaire that maps a user's answers about a new, unlabeled dataset to the most similar platform and recommends the tool that performed best there. If it is right, researchers and practitioners get a low-barrier way to pick a trustworthy tool without labeled data or NLP expertise. The paper is an extension of prior cross-platform comparisons, aiming to turn observed variability into actionable selection guidance.

What carries the argument

The load-bearing mechanism is the characteristic-to-platform mapping: 13 manually coded linguistic features (direct emotion, emphasized positivity, technical focus, balanced critique, progress sharing, gratitude, inquisitiveness, help seeking or offering, compliments, bug-fix requests, constructive criticism, username mentioning, name mentioning) plus 8 statistics (characters per document, characters per word, words per document, capitalized words, spelling mistakes, emoticons, question marks, exclamation marks). The questionnaire converts user answers about these features into interval matches (True, Likely, Unlikely, Untrue), assigns a point to each platform whose precomputed profile falls in the user's interval, uses absolute difference to match statistics, and recommends the highest-scoring platform's best tool. The mapping is what connects an unlabeled dataset to a tool recommendation without retraining or annotation.

What would settle it

Collect a new labeled dataset from a platform or communication channel not among the five studied here, run the questionnaire's recommended tool against a fixed default such as SetFit, and check whether the recommended tool wins significantly more often than the default across several such datasets. A single held-out dataset where the recommended tool finishes below the default would directly test the predictive link.

Watch

Extended reading notes

Core claim

The central discovery is that platform communication styles are measurably distinct and that those distinctions carry information about which sentiment-analysis tool will do well. SetFit and RoBERTa achieve the strongest overall F1 on most of the ten datasets, yet no tool wins everywhere; the best choice tracks dataset characteristics such as direct emotion, gratitude, technical focus, and document length. The paper's proposed mapping turns these empirical regularities into a concrete procedure: interval-code the 13 linguistic features, take absolute-distance matching on the 8 statistics, count points per platform, and recommend the empirically best tool for the winning platform, falling back to SetFit or SentiStrength-SE in ambiguous cases. The authors present the mapping as a practical, low-barrier selection mechanism rather than a definitive ranking of tools.

Load-bearing premise

The mapping assumes that the platform whose profile best matches a user's answers will also be the platform whose best-performing tool ranks highest on the user's dataset, a link the paper asserts but does not validate on held-out data.

Editorial extensions

If this is right

  • If a user's dataset resembles a platform profile, the tool that topped that platform's benchmark should outperform a fixed default tool on the new data.
  • Transformer-based models (SetFit, RoBERTa, ELECTRA) become the recommended default for most developer-communication contexts, since they dominate SE-specific and lexicon tools on the evaluated datasets.
  • The ambiguous fallback (SetFit or SentiStrength-SE) gives a practical answer for datasets that match no platform profile, supporting robustness under dataset drift.
  • The questionnaire lowers the barrier to evidence-based tool selection because it requires only subjective feature answers, not labeled data or computational resources.
  • Platform names are a proxy rather than the target: the same machinery can be reused to match datasets directly by profile, for example via clustering.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to hold out entire datasets and check whether the questionnaire's recommended tool actually beats a fixed baseline such as SetFit on the held-out data; the paper asserts this predictive link but does not report such validation.
  • The 13 linguistic features are hand-coded with high inter-rater agreement, so the mapping could be automated with a classifier trained on those codes, making the questionnaire objective rather than subjective.
  • Because class balance varies strongly across datasets, the F1-based recommendation may flip if users weight recall of negative comments more heavily; a risk-weighted variant of the mapping would be a natural next step.
  • The claim that platform-level similarity predicts tool ranking could be probed by a correlation analysis between pairwise characteristic distances and pairwise tool-performance differences, which the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper analyzes the linguistic and statistical characteristics of 10 labeled sentiment-analysis datasets from five software-engineering platforms, evaluates 14 sentiment-analysis tools on those datasets, and uses the resulting feature and performance tables to build a questionnaire that recommends a sentiment-analysis tool for a new, unlabeled dataset. The main empirical contributions are the linguistic-feature and statistical tables (Tables V and VI), the 14×10 performance table (Table VII), and the platform-matching questionnaire (Section III-E5 and Table VIII). The paper claims that dataset characteristics can be leveraged to improve tool selection, with transformer-based models such as SetFit and RoBERTa performing strongly across datasets.

Significance. If the proposed mapping worked, it would be a practical, low-cost tool-selection aid for practitioners analyzing unlabeled software-engineering text, and the paper would provide a useful cross-platform benchmark of 14 sentiment tools. The dataset characterization is reasonably careful: the authors report Fleiss' kappa and raw agreement for the linguistic coding, make the labeling data available on figshare, and explicitly discuss threats to validity. However, the central claim that the questionnaire improves tool selection is not tested, and the paper's own Table VII suggests that the improvement over an always-SetFit policy would be very small, since SetFit is the best tool on 7 of 10 datasets and within 0.02–0.03 of the best on the other three. The paper also identifies future validation as needed in Section V-D, which is appropriate but leaves the headline contribution unsupported.

major comments (4)
  1. [III-E5 and IV-D] The central claim that the questionnaire improves tool selection is not evaluated against any baseline. The mapping and the per-platform "best tool" recommendations are derived from the same pooled 10-dataset evaluation, so there is no held-out or external check of whether the recommendations generalize. No confidence intervals or significance tests accompany the F1 values, even though many differences are small. Section V-D itself states that "the questionnaire should be validated in practice to assess its utility and accuracy in real-world scenarios." This gap is load-bearing because the abstract and RQ3 claim that dataset characteristics improve tool selection. Concretely, from Table VII, SetFit is the best overall tool on App, Code, GH1, GH3, Jira1, SO1, and SO3, and within 0.02–0.03 of the best on GH2, Jira2, and SO2; an omniscient selector that always picks the true best tool would beat an always-SetFit policy by only 0.007 average overall F1 on these datasets. Please add a validation experiment, e.g., leave-one-dataset-out or simulated questionnaire users, comparing recommended tools against fixed defaults, and report confidence intervals or significance tests.
  2. [Table VII, SO2 row] The reported overall score for SentiSW on SO2 is 0.68, but the reported micro- and macro-averaged F1 are 0.79 and 0.78, whose mean is 0.785. This internal inconsistency appears in the central performance table and must be corrected, and any downstream recommendation or summary derived from that entry should be recomputed.
  3. [III-E5c and Table VII] The manuscript does not specify how a platform-level "best tool" is derived from the per-dataset results in Table VII. Within Jira, Jira1's best overall score is SetFit (0.97) while Jira2's best is ALBERT, ELECTRA, RoBERTa, or XLNet (0.84); within Stack Overflow, SO1 and SO3 prefer SetFit while SO2 prefers RoBERTa. The questionnaire recommends "its empirically best-performing tool" for the matched platform, but the aggregation rule (mean, median, win count, or something else) is not defined. Since RQ3's output depends directly on this rule, it must be stated and justified.
  4. [III-E5 and Table VIII] The matching procedure weights all 13 linguistic features and 8 statistics equally, even though several inputs are non-discriminative: five features (L5, L7, L10, L12, L13) are uniform across all platforms in Table VIII, and statistics such as characters per word are nearly identical across platforms in Table VI. These constant or near-constant inputs can dilute or dominate the platform score. The Likert interval boundaries and the equal feature/statistic weights are free parameters, and no sensitivity analysis is reported. Please state which features and statistics actually separate platforms and test whether the recommendations are robust to dropping non-discriminative inputs or to changing the interval boundaries.
minor comments (4)
  1. [V-B] The text says "Every linguistic feature shows at least a two-digit range between platforms," but Table V reports Inquisitive with a range of only 6.8 percentage points, which is not a two-digit range; please correct this statement.
  2. [III-D] The URL for the scikit-learn classification_report is broken across a line break; please provide a proper citation or a working link.
  3. [III-E5 and Table VIII] The mapping description for "Not specified" and "All" entries is unclear: please clarify whether a "Not specified" answer for a feature with an "All" entry gives a point to every platform, to the ambiguous bucket, or to neither, and state how "Ambiguous" points interact with the final recommendation in a tie.
  4. [VI] The conclusion states that the questionnaire "allows users to describe their dataset and receive evidence-based tool recommendations," but Section V-D defers validation to future work; please qualify the conclusion to reflect that the mapping is currently a proposed, not validated, mechanism.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the recommendation approach is an unvalidated empirical heuristic rather than a self-referential one.

full rationale

The paper's central chain—compute platform-level linguistic and statistical profiles, map a user's questionnaire answers to the closest platform, then output that platform's empirically best tool—is an inductive heuristic, not a derivation that equals its inputs by construction. The platform profiles are defined from the datasets' features, independently of the tool rankings, and the tool rankings come from Table VII, which is computed from external datasets and tools. No equation in the paper defines the recommended tool as a fitted parameter or as the questionnaire answer itself. The fallback recommendation of SetFit and SentiStrength-SE is justified from the same empirical evaluation, not from a self-citation. The paper's many self-citations appear in related work and dataset sourcing, but the load-bearing empirical content is the authors' own evaluation of 14 tools on 10 datasets. The main weakness is external validity: the questionnaire is never tested on held-out data, and the paper explicitly states in Section V-D that 'the questionnaire should be validated in practice to assess its utility and accuracy in real-world scenarios.' That is a missing-validation threat or overfitting risk, not circularity. There is also no baseline comparison to an always-SetFit policy, so the claimed improvement is unquantified; again, this is an empirical gap, not a definitional reduction. Therefore, no circular step is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen Likert intervals, equal feature weights, the reliability of existing labeled datasets, the validity of manually coded features, and the use of platform as a proxy for communication style. No new physical or conceptual entities are introduced.

free parameters (2)
  • Likert interval boundaries for linguistic features = 0-25%, 25-50%, 50-75%, 75-100%
    The four frequency intervals in Table II are chosen by hand and directly determine which platform receives a point in the questionnaire scoring.
  • Equal feature and statistic weights = 1 point per feature or statistic
    Every linguistic feature and every statistic contributes equally to the platform score, with no empirical justification or weighting based on predictive power.
assumptions (4)
  • domain assumption Gold labels in the 10 datasets are sufficiently reliable for evaluating sentiment tools.
    The paper builds on existing polarity and emotion labels, while Section V-C acknowledges that gold labels reflect subjective annotator judgments that may not generalize.
  • domain assumption The 13 manually coded linguistic features capture communication differences relevant to sentiment analysis.
    Feature categories were derived by the authors through collaborative coding (Section III-E2); their relevance to tool performance is asserted, not demonstrated.
  • domain assumption Platform is a valid proxy for communication style.
    The questionnaire maps new datasets to platform profiles (Section III-E5), although the authors note this is a simplification and mention future clustering approaches.
  • domain assumption Emotion-to-polarity mappings from prior heuristics preserve the sentiment signal.
    Section III-B maps emotions to polarity using heuristics from prior work and excludes ambiguous cases, which may alter dataset distributions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards Trustworthy Sentiment Analysis in Software Engineering: Dataset Characteristics and Tool Selection." pith.science (2026). https://pith.science/paper/33A5AAHT

@misc{pith2026250702137,
  author       = {Pith},
  title        = {Pith review of: Towards Trustworthy Sentiment Analysis in Software Engineering: Dataset Characteristics and Tool Selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/33A5AAHT}},
  note         = {Machine review of arXiv:2507.02137}
}
read the original abstract

Software development relies heavily on text-based communication, making sentiment analysis a valuable tool for understanding team dynamics and supporting trustworthy AI-driven analytics in requirements engineering. However, existing sentiment analysis tools often perform inconsistently across datasets from different platforms, due to variations in communication style and content. In this study, we analyze linguistic and statistical features of 10 developer communication datasets from five platforms and evaluate the performance of 14 sentiment analysis tools. Based on these results, we propose a mapping approach and questionnaire that recommends suitable sentiment analysis tools for new datasets, using their characteristic features as input. Our results show that dataset characteristics can be leveraged to improve tool selection, as platforms differ substantially in both linguistic and statistical properties. While transformer-based models such as SetFit and RoBERTa consistently achieve strong results, tool effectiveness remains context-dependent. Our approach supports researchers and practitioners in selecting trustworthy tools for sentiment analysis in software engineering, while highlighting the need for ongoing evaluation as communication contexts evolve.

Figures

Figures reproduced from arXiv: 2507.02137 by the authors.

Figure 1
Figure 1. Overview of the study design. linguistic and statistical characteristics? RQ2: How do sentiment analysis tools perform on communi￾cation data from different SE platforms? RQ3: What is the best sentiment analysis tool for a given, unlabeled communication dataset based on its characteristics? B. Data We selected sentiment analysis datasets from a recent sys￾tematic literature review [3], focusing on datasets with pola… view at source ↗
Figure 2
Figure 2. Steps of linguistic feature coding and platform analysis. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Automatic Generation of Explainability Requirements and Software Explanations From User Reviews

    cs.SE 2025-07 conditional novelty 6.0 of 10

    A ChatGPT-based pipeline can draft explainability requirements and explanations from app reviews, but engineers still prefer manually written requirements and users find AI explanations stylish yet less correct.

Reference graph

Works this paper leans on

51 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Development and application of sentiment analysis tools in software engineering: A systematic literature review,

    M. Obaidi and J. Kl ¨under, “Development and application of sentiment analysis tools in software engineering: A systematic literature review,” in International Conference on Evaluation and Assessment in Software Engineering, ACM. New York, NY , USA: Association for Computing Machinery, 2021

    work page 2021

  2. [2]

    Sentiment analysis tools in software engineering: A systematic mapping study,

    M. Obaidi, L. Nagel, A. Specht, and J. Kl ¨under, “Sentiment analysis tools in software engineering: A systematic mapping study,” Information and Software Technology , vol. 151, p. 107018, 2022

    2022

  3. [3]

    Opinion mining for software development: A systematic literature review,

    B. Lin, N. Cassee, A. Serebrenik, G. Bavota, N. Novielli, and M. Lanza, “Opinion mining for software development: A systematic literature review,” ACM Trans. Softw. Eng. Methodol. , vol. 31, no. 3, mar 2022

    work page 2022

  4. [4]

    Dataset: Systematic mapping study on the development and application of sentiment analysis tools in software engineering,

    M. Obaidi, L. Nagel, A. Specht, and J. Kl ¨under, “Dataset: Systematic mapping study on the development and application of sentiment analysis tools in software engineering,” Mar. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.4726650

    work page doi:10.5281/zenodo.4726650 2022

  5. [5]

    A gold standard for emotion annotation in stack overflow,

    N. Novielli, F. Calefato, and F. Lanubile, “A gold standard for emotion annotation in stack overflow,” in MSR, ser. MSR ’18. New York, NY , USA: Association for Computing Machinery, 2018

    work page 2018

  6. [6]

    Sentiment polarity detection for software development,

    F. Calefato, F. Lanubile, F. Maiorano, and N. Novielli, “Sentiment polarity detection for software development,” Empirical Software En- gineering, vol. 23, no. 3, 2018

    work page 2018

  7. [7]

    From textual to verbal communication: Towards applying sentiment analysis to a software project meeting,

    M. Herrmann and J. Kl ¨under, “From textual to verbal communication: Towards applying sentiment analysis to a software project meeting,” in REW. Piscataway, NJ, USA: IEEE, 2021

    work page 2021

  8. [8]

    Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text,

    M. R. Islam and M. F. Zibran, “Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text,” JSS, vol. 145, 2018

    work page 2018

Show all 51 references
  1. [9]

    Senticr: A customized sentiment analysis tool for code review interactions,

    T. Ahmed, A. Bosu, A. Iqbal, and S. Rahimi, “Senticr: A customized sentiment analysis tool for code review interactions,” in ASE. IEEE, 2017

    2017

  2. [10]

    Automatic classification of statements in meetings of development teams,

    M. Herrmann, “Automatic classification of statements in meetings of development teams,” Bachelor’s Thesis, Gottfried Wilhelm Leibniz Universit¨at Hannover, March 2021

    2021

  3. [11]

    A german gold-standard dataset for sentiment analysis in software engineering,

    M. Obaidi, M. Herrmann, E. Schmid, R. Ochsner, K. Schneider, and J. Kl ¨under, “A german gold-standard dataset for sentiment analysis in software engineering,” in 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW) , 2025

    2025

  4. [12]

    What is needed to apply sentiment analysis in real software projects: A feasibility study in industry,

    A. Specht, M. Obaidi, L. Nagel, M. Stess, and J. Kl ¨under, “What is needed to apply sentiment analysis in real software projects: A feasibility study in industry,” in Human-Centered Software Engineering . Cham: Springer Nature Switzerland, 2024, pp. 105–129

    2024

  5. [13]

    Towards understanding emotional response to requirements changes in agile teams,

    K. Madampe, R. Hoda, and P. Singh, “Towards understanding emotional response to requirements changes in agile teams,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: New Ideas and Emerging Results , ser. ICSE-NIER ’20. New York, NY , USA: ...

    2020

  6. [14]

    Can we use se-specific sentiment analysis tools in a cross-platform setting?

    N. Novielli, F. Calefato, D. Dongiovanni, D. Girardi, and F. Lanubile, “Can we use se-specific sentiment analysis tools in a cross-platform setting?” in Proceedings of the 17th International Conference on Mining Software Repositories, ser. MSR ’20. New York, NY , USA: Associat...

    2020

  7. [15]

    On the lim- itations of combining sentiment analysis tools in a cross-platform setting,

    M. Obaidi, H. Holm, K. Schneider, and J. Kl ¨under, “On the lim- itations of combining sentiment analysis tools in a cross-platform setting,” in Product-F ocused Software Process Improvement , D. Taibi, M. Kuhrmann, T. Mikkonen, J. Kl ¨under, and P. Abrahamsson, Eds. Cham: Spr...

    2022

  8. [16]

    Sentiment analysis for software engineering: How far can pre-trained transformer models go?

    T. Zhang, B. Xu, F. Thung, S. A. Haryono, D. Lo, and L. Jiang, “Sentiment analysis for software engineering: How far can pre-trained transformer models go?” in ICSME, 2020

    2020

  9. [17]

    Sentiment analysis for software engineering: How far can we go?

    B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, and R. Oliveto, “Sentiment analysis for software engineering: How far can we go?” in Proceedings of the 40th International Conference on Software Engineer- ing, ser. ICSE ’18. New York, NY , USA: Association for Computing ...

    2018

  10. [18]

    An empirical study of the effectiveness of an ensemble of stand-alone sentiment detection tools for software engineering datasets,

    G. Uddin, Y .-G. Gu ´eh´enuc, F. Khomh, and C. K. Roy, “An empirical study of the effectiveness of an ensemble of stand-alone sentiment detection tools for software engineering datasets,” ACM Trans. Softw. Eng. Methodol. , vol. 31, no. 3, apr 2022

    2022

  11. [19]

    Classifying emotions in stack overflow and jira using a multi-label approach,

    L. A. Cabrera-Diego, N. Bessis, and I. Korkontzelos, “Classifying emotions in stack overflow and jira using a multi-label approach,” Knowledge-Based Systems , vol. 195, 2020

    2020

  12. [20]

    Parallel worlds: Repeated initializations of the same team to improve team viability,

    M. E. Whiting, I. Gao, M. Xing, N. J. Diarrassouba, T. Nguyen, and M. S. Bernstein, “Parallel worlds: Repeated initializations of the same team to improve team viability,” Proc. ACM Hum.-Comput. Interact. , vol. 4, no. CSCW1, May 2020

    2020

  13. [21]

    Mostafa and M

    L. Mostafa and M. Abd Elghany, INVESTIGATING GAME DEVELOP- ERS’ GUILT EMOTIONS USING SENTIMENT ANALYSIS . IJSEA, 2018, vol. 9

    2018

  14. [22]

    Automatic mining of opinions expressed about apis in stack overflow,

    G. Uddin and F. Khomh, “Automatic mining of opinions expressed about apis in stack overflow,” IEEE Transactions on Software Engineering , vol. 47, no. 3, 2021

    2021

  15. [23]

    A gold standard for polarity of emotions of software developers in github,

    N. Novielli, F. Calefato, D. Dongiovanni, D. Girardi, and F. Lanubile, “A gold standard for polarity of emotions of software developers in github,” 3 2020

    2020

  16. [24]

    Entity-level sentiment analysis of issue comments,

    J. Ding, H. Sun, X. Wang, and X. Liu, “Entity-level sentiment analysis of issue comments,” in Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering , ser. SEmotion ’18. New York, NY , USA: Association for Computing Machinery, 2018

    2018

  17. [25]

    Sentiment and politeness analysis tools on developer discussions are unreliable, but so are people,

    N. Imtiaz, J. Middleton, P. Girouard, and E. Murphy-Hill, “Sentiment and politeness analysis tools on developer discussions are unreliable, but so are people,” in Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering , ser. SEmotion ’18. Ne...

    2018

  18. [26]

    The emotional side of software developers in jira,

    M. Ortu, A. Murgia, G. Destefanis, P. Tourani, R. Tonelli, M. Marchesi, and B. Adams, “The emotional side of software developers in jira,” in Proceedings of the 13th International Conference on Mining Software Repositories, ser. MSR ’16. New York, NY , USA: Association for Com...

    2016

  19. [27]

    Deva: sensing emotions in the valence arousal space in software engineering text,

    M. R. Islam and M. F. Zibran, “Deva: sensing emotions in the valence arousal space in software engineering text,” in Applied computing 2018 , H. M. Haddad, R. L. Wainwright, and R. Chbeir, Eds. New York, NY: Association for Computing Machinery Inc. (ACM), 2018

    2018

  20. [28]

    On the sub- jectivity of emotions in software projects: How reliable are pre-labeled data sets for sentiment analysis?

    M. Herrmann, M. Obaidi, L. Chazette, and J. Kl ¨under, “On the sub- jectivity of emotions in software projects: How reliable are pre-labeled data sets for sentiment analysis?” JSS, vol. 193, 2022

    2022

  21. [29]

    On the potentials of realtime sentiment analysis on text-based communication in software projects,

    L. Schroth, M. Obaidi, A. Specht, and J. Kl ¨under, “On the potentials of realtime sentiment analysis on text-based communication in software projects,” in Human-Centered Software Engineering . Cham: Springer International Publishing, 2022

    2022

  22. [30]

    Sentiment strength detec- tion for the social web,

    M. Thelwall, K. Buckley, and G. Paltoglou, “Sentiment strength detec- tion for the social web,” Journal of the American Society for Information Science and Technology , vol. 63, no. 1, 2012

    2012

  23. [31]

    Achiev- ing reliable sentiment analysis in the software engineering domain using bert,

    E. Biswas, M. E. Karabulut, L. Pollock, and K. Vijay-Shanker, “Achiev- ing reliable sentiment analysis in the software engineering domain using bert,” in ICSME, 2020

    2020

  24. [32]

    Sentimoji: An emoji- powered learning approach for sentiment analysis in software engineer- ing,

    Z. Chen, Y . Cao, X. Lu, Q. Mei, and X. Liu, “Sentimoji: An emoji- powered learning approach for sentiment analysis in software engineer- ing,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the F oundations of So...

    2019

  25. [33]

    Senti-analyzer: Joint sen- timent analysis for text-based and verbal communication in software projects,

    M. Herrmann, M. Obaidi, and J. Kl ¨under, “Senti-analyzer: Joint sen- timent analysis for text-based and verbal communication in software projects,” Leibniz University Hannover, Tech. Rep., 2022

    2022

  26. [34]

    Revisiting sentiment analysis for software engineering in the era of large language models,

    T. Zhang, I. C. Irsan, F. Thung, and D. Lo, “Revisiting sentiment analysis for software engineering in the era of large language models,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 3, Feb. 2025

    2025

  27. [35]

    Different and similar percep- tions of communication among software developers,

    M. Herrmann, M. Obaidi, and J. Kl ¨under, “Different and similar percep- tions of communication among software developers,” Information and Software Technology, vol. 181, p. 107698, 2025

    2025

  28. [36]

    Modeling communica- tion perception in development teams using monte carlo methods,

    M. Herrmann, M. Obaidi, and J. Kl ¨under, “Modeling communica- tion perception in development teams using monte carlo methods,” in Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering , ser. EASE ’25. Association for Computing M...

    2025

  29. [37]

    Dataset: Sentisurvey for sentiment analysis in software projects,

    M. Obaidi, M. Herrmann, L. Chazette, and J. Kl ¨under, “Dataset: Sentisurvey for sentiment analysis in software projects,” Jun. 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6611728

    2022 doi

  30. [38]

    Do users’ explainability needs in software change with mood?

    M. Obaidi, J. Droste, H. Deters, M. Herrmann, J. Kl ¨under, and K. Schneider, “Do users’ explainability needs in software change with mood?” in Requirements Engineering: F oundation for Software Quality: 31st International Working Conference . Springer Nature Switzerland, 2025

    2025

  31. [39]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019

    2019

  32. [40]

    Efficient few-shot learning without prompts,

    L. Tunstall, N. Reimers, U. E. S. Jo, L. Bates, D. Korat, M. Wasserblat, and O. Pereg, “Efficient few-shot learning without prompts,” 2022

    2022

  33. [41]

    Identifying the mood of a software development team by analyzing text-based communication in chats with machine learning,

    J. Kl ¨under, J. Horstmann, and O. Karras, “Identifying the mood of a software development team by analyzing text-based communication in chats with machine learning,” in Human-Centered Software Engineer- ing, R. Bernhaupt, C. Ardito, and S. Sauer, Eds. Cham: Springer Internati...

    2020

  34. [42]

    On the use of emoticons in open source software development,

    M. Claes, M. M ¨antyl¨a, and U. Farooq, “On the use of emoticons in open source software development,” in Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ser. ACM Digital Library, M. Oivo, Ed. New York, NY: ACM, 2018

    2018

  35. [43]

    Measuring nominal scale agreement among many raters

    J. L. Fleiss, “Measuring nominal scale agreement among many raters.” Psychological Bulletin, vol. 76, no. 5, 1971

    1971

  36. [44]

    The measurement of observer agreement for categorical data

    J. Landis and G. Koch, “The measurement of observer agreement for categorical data.” Biometrics, vol. 33 1, 1977

    1977

  37. [45]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, and A. Wessl´en, Experimentation in software engineering . Springer, 2012

    2012

  38. [46]

    Are bullies more produc- tive? empirical study of affectiveness vs. issue fixing time,

    Marco Ortuand Bram Adams, Giuseppe Destefanis, Parastou Tourani, Michele Marchesi, and Roberto Tonelli, “Are bullies more produc- tive? empirical study of affectiveness vs. issue fixing time,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories , 2015

    2015

  39. [47]

    Do developers feel emotions? an exploratory analysis of emotions in software artifacts,

    A. Murgia, P. Tourani, B. Adams, and M. Ortu, “Do developers feel emotions? an exploratory analysis of emotions in software artifacts,” in MSR’14, ser. MSR 2014. New York, NY , USA: Association for Computing Machinery, 2014

    2014

  40. [48]

    Exploiting the unique expression for improved sentiment analysis in software engineering text,

    K. Sun, H. Gao, H. Kuang, X. Ma, G. Rong, D. Shao, and H. Zhang, “Exploiting the unique expression for improved sentiment analysis in software engineering text,” in 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC) , 2021, pp. 149–159

    2021

  41. [49]

    Emotion knowledge: Further exploration of a prototype approach,

    P. R. Shaver, J. C. Schwartz, D. Kirson, and C. O’Connor, “Emotion knowledge: Further exploration of a prototype approach,” Journal of Personality and Social Psychology , vol. 52, no. 6, pp. 1061–1086, 1987

    1987

  42. [50]

    W. G. Parrott, Emotions in social psychology: Essential readings . psychology press, 2001

    2001

  43. [51]

    Dataset: Towards Trustworthy Sentiment Analysis in Software Engineering: Dataset Characteristics and Tool Selection,

    M. Obaidi, M. Herrmann, J. Kl ¨under, and K. Schneider, “Dataset: Towards Trustworthy Sentiment Analysis in Software Engineering: Dataset Characteristics and Tool Selection,” 7 2025. [Online]. Available: https://doi.org/10.6084/m9.figshare.29250935.v1

    2025 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.