Pith. sign in

REVIEW 4 major objections 5 minor 62 references

A German Gold-Standard Dataset for Sentiment Analysis in Software Engineering

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces the first German-language gold-standard dataset for sentiment analysis in software engineering, built from 5,949 forum statements.

desk verdict First German SE-specific sentiment gold-standard, but the sampling is GerVADER-conditioned and the representativeness claim overreaches. read the letter →

arxiv 2507.07325 v1 pith:UELQ4EDE submitted 2025-07-09 cs.SE

Martin Obaidi , Marc Herrmann , Elisa Schmid , Raymond Ochsner , Kurt Schneider , Jil Klünder This is my paper · ORCID
classification cs.SE
keywords Germansentimentanalysisgold-standarddatasetsoftwareengineeringemotionannotationdevelopercommunicationinterraterreliabilityAndroid-Hilfeforum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sentiment analysis of developer communication is mostly built on English gold standards, leaving German-speaking software teams without a trusted benchmark. This paper builds a German-language gold-standard dataset by crawling 20,380 statements from the Android-Hilfe.de forum, pre-balancing them with a sentiment tool, and having three raters label 5,949 unique statements with one of six basic emotions. The authors report high interrater agreement in the final annotation round and argue that the dataset is valid and reliable enough to support German software-engineering sentiment research. They also benchmark four existing German sentiment tools against the dataset and find that none performs strongly, with SentiStrength DE reaching the best accuracy (0.72) and BertDE the worst (0.36). The paper's central claim is that this dataset can serve as the missing German foundation for training and evaluating domain-specific sentiment tools.

What carries the argument

The load-bearing object is the annotated dataset itself: 5,949 statements, each carrying a single label from a six-emotion hierarchical model, with those labels mappable to polarity (Love, Joy, and Positive Surprise to Positive; Negative Surprise, Anger, Sadness, and Fear to Negative). The dataset is produced by a pipeline whose steps all matter: a crawler extracts 20,380 German statements from Android-Hilfe.de; GerVADER pre-sorts them so that the 2,000 most positive, 2,000 most negative, and 2,000 most neutral statements are selected; and a workshop of five German-speaking computer science students annotates the subset, with an intermediate consensus round after the first 100 statements per rater. The intermediate discussion is a deliberate mechanism: the final-round multi-rater $\kappa$ for emotion labels rises from 0.50 to 0.71, and the $\kappa$ for Love rises from 0.04 to 0.85. This pipeline is what lets the paper claim the resulting labels are reliable enough to serve as a benchmark.

What would settle it

Re-annotate a random sample of, say, 300 statements drawn from the full 20,380 without any sentiment pre-sorting, using the same three-rater procedure; if the neutral share or the tool accuracy figures differ substantially from the published 69.78% and 0.72, the pre-sorting assumption is falsified.

Watch

Extended reading notes

Core claim

The central claim is that 5,949 unique German developer statements, each labeled by three raters into one of six basic emotions from a hierarchical emotion model, constitute a valid gold standard for sentiment analysis in software engineering. The final annotation round reached a multi-rater $\kappa$ of 0.71 for the six emotion labels and 0.73 when labels are mapped to polarity, with an overall raw agreement of 0.81; the authors interpret these values as comparable to existing English software-engineering gold standards. The dataset is deliberately imbalanced: 69.78% of statements are neutral, 21.36% positive, and 8.85% negative. When four German sentiment tools are scored against the human labels, none reaches acceptable performance, and the authors take this as evidence that no existing German tool is adequate for the software-engineering domain and that a domain-specific German model is needed.

Load-bearing premise

The evaluation stands on the assumption that GerVADER's selection of the 2,000 most positive, 2,000 most neutral, and 2,000 most negative statements from the 20,380 crawled posts yields a corpus whose labels and tool scores represent German developer communication; if that pre-sorting skews the sample, the 69.78% neutral share and all tool accuracy figures fail to generalize.

Editorial extensions

If this is right

  • The dataset can be used as training data for a German machine-learning sentiment classifier, and the paper suggests BertDE could improve substantially if fine-tuned on it.
  • Benchmarking results imply that SentiStrength DE is currently the best available German tool for software-engineering text, yet its accuracy of 0.72 and low $\kappa$ still fall short of what should be trusted.
  • The 69.78% neutral share implies that German developer forum communication is mostly free of explicit emotion, so tools for this domain should not over-trigger on routine technical statements.
  • The large agreement gain after the intermediate consensus round supports building such rounds into future annotation campaigns.
  • Because the dataset is published openly, it gives the German-speaking software-engineering community a reusable resource for training and benchmarking, filling a gap that previously had only English counterparts.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: if the GerVADER pre-sorting is biased, the neutral-heavy distribution may reflect the selection procedure rather than German developer communication, so a random sample from the full 20,380 statements should be annotated to test representativeness.
  • Our inference: the same corpus could support a test of whether large-language-model filtering matches human emotion judgments on German software-engineering text, though the paper only names this as future work.
  • Our inference: because all raters were male computer science students aged 20 to 25, the emotion labels may carry a cohort-specific reading; re-annotation by a more diverse rater pool would test how much of the gold standard is rater-dependent.
  • Our inference: the dataset's low negative-class F1 scores across tools suggest that any German software-engineering sentiment model trained on this corpus will need extra negative examples, since the current balance underrepresents anger and fear.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a German-language sentiment-analysis dataset for software engineering, constructed from 20,380 statements crawled from the Android-Hilfe.de forum. The authors pre-select 2,000 statements per polarity class using GerVADER, yielding 6,000 statements, which were labeled by five German-speaking computer science students (three raters per statement after one rater withdrew) using a six-emotion model based on Shaver et al., plus a neutral class. The final dataset contains 5,949 statements. The paper reports inter-rater agreement before and after an intermediate discussion round, evaluates the dataset with four German sentiment-analysis tools (GerVADER, SentiStrength DE, TextBlobDE, BertDE), and concludes that the dataset is a valid and robust gold standard and that existing German tools are inadequate for the software-engineering domain.

Significance. If the representativeness and validity claims are properly supported, this would be a useful resource: it is, to my knowledge, the first German-language SE sentiment dataset, it is openly available on Zenodo, the annotation process is transparently described, and the multi-rater design with an intermediate discussion is a methodological strength. The comparison of four German tools provides a reproducible baseline. However, the central claim of validity and robustness is weakened by the GerVADER-conditioned sampling procedure and by the low per-class inter-rater reliability for several emotion classes; these issues need to be addressed before the dataset can be recommended as a general-purpose gold standard.

major comments (4)
  1. [III-A3, IV-B, Fig. 1] The dataset is not a representative sample of German developer communication. The construction ranks all 20,380 crawled statements by GerVADER polarity and keeps the 2,000 most extreme statements in each polarity class, which forces a 33/33/33 GerVADER split. The human labels in Table I are 69.78% neutral, 21.36% positive, and 8.85% negative, meaning that many statements that GerVADER scored as strongly positive or strongly negative were judged neutral by human raters. The final dataset is therefore conditioned on GerVADER's scoring function, not on the distribution of developer communication. This directly undermines the abstract's claim that the dataset is 'sufficiently valid and robust to support sentiment analysis in the German-speaking software engineering community.' Please either temper the claim to describe a reusable, stratified benchmark for tool development, or add an explicit external-validity discussion of the selection effect and its consequences.
  2. [V-C] The threats-to-validity section discusses rater demographics, reliance on English-language comparisons, and the data source, but never lists GerVADER pre-filtering as a threat to external validity or representativeness. This omission is load-bearing: the gap between the construction procedure and the general validity claim is exactly the pre-selection step. The section should be revised to acknowledge that all downstream statistics--including the class distribution, per-class F1 values in Table IV, and Cohen's kappa values in Table V--describe performance on GerVADER's extreme-score regions, not on typical forum posts.
  3. [IV-A2, Table II] The claim of 'high interrater agreement and reliability' is overstated for the emotion-level labels. In the final round, Fleiss' kappa is 0.47 for Joy, 0.28 for Positive Surprise, 0.10 for Negative Surprise, 0.37 for Anger, and 0.34 for Fear; only Neutral, Love, and Sadness reach values above 0.6. The high percentage agreement is driven largely by the dominant Neutral class. The abstract and Section V-A should report these per-class values and qualify the reliability claim accordingly, rather than relying on the overall Fleiss' kappa of 0.71.
  4. [IV-C, Tables IV and V] Evaluating GerVADER on a dataset that was pre-selected by GerVADER introduces a methodological circularity. Although the human labels are independent of GerVADER, the sample is enriched for statements on which GerVADER is confident, so the reported accuracy, F1-scores, and Cohen's kappa values do not estimate performance on a random sample of German SE text. This should be acknowledged explicitly, and the tool-comparison conclusions should be framed as applying to the extreme-stratified sample rather than to German developer communication in general.
minor comments (5)
  1. [V-A] The sentence 'this work succeeded in achieving the goal of create a German gold-standard dataset' contains a grammatical error; it should read 'the goal of creating.'
  2. [III-B2] The sentence 'while three others, were writing their theses at the time of the workshop' contains an errant comma after 'others.'
  3. [Table II] The row 'Diff. Fleiss' K' leaves the Positive Surprise cell empty because Round 1 did not distinguish surprise polarities; this should be stated explicitly in the table caption or a note to avoid confusion.
  4. [III-A3] The example 'sent from my iPhone XR' is presented as translated to English, but the quoted text is already in English; please clarify whether the original German signature was translated for the paper.
  5. [III-B1] The phrase 'to agree on a single emotion' may be misread as requiring consensus among raters; it would be clearer to say 'to select a single emotion' for the final label.

Circularity Check

1 steps flagged · score 4.0 of 10

GerVADER conditions the sample it is later evaluated on; human-labeled dataset core remains independent.

  1. other [Section III-A3 (Dataset Composition); Section IV-C2 (Tool Results), Tables IV-V]
    "To ensure a balanced dataset, we performed preliminary sorting using the sentiment analysis tool GerVADER [49] ... Subsequently, the 2,000 statements with the highest scores for each of the three polarities were selected and merged to form a dataset of 6,000 statements. ... For the evaluation with sentiment analysis tools, the lexicon-based GerVADER [49] ... were chosen."

    The evaluation set is defined by GerVADER's own polarity scores: it is the union of the 2,000 statements with the most extreme GerVADER scores in each polarity class. The same GerVADER tool is then scored against human labels on that set (Tables IV and V). GerVADER's reported accuracy, macro-F1, and Cohen's kappa therefore describe the tool on its own selected extremes, not on representative German SE developer statements; by construction, the test distribution is GerVADER-conditioned. The human labels are independent, so the dataset-validity claim does not collapse, but the 'evaluation with existing German sentiment analysis tools confirms the lack of domain-specific solutions' conclusion is partly self-referential for the tool that shaped the sample.

full rationale

The core deliverable is a 5,949-statement human-labeled German SE sentiment dataset. Labels come from student raters using a Shaver et al. emotion model, with majority voting and interrater-reliability metrics; this part is independent of sentiment tools and is the main evidence for dataset validity. The circularity is confined to the tool-evaluation pass: Section III-A3 constructs the dataset by selecting the top 2,000 statements per GerVADER polarity, and Section IV-C then reports GerVADER's accuracy, macro-F1, and Cohen's kappa on that same GerVADER-conditioned sample (Tables IV and V). This makes GerVADER's reported performance a statistic over the extreme tails of its own scoring function, not a benchmark on representative developer communication. The threats-to-validity section does not list this pre-filtering as a limitation. No load-bearing self-citation chain is present: the authors' prior work is used for background and comparison, not to justify the dataset's construction, and no uniqueness or ansatz is imported from the authors' own papers. Given that the human-label construction is independent, the central dataset claim survives, but the GerVADER evaluation contribution is partially self-referential.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The dataset's validity and the tool comparisons rest on several unverified domain assumptions: the representativeness of one Android forum, the suitability of the Shaver emotion model for SE text, the Neutral default rule, and the use of GerVADER to pre-select the sample. None of these is independently validated, and per-class interrater reliability is low for several emotions.

free parameters (3)
  • per-polarity sample size = 2,000
    The 2,000 statements with the highest GerVADER scores in each polarity were selected to form a 6,000-statement balanced pool (Section III-A3). This number is a design choice, not derived from a requirement.
  • post length cutoff = 200 characters
    The crawler excluded posts longer than 200 characters (Section III-A2). This arbitrary threshold shapes the corpus and is not justified by analysis.
  • initial calibration sample size = 20 sample statements and 100 initial labels per rater
    The workshop used 20 sample statements for guideline training and 100 initial statements per rater for agreement checks (Section III-B3). These sizes are ad hoc choices.
assumptions (6)
  • domain assumption Android-Hilfe.de developer statements are representative of German software engineering communication.
    The dataset is drawn from a single Android development forum; no evidence shows transfer to other German SE contexts (Section III-A1).
  • domain assumption The Shaver et al. emotion model plus Surprise is appropriate for coding developer statements.
    Annotation guideline is based on this model (Section III-B1); the model was developed for general emotion, not SE-specific text.
  • domain assumption Statements with no identifiable emotion should be labeled Neutral.
    This rule drives the 69.78% neutral class and affects all downstream metrics (Section III-B1).
  • domain assumption GerVADER scores provide a meaningful basis for pre-sorting and balancing.
    The dataset sample depends on GerVADER's polarity ranking; if GerVADER is inaccurate in this domain, the sample is biased (Section III-A3).
  • standard math Fleiss' Kappa and Cohen's Kappa are appropriate agreement measures for this annotation design.
    Used to quantify interrater reliability (Section III-C2); standard tools but sensitive to class imbalance.
  • domain assumption Majority voting with discussion among three raters yields valid ground-truth labels.
    Final labels were determined after group discussion to resolve 1,205 disagreements (Section IV-A2); no external validation of these consensus labels is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A German Gold-Standard Dataset for Sentiment Analysis in Software Engineering." pith.science (2026). https://pith.science/paper/UELQ4EDE

@misc{pith2026250707325,
  author       = {Pith},
  title        = {Pith review of: A German Gold-Standard Dataset for Sentiment Analysis in Software Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UELQ4EDE}},
  note         = {Machine review of arXiv:2507.07325}
}
read the original abstract

Sentiment analysis is an essential technique for investigating the emotional climate within developer teams, contributing to both team productivity and project success. Existing sentiment analysis tools in software engineering primarily rely on English or non-German gold-standard datasets. To address this gap, our work introduces a German dataset of 5,949 unique developer statements, extracted from the German developer forum Android-Hilfe.de. Each statement was annotated with one of six basic emotions, based on the emotion model by Shaver et al., by four German-speaking computer science students. Evaluation of the annotation process showed high interrater agreement and reliability. These results indicate that the dataset is sufficiently valid and robust to support sentiment analysis in the German-speaking software engineering community. Evaluation with existing German sentiment analysis tools confirms the lack of domain-specific solutions for software engineering. We also discuss approaches to optimize annotation and present further use cases for the dataset.

Figures

Figures reproduced from arXiv: 2507.07325 by the authors.

Figure 1
Figure 1. Composition of the dataset at different phases [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 55 canonical work pages

  1. [1]

    Emotion knowledge: further exploration of a prototype approach

    P. R. Shaver, J. C. Schwartz, D. Kirson, and C. O’Connor, “Emotion knowledge: further exploration of a prototype approach.” Journal of personality and social psychology , vol. 52 6, 1987

    work page 1987

  2. [2]

    The influence of sad mood on cognition

    L. G. Chepenik, L. A. Cornew, and M. J. Farah, “The influence of sad mood on cognition.” Emotion, vol. 7, no. 4, p. 802, 2007

    work page 2007

  3. [3]

    Different and similar percep- tions of communication among software developers,

    M. Herrmann, M. Obaidi, and J. Kl ¨under, “Different and similar percep- tions of communication among software developers,” Information and Software Technology, vol. 181, p. 107698, 2025

    work page 2025

  4. [4]

    Happy software devel- opers solve problems better: psychological measurements in empirical software engineering,

    D. Graziotin, X. Wang, and P. Abrahamsson, “Happy software devel- opers solve problems better: psychological measurements in empirical software engineering,” PeerJ, vol. 2, 2014

    work page 2014

  5. [5]

    Automating developer chat mining,

    S. Pan, L. Bao, X. Ren, X. Xia, D. Lo, and S. Li, “Automating developer chat mining,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) , 2021, pp. 854–866

    work page 2021

  6. [6]

    An empirical study of developer discussions in the gitter platform,

    O. Ehsan, S. Hassan, M. E. Mezouar, and Y . Zou, “An empirical study of developer discussions in the gitter platform,” ACM Trans. Softw. Eng. Methodol., vol. 30, no. 1, Dec. 2021

    work page 2021

  7. [7]

    Modeling communication perception in development teams using monte carlo methods,

    M. Herrmann, M. Obaidi, and J. Kl ¨under, “Modeling communication perception in development teams using monte carlo methods,” in EASE ’25’. Association for Computing Machinery, 2025

    work page 2025

  8. [8]

    From textual to verbal communication: Towards applying sentiment analysis to a software project meeting,

    M. Herrmann and J. Kl ¨under, “From textual to verbal communication: Towards applying sentiment analysis to a software project meeting,” in REW. IEEE, 2021

    work page 2021

Show all 62 references
  1. [9]

    Automatic classification of statements in meetings of development teams,

    M. Herrmann, “Automatic classification of statements in meetings of development teams,” Bachelor’s Thesis, Gottfried Wilhelm Leibniz Universit¨at Hannover, March 2021

    2021

  2. [10]

    Characterising developer sentiment in software components: An exploratory study of gentoo,

    T. R. Tulili, A. Rastogi, and A. Capiluppi, “Characterising developer sentiment in software components: An exploratory study of gentoo,”

  3. [11]

    Analyzing the developer’s sentiment in software components: A decade-long study of the apache project,

    ——, “Analyzing the developer’s sentiment in software components: A decade-long study of the apache project,” in HUCAPP, 2025

    2025

  4. [12]

    Estimating trust in virtual teams - a framework based on sentiment analysis,

    G. A. M. da Cruz, E. H. M. Huzita, and V . D. Feltrim, “Estimating trust in virtual teams - a framework based on sentiment analysis,” in ICEIS, INSTICC. SciTePress, 2016, pp. 464–471

    2016

  5. [13]

    ”how was your weekend?

    C. Miller, P. Rodeghero, M.-A. Storey, D. Ford, and T. Zimmermann, “”how was your weekend?” software development teams working from home during covid-19,” in ICSE ’21 , 2021, pp. 624–636

    2021

  6. [14]

    Do users’ explainability needs in software change with mood?

    M. Obaidi, J. Droste, H. Deters, M. Herrmann, J. Kl ¨under, and K. Schneider, “Do users’ explainability needs in software change with mood?” in REFSQ’25, 2025

    2025

  7. [15]

    Sentiment analysis of github commit comments using machine learning techniques,

    S. Patel, G. Zagade, and R. Sharma, “Sentiment analysis of github commit comments using machine learning techniques,” in IDICAIEI, 2024, pp. 1–6

    2024

  8. [16]

    Sentiment analysis for software engineering: How far can we go?

    B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, and R. Oliveto, “Sentiment analysis for software engineering: How far can we go?” in ICSE ’18 . ACM, 2018

    2018

  9. [17]

    Revisiting sentiment analysis for software engineering in the era of large language models,

    T. Zhang, I. C. Irsan, F. Thung, and D. Lo, “Revisiting sentiment analysis for software engineering in the era of large language models,” ACM Trans. Softw. Eng. Methodol. , vol. 34, no. 3, Feb. 2025

    2025

  10. [18]

    Senti-analyzer: Joint sen- timent analysis for text-based and verbal communication in software projects,

    M. Herrmann, M. Obaidi, and J. Kl ¨under, “Senti-analyzer: Joint sen- timent analysis for text-based and verbal communication in software projects,” Leibniz University Hannover, Tech. Rep., 2022

    2022

  11. [19]

    Dataset: On the Subjectivity of Emotions in Software Projects: How Reliable are Pre-Labeled Data Sets for Sentiment Analysis?

    M. Obaidi, M. Herrmann, L. Chazette, and J. Kl ¨under, “Dataset: On the Subjectivity of Emotions in Software Projects: How Reliable are Pre-Labeled Data Sets for Sentiment Analysis?” Jun. 2022

    2022

  12. [20]

    Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text,

    M. Islam and M. Zibran, “Sentistrength-se: Exploiting domain specificity for improved sentiment analysis in software engineering text,” Journal of Systems and Software , vol. 145, pp. 125–146, 2018

    2018

  13. [21]

    On the sub- jectivity of emotions in software projects: How reliable are pre-labeled data sets for sentiment analysis?

    M. Herrmann, M. Obaidi, L. Chazette, and J. Kl ¨under, “On the sub- jectivity of emotions in software projects: How reliable are pre-labeled data sets for sentiment analysis?” JSS, vol. 193, 2022

    2022

  14. [22]

    Sentiment polarity detection for software development,

    F. Calefato, F. Lanubile, F. Maiorano, and N. Novielli, “Sentiment polarity detection for software development,” EMSE, vol. 23, 2018

    2018

  15. [23]

    Development and application of sentiment analysis tools in software engineering: A systematic literature review,

    M. Obaidi and J. Kl ¨under, “Development and application of sentiment analysis tools in software engineering: A systematic literature review,” in EASE ’21’ . ACM, 2021

    2021

  16. [24]

    Sentiment analysis tools in software engineering: A systematic mapping study,

    M. Obaidi, L. Nagel, A. Specht, and J. Kl ¨under, “Sentiment analysis tools in software engineering: A systematic mapping study,” Information and Software Technology , vol. 151, p. 107018, 2022

    2022

  17. [25]

    A Twitter corpus and benchmark resources for German sentiment analysis,

    M. Cieliebak, J. M. Deriu, D. Egger, and F. Uzdilli, “A Twitter corpus and benchmark resources for German sentiment analysis,” in Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media , Apr. 2017, pp. 45–51

    2017

  18. [26]

    Fine-grained german sentiment analysis on social media,

    S. Momtazi, “Fine-grained german sentiment analysis on social media,” in LREC’12, 2012, pp. 1215–1220

    2012

  19. [27]

    Towards trust- worthy sentiment analysis in software engineering: Dataset characteris- tics and tool selection,

    M. Obaidi, M. Herrmann, J. Kl ¨under, and K. Schneider, “Towards trust- worthy sentiment analysis in software engineering: Dataset characteris- tics and tool selection,” in 2025 IEEE 33rd International Requirements Engineering Conference Workshops (REW) , 2025

    2025

  20. [28]

    On the limitations of combining sentiment analysis tools in a cross-platform setting,

    M. Obaidi, H. Holm, K. Schneider, and J. Kl ¨under, “On the limitations of combining sentiment analysis tools in a cross-platform setting,” in PROFES. Cham: Springer International Publishing, 2022

    2022

  21. [29]

    Can we use se-specific sentiment analysis tools in a cross-platform setting?

    N. Novielli, F. Calefato, D. Dongiovanni, D. Girardi, and F. Lanubile, “Can we use se-specific sentiment analysis tools in a cross-platform setting?” in MSR ’20 . ACM, 2020

    2020

  22. [30]

    On the potentials of realtime sentiment analysis on text-based communication in software projects,

    L. Schroth, M. Obaidi, A. Specht, and J. Kl ¨under, “On the potentials of realtime sentiment analysis on text-based communication in software projects,” in HCSE. Springer International Publishing, 2022

    2022

  23. [31]

    Automating explanation need management in app reviews: A case study from the navigation app industry,

    M. Obaidi, N. V oß, J. Droste, H. Deters, M. Herrmann, J. Fischbach, and K. Schneider, “Automating explanation need management in app reviews: A case study from the navigation app industry,” in ICSE- SEIP’25, 2025

    2025

  24. [32]

    From app features to explanation needs: Analyzing correlations and predictive potential,

    M. Obaidi, K. Qengaj, J. Droste, H. Deters, M. Herrmann, E. Schmid, J. Kl ¨under, and K. Schneider, “From app features to explanation needs: Analyzing correlations and predictive potential,” in 33rd International Requirements Engineering Conference Workshops (REW) , 2025

    2025

  25. [33]

    How does users’ app knowledge influence the preferred level of detail and format of software explanations?

    M. Obaidi, J. Fischbach, M. Herrmann, H. Deters, J. Droste, J. Kl ¨under, and K. Schneider, “How does users’ app knowledge influence the preferred level of detail and format of software explanations?” in REFSQ’25, 2025

    2025

  26. [34]

    A study on the men- tal models of users concerning existing software,

    M. Anders, M. Obaidi, B. Paech, and K. Schneider, “A study on the men- tal models of users concerning existing software,” in REFSQ. Springer International Publishing, 2022, pp. 235–250

    2022

  27. [35]

    What can be concluded from user feedback? - an empirical study,

    M. Anders, M. Obaidi, A. Specht, and B. Paech, “What can be concluded from user feedback? - an empirical study,” in REW, 2023, pp. 122–128

    2023

  28. [36]

    How to elicit explainability requirements? a com- parison of interviews, focus groups, and surveys,

    M. Obaidi, J. Droste, H. Deters, M. Herrmann, R. Ochsner, J. Kl ¨under, and K. Schneider, “How to elicit explainability requirements? a com- parison of interviews, focus groups, and surveys,” in 2025 IEEE 33rd International Requirements Engineering Conference (RE) , Sep. 2025

    2025

  29. [37]

    How do users like this feature? a fine grained sentiment analysis of app reviews,

    E. Guzman and W. Maalej, “How do users like this feature? a fine grained sentiment analysis of app reviews,” in RE, 2014

    2014

  30. [38]

    Ulich and P

    D. Ulich and P. Mayring, Psychologie der Emotionen , ser. Grundriss der Psychologie. Kohlhammer, 2003

    2003

  31. [39]

    Core affect and the psychological construction of emotion

    J. A. Russell, “Core affect and the psychological construction of emotion.” Psychological Review, vol. 110 1, 2003

    2003

  32. [40]

    Mehrabian and J

    A. Mehrabian and J. A. Russell, An approach to environmental psychol- ogy. the MIT Press, 1974

    1974

  33. [41]

    C. E. Osgood, G. J. Suci, and P. H. Tannenbaum, The measurement of meaning. University of Illinois press, 1957

    1957

  34. [42]

    Basic emotions, relations among emotions, and emotion- cognition relations,

    C. E. Izard, “Basic emotions, relations among emotions, and emotion- cognition relations,” Psychological Review, vol. 99, no. 3, pp. 561–565, 1992

    1992

  35. [43]

    A psychoevolutionary theory of emotions,

    R. Plutchik, “A psychoevolutionary theory of emotions,” Social Science Information, vol. 21, 1982

    1982

  36. [44]

    W. G. Parrott, Emotions in social psychology: Essential readings . psychology press, 2001

    2001

  37. [45]

    The emotional side of software developers in jira,

    M. Ortu, A. Murgia, G. Destefanis, P. Tourani, R. Tonelli, M. Marchesi, and B. Adams, “The emotional side of software developers in jira,” in 13th International Conference on Mining Software Repositories , ser. MSR ’16. ACM, 2016

    2016

  38. [46]

    Evaluation datasets for twitter sentiment analysis: a survey and a new dataset, the sts-gold,

    H. Saif, M. Fern ´andez, Y . He, and H. Alani, “Evaluation datasets for twitter sentiment analysis: a survey and a new dataset, the sts-gold,” in ESSEM 2013 , 2013

    2013

  39. [47]

    Boland, A

    K. Boland, A. Wira-Alam, and R. Messerschmidt, Creating an Anno- tated Corpus for Sentiment Analysis of German Product Reviews , ser. GESIS-Technical Reports. Mannheim: GESIS - Leibniz-Institut f ¨ur Sozialwissenschaften, 2013, vol. 2013/05

    2013

  40. [48]

    SCARE — the sentiment corpus of app reviews with fine-grained annotations in German,

    M. S ¨anger, U. Leser, S. Kemmerer, P. Adolphs, and R. Klinger, “SCARE — the sentiment corpus of app reviews with fine-grained annotations in German,” in LREC’16). ELRA, May 2016, pp. 1114–1121

    2016

  41. [49]

    Gervader-a german adaptation of the vader sentiment analysis tool for social media texts

    K. Tymann, M. Lutz, P. Palsbr ¨oker, and C. Gips, “Gervader-a german adaptation of the vader sentiment analysis tool for social media texts.” in LWDA, 2019, pp. 178–189

    2019

  42. [50]

    A gold standard for emotion annotation in stack overflow,

    N. Novielli, F. Calefato, and F. Lanubile, “A gold standard for emotion annotation in stack overflow,” in MSR’18. ACM, 2018

    2018

  43. [51]

    Sentiment analysis for software engineering: How far can pre-trained transformer models go?

    T. Zhang, B. Xu, F. Thung, S. A. Haryono, D. Lo, and L. Jiang, “Sentiment analysis for software engineering: How far can pre-trained transformer models go?” in ICSME, 2020

    2020

  44. [52]

    The measurement of observer agreement for categorical data

    J. Landis and G. Koch, “The measurement of observer agreement for categorical data.” Biometrics, vol. 33 1, 1977

    1977

  45. [53]

    Dataset: A german gold-standard dataset for sentiment analysis in software engineering,

    M. Obaidi, M. Herrmann, E. Schmid, R. Ochsner, K. Schneider, and J. Kl ¨under, “Dataset: A german gold-standard dataset for sentiment analysis in software engineering,” Jul. 2025. [Online]. Available: https://doi.org/10.5281/zenodo.15851545

    2025 doi

  46. [54]

    Training a broad-coverage german sentiment classification model for dialog sys- tems,

    O. Guhr, A.-K. Schumann, F. Bahrmann, and H. J. B ¨ohme, “Training a broad-coverage german sentiment classification model for dialog sys- tems,” in Proceedings of The 12th Language Resources and Evaluation Conference, 2020, pp. 1627–1632

    2020

  47. [55]

    Sentiment strength detection in short informal text,

    M. Thelwall, K. Buckley, G. Paltoglou, and A. Di Caiand Kappas, “Sentiment strength detection in short informal text,” JASIST, vol. 61, no. 12, 2010

    2010

  48. [56]

    Deva: sensing emotions in the valence arousal space in software engineering text,

    M. R. Islam and M. F. Zibran, “Deva: sensing emotions in the valence arousal space in software engineering text,” in Applied computing 2018 . Association for Computing Machinery Inc. (ACM), 2018

    2018

  49. [57]

    SentiCR: A customized sentiment analysis tool for code review interactions,

    T. Ahmed, A. Bosu, A. Iqbal, and S. Rahimi, “SentiCR: A customized sentiment analysis tool for code review interactions,” in ASE. Piscat- away, NJ, USA: IEEE, 2017

    2017

  50. [58]

    Sentiment and politeness analysis tools on developer discussions are unreliable, but so are people,

    N. Imtiaz, J. Middleton, P. Girouard, and E. Murphy-Hill, “Sentiment and politeness analysis tools on developer discussions are unreliable, but so are people,” in 3rd International Workshop on Emotion Awareness in Software Engineering, ser. SEmotion ’18, 2018, p. 55–61

    2018

  51. [59]

    Sentiment analysis for software engineering: How far can pre-trained transformer models go?

    T. Zhang, B. Xu, F. Thung, S. A. Haryono, D. Lo, and L. Jiang, “Sentiment analysis for software engineering: How far can pre-trained transformer models go?” in ICSME. IEEE, 2020

    2020

  52. [60]

    A theory of learning from different domains,

    S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan, “A theory of learning from different domains,” Machine learning, vol. 79, no. 1, pp. 151–175, 2010

    2010

  53. [61]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, and A. Wessl´en, Experimentation in software engineering . Springer, 2012

    2012

  54. [2024]

    Available: https://arxiv.org/abs/2405.16981

    [Online]. Available: https://arxiv.org/abs/2405.16981

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.