Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Modeling Communication Perception in Development Teams Using Monte Carlo Methods

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper derives a closed-form worst-case agreement formula for random subsets of raters and argues that omitting even one member of a 7-person team can misrepresent the team's mood.

desk verdict A transparently preliminary curve fit to Monte Carlo minima that is honest about its limits, but the central formula likely describes the simulation budget as much as the team; worth a serious referee, not yet citable as established. read the letter →

arxiv 2504.17610 v1 pith:VLD7UDHH submitted 2025-04-24 cs.SE

Marc Herrmann , Martin Obaidi , Jil Klünder This is my paper · ORCID
classification cs.SE MSC 68N3065C0562J02
keywords softwareengineeringcommunicationmoodanalysissentimentperceptionagreementMonteCarloexperimentnonlinearregression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how many members of a software development team must report their sentiment for the measured team mood to be trustworthy. Using a Monte Carlo experiment that draws random subsets of 45 developers who labeled 100 project statements, it finds that the interrater agreement of small subsets fluctuates strongly, and it fits a closed-form approximation to the worst-case agreement drop: $\min \kappa_n \approx 2\hat{\kappa}(n-2)/(k/10+n-2) - \hat{\kappa}$, where $\hat{\kappa}$ is the whole team's agreement and $n$ is the subset size. Applying the model to a typical 7-member team, the paper finds that omitting a single member moves the 68% confidence interval by about $\pm 19\%$ of $\hat{\kappa}$, and that larger omissions quickly make the measured mood unreliable. The paper therefore recommends including all team members in any mood survey, while explicitly noting that the model is preliminary and not yet proven general.

What carries the argument

The machinery is the rational regression function $f(n,k,\hat{\kappa}) = 2\hat{\kappa}(n-2)/(k/10+n-2) - \hat{\kappa}$, which models the minimum agreement curve of random subsets. It is the end product of a nonlinear regression that starts with a four-parameter rational form and, by iteratively replacing fitted regressors with observed constants, reduces to this expression in which the team size $k$ enters only through the additive term $k/10$ and the whole-team agreement $\hat{\kappa}$ enters as a linear scale factor. The function does the work of converting the empirical minima of 1,000 bootstrap draws into a closed-form lower bound that can be evaluated for any subset size before any surveying is done.

What would settle it

Recompute the Monte Carlo minima on the same 45-rater data with $m=100$ and with $m=10{,}000$ draws: if the fitted constants $a=2\hat{\kappa}$ and $b=k/10$ move materially, the formula is an artifact of the chosen $m=1000$. A second, independent check is the perfect-agreement case $\hat{\kappa}=1$, where the formula predicts $\min \kappa_n < 1$ for $n<k$ even though any subset of a perfectly agreeing team must have agreement 1.

Watch

Extended reading notes

Core claim

The paper's central claim is a worst-case law for random subsets of raters: for a team of $k$ members with overall agreement $\hat{\kappa}$, the minimum agreement of any random $n$-member subset is approximated by $\min \kappa_n \approx 2\hat{\kappa}(n-2)/(k/10+n-2) - \hat{\kappa}$ for $2 \le n \le k$. The authors arrive at this by bootstrapping subsets of various sizes, extracting the minimum agreement value $\min \kappa_n$ over 1,000 random draws, and fitting a rational function whose four free parameters successively collapse to constants (horizontal shift 2, vertical offset $-\hat{\kappa}$, numerator scale $2\hat{\kappa}$, and denominator shift $k/10$), leaving a formula that depends only on $n$, $k$, and $\hat{\kappa}$. They report a fit of $R^2 \ge 0.9$ across team sizes $3 \le k \le 45$, and use the formula together with coefficient-of-variation intervals to show that a 7-member team's agreement becomes materially uncertain once even one member is excluded. The authors themselves caution that general applicability is not yet confirmed.

Load-bearing premise

The load-bearing premise is that the minimum agreement observed over 1,000 random draws is a stable property of the rater pool, whereas the minimum of a sample is an extreme-order statistic that keeps falling as the number of draws grows, and the paper never tests this dependence on the draw count.

Editorial extensions

If this is right

  • For a team of 7 members with the observed agreement $\hat{\kappa}=0.2193$, the 68% interval for a 6-member subset is roughly $\hat{\kappa} \pm 19\%$; for 5 members it is $\hat{\kappa} \pm 31\%$; and for 2 members it reaches $\hat{\kappa} \pm 119\%$, so subsets quickly become unreliable.
  • Equation (11) gives a project lead a way to predict, before surveying, the worst-case agreement drop for any subset size $n$, using only the team size and the whole team's overall agreement.
  • The fitted constants ($a=2\hat{\kappa}$, $b=k/10$) indicate that the minimum-agreement curve is determined entirely by team size and overall agreement, with no additional team-specific free parameters.
  • Under the model, the only way to keep the measured agreement representative of the whole team is to include all $k$ members in the survey, because even $n=k-1$ carries a substantial worst-case deviation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the target $\min \kappa_n$ is a minimum over $m$ draws, the fitted constants $a$ and $b$ are likely to depend on the simulation budget; refitting the model at $m=100$ and $m=10^{4}$ would reveal whether the formula describes the team or the sampling procedure.
  • The same rational-curve approach could be applied to other subjective labeling tasks or to two-rater agreement coefficients, but the empirical constant $k/10$ would need to be re-estimated for each new rater population.
  • If the formula holds, it can be inverted to compute the smallest $n$ that keeps the worst-case agreement drop below a chosen tolerance, turning the model into a sampling-budget rule for mood surveys.
  • The model's breakdown at $\hat{\kappa}=1$ hints that the true relationship must saturate at perfect agreement, suggesting the fitted rational form is an approximation to a different extreme-order-statistics law for the minimum of permutation-sampled agreement values.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper investigates how interrater agreement (Fleiss' κ) changes when a random subset of developers is drawn from a larger team, using a Monte Carlo experiment based on 45 complete-case raters from an existing sentiment-perception survey. The authors compute, for each subset size n, the minimum of m=1000 simulated κ_n values, fit a rational function to those minima, and arrive at the closed-form approximation in Eq. (11): min κ_n ≈ 2κ̂·(n−2)/(k/10+n−2) − κ̂. They report an in-sample R² of 0.95 (Figure 2), explicitly disclaim general applicability of the model, and use a separate Monte Carlo analysis with k=7 to compute coefficients of variation, concluding that omitting even one member of a seven-person team can substantially misrepresent the team's overall mood. The paper is openly framed as a preliminary model and lists several threats to validity.

Significance. If the proposed model were validated, it would provide a practically useful closed-form estimate of the worst-case agreement drop in random subsets of raters, directly informing mood-survey design and sentiment-dataset construction in software engineering. The paper makes its raw survey data available via Zenodo and is transparent about the exploratory nature of the analysis. However, the central claim is currently supported only by an in-sample fit to Monte Carlo minima that depend on the simulation budget m; no out-of-sample validation, m-sensitivity analysis, or independent confirmation is provided. The work is therefore better characterized as a descriptive case study with a plausible but unproven empirical formula than as an established model of team perception.

major comments (4)
  1. [§3.5.2 / Table 1] In Section 3.5.2, the quantity min κ_n is defined as the minimum over m=1000 Monte Carlo draws (Table 1, Step 4). For n<k, an empirical minimum is an extreme-order statistic that is nonincreasing in m and converges to the true subset minimum only as m→∞; for k=45 the number of subsets is astronomically large, so the fitted minima depend on the chosen simulation budget. The paper cites the law of large numbers in §3.5.1, but that theorem applies to averages rather than extremes, and no sensitivity analysis with respect to m is reported. Eq. (11) is therefore fitted to a target that may be an artifact of m=1000, and the model cannot yet be claimed to describe the rater population.
  2. [§3.5.2, Eq. (11); §4.2, Figure 2] The constants in Eq. (11) were obtained by iteratively fitting the regression model to the Monte Carlo minima and then replacing regressors with their fitted values (Section 3.5.2), and the reported R²=0.95 (Figure 2) is an in-sample fit to those same minima. This circular procedure does not provide evidence that the formula predicts minimum agreement on new teams or new Monte Carlo runs. An out-of-sample evaluation, such as fitting on one subset of teams or runs and testing on a held-out subset, or validation on an independent rater dataset, is required before Theorem 1 can be presented as a model rather than a descriptive curve.
  3. [§4.2, Eq. (11)] Eq. (11) is inconsistent with the boundary condition at n=k: when the subset is the whole team, the minimum agreement equals κ̂ by definition, but the formula yields a value strictly below κ̂ for all finite k (e.g., approximately 0.811κ̂ for k=45, as the authors acknowledge in the text following Figure 2). For a function intended to describe the minimum possible agreement, this boundary violation should be repaired, or the model's domain should be explicitly restricted to n<k with a statement about the size of the discrepancy.
  4. [§3.5.3, Table 2] The RQ3 analysis computes the coefficients of variation cv_n from the same simulation and the same 45 raters used to build the model, with m=j=100 repetitions, and then invokes the central limit theorem and the empirical rule (Eqs. (1) and (4)) to form intervals for κ_n. Because κ_n is bounded and can be strongly skewed for small n, the normality assumption is questionable, and no confidence intervals or bootstrap validation are provided for cv_n. The practical recommendation that omitting one member of a seven-person team misrepresents the team mood is therefore a descriptive statement about this simulation, not a validated empirical claim.
minor comments (5)
  1. [§3.5.1] The statement that analyzing all possible subsets would lead to '44!≈2.7×10^54 combinatory possibilities' is not correct: the number of nonempty subsets of 45 raters is 2^45−1, and 45! (or 44!) would count ordered sequences, not subsets.
  2. [§3.5.1] The phrase 'ensuring valid statistical interference' should read 'statistical inference,' and the citation to the law of large numbers should be clarified because that theorem does not justify stability of minima.
  3. [§4.2] Labeling Eq. (11) as 'Theorem 1' is misleading because no formal proof is given; the authors themselves state that they cannot confirm general applicability. Renaming it an 'empirical model' or 'conjecture' would better match its evidential status.
  4. [§4.1 / Figure 1] The claim that 'the data showed great uniformity' across repetitions of the Monte Carlo experiment is not supported by quantitative evidence; please report summary statistics or show multiple independent runs.
  5. [§5.4.3] The 'Computational Limitations' threat correctly identifies that not all combinations are enumerated, but it is listed only as a threat to validity; the paper would be stronger if this limitation were addressed directly by an m-sensitivity analysis.

Circularity Check

1 steps flagged · score 6.0 of 10

Eq. (11) is fitted to the same Monte Carlo minima that Theorem 1 then 'predicts'; the central closed-form result is an in-sample curve fit, not an independent derivation.

  1. fitted input called prediction [Section 3.5.2 and Section 4.2, Eq. (11) / Theorem 1]
    "we determined the minimum value of agreement (i.e., min κ_n) for each n raters resulting from m repeated cycles as defined within algorithm 1. We contemplated these minimum values ... and tried fitting a mathematical model to the values using non-linear multiple regression analysis ... To evaluate the fit of the model f to the true values min κ_n, we computed the coefficient of determination R^2. ... By inserting b = k/10 into f , we can therefore obtain our final regression model: f(n,k,κ̂)=2·κ̂·(n−2)/(k/10+(n−2))−κ̂ ... Theorem 1: ... we find that minκ_n {κ_n} ≈ 2·κ̂·(n−2)/(k/10+n−2)−κ̂"

    The target of the regression, min κ_n, is exactly the quantity Theorem 1 presents as a result. The constants in Eq. (11) (the shift d=2, the relation c=κ̂−a, the factor a=2κ̂, and b=k/10) are obtained by fitting to those same Monte Carlo minima, with iterative replacement of regressors, and the only evidence offered is the in-sample R² in Figure 2. Therefore the theorem's '≈' is a restatement of the fitted curve rather than an independent prediction; the agreement between Eq. (11) and min κ_n is forced by the least-squares construction. Moreover, min κ_n is an extreme-order statistic over m=1000 draws, so the fitted constants inherit the simulation budget, and no m-sensitivity or out-of-sample check is reported.

full rationale

The paper's central mathematical claim is not independently derived: Eq. (11) is a nonlinear regression fitted to the Monte Carlo minima min κ_n, and Theorem 1 then presents that fitted expression as a general result for the same quantity. This is the classic in-sample fit renamed as a prediction, so the match with R²=0.95 is by construction rather than evidence of predictive validity. The paper honestly disclaims general applicability and lists computational limitations, and the self-citations to [14], [16], and [38] serve as data/background rather than load-bearing theorems, so no additional circularity of the self-citation or uniqueness-import kind is present. However, because the central closed-form result reduces to the regression target itself, a partial circularity score of 6 is warranted.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The final formula contains no overt free parameters, but every constant in it (2, 1/10, and the shift terms) was fitted to the same Monte Carlo data it describes. The analysis also depends on distributional and representativeness assumptions that are acknowledged but untested.

free parameters (4)
  • numerator factor 2 (a) = 2
    Regressor a stabilized at 2·κ̂ across Monte Carlo runs; substituted into Eq. (10) as a constant rather than derived from first principles.
  • denominator slope 1/10 (b = k/10) = k/10
    Regressor b was observed to scale linearly with team size k for k=3..45; the factor 1/10 is fitted to these runs and used in Eq. (11).
  • horizontal shift d = 2
    Set to 2 because the minima start at n=2; adopted from iterative regression observations, not from theory.
  • vertical shift c = κ̂ − a
    Observed to stabilize at κ̂ minus regressor a; substituted into Eq. (9).
assumptions (6)
  • domain assumption Fleiss' kappa is an appropriate agreement measure for subsets as small as n=2 raters on three nominal categories.
    Used to define agreement throughout; the small-sample behavior of kappa at n=2 is not discussed, though it becomes a two-rater coefficient with substantial variance.
  • domain assumption The distribution of κ_n for random subsets is approximately normal, so the empirical rule can be applied.
    Section 3.5.3 invokes the central limit theorem for κ_n even for n=2, despite kappa being bounded and the distribution skewed for low agreement.
  • ad hoc to paper The 45 complete-case raters are representative of arbitrary development teams.
    The entire Monte Carlo experiment resamples from this single pool; the authors acknowledge in §5.4.4 that the sample is mostly students and generalizability is limited.
  • ad hoc to paper The mean of subset kappas equals the full-team kappa after averaging over j teams (μ̄_n = κ̂).
    Needed to build the ±cv·κ̂ intervals in Eq. (4); the paper admits the mean is biased by the selected subset P and assumes regression to the mean.
  • ad hoc to paper The minimum over m=1000 draws converges to a stable model target independent of m.
    The fitted model describes these empirical minima, but extreme-order statistics shift with m; no justification for m=1000 is given.
  • domain assumption Missing annotations are missing at random, so complete-case selection does not bias results.
    Section 5.4.2 raises this threat; if missingness is not MCAR/MAR, the 45-raters sample is selected and the agreement values may be biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modeling Communication Perception in Development Teams Using Monte Carlo Methods." pith.science (2026). https://pith.science/paper/VLD7UDHH

@misc{pith2026250417610,
  author       = {Pith},
  title        = {Pith review of: Modeling Communication Perception in Development Teams Using Monte Carlo Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VLD7UDHH}},
  note         = {Machine review of arXiv:2504.17610}
}
read the original abstract

Software development is a collaborative task involving diverse development teams, where toxic communication can negatively impact team mood and project success. Mood surveys enable the early detection of underlying tensions or dissatisfaction within development teams, allowing communication issues to be addressed before they escalate, fostering a positive and productive work environment. The mood can be surveyed indirectly by analyzing the text-based communication of the team. However, emotional subjectivity leads to varying sentiment interpretations across team members; a statement perceived neutrally by one developer might be seen as problematic by another developer with a different conversational culture. Early identification of perception volatility can help prevent misunderstandings and enhance team morale while safeguarding the project. This paper analyzes the diversity of perceptions within arbitrary development teams and determines how many team members should report their sentiment to accurately reflect the team's mood. Through a Monte Carlo experiment involving 45 developers, we present a preliminary mathematical model to calculate the minimum agreement among a subset of developers based on the whole team's agreement. This model can guide leadership in mood assessment, demonstrating that omitting even a single member in an average-sized 7-member team can misrepresent the overall mood. Therefore, including all developers in mood surveying is recommended to ensure a reliable evaluation of the team's mood.

Figures

Figures reproduced from arXiv: 2504.17610 by the authors.

Figure 1
Figure 1. Violin plot of the agreement values 𝜿𝒏 resulting from one run of the Monte Carlo experiment (cf [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Comparison of the minimum agreement min𝜿𝒏 from the Monte Carlo experiment and our deducted model 𝒇 (𝒏, 𝒌, 𝜿ˆ) used for regression. generally produces results fitting the minimum agreement values min𝜅𝑛 with a 𝑅 2 -value of 0.9 and above. Lower 𝑅 2 -values could be explained by the randomly generated plot having a flatter curvature because more agreeing raters were selected in the beginning by chance of the Monte Carl… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 29 canonical work pages

  1. [1]

    Toufique Ahmed, Amiangshu Bosu, Anindya Iqbal, and Shahram Rahimi. 2017. SentiCR: A customized sentiment analysis tool for code review interactions. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, Piscataway, NJ, USA, 106–111. https://doi.org/10.1109/ASE.2017. 8115623

    doi:10.1109/ase.2017 2017

  2. [2]

    Joseph Bamidele Awotunde, Femi Emmanuel Ayo, Roseline Oluwaseun Ogun- dokun, Opeyemi Emmanuel Matiluko, and Emmanuel Abidemi Adeniyi. 2020. In- vestigating the Roles of Effective Communication Among Stakeholders in Collab- orative Software Development Projects. In Computational Science and Its Applica- tions – ICCSA 2020 , Osvaldo Gervasi, Beniamino Murga...

    work page 2020

  3. [3]

    Carver, and Kimberly Stowers

    Eduardo Caballero-Espinosa, Jeffrey C. Carver, and Kimberly Stowers. 2023. Community smells—The sources of social debt: A systematic literature review. Information and Software Technology 153 (2023), 107078. https://doi.org/10.1016/ j.infsof.2022.107078

    arXiv 2023

  4. [4]

    Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20, 1 (1960), 37–46. https://doi.org/10.1177/ 001316446002000104

    work page 1960

  5. [5]

    Frederik Michel Dekking, Cornelis Kraaikamp, Hendrik Paul Lopuhaä, and Lu- dolf Erwin Meester. 2005. The law of large numbers . Springer London, London, 181–194. https://doi.org/10.1007/1-84628-168-7_13

    work page doi:10.1007/1-84628-168-7_13 2005

  6. [6]

    Tom DeMarco and Tim Lister. 2013. Peopleware: Productive Projects and Teams (3rd Edition) (3rd ed.). Addison-Wesley Professional, Boston, MA, USA

    work page 2013

  7. [8]

    K. Emam. 1999. Benchmarking Kappa: Interrater Agreement in Software Process Assessments. Empirical Software Engineering 4 (1999), 113–133. https://doi.org/ 10.1023/A:1009820201126

    doi:10.1023/a:1009820201126 1999

  8. [9]

    Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 5 (1971), 378—382. https://doi.org/10.1037/h0031619

    doi:10.1037/h0031619 1971

Show all 70 references
  1. [10]

    Daniel Graziotin, Xiaofeng Wang, and Pekka Abrahamsson. 2014. Happy soft- ware developers solve problems better: psychological measurements in empirical software engineering. PeerJ 2 (2014), e289. https://doi.org/10.7717/peerj.289

    2014 doi

  2. [11]

    Harris, K

    Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fe...

    2020

  3. [12]

    David Heise. 2014. Cultural variations in sentiments. SpringerPlus 3 (04 2014),

    2014

  4. [13]

    Marc Herrmann and Jil Klünder. 2021. From Textual to Verbal Communication: Towards Applying Sentiment Analysis to a Software Project Meeting. In2021 IEEE 29th International Requirements Engineering Conference Workshops (REW) . IEEE, New York, NY, USA, 371–376. https://doi.org/...

    2021

  5. [14]

    Marc Herrmann, Martin Obaidi, Larissa Chazette, and Jil Klünder. 2022. On the subjectivity of emotions in software projects: How reliable are pre-labeled data sets for sentiment analysis? Journal of Systems and Software 193 (2022), 111448. https://doi.org/10.1016/j.jss.2022.111448

    2022

  6. [15]

    Marc Herrmann, Martin Obaidi, and Jil Klünder. 2022. SEnti-Analyzer: Joint Sentiment Analysis For Text-Based and Verbal Communication in Software Projects . Technical Report. Leibniz University Hannover. https://doi.org/10.48550/ARXIV. 2206.10993

    2022 doi arXiv

  7. [16]

    Marc Herrmann, Martin Obaidi, and Jil Klünder. 2025. Different and similar perceptions of communication among software developers. Information and Software Technology 181 (2025), 107698. https://doi.org/10.1016/j.infsof.2025. 107698

    2025 doi

  8. [17]

    John D Hunter. 2007. Matplotlib: A 2D graphics environment. Computing in science & engineering 9, 3 (2007), 90–95. https://doi.org/10.1109/MCSE.2007.55

    2007 doi

  9. [18]

    Nasif Imtiaz, Justin Middleton, Peter Girouard, and Emerson Murphy-Hill. 2018. Sentiment and Politeness Analysis Tools on Developer Discussions Are Un- reliable, but so Are People. In Proceedings of the 3rd International Workshop on Emotion A wareness in Software Engineering (...

    2018

  10. [19]

    Md Rakibul Islam and Minhaz F. Zibran. 2018. DEVA: sensing emotions in the valence arousal space in software engineering text. In Applied computing 2018, Hisham M. Haddad, Roger L. Wainwright, and Richard Chbeir (Eds.). Association for Computing Machinery Inc. (ACM), New York,...

    2018

  11. [20]

    John A. Johnson. 2017. Big-Five Model. Springer International Publishing, Cham, Chapter 1, 1–16. https://doi.org/10.1007/978-3-319-28099-8_1212-1

    2017 doi

  12. [21]

    Robbert Jongeling, Proshanta Sarkar, Subhajit Datta, and Alexander Serebrenik

  13. [22]

    Ho-Won Jung. 2003. Evaluating interrater agreement in SPICE-based assessments. Comput. Stand. Interfaces 25 (2003), 477–499. https://doi.org/10.1016/S0920- 5489(03)00018-7

    2003 doi

  14. [23]

    Kraut and Lynn A

    Robert E. Kraut and Lynn A. Streeter. 1995. Coordination in Software Develop- ment. Commun. ACM 38, 3 (1995), 69–81. https://doi.org/10.1145/203330.203345

    1995

  15. [24]

    Pierre-Simon Laplace. 1810. Sur les approximations des formules qui sont fonc- tions de tres grands nombres et sur leur application aux probabilites. Œuvres complètes 12 (1810), 301–345

  16. [25]

    Grischa Liebel, Jil Klünder, Regina Hebig, Christopher Lazik, Inês Nunes, Isabella Graßl, Jan-Philipp Steghöfer, Joeri Exelmans, Julian Oertel, Kai Marquardt, Katha- rina Juhnke, Kurt Schneider, Lucas Gren, Lucia Happe, Marc Herrmann, Marvin Wyrich, Matthias Tichy, Miguel Goul...

    2024

  17. [26]

    Bin Lin, Nathan Cassee, Alexander Serebrenik, Gabriele Bavota, Nicole Novielli, and Michele Lanza. 2022. Opinion Mining for Software Development: A System- atic Literature Review. ACM Trans. Softw. Eng. Methodol. 31, 3, Article 38 (March 2022), 41 pages. https://doi.org/10.114...

    2022 doi

  18. [27]

    Bin Lin, Fiorella Zampetti, Gabriele Bavota, Massimiliano Di Penta, Michele Lanza, and Rocco Oliveto. 2018. Sentiment Analysis for Software Engineering: How Far Can We Go?. In Proceedings of the 40th International Conference on Soft- ware Engineering (Gothenburg, Sweden) (ICSE...

    2018

  19. [28]

    Marjaana Lindeman and Markku Verkasalo. 2005. Measuring Values With the Short Schwartz’s Value Survey. Journal of Personality Assessment 85, 2 (2005), 170–178. https://doi.org/10.1207/s15327752jpa8502_09 PMID: 16171417

    2005 doi

  20. [29]

    Daniel Graziotin, Xiaofeng Wang, and Pekka Abrahamsson. 2015. How do you feel, developer? An explanatory theory of the impact of affects on programming performance. PeerJ Computer Science 1 (2015), e18. https://doi.org/10.7717/peerj- cs.18

    2015 doi

  21. [30]

    Walaa Medhat, Ahmed Hassan, and Hoda Korashy. 2014. Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal 5, 4 (2014), 1093–1113. https://doi.org/10.1016/j.asej.2014.04.011

    2014 doi

  22. [31]

    Did You Miss My Comment or What?

    Courtney Miller, Sophie Cohen, Daniel Klug, Bogdan Vasilescu, and Christian KaUstner. 2022. "Did You Miss My Comment or What?": Understanding Toxicity in Open Source Discussions. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylva...

    2022

  23. [32]

    George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review 63, 2 (1956), 81. https://doi.org/10.1037/h0043158

    1956 doi

  24. [33]

    Alessandro Murgia, Parastou Tourani, Bram Adams, and Marco Ortu. 2014. Do Developers Feel Emotions? An Exploratory Analysis of Emotions in Software Artifacts. In Proceedings of the 11th Working Conference on Mining Software Repositories (Hyderabad, India)(MSR 2014). Associatio...

    2014

  25. [34]

    Nico J. D. Nagelkerke. 1991. A note on a general definition of the coefficient of determination. Biometrika 78, 3 (09 1991), 691–692. https://doi.org/10.1093/ biomet/78.3.691

    1991

  26. [35]

    Nicole Novielli, Fabio Calefato, Davide Dongiovanni, Daniela Girardi, and Filippo Lanubile. 2020. A gold standard for polarity of emotions of software developers in GitHub. https://doi.org/10.6084/m9.figshare.11604597.v1

    2020 doi

  27. [36]

    Nicole Novielli, Fabio Calefato, and Filippo Lanubile. 2018. A Gold Standard for Emotion Annotation in Stack Overflow. In 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR) (Gothenburg, Sweden) (MSR ’18). Association for Computing Machinery, New ...

    2018

  28. [37]

    Nicole Novielli and Alexander Serebrenik. 2023. Emotion Analysis in Software Ecosystems. Springer International Publishing, Cham, 105–127. https://doi.org/ 10.1007/978-3-031-36060-2_5

    2023 doi

  29. [38]

    Martin Obaidi, Marc Herrmann, Larissa Chazette, and Jil Klünder. 2022. Dataset: On the Subjectivity of Emotions in Software Projects: How Reliable are Pre- Labeled Data Sets for Sentiment Analysis? https://doi.org/10.5281/zenodo. 6611728

    2022 doi

  30. [39]

    Martin Obaidi and Jil Klünder. 2021. Development and Application of Sentiment Analysis Tools in Software Engineering: A Systematic Literature Review. In Evaluation and Assessment in Software Engineering (EASE’21) . Association for Computing Machinery, New York, NY, USA, 80–89....

    2021

  31. [40]

    Martin Obaidi, Lukas Nagel, Alexander Specht, and Jil Klünder. 2022. Sentiment analysis tools in software engineering: A systematic mapping study. Information and Software Technology 151 (2022), 107018. https://doi.org/10.1016/j.infsof.2022. 107018

    2022 doi

  32. [41]

    Marco Ortu, Alessandro Murgia, Giuseppe Destefanis, Parastou Tourani, Roberto Tonelli, Michele Marchesi, and Bram Adams. 2016. The Emotional Side of Software Developers in JIRA. In Proceedings of the 13th International Conference on Mining Software Repositories (Austin, Texas)...

    2016

  33. [42]

    W Gerrod Parrott. 2001. Emotions in social psychology: Essential readings . Psy- chology Press, Hove, East Sussex, United Kingdom

    2001

  34. [43]

    Curran, Kenneth A

    Pamela Paxton, Patrick J. Curran, Kenneth A. Bollen, Jim Kirby, and Feinian Chen. 2001. Monte Carlo Experiments: Design and Implementation. Structural Equation Modeling: A Multidisciplinary Journal 8, 2 (2001), 287–312. https: //doi.org/10.1207/S15328007SEM0802_7

    2001 doi

  35. [44]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cour- napeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine ...

    2011

  36. [45]

    Daniel Preoţiuc-Pietro and Lyle Ungar. 2018. User-Level Race and Ethnicity Predictors from Twitter Text. In Proceedings of the 27th International Conference on Computational Linguistics , Emily M. Bender, Leon Derczynski, and Pierre Isabelle (Eds.). Association for Computation...

    2018

  37. [46]

    Muhammad Saqib Rehan, Mirza Adnan Baig, Arsalan Nasir, Shereen‘ Gul, and Rabia Ashraf. 2023. Age, Gender, and Sentiments: Navigating the Landscape of Review Interpretation. International Journal of Emerging Engineering and Technology 2, 1 (Jul. 2023), 92–98. https://doi.org/10...

    2023 doi

  38. [47]

    Donald B. Rubin. 1976. Inference and missing data. Biometrika 63, 3 (1976), 581–592. https://doi.org/10.1093/biomet/63.3.581

    1976 doi

  39. [48]

    James A Russell and Albert Mehrabian. 1977. Evidence for a three-factor theory of emotions. Journal of Research in Personality 11, 3 (1977), 273–294. https: //doi.org/10.1016/0092-6566(77)90037-X

    1977 doi

  40. [49]

    Kurt Schneider, Jil Klünder, Fabian Kortum, Lisa Handke, Julia Straube, and Simone Kauffeld. 2018. Positive affect through interactions in meetings: The role of proactive and supportive statements. Journal of Systems and Software 143 (2018), 59–70. https://doi.org/10.1016/j.js...

    2018 doi

  41. [50]

    Lennart Schroth, Martin Obaidi, Alexander Specht, and Jil Klünder. 2022. On the Potentials of Realtime Sentiment Analysis on Text-Based Communication in Software Projects. In Human-Centered Software Engineering. Springer Interna- tional Publishing, Cham, 90–109. https://doi.or...

    2022 doi

  42. [51]

    Schwartz

    Shalom H. Schwartz. 1992. Universals in the Content and Structure of Values: Theoretical Advances and Empirical Tests in 20 Countries. In Advances in Exper- imental Social Psychology, Mark P. Zanna (Ed.). Vol. 25. Academic Press, London, 1–65. https://doi.org/10.1016/S0065-260...

    1992 doi

  43. [52]

    Shaver, Judith C

    Phillip R. Shaver, Judith C. Schwartz, Donald Kirson, and Cary O’Connor. 1987. Emotion knowledge: further exploration of a prototype approach. Journal of personality and social psychology 52 6 (1987), 1061–86

    1987

  44. [53]

    Alexander Specht, Martin Obaidi, Lukas Nagel, Marek Stess, and Jil Klünder

  45. [54]

    Kai Stapel and Kurt Schneider. 2014. Managing knowledge on communication and information flow in global software projects. Expert Sys: J. Knowl. Eng. 31, 3 (July 2014), 234–252. https://doi.org/10.1111/exsy.649

    2014 doi

  46. [55]

    Czesław Ste ¸pniak. 2011. Coefficient of Variation . Springer Berlin Heidelberg, Berlin, Heidelberg, 267–267. https://doi.org/10.1007/978-3-642-04898-2_177

    2011 doi

  47. [56]

    Sutherland, J.O

    J. Sutherland, J.O. Coplien, and C.O. Ramos. 2019. A Scrum Book: The Spirit of the Game. Pragmatic Bookshelf, Raleigh, NC, USA

    2019

  48. [57]

    Tamburri, Ivan S

    Damian A. Tamburri, Ivan S. Razo-Zapata, Héctor Fernández, and Cédric Tedeschi

  49. [58]

    Tien Rahayu Tulili, Andrea Capiluppi, and Ayushi Rastogi. 2022. Burnout in software engineering: A systematic mapping study. Information and Software Technology 155 (2022), 107116. https://doi.org/10.1016/j.infsof.2022.107116

    2022

  50. [59]

    Gias Uddin and Foutse Khomh. 2021. Automatic Mining of Opinions Expressed About APIs in Stack Overflow. IEEE Transactions on Software Engineering 47, 3 (2021), 522–559. https://doi.org/10.1109/TSE.2019.2900245

    2021

  51. [60]

    Lorenzo Villarroel, Gabriele Bavota, Barbara Russo, Rocco Oliveto, and Massimil- iano Di Penta. 2016. Release planning of mobile apps based on user reviews. In Proceedings of the 38th International Conference on Software Engineering (Austin, Texas) (ICSE ’16). Association for ...

    2016

  52. [61]

    Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J

    Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jar- rod Millman, Nikolay Mayorov, Andrew R. J. Nelson,...

    2020

  53. [62]

    Sai Datta Vishnubhotla, Emilia Mendes, and Lars Lundberg. 2020. Investigating the relationship between personalities and agile team climate of software profes- sionals in a telecom company. Information and Software Technology 126 (2020), 106335. https://doi.org/10.1016/j.infso...

    2020

  54. [63]

    Wes McKinney. 2010. Data Structures for Statistical Computing in Python. In Proceedings of the 9th Python in Science Conference . SciPy, Austin, TX, USA, 56–61. https://doi.org/10.25080/Majora-92bf1922-00a

    2010 doi

  55. [64]

    Emily Winter, Steve Forshaw, and Maria Angela Ferrario. 2018. Measuring human values in software engineering. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (Oulu, Finland) (ESEM ’18). Association for Computing Mac...

    2018

  56. [65]

    Ohlsson, Björn Regnell, and Anders Wesslén

    Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering . Springer, Berlin, Germany. https://doi.org/10.1007/978-3-642-29044-2

    2012 doi

  57. [66]

    Rebekka Wohlrab, Marc Herrmann, Marvin Lazik, Christopherand Wyrich, Inês Nunes, Kurt Schneider, Lucas Gren, and Robert Heinrich. 2025. Supporting Value- Aware Software Engineering Through Traceability and Value Tactics. InProduct- Focused Software Process Improvement , Dietma...

    2025 doi

  58. [67]

    Umer Zaman, Zulaikha Jabbar, Shahid Nawaz, and Mazhar Abbas. 2019. Under- standing the soft side of software projects: An empirical study on the interactive effects of social skills and political skills on complexity – performance rela- tionship. International Journal of Proje...

    2019 doi

  59. [170]

    https://doi.org/10.1186/2193-1801-3-170

    doi

  60. [2012]

    In Proceedings of the 4th International Workshop on Principles of Engineering Service-Oriented Systems (Zurich, Switzer- land) (PESOS ’12)

    Simulating awareness in global software engineering: a comparative analy- sis of scrum and agile service networks. In Proceedings of the 4th International Workshop on Principles of Engineering Service-Oriented Systems (Zurich, Switzer- land) (PESOS ’12). IEEE Press, Piscataway...

  61. [2017]

    Empirical Software Engineering 22, 5 (2017), 2543–2584

    On negative results when using sentiment analysis tools for software engineering research. Empirical Software Engineering 22, 5 (2017), 2543–2584. https://doi.org/10.1007/s10664-016-9493-x

    2017 doi

  62. [2024]

    In Human-Centered Software Engineering

    What is Needed to Apply Sentiment Analysis in Real Software Projects: A Feasibility Study in Industry. In Human-Centered Software Engineering. Springer Nature Switzerland, Cham, 105–129. https://doi.org/10.1007/978-3-031-64576- 1_6

    doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.