pith. sign in

arxiv: 1907.07874 · v1 · pith:WYBEYLZBnew · submitted 2019-07-18 · 💻 cs.SE

A Study on the Prevalence of Human Values in Software Engineering Publications, 2015-2018

Harsha Perera , Arif Nurwidyantoro , Waqar Hussain , Davoud Mougouei , Jon Whittle , Rifat Ara Shams , Gillian Oliver This is my paper

Pith reviewed 2026-05-24 20:01 UTC · model grok-4.3

classification 💻 cs.SE
keywords human valuessoftware engineeringpublication analysisprevalenceconferencesjournalsvalue classification
0
0 comments X

The pith

Software engineering publications from 2015-2018 rarely address human values directly.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper applies a social science value structure to classify recent top-tier SE conference and journal papers according to their relevance to human values such as equality and fairness. It finds that only a small proportion of publications directly consider these values. For most individual values, very few or no relevant papers appear at all. Relevant publications occur more often in conferences than in journals. These patterns indicate limited technical and methodological support for engineering human values into software.

Core claim

The paper classifies publications from 2015-2018 against a value structure adopted from social sciences and reports three main results: only a small proportion of the publications directly consider values and are therefore classified as relevant; for the majority of the values, very few or no relevant publications were found; and the prevalence of relevant publications was higher in SE conferences compared to SE journals.

What carries the argument

Classification of each publication as relevant or not to specific human values using the adopted social science value structure.

If this is right

  • Most human values receive little or no direct attention in recent SE research.
  • Conferences incorporate value considerations more often than journals do.
  • Engineering human values into software lacks widespread methodological support in the published literature.
  • Negative socio-economic impacts from software can arise when values such as equality and fairness are overlooked.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Researchers could test whether adding value-focused tracks or review criteria at conferences would increase the proportion of relevant papers.
  • The gap between conferences and journals suggests that publication venue itself influences how often values appear in SE work.
  • Extending the analysis to earlier or later years would show whether the observed low prevalence is stable or changing.

Load-bearing premise

The value structure from social sciences supplies a valid and complete basis for deciding whether an SE publication addresses human values, and the classification decisions are consistent.

What would settle it

A re-examination of the same set of 2015-2018 publications that applies a different classification method and identifies a substantially larger share as directly considering human values.

Figures

Figures reproduced from arXiv: 1907.07874 by Arif Nurwidyantoro, Davoud Mougouei, Gillian Oliver, Harsha Perera, Jon Whittle, Rifat Ara Shams, Waqar Hussain.

Figure 1
Figure 1. Figure 1: Schwartz Values Structure [30, 33] (adopted from [17]). Words in black [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Relevance of SE publications to human values [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Relevant publications per year. be further clarified in a SE context before they can be used by SE researchers and practitioners. In the attempt to understand which values are most commonly considered in SE research, we found ( [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The level of attention given to 58 values in the Schwartz Value Structure. Publications [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The number of relevant publications per value [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Relevant publications per value category [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Proportion of values relevant publications in SE [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 10
Figure 10. Figure 10: The distribution of publications relevant to different values by venue/track; relevant [PITH_FULL_IMAGE:figures/full_fig_p009_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Publications relevant to different value [PITH_FULL_IMAGE:figures/full_fig_p010_11.png] view at source ↗
read the original abstract

Failure to account for human values in software (e.g., equality and fairness) can result in user dissatisfaction and negative socio-economic impact. Engineering these values in software, however, requires technical and methodological support throughout the development life cycle. This paper investigates to what extent software engineering (SE) research has considered human values. We investigate the prevalence of human values in recent (2015 - 2018) publications at some of the top-tier SE conferences and journals. We classify SE publications, based on their relevance to different values, against a widely used value structure adopted from social sciences. Our results show that: (a) only a small proportion of the publications directly consider values, classified as relevant publications; (b) for the majority of the values, very few or no relevant publications were found; and (c) the prevalence of the relevant publications was higher in SE conferences compared to SE journals. This paper shares these and other insights that motivate research on human values in software engineering.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper reports an empirical classification of software engineering publications from 2015-2018 at selected top-tier conferences and journals. Publications are labeled as relevant or not to individual human values drawn from a social-science taxonomy; the central findings are that only a small proportion are relevant, that most values have zero or near-zero relevant papers, and that relevant papers appear more frequently in conferences than in journals.

Significance. If the classification judgments prove reliable, the study would document a measurable gap between the importance of human values in software systems and the attention they receive in the SE literature, supplying a concrete baseline that could guide future empirical and methodological work on value-sensitive design.

major comments (2)
  1. [§3] §3 (Classification procedure): the manuscript supplies no total sample size, no inter-rater agreement statistic (e.g., Cohen’s κ or percentage agreement), no explicit coding protocol distinguishing “direct” from indirect mention of a value, and no exclusion criteria. These omissions make the headline prevalence figures and the conference-versus-journal comparison impossible to evaluate.
  2. [§3.2] §3.2 (Value taxonomy): the decision to import an unmodified social-science value structure is presented without any SE-specific validation, pilot coding, or discussion of how relevance is operationalized for technical papers; because the boundary between relevant and irrelevant is therefore unanchored, modest shifts in coding rules could materially change the reported proportions.
minor comments (2)
  1. [Results] Table 1 (or equivalent results table) should report the raw counts alongside the percentages so readers can judge the absolute scale of the “small proportion” claim.
  2. [Abstract] The abstract would benefit from a single sentence stating the total number of papers examined and the number of coders.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. We address each major comment below and indicate the revisions that will be made to improve methodological transparency.

read point-by-point responses

  1. Referee: [§3] §3 (Classification procedure): the manuscript supplies no total sample size, no inter-rater agreement statistic (e.g., Cohen’s κ or percentage agreement), no explicit coding protocol distinguishing “direct” from indirect mention of a value, and no exclusion criteria. These omissions make the headline prevalence figures and the conference-versus-journal comparison impossible to evaluate.

    Authors: We agree that these details are necessary for readers to evaluate the reported prevalence figures and comparisons. The total sample size of publications examined, inter-rater agreement statistics, explicit coding protocol (including the distinction between direct and indirect mentions), and exclusion criteria will be added to §3 in the revised manuscript. revision: yes

  2. Referee: [§3.2] §3.2 (Value taxonomy): the decision to import an unmodified social-science value structure is presented without any SE-specific validation, pilot coding, or discussion of how relevance is operationalized for technical papers; because the boundary between relevant and irrelevant is therefore unanchored, modest shifts in coding rules could materially change the reported proportions.

    Authors: The taxonomy was adopted because it is a well-established and validated instrument from the social sciences that offers a comprehensive set of human values. We acknowledge that the current manuscript does not include SE-specific validation steps or a detailed operationalization of relevance for technical papers. In the revision we will expand §3.2 with an explicit discussion of how relevance was operationalized, including any pilot coding performed, to better anchor the classification boundaries. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical classification study with external taxonomy

full rationale

The paper is a literature classification exercise that counts how many SE publications directly address human values drawn from an external social-science taxonomy. No equations, fitted parameters, predictions, or derivations appear. The prevalence results are direct tallies from applying the imported structure; they do not reduce by construction to any quantity defined inside the paper or via self-citation chains. The central claim therefore remains independent of the authors' prior outputs and meets the criteria for a self-contained empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the appropriateness of the imported social-science value taxonomy for SE and on the validity of the manual relevance judgments; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption The value structure adopted from social sciences is suitable for classifying relevance of SE publications to human values.
    The abstract states that publications were classified against this structure without further justification of its fit to software engineering.

pith-pipeline@v0.9.0 · 5731 in / 1137 out tokens · 22402 ms · 2026-05-24T20:01:09.410189+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

49 extracted references · 49 canonical work pages

  1. [1]

    Huib Aldewereld, Virginia Dignum, and Yao-hua Tan. 2015. Design for values in software development. Handbook of Ethics, Values, and Technological Design: Sources, Theory, Values and Application Domains (2015), 831–845

    work page 2015

  2. [2]

    Julia Angwin, Jeff Larson, Lauren Kirchner, and Surya Mattu. 2016. Ma- chine Bias. https://www.propublica.org/article/machine-bias-risk-assessments- in-criminal-sentencing

    work page 2016

  3. [3]

    Nick Baker. 2019. Molly Russell: Instagram bans graphic self-harm images after suicide of UK teen. https://www.sbs.com.au/news/ molly-russell-instagram-bans-graphic-self-harm-images-after-suicide-of-uk-teen

    work page 2019

  4. [4]

    Antonia Bertolino, Antonello Calabrò, Francesca Lonetti, Eda Marchetti, and Breno Miranda. 2018. A categorization scheme for software engineering con- ference papers and its application. Journal of Systems and Software 137 (2018), 114–129. https://doi.org/10.1016/j.jss.2017.11.048

    work page doi:10.1016/j.jss.2017.11.048 2018

  5. [5]

    Corinne Cath, Sandra Wachter, Brent Mittelstadt, Mariarosaria Taddeo, and Luciano Floridi. 2018. Artificial intelligence and the ’good society’: the US, EU, and UK approach. Science and engineering ethics 24, 2 (2018), 505–528

    work page 2018

  6. [6]

    An-Shou Cheng and Kenneth R Fleischmann. 2010. Developing a meta-inventory of human values. In Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem-Volume 47. American Society for Information Science, 3

    work page 2010

  7. [7]

    Paresh Dave. 2018. Google bars uses of its artificial intelligence tech in weapons. https://www.reuters.com/article/us-alphabet-ai/google-bars-uses-of- its-artificial-intelligence-tech-in-weapons-idUSKCN1J32M7

    work page 2018

  8. [8]

    Roel van Dijk, Christophe Creeten, Jeroen van der Ham, and Jeroen van den Bos

    work page

  9. [9]

    In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering

    Model-driven software engineering in practice: privacy-enhanced filtering of network traffic. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 860–865

    work page 2017

  10. [10]

    Amitai Etzioni and Oren Etzioni. 2017. Incorporating ethics into artificial intelli- gence. The Journal of Ethics 21, 4 (2017), 403–418

    work page 2017

  11. [11]

    Maria Angela Ferrario, Will Simm, Stephen Forshaw, Adrian Gradinar, Mar- cia Tavares Smith, and Ian Smith. 2016. Values-first SE: research principles in practice. In Proceedings of the 38th International Conference on Software Engineer- ing Companion. ACM, 553–562

    work page 2016

  12. [12]

    Maria Angela Ferrario, Will Simm, Peter Newman, Stephen Forshaw, and Jon Whittle. 2014. Software Engineering for ’Social Good’: Integrating Action Re- search, Participatory Design, and Agile Development. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion 2014). ACM, New York, NY, USA, 520–523. https://doi...

    work page doi:10.1145/2591062 2014

  13. [13]

    Mary Flanagan, Daniel C Howe, and Helen Nissenbaum. 2005. Values at play: Design tradeoffs in socially-oriented game design. In Proceedings of the SIGCHI conference on human factors in computing systems . ACM, 751–760

    work page 2005

  14. [14]

    Batya Friedman. 1996. Value-sensitive design. interactions 3, 6 (1996), 16–23

    work page 1996

  15. [15]

    Batya Friedman and Peter H Kahn Jr. 2007. Human values, ethics, and design. In The human-computer interaction handbook . CRC Press, 1223–1248

    work page 2007

  16. [16]

    Glass, Iris Vessey, and Venkataraman Ramesh

    Robert L. Glass, Iris Vessey, and Venkataraman Ramesh. 2002. Research in software engineering: an analysis of the literature. Information and Software technology 44, 8 (2002), 491–506

    work page 2002

  17. [17]

    Preston Gralla. 2016. Amazon Prime and the racist algorithms. https://www.computerworld.com.au/article/599661/amazon-prime-racist- algorithms

    work page 2016

  18. [18]

    Tim Holmes, Elena Blackmore, Richard Hawkins, and Tom Wakeford. 2011. The common cause handbook. Public Interest Research Center

    work page 2011

  19. [19]

    John PA Ioannidis, Daniele Fanelli, Debbie Drake Dunne, and Steven N Goodman

    work page

  20. [20]

    PLoS biology 13, 10 (2015), e1002264

    Meta-research: evaluation and improvement of research methods and practices. PLoS biology 13, 10 (2015), e1002264

    work page 2015

  21. [21]

    J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159–174

    work page 1977

  22. [22]

    Michela Montesi and Patricia Lago. 2008. Software engineering article types: An analysis of the literature. Journal of Systems and Software 81, 10 (2008), 1694–1714

    work page 2008

  23. [23]

    Rodrigo Morales, Rubén Saborido, Foutse Khomh, Francisco Chicano, and Giu- liano Antoniol. 2018. Earmo: An energy-aware refactoring approach for mobile apps. IEEE Transactions on Software Engineering 44, 12 (2018), 1176–1206

    work page 2018

  24. [24]

    Davoud Mougouei, Harsha Perera, Waqar Hussain, Rifat Shams, and Jon Whittle

    work page

  25. [25]

    In Pro- ceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering - ESEC/FSE

    Operationalizing human values in software: a research roadmap. In Pro- ceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering - ESEC/FSE

    work page 2018

  26. [26]

    https://doi.org/10.1145/3236024.3264843

    780–784. https://doi.org/10.1145/3236024.3264843

    work page doi:10.1145/3236024.3264843

  27. [27]

    Mark O Riedl and Brent Harrison. 2016. Using stories to teach human values to artificial agents. In Workshops at the Thirtieth AAAI Conference on Artificial Intelligence

    work page 2016

  28. [28]

    Adrián Riesco and Kazuhiro Ogata. 2018. Prove It&Excl; Inferring Formal Proof Scripts from CafeOBJ Proof Scores.ACM Trans. Softw. Eng. Methodol.27, 2, Article 6 (July 2018), 32 pages. https://doi.org/10.1145/3208951

    work page doi:10.1145/3208951 2018

  29. [29]

    Milton Rokeach. 1973. The nature of human values. Free press

    work page 1973

  30. [30]

    Justin Sablich. 2017. ’Price Gouging’ and Hurricane Irma: What Happened and What to Do. https://www.nytimes.com/2017/09/17/travel/price-gouging- hurricane-irma-airlines.html

    work page 2017

  31. [31]

    Shalom H Schwartz. 1992. Universals in the content and structure of values: The- oretical advances and empirical tests in 20 countries. In Advances in experimental social psychology. Vol. 25. Elsevier, 1–65

    work page 1992

  32. [32]

    Shalom H Schwartz. 1994. Are there universal aspects in the structure and contents of human values? Journal of social issues 50, 4 (1994), 19–45

    work page 1994

  33. [33]

    Shalom H Schwartz. 2005. Basic human values: Their content and structure across countries. Valores e comportamento nas organizações (2005), 21–55

    work page 2005

  34. [34]

    Shalom H Schwartz. 2006. Les valeurs de base de la personne: théorie, mesures et applications. Revue française de sociologie 47, 4 (2006), 929–968

    work page 2006

  35. [35]

    Shalom H Schwartz. 2007. Basic human values: Theory, methods, and application. Risorsa Uomo (2007)

    work page 2007

  36. [36]

    Schwartz

    Shalom H. Schwartz. 2012. An Overview of the Schwartz Theory of Basic Values. Online Readings in Psychology and Culture 2, 1 (2012), 12–13. https://doi.org/10. 9707/2307-0919.1116

    work page arXiv 2012

  37. [37]

    Shalom H Schwartz and Klaus Boehnke. 2004. Evaluating the structure of human values with confirmatory factor analysis. Journal of research in personality 38, 3 (2004), 230–255

    work page 2004

  38. [38]

    Mary Shaw. 2003. Writing good software engineering research papers. InSoftware Engineering, 2003. Proceedings. 25th International Conference on . IEEE, 726–736

    work page 2003

  39. [39]

    Dag IK Sjøberg, Jo Erskine Hannay, Ove Hansen, Vigdis By Kampenes, Amela Karahasanovic, N-K Liborg, and Anette C Rekdal. 2005. A survey of controlled experiments in software engineering. IEEE transactions on software engineering 31, 9 (2005), 733–753

    work page 2005

  40. [40]

    David Smith. 2018. https://www.theguardian.com/technology/2018/apr/11/ zuckerberg-hearing-facebook-tracking-questions-house-back-foot. Zuckerberg put on back foot as House grills Facebook CEO over user tracking

    work page 2018

  41. [41]

    Igor Steinmacher, Tayana Uchoa Conte, Christoph Treude, and Marco Aurélio Gerosa. 2016. Overcoming Open Source Project Entry Barriers with a Portal for Newcomers. In Proceedings of the 38th International Conference on Software Engineering (ICSE ’16). ACM, New York, NY, USA, 273–284. https://doi.org/10. 1145/2884781.2884806

    work page arXiv 2016

  42. [42]

    Klaas-Jan Stol and Brian Fitzgerald. 2015. A holistic overview of software engi- neering research strategies. In Proceedings of the Third International Workshop on Conducting Empirical Studies in Industry . IEEE Press, 47–54

    work page 2015

  43. [43]

    Tarja Systä, Maarit Harsu, and Kai Koskimies. 2012. Inbreeding in software engineering conferences

    work page 2012

  44. [44]

    Sarah Thew and Alistair Sutcliffe. 2018. Value-based requirements engineering: method and experience. Requirements Engineering 23, 4 (2018), 443–464. OVIS, Monash University, Australia H. Perera et al

    work page 2018

  45. [45]

    Bogdan Vasilescu, Alexander Serebrenik, Tom Mens, Mark GJ van den Brand, and Ekaterina Pek. 2014. How healthy are software engineering conferences? Science of Computer Programming 89 (2014), 251–272

    work page 2014

  46. [46]

    Iris Vessey, Venkataraman Ramesh, and Robert L Glass. 2002. Research in infor- mation systems: An empirical study of diversity in the discipline and its journals. Journal of Management Information Systems 19, 2 (2002), 129–174

    work page 2002

  47. [47]

    Roel Wieringa, Neil Maiden, Nancy Mead, and Colette Rolland. 2006. Require- ments engineering paper classification and evaluation criteria: a proposal and a discussion. Requirements Engineering 11, 1 (2006), 102–107

    work page 2006

  48. [48]

    Carmen Zannier, Grigori Melnik, and Frank Maurer. 2006. On the success of empirical studies in the international conference on software engineering. In Proceedings of the 28th international conference on Software engineering . ACM, 341–350

    work page 2006

  49. [49]

    Marvin V Zelkowitz and Dolores Wallace. 1997. Experimental validation in software engineering. Information and Software Technology 39, 11 (1997), 735– 743

    work page 1997