pith. sign in

arxiv: 2605.03257 · v1 · submitted 2026-05-05 · 💻 cs.SE

Operationalizing Software Engineering Theories for Practical Validation

Isaque Alves , Fabio Kon , Jessica Diaz , Carla Rocha This is my paper

Pith reviewed 2026-05-07 16:05 UTC · model grok-4.3

classification 💻 cs.SE
keywords operationalizationsoftware engineering theoryempirical validationtheory buildingDevOps team taxonomiesnon-causal hypothesesmethodological guidelinesocio-technical systems
0
0 comments X

The pith

A systematic procedure turns abstract software engineering theories into measurable variables, indicators, and testable hypotheses.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a structured way to translate high-level ideas in software engineering theories into concrete, measurable parts ready for real-world testing. This matters in an applied field where theories need to guide decisions about teams, processes, and systems rather than stay at the level of general concepts. The procedure defines variables, picks indicators, and creates hypotheses in a way that keeps a traceable link from the original idea to the data that would check it. A sympathetic reader would value this because it makes theoretical work more likely to produce reliable, usable results for practitioners. The steps are shown in detail through one example theory about how DevOps teams are organized.

Core claim

The paper establishes a systematic procedure for the operationalization phase that translates abstract concepts in software engineering theories into measurable variables and indicators, then derives non-causal hypotheses to enable empirical validation. This produces a replicable methodological guideline with a transparent chain of evidence from theory to testable elements, shown through its application to the DevOps Team Taxonomies Theory.

What carries the argument

The systematic procedure for operationalization that defines variables from abstract concepts, selects indicators, and derives non-causal hypotheses.

If this is right

  • Theories gain both rigor and direct practical utility once translated through the defined steps.
  • Researchers receive a replicable guideline that keeps a clear record from original concept to data collection.
  • Empirical validation becomes more feasible for socio-technical theories, yielding actionable guidance for practitioners.
  • Application to existing theories such as DevOps team taxonomies demonstrates how the steps produce testable elements.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Adoption across studies could reduce differences in how software engineering theories are turned into evidence.
  • The approach might shorten the distance between academic descriptions of development practices and day-to-day decisions in teams.
  • Trying the same steps on additional theories would show whether the procedure works beyond the single illustration given.

Load-bearing premise

That extending an existing operationalization framework to include non-causal hypotheses will preserve rigor and provide a transparent chain of evidence without introducing selection bias in socio-technical software engineering contexts.

What would settle it

If researchers apply the procedure to derive variables and hypotheses from the DevOps Team Taxonomies Theory and then collect data to test them, but find the hypotheses cannot be clearly measured or the evidence chain breaks, the procedure would not achieve its stated goal.

Figures

Figures reproduced from arXiv: 2605.03257 by Carla Rocha, Fabio Kon, Isaque Alves, Jessica Diaz.

Figure 1
Figure 1. Figure 1: Methodological framework for theory-building. view at source ↗
Figure 2
Figure 2. Figure 2: Operationalization requires two steps: (A) defining concept variables and indicators, and (B) translating propositions view at source ↗
Figure 3
Figure 3. Figure 3: T3 instantiation of an Enabler (Platform) Team. The team leverages technology to automate workflows and deliver view at source ↗
Figure 4
Figure 4. Figure 4: The path from concepts to testable hypotheses (via view at source ↗
read the original abstract

Software Engineering often adapts theory-building frameworks from the social sciences to address socio-technical complexity. The key phases of the theory-building process are conceptual development, operationalization, testing, and application. Operationalization translates abstract concepts into measurable elements for empirical validation. This phase is essential for delivering the practical utility required by an applied science like Software Engineering. We propose a systematic procedure for the operationalization phase that bridges the gap between abstract concepts and empirical validation, ensuring the resulting theory is both rigorous and practically useful. We extend the operationalization framework proposed by Sj{\o}berg et al. and formulate non-causal hypotheses following Dubin's approach. Our procedure defines variables, selects indicators, and systematically derives hypotheses. We present a replicable, evidence-based methodological guideline that preserves a clear chain of evidence and supports practical validation. We illustrate the procedure using the DevOps Team Taxonomies Theory. This guideline provides a transparent chain of evidence from theory to testable elements, empowering researchers to ground theoretical advancements in empirical evidence and deliver actionable insights for practitioners.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper proposes a systematic procedure for the operationalization phase of theory-building in software engineering. It extends the operationalization framework from Sjørberg et al. by incorporating Dubin's approach to non-causal hypotheses. The procedure covers defining variables, selecting indicators, and systematically deriving hypotheses, with the goal of creating a replicable, evidence-based guideline that maintains a transparent chain of evidence from abstract concepts to testable elements. The proposal is illustrated using the DevOps Team Taxonomies Theory.

Significance. If the procedure is sound and adopted, it would offer software engineering researchers a structured method to translate abstract theories into measurable components suitable for empirical validation. This addresses a recognized challenge in socio-technical SE research by emphasizing non-causal hypotheses, potentially improving the practical utility of theories. The single illustration demonstrates applicability but leaves the broader impact dependent on future adoption and testing.

minor comments (2)
  1. [Abstract] Abstract: The repeated emphasis on 'rigorous and practically useful' and 'transparent chain of evidence' could be streamlined to avoid redundancy while preserving the core message.
  2. [Illustration] The illustration section would benefit from a summary table mapping abstract concepts to specific variables, indicators, and derived hypotheses to enhance clarity and replicability.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for the positive assessment of our manuscript and the recommendation for minor revision. The provided summary correctly captures the core contribution: a replicable guideline extending Sjørberg et al.'s operationalization framework by incorporating Dubin's non-causal hypotheses, with variables, indicators, and hypothesis derivation illustrated via the DevOps Team Taxonomies Theory. No major comments were specified in the report.

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper proposes a methodological procedure for operationalizing SE theories by extending the external framework from Sjørberg et al. and incorporating Dubin's approach to non-causal hypotheses. The derivation chain defines variables, selects indicators, and derives hypotheses in a replicable guideline, illustrated on the DevOps Team Taxonomies Theory. No self-definitional reductions, fitted inputs renamed as predictions, or load-bearing self-citations appear; the central claim remains a conceptual extension with an independent chain of evidence from abstract concepts to testable elements, self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that social-science theory-building phases transfer directly to SE without loss of validity and that the chosen extensions maintain rigor; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Theory-building phases (conceptual development, operationalization, testing, application) from social sciences apply to software engineering socio-technical systems.
    Invoked as the foundation for adapting and extending the frameworks of Sjørberg et al. and Dubin.

pith-pipeline@v0.9.0 · 5472 in / 1292 out tokens · 53420 ms · 2026-05-07T16:05:45.031138+00:00 · methodology

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

34 extracted references · 2 canonical work pages

  1. [1]

    Pritha Bhandari. 2022. Operationalization. A Guide with Examples, Pros & Cons. https://www.scribbr.com/methodology/operationalization/

    2022

  2. [2]

    2014.Constructing Grounded Theory

    Kathy Charmaz. 2014.Constructing Grounded Theory. Sage 2nd Ed., London

    2014

  3. [3]

    1990.Basics of Qualitative Research: Grounded Theory Procedures and Techniques

    Juliet Corbin and Anselm Strauss. 1990.Basics of Qualitative Research: Grounded Theory Procedures and Techniques. SAGE Publication, London

    1990

  4. [4]

    Jessica Díaz, Jorge Pérez, Isaque Alves, Fabio Kon, Leonardo Leite, Paulo Meirelles, and Carla Rocha. 2024. Harmonizing DevOps taxonomies—A grounded theory study.Journal of Systems and Software208 (2024), 111908

    2024

  5. [5]

    1978.Theory building: a practical guide to the construction and testing of theoretical models

    Robert Dubin. 1978.Theory building: a practical guide to the construction and testing of theoretical models. Free Press, New York

    1978

  6. [6]

    2018.Accelerate: The science of lean software and devops: Building and scaling high performing technology organizations

    Nicole Forsgren, Jez Humble, and Gene Kim. 2018.Accelerate: The science of lean software and devops: Building and scaling high performing technology organizations. IT Revolution, Portland

    2018

  7. [7]

    Dennis A Gioia and Evelyn Pitre. 1990. Multiparadigm perspectives on theory building.Academy of management review15, 4 (1990), 584–602

    1990

  8. [8]

    1967.The Discovery of Grounded Theory: Strategies for Qualitative Research

    Barney Glaser and Anselm Strauss. 1967.The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine de Gryter, New York

    1967

  9. [9]

    Shirley Gregor. 2006. The nature of theory in information systems.MIS quarterly 30, 3 (2006), 611–642

    2006

  10. [10]

    Rashina Hoda. 2022. Socio-Technical Grounded Theory for Software Engineering. IEEE Trans. Softw. Eng.48, 10 (oct 2022), 3808–3832. doi:10.1109/TSE.2021.3106280

    work page doi:10.1109/tse.2021.3106280 2022

  11. [11]

    2016.An enquiry concerning human understanding

    David Hume. 2016.An enquiry concerning human understanding. Routledge, New York. 183–276 pages

    2016

  12. [12]

    2019.Theory construction and model-building skills: A practical guide for social scientists

    James Jaccard and Jacob Jacoby. 2019.Theory construction and model-building skills: A practical guide for social scientists. Guilford publications, Nova York

    2019

  13. [13]

    Leonardo Leite, Nelson Lago, Claudia Melo, Fabio Kon, and Paulo Meirelles

  14. [14]

    A theory of organizational structures for development and infrastructure professionals.IEEE Transactions on Software Engineering49, 4 (2022), 1898–1911

    2022

  15. [16]

    Leonardo Leite, Gustavo Pinto, Fabio Kon, and Paulo Meirelles. 2021. The or- ganization of software teams in the quest for continuous delivery: A grounded theory approach.Information and Software Technology139 (2021), 106672

    2021

  16. [17]

    Daniel López-Fernández, Jessica Díaz, Javier García, Jorge Pérez, and Ángel González-Prieto. 2021. DevOps team structures: Characterization and implica- tions.IEEE transactions on software engineering48, 10 (2021), 3716–3736

    2021

  17. [18]

    Welder Pinheiro Luz, Gustavo Pinto, and Rodrigo Bonifácio. 2019. Adopting DevOps in the real world: A theory, a model, and a case study.Journal of Systems and Software157 (2019), 110384

    2019

  18. [19]

    Susan A. Lynham. 2002. The General Method of Theory-Building Research in Applied Disciplines.Advances in Developing Human Resources4, 3 (2002), 221–241

    2002

  19. [20]

    Macarthy and Julian M

    Ruth W. Macarthy and Julian M. Bass. 2020. An Empirical Taxonomy of DevOps in Practice. In2020 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). IEEE, New Jersey, 221–228

    2020

  20. [21]

    2011.Abduction, reason and science: Processes of discovery and explanation

    Lorenzo Magnani. 2011.Abduction, reason and science: Processes of discovery and explanation. Springer Science & Business Media, New York

    2011

  21. [22]

    Niall Richard Murphy, Liz Fong-Jones, Betsy Beyer, Todd Underwood, Laura Nolan, and Dave Rensin. 2018. Site Reliability Engineering book, Chapter 1 - How SRE Relates to DevOps. https://sre.google/workbook/how-sre-relates/

    2018

  22. [23]

    Kristian Nybom, Jens Smeds, and Ivan Porres. 2016. On the impact of mixing re- sponsibilities between devs and ops. InInternational Conference on Agile Software Development. Springer, Springer International, Publishing, Cham, 131–143

    2016

  23. [24]

    Jorge Pérez, Jessica Díaz, Ángel González-Prieto, and Sergio Gil-Borrás. 2024. Theory building for empirical software engineering in qualitative research: Op- erationalization.arXiv preprint arXiv:2412.023841-22 (2024), 1898–1911

    work page arXiv 2024

  24. [25]

    1959.The logic of scientific discovery

    Karl Popper. 1959.The logic of scientific discovery. Routledge, New York

    1959

  25. [26]

    Puppet and CircleCI. 2020. 2020 State of DevOps Report. https://www2.circleci. com/2020-state-of-devops-report.html

    2020

  26. [27]

    Paul Ralph. 2018. Toward methodological guidelines for process theories and taxonomies in software engineering.IEEE Transactions on Software Engineering 45, 7 (2018), 712–735

    2018

  27. [28]

    Mojtaba Shahin, Mansooreh Zahedi, Muhammad Ali Babar, and Liming Zhu

  28. [29]

    InProceedings of the 21st International Conference on evaluation and assessment in software engineering

    Adopting continuous delivery and deployment: Impacts on team struc- tures, collaboration and responsibilities. InProceedings of the 21st International Conference on evaluation and assessment in software engineering. Association for Computing Machinery, New York, 384–393

  29. [30]

    Dag IK Sjøberg and Gunnar Rye Bergersen. 2022. Construct validity in software engineering.IEEE Transactions on Software Engineering49, 3 (2022), 1374–1396

    2022

  30. [31]

    Sjøberg, Tore Dybå, Bente Anda, and Jo Erskine Hannay

    Dag I.K. Sjøberg, Tore Dybå, Bente Anda, and Jo Erskine Hannay. 2008.Building Theories in Software Engineering. Springer London, London, 312–336

    2008

  31. [32]

    2019.Team topologies: organizing business and technology teams for fast flow

    Matthew Skelton and Manuel Pais. 2019.Team topologies: organizing business and technology teams for fast flow. It Revolution, Portland

    2019

  32. [33]

    2014.Abductive reasoning

    Douglas Walton. 2014.Abductive reasoning. University of Alabama, Alabama

    2014

  33. [34]

    Xin Zhou, Huang Huang, He Zhang, Xin Huang, Dong Shao, and Chenxin Zhong

  34. [35]

    In2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)

    A Cross-Company Ethnographic Study on Software Teams for DevOps and Microservices: Organization, Benefits, and Issues. In2022 IEEE/ACM 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, Association for Computing Machinery, New York, 1–10