pith. sign in

arxiv: 1906.11351 · v1 · pith:TAEJOQSOnew · submitted 2019-06-26 · 💻 cs.SE

Software Engineering Research Community Viewpoints on Rapid Reviews

Bruno Cartaxo , Gustavo Pinto , Baldoino Fonseca , M\'arcio Ribeiro , Pedro Pinheiro , Sergio Soares , Maria Teresa Baldassarre This is my paper

Pith reviewed 2026-05-25 15:04 UTC · model grok-4.3

classification 💻 cs.SE
keywords rapid reviewsQ-methodologysoftware engineering researchresearcher viewpointsevidence-based software engineeringsystematic reviewsfactors
0
0 comments X

The pith

The software engineering research community holds four distinct viewpoints on using rapid reviews to deliver evidence to practitioners.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper applies Q-methodology to map software engineering researchers' attitudes toward rapid reviews, a method already used in health fields to speed up delivery of relevant evidence to practice. Thirty-seven researchers rated their agreement with fifty opinion statements, and the analysis extracted four factors that group their responses. One factor collects researchers who remain undecided and want more proof of value. A second is favorable overall but insists on minimum standards being set first. A third is skeptical and stresses that only high-quality evidence matters. The fourth is pragmatic and sees rapid reviews as usable when context and constraints allow. A sympathetic reader cares because the study also surfaces shared ground across all four factors, such as the belief that both rapid reviews and systematic reviews can be executed well or poorly.

Core claim

By inviting 37 researchers to sort 50 statements about rapid reviews according to their level of agreement, the study extracts four factors. Factor A captures undecided researchers who need additional evidence before they would use rapid reviews. Factor B researchers are generally positive yet emphasize the need to define minimum standards. Factor C researchers are more skeptical and reinforce the importance of high-quality evidence. Factor D researchers hold a pragmatic view that rapid reviews can be applied depending on the specific context and constraints faced by practitioners. All four factors agree that both rapid reviews and systematic reviews can be conducted either poorly or well.

What carries the argument

Q-methodology, which factors researchers' numerical ratings of opinion statements to surface shared viewpoints as distinct factors.

If this is right

  • All viewpoints converge on the idea that review quality depends on execution, so efforts to improve practice should focus on guidelines rather than choosing between rapid reviews and systematic reviews.
  • The call for minimum standards from the positive group suggests that establishing such standards could increase acceptance of rapid reviews.
  • The pragmatic factor implies that rapid reviews may see earlier adoption in settings where time or resources are limited.
  • The undecided factor indicates that additional validation studies comparing rapid reviews to traditional methods would be needed to shift opinion.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • These four factors could be used as a diagnostic tool to segment the community and tailor communication about rapid reviews accordingly.
  • Longitudinal tracking of researchers aligned with each factor might reveal whether viewpoint predicts actual use of rapid reviews in published work.
  • The shared agreement on execution quality points to a possible next step of developing concrete checklists that apply equally to rapid and systematic reviews.

Load-bearing premise

The sample of 37 invited researchers and the set of 50 opinion statements are sufficient to represent and distinguish the main viewpoints held by the broader software engineering research community.

What would settle it

A follow-up study that recruits a substantially larger and more diverse group of researchers or uses a different collection of statements and fails to recover the same four factors would indicate that the identified viewpoints do not generalize.

Figures

Figures reproduced from arXiv: 1906.11351 by Baldoino Fonseca, Bruno Cartaxo, Gustavo Pinto, M\'arcio Ribeiro, Maria Teresa Baldassarre, Pedro Pinheiro, Sergio Soares.

Figure 2
Figure 2. Figure 2: 3rd step: Classifying statements in three levels of agreement. The [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: Q-SORT structure. The number under parentheses represents the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: 4th step: Ranking statements in seven agreement levels according to [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Factor analysis steps (rounded rectangles), and decisions made during [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
read the original abstract

Background: One of the most important current challenges of Software Engineering (SE) research is to provide relevant evidence to practice. In health related fields, Rapid Reviews (RRs) have shown to be an effective method to achieve that goal. However, little is known about how the SE research community perceives the potential applicability of RRs. Aims: The goal of this study is to understand the SE research community viewpoints towards the use of RRs as a means to provide evidence to practitioners. Method: To understand their viewpoints, we invited 37 researchers to analyze 50 opinion statements about RRs, and rate them according to what extent they agree with each statement. Q-Methodology was employed to identify the most salient viewpoints, represented by the so called factors. Results: Four factors were identified: Factor A groups undecided researchers that need more evidence before using RRs; Researchers grouped in Factor B are generally positive about RRs, but highlight the need to define minimum standards; Factor C researchers are more skeptical and reinforce the importance of high quality evidence; Researchers aligned to Factor D have a pragmatic point of view, considering RRs can be applied based on the context and constraints faced by practitioners. Conclusions: In conclusion, although there are opposing viewpoints, there are also some common grounds. For example, all viewpoints agree that both RRs and Systematic Reviews can be poorly or well conducted.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 1 minor

Summary. The paper reports a Q-methodology study in which 37 software engineering researchers rated 50 opinion statements about Rapid Reviews (RRs). Four factors were extracted and interpreted as: Factor A (undecided researchers needing more evidence), Factor B (generally positive but calling for minimum standards), Factor C (skeptical, stressing high-quality evidence), and Factor D (pragmatic, context- and constraint-dependent). The conclusions highlight both opposing viewpoints and common ground, such as agreement that RRs and systematic reviews can be conducted well or poorly.

Significance. If the factor structure is robust, the study offers a structured map of researcher attitudes toward RRs that could inform guidelines, training, and adoption strategies aimed at closing the research-practice gap in SE. The choice of Q-methodology is well-suited to surfacing subjective viewpoints and the identification of shared ground across factors is a constructive finding.

major comments (3)
  1. [Method] Method section (as described in the abstract): the selection process for the 37 invited researchers is not described, including any sampling frame, invitation criteria, or steps taken to ensure coverage of experience, geography, or subfields. Because the central claim is that the four factors represent salient community viewpoints, this omission is load-bearing; without it the factors cannot be distinguished from artifacts of a convenience sample.
  2. [Method] Method section: no information is supplied on how the 50 opinion statements were generated, sourced, or validated to ensure they exhaustively sample the discourse on RRs. Q-methodology results are only as generalizable as the statement set; absent this justification the four-factor solution may simply reflect the particular statements chosen rather than stable community viewpoints.
  3. [Results] Results section: the paper states that four factors were identified but supplies no details on factor extraction criteria (e.g., eigenvalues, variance explained), loading thresholds, or any post-hoc checks for sorting bias or respondent consistency. These omissions prevent assessment of whether the reported factor interpretations are statistically supported.
minor comments (1)
  1. [Abstract] Abstract: the phrasing 'we invited 37 researchers' could be clarified to indicate whether participation was by invitation only or open, to help readers gauge selection effects.

Simulated Author's Rebuttal

3 responses · 0 unresolved

Thank you for the detailed and constructive review of our manuscript. We have carefully considered each of the major comments and provide point-by-point responses below. We plan to make revisions to address the identified omissions in the method and results descriptions.

read point-by-point responses

  1. Referee: [Method] Method section (as described in the abstract): the selection process for the 37 invited researchers is not described, including any sampling frame, invitation criteria, or steps taken to ensure coverage of experience, geography, or subfields. Because the central claim is that the four factors represent salient community viewpoints, this omission is load-bearing; without it the factors cannot be distinguished from artifacts of a convenience sample.

    Authors: We agree that the participant selection process requires additional detail to support claims about community viewpoints. In the revised manuscript we will expand the Method section to describe the purposive sampling frame: researchers were identified from recent publications and conference activity in evidence-based software engineering and invited by email, with the goal of including a range of experience levels, subfields, and geographic locations. We will also explicitly note the limitations of this approach. revision: yes

  2. Referee: [Method] Method section: no information is supplied on how the 50 opinion statements were generated, sourced, or validated to ensure they exhaustively sample the discourse on RRs. Q-methodology results are only as generalizable as the statement set; absent this justification the four-factor solution may simply reflect the particular statements chosen rather than stable community viewpoints.

    Authors: We acknowledge the need for explicit justification of the statement set. The 50 statements were compiled from a review of the rapid-review literature in health sciences and software engineering together with team discussions to capture divergent opinions. In the revision we will add a subsection detailing the concourse sampling, statement selection criteria, and any pilot validation steps used to ensure reasonable coverage of the discourse on RRs. revision: yes

  3. Referee: [Results] Results section: the paper states that four factors were identified but supplies no details on factor extraction criteria (e.g., eigenvalues, variance explained), loading thresholds, or any post-hoc checks for sorting bias or respondent consistency. These omissions prevent assessment of whether the reported factor interpretations are statistically supported.

    Authors: We agree that the statistical details of the factor solution must be reported for transparency and replicability. The revised Results section will include the eigenvalues and variance explained by each factor, the loading threshold applied, and any post-extraction checks performed for respondent consistency or sorting anomalies, following standard Q-methodology reporting practices. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical Q-methodology application with no derivations or self-referential reductions.

full rationale

The paper reports results from applying standard Q-methodology to primary survey data (37 researchers rating 50 statements). Factors are extracted directly from participant ratings via established Q-method procedures; no equations, fitted parameters, predictions, or derivations are present that could reduce to inputs by construction. No self-citation chains or ansatzes underpin the central claims. The work is self-contained empirical reporting, with methodological choices (sample, statements) open to external critique on validity but not creating definitional or fitted circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the assumption that Q-methodology applied to this sample and statement set yields stable, representative factors; no free parameters, invented entities, or additional axioms beyond standard survey methodology are stated in the abstract.

axioms (1)
  • domain assumption Q-methodology can reliably extract distinct shared viewpoints from participants' ratings of opinion statements
    The entire factor analysis in the Results section depends on this statistical premise.

pith-pipeline@v0.9.0 · 5792 in / 1262 out tokens · 26150 ms · 2026-05-25T15:04:22.554631+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 49 canonical work pages

  1. [1]

    H. Abdi. Factor rotations in factor analyses. Encyclopedia for Research Methods for the Social Sciences. Sage: Thousand Oaks, CA , pages 792– 795, 2003

    work page 2003

  2. [2]

    Armatas, T

    C. Armatas, T. Venn, and A. Watson. Understanding social–ecological vulnerability with q-methodology: a case study of water-based ecosys- tem services in wyoming, usa. Sustainability Science , 12(1):105–121, 2017

    work page 2017

  3. [3]

    Badampudi, C

    D. Badampudi, C. Wohlin, and T. Gorschek. Contextualizing research evidence through knowledge translation in software engineering. In Proceedings of the Evaluation and Assessment on Software Engineering, EASE ’19, pages 306–311, New York, NY , USA, 2019. ACM

    work page 2019

  4. [4]

    Baker, J

    R. Baker, J. Wildman, H. Mason, and C. Donaldson. Q-ing for healtha new approach to eliciting the public’s views on health care resource allocation. Health economics, 23(3):283–297, 2014

    work page 2014

  5. [5]

    J. Batten. Comment on editorial literature reviews as a research strategy. Journal of School Nursing , 2012

    work page 2012

  6. [6]

    Begel and T

    A. Begel and T. Zimmermann. Analyze this! 145 questions for data scientists in software engineering. In 36th International Conference on Software Engineering, ICSE ’14, Hyderabad, India - May 31 - June 07, 2014, pages 12–13, 2014

    work page 2014

  7. [7]

    L. Best, A. Stevens, and D. ColinJones. Rapid and responsive health technology assessment: The development and evaluation process in the south and west region of england. Journal of Clinical Effectiveness , 1997

    work page 1997

  8. [8]

    Borges, W

    A. Borges, W. Ferreira, E. Barreiros, A. Almeida, L. Fonseca, E. Teix- eira, D. Silva, A. Alencar, and S. Soares. Support mechanisms to conduct empirical studies in software engineering. In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM ’14, pages 50:1–50:4, New York, NY , USA,

    work page

  9. [9]

    Borges, W

    A. Borges, W. Ferreira, E. Barreiros, A. Almeida, L. Fonseca, E. Teix- eira, D. Silva, A. Alencar, and S. Soares. Support mechanisms to conduct empirical studies in software engineering: A systematic mapping study. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering , EASE ’15, pages 22:1–22:14, 2015

    work page 2015

  10. [10]

    S. R. Brown. Political subjectivity: Applications of Q methodology in political science. Yale University Press, 1980

    work page 1980

  11. [11]

    S. R. Brown. A primer on q methodology. Operant subjectivity , 16(3/4):91–138, 1993

    work page 1993

  12. [12]

    Budgen, S

    D. Budgen, S. M. Charters, M. Turner, P. Brereton, B. A. Kitchenham, and S. G. Linkman. Investigating the applicability of the evidence-based paradigm to software engineering. In Proceedings of the 2006 interna- tional workshop on Workshop on interdisciplinary software engineering research, WISER 2006, Shanghai, China, May 20, 2006 , pages 7–14, 2006

    work page 2006

  13. [13]

    Cartaxo, G

    B. Cartaxo, G. Pinto, and S. Soares. The role of rapid reviews in supporting decision-making in software engineering practice. In Proceedings of the 22Nd International Conference on Evaluation and Assessment in Software Engineering 2018, EASE’18, pages 24–34, New York, NY , USA, 2018. ACM

    work page 2018

  14. [14]

    Cartaxo, G

    B. Cartaxo, G. Pinto, and S. Soares. Towards a model to transfer knowledge from software engineering research to practice. Information & Software Technology, 97:80–82, 2018

    work page 2018

  15. [15]

    Cartaxo, G

    B. Cartaxo, G. Pinto, E. Vieira, and S. Soares. Evidence briefings: Towards a medium to transfer knowledge from systematic reviews to practitioners. Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2016

    work page 2016

  16. [16]

    J. C. Carver, O. Dieste, N. A. Kraft, D. Lo, and T. Zimmermann. How practitioners perceive the relevance of esem research. In Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , ESEM ’16, pages 56:1–56:10, 2016

    work page 2016

  17. [17]

    D. S. Cruzes and T. Dyb. Research synthesis in software engineering: A tertiary study. Information and Software Technology , 53(5):440 – 455,

    work page

  18. [18]

    Special Section on Best Papers from {XP2010}

    work page

  19. [19]

    F. Q. da Silva, A. L. Santos, S. Soares, A. C. C. Frana, C. V . Monteiro, and F. F. Maciel. Six years of systematic literature reviews in software engineering: An updated tertiary study. Information and Software Technology, 53(9):899 – 913, 2011. Studying work practices in Global Software Engineering

    work page 2011

  20. [20]

    Davies and I

    B. Davies and I. Hodge. Exploring environmental perspectives in lowland agriculture: Aq methodology study in east anglia, uk.Ecological economics, 61(2-3):323–333, 2007

    work page 2007

  21. [21]

    T. Dyba, T. Dingsoyr, and G. K. Hanssen. Applying systematic reviews to diverse study types: An experience report. In Empirical Software Engineering and Measurement, 2007. ESEM 2007. First International Symposium on, pages 225–234. IEEE, 2007

    work page 2007

  22. [22]

    Dyb ˚a, B

    T. Dyb ˚a, B. A. Kitchenham, and M. Jørgensen. Evidence-based software engineering for practitioners. IEEE Software, 22(1):58–65, 2005

    work page 2005

  23. [23]

    J. M. Good. Introduction to william stephenson’s quest for a science of subjectivity. Psychoanalysis and history , 12(2):211–243, 2010

    work page 2010

  24. [24]

    D. Hailey. A preliminary survey on the influence of rapid health technology assessments. International journal of technology assessment in health care , 2009

    work page 2009

  25. [25]

    Hailey, P

    D. Hailey, P. Corabian, and W. Schneider. The use and impact of rapid health technology assessments. International journal of technology assessment in health care , 2000

    work page 2000

  26. [26]

    J. F. Hair, W. C. Black, B. J. Babin, and R. E. Anderson. Multivariate data analysis: Pearson new international edition, always learning, 2014

    work page 2014

  27. [27]

    Hassler et al

    E. Hassler et al. Outcomes of a community workshop to identify and rank barriers to the systematic literature review process. EASE, 2014

    work page 2014

  28. [28]

    S. E. Kelly, D. Moher, and T. J. Clifford. Expediting evidence synthesis for healthcare decision-making: exploring attitudes and perceptions towards rapid reviews using q methodology. PeerJ, 4:e2522, 2016

    work page 2016

  29. [29]

    B. A. Kitchenham. Evidence-based software engineering and systematic literature reviews. In Product-Focused Software Process Improvement, 7th International Conference, PROFES 2006, Amsterdam, The Nether- lands, June 12-14, 2006, Proceedings , page 3, 2006

    work page 2006

  30. [30]

    B. A. Kitchenham, D. Budgen, P. Brereton, and S. G. Linkman. Re- alising evidence-based software engineering. ACM SIGSOFT Software Engineering Notes, 30(4):1–3, 2005

    work page 2005

  31. [31]

    B. A. Kitchenham, T. Dyba, and M. Jorgensen. Evidence-based software engineering. In Proceedings of the 26th International Conference on Software Engineering , ICSE ’04, pages 273–281, Washington, DC, USA, 2004. IEEE Computer Society

    work page 2004

  32. [32]

    M. A. Lawani et al. Five shared decision-making tools in 5 months: use of rapid reviews to develop decision boxes for seniors living with dementia and their caregivers. Systematic reviews, 2017

    work page 2017

  33. [33]

    D. Lo, N. Nagappan, and T. Zimmermann. How practitioners perceive the relevance of software engineering research. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering , ESEC/FSE 2015, pages 415–425, 2015

    work page 2015

  34. [34]

    McGregor and J

    M. McGregor and J. M. Brophy. End-user involvement in health technology assessment (hta) development: a way to increase impact. International Journal of Technology Assessment in Health Care , 2005

    work page 2005

  35. [35]

    Moher, L

    D. Moher, L. Stewart, and P. Shekelle. All in the family: systematic reviews, rapid reviews, scoping reviews, realist reviews, and more. Systematic Reviews, 2015

    work page 2015

  36. [36]

    S. A. Mulaik. Foundations of factor analysis. Chapman and Hall/CRC, 2009

    work page 2009

  37. [37]

    M. Niazi. Do systematic literature reviews outperform informal literature reviews in the software engineering domain? an initial case study. Arabian Journal for Science and Engineering , 40(3):845–855, 2015

    work page 2015

  38. [38]

    J. B. Paige and K. H. Morin. Q-sample construction: A critical step for a q-methodological study. Western journal of nursing research, 38(1):96– 110, 2016

    work page 2016

  39. [39]

    E. Rogers. Diffusion of Innovations, 5th Edition . Free Press, 2003

    work page 2003

  40. [40]

    R. E. S. Santos and F. Q. B. d. Silva. Motivation to perform systematic reviews and their impact on software engineering practice. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, pages 292–295, 2013

    work page 2013

  41. [41]

    Taylor-Phillips, J

    S. Taylor-Phillips, J. Geppert, C. Stinton, K. Freeman, S. Johnson, H. Fraser, P. Sutcliffe, and A. Clarke. Comparison of a full systematic review versus rapid review approaches to assess a newborn screening test for tyrosinemia type 1. Research Synthesis Methods , 2017

    work page 2017

  42. [42]

    Tricco, E

    A. Tricco, E. V . Langlois, and S. E. Straus. Rapid reviews to strengthen health policy and systems: a practical guide. World Health Organization. http://apps. who. int/iris/bitstream/10665/258698/1/9789241512763- eng. pdf, 2017

    work page 2017

  43. [43]

    A. C. Tricco, J. Antony, W. Zarin, L. Strifler, M. Ghassemi, J. Ivory, L. Perrier, B. Hutton, D. Moher, and S. E. Straus. A scoping review of rapid review methods. BMC Medicine, 2015

    work page 2015

  44. [44]

    Watts and P

    S. Watts and P. Stenner. Doing q methodology: theory, method and interpretation. Qualitative research in psychology , 2(1):67–91, 2005

    work page 2005

  45. [45]

    Winter, S

    E. Winter, S. Forshaw, and M. A. Ferrario. Measuring human values in software engineering. In Proceedings of the 12th ACM/IEEE Interna- tional Symposium on Empirical Software Engineering and Measurement, ESEM ’18, pages 48:1–48:4, 2018

    work page 2018

  46. [46]

    A. Zabala. Package qmethod, 2018

    work page 2018

  47. [47]

    Zabala and U

    A. Zabala and U. Pascual. Bootstrapping q methodology to improve the understanding of human perspectives. PloS one, 11(2):e0148087, 2016

    work page 2016

  48. [48]

    Zechmeister and I

    I. Zechmeister and I. Schumacher. The impact of health technology assessment reports on decision making in austria. International journal of technology assessment in health care , 2012

    work page 2012

  49. [49]

    Y . Zhou, H. Zhang, X. Huang, S. Yang, M. A. Babar, and H. Tang. Quality assessment of systematic reviews in software engineering: A tertiary study. In Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering , EASE ’15, pages 14:1–14:14, New York, NY , USA, 2015. ACM

    work page 2015