Why Should the Q-method be Integrated Into the Design Science Research? A Systematic Mapping Study

Irawan Nurhas; Jan Pawlowski; Stefan Geisler

REVIEW 4 major objections 4 minor 14 references

Why Should the Q-method be Integrated Into the Design Science Research? A Systematic Mapping Study

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The Q-method can support every main stage of design science research in information systems, a systematic mapping of 45 studies concludes.

desk verdict A useful first systematic map of Q-method use in IS, but the headline claim that Q-method supports every DSR stage rests on the authors' own classification of papers that never used DSR. read the letter →

arxiv 1908.05203 v2 pith:WO6NDKOX submitted 2019-08-14 cs.HC cs.CY

Irawan Nurhas , Stefan Geisler , Jan Pawlowski This is my paper · ORCID

classification cs.HCcs.CY

keywords Q-methodologyQ-sortDesignScienceResearchInformationSystemssystematicmappingstudysubjectivitymeasurementsystemevaluationDSRprocess

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that the Q-method, a sorting technique that measures people's subjective viewpoints, can be integrated into every main stage of design science research (DSR) in information systems. The authors systematically mapped 45 studies that used the Q-method in information-systems research and classified each one by the DSR stage it could most plausibly support: problem identification, solution definition, or evaluation. A sympathetic reader would take the central claim to be that no DSR stage is out of reach for the Q-method, and that the method is especially useful for evaluating systems in the system analysis and design topic. This matters because design science depends on subjective human perception, and the Q-method offers a structured, mixed qualitative-quantitative way to capture it.

What carries the argument

The central object is the Q-method (also called Q-sort or Q-methodology), a procedure for measuring human subjectivity in which participants rank a set of statements, the concourse, into a forced-choice distribution, and factor analysis of the sorted cards reveals shared viewpoints. The other load-bearing component is the DSR process model, simplified into three main stages: problem identification and motivation, solution definition and development, and evaluation. The mapping machinery works by taking each of the 45 publications and classifying it according to which comparable DSR process the Q-method use most plausibly supports, along with the role of the Q-method, the sources of its statements, the IS topic, and the knowledge contribution. This classification produces the bubble plots and tables from which the paper argues that the Q-method is flexible enough to carry every main DSR phase, especially evaluation.

What would settle it

An independent audit of Table 1 could settle the claim: if raters re-read the 45 papers and find that a large share contain no activity matching the design-science stage they were assigned to, then the mapping loses its empirical foundation. A second test would be to run a design science project that uses the Q-method at all three stages and check whether each stage actually yields usable stage-specific output.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the Q-method can be used to support each main research stage of DSR processes and can serve as a useful tool to evaluate a system in the information-systems topic of system analysis and design. The mapping shows that the Q-method is used mostly as a supporting method rather than the main method, that it appears across all three main DSR stages when studies are reclassified by comparable process, and that evaluation-driven use dominates in system analysis and design. The authors also report that none of the 45 studies explicitly used DSR, which they read as a gap rather than a contradiction: the Q-method's flexibility in collecting statements and its ability to reach consensus on user preferences make it a candidate for future DSR-based information-systems studies.

Load-bearing premise

The load-bearing premise is that the authors' assignment of 45 studies, none of which explicitly uses design science research, to stages of a design science process is accurate enough to show the Q-method supports every stage.

Editorial extensions

If this is right

Design science researchers can treat the Q-method as a stage-neutral tool, using it to identify problems, define solutions, and evaluate artifacts rather than restricting it to evaluation alone.
In problem identification, the concourse itself becomes the problem-framing device: statements drawn from literature, users, experts, or digital sources can expose barriers and challenges.
In solution definition, Q-sorting lets stakeholders rank design options by preference, providing consensus on which artifact to build.
In evaluation, Q-method results supply both quantitative rankings and qualitative explanations of user viewpoints, which can be used to assess usability and acceptance.
Future DSR studies can draw the concourse from big data sources such as social media, extending the method beyond literature-based statements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: if the mapping holds, the Q-method offers a transferable subjectivity-measurement template not only for information-systems design but for any design discipline where user perception shapes the artifact.
Editorial inference: a testable next step would be to run a DSR project that uses the Q-method at all three stages and compare its problem statements, design choices, and evaluation findings against a parallel project using conventional surveys.
Editorial inference: because all 45 mapped studies lack explicit DSR framing, the paper's conclusion should be read as a supported programmatic proposal rather than a demonstration; the first explicit Q-method-in-DSR case study would be the decisive evidence.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a legitimate first systematic map of Q-method use in information systems, and the proposed DSR integration framework is a real contribution. But the paper's central claim — that the Q-method can support all main DSR stages — is only as strong as the authors' interpretive coding of 45 papers that, by the paper's own admission, never explicitly used DSR. The mapping is not independent of the coders' judgment.

Credit where it is due. The paper follows a recognized mapping protocol, builds on Peffers et al.'s DSR process and Gregor and Hevner's contribution framework, and compiles a genuinely useful table classifying each paper by role of the Q-method, concourse source, IS topic, DSR process, and knowledge contribution. Figure 4 gives DSR researchers a concrete starting point for where Q-sort could plug into problem identification, solution definition, and evaluation. As a scoping exercise, this is well worth having.

The soft spots are real but not fatal. The search reporting is sloppy: the text says 47 publications from Basket-of-Eight journals and 211 from the ACM Digital Library, though the context clearly means the AIS Digital Library, and the path from 258 to 70 to 45 is not fully accounted. More importantly, the stress-test note lands: because no selected study used DSR, the "Comparable DSR process" column in Table 1 is the authors' classification, and no coding protocol or inter-rater reliability is reported. The percentages in Figures 2 and 3 therefore inherit the authors' interpretive choices. The paper does show some awareness — in Section 4.2 it quietly narrows the problem-identification role to a supporting tool, which is more defensible than the abstract's blanket claim — but the limitations section never addresses the reproducibility of the DSR-stage coding.

The audience is IS researchers interested in mixed-methods DSR and methodologists looking for a corpus of Q-method applications. This deserves a serious referee, not a desk reject. I would send it to review with a request to fix the search reporting and either report coding reliability or reframe the conclusion to match the evidence. The framework is worth publishing; the strongest version of the claim needs more support.

Referee Report

4 major / 4 minor

Summary. This paper reports a systematic mapping study of Q-method use in Information Systems (IS) research. The authors search Basket-of-Eight journals and the AIS digital library, identify 45 publications that use the Q-method, and classify each paper by IS topic, role of the Q-method, concourse source type, knowledge contribution, and a "comparable DSR process" (problem identification, solution definition, evaluation). Based on this classification, the paper proposes an integration framework for the Q-method into Design Science Research (DSR) and claims that the Q-method can support every main stage of the DSR process. The central conclusion is stated in the abstract and in Section 4.2: the Q-methodology "can be used to support each main research stage of DSR processes" and can serve as a useful tool for evaluating systems in the IS topic of system analysis and design.

Significance. If the mapping were reliable, the paper would be a useful starting point for DSR researchers, consolidating scattered Q-method applications and linking them to recognized DSR stages. The study has clear strengths: it follows a systematic mapping structure, formulates explicit guideline questions, presents a detailed mapping table, and builds on established frameworks (Peffers et al., 2007; Gregor & Hevner, 2013). The proposed integration framework is a plausible conceptual contribution. However, the central empirical claim is currently not supported by the corpus: the authors concede that no included publication explicitly used DSR, so the conclusion that the Q-method supports all DSR stages rests entirely on the authors' own assignment of each paper to a "comparable DSR process." That assignment is presented without a coding protocol, reliability check, or external validation, making the main result non-reproducible and indistinguishable from coder judgment. The paper is therefore best treated as a promising proposal whose evidence base needs substantial methodological strengthening.

major comments (4)

[Section 2.2] The screening pipeline is not fully reported: the search yields 47 publications from Basket-of-Eight journals and 211 from the ACM Digital Library (a likely typo for AIS Digital Library), then the text says "by applying to 70 publications the exclusion criteria" before reaching the final 45. The reduction from 258 to 70 is unaccounted for, and the final count is not reconciled with the inclusion criteria. This makes the systematic-mapping evidence base non-reproducible.
[Section 3 and Section 4.2] The paper concedes in Section 3 that "no publications explicitly pointed to the use of DSR in their studies," yet the conclusion in Section 4.2 that "the Q-method can be used to support all the main DSR processes" is drawn from the authors' assignment of each paper to a "Comparable DSR process" in Table 2. No coding protocol, inter-rater reliability check, or external validation is provided for this assignment, so the central claim is currently indistinguishable from the coders' judgment.
[Section 2.3 and Table 2] The classification variables are described, but the table contains entries such as Campbell (2015) classified as "Major support" while its concourse source is marked "not mentioned." The treatment of missing or inconsistent data is not explained. Because Figures 2 and 3 aggregate percentages over the same rows, missing or inconsistently coded values propagate directly into the quantitative results that ground the integration framework.
[Section 4.2 and Figure 2] The statement that the Q-method "with 66.6% dominates the IS-topic system analysis and design" is not clearly tied to a specific figure or calculation. Section 3 reports 37.8% (Q-method as minor support in that topic) and 40% (evaluation-driven papers in that topic), and these numbers are not reconciled with the 66.6% claim. The reader cannot verify the quantitative basis of the conclusion.

minor comments (4)

[Section 2.2] The search is described as targeting the Association for Information Systems digital library, but the screening text refers to the "ACM Digital Library" when reporting 211 publications; this should be corrected.
[Table 2 and Appendix 2] The text refers to "Table 1" for the mapping overview, while the table caption reads "Table 2"; additionally, Appendix 2 is listed but appears empty in the manuscript, so the year distribution cannot be verified.
[Section 3] The sentence "approximately 35 years elapse from the first introduction of the Q-method in 1953 to first use of the method for IS study in 1988" is arithmetically correct but awkwardly worded; consider revising for clarity.
[Section 4.3] The phrase "this study pioneer s the initial step" contains a spacing typo and should read "pioneers the initial step."

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the DSR-stage categories come from external literature and the central claim is an empirical generalization from the mapping, not a restatement of the study's own inputs.

full rationale

The paper's central claim—that the Q-method can support each main DSR process—is an inductive generalization from a systematic mapping of 45 IS publications. The DSR process categories are taken from external sources (Peffers et al., 2007; Hevner & Chatterjee, 2010), and the Q-method corpus is independently defined by inclusion/exclusion criteria. The authors explicitly acknowledge that 'no publications explicitly pointed to the use of DSR in their studies,' and therefore assign each paper to a 'Comparable DSR process' in Table 1. This assignment is interpretive and would benefit from inter-rater reliability or external validation, but that is a correctness and reproducibility concern, not circularity: the classification could have produced different distributions, and the conclusion is not equivalent by definition to the coding scheme. The one self-citation (Pawlowski et al., 2015) is used to motivate interest in well-being and positive computing, but it is not load-bearing for the derivation of the Q-method's applicability to DSR stages. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The paper's limitations section even identifies the interpretive nature of its topic classification, which further shows that the authors did not conceal the dependence on judgment. Accordingly, the central derivation is self-contained as a mapping study, and the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. It relies on domain assumptions about Q-method and DSR, and on an ad hoc interpretive mapping from non-DSR papers to DSR stages. These axioms are load-bearing because the conclusion depends on them.

assumptions (3)

domain assumption Q-methodology is a valid instrument for eliciting and quantifying subjective viewpoints.
The paper treats Q-method validity as established by prior literature (Stephenson 1953; Watts & Stenner 2012); it is not tested here.
domain assumption Design Science Research can be decomposed into problem identification, solution definition, and evaluation for the classification.
The three-process decomposition is taken from Peffers et al. (2007) and Hevner & Chatterjee (2010) and used to categorize the 45 papers.
ad hoc to paper The authors' assignment of each publication to a 'comparable DSR process' is a valid representation of how Q-method would function in DSR.
This is the key interpretive step: none of the 45 papers explicitly used DSR, so the mapping relies on the authors' judgment, not on the papers' own claims.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Why Should the Q-method be Integrated Into the Design Science Research? A Systematic Mapping Study." pith.science (2026). https://pith.science/paper/WO6NDKOX

@misc{pith2026190805203,
  author       = {Pith},
  title        = {Pith review of: Why Should the Q-method be Integrated Into the Design Science Research? A Systematic Mapping Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WO6NDKOX}},
  note         = {Machine review of arXiv:1908.05203}
}

read the original abstract

The Q-method has been utilized over time in various areas, including information systems. In this study, we used a systematic mapping to illustrate how the Q-method was applied within Information Systems (IS) community and proposing towards the integration of Q-method into the Design Sciences Research (DSR) process as a tool for future research DSR-based IS studies. In this mapping study, we collected peer-reviewed journals from Basket-of-Eight journals and the digital library of the Association for Information Systems (AIS). Then we grouped the publications according to the process of DSR, and different variables for preparing Q-method from IS publications. We found that the potential of the Q-methodology can be used to support each main research stage of DSR processes and can serve as the useful tool to evaluate a system in the IS topic of system analysis and design

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 14 canonical work pages

[1]

Abbasi, A., Sarker, S., & Chiang, R. H. L. (2016). Big data research in information systems: Toward an inclusive research agenda. Journal of the Association for Information Systems, 17(2). Barbosa, O., Santos, R. P., Alves, C., Werner, C., & Jansen, S. (2013). Software ecosystems – analyzing and managing business networks in the software industry. Jansen ...

work page 2016
[2]

Gregor, S., & Hevn er, A. R. (2013). Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2). Gregory, R. W. (2011). Design science research and the grounded theory method: Characteristics, dif- ferences, and complementary uses. In Theory-guided modeling and empiricism in information sys- tems research (pp. 111–127). Sp...

work page 2013
[9]

S., Wright, R

Topi, H., Valacich, J. S., Wright, R. T., Kaiser, K., Nunamaker Jr, J. F., Sipior, J. C., & Vreede, G. -J. de. (2010). IS 2010: Curriculum guidelines for undergraduate degree programs in information sys- tems. Communications of the Association for Information Systems, 26(1),

work page 2010
[18]

/Towards the Integration of the Q method into a DSR Study Tenth Scandinavian Conference on Information Systems (SCIS2019), Nokia, Finland

Nurhas et al. /Towards the Integration of the Q method into a DSR Study Tenth Scandinavian Conference on Information Systems (SCIS2019), Nokia, Finland. 18 Tractinsky, N., & Jarvenpaa, S. L. (1995). Information systems design decisions in a global versus domestic context. MIS Quarterly, 507–534. Venkatesh, V., & Davis, F. D. (2000). A theoretical extensio...

work page 1995
[25]

A., Henningsson, S., Hrastinski, S., & Keller, C

Carlsson, S. A., Henningsson, S., Hrastinski, S., & Keller, C. (2011). Socio -technical IS design sci- ence research: Developing design theory for IS integration mana gement. Information Systems and E-Business Management, 9(1), 109–131. Chang, C.-C. (2012). Exploring IT entrepreneurs' dynamic capabilities using Q -technique. Industrial Management & Data S...

work page 2011
[39]

Eden, S., Donaldson, A., & Walker, G. (2005). Structuring subjectivities? Using Q methodology in human geography. Area, 37(4), 413–422. Engström, E., & Runeson, P. (2011). Softwar e product line testing –a systematic mapping study. In- formation and Software Technology, 53(1), 2–13. Nurhas et al. /Towards the Integration of the Q method into a DSR Study T...

work page 2005
[580]

R., Kettinger, W

Grover, V., Jeong, S. R., Kettinger, W. J., & Teng, J. T. C. (1995). The implementation of business process reengineering. Journal of Management Information Systems, 12(1), 109–144. Hevner, A., & Chatterjee, S. (2010). De sign science research in information systems. In Design re- search in information systems (pp. 9–22). Springer. Hrastinski, S., Carlsso...

work page 1995
[593]

Doherty, M. J. (2012). Examining Project Manager Insights of Agile and Traditional Success Factors for Information Technology Projects: A Q -Methodology Study. PhD, Marian University, Doctoral Dissertation. Doody, D. G., Kearney, P., Barry, J., Moles, R., & O’Regan, B. (2009). Evaluation of the Q-method as a method of public participation in the selection...

work page 2012

Show all 14 references

[659]

Söllner, M., Bitzer, P., Janson, A., & Leimeister, J. M. (2018). Process is king: Evaluating the perfor- mance of technology -mediated learning in vocational software training. Journal of Information Technology, 33(3), 233–253. Stephenson, W. (1953). The study of behavior; Q-t...

2018
[2008]

Iivari, J

On How to Develop Design Theories for IS Use and Management. Iivari, J. (2015). Distinguishing and contrasting two strategies for design science research. European Journal of Information Systems, 24(1), 107–115. Jahng, J., Jain, H., & Ramamurthy, K. (2007). Effects of interact...

2015
[2009]

Benlian, A., Hilkert, D., & Hess, T

Soft design science methodology: ACM. Benlian, A., Hilkert, D., & Hess, T. (2015). How open is this platform? The meaning and measure- ment of platform openness from the complementors’ perspective. Journal of Information Technol- ogy, 30(3), 209–228. Bhattacherjee, A. (2002). ...

2015
[2012]

Nadkarni, S., & Gupta, R

Combining case study, design science and action research methods for ef- fective collaboration engineering research efforts: IEEE. Nadkarni, S., & Gupta, R. (2007). A task -based model of perceived website complexity. MIS Quarter- ly, 501–524. Newman, I., & Ramlo, S. (2010). U...

2007
[2014]

Saeed, K

Artifact Evaluation in Information Systems De- sign-Science Research-a Holistic View. Saeed, K. A., Grover, V., & Hwang, Y. (2005). The relationship of e -commerce competence to cus- tomer value and firm performance: An empirical investigation. Journal of Management Infor- mat...

2005
[2015]

Messerschmidt, C

A Method for Measur- ing User Preferences in Information Systems Design Choices. Messerschmidt, C. M., & Hinz, O. (2013). Explaining the adoption of grid computing: An integrated institutional theory and organizational capability approach. The Journal of Strate gic Information...

2013

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Abbasi, A., Sarker, S., & Chiang, R. H. L. (2016). Big data research in information systems: Toward an inclusive research agenda. Journal of the Association for Information Systems, 17(2). Barbosa, O., Santos, R. P., Alves, C., Werner, C., & Jansen, S. (2013). Software ecosystems – analyzing and managing business networks in the software industry. Jansen ...

work page 2016

[2] [2]

Gregor, S., & Hevn er, A. R. (2013). Positioning and presenting design science research for maximum impact. MIS Quarterly, 37(2). Gregory, R. W. (2011). Design science research and the grounded theory method: Characteristics, dif- ferences, and complementary uses. In Theory-guided modeling and empiricism in information sys- tems research (pp. 111–127). Sp...

work page 2013

[3] [9]

S., Wright, R

Topi, H., Valacich, J. S., Wright, R. T., Kaiser, K., Nunamaker Jr, J. F., Sipior, J. C., & Vreede, G. -J. de. (2010). IS 2010: Curriculum guidelines for undergraduate degree programs in information sys- tems. Communications of the Association for Information Systems, 26(1),

work page 2010

[4] [18]

/Towards the Integration of the Q method into a DSR Study Tenth Scandinavian Conference on Information Systems (SCIS2019), Nokia, Finland

Nurhas et al. /Towards the Integration of the Q method into a DSR Study Tenth Scandinavian Conference on Information Systems (SCIS2019), Nokia, Finland. 18 Tractinsky, N., & Jarvenpaa, S. L. (1995). Information systems design decisions in a global versus domestic context. MIS Quarterly, 507–534. Venkatesh, V., & Davis, F. D. (2000). A theoretical extensio...

work page 1995

[5] [25]

A., Henningsson, S., Hrastinski, S., & Keller, C

Carlsson, S. A., Henningsson, S., Hrastinski, S., & Keller, C. (2011). Socio -technical IS design sci- ence research: Developing design theory for IS integration mana gement. Information Systems and E-Business Management, 9(1), 109–131. Chang, C.-C. (2012). Exploring IT entrepreneurs' dynamic capabilities using Q -technique. Industrial Management & Data S...

work page 2011

[6] [39]

Eden, S., Donaldson, A., & Walker, G. (2005). Structuring subjectivities? Using Q methodology in human geography. Area, 37(4), 413–422. Engström, E., & Runeson, P. (2011). Softwar e product line testing –a systematic mapping study. In- formation and Software Technology, 53(1), 2–13. Nurhas et al. /Towards the Integration of the Q method into a DSR Study T...

work page 2005

[7] [580]

R., Kettinger, W

Grover, V., Jeong, S. R., Kettinger, W. J., & Teng, J. T. C. (1995). The implementation of business process reengineering. Journal of Management Information Systems, 12(1), 109–144. Hevner, A., & Chatterjee, S. (2010). De sign science research in information systems. In Design re- search in information systems (pp. 9–22). Springer. Hrastinski, S., Carlsso...

work page 1995

[8] [593]

Doherty, M. J. (2012). Examining Project Manager Insights of Agile and Traditional Success Factors for Information Technology Projects: A Q -Methodology Study. PhD, Marian University, Doctoral Dissertation. Doody, D. G., Kearney, P., Barry, J., Moles, R., & O’Regan, B. (2009). Evaluation of the Q-method as a method of public participation in the selection...

work page 2012