{"id":"099190f8-e605-4bf5-84fd-f1d0643c23b7","arxiv_id":"1908.05203","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A systematic mapping study of 45 IS papers suggests the Q-method can support problem identification, solution definition, and evaluation in Design Science Research, and includes a proposed integration framework.","lead":"This paper reviews 45 information systems publications that use the Q-method and argues that this sorting technique can support every major stage of Design Science Research. It then proposes a framework for inserting Q-sorting into DSR studies to capture user subjectivity during design and evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that the Q-method supports every DSR stage rests on an untested interpretive mapping of non-DSR papers; without reliability data, the central result is indistinguishable from the coders' judgment.","rationale":"The reader and I identify the same choke point. The paper is honest about the gap in Section 3, noting that no publication explicitly used DSR, and Section 4.2 itself uses the qualifier 'comparable.' However, the mapping is not mechanical: collapsing six Peffers et al. processes into three stages and then assigning each Q-method paper to a stage requires a judgment about the dominant purpose served by the Q-method in that study. No inter-rater reliability or external validation is reported, and the limitation section does not flag this. A re-coding experiment is a low-cost, decisive way to check whether the central conclusion survives, and it should be reported in any revision. I do not see a reason to move from CONDITIONAL; the concern is exactly what makes the current verdict conditional. A separate search-process inconsistency in Section 2.2 (AIS DL versus ACM DL) also needs correction, but it does not displace the mapping as the load-bearing issue.","tokens_in":17274,"tokens_out":3506,"duration_ms":36231,"concrete_test":"Ask two independent coders, blind to the paper's Table 1 assignments, to classify all 45 publications into the three DSR stages (problem identification, solution definition, evaluation) using only the Section 2.1.2 definitions and Peffers et al.'s six processes; report Cohen's kappa. Then recompute the Figure 2 and Figure 3 distributions using only the cases on which both coders agree, and compare these with the published percentages. If kappa is below 0.6, or the recomputed DSR-process distribution shifts by more than 10 percentage points in any stage, the claim that the mapping supports 'all main DSR processes' should be treated as unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing bridge is Table 1's 'Comparable DSR process' column. Section 3 concedes that 'no publications explicitly pointed to the use of DSR in their studies,' so the Section 4.2 conclusion that 'the Q-method can be used to support all the main DSR processes' cannot be read directly from the corpus. It is produced entirely by the authors' assignment of each non-DSR paper to one of the three DSR stages derived in Section 2.1.2. The paper provides no coding protocol for that assignment and no reliability check; the label 'comparable' in the table and text makes the interpretive nature explicit. Because the mapping percentages in Figures 2 and 3 feed directly into the proposed integration framework, a non-reproducible coding makes the central result non-reproducible. The concern is not that the mapping is definitely wrong, but that the central claim is currently indistinguishable from the coders' judgment.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a systematic mapping study of Q-method use in Information Systems (IS) research. The authors search Basket-of-Eight journals and the AIS digital library, identify 45 publications that use the Q-method, and classify each paper by IS topic, role of the Q-method, concourse source type, knowledge contribution, and a \"comparable DSR process\" (problem identification, solution definition, evaluation). Based on this classification, the paper proposes an integration framework for the Q-method into Design Science Research (DSR) and claims that the Q-method can support every main stage of the DSR process. The central conclusion is stated in the abstract and in Section 4.2: the Q-methodology \"can be used to support each main research stage of DSR processes\" and can serve as a useful tool for evaluating systems in the IS topic of system analysis and design.","tokens_in":17491,"tokens_out":3938,"duration_ms":37309,"significance":"If the mapping were reliable, the paper would be a useful starting point for DSR researchers, consolidating scattered Q-method applications and linking them to recognized DSR stages. The study has clear strengths: it follows a systematic mapping structure, formulates explicit guideline questions, presents a detailed mapping table, and builds on established frameworks (Peffers et al., 2007; Gregor & Hevner, 2013). The proposed integration framework is a plausible conceptual contribution. However, the central empirical claim is currently not supported by the corpus: the authors concede that no included publication explicitly used DSR, so the conclusion that the Q-method supports all DSR stages rests entirely on the authors' own assignment of each paper to a \"comparable DSR process.\" That assignment is presented without a coding protocol, reliability check, or external validation, making the main result non-reproducible and indistinguishable from coder judgment. The paper is therefore best treated as a promising proposal whose evidence base needs substantial methodological strengthening.","major_comments":[{"comment":"The screening pipeline is not fully reported: the search yields 47 publications from Basket-of-Eight journals and 211 from the ACM Digital Library (a likely typo for AIS Digital Library), then the text says \"by applying to 70 publications the exclusion criteria\" before reaching the final 45. The reduction from 258 to 70 is unaccounted for, and the final count is not reconciled with the inclusion criteria. This makes the systematic-mapping evidence base non-reproducible.","section":"Section 2.2"},{"comment":"The paper concedes in Section 3 that \"no publications explicitly pointed to the use of DSR in their studies,\" yet the conclusion in Section 4.2 that \"the Q-method can be used to support all the main DSR processes\" is drawn from the authors' assignment of each paper to a \"Comparable DSR process\" in Table 2. No coding protocol, inter-rater reliability check, or external validation is provided for this assignment, so the central claim is currently indistinguishable from the coders' judgment.","section":"Section 3 and Section 4.2"},{"comment":"The classification variables are described, but the table contains entries such as Campbell (2015) classified as \"Major support\" while its concourse source is marked \"not mentioned.\" The treatment of missing or inconsistent data is not explained. Because Figures 2 and 3 aggregate percentages over the same rows, missing or inconsistently coded values propagate directly into the quantitative results that ground the integration framework.","section":"Section 2.3 and Table 2"},{"comment":"The statement that the Q-method \"with 66.6% dominates the IS-topic system analysis and design\" is not clearly tied to a specific figure or calculation. Section 3 reports 37.8% (Q-method as minor support in that topic) and 40% (evaluation-driven papers in that topic), and these numbers are not reconciled with the 66.6% claim. The reader cannot verify the quantitative basis of the conclusion.","section":"Section 4.2 and Figure 2"}],"minor_comments":[{"comment":"The search is described as targeting the Association for Information Systems digital library, but the screening text refers to the \"ACM Digital Library\" when reporting 211 publications; this should be corrected.","section":"Section 2.2"},{"comment":"The text refers to \"Table 1\" for the mapping overview, while the table caption reads \"Table 2\"; additionally, Appendix 2 is listed but appears empty in the manuscript, so the year distribution cannot be verified.","section":"Table 2 and Appendix 2"},{"comment":"The sentence \"approximately 35 years elapse from the first introduction of the Q-method in 1953 to first use of the method for IS study in 1988\" is arithmetically correct but awkwardly worded; consider revising for clarity.","section":"Section 3"},{"comment":"The phrase \"this study pioneer s the initial step\" contains a spacing typo and should read \"pioneers the initial step.\"","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is worth publishing if the mapping is made reproducible and the conclusion is appropriately qualified. The most serious issue is that the \"comparable DSR process\" coding is untested and the screening numbers are inconsistent; these are fixable within a revision. I would not recommend rejection because the proposed integration framework can stand as a conceptual contribution even if the empirical support is weakened. The disparity between the stated sources (AIS Digital Library) and the reported \"ACM Digital Library\" count should be checked by the authors before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this is a legitimate first systematic map of Q-method use in information systems, and the proposed DSR integration framework is a real contribution. But the paper's central claim — that the Q-method can support all main DSR stages — is only as strong as the authors' interpretive coding of 45 papers that, by the paper's own admission, never explicitly used DSR. The mapping is not independent of the coders' judgment.\n\nCredit where it is due. The paper follows a recognized mapping protocol, builds on Peffers et al.'s DSR process and Gregor and Hevner's contribution framework, and compiles a genuinely useful table classifying each paper by role of the Q-method, concourse source, IS topic, DSR process, and knowledge contribution. Figure 4 gives DSR researchers a concrete starting point for where Q-sort could plug into problem identification, solution definition, and evaluation. As a scoping exercise, this is well worth having.\n\nThe soft spots are real but not fatal. The search reporting is sloppy: the text says 47 publications from Basket-of-Eight journals and 211 from the ACM Digital Library, though the context clearly means the AIS Digital Library, and the path from 258 to 70 to 45 is not fully accounted. More importantly, the stress-test note lands: because no selected study used DSR, the \"Comparable DSR process\" column in Table 1 is the authors' classification, and no coding protocol or inter-rater reliability is reported. The percentages in Figures 2 and 3 therefore inherit the authors' interpretive choices. The paper does show some awareness — in Section 4.2 it quietly narrows the problem-identification role to a supporting tool, which is more defensible than the abstract's blanket claim — but the limitations section never addresses the reproducibility of the DSR-stage coding.\n\nThe audience is IS researchers interested in mixed-methods DSR and methodologists looking for a corpus of Q-method applications. This deserves a serious referee, not a desk reject. I would send it to review with a request to fix the search reporting and either report coding reliability or reframe the conclusion to match the evidence. The framework is worth publishing; the strongest version of the claim needs more support.","headline":"A useful first systematic map of Q-method use in IS, but the headline claim that Q-method supports every DSR stage rests on the authors' own classification of papers that never used DSR.","tokens_in":17955,"tokens_out":1938,"would_cite":false,"duration_ms":20420,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Q-method can support every main stage of design science research in information systems, a systematic mapping of 45 studies concludes.","keywords":["Q-methodology","Q-sort","Design Science Research","Information Systems","systematic mapping study","subjectivity measurement","system evaluation","DSR process"],"falsifier":"An independent audit of Table 1 could settle the claim: if raters re-read the 45 papers and find that a large share contain no activity matching the design-science stage they were assigned to, then the mapping loses its empirical foundation. A second test would be to run a design science project that uses the Q-method at all three stages and check whether each stage actually yields usable stage-specific output.","tokens_in":17105,"feed_emoji":"🗺️","tokens_out":8383,"duration_ms":68906,"temperature":0.7,"pith_summary":"The paper tries to show that the Q-method, a sorting technique that measures people's subjective viewpoints, can be integrated into every main stage of design science research (DSR) in information systems. The authors systematically mapped 45 studies that used the Q-method in information-systems research and classified each one by the DSR stage it could most plausibly support: problem identification, solution definition, or evaluation. A sympathetic reader would take the central claim to be that no DSR stage is out of reach for the Q-method, and that the method is especially useful for evaluating systems in the system analysis and design topic. This matters because design science depends on subjective human perception, and the Q-method offers a structured, mixed qualitative-quantitative way to capture it.","feed_headline":"A 45-study map finds the Q-method fits every design-science stage","feed_subtitle":"Ranking user viewpoints by preference gives designers a subjectivity tool for every design phase.","key_machinery":"The central object is the Q-method (also called Q-sort or Q-methodology), a procedure for measuring human subjectivity in which participants rank a set of statements, the concourse, into a forced-choice distribution, and factor analysis of the sorted cards reveals shared viewpoints. The other load-bearing component is the DSR process model, simplified into three main stages: problem identification and motivation, solution definition and development, and evaluation. The mapping machinery works by taking each of the 45 publications and classifying it according to which comparable DSR process the Q-method use most plausibly supports, along with the role of the Q-method, the sources of its statements, the IS topic, and the knowledge contribution. This classification produces the bubble plots and tables from which the paper argues that the Q-method is flexible enough to carry every main DSR phase, especially evaluation.","core_discovery":"On the paper's own terms, the central discovery is that the Q-method can be used to support each main research stage of DSR processes and can serve as a useful tool to evaluate a system in the information-systems topic of system analysis and design. The mapping shows that the Q-method is used mostly as a supporting method rather than the main method, that it appears across all three main DSR stages when studies are reclassified by comparable process, and that evaluation-driven use dominates in system analysis and design. The authors also report that none of the 45 studies explicitly used DSR, which they read as a gap rather than a contradiction: the Q-method's flexibility in collecting statements and its ability to reach consensus on user preferences make it a candidate for future DSR-based information-systems studies.","pith_inferences":["Editorial inference: if the mapping holds, the Q-method offers a transferable subjectivity-measurement template not only for information-systems design but for any design discipline where user perception shapes the artifact.","Editorial inference: a testable next step would be to run a DSR project that uses the Q-method at all three stages and compare its problem statements, design choices, and evaluation findings against a parallel project using conventional surveys.","Editorial inference: because all 45 mapped studies lack explicit DSR framing, the paper's conclusion should be read as a supported programmatic proposal rather than a demonstration; the first explicit Q-method-in-DSR case study would be the decisive evidence."],"forward_implications":["Design science researchers can treat the Q-method as a stage-neutral tool, using it to identify problems, define solutions, and evaluate artifacts rather than restricting it to evaluation alone.","In problem identification, the concourse itself becomes the problem-framing device: statements drawn from literature, users, experts, or digital sources can expose barriers and challenges.","In solution definition, Q-sorting lets stakeholders rank design options by preference, providing consensus on which artifact to build.","In evaluation, Q-method results supply both quantitative rankings and qualitative explanations of user viewpoints, which can be used to assess usability and acceptance.","Future DSR studies can draw the concourse from big data sources such as social media, extending the method beyond literature-based statements."],"supporting_citations":[{"why":"Supplies the systematic-mapping methodology that structures the entire literature search and classification.","marker":"Petersen et al. (2008)"},{"why":"Defines the DSR process model from which the three main stages are drawn.","marker":"Peffers et al. (2007)"},{"why":"Simplifies DSR into problem-driven, solution-driven, and evaluation-driven processes used as the classification variables.","marker":"Hevner and Chatterjee (2010)"},{"why":"The Q-method primer for information-systems research that frames how the method should be implemented and how its role is classified.","marker":"Thomas and Watson (2002)"},{"why":"Original source of the Q-technique and the basis for describing the method as measuring subjectivity.","marker":"Stephenson (1953)"},{"why":"Provides the operational definitions of concourse, Q-sort, and analysis used to classify statement sources and integration activities.","marker":"Watts and Stenner (2012)"},{"why":"Supplies the knowledge-contribution framework used to classify each mapped publication's contribution type.","marker":"Gregor and Hevner (2013)"},{"why":"Empirical basis for the claim that the Q-method captures user preferences better than rating, ranking, or maximum-different scaling in design choices.","marker":"Matzner et al. (2015)"},{"why":"Provides the IS curriculum topic classification used for the thematic grouping of publications.","marker":"Topi et al. (2010)"},{"why":"Provides the episodic DSR activities underpinning the proposed integration framework.","marker":"Baskerville et al. (2009)"}],"fun_headline_variants":["Q-method fits all design-science stages, 45-study map shows","Why the Q-method belongs in design science research","Q-method: a tool for every design-science phase?","45 studies reveal Q-method's fit for DSR stages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the authors' assignment of 45 studies, none of which explicitly uses design science research, to stages of a design science process is accurate enough to show the Q-method supports every stage.","fun_headline_variants_meta":{"raw":{"variants":["Q-method fits all design-science stages, 45-study map shows","Why the Q-method belongs in design science research","Q-method: a tool for every design-science phase?","45 studies reveal Q-method's fit for DSR stages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1329,"prompt_tokens":848,"completion_tokens":481,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":464,"tokens_out":481,"duration_ms":4924,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:19:54.943317+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent audit of Table 1 could settle the claim: if raters re-read the 45 papers and find that a large share contain no activity matching the design-science stage they were assigned to, then the mapping loses its empirical foundation. A second test would be to run a design science project that uses the Q-method at all three stages and check whether each stage actually yields usable stage-specific output.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the knowledge-contribution framework used to classify each mapped publication's contribution type."},{"cited_title":"S., Wright, R","cited_arxiv_id":null,"evidence_quote":"Provides the IS curriculum topic classification used for the thematic grouping of publications."}],"review_version":1}