Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Exploring the relations between net benefits of IT projects and CIOs' perception of quality of software development disciplines

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that software development disciplines fall into distinct Kano quality categories, with testing most strongly tied to IT project net benefits.

desk verdict Plausible empirical study; the Kano classification is read off plots by eye and needs a formal rule to be credible. read the letter →

arxiv 1908.04070 v1 pith:R2TDWB5Z submitted 2019-08-12 cs.SE cs.CY

Damjan Vavpotič , Marko Robnik-Šikonja , Tomaž Hovelja This is my paper · ORCID
classification cs.SEcs.CY
keywords KanomodelsoftwaredevelopmentdisciplinesITprojectnetbenefitsOrdEvalReliefFCIOsurveyprocessquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that how thoroughly a software development method is applied in individual disciplines—requirements acquisition, design, coding and integration, testing, deployment, and project management—is measurably tied to the net benefits of IT projects. Using survey responses from 113 CIOs of large Slovenian enterprises, it ranks disciplines by strength of association and then classifies each discipline by Kano's quality categories. The central result is that testing shows the strongest association with net benefits and is a must-be quality, deployment is also must-be, coding and integration is one-dimensional, and requirements acquisition is attractive. If true, managers can treat changes to testing and deployment routines as risky disruptions, while experimenting with requirements acquisition is comparatively safe.

What carries the argument

The central machinery is the pair of attribute-evaluation algorithms ReliefF and OrdEval. ReliefF ranks disciplines by the strength of their association with net benefits while taking attribute interdependencies into account. OrdEval, an ordinal attribute evaluator built on ReliefF, computes upward and downward reinforcement factors for each value of a discipline's satisfaction score; the pattern of statistically significant bars in its visual output indicates whether a discipline behaves as must-be, one-dimensional, attractive, indifferent, or reverse Kano quality. The Kano model supplies the interpretive categories, and the paper's key analytic step is reading OrdEval bar plots to assign each discipline to a Kano category.

What would settle it

Re-survey the same type of respondents with Kano's original paired functional and dysfunctional questions for each discipline, or apply an explicit scoring rule to the OrdEval reinforcement bars, and check whether testing and deployment still classify as must-be, coding as one-dimensional, and requirements acquisition as attractive.

Watch

Extended reading notes

Core claim

The paper's central claim is that software development disciplines differ in Kano quality type according to CIOs' perception of how well the discipline's methodology is applied. Based on OrdEval reinforcement patterns, testing and deployment are must-be qualities, coding and integration is one-dimensional, requirements acquisition is attractive, and system design and architecture as well as project management are inconclusive on average. The paper also claims that ReliefF ranking shows testing has the strongest positive association with project net benefits, coding and integration ranks second, and project management shows no positive association on average.

Load-bearing premise

The load-bearing assumption is that the visual pattern of statistically significant bars in the OrdEval plots maps unambiguously onto Kano's theoretical categories; the paper provides no formal rule or independent validation for that mapping.

Editorial extensions

If this is right

  • Testing is the discipline whose application quality is most strongly tied to IT project net benefits, so improvement efforts aimed at testing are the likeliest to pay off.
  • Testing and deployment are must-be quality: cutting back on their established methodology risks significant dissatisfaction and lost benefits, while pushing them beyond basic levels yields little extra satisfaction.
  • Coding and integration is one-dimensional: raising its methodology quality should produce a steady, proportional increase in benefits.
  • Requirements acquisition is attractive: adopting improved techniques can raise satisfaction and benefits substantially, with less risk of disrupting routines.
  • Project management shows no average positive association with net benefits, although a subgroup of CIOs perceives it as attractive.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Going beyond the paper, the same OrdEval-Kano mapping could be validated by administering the original Kano questionnaire alongside the short survey; if the two classifications agree, the short instrument could replace the longer one in process-quality audits.
  • Going beyond the paper, the must-be versus attractive split suggests a resource-allocation rule: protect the must-be disciplines' baseline routines, then invest surplus improvement effort in attractive or one-dimensional disciplines.
  • Going beyond the paper, the ranking is based on CIO perception in large Slovenian enterprises; a direct test would repeat the survey in mid-size firms or other countries to see whether the Kano types and the testing-first ranking persist.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper analyzes survey data from 113 CIOs of the largest Slovenian enterprises to relate the perceived quality of software development methodology (SDM) application in individual development disciplines to the net benefits of IT projects. Using the ReliefF algorithm for attribute ranking and the OrdEval algorithm to characterize nonlinear, asymmetric effects of individual attribute values on the outcome, the authors report that Testing has the strongest positive association with net benefits, followed by Coding and integration, while Project management shows no positive association. Based on visual interpretation of OrdEval reinforcement plots, they assign disciplines to Kano quality categories: Testing and Deployment as must-be, Coding and integration as one-dimensional, Requirements acquisition as attractive, with System design and architecture inconclusive and Project management inconclusive on average but attractive for a subgroup. The paper draws prescriptive conclusions, e.g., that enterprises should be cautious when altering testing and deployment but may experiment with requirements acquisition.

Significance. If the findings hold, the paper offers a practical approach to evaluating software development disciplines through a nonlinear quality lens, which is a genuine contribution to IT project management and Kano-model research. The study uses well-established algorithms (ReliefF, OrdEval) implemented in a public R package, and the authors explicitly acknowledge the non-random sample and the retrospective survey design as limitations. The core correlational claim—that SDM application in some disciplines is associated with project net benefits—is plausible and consistent with prior work. However, the more distinctive contribution, the Kano classification in Table 1, currently rests on an unformalized visual reading of OrdEval plots, with no decision rule, no inter-rater reliability check, and no validation against an independent Kano questionnaire. Given that the prescriptive advice depends directly on this classification, the significance is conditional on addressing that methodological gap.

major comments (3)
  1. [Section 4, Table 1, Figs. 4-9] The assignment of disciplines to Kano quality categories is based on a visual interpretation of statistically significant reinforcement bars, but no formal decision rule is stated that maps a pattern of significant upward/downward reinforcements to a Kano category (e.g., must-be vs. attractive). The text in Section 6 concedes that subjective classification is present in the Kano model and suggests future integration with quantitative Kano models. Because the prescriptive conclusions about caution in altering testing/deployment versus experimenting with requirements acquisition depend directly on this classification, the manuscript needs a reproducible rule, an inter-rater reliability check, or a validation against the original Kano questionnaire on a subsample. Without this, the must-be/attractive distinction is not supported beyond the authors' visual impression.
  2. [Section 4, Fig. 3] The claim that 'Testing discipline has the strongest association' with net benefits, with Coding and integration second, is reported without any uncertainty intervals or significance tests on the ReliefF scores. With n = 113 and six disciplines, the differences between adjacent ranks may be within sampling noise. Please provide bootstrap confidence intervals or a permutation test for the ReliefF values, and address the multiple-comparisons issue when interpreting the ranking and the OrdEval significance bars.
  3. [Section 3] The OrdEval hyperparameters (e.g., number of nearest neighbors, number of iterations, missing-value handling parameters) are not reported. Since the OrdEval visualization and the resulting Kano classification depend on these settings, the reproducibility of the central classification requires reporting the exact parameter values and, ideally, a sensitivity analysis showing that the classification is stable across reasonable hyperparameter choices.
minor comments (4)
  1. [Section 6, Conclusion] The conclusion uses causal language such as 'Testing providing the highest net benefits' and 'increasing its quality will result in a stable growth of benefits,' but the study is cross-sectional and observational. Please temper these statements to reflect association rather than causation.
  2. [Introduction, Fig. 1 caption] The caption states an assumption that the scales are 7-point Likert, which is a modeling assumption for the idealized curves; clarify that this is an illustrative assumption and not a property of the collected data.
  3. [Section 4, Figs. 4-9] The text describes reinforcement bars as 'statistically significant' without stating that significance is determined by bars extending beyond the box-and-whiskers whiskers at the 95% level. Please state this criterion explicitly in the results section or in the figure captions.
  4. [Section 2, OrdEval description] The paper alternates between 'CIOs' satisfaction with the application of SDM in a discipline' and 'net benefits of IT projects' as the response variable. Clarify which variable is used as the OrdEval/ReliefF response in the analysis, as this is essential for interpreting the results.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the analysis applies published, code-reproduced ReliefF/OrdEval methods to new survey data, and the Kano categories are inferred from data patterns rather than assumed.

full rationale

The paper's derivation chain is self-contained with respect to its empirical claims. The ReliefF ranking and OrdEval-based Kano classification are computed from the survey data via published algorithms (ReliefF, OrdEval) implemented in the CORElearn package. The Kano categories in Table 1 are inferred from the pattern of statistically significant reinforcement bars in Figs. 4-9, not assumed in advance; the mapping from bar patterns to Kano categories is stated in the paper (Fig. 2 and accompanying text) and is an interpretive rule, not a definitional identity. The self-citations to OrdEval and ReliefF are to peer-reviewed, code-reproduced methods and therefore constitute independent support rather than circular reasoning. The paper's own acknowledgement of 'subjective classification present in the Kano model' is a limitation on reliability, not evidence that the result is equivalent to its input. No fitted parameter is renamed as a prediction, and no uniqueness theorem is invoked to force the choice. Hence no circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new entities. Its main extra assumptions are the validity of self-reported CIO perceptions as measures of process quality, the ad hoc visual mapping from OrdEval output to Kano categories, and the representativeness of the small non-random sample. These assumptions are load-bearing for the classification results.

free parameters (1)
  • OrdEval hyperparameters (e.g., number of nearest neighbors, number of iterations)
    The paper does not report the settings used for the OrdEval algorithm, which can affect the reinforcement factors and confidence intervals.
assumptions (3)
  • domain assumption CIO satisfaction scores on a 7-point Likert scale measure the quality of SDM application in each discipline.
    The entire analysis uses self-reported perceptions as proxies for actual process quality without independent validation.
  • ad hoc to paper The shape of statistically significant OrdEval reinforcement bars maps to Kano quality categories.
    The mapping from Fig. 4-9 to Table 1 is done by visual inspection with no explicit rule or validation.
  • domain assumption The sample of 113 CIOs from the top 1000 Slovenian enterprises supports generalization to the broader population.
    The authors acknowledge the sample is non-random and the response rate is 11.3%, so the representativeness is limited.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the relations between net benefits of IT projects and CIOs' perception of quality of software development disciplines." pith.science (2026). https://pith.science/paper/R2TDWB5Z

@misc{pith2026190804070,
  author       = {Pith},
  title        = {Pith review of: Exploring the relations between net benefits of IT projects and CIOs' perception of quality of software development disciplines},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R2TDWB5Z}},
  note         = {Machine review of arXiv:1908.04070}
}
read the original abstract

Software development enterprises are under consistent pressure to improve their management techniques and development processes. These are comprised of several disciplines like requirements acquisition, design, coding, testing, etc. that must be continuously improved and individually tailored to suit specific software development project. This paper presents an evaluation approach that enables the enterprises to increase development process net benefits by improving disciplines' quality and increasing developers' satisfaction. Our approach builds on Kano's model of quality. Based on an empirical study of top 1000 enterprises from Slovenia we find that application of software development methodologies in individual development disciplines significantly relates to net benefits of IT projects. The results show that different types of Kano quality are present in individual disciplines. Enterprises should be cautious when altering must-be quality disciplines like testing or deployment as they can significantly disrupt the established routines, cause great dissatisfaction between developers and significantly reduce benefits. On the other hand, changing the attractive quality disciplines like requirements acquisition can notably increase developers' satisfaction and benefits but is less likely to disrupt the established routines.

Figures

Figures reproduced from arXiv: 1908.04070 by the authors.

Figure 1
Figure 1. Graphs of idealized attribute types based on Kano qua [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Visualization provided by the OrdEval algorithm for t [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Association between CIOs’ satisfaction with SDM appli [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: shows the statistically significant positive influence of SDM application in Requirements acquisition discipline on CIO satisfaction with the discipline when attribute value increases from 3 to 4, 4 to 5 and from 6 to 7 and statistically significant negative influence …
Figure 5
Figure 5. Figure 5: OrdEval results for System design and architecture [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: shows the statistically significant positive influence of SDM application in Coding and integration discipline on CIO satisfaction with the discipline when attribute value increases from 2 to 3 and from 4 to 5 and statistically significant negative influence when the a…
Figure 7
Figure 7. Figure 7: OrdEval results for Testing SDM application in Deployment discipline is generally perceived as must-be quality as indicated by the bottom right bar showing statistically significant and strong effect on CIO satisfaction with the discipline in [PITH_FULL_IMAGE:figures/…
Figure 8
Figure 8. Figure 8: OrdEval results for Deployment [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: OrdEval results for Project management 5 Discussion The attribute evaluation with ReliefF ( [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 canonical work pages

  1. [1]

    Exploring the relations between net benefits of IT projects and CIOs’ perception of quality of software development disciplines assoc. prof. dr. Damjan Vavpotiča,*, prof. dr. Marko Robnik-Šikonjab, assoc. prof. dr. Tomaž Hoveljac University of Ljubljana, Faculty of Computer and Information Sc ience, Večna pot 113, SI-1000 Ljubljana, Slovenia * correspondi...

  2. [2]

    The information the OrdEval algorithm provides has no parallel in standard approaches like multiple regression analysis

    Figure 2: Visualization provided by the OrdEval algorithm for three idealized attributes: one-dimensional (left), must be (middle), and attractive (right). The information the OrdEval algorithm provides has no parallel in standard approaches like multiple regression analysis. Firstly, there is substantial context sensitivity. Typically, the attributes are...

    work page 2002

  3. [3]

    shows that appli cation of SDM in Requirements acquisition discipline importantly affects the net benefits of IT projects. OrdEval further identifies the attractive Kano quality of SDM in Requirements acquisition discipline as having the highest statistically significant inf luence on CIO satisfaction (Fig. 4). These OrdEval results might be surprising si...

    work page 2009

  4. [6]

    2 Background and related work According to the CHAOS report (Standish Group 2015), only 29% of software development projects are completed on-time and on-budget, with all functions as initially specifie d. Another 52% of projects are completed and operational but over-budget, over the time estimate, and offer fewer functions than originally specified, whi...

    work page 2015

  5. [7]

    take only dependence between one attribute and satisfaction into account. More advanced measures take into account conditional dependencies between a response variable and several attributes, the best kn own such measure being ReliefF (Robnik-Šikonja & Kononenko 2003). These attribute quality evaluation measures are concerned with the predictive power of ...

    work page 2003

  6. [8]

    It was initially developed for customer satisfaction research in marketing

    is an attribute evaluat ion measure building upon ReliefF. It was initially developed for customer satisfaction research in marketing. The approach evaluates ordinal attributes, i.e. survey questions, based on their relation to the expected outco me. Different to ReliefF and other attribute evaluation measures it analyses each attribute’s value separate l...

    work page 2015

  7. [12]

    The agreement statements about the net benefits of SDM disciplines for the deployed solution were analyzed and assigned to Kano quality categories

    and i ncluded requirements acquisition, system design and architecture, coding and integration, testing, and deployment. The agreement statements about the net benefits of SDM disciplines for the deployed solution were analyzed and assigned to Kano quality categories. We use two attribute evaluation measures: ReliefF and OrdEval, both implemented in R pac...

    work page 2016

  8. [16]

    Such approaches lead to higher level of efficiency in coding and integration discipline (Karvonen et al

    can be considerably reduced by introducing the practice of continuous integration. Such approaches lead to higher level of efficiency in coding and integration discipline (Karvonen et al. 2017). This matches our findings regarding the positive contribution of SDM in Coding and integration. The overall importance of Testing discipline is confirmed by Relie...

    work page 2017

Show all 21 references
  1. [17]

    As shown by ReliefF, the satisfaction of CIOs with the application of SDM in the Deployment discipline positively associates with net benefits of IT projects

    show ing that additional benefits are related to SDM defining higher levels of testing automation. As shown by ReliefF, the satisfaction of CIOs with the application of SDM in the Deployment discipline positively associates with net benefits of IT projects . These benefits pro...

    2017

  2. [19]

    Journal of systems and software, 86(8), pp.1978–2001

    An orchestrated survey of methodologies for automated software test case generation. Journal of systems and software, 86(8), pp.1978–2001. Barr, E.T. et al.,

    1978

  3. [20]

    Journal of the Japanese Society for Quality Control, 14(2), pp.147–156

    Attractive Quality and Must-Be Quality. Journal of the Japanese Society for Quality Control, 14(2), pp.147–156. Available at: http://ci.nii.ac.jp/naid/110003158895/en/. Karlsson, F. & Agerfalk, P.,

  4. [21]

    Empirical software engineering, 21(5), pp.1990–2032

    Change-based test selection: an empirical evaluation. Empirical software engineering, 21(5), pp.1990–2032. Standish Group,

    1990

  5. [1984]

    The Kano model was developed to categorize the qual ity attributes based on their ability to satisfy customers’ needs

    has become one of the most popular two-dimensional, nonlinear, asymmetrical models for eva luation of quality, and is adopted in a range of industries. The Kano model was developed to categorize the qual ity attributes based on their ability to satisfy customers’ needs. The fi...

    2008

  6. [2003]

    due to the rich variation in situational contexts (Clarke & O’Connor 2 012). Consequentially, it is important that the quality assessment of the software development process is performed for the key parts such as process disciplines and not only for the process as a whole (Hov...

    2015

  7. [2007]

    one size fits all

    and the Capability Maturity Model Integration (CMMI) (Kneuper 2009). These frameworks recognize that the software development process can have various degrees of maturity, wh ere increased process maturity r esults in increased predictability in relation to quality levels (Cla...

    2009

  8. [2009]

    The starting population was the top 1000 enterprises in Slovenia based on the creation of added value in 2014 including IBM, Microsoft, Novartis, Goodyear, KPMG, etc

    this study has also focused on the largest non-financial enterprises in a country. The starting population was the top 1000 enterprises in Slovenia based on the creation of added value in 2014 including IBM, Microsoft, Novartis, Goodyear, KPMG, etc . We sent the surveys to CIO...

    2014

  9. [2012]

    However, according to (Wells 2012), the majority of project managers with medium experience do not perceive the project management methodology as helpful

    who suggests that project management methodologies help inexperienced managers, who use them as a guidance tool, while they also help very experienced manag ers, who understand the value of promotion of standardization and uniformity of processes and procedures in m ultiple pr...

    2012

  10. [2013]

    and are probably perceived as innovations by the CIOs of the studied enterprises. Moreover, i n an environment of ever changing customer requirements and technological changes there is a need for cont inuous reflection to decide on the best course of action (Kakar 2017). The n...

    2017

  11. [2014]

    or big data (Zhao et al. 2017). Simple attribute evaluation measures like Gini index (Breiman e t al

    2017

  12. [2016]

    Such response rate is typical for mail surveys conducted in enterpri ses in Slovenia (Hovelja 2008)

    Based on personal and phone communication with the CIOs i nvolved in the study we found that the relatively low 11.3% response rate was mainly caused by the lack of time needed to fill out the questionnaire. Such response rate is typical for mail surveys conducted in enterpri ...

    2008

  13. [2017]

    with high importance to software project success (Fernandez et al. 2017). The results can be explained by the fact that recently many new requirement acquisition techniques and approa ches have been developed (Ernst et al. 2014; Lauesen & Kuhail 2012; Lucassen et al. 2016; Ras...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.