REVIEW 3 major objections 6 minor 1 cited by
Using Sustainability Impact Scores for Software Architecture Evaluation
T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Technical quality choices carry measurable sustainability consequences that a weighted impact score can expose.
desk verdict The proposed SIS extension is a real step forward for sustainability-aware architecture evaluation, but the paper's headline claim of cross-dimension comparability is undermined by a flawed normalization that anchors to observed minima, not an absolute lower bound. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the Sustainability Impact Score (SIS): a weighted sum over dependency matrices that pair quality attributes of two sustainability dimensions, with each cross-effect recorded as $+1$, $-1$, or $0$. Priorities in the sum are derived from a utility matrix that maps scenarios to quality attributes and scores each attribute by importance and risk, normalized to $[0.1,1]$. To make scores comparable across dimension pairs, the paper normalizes each score against a theoretical-optimal dependency matrix in which negative effects are minimized, converting the result to a percentage. This machinery turns subjective architecture reasoning—which quality attributes matter and how they interact—into a single numeric comparison per dimension pair.
What would settle it
Recompute the SIS for the single-model and multi-model approaches using effect magnitudes estimated from actual system measurements (for example, the monetary cost or energy use associated with each dependency) instead of uniform $\pm 1$; if any dimension-pair ranking between the two approaches reverses, the equal-magnitude assumption is not sufficient for the reported comparison.
Extended reading notes
Core claim
On its own terms, the paper argues that a risk- and importance-weighted Sustainability Impact Score can turn architecture trade-offs into comparable numbers. The score is computed for pairs of sustainability dimensions as $\mathrm{SIS}_{\mathrm{dim1},\mathrm{dim2}} = \sum_{i,j}(\mathrm{Priority}_{\mathrm{dim1},i} + \mathrm{Priority}_{\mathrm{dim2},j}) \times \mathrm{Impact}_{ij}$, with effects taken as $+1$, $-1$, or $0$ from dependency matrices; priorities come from a weighted combination of each quality attribute's importance and risk. Compared with a theoretical-optimal dependency matrix, normalized scores express how close an architecture comes to fully supporting a dimension. In the MMvIB case, the multi-model approach scores highest for social and economic support while both approaches underperform on the environmental dimension, and the analysis exposes chains such as traceability improving reproducibility, transparency, stakeholder stake, and ultimately monetary cost. The paper's central claim is that these cross-dimension effects are real, often unrecognized, and can be surfaced by the SIS procedure.
Load-bearing premise
The load-bearing simplification is that every quality attribute's effect on another has the same magnitude, recorded as $+1$, $-1$, or $0$; if real effects differ in strength, the scores and the ranking of architectures could change.
Editorial extensions
If this is right
- Architects can compare alternative designs by normalized SIS per sustainability dimension pair, not just within one pair.
- Enabling and systemic sustainability effects, such as reproducibility requirements increasing resource utilization, become visible at evaluation time instead of after deployment.
- High-priority technical attributes that harm another dimension can be explicitly weighed against the consequences of changing them, as in the reproducibility-resource utilization trade-off.
- Organizations gain a repeatable, documented path toward sustainability reporting obligations, including preparing for regulations like CSRD.
- The relative comparison to a theoretical optimal provides a benchmark for how much sustainability support an architecture is leaving on the table.
Reading between the lines
- Beyond the paper, the equal-magnitude effect assumption could be stress-tested by replacing uniform $\pm 1$ values with measured strengths and re-running the MMvIB comparison; the ranking would hold only if it survives that perturbation.
- Beyond the paper, the normalized SIS could be inverted into a design objective, letting architects search over workflow configurations for the highest sustainability support across all dimension pairs.
- Beyond the paper, if external benchmarks such as carbon budgets become available, the relative percentages could be anchored to absolute targets, turning the score from a comparison device into a compliance metric.
- Beyond the paper, multi-stakeholder projects would need an explicit consensus procedure for risk, importance, and effect values; the utility-matrix structure suggests a group-weighted aggregation as a natural next step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an improved Sustainability Impact Score (SIS) for software architecture evaluation. The method extends prior work by computing SIS for any pair of sustainability dimensions, deriving QA priorities from stakeholder-provided importance and risk levels via a weighted sum, and normalizing scores relative to a theoretical-optimal DMatrix. The approach is demonstrated on a single industrial case study (MMvIB) comparing a single-model and a multi-model architecture. The paper reports inter-QA trade-offs and normalized SIS percentages for the T-Ec, T-En, T-S, and S-Ec dimension pairs, concluding that the multi-model approach is better for the economic and social dimensions and that both approaches underperform on the environmental dimension.
Significance. If the method were sound, it would provide a lightweight, practitioner-oriented tool for early sustainability assessment in architecture evaluation, complementing heavier scenario-based methods such as ATAM. The risk- and importance-based prioritization is a reasonable adaptation of ATAM's utility tree concept, and the authors ship a replication package, which is a concrete strength. The central limitation claims of the prior SIS (Section 2.3) are addressed in principle. However, the significance is conditional on fixing a load-bearing normalization problem and on treating the single-participant case study as an illustration rather than as evidence of generalizable empirical findings. The paper is honest in acknowledging several threats to validity, but the main contribution as stated, cross-dimension comparability of SIS values, is not currently established.
major comments (3)
- [Section 5.3] The normalization in Eq. (4) uses the observed minimum SIS per dimension pair as the lower bound, not a common absolute lower bound. In Table 3, for T-Ec the minimum is -0.425 (Single Model), while for T-En the minimum is -1.05 (Multi Model). Consequently, a normalized value of 0% corresponds to different raw SIS values in different rows and columns, so the normalized percentages are not on a common scale. This directly contradicts the claim in Section 5.3 that 'the normalization allowed us to compare the SIS values across dimensions,' and it fails to overcome limitation (ii) of Section 2.3. The issue is independent of the equal-magnitude assumption: even with exact effect magnitudes, the percentages cannot be compared across pairs because the anchor point varies. Section 6 even states that 'we do not have a lower bound for a SIS value,' so the min(SIS) used in Eq. (4) is only an artifact of the alternatives considered. The authors should either define a theoretical lower bound (e.g., a worst-case DMatrix) that is common across all pairs, or explicitly restrict all comparative claims to within-pair comparisons.
- [Section 5.3] The assumption that all QA effects have the same magnitude (+1 or -1) is load-bearing for the reported scores. The authors acknowledge this explicitly: 'one QA may affect another QA more positively or negatively than another QA.' Because all effects are equal-magnitude, the SIS values reduce to weighted counts of positive and negative edges, and the differences between alternatives (e.g., 76.51% vs 28.41% for T-Ec) depend entirely on which edges are present. No sensitivity analysis is provided to show how the rankings would change if magnitudes were differentiated or if the effect signs were assigned differently. At a minimum, the scores should be described as ordinal and the reported percentages should be accompanied by a sensitivity analysis over plausible effect-weight variations.
- [Sections 4 and 5] The central empirical findings rest on inputs from a single study participant, who supplied the QA set, the effect signs, the importance and risk levels, and the theoretical-optimal DMatrix. The weights wI and wR are fixed to 0.5 without exploration, and no inter-rater comparison or independent validation is reported. The conclusions such as 'the multi model approach performs better across economic and social dimensions' are deterministic transformations of those inputs, so they cannot be read as robust empirical findings. The paper should either add a sensitivity analysis that varies the inputs within plausible ranges, or explicitly frame the case study as a feasibility demonstration rather than as evidence that these particular trade-offs hold for the MMvIB system generally.
minor comments (6)
- [Section 5.3] There is a typo: 'Is it theocratically possible' should be 'Is it theoretically possible.'
- [Section 3] The phrase 'containerless provided sustainability support' appears to be a typo; the intended word is likely 'containerization.'
- [Equations (1) and (2)] The summation notation 'n,mX' is difficult to read; please use standard double-sum notation such as \sum_{i=1}^n \sum_{j=1}^m.
- [Table 2] The 'x' marks in the sustainability-dimension columns are not explained; a legend is needed to clarify that they indicate the dimension to which each QA belongs.
- [Section 2.3] The example 'T-Ec with E-Ec' likely contains a typo; it should probably be 'T-Ec with En-Ec' or 'T-Ec with S-Ec,' since no 'E-Ec' dimension appears elsewhere in the paper.
- [Section 6] The statement 'we do not have a lower bound for a SIS value' conflicts with Eq. (4), which uses min(SIS) as a lower bound. Please state explicitly that min(SIS) is the observed minimum among the compared alternatives, not an absolute lower bound.
Circularity Check
Cross-pair comparability of the normalized SIS is an artifact of pair-relative min-max anchoring; the rest of the case study is a transparent weighted-sum application.
-
self definitional
[Section 4, Eq. 4; Table 3; Section 6, 'Inferring the SIS values']
"Normalized SIS (%) = SIS − min(SIS ) T O(SIS ) − min(SIS ) × 100 (4) TO = Theoretical Optimal ... This makes the SIS comparable across different dimension pairs. ... Since we do not have a lower bound for a SIS value, an open question remains: how bad are the lower values of SIS?"
In Eq. 4 the zero point is min(SIS), which Table 3 shows is the lowest observed SIS within each dimension pair: T-Ec min = -0.425, T-En min = -1.05, T-S min = 1.75, S-Ec min = 0. The 0% anchor therefore differs between pairs, and Section 6 concedes 'we do not have a lower bound for a SIS value.' Hence a normalized percentage is a within-pair distance from that pair's observed worst case, not a value on a shared scale.
full rationale
The main SIS computation (Eqs. 2-3) is a transparent weighted-sum aggregation of stakeholder-elicited priorities and effects; presenting a case-study application of such a score is not circular in itself, and the paper does not claim external validation. The self-citations to prior SIS work and the SAF Toolkit are used as normal incremental references, not as a uniqueness theorem or a substitute for argument. The load-bearing circularity is confined to the normalization step: the cross-dimension comparability that the paper claims as an improvement is built into Eq. 4's use of the per-pair observed minimum as a zero point, so the comparative percentages reduce to the chosen rescaling by construction rather than to an independent measurement. For this reason, the paper's most quantitative comparative claims should be read as illustrations of the method, not as validated cross-pair measurements.
Assumptions & free parameters
free parameters (4)
- Importance and risk levels (I, R) for each quality attribute =
18 values as given in Table 2 (e.g., Traceability I=3, R=3; Reproducibility I=2, R=1)
- Weights wI and wR in priority function =
0.5 and 0.5 (Section 5.1)
- Impact values (+1, -1, 0) in DMatrices =
Set for each pair of QAs; contained in the replication package
- Theoretical optimal DMatrix =
Not given in full; described in Section 5.3 and replication package
assumptions (4)
- domain assumption Quality-attribute effects can be represented as +1, -1, or 0 with equal magnitude
- domain assumption The Dependency Matrix framework from [7] captures interdimensional dependencies
- domain assumption The Sustainability Quality Model from the authors' SAF Toolkit [20] correctly maps quality concerns to sustainability dimensions
- domain assumption A single participant's knowledge is sufficient to identify all relevant quality concerns and effects
Cite this review
Pith. "Pith review of Using Sustainability Impact Scores for Software Architecture Evaluation." pith.science (2026). https://pith.science/paper/NKREWFS6
@misc{pith2026250117004,
author = {Pith},
title = {Pith review of: Using Sustainability Impact Scores for Software Architecture Evaluation},
year = {2026},
howpublished = {\url{https://pith.science/paper/NKREWFS6}},
note = {Machine review of arXiv:2501.17004}
}
read the original abstract
For future regulatory compliance, organizations must assess and report on the state of sustainability in terms of its impacts over time. Sustainability, being a multidimensional concern, is complex to quantify. This complexity further increases with the interdependencies of the quality concerns across different sustainability dimensions. The research literature lacks a holistic way to evaluate sustainability at the software architecture level. With this study, our aim is to identify quality attribute (QA) trade-offs at the software architecture level and quantify the related sustainability impact. To this aim we present an improved version of the Sustainability Impact Score (SIS), building on our previous work. The SIS facilitates the identification and quantification of trade-offs in terms of their sustainability impact, leveraging a risk- and importance-based prioritization mechanism. To evaluate our approach, we apply it to an industrial case study involving a multi-model framework for integrated decision-making in the energy sector. Our study reveals that technical quality concerns have significant, often unrecognized impacts across sustainability dimensions. The SIS coupled with QA trade-offs can help practitioners make informed decisions that align with their sustainability goals. Early evaluations can help organizations mitigate sustainability risks by taking preventive actions.
Figures
Forward citations
Cited by 1 Pith paper
-
Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices
A 29-participant workshop synthesized a research agenda for reducing AI's environmental footprint through software engineering, covering measurement, benchmarking, architecture, empirical methods, and education.
Reference graph
Works this paper leans on
-
[1]
Security risk analysis of software architecture based on AHP
Abdulaziz Alkussayer and William H Allen. Security risk analysis of software architecture based on AHP. In 7th Int. Conf. on Networked Computing . IEEE, 2011
work page 2011
-
[2]
Software Architecture in Practice
Len Bass, Paul Clements, and Rick Kazman. Software Architecture in Practice. Addison-Wesley Professional, 3rd edition, 2012
work page 2012
-
[3]
A Handbook on Multi-Attribute Decision-Making Methods
Omid Bozorg-Haddad, Hugo Lo ´aiciga, and Babak Zolghadr-Asli. A Handbook on Multi-Attribute Decision-Making Methods . Wiley, 2021
work page 2021
-
[4]
Software architecture evolution through evolvability analysis
Hongyu Pei Breivold, Ivica Crnkovic, and Magnus Larsson. Software architecture evolution through evolvability analysis. Journal of Systems and Software , 85(11), 2012
work page 2012
-
[5]
Using Multi-criteria Analysis to Evaluate En- terprise Architecture Scenarios
Francisco Cansado, Andr ´e Vasconcelos, and Gonc ¸alo Santos. Using Multi-criteria Analysis to Evaluate En- terprise Architecture Scenarios. In Proceedings of the 14th International Conference on Enterprise Informa- tion Systems (ICEIS) . SciTePress, 2012
work page 2012
-
[6]
Nelly Condori-Fernandez, Patricia Lago, Miguel R. Luaces, and Angeles S. Places. An Action Research for Improving the Sustainability Assessment Framework Instruments. Sustainability, 12(4), 2020
work page 2020
-
[7]
Defining Interdi- mensional Dependencies of the Sustainability-Quality Model
Nelly Condori-Fernandez, Patricia Lago, Alejandro Catala, and Miguel R Luaces. Defining Interdi- mensional Dependencies of the Sustainability-Quality Model. Technical report, VU Amsterdam, 2024
work page 2024
-
[8]
Udaya Dayanandan and Vivekanandan Kalimuthu. Software Architectural Quality Assessment Model for Security Analysis Using Fuzzy Analytical Hierarchy Process (FAHP) Method. 3D Research, 2018
work page 2018
Show all 29 references
-
[9]
A Fuzzy Analytical Hierarchy Process (FAHP) Based Software Quality Assessment Model: Maintainability Analysis
Udaya Dayanandan and Vivekanandan Kalimuthu. A Fuzzy Analytical Hierarchy Process (FAHP) Based Software Quality Assessment Model: Maintainability Analysis. International Journal of Intelligent Engi- neering and Systems , 2018
2018
-
[10]
Sustainability competencies and skills in software engineering: An industry perspective
Heldal et al. Sustainability competencies and skills in software engineering: An industry perspective. Journal of Systems and Software , 211, 2024
2024
-
[11]
Sustainable software engineering: Reflec- tions on advances in research and practice
Venters et al. Sustainable software engineering: Reflec- tions on advances in research and practice. Information and Software Technology , 164, 2023
2023
-
[12]
CSRD directive, 2021
European Commission. CSRD directive, 2021. https: //x.gd/hecGA
2021
-
[13]
A Review of Software Architecture Evaluation Methods for Sustainability As- sessment
Iffat Fatima and Patricia Lago. A Review of Software Architecture Evaluation Methods for Sustainability As- sessment. In 2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C) , 2023
2023
-
[14]
Software Architecture Assessment for Sustainability: A Case Study
Iffat Fatima and Patricia Lago. Software Architecture Assessment for Sustainability: A Case Study. In Soft- ware Architecture. Springer Nature Switzerland, 2024
2024
-
[15]
Replication Package: Using Sustainability Impact Scores for Software Architecture Evaluation (1.0), 2024
Iffat Fatima, Patricia Lago, Vasilios Andrikopoulos, and Bram van der Waaij. Replication Package: Using Sustainability Impact Scores for Software Architecture Evaluation (1.0), 2024. URL http://doi.org/10.5281/ zenodo.14732178
2024
-
[16]
ICT for sustainability: An emerging research field
Lorenz M Hilty and Bernard Aebischer. ICT for sustainability: An emerging research field. In Advances in Intelligent Systems and Computing , Advances in intelligent systems and computing. Springer, 2015
2015
-
[17]
An Overview of Corporate Sustainability Reporting Legislation in the European Union
Katrin Hummel and Dominik Jobst. An Overview of Corporate Sustainability Reporting Legislation in the European Union. Accounting in Europe , 21(3), 2024
2024
-
[18]
Jansen and J
A. Jansen and J. Bosch. Software Architecture as a Set of Architectural Design Decisions. In 5th Working IEEE/IFIP Conference on Software Architecture, 2005
2005
-
[19]
Jeromy Carri`ere, and Steven G
Rick Kazman, Mario Barbacci, Mark Klein, S. Jeromy Carri`ere, and Steven G. Woods. Experience with per- forming architecture tradeoff analysis. In Proceedings of the 1999 Int. Conf. on Software Engineering , 1999
1999
-
[20]
P. Lago, N. Condori Fernandez, I. Fatima, M. Funke, and I. Malavolta. The sustainability assessment frame- work toolkit: a decade of modeling experience. Soft- ware and Systems Modeling , 2024. URL https://doi. org/10.1007/s10270-024-01230-9
2024 doi
-
[21]
Framing Sustainability as a Property of Software Quality
Patricia Lago, Sedef Akinli Koc ¸ak, Ivica Crnkovic, and Birgit Penzenstadler. Framing Sustainability as a Property of Software Quality. Comm. ACM, 58(10), 2015
2015
-
[22]
Containerization vs
Chandrakanth Lekkala. Containerization vs. Serverless Architectures for Data Pipelines. Journal of Engineer- ing and Applied Sciences Technology , 2023
2023
-
[23]
Multimodelling documentation, 2023
Edwin Matthijssen, Ewoud Werkman, and Serkan Ke- skin. Multimodelling documentation, 2023. URL multimodelling.readthedocs.io/en/latest. [2024-06-17]
2023
-
[24]
Toward an architec- tural knowledge base for wireless service engineering
E Niemela, J Kalaoja, and P Lago. Toward an architec- tural knowledge base for wireless service engineering. IEEE Transactions on Software Engineering , 31(5): 361–379, May 2005
2005
-
[25]
Sheikhvand
Mahsa Razavi and K. Sheikhvand. An Approach towards Enterprise Architecture Analysis using AHP and Fuzzy AHP. International Journal of Machine Learning and Computing , 2012
2012
-
[26]
A new AHP-based approach towards Enterprise Ar- chitecture quality attribute analysis
Mahsa Razavi Davoudi and Fereidoon Shams Aliee. A new AHP-based approach towards Enterprise Ar- chitecture quality attribute analysis. In 2009 Third International Conference on Research Challenges in Information Science , 2009
2009
-
[27]
The analytic hierarchy process
Thomas L Saaty. The analytic hierarchy process. Agricultural Economics Review , 70(804), 1980
1980
-
[28]
Why Software Ar- chitects Must Be Involved in the Earliest Systems Engineering Activities
Sarah Sheard, Suzanne Miller, Bill Nichols, Donald Firesmith, and Michael Phillips. Why Software Ar- chitects Must Be Involved in the Earliest Systems Engineering Activities. Carnegie Mellon University, Software Engineering Institute’s Insights (blog), 2019
2019
-
[29]
Ohlsson, Bj ¨orn Regnell, and Anders Wessl ´en
Claes Wohlin, Per Runeson, Martin H ¨ost, Magnus C. Ohlsson, Bj ¨orn Regnell, and Anders Wessl ´en. Experi- mentation in Software Engineering . Springer, 2012
2012
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.