Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Using Sustainability Impact Scores for Software Architecture Evaluation

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Technical quality choices carry measurable sustainability consequences that a weighted impact score can expose.

desk verdict The proposed SIS extension is a real step forward for sustainability-aware architecture evaluation, but the paper's headline claim of cross-dimension comparability is undermined by a flawed normalization that anchors to observed minima, not an absolute lower bound. read the letter →

arxiv 2501.17004 v1 pith:NKREWFS6 submitted 2025-01-28 cs.SE

classification cs.SE
keywords softwarearchitectureevaluationsustainabilityimpactscorequalityattributetrade-offsdimensionstrade-offanalysismulti-modelsystemsdecisionmapsreporting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that sustainability impact at the software architecture level can be quantified with an improved Sustainability Impact Score (SIS), a weighted sum of quality-attribute effects across sustainability dimensions. The authors apply the score to a real energy-sector decision-support system and report that technical quality attributes have significant, often hidden impacts on economic, environmental, and social concerns. If correct, the score gives architects a way to compare design options by their sustainability consequences before implementation, and gives organizations a documented basis for sustainability reporting such as the CSRD regulation. The central pay-off is early visibility: trade-offs that usually surface only as side effects can be made explicit and managed during architecture evaluation.

What carries the argument

The central mechanism is the Sustainability Impact Score (SIS): a weighted sum over dependency matrices that pair quality attributes of two sustainability dimensions, with each cross-effect recorded as $+1$, $-1$, or $0$. Priorities in the sum are derived from a utility matrix that maps scenarios to quality attributes and scores each attribute by importance and risk, normalized to $[0.1,1]$. To make scores comparable across dimension pairs, the paper normalizes each score against a theoretical-optimal dependency matrix in which negative effects are minimized, converting the result to a percentage. This machinery turns subjective architecture reasoning—which quality attributes matter and how they interact—into a single numeric comparison per dimension pair.

What would settle it

Recompute the SIS for the single-model and multi-model approaches using effect magnitudes estimated from actual system measurements (for example, the monetary cost or energy use associated with each dependency) instead of uniform $\pm 1$; if any dimension-pair ranking between the two approaches reverses, the equal-magnitude assumption is not sufficient for the reported comparison.

Watch

Extended reading notes

Core claim

On its own terms, the paper argues that a risk- and importance-weighted Sustainability Impact Score can turn architecture trade-offs into comparable numbers. The score is computed for pairs of sustainability dimensions as $\mathrm{SIS}_{\mathrm{dim1},\mathrm{dim2}} = \sum_{i,j}(\mathrm{Priority}_{\mathrm{dim1},i} + \mathrm{Priority}_{\mathrm{dim2},j}) \times \mathrm{Impact}_{ij}$, with effects taken as $+1$, $-1$, or $0$ from dependency matrices; priorities come from a weighted combination of each quality attribute's importance and risk. Compared with a theoretical-optimal dependency matrix, normalized scores express how close an architecture comes to fully supporting a dimension. In the MMvIB case, the multi-model approach scores highest for social and economic support while both approaches underperform on the environmental dimension, and the analysis exposes chains such as traceability improving reproducibility, transparency, stakeholder stake, and ultimately monetary cost. The paper's central claim is that these cross-dimension effects are real, often unrecognized, and can be surfaced by the SIS procedure.

Load-bearing premise

The load-bearing simplification is that every quality attribute's effect on another has the same magnitude, recorded as $+1$, $-1$, or $0$; if real effects differ in strength, the scores and the ranking of architectures could change.

Editorial extensions

If this is right

  • Architects can compare alternative designs by normalized SIS per sustainability dimension pair, not just within one pair.
  • Enabling and systemic sustainability effects, such as reproducibility requirements increasing resource utilization, become visible at evaluation time instead of after deployment.
  • High-priority technical attributes that harm another dimension can be explicitly weighed against the consequences of changing them, as in the reproducibility-resource utilization trade-off.
  • Organizations gain a repeatable, documented path toward sustainability reporting obligations, including preparing for regulations like CSRD.
  • The relative comparison to a theoretical optimal provides a benchmark for how much sustainability support an architecture is leaving on the table.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the equal-magnitude effect assumption could be stress-tested by replacing uniform $\pm 1$ values with measured strengths and re-running the MMvIB comparison; the ranking would hold only if it survives that perturbation.
  • Beyond the paper, the normalized SIS could be inverted into a design objective, letting architects search over workflow configurations for the highest sustainability support across all dimension pairs.
  • Beyond the paper, if external benchmarks such as carbon budgets become available, the relative percentages could be anchored to absolute targets, turning the score from a comparison device into a compliance metric.
  • Beyond the paper, multi-stakeholder projects would need an explicit consensus procedure for risk, importance, and effect values; the utility-matrix structure suggests a group-weighted aggregation as a natural next step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an improved Sustainability Impact Score (SIS) for software architecture evaluation. The method extends prior work by computing SIS for any pair of sustainability dimensions, deriving QA priorities from stakeholder-provided importance and risk levels via a weighted sum, and normalizing scores relative to a theoretical-optimal DMatrix. The approach is demonstrated on a single industrial case study (MMvIB) comparing a single-model and a multi-model architecture. The paper reports inter-QA trade-offs and normalized SIS percentages for the T-Ec, T-En, T-S, and S-Ec dimension pairs, concluding that the multi-model approach is better for the economic and social dimensions and that both approaches underperform on the environmental dimension.

Significance. If the method were sound, it would provide a lightweight, practitioner-oriented tool for early sustainability assessment in architecture evaluation, complementing heavier scenario-based methods such as ATAM. The risk- and importance-based prioritization is a reasonable adaptation of ATAM's utility tree concept, and the authors ship a replication package, which is a concrete strength. The central limitation claims of the prior SIS (Section 2.3) are addressed in principle. However, the significance is conditional on fixing a load-bearing normalization problem and on treating the single-participant case study as an illustration rather than as evidence of generalizable empirical findings. The paper is honest in acknowledging several threats to validity, but the main contribution as stated, cross-dimension comparability of SIS values, is not currently established.

major comments (3)
  1. [Section 5.3] The normalization in Eq. (4) uses the observed minimum SIS per dimension pair as the lower bound, not a common absolute lower bound. In Table 3, for T-Ec the minimum is -0.425 (Single Model), while for T-En the minimum is -1.05 (Multi Model). Consequently, a normalized value of 0% corresponds to different raw SIS values in different rows and columns, so the normalized percentages are not on a common scale. This directly contradicts the claim in Section 5.3 that 'the normalization allowed us to compare the SIS values across dimensions,' and it fails to overcome limitation (ii) of Section 2.3. The issue is independent of the equal-magnitude assumption: even with exact effect magnitudes, the percentages cannot be compared across pairs because the anchor point varies. Section 6 even states that 'we do not have a lower bound for a SIS value,' so the min(SIS) used in Eq. (4) is only an artifact of the alternatives considered. The authors should either define a theoretical lower bound (e.g., a worst-case DMatrix) that is common across all pairs, or explicitly restrict all comparative claims to within-pair comparisons.
  2. [Section 5.3] The assumption that all QA effects have the same magnitude (+1 or -1) is load-bearing for the reported scores. The authors acknowledge this explicitly: 'one QA may affect another QA more positively or negatively than another QA.' Because all effects are equal-magnitude, the SIS values reduce to weighted counts of positive and negative edges, and the differences between alternatives (e.g., 76.51% vs 28.41% for T-Ec) depend entirely on which edges are present. No sensitivity analysis is provided to show how the rankings would change if magnitudes were differentiated or if the effect signs were assigned differently. At a minimum, the scores should be described as ordinal and the reported percentages should be accompanied by a sensitivity analysis over plausible effect-weight variations.
  3. [Sections 4 and 5] The central empirical findings rest on inputs from a single study participant, who supplied the QA set, the effect signs, the importance and risk levels, and the theoretical-optimal DMatrix. The weights wI and wR are fixed to 0.5 without exploration, and no inter-rater comparison or independent validation is reported. The conclusions such as 'the multi model approach performs better across economic and social dimensions' are deterministic transformations of those inputs, so they cannot be read as robust empirical findings. The paper should either add a sensitivity analysis that varies the inputs within plausible ranges, or explicitly frame the case study as a feasibility demonstration rather than as evidence that these particular trade-offs hold for the MMvIB system generally.
minor comments (6)
  1. [Section 5.3] There is a typo: 'Is it theocratically possible' should be 'Is it theoretically possible.'
  2. [Section 3] The phrase 'containerless provided sustainability support' appears to be a typo; the intended word is likely 'containerization.'
  3. [Equations (1) and (2)] The summation notation 'n,mX' is difficult to read; please use standard double-sum notation such as \sum_{i=1}^n \sum_{j=1}^m.
  4. [Table 2] The 'x' marks in the sustainability-dimension columns are not explained; a legend is needed to clarify that they indicate the dimension to which each QA belongs.
  5. [Section 2.3] The example 'T-Ec with E-Ec' likely contains a typo; it should probably be 'T-Ec with En-Ec' or 'T-Ec with S-Ec,' since no 'E-Ec' dimension appears elsewhere in the paper.
  6. [Section 6] The statement 'we do not have a lower bound for a SIS value' conflicts with Eq. (4), which uses min(SIS) as a lower bound. Please state explicitly that min(SIS) is the observed minimum among the compared alternatives, not an absolute lower bound.

Circularity Check

1 steps flagged · score 5.0 of 10

Cross-pair comparability of the normalized SIS is an artifact of pair-relative min-max anchoring; the rest of the case study is a transparent weighted-sum application.

  1. self definitional [Section 4, Eq. 4; Table 3; Section 6, 'Inferring the SIS values']
    "Normalized SIS (%) = SIS − min(SIS ) T O(SIS ) − min(SIS ) × 100 (4) TO = Theoretical Optimal ... This makes the SIS comparable across different dimension pairs. ... Since we do not have a lower bound for a SIS value, an open question remains: how bad are the lower values of SIS?"

    In Eq. 4 the zero point is min(SIS), which Table 3 shows is the lowest observed SIS within each dimension pair: T-Ec min = -0.425, T-En min = -1.05, T-S min = 1.75, S-Ec min = 0. The 0% anchor therefore differs between pairs, and Section 6 concedes 'we do not have a lower bound for a SIS value.' Hence a normalized percentage is a within-pair distance from that pair's observed worst case, not a value on a shared scale.

full rationale

The main SIS computation (Eqs. 2-3) is a transparent weighted-sum aggregation of stakeholder-elicited priorities and effects; presenting a case-study application of such a score is not circular in itself, and the paper does not claim external validation. The self-citations to prior SIS work and the SAF Toolkit are used as normal incremental references, not as a uniqueness theorem or a substitute for argument. The load-bearing circularity is confined to the normalization step: the cross-dimension comparability that the paper claims as an improvement is built into Eq. 4's use of the per-pair observed minimum as a zero point, so the comparative percentages reduce to the chosen rescaling by construction rather than to an independent measurement. For this reason, the paper's most quantitative comparative claims should be read as illustrations of the method, not as validated cross-pair measurements.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests entirely on participant-provided inputs and design choices (weights, normalization bounds, equal-impact magnitudes). There are no measured quantities, external benchmarks, or validated constants. The method is transparent, but every number in the case study traces back to a judgment call.

free parameters (4)
  • Importance and risk levels (I, R) for each quality attribute = 18 values as given in Table 2 (e.g., Traceability I=3, R=3; Reproducibility I=2, R=1)
    These levels are chosen by the single participant and determine the priority values via Eq. 3. They are not derived from measurement or external data.
  • Weights wI and wR in priority function = 0.5 and 0.5 (Section 5.1)
    The paper says stakeholders can set these weights; in the case study they were fixed to equal 0.5, which is a modeling choice.
  • Impact values (+1, -1, 0) in DMatrices = Set for each pair of QAs; contained in the replication package
    The participant assigns the direction of effect between QAs on different dimensions. These are hand-assigned and not measured.
  • Theoretical optimal DMatrix = Not given in full; described in Section 5.3 and replication package
    The benchmark used in Eq. 4 is constructed by the researchers and participant, and it directly sets the 100% reference point. The lower bound for normalization is the observed minimum among alternatives, which the paper does not justify.
assumptions (4)
  • domain assumption Quality-attribute effects can be represented as +1, -1, or 0 with equal magnitude
    The paper explicitly states this simplification in Section 5.3 and leaves magnitude differentiation to future work.
  • domain assumption The Dependency Matrix framework from [7] captures interdimensional dependencies
    The DMatrix representation is adopted without critical examination; the paper relies on it as the representation of QA effects.
  • domain assumption The Sustainability Quality Model from the authors' SAF Toolkit [20] correctly maps quality concerns to sustainability dimensions
    This is used in Section 5.1 (Table 2) to label QAs as technical, economic, environmental, or social; the model comes from the authors' own toolkit.
  • domain assumption A single participant's knowledge is sufficient to identify all relevant quality concerns and effects
    The paper uses one architect as the sole data source; Section 7 acknowledges this as an internal validity threat.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using Sustainability Impact Scores for Software Architecture Evaluation." pith.science (2026). https://pith.science/paper/NKREWFS6

@misc{pith2026250117004,
  author       = {Pith},
  title        = {Pith review of: Using Sustainability Impact Scores for Software Architecture Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NKREWFS6}},
  note         = {Machine review of arXiv:2501.17004}
}
read the original abstract

For future regulatory compliance, organizations must assess and report on the state of sustainability in terms of its impacts over time. Sustainability, being a multidimensional concern, is complex to quantify. This complexity further increases with the interdependencies of the quality concerns across different sustainability dimensions. The research literature lacks a holistic way to evaluate sustainability at the software architecture level. With this study, our aim is to identify quality attribute (QA) trade-offs at the software architecture level and quantify the related sustainability impact. To this aim we present an improved version of the Sustainability Impact Score (SIS), building on our previous work. The SIS facilitates the identification and quantification of trade-offs in terms of their sustainability impact, leveraging a risk- and importance-based prioritization mechanism. To evaluate our approach, we apply it to an industrial case study involving a multi-model framework for integrated decision-making in the energy sector. Our study reveals that technical quality concerns have significant, often unrecognized impacts across sustainability dimensions. The SIS coupled with QA trade-offs can help practitioners make informed decisions that align with their sustainability goals. Early evaluations can help organizations mitigate sustainability risks by taking preventive actions.

Figures

Figures reproduced from arXiv: 2501.17004 by the authors.

Figure 1
Figure 1. DMatrix Overview 2.2. Sustainability Impact Score In our previous work [14], we introduced the concept of a SIS, which quantifies the impact of QAs within the technical dimension T about QAs within other sustainability dimensions dim(Ec, En, S). Taking inspiration from the weighted sum model [3], this score uses QA priority values as weights to assess the impact of QA. Subsequently, it computes the cumulative sum of… view at source ↗
Figure 2
Figure 2. DMatrices for a. Serverless (left) b. Containerization (middle) c. Theoretical Optimal (right) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Decision Map showing quality concerns of the MMvIB IT Architecture using Multi Model Approach [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Greening AI-enabled Systems with Software Engineering: A Research Agenda for Environmentally Sustainable AI Practices

    cs.SE 2025-06 accept novelty 4.0 of 10

    A 29-participant workshop synthesized a research agenda for reducing AI's environmental footprint through software engineering, covering measurement, benchmarking, architecture, empirical methods, and education.

Reference graph

Works this paper leans on

29 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Security risk analysis of software architecture based on AHP

    Abdulaziz Alkussayer and William H Allen. Security risk analysis of software architecture based on AHP. In 7th Int. Conf. on Networked Computing . IEEE, 2011

  2. [2]

    Software Architecture in Practice

    Len Bass, Paul Clements, and Rick Kazman. Software Architecture in Practice. Addison-Wesley Professional, 3rd edition, 2012

  3. [3]

    A Handbook on Multi-Attribute Decision-Making Methods

    Omid Bozorg-Haddad, Hugo Lo ´aiciga, and Babak Zolghadr-Asli. A Handbook on Multi-Attribute Decision-Making Methods . Wiley, 2021

  4. [4]

    Software architecture evolution through evolvability analysis

    Hongyu Pei Breivold, Ivica Crnkovic, and Magnus Larsson. Software architecture evolution through evolvability analysis. Journal of Systems and Software , 85(11), 2012

  5. [5]

    Using Multi-criteria Analysis to Evaluate En- terprise Architecture Scenarios

    Francisco Cansado, Andr ´e Vasconcelos, and Gonc ¸alo Santos. Using Multi-criteria Analysis to Evaluate En- terprise Architecture Scenarios. In Proceedings of the 14th International Conference on Enterprise Informa- tion Systems (ICEIS) . SciTePress, 2012

  6. [6]

    Luaces, and Angeles S

    Nelly Condori-Fernandez, Patricia Lago, Miguel R. Luaces, and Angeles S. Places. An Action Research for Improving the Sustainability Assessment Framework Instruments. Sustainability, 12(4), 2020

  7. [7]

    Defining Interdi- mensional Dependencies of the Sustainability-Quality Model

    Nelly Condori-Fernandez, Patricia Lago, Alejandro Catala, and Miguel R Luaces. Defining Interdi- mensional Dependencies of the Sustainability-Quality Model. Technical report, VU Amsterdam, 2024

  8. [8]

    Software Architectural Quality Assessment Model for Security Analysis Using Fuzzy Analytical Hierarchy Process (FAHP) Method

    Udaya Dayanandan and Vivekanandan Kalimuthu. Software Architectural Quality Assessment Model for Security Analysis Using Fuzzy Analytical Hierarchy Process (FAHP) Method. 3D Research, 2018

Show all 29 references
  1. [9]

    A Fuzzy Analytical Hierarchy Process (FAHP) Based Software Quality Assessment Model: Maintainability Analysis

    Udaya Dayanandan and Vivekanandan Kalimuthu. A Fuzzy Analytical Hierarchy Process (FAHP) Based Software Quality Assessment Model: Maintainability Analysis. International Journal of Intelligent Engi- neering and Systems , 2018

  2. [10]

    Sustainability competencies and skills in software engineering: An industry perspective

    Heldal et al. Sustainability competencies and skills in software engineering: An industry perspective. Journal of Systems and Software , 211, 2024

  3. [11]

    Sustainable software engineering: Reflec- tions on advances in research and practice

    Venters et al. Sustainable software engineering: Reflec- tions on advances in research and practice. Information and Software Technology , 164, 2023

  4. [12]

    CSRD directive, 2021

    European Commission. CSRD directive, 2021. https: //x.gd/hecGA

  5. [13]

    A Review of Software Architecture Evaluation Methods for Sustainability As- sessment

    Iffat Fatima and Patricia Lago. A Review of Software Architecture Evaluation Methods for Sustainability As- sessment. In 2023 IEEE 20th International Conference on Software Architecture Companion (ICSA-C) , 2023

  6. [14]

    Software Architecture Assessment for Sustainability: A Case Study

    Iffat Fatima and Patricia Lago. Software Architecture Assessment for Sustainability: A Case Study. In Soft- ware Architecture. Springer Nature Switzerland, 2024

  7. [15]

    Replication Package: Using Sustainability Impact Scores for Software Architecture Evaluation (1.0), 2024

    Iffat Fatima, Patricia Lago, Vasilios Andrikopoulos, and Bram van der Waaij. Replication Package: Using Sustainability Impact Scores for Software Architecture Evaluation (1.0), 2024. URL http://doi.org/10.5281/ zenodo.14732178

  8. [16]

    ICT for sustainability: An emerging research field

    Lorenz M Hilty and Bernard Aebischer. ICT for sustainability: An emerging research field. In Advances in Intelligent Systems and Computing , Advances in intelligent systems and computing. Springer, 2015

  9. [17]

    An Overview of Corporate Sustainability Reporting Legislation in the European Union

    Katrin Hummel and Dominik Jobst. An Overview of Corporate Sustainability Reporting Legislation in the European Union. Accounting in Europe , 21(3), 2024

  10. [18]

    Jansen and J

    A. Jansen and J. Bosch. Software Architecture as a Set of Architectural Design Decisions. In 5th Working IEEE/IFIP Conference on Software Architecture, 2005

  11. [19]

    Jeromy Carri`ere, and Steven G

    Rick Kazman, Mario Barbacci, Mark Klein, S. Jeromy Carri`ere, and Steven G. Woods. Experience with per- forming architecture tradeoff analysis. In Proceedings of the 1999 Int. Conf. on Software Engineering , 1999

  12. [20]

    P. Lago, N. Condori Fernandez, I. Fatima, M. Funke, and I. Malavolta. The sustainability assessment frame- work toolkit: a decade of modeling experience. Soft- ware and Systems Modeling , 2024. URL https://doi. org/10.1007/s10270-024-01230-9

  13. [21]

    Framing Sustainability as a Property of Software Quality

    Patricia Lago, Sedef Akinli Koc ¸ak, Ivica Crnkovic, and Birgit Penzenstadler. Framing Sustainability as a Property of Software Quality. Comm. ACM, 58(10), 2015

  14. [22]

    Containerization vs

    Chandrakanth Lekkala. Containerization vs. Serverless Architectures for Data Pipelines. Journal of Engineer- ing and Applied Sciences Technology , 2023

  15. [23]

    Multimodelling documentation, 2023

    Edwin Matthijssen, Ewoud Werkman, and Serkan Ke- skin. Multimodelling documentation, 2023. URL multimodelling.readthedocs.io/en/latest. [2024-06-17]

  16. [24]

    Toward an architec- tural knowledge base for wireless service engineering

    E Niemela, J Kalaoja, and P Lago. Toward an architec- tural knowledge base for wireless service engineering. IEEE Transactions on Software Engineering , 31(5): 361–379, May 2005

  17. [25]

    Sheikhvand

    Mahsa Razavi and K. Sheikhvand. An Approach towards Enterprise Architecture Analysis using AHP and Fuzzy AHP. International Journal of Machine Learning and Computing , 2012

  18. [26]

    A new AHP-based approach towards Enterprise Ar- chitecture quality attribute analysis

    Mahsa Razavi Davoudi and Fereidoon Shams Aliee. A new AHP-based approach towards Enterprise Ar- chitecture quality attribute analysis. In 2009 Third International Conference on Research Challenges in Information Science , 2009

  19. [27]

    The analytic hierarchy process

    Thomas L Saaty. The analytic hierarchy process. Agricultural Economics Review , 70(804), 1980

  20. [28]

    Why Software Ar- chitects Must Be Involved in the Earliest Systems Engineering Activities

    Sarah Sheard, Suzanne Miller, Bill Nichols, Donald Firesmith, and Michael Phillips. Why Software Ar- chitects Must Be Involved in the Earliest Systems Engineering Activities. Carnegie Mellon University, Software Engineering Institute’s Insights (blog), 2019

  21. [29]

    Ohlsson, Bj ¨orn Regnell, and Anders Wessl ´en

    Claes Wohlin, Per Runeson, Martin H ¨ost, Magnus C. Ohlsson, Bj ¨orn Regnell, and Anders Wessl ´en. Experi- mentation in Software Engineering . Springer, 2012

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.