Supporting architecture evaluation for ATAM scenarios with LLMs

J. Andr\'es D\'iaz-Pace; Jennifer P\'erez; Rafael Capilla; Vanessa Rodr\'iguez-Horcajo; Yamid Ram\'irez

REVIEW 4 major objections 6 minor 15 references

Supporting architecture evaluation for ATAM scenarios with LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM fed an architecture assignment and prompted stepwise detects more ATAM risks, sensitivity points, and tradeoffs than a student evaluation team, enabling partial automation of architecture evaluation.

desk verdict A useful preliminary probe of LLM-assisted ATAM that overstates its own evidence: 'more' is shown, 'better and more accurate' is not. read the letter →

arxiv 2506.00150 v1 pith:GZX6AA2Y submitted 2025-05-30 cs.SE cs.AI

Rafael Capilla , J. Andrés Díaz-Pace , Yamid Ramírez , Jennifer Pérez , Vanessa Rodríguez-Horcajo This is my paper

classification cs.SEcs.AI

keywords ATAMarchitectureevaluationquality-attributescenarioslargelanguagemodelsriskidentificationsensitivitypointstradeoffanalysisscenarioselectionretrieval-augmentedgeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports an initial study asking whether a large language model can take over parts of ATAM, a manual software-architecture evaluation method that relies on stakeholder brainstorming to identify quality scenarios. The authors fed nine student ATAM assignments into MS Copilot using a retrieval-augmented prompt sequence and compared Copilot's assessment of risks, sensitivity points, tradeoffs, and final scenario selections with the students' own results. Their central claim is that the LLM produced in most cases better and more accurate results than the students, finding additional risks and sensitivity points and identifying more quality-attribute tradeoffs, while agreeing with the students' chosen scenarios in 26 of 30 cases. If true, this matters because architecture evaluation is labor-intensive and slow, and even partial automation could shorten brainstorming sessions and give architects a second opinion on which scenarios deserve attention.

What carries the argument

The carrying mechanism is a four-prompt protocol applied to MS Copilot: identify risks, identify sensitivity points, perform a tradeoff analysis, and select scenarios, all over a retrieval-augmented document set taken from each student project. The central object is the ATAM utility tree, the structured collection of quality scenarios that ties each scenario to a quality attribute, a stimulus, and a measurable response; the prompts ask the LLM to re-derive the evaluation judgments, namely risks, sensitivity points, and tradeoffs, that ATAM teams would normally produce manually.

What would settle it

Have a panel of independent ATAM experts sort every risk, sensitivity point, and tradeoff that Copilot reported for the seven analyzed groups into 'true concern for this architecture' and 'not a real concern', then compare those precision rates with the same sort for the students' lists. If Copilot's added items are mostly judged irrelevant or hallucinated, the paper's claim that the LLM is 'better and more accurate' would fail; if they are mostly judged real, the claim survives.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an LLM equipped with a retrieval-augmented context containing the project's architecture description and prompted stepwise can reproduce and extend the judgment of novice ATAM evaluators: it detects risks and sensitivity points the students missed, reports more tradeoff pairs between quality attributes, and selects scenarios that mostly align with the students' picks. The paper frames this as evidence that generative AI can partially automate architecture evaluation and improve human decision-making, not replace it.

Load-bearing premise

The central claim assumes that the extra risks, trouble spots, and trade-offs the chatbot listed are genuine findings rather than plausible-sounding but irrelevant or invented items, because the paper assigns no severity or ground-truth verification to those additions.

Editorial extensions

If this is right

An LLM second opinion can surface risks and sensitivity points a human team overlooked, so scenario choices can be revisited before architecture changes are made.
Because the LLM identified more tradeoffs, including pairs involving qualities the students were told to ignore, prioritization becomes a matter of deciding which tradeoffs actually matter rather than which ones the team happened to notice.
If the 26-of-30 agreement on final scenario selections holds, LLM output can serve as a screening pass that flags the small number of cases where human and machine disagree for closer review.
ATAM phases that are currently conducted in long brainstorming sessions can be partially automated, reducing the manual effort of scenario assessment and selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not measure precision: Copilot's extra risks and tradeoffs could be partly plausible filler. A natural extension, which the authors do not run, is to have experienced ATAM architects rate each added item as a genuine architectural concern or as a hallucination; the central claim would need to be re-stated if many items fail that test.
Severity is deferred to future work, but severity changes the practical conclusion: a selection agreement of 26/30 is less reassuring if Copilot favors scenarios with low-severity risks, because then its 'better and more accurate' risk lists do not align with its scenario choices.
The comparison uses novice students as the human baseline; applying the same four-prompt protocol to professional architects would reveal whether the LLM advantage shrinks, disappears, or reverses when the human side has more experience.
The authors plan to try alternative LLMs; until then the reported benefits are attributable to this particular tool and prompt sequence, not to LLMs in general.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The genuinely new thing here is an applied RAG-prompted LLM workflow for ATAM scenario evaluation, tested on seven student projects with a public reproducibility kit. That is a legitimate and useful step for the architecture-evaluation community, and the paper is transparent about many of its limits: it says severity analysis is deferred, says tradeoff accuracy needs deeper study, and openly notes that some of Copilot's tradeoffs involve qualities the students were told to ignore. The per-group tables are detailed enough to let a reader re-interpret the data independently. I give it credit for that.

The soft spot is the load-bearing claim. The abstract says the LLM produces "better and more accurate" results in most cases, but the evidence only supports "more." There is no ground-truth rubric for risks, sensitivity points, or tradeoffs; the co-authors tested the LLM's outputs without reporting criteria or inter-rater reliability. Two groups were dropped post hoc for low-quality scenarios, which weakens the comparison. And the tradeoff tables count items like cost and usability even though the text says students were told to ignore those qualities — that directly inflates Copilot's advantage. Without excluding out-of-scope items or measuring against an independent standard, "more" cannot be read as "better." The paper even concedes this at the end of Section 4, so the abstract is simply ahead of the data.

These problems are fixable rather than fatal. The authors could exclude the qualities students were told to ignore, add even a rough rubric with independent ratings, and soften the abstract to "more complete" or "potentially useful." The current version would not be a strong archival publication as-is, but it is an honest initial study with reproducible artifacts and a clear prompt design.

Who is this for? People working on LLM support for architecture evaluation or ATAM automation will want to see it, and the reproducibility kit makes it a usable baseline. It is not a breakthrough and the impact stays within software architecture. I would send it to peer review rather than desk-rejecting, because the topic is timely and the artifacts are real, but I would expect revision before acceptance. The main message to the authors should be: match the claim to the evidence.

Referee Report

4 major / 6 minor

Summary. The paper proposes using MS Copilot with a retrieval-augmented generation (RAG) setup to partially automate ATAM-based architecture evaluation, focusing on risk identification, sensitivity points, tradeoff analysis, and scenario selection. The authors compare the outputs of seven student teams (after excluding two teams) with Copilot's answers to four prompts, report tables of risks, sensitivity points, tradeoffs, and selected scenarios, and conclude that the LLM produces better and more accurate results in most cases, improving human decision-making. The manuscript includes an anonymized reproducibility kit with input artifacts and prompts.

Significance. If the central claim were established, the paper would provide a useful proof-of-concept that an LLM with project context can augment the brainstorming and scenario-selection phases of ATAM, potentially reducing manual evaluation effort. The paper deserves credit for a concrete prompt sequence, a reproducibility kit, and a transparent presentation of per-group raw results, which are appropriate for an initial exploratory study. Its current significance is limited, however, by the absence of an independent ground-truth rubric and by several count-based comparisons that conflate 'more items' with 'more accurate items.'

major comments (4)

[Abstract and Section 4 (Prompt #1/#2, Tables 1-3)] The central claim that the LLM produces 'better and more accurate results regarding the risks, sensitivity points and tradeoff analysis' is not established by the reported data. The only validation described is that 'the co-authors tested the validity and accuracy of the scenarios' using Copilot itself, with no independent gold standard, rubric, or inter-rater reliability measure. Since the paper also states that severity classification is deferred to future work, the additional risks and sensitivity points in Table 1 and the scenarios in Table 3 cannot be counted as true positives. The conclusion should be limited to 'more exhaustive' unless the authors add an expert-based ground-truth evaluation with precision/recall or equivalent measures.
[Section 4, Table 2 and the paragraph after Table 2] The tradeoff comparison is inflated by out-of-scope items. The text explicitly concedes that 'some of these tradeoffs belong to certain qualities (e.g., cost, usability) that the students were told to ignore,' yet Table 2 counts exactly such tradeoffs in Copilot's favor, e.g., G6 Availability-Cost, G7 Scalability-Cost and Security-Usability, and G8 Availability-Cost and Security-Usability. Because those qualities were excluded by the study design, these rows should not be scored as Copilot advantages; the analysis should either remove out-of-scope tradeoffs or report a sensitivity analysis with and without them.
[Section 3, group exclusion] The post hoc exclusion of two of nine student groups is a load-bearing methodological decision. The manuscript says the two groups were discarded because 'the quality of the scenarios was rather low and they provided several unclear scenarios,' but it gives no criteria, no independent assessment, and no transparency about which groups were excluded or how their results would have affected the comparison. This selection can bias every subsequent count in favor of the LLM. The authors should report all nine groups, or pre-register explicit exclusion criteria and show that the conclusions are robust to including the discarded groups.
[Section 4, Prompt #1/#2, Table 1 and the following discussion] The additional risks and sensitivity points are assumed to be relevant simply because Copilot produced them and the authors did not discard them. This assumption is stated in the text: 'If a risk identified for a chosen scenario is not critical, we assume that selecting such scenario would not harm the system.' The paper then lists 'scenarios selected with risks' without assessing whether those risks are critical, probable, or business-relevant. Without such an assessment, the counts in Table 1 support only a claim about the LLM's verbosity, not about the quality of the scenario selection. A severity-based re-scoring or an expert review of each additional item is needed before the 'more accurate' wording can be used.

minor comments (6)

[Keywords and Abstract] The keyword list contains a formatting error: 'Architecture evaluation,·Quality-attribute scenarios·Architecture tradeoffs·Large Language Models·Design assistance' should be separated by commas or semicolons rather than centered dots.
[Section 3] The two excluded groups are never identified, and the group numbering in Tables 1-3 skips G5, which makes the data hard to follow. Please clarify whether the numbering reflects original group labels or a renumbering after exclusion.
[Table 2, row G7] In the Copilot 'Wining scenarios after tradeoff' column for G7, the entry '1,2' should presumably be '1.2' to match the scenario identifier format used everywhere else.
[Section 3 and Appendix/Reproducibility text] The RAG strategy is mentioned but not described: the manuscript does not state how the PDF documents were chunked, embedded, or retrieved, or which prompts were used to inject the context. Since the reproducibility kit is a strength, adding these details would make the study actually repeatable.
[Bibliography] Reference [3] lists 'Clements, P., Kazman' without full author names; either the author list is incomplete (likely R. Kazman and/or R. Klein are missing) or the citation was truncated.
[Section 4, Prompt #4, Table 3] The statement that 'only in 4 cases out of 30 the scenarios selected by MS Copilot diverged from the selection of the students' is presented without any chance baseline or agreement measure (e.g., Cohen's kappa). Since scenario sets are small and many scenarios overlap by construction, a high match rate can occur by chance; a simple agreement statistic would help the reader interpret the result.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical comparison, and no claimed result reduces to its own input by construction.

full rationale

The paper does not present a derivation chain in which a quantity is defined in terms of the target result or in which a fitted parameter is later called a prediction. The central claim is empirical: MS Copilot's outputs for risks, sensitivity points, and tradeoffs are compared with students' outputs. The LLM's responses are not a deterministic consequence of the students' labeled results, as Tables 1-3 show both additional and divergent items. The co-authors' manual assessment of validity and accuracy is external to any formal derivation, even if under-specified. The self-citations [8] and [10] appear only as background or framing and do not carry the load-bearing empirical comparison. Concerns about leakage (feeding the students' full PDFs into the RAG prompt), the lack of a gold-standard rubric, and the inclusion of out-of-scope tradeoffs are validity threats for the empirical conclusion, not circular reasoning. No equation or argument in the paper reduces to its own inputs, so the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No numeric free parameters are fit; the argument is empirical. The load-bearing axioms are that student ATAM artifacts are a fair test bed and that the authors' reading of Copilot responses is a valid measure of correctness.

assumptions (3)

domain assumption Student ATAM assignments are an acceptable proxy for real architecture evaluation output.
Section 3 uses nine course projects as the input corpus and compares LLM output against them; the quality and correctness of student outputs is not independently validated.
ad hoc to paper MS Copilot's outputs are considered valid additional ATAM findings whenever they are not contradicted by the authors' inspection.
Section 4 treats Copilot's extra risks and sensitivity points as relevant additions, with severity assessment deferred to future work.
domain assumption The ATAM background and scenario semantics from [3] and [6] are correct and apply unchanged to LLM prompting.
Section 2 relies on ATAM definitions to map LLM answers to risks, sensitivity points, and tradeoffs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Supporting architecture evaluation for ATAM scenarios with LLMs." pith.science (2026). https://pith.science/paper/GZX6AA2Y

@misc{pith2026250600150,
  author       = {Pith},
  title        = {Pith review of: Supporting architecture evaluation for ATAM scenarios with LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GZX6AA2Y}},
  note         = {Machine review of arXiv:2506.00150}
}

read the original abstract

Architecture evaluation methods have long been used to evaluate software designs. Several evaluation methods have been proposed and used to analyze tradeoffs between different quality attributes. Having competing qualities leads to conflicts for selecting which quality-attribute scenarios are the most suitable ones that an architecture should tackle and for prioritizing the scenarios required by the stakeholders. In this context, architecture evaluation is carried out manually, often involving long brainstorming sessions to decide which are the most adequate quality scenarios. To reduce this effort and make the assessment and selection of scenarios more efficient, we suggest the usage of LLMs to partially automate evaluation activities. As a first step to validate this hypothesis, this work studies MS Copilot as an LLM tool to analyze quality scenarios suggested by students in a software architecture course and compares the students' results with the assessment provided by the LLM. Our initial study reveals that the LLM produces in most cases better and more accurate results regarding the risks, sensitivity points and tradeoff analysis of the quality scenarios. Overall, the use of generative AI has the potential to partially automate and support the architecture evaluation tasks, improving the human decision-making process.

Figures

Figures reproduced from arXiv: 2506.00150 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 12 canonical work pages

[1]

Adv Neural Inf Process Sys 33, 1877--1901 (2020)

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Adv Neural Inf Process Sys 33, 1877--1901 (2020)

work page 2020
[2]

SEI series in software engineering, Addison-Wesley (2024)

Cervantes, H., Kazman, R.: Designing Software Architectures: A Practical Approach 2nd Edition. SEI series in software engineering, Addison-Wesley (2024)

work page 2024
[3]

SEI series in software engineering, Addison-Wesley (2001)

Clements, P., Kazman: Evaluating Software Architectures: Methods and Case Studies. SEI series in software engineering, Addison-Wesley (2001)

work page 2001
[4]

Procedia CIRP 60, 128--132 (2017), complex Systems Engineering and Development Proceedings of the 27th CIRP Design Conference Cranfield University, UK 10th – 12th May 2017

Darwish, M., Shehab, E.: Framework for engineering design systems architectures evaluation and selection: Case study. Procedia CIRP 60, 128--132 (2017), complex Systems Engineering and Development Proceedings of the 27th CIRP Design Conference Cranfield University, UK 10th – 12th May 2017

work page 2017
[5]

In: 2009 16th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems

Ibrahim, H., Far, B.H., Eberlein, A.: Tradeoff and sensitivity analysis of a hybrid model for ranking commercial off-the-shelf products. In: 2009 16th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems. pp. 119--127 (2009)

work page 2009
[6]

Kazman, R., Klein, M., Clements, P.: Atam:sm method for architecture evaluation. Tech. Rep. CMU/SEI-2000-TR-004, Carnegie Mellon University - Software Engineering Institute (2000)

work page 2000
[7]

u ttler, H., Lewis, M., Yih, W.t., Rockt \

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.t., Rockt \"a schel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv Neural Inf Process Syst 33, 9459--9474 (2020)

work page 2020
[8]

Computing 102(2), 551--572 (2020)

Lytra, I., Carrillo, C., Capilla, R., Zdun, U.: Quality attributes use in architecture design decision methods: research and practice. Computing 102(2), 551--572 (2020)

work page 2020

Show all 15 references

[9]

Ozkaya, I.: Can architecture knowledge guide software development with generative ai? IEEE Software 40(5), 4--8 (2023)

2023
[10]

Razavian, M., Tang, A., Capilla, R., Lago, P.: In two minds: how reflections influence software design thinking. J. Softw.: Evol. Process 28(6), 394--426 (2016)

2016
[11]

CoRR abs/2303.07839 (2023)

White, J., Hays, S., Fu, Q., Spencer - Smith, J., Schmidt, D.C.: Chatgpt prompt patterns for improving code quality, refactoring, requirements elicitation, and software design. CoRR abs/2303.07839 (2023)

2023 arXiv
[12]

, " * write output.state after.block = add.period write

ENTRY address author booktitle chapter doi edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION in...
[13]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
[14]

, " * write output.state after.block = add.period write newline

ENTRY address author booktitle chapter edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.se...
[15]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Adv Neural Inf Process Sys 33, 1877--1901 (2020)

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Adv Neural Inf Process Sys 33, 1877--1901 (2020)

work page 2020

[2] [2]

SEI series in software engineering, Addison-Wesley (2024)

Cervantes, H., Kazman, R.: Designing Software Architectures: A Practical Approach 2nd Edition. SEI series in software engineering, Addison-Wesley (2024)

work page 2024

[3] [3]

SEI series in software engineering, Addison-Wesley (2001)

Clements, P., Kazman: Evaluating Software Architectures: Methods and Case Studies. SEI series in software engineering, Addison-Wesley (2001)

work page 2001

[4] [4]

Procedia CIRP 60, 128--132 (2017), complex Systems Engineering and Development Proceedings of the 27th CIRP Design Conference Cranfield University, UK 10th – 12th May 2017

Darwish, M., Shehab, E.: Framework for engineering design systems architectures evaluation and selection: Case study. Procedia CIRP 60, 128--132 (2017), complex Systems Engineering and Development Proceedings of the 27th CIRP Design Conference Cranfield University, UK 10th – 12th May 2017

work page 2017

[5] [5]

In: 2009 16th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems

Ibrahim, H., Far, B.H., Eberlein, A.: Tradeoff and sensitivity analysis of a hybrid model for ranking commercial off-the-shelf products. In: 2009 16th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems. pp. 119--127 (2009)

work page 2009

[6] [6]

Kazman, R., Klein, M., Clements, P.: Atam:sm method for architecture evaluation. Tech. Rep. CMU/SEI-2000-TR-004, Carnegie Mellon University - Software Engineering Institute (2000)

work page 2000

[7] [7]

u ttler, H., Lewis, M., Yih, W.t., Rockt \

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K \"u ttler, H., Lewis, M., Yih, W.t., Rockt \"a schel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv Neural Inf Process Syst 33, 9459--9474 (2020)

work page 2020

[8] [8]

Computing 102(2), 551--572 (2020)

Lytra, I., Carrillo, C., Capilla, R., Zdun, U.: Quality attributes use in architecture design decision methods: research and practice. Computing 102(2), 551--572 (2020)

work page 2020