{"id":"c938082b-64e8-417e-96dd-7bf49c1ec26d","arxiv_id":"2507.14554","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A five-year analysis of practitioner conference talks shows a few cloud-native technologies, led by Kubernetes, dominate industry discourse, while planning and coding phases receive far less attention than deployment and operations.","lead":"This study analyzed titles of 5,677 talks from eight practitioner software conferences (2020-2024) using large language models to extract technologies, purposes, and contexts. It finds that Kubernetes, Cloud Native, Serverless, and Containers dominate industry talk frequency and network centrality, while early DevOps phases like planning and coding receive little attention.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"KubeCon supplies roughly 43% of the corpus and its name literally includes 'Kubernetes' and 'Cloud Native'; the top-4 frequency/centrality finding is therefore not yet separable from the sampling frame. Recompute without KubeCon before trusting the dominance claim.","rationale":"I read the paper as a descriptive empirical study whose central claim is that a small set of cloud-native technologies dominates practitioner conference discourse. The authors provide a replication package and an LLM-extraction pipeline with human validation, which is real methodological support. However, the corpus construction makes the headline result fragile: KubeCon is both the largest single source (42.8%) and the venue whose name encodes the two top findings. A robustness analysis excluding KubeCon or equal-weighting conferences is the minimal check that would separate a property of practitioner practice from a property of the sampling frame. The reader's weakest assumption identifies the same issue, so I agree. I did not find a separate internal inconsistency that is more load-bearing than this: Table 9's Code percentage (3.11% vs 94 entries) is a likely typo that should be corrected, but it does not threaten the main dominance claim as directly as the KubeCon share does. The verdict should remain CONDITIONAL: the paper is a useful descriptive analysis but its central generalization needs the robustness check (and Table 9 fix) before acceptance.","tokens_in":34989,"tokens_out":7542,"duration_ms":91615,"concrete_test":"Re-run RQ1.1 and RQ1.2 (frequency quartiles, weighted degree, closeness, betweenness, community detection) on the 3,250 talks from non-KubeCon conferences, or with all eight conferences weighted equally. If Kubernetes and Cloud Native no longer rank in the top-4 on both frequency and all three centrality measures, the headline dominance is a sampling artifact and the generalizable claim should be revised or rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 3 shows KubeCon (EU, NA, China) contributes 2,427 of 5,677 included talks (42.8%). KubeCon+CloudNativeCon is a venue whose brand names the two most-reported technologies; talks there are structurally likely to mention Kubernetes or Cloud Native in their titles regardless of software-architecture trends. The top-4 dominance in Table 8 and the centrality rankings in Figure 5 are computed on title-derived co-occurrences, so the dense KubeCon sub-corpus inflates weighted degree, closeness, and betweenness for these four terms. Section 3.2.2 reports that the four core technologies connect to 89 of 112 nodes, meaning the network backbone is largely built from KubeCon-style co-occurrences. Without a robustness check that excludes KubeCon or weights conferences equally, the central claim that these technologies dominate practitioner software architecture discourse cannot be distinguished from the claim that these technologies dominate KubeCon. Section 6.4 acknowledges external-validity limits but does not quantify the effect. The term list also mixes tools with paradigms ('Cloud Native', 'Serverless', 'Cloud'), further inflating these categories. The dominance finding is thus a plausible property of this corpus but is not yet established as a property of practitioner practice at large.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports an empirical, large-scale review of software architecture trends as reflected in practitioner conference talks. The authors collected 5,677 talk titles from eight industry conferences over 2020–2024, used an LLM pipeline (with three validator LLMs and human validation) to extract technologies, purposes, and contexts, and then applied frequency analysis, co-occurrence network analysis, centrality metrics, and Louvain community detection. The central claims are that Kubernetes, Cloud Native, Serverless, and Containers dominate practitioner discourse by both frequency and network centrality; that practitioner attention is concentrated in the Build, Deploy, Operate, and Monitor DevOps phases; and that five technology communities can be identified, covering deployment automation, service communication, cloud AI, observability, and cloud-edge collaboration. The paper also reports a relative lack of practitioner focus on early lifecycle phases such as Plan, Code, and Release.","tokens_in":35309,"tokens_out":3997,"duration_ms":52830,"significance":"If the dominance claim is robust, the study would be a useful complement to existing practitioner-oriented trend reports such as the DORA reports and the Thoughtworks Technology Radar, and it would provide an evidence base for aligning academic research agendas with industry practice. The main strengths are the size of the corpus (5,677 talks), the explicit validation protocol involving multiple LLMs and human experts, and the public replication package. However, the central inference depends on the representativeness of the conference sample and on title-level extraction; these are not merely presentation concerns, because they directly determine whether the top-four finding is a property of software architecture practice or of the KubeCon-dominated corpus. For this reason the paper's current form does not yet establish the headline claim as stated, although the claim is plausibly correct and testable with additional robustness analyses.","major_comments":[{"comment":"The sampling-frame concern raised by the reader's report is real and land on the central claim. KubeCon editions contribute 2,427 of the 5,677 included talks (42.8% in Table 3), and the venue name itself contains the two most frequent extracted technologies, 'Kubernetes' and 'Cloud Native'. Because the co-occurrence network and all three centrality measures in Section 3.2 are computed from talk titles, the dense KubeCon sub-corpus can dominate both frequency counts and structural metrics. As reported in Section 3.2.2, the four core technologies connect to 89 of the other 108 nodes, which is exactly the pattern one would expect from a corpus in which one conference contributes nearly half of the talks and is named after two of the four 'core' terms. Section 6.4 acknowledges external-validity limits in general terms but does not quantify the effect. The authors should re-run the frequency and centrality analysis excluding all KubeCon editions, and also with conference-level equal weighting or stratified re-sampling, and report whether Kubernetes, Cloud Native, Serverless, and Containers still occupy the top ranks. Without such a robustness check, the dominance finding is not separable from the dominance of KubeCon in the sample.","section":"Section 2.2, Table 3, Section 3.2.1"},{"comment":"The study extracts technologies, purposes, and contexts exclusively from talk titles. This is a substantial construct-validity risk, and the manuscript's own discussion in Section 6.1 acknowledges that titles 'may not fully capture the depth or nuance of the talk content.' The risk is not symmetric across the DevOps phases: talks about deployment and monitoring tools are likely to name the tool in the title, while talks about planning, architecture design, or coding practices may use abstract terms that do not map to the extracted technology list. Inclusion criterion E3 also requires titles to reference software architecture-related terms, which can exclude relevant practitioner talks and further skew the phase distribution. The authors should present a sensitivity analysis using a sample of talk abstracts or full descriptions, or at least quantify how many of the 5,677 titles mention a technology explicitly versus how many required LLM inference. At present the 'DevOps phase gap' finding may be an artifact of title-level extraction rather than a property of practitioner discourse.","section":"Section 2.3, Section 6.1"},{"comment":"Several analysis choices are reasonable but are not tested for robustness, and they directly shape the headline results. First, the extracted term list mixes concrete tools (Kubernetes, Prometheus) with broad paradigms and categories (Cloud Native, Cloud, Serverless, AI), which inflates the apparent dominance of cloud-native concepts and makes the DevOps classification partly tautological: a term labeled 'Cloud Native' is very likely to be classified into cloud-centric DevOps phases. Second, the frequency threshold of at least 2 appearances and the focus on the fourth quartile are arbitrary; the paper does not report how the top-ten list changes if thresholds are varied. Third, the Louvain community detection in Section 3.2.3 is performed on a graph from which the four core technologies and 13 exclusively linked nodes are removed before clustering. The removal is justified by reference to Wen et al. (2011), but the number 13 and the definition of 'exclusively linked' are not described precisely, and no alternative community-detection runs (e.g., without removal, or with different resolution parameter gamma) are reported. The five-community structure is presented as a main result; it should be accompanied by sensitivity analyses showing that it is not an artifact of these choices.","section":"Section 2.3.2, Section 2.4, Section 3.2.3"}],"minor_comments":[{"comment":"The Code phase row reports # = 94 but a percentage of 3.11%, which is not consistent with 94 out of 450 technologies (20.9%). The Plan, Code, and Release percentages appear to be percentages of technology-phase assignments rather than percentages of technologies; the table should state the denominator clearly, and the text claiming that Plan, Code, and Release are 'each less than 10%' should be reconciled with the reported counts.","section":"Table 9"},{"comment":"The sentence 'we grouped the words into quartiles based on their frequency and present only the fourth quartile, i.e., the top 25 most frequently occurring words' is internally inconsistent: the fourth quartile contains 120 technologies according to the preceding paragraph. Please correct the wording to match the actual number.","section":"Section 3.1"},{"comment":"The text says 'such three technologies connect to over 80% of all other technologies' immediately after listing four core technologies (Kubernetes, Cloud Native, Serverless, and Containers). This inconsistency appears multiple times and should be fixed throughout the discussion.","section":"Section 4.1"},{"comment":"The sentence about prompt engineering says the authors employed LLMs for 'classifying refactoring motivations,' but the study is about technology adoption purposes and contexts, not refactoring motivations. This appears to be a copy-paste error from another paper and should be corrected.","section":"Section 2.3.1"},{"comment":"The claim that 'only research can provide a more holistic lens on architectural design, quality, and evolution' is a normative conclusion that goes beyond the descriptive data. The manuscript should either soften this statement or support it with evidence from the related-work comparison rather than presenting it as an empirical result.","section":"Section 4.3 and Section 7"},{"comment":"There are numerous typographical errors that should be corrected in revision, including 'adotpion', 'Futhermore', 'practioner', 'Holististic', 'Garter' for Gartner in Table 3, and inconsistent capitalization of 'Container' versus 'Containers' when referring to the same concept.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid descriptive empirical study with a clear methodology and a public replication package, but the headline finding is currently confounded by the KubeCon-dominated sample. The requested robustness analyses are within the scope of a revision and should be feasible with the existing data. I would also suggest the editors ask the authors to reduce reliance on their own prior work (e.g., Esposito et al. 2025a) for the interpretive claim that research provides the only holistic lens, since that framing goes beyond the data and is cited from a preprint by the same group."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is a new dataset and a transparent method: 5,677 practitioner talk titles from eight conferences (2020-2024), extracted with an LLM pipeline and human validation, with a replication package. The paper's real contribution is showing you can mine practitioner conference titles for trend evidence. The headline finding — Kubernetes, Cloud Native, Serverless, Containers dominate — is plausible and lines up with industry reports, but the paper doesn't yet separate it from the sampling frame. KubeCon alone supplies 43% of the corpus, and the venue name literally contains two of the four dominant terms.\n\nWhat's genuinely good: the LLM validation is carefully designed (one extractor, three validators, three-step human check on a statistically significant sample), and the reported accuracy figures are credible. The DevOps phase observation — heavy attention to Build/Deploy/Operate/Monitor, almost none to Plan/Code/Release — is interesting and worth following up. The five communities are descriptive but coherent.\n\nThe soft spots are real and, mostly, fixable. The biggest is the KubeCon confound. The four core technologies connect to 89 of 108 other nodes, so the network backbone is largely KubeCon-shaped. Section 6.4 mentions limited generalizability in prose but never quantifies it. A robustness check excluding KubeCon, or weighting conferences equally, is essential before anyone should trust the dominance claim as a property of practitioner practice rather than of this corpus. Table 9 has an arithmetic slip: Code is listed as 94 technologies (20.9%) but labeled 3.11%, and the text says Code is under 10%. Also, the paper says general-purpose terms like Cloud and AI were excluded from the co-occurrence network, yet Cloud and Generative AI appear as central nodes in Community 3. That's contradictory; needs reconciliation.\n\nWho this is for: SE researchers who want a descriptive map of practitioner discourse, and anyone interested in LLM-based classification in empirical SE. The paper deserves a serious referee — the data are shared, the method is reproducible, and the limitations are partially acknowledged. But a referee should push for the robustness analysis and the table fixes. I'd cite the corpus and method, not the dominance claim. Verdict: conditional — substantial revision.","headline":"Useful dataset and method, but the Kubernetes-dominance claim is not yet separable from the 43% KubeCon share of the corpus.","tokens_in":35785,"tokens_out":3813,"would_cite":true,"duration_ms":40085,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Four cloud-native tools dominate practitioner architecture talks.","keywords":["software architecture trends","practitioner conferences","technology co-occurrence network","centrality analysis","DevOps phases","cloud-native","large language models","community detection"],"falsifier":"Re-run the same extraction and network pipeline on a different corpus of practitioner talks that deliberately excludes KubeCon and other cloud-dominated conferences; if the centrality of Kubernetes, Cloud Native, Serverless, and Containers collapses, the original finding is a property of the conference selection. Alternatively, analyze talk abstracts or transcripts rather than titles for the same corpus and check whether early DevOps phases (Plan, Code, Release) show more than 10 percent of technologies when full content is considered.","tokens_in":34825,"feed_emoji":"☁️","tokens_out":5476,"duration_ms":53679,"temperature":0.7,"pith_summary":"This paper tries to establish that a handful of cloud-native infrastructure technologies—Kubernetes, Cloud Native, Serverless, and Containers—have come to dominate how practitioners talk about software architecture, and that this talk is concentrated in the later DevOps phases. It analyzes 5,677 talk titles from eight practitioner conferences over 2020–2024, using large language models with human validation to extract technologies, purposes, and contexts. If the finding holds, academic research agendas and practitioner tooling investments should treat these few technologies as the gravitational center of current practice, while early-lifecycle phases like planning and coding are comparatively neglected. The result would matter because it gives a data-driven picture of where industry attention actually sits.","feed_headline":"Four cloud-native tools dominate practitioner architecture talks","feed_subtitle":"A 5,677-talk review shows later DevOps stages get the attention; planning and coding are left behind.","key_machinery":"The load-bearing mechanism is a technology co-occurrence network built from talk titles: nodes are the 450 technologies, and edges count how often two technologies appear in the same title. On this graph the paper computes three classic centrality metrics—weighted degree (connection strength), closeness centrality (proximity to all other nodes), and betweenness centrality (role as a bridge)—and then applies the Louvain method for community detection to reveal cohesive clusters. This network machinery is what turns raw mention counts into a structural claim about which technologies are genuinely central, and it is what produces the five identified communities. The other central piece is the classification of each technology against the eight DevOps phases, which grounds the claim that practitioner attention skews to later pipeline stages.","core_discovery":"The paper's central claim is that practitioner discourse on software architecture over the past five years is organized around a small set of core technologies. Among 450 technologies extracted from conference talk titles, Kubernetes, Cloud Native, Serverless, and Containers lead in both raw frequency and structural centrality: they connect to over 80 percent of the other technologies in the co-occurrence network and rank highest on weighted degree, closeness, and betweenness. These core technologies are mostly discussed in connection with the Build, Deploy, Operate, and Monitor phases of the DevOps pipeline, while Plan, Code, and Release receive little technological attention. Practitioners frame most talks as introductions and overviews, with deployment, communication, AI, and observability as the dominant purposes and contexts. The paper also identifies five technology communities—deployment automation, service communication, cloud AI, observability, and cross-cloud edge collaboration—that reflect the main subfields of current practice.","pith_inferences":["If the dominance of KubeCon in the corpus (about 43 percent of talks) reflects conference scheduling rather than practice, the same pipeline applied to a non-Kubernetes-centric conference set might shrink the centrality gap; this is a testable consequence of the sampling choice.","The finding that most talks are 'Introduction & Overview' suggests that conference discourse measures technology awareness diffusion more than deep adoption; the paper's own data cannot distinguish hype from production use.","The centrality analysis, applied to talk titles only, treats a technology mentioned in a title as the talk's subject; analyzing abstracts or transcripts could reveal whether early-phase technologies like planning tools are discussed under non-architectural vocabulary.","The five communities map naturally onto a maturity narrative: automation and observability are mature topics, while cloud-AI and cloud-edge are still forming, so the same method applied to 2025–2029 talks should show community boundary changes."],"forward_implications":["A small set of cloud-native infrastructure tools functions as the backbone of practitioner architecture discourse; new technologies gain visibility mainly by connecting to this core rather than by forming independent hubs.","Tools supporting Plan, Code, and Release are scarce in practitioner conferences, suggesting either a tooling gap or a blind spot in how early-lifecycle work is discussed.","Most technologies are classified as usable in both cloud and on-premise environments, indicating that hybrid deployment is the assumed default in current practice.","The rise of Generative AI and WebAssembly in later years, visible mainly in betweenness centrality, implies these are emerging bridge technologies rather than yet-established architectural centers.","Because the core technologies span many DevOps phases and purposes, practitioner talks treat them as generic infrastructure rather than niche tools."],"supporting_citations":[{"why":"Supplies the eight-phase DevOps framework (plan, code, build, test, release, deploy, operate, monitor) used to classify every extracted technology.","marker":"(Bass et al., 2015)"},{"why":"The Louvain method for community detection that produces the five technology communities.","marker":"(Blondel et al., 2008)"},{"why":"Gephi, the network analysis tool used to construct the co-occurrence graph and compute centrality and visualization.","marker":"(Bastian et al., 2009)"},{"why":"Provides the guidance for the three classic centrality metrics (closeness, betweenness, and their interpretation) applied to the network.","marker":"(Brandes et al., 2016)"},{"why":"Supports the targeted removal of core nodes to refine the graph before community detection, avoiding modularity skew.","marker":"(Wen et al., 2011)"},{"why":"Chain-of-thought prompting, the technique used to instruct the large reasoning model during extraction and classification.","marker":"(Wei et al., 2022)"},{"why":"Systematic-review methodology used to validate the applicability of the inclusion and exclusion criteria on a pilot sample.","marker":"(Kitchenham and Brereton, 2013)"}],"fun_headline_variants":["Kubernetes and serverless dominate five years of architecture talks","Cloud-native tools lead practitioner architecture talks","Four technologies tower over 5,677 architecture talks","Architecture talks focus on deploy and operate, not plan and code","Five-year review: a few core tools rule architecture practice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the premise that talk titles from these eight conferences give a representative picture of software architecture practice; if the conference lineup is skewed toward Kubernetes-centric and cloud-devops topics, the dominance findings would be an artifact of the sampling frame rather than of practice itself.","fun_headline_variants_meta":{"raw":{"variants":["Kubernetes and serverless dominate five years of architecture talks","Cloud-native tools lead practitioner architecture talks","Four technologies tower over 5,677 architecture talks","Architecture talks focus on deploy and operate, not plan and code","Five-year review: a few core tools rule architecture practice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000852,"raw_usage":{"total_tokens":3719,"prompt_tokens":979,"completion_tokens":2740,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":2661}},"tokens_in":595,"tokens_out":2740,"duration_ms":24648,"temperature":1.0,"reasoning_tokens":2661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:53:15.349712+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same extraction and network pipeline on a different corpus of practitioner talks that deliberately excludes KubeCon and other cloud-dominated conferences; if the centrality of Kubernetes, Cloud Native, Serverless, and Containers collapses, the original finding is a property of the conference selection. Alternatively, analyze talk abstracts or transcripts rather than titles for the same corpus and check whether early DevOps phases (Plan, Code, Release) show more than 10 percent of technologies when full content is considered.","supporting_citations":[{"cited_title":", author Weber, I","cited_arxiv_id":null,"evidence_quote":"Supplies the eight-phase DevOps framework (plan, code, build, test, release, deploy, operate, monitor) used to classify every extracted technology."},{"cited_title":", author Guillaume, J.L","cited_arxiv_id":null,"evidence_quote":"The Louvain method for community detection that produces the five technology communities."},{"cited_title":", author Heymann, S","cited_arxiv_id":null,"evidence_quote":"Gephi, the network analysis tool used to construct the co-occurrence graph and compute centrality and visualization."},{"cited_title":", author Borgatti, S.P","cited_arxiv_id":null,"evidence_quote":"Provides the guidance for the three classic centrality metrics (closeness, betweenness, and their interpretation) applied to the network."},{"cited_title":", author Leicht, E","cited_arxiv_id":null,"evidence_quote":"Supports the targeted removal of core nodes to refine the graph before community detection, avoiding modularity skew."},{"cited_title":", author Wang, X","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting, the technique used to instruct the large reasoning model during extraction and classification."},{"cited_title":", author Brereton, P","cited_arxiv_id":null,"evidence_quote":"Systematic-review methodology used to validate the applicability of the inclusion and exclusion criteria on a pilot sample."}],"review_version":1}