Pith. sign in

REVIEW 4 major objections 5 minor 21 references

A Grey Literature Review of AI-Native Applications

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI-native applications form a distinct software class with AI as the core engine and an inherently probabilistic, non-deterministic nature.

desk verdict Useful structured synthesis of the AI-native discourse, but the 'two core pillars' claim is partly an artifact of the search criteria and the paper needs consistency fixes before it is citable. read the letter →

arxiv 2509.13144 v2 pith:6H6JBL3E submitted 2025-09-16 cs.SE

Lingli Cao , Shanshan Li , Ying Fan , Danyang Li , Chenxing Zhong This is my paper · ORCID
classification cs.SE
keywords AI-nativeapplicationsgreyliteraturereviewsoftwareengineeringparadigmqualityattributestechnologystackLLMorchestrationobservabilityprobabilistic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This grey-literature review tries to establish that “AI-native” is more than a buzzword: it names a distinct class of software in which AI is not an add-on but the system’s central intelligence, making applications inherently probabilistic. The authors synthesize 106 blogs, reports, and open-source projects to extract seven core definitional elements, eight key characteristics, seven prominent quality attributes, and eight recurring technology stacks. They propose a dual-layered engineering blueprint to guide practitioners in designing, evaluating, and operating such systems. A sympathetic reader would care because it offers a shared vocabulary and a first architectural map for a fast-moving field that currently lacks one.

What carries the argument

The central machinery is the grey-literature review protocol itself: a structured search of search engines and an open-source code-hosting platform, inclusion/exclusion criteria requiring sources to focus primarily on AI-native applications, quality scoring, and thematic analysis that lets definitions and themes emerge inductively rather than imposing a prior definition. The analytic output—frequency-coded themes—carries the argument: the two pillars and all subsequent claims are built from the recurrence of these themes across 106 heterogeneous sources.

What would settle it

Conduct a matched survey of LLM-powered applications that never use the label “AI-native” (e.g., internal enterprise tools, academic systems) and compare their architecture, quality concerns, and stack against the two pillars. If the unlabeled set shows the same probabilistic-AI-center pattern, the pillars generalize; if the labeled set differs mainly in self-description, the review’s definition is an artifact of corpus selection.

Watch

Extended reading notes

Core claim

The paper’s central claim is that AI-native applications are defined by two load-bearing pillars: AI as the system’s intelligence paradigm (removing it voids the application’s purpose) and their probabilistic, non-deterministic behavior. Synthesizing 106 grey literature sources, the authors extract seven core definitional elements (AI as first principle, foundation models, agentic orchestration, context/memory, multimodal interaction, generative UI, on-device execution), eight key characteristics, seven quality attributes led by reliability, usability, observability, and cost-aware performance, and a typical stack of LLM orchestration, vector databases, and AI-specific observability platform

Load-bearing premise

The load-bearing premise is that the 106 sources that explicitly use the term “AI-native” and rank high in search results or popular open-source repositories are representative of the entire category; if the term is used mainly by a self-selected, marketing-forward group, the “defining characteristics” may reflect vocabulary rather than engineering reality.

Editorial extensions

If this is right

  • Practitioners get a concrete architecture map: an orchestration-centric stack (models plus orchestration, memory, and retrieval) with hybrid cloud/on-device deployment, and cost-efficiency as a first-order design constraint.
  • Quality assurance must add AI-specific observability—monitoring model behavior, data drift, output quality, and token usage—alongside traditional reliability and usability testing.
  • Post-deployment monitoring and maintenance are the field’s biggest blind spot, with only a handful of the 106 sources covering that lifecycle stage; the blueprint flags this as a priority for tooling and research.
  • The distinction “AI-native vs AI-assisted” becomes testable: an app is AI-native only if removing the AI core leaves it non-functional, as with answer engines built from the ground up around the model versus traditional search with AI features appended.
  • The emerging stack—foundation models, orchestration frameworks, vector databases, and UI tooling—provides a baseline for technology selection, while observability tooling remains nascent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The review’s findings likely say as much about the word “AI-native” as about the systems it labels; a parallel survey of LLM-based applications that never use the term might reveal the same architectural patterns, which would strengthen the blueprint’s generality.
  • The dual-layered blueprint could be evaluated empirically by using it to build a new AI-native application and checking whether its design guidelines reduce integration and reliability failures.
  • The pronounced gap between the recognized need for observability and the scarcity of observability tooling suggests a near-term instrumentation niche—standardized logging and evaluation for agentic, retrieval-augmented workflows.
  • If the two-pillar definition becomes canonical, it will sharpen the AI-assisted vs AI-native boundary and could reshape how enterprise software is categorized, funded, and regulated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper reports a grey literature review of 'AI-native applications' based on Google, Bing, and GitHub sources, following the Garousi et al. GLR guidelines. The authors retrieve 1,576 candidate entries, screen to 106 (or 112; inconsistent) sources, and apply thematic analysis to answer four RQs: key characteristics, quality attributes, technology stacks, and opportunities/challenges. They report 7 core definitional elements, 8 key characteristics, 7 quality attributes, 8 technology stack categories, 9 opportunities, and 7 challenges, and they claim a 'first dual-layered engineering blueprint' for AI-native applications. The central thesis is that AI-native applications are distinguished by two core pillars: AI as the central intelligence paradigm and their inherently probabilistic, non-deterministic nature.

Significance. If the synthesis is accurate, the paper provides a useful, practitioner-oriented map of an emerging engineering paradigm and a basis for future formalization. Strengths include the explicit GLR protocol, the use of quality assessment criteria, independent coding and cross-checking, clear search strings, and a public replication package on GitHub. However, the headline claims ('two core pillars', 'dual-layered blueprint', 'distinguished by') are not fully supported by the reported results and are partly built into the selection criteria. The contribution is therefore a well-structured descriptive review whose interpretive claims need to be either more carefully derived or more cautiously worded.

major comments (4)
  1. [§3.3 vs. §4] The final corpus size is reported inconsistently: §3.3 ends with 'the GLR process yielded 112 grey literature sources,' while §4 and the abstract say 106. The annual counts in §4 (2+27+48+29=106) support 106. The discrepancy must be resolved because all frequency counts (e.g., 73 of 106 vs 73 of 112) and the flow diagram depend on it.
  2. [Abstract and §4] The abstract's statement that AI-native applications 'are distinguished by two core pillars' is not backed by a corresponding results subsection. §4.1 presents seven core elements and eight characteristics; §4.2 presents seven quality attributes. No analysis names or derives two pillars, and 'probabilistic, non-deterministic nature' appears as a framing remark in §4.2 rather than as an empirically grounded theme. Either add an explicit two-pillar synthesis with evidence mapping or soften the abstract.
  3. [Abstract, §1, §4.3, §5] The contribution 'first dual-layered engineering blueprint' is never presented. §4.3 discusses technology stack layers and §5 gives implications, but there is no blueprint with layers, components, and design guidelines. If the blueprint is intended to be the two 'application mind' layers in §4.3, that connection is not made. The claim should be operationalized with a figure/table or removed.
  4. [§3.3 and Table 1 (I1)] The selection design makes the 'distinguishing' claim partly circular. The search string is restricted to sources explicitly labeled AI-native, and criterion I1 requires that the study 'primarily focuses on AI-native applications.' Thus the finding that AI centrality is a core element mostly restates the inclusion rule. The paper's own §6 acknowledges selection bias generally but does not address this specific threat. I recommend tempering the 'distinguished by' and 'setting them apart' claims, or adding a comparison corpus (e.g., sources on 'LLM-powered' or 'AI-assisted' applications) to test whether the themes are unique.
minor comments (5)
  1. [§4.4 Summary] The RQ4 summary is corrupted: after 'And seven challenges:' the text repeats the nine opportunities verbatim. Replace it with the seven challenges enumerated earlier in §4.4.
  2. [Table 1 (E2)] E2 as written ('The study may take the form of a scientific paper, book, video, job posting, training announcement, etc.') is not an exclusion criterion; it appears to be an inclusion note. Please rephrase or correct.
  3. [§3.7 and §6] The text attributes 'Braun and Clarke's six-phase framework' to reference [17], but [17] is Cruzes and Dyba (2011), 'Recommended steps for thematic synthesis in software engineering.' Add the correct Braun and Clarke citation or rephrase.
  4. [Throughout] There are numerous typos and duplicated phrases, including 'appclications', 'tranformative', 'intrisic', 'depolyment', 'addressees', 'accross', 'empricial', 'appplicaions', 'Ai-native' in the abstract, and 'This study identifies This study identified' in §4.4. A thorough proofreading pass is needed.
  5. [Figure 1] The text says 'Figure 1: Study identification process' but no flow diagram is visible in the manuscript; ensure the figure is included and that its numbers match §3.3 and §4.

Circularity Check

1 steps flagged · score 1.0 of 10

Minor definitional caveat: the 'AI as core' pillar is partly built into the term-based corpus selection; no fitted-parameter or derivation-level circularity.

  1. self definitional [Section 3.3 (search strategy) and Table 1 criterion I1; abstract Results paragraph]
    "we restricted our search to literature explicitly labeled as involving AI-native applications... I1 The study primarily focuses on AI-native applications."

    The search string and inclusion criterion define the corpus by the label 'AI-native'. The abstract then presents 'the central role of AI as the system's intelligence paradigm' as an empirical finding that distinguishes AI-native applications. Since 'AI-native' semantically means that AI is native/inherent, the high salience of AI centrality is partly a consequence of the sampling frame, not an independent discovery. The paper also lacks a comparison corpus of non-AI-native (e.g., AI-assisted) systems, so the 'distinguished by' wording goes beyond what the selected sources can show. This is a minor, partial definitional circularity; the frequency distributions, quality attributes, and technology-stack syntheses are still empirical summaries.

full rationale

This is a grey literature review, not a derivation with fitted parameters, so the core circularity patterns (fitted input called prediction, self-citation chains, uniqueness theorems) do not apply. The only substantive concern is that the corpus was selected using the term 'AI-native' and criterion I1 required sources to primarily focus on AI-native applications. That makes the finding that AI is a core pillar partly semantic/definitional rather than fully empirical. The paper's own Section 6 (External Validity) acknowledges the selection-bias limitation, but the abstract and contributions do not carry that caveat. However, the remaining results—7 core elements, 8 characteristics, 7 quality attributes, 8 technology stacks, and lifecycle coverage—are reported as frequency-based thematic syntheses of the included sources; they are not predicted from or equivalent to the selection criteria. Self-references in the paper are not load-bearing to these claims. Therefore the overall circularity is minimal.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The review relies on the representativeness of a self-selected corpus, on frequency counts as indicators of importance, on the reliability of qualitative coding, and on a consistent meaning of 'AI-native' across sources. No new entities are introduced beyond the proposed (but under-specified) 'dual-layered blueprint'.

assumptions (4)
  • domain assumption The 106 selected grey literature sources are representative of the AI-native application discourse.
    Search and selection restrict to English, search-engine-indexed, or popular GitHub sources; stated in Sections 3.3 to 3.5.
  • domain assumption Frequency of mention in grey literature is a valid proxy for importance.
    Used throughout Section 4 to rank characteristics, quality attributes, and challenges; e.g., Section 4.2 treats 'most frequently cited' as 'forefront of developer concerns'.
  • domain assumption Thematic coding by the authors is reliable and reproducible.
    Section 3.7 claims all data derived without subjective interpretation, but thematic analysis is interpretive; threats are discussed in Section 6.
  • domain assumption The term 'AI-native' has a stable referent across sources.
    Sources define AI-native differently (Sapphire vs Ericsson); the paper synthesizes bottom-up, but construct validity is discussed in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Grey Literature Review of AI-Native Applications." pith.science (2026). https://pith.science/paper/6H6JBL3E

@misc{pith2026250913144,
  author       = {Pith},
  title        = {Pith review of: A Grey Literature Review of AI-Native Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6H6JBL3E}},
  note         = {Machine review of arXiv:2509.13144}
}
read the original abstract

Background: The rapid advancement of large language models (LLMs) has given rise to AI-native applications, a new paradigm in software engineering that fundamentally redefines how software is designed, developed, and evolved. Despite their growing prominence, AI-native applications still lack a unified engineering definition and architectural blueprint, leaving practitioners without systematic guidance for system design, quality assurance, and technology selection. Objective: This study seeks to establish a comprehensive understanding of AI-native applications by identifying their defining characteristics, key quality attributes, and typical technology stacks, as well as by clarifying the opportunities and challenges they present. Method: We conducted a grey literature review, integrating conceptual perspectives retrieved from targeted Google and Bing searches with practical insights derived from leading open-source projects on GitHub. A structured protocol encompassing source selection, quality assessment, and thematic analysis was applied to synthesize findings across heterogeneous sources. Results: We finally identified 106 studies based on the selection criteria. The analysis reveals that AI-native applications are distinguished by two core pillars: the central role of AI as the system's intelligence paradigm and their inherently probabilistic, non-deterministic nature. Critical quality attributes include reliability, usability, performance efficiency, and AI-specific observability. In addition, a typical technology stack has begun to emerge, comprising LLM orchestration frameworks, vector databases, and AI-native observability platforms. These systems emphasize response quality, cost-effectiveness, and outcome predictability, setting them apart from conventional software systems. Conclusion: This study is the first to propose a dual-layered engineering blueprint...

Figures

Figures reproduced from arXiv: 2509.13144 by the authors.

Figure 1
Figure 1. Study identification process The overall retrieval process is shown in [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Distribution of reviewed studies per year [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. The core elements related to the definition of AI-native applications [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The key characteristics of AI-Native applications [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Concerned quality attributes for AI-native applications [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Quality attribute details for AI-native applications [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: The common technology stacks used in the development of AI-native applications [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Lifecycle coverage of the AI-native application technology stacks [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Opportunities presented by AI-Native applications [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Challenges presented by AI-Native applications [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 5 linked inside Pith

  1. [17]

    D. S. Cruzes, T. Dyba, Recommended steps for thematic synthesis in soft- ware engineering, in: 2011 international symposium on empirical software engineering and measurement, IEEE, 2011, pp. 275–284. 27

    2011

  2. [1]

    Feuerriegel, J

    S. Feuerriegel, J. Hartmann, C. Janiesch, P. Zschech, Generative ai, Busi- ness & Information Systems Engineering 66 (1) (2024) 111–126

    2024

  3. [2]

    Fui-Hoon Nah, R

    F. Fui-Hoon Nah, R. Zheng, J. Cai, K. Siau, L. Chen, Generative ai and chatgpt: Applications, challenges, and ai-human collaboration (2023)

    2023

  4. [3]

    A. E. Hassan, G. A. Oliva, D. Lin, B. Chen, Z. Ming, et al., Towards ai- native software engineering (se 3.0): A vision and a challenge roadmap, arXiv preprint arXiv:2410.06107 (2024)

    arXiv 2024

  5. [4]

    C. Gao, K. Burke, L. Segall, J. DellaPasqua, A. Reddy, M. Liao, Ai-native applications: A framework for evaluating the future of enterprise software 26 (2025). URLhttps://sapphireventures.com/blog/ai-native-applications/ ?utm_source=chatgpt.com

    2025

  6. [5]

    Hymel, The ai-native software development lifecycle: A theoretical and practical new methodology, arXiv preprint arXiv:2408.03416 (2024)

    C. Hymel, The ai-native software development lifecycle: A theoretical and practical new methodology, arXiv preprint arXiv:2408.03416 (2024)

    arXiv 2024

  7. [6]

    Ericsson, Ai-native: Building the next generation of intelligent software, https://www.ericsson.com/49341a/assets/local/reports-papers/ white-papers/ai-native.pdf, accessed: 2024-06-04 (2024)

    2024

  8. [7]

    Zheng, S

    R. Zheng, S. Dou, S. Gao, Y. Hua, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, Y. Zhou, et al., Secrets of rlhf in large language models part i: Ppo, arXiv preprint arXiv:2307.04964 (2023)

    arXiv 2023

Show all 21 references
  1. [8]

    X. Lin, W. Wang, Y. Li, S. Yang, F. Feng, Y. Wei, T.-S. Chua, Data- efficient fine-tuning for llm-based recommendation, in: Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, 2024, pp. 365–374

    2024

  2. [9]

    Y. Yu, G. Rong, H. Shen, H. Zhang, D. Shao, M. Wang, Z. Wei, Y. Xu, J. Wang, Fine-tuning large language models to improve accuracy and com- prehensibility of automated code review, ACM transactions on software engineering and methodology 34 (1) (2024) 1–26

    2024

  3. [10]

    L. Cao, H. Zhang, S. Li, D. Li, Y. Yang, C. Zhong, X. Zhou, Y. Xie, Enhancing the accuracy and comprehensibility in architectural tactics de- tection via small model-augmented prompt engineering, arXiv preprint arXiv:2503.03609 (2025)

    2025 arXiv

  4. [11]

    Gozalo-Brizuela, E

    R. Gozalo-Brizuela, E. C. Garrido-Merchán, A survey of generative ai ap- plications, arXiv preprint arXiv:2306.02781 (2023)

    2023 arXiv

  5. [12]

    J. D. Weisz, M. Muller, J. He, S. Houde, Toward general design principles for generative ai applications, arXiv preprint arXiv:2301.05578 (2023)

    2023 arXiv

  6. [13]

    J. D. Weisz, J. He, M. Muller, G. Hoefer, R. Miles, W. Geyer, Design principles for generative ai applications, in: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 2024, pp. 1–22

    2024

  7. [14]

    Fowler, Patterns of generative ai (2025)

    M. Fowler, Patterns of generative ai (2025). URLhttps://martinfowler.com/articles/gen-ai-patterns/

    2025

  8. [15]

    Garousi, M

    V. Garousi, M. Felderer, M. V. Mäntylä, Guidelines for including grey liter- ature and conducting multivocal literature reviews in software engineering, Information and software technology 106 (2019) 101–121

    2019

  9. [16]

    Garousi, M

    V. Garousi, M. Felderer, M. V. Mäntylä, A. Rainer, Benefitting from the grey literature in software engineering research, in: Contemporary Empir- ical Methods in Software Engineering, Springer, 2020, pp. 385–413

    2020

  10. [18]

    Corbin, A

    J. Corbin, A. Strauss, Basics of qualitative research: Techniques and pro- cedures for developing grounded theory, Sage publications, 2014

    2014

  11. [19]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küt- tler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrieval-augmented gen- eration for knowledge-intensive nlp tasks, Advances in neural information processing systems 33 (2020) 9459–9474

    2020

  12. [20]

    Shull, J

    F. Shull, J. Singer, D. I. Sjøberg, Guide to advanced empirical software engineering, Vol. 93, Springer, 2008

    2008

  13. [21]

    Wohlin, P

    C. Wohlin, P. Runeson, M. Höst, M. C. Ohlsson, B. Regnell, A. Wesslén, et al., Experimentation in software engineering, Vol. 236, Springer, 2012. 28

    2012

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.