pith. sign in

arxiv: 2607.02453 · v1 · pith:BJUK5JTTnew · submitted 2026-07-02 · 💻 cs.MA

Adoption and Ecosystem Health: A Longitudinal Analysis of Open-Source Multi-Agent Frameworks

Xi Zhang (Cisco Systems) , Papi Menon (Cisco Systems) , Vivian Chu (Cisco Systems) , Koray Cosguner (Indiana University) This is my paper

Pith reviewed 2026-07-03 02:41 UTC · model grok-4.3

classification 💻 cs.MA
keywords open-source frameworksAI agentsecosystem healthGitHub metricscontributor densityretentionmulti-agent systemsadoption analysis
0
0 comments X

The pith

Contributor density, cross-ecosystem engagement, and retention better indicate open-source AI agent framework health than star counts alone.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tracks 15 open-source AI agent frameworks with GitHub data spanning late 2022 to early 2026, including over 800,000 stars and nearly one million user profiles. It shows that star counts often track hype cycles rather than real adoption, with examples like AutoGPT converting far fewer contributors per star than other projects. Contributor density, defined here as contributors per thousand stars, plus patterns of cross-framework participation and early retention, separate superficial visibility from sustained community activity. LangChain emerges as a central node attracting the large majority of contributors who work across multiple frameworks. These metrics together give engineering teams a clearer basis for choosing frameworks than popularity signals provide.

Core claim

Ecosystem health in open-source multi-agent frameworks is better measured by contributor density, cross-ecosystem engagement, and retention than by stars alone, as shown by analysis of 808,042 stars, 73,997 pull requests, 86,241 commits, and 987,330 user profiles across 15 repositories from late 2022 to early 2026.

What carries the argument

Contributor density, defined as contributors per 1,000 stars, which quantifies conversion from visibility to active participation and distinguishes hype from adoption.

Load-bearing premise

The GitHub metrics collected accurately and unbiasedly reflect awareness, adoption, and retention across the 15 selected frameworks without significant platform artifacts or selection effects.

What would settle it

A controlled comparison showing that star counts predict actual framework usage, project longevity, or developer satisfaction more accurately than contributor density or retention curves.

Figures

Figures reproduced from arXiv: 2607.02453 by Koray Cosguner (Indiana University), Papi Menon (Cisco Systems), Vivian Chu (Cisco Systems), Xi Zhang (Cisco Systems).

Figure 1
Figure 1. Figure 1: Cumulative GitHub Stars Over Time Across 15 Open-Source AI Frameworks 3.1 Cumulative Star Trajectories: A Cross-Framework Overview AutoGPT's surge in April 2023 marked a significant inflection point in the early ecosystem, accumulating 111,967 stars in a single month and surpassing all other frameworks in this dataset. This spike was largely driven by widely circulated social media demonstrations in which … view at source ↗
Figure 2
Figure 2. Figure 2: Star Accumulation Chart Normalized to Months Since Launch [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Cumulative Stars for First Party SDKs As shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 6
Figure 6. Figure 6: Repo Contributors and Profile Repo Name Total Contributors Median Account Age (yrs) % Company Affiliated langchain 5362 8.9 37.7 langgraph 607 7.5 38.4 langflow 707 7.9 39.5 MetaGPT 257 7 30.6 autogen 869 8.8 39.7 crewAI 645 8.4 38.9 semantic-kernel 618 10.1 49.2 agentscope 145 7 26.4 smolagents 395 8.4 35.9 mastra 484 9.2 44.5 openai-agents-python 438 6.3 36.9 adk-python 621 7.6 39.2 pydantic-ai 675 9 44.… view at source ↗
Figure 9
Figure 9. Figure 9: Contributor Density Ratio [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
read the original abstract

Since ChatGPT's launch in November 2022, open-source agentic frameworks have proliferated, making framework selection important for engineering teams while obscured by popularity signals such as GitHub stars. This paper analyzes 15 major open-source AI agent framework repositories from late 2022 to early 2026, using 808,042 stars, 73,997 pull requests, 86,241 commits, and 987,330 user profiles to assess ecosystem health across awareness, adoption, and retention. Three findings emerge. First, headline popularity is unreliable. Star counts reflect hype cycles and inorganic activity. AutoGPT gained 111,967 stars in one month but converted fewer than 9 contributors per 1,000 stars, defined as contributor density in this research, compared with LangChain's 41. Lower-profile frameworks such as Pydantic-AI show higher contributor density, indicating deeper adoption. Second, mapping awareness against adoption shows that visibility and engagement diverge. MetaGPT and LangFlow have contributor density ratios below 5 even with their high visibility. Openai-agents-python's limited contributor base suggests institutional backing alone does not ensure community depth. By analyzing cross-framework contribution, we discover that LangChain functions as a shared infrastructure, attracting 82.5% of cross-ecosystem contributors. Third, retention drops most steeply in the first 30 days of initial contribution and stabilizes near 90 days. Overall, ecosystem health is better measured by contributor density, cross-ecosystem engagement, and retention than by stars alone. These metrics offer teams a more robust basis for framework evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper analyzes GitHub data from 15 open-source multi-agent frameworks (808k stars, 74k PRs, 86k commits, 987k profiles) over late 2022 to early 2026. It argues that star counts are unreliable due to hype and inorganic activity, and that contributor density (contributors per 1,000 stars), cross-ecosystem engagement, and retention (steep drop in first 30 days, stabilizing at 90 days) are superior metrics for awareness, adoption, and retention. Specific claims include AutoGPT's low density (<9/1k stars) vs. LangChain (41/1k), divergence in visibility/engagement for MetaGPT/LangFlow, LangChain attracting 82.5% of cross-contributors, and Pydantic-AI showing higher density.

Significance. If the results hold after validation, the work could influence how engineering teams select AI agent frameworks by prioritizing depth and retention metrics over visibility. The longitudinal scale and concrete examples (e.g., LangChain's infrastructure role) add potential value for the open-source AI community if the metrics are shown to be less biased than stars.

major comments (3)
  1. [Abstract] Abstract: The central claim that contributor density, cross-ecosystem engagement, and retention are demonstrably superior proxies for awareness/adoption/retention (and less susceptible to artifacts than stars) is load-bearing but unsupported by any described validation, robustness tests, external benchmarks, or controls for bots/inorganic activity.
  2. [Abstract] Abstract: No information is given on selection criteria or justification for the 15 frameworks, which is critical for assessing sample bias and whether the divergence findings generalize beyond the chosen set.
  3. [Abstract] Abstract: Retention analysis lacks any description of computation (definition of 'initial contribution', handling of repeat contributors, bot filtering, or profile deduplication), undermining the third finding on 30/90-day patterns.
minor comments (2)
  1. [Abstract] Abstract: The exact date ranges for 'late 2022 to early 2026' and data collection endpoints should be specified to enable reproducibility.
  2. [Abstract] Abstract: 'Contributor density' is defined inline but would benefit from an explicit formula or table showing per-framework values for transparency.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below and indicate the revisions that will be incorporated.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The central claim that contributor density, cross-ecosystem engagement, and retention are demonstrably superior proxies for awareness/adoption/retention (and less susceptible to artifacts than stars) is load-bearing but unsupported by any described validation, robustness tests, external benchmarks, or controls for bots/inorganic activity.

    Authors: The manuscript's primary evidence consists of the observed empirical divergences (e.g., AutoGPT's star surge versus low contributor density, LangChain's cross-contributor share) that illustrate the limitations of star counts. We acknowledge that the initial submission does not include explicit robustness tests or external validation of the metrics' superiority. In the revised manuscript we will add a dedicated robustness subsection describing bot-filtering procedures, sensitivity analyses on inorganic activity, and a comparison against an external adoption signal where available. revision: yes

  2. Referee: [Abstract] Abstract: No information is given on selection criteria or justification for the 15 frameworks, which is critical for assessing sample bias and whether the divergence findings generalize beyond the chosen set.

    Authors: The 15 frameworks were chosen as the most prominent open-source multi-agent repositories by GitHub activity and community visibility at the start of the observation window. We agree that explicit criteria are required for reproducibility and bias assessment. The revised methods section will include a clear selection protocol together with a discussion of scope and generalizability. revision: yes

  3. Referee: [Abstract] Abstract: Retention analysis lacks any description of computation (definition of 'initial contribution', handling of repeat contributors, bot filtering, or profile deduplication), undermining the third finding on 30/90-day patterns.

    Authors: The abstract omits these operational details. The underlying analysis defines an initial contribution as the first commit or merged PR associated with a unique GitHub user ID, applies username-pattern bot filtering, and treats repeat contributors by retaining only the first event per profile. We will expand both the abstract and the methods section to document these steps fully, including any additional deduplication logic. revision: yes

Circularity Check

0 steps flagged

Empirical metric definitions and observations contain no circular reductions

full rationale

The paper is a longitudinal empirical analysis of GitHub repository data for 15 frameworks. It defines contributor density directly as contributors per 1,000 stars from the raw counts (808k stars, 74k PRs, etc.) and reports observed divergences (e.g., AutoGPT vs. LangChain) without any equations, fitted parameters, predictions that reduce to inputs, or load-bearing self-citations. The central claim that density, cross-ecosystem engagement, and retention are superior proxies follows from the data patterns themselves rather than from any definitional equivalence or imported uniqueness result. No derivation chain reduces to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no information on free parameters, axioms, or invented entities; the analysis appears purely observational.

pith-pipeline@v0.9.1-grok · 5838 in / 1062 out tokens · 21087 ms · 2026-07-03T02:41:57.829418+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages · 4 internal anchors

  1. [1]

    Adoption and Ecosystem Health: A Longitudinal Analysis of Open-Source Multi-Agent Frameworks Xi Zhang, PhD Cisco Systems Papi Menon Cisco Systems Vivian Chu Cisco Systems Koray Cosguner, PhD Indiana University Abstract Since ChatGPT’s launch in November 2022, open-source agentic frameworks have proliferated, making framework selection important for engine...

    work page 2022

  2. [2]

    Introduction The evolution of agentic AI frameworks reflects a broader industry understanding of deploying reliable AI in real-world settings, beyond mere software adoption [1]. Following ChatGPT’s public launch in late 2022 [2], the 2023–2024 period signaled the rise of autonomous agents, with frameworks like LangChain enabling composable LLM pipelines a...

    work page 2022

  3. [3]

    Data This study draws on publicly available data from GitHub, collected via the GitHub REST API and GraphQL API. The dataset spans multiple dimensions of repository activities, including star events, commit histories, pull request records, and user profile, enabling analysis across three analytical layers: community awareness, active adoption, and contrib...

    work page 2026

  4. [4]

    langgraph langchain-ai Aug, 2023 Stateful multi-actor graph orchestration autogen microsoft Aug, 2023 Conversational multi-agent framework

    work page 2023

  5. [5]

    All repositories were accessed on March 10,

    crewAI crewAIInc Oct, 2023 Role-based multi-agent task orchestration agentscope agentscope-ai Jan, 2024 Multi-agent platform with fault tolerance pydantic-ai pydantic Jun, 2024 Type-safe agent framework built on Pydantic mastra mastra-ai Aug, 2024 TypeScript-native agent and workflow framework smolagents huggingface Dec, 2024 Lightweight code-first agent ...

    work page 2023

  6. [6]

    Collection covered each repository’s full public history from its creation date through March 2026, spanning repositories with a creation window from October 2022 to May

    2.2 Data Collected Data was collected from the GitHub API across four event types for each of the 15 repositories: stars, pull requests, commits, and user profiles. Collection covered each repository’s full public history from its creation date through March 2026, spanning repositories with a creation window from October 2022 to May

    work page 2026

  7. [7]

    Awareness The primary gauge of public awareness for an open-source project is its GitHub star count. Across the 15 projects that were studied in this research, star growth varied considerably (see Figure 1 for cumulative star trajectories plotted against calendar time and Appendix A for individual star histories showing both cumulative accumulation and mo...

    work page 2023

  8. [8]

    The three repositories cluster between 14,000 and 21,000 stars at month 6, indicating that institutional origin provides a significant, though not dominant, advantage at launch

    Google ADK recorded 14,204 stars at month 6, with no data available afterward because the repository had not yet reached month 12 at the time of data collection. The three repositories cluster between 14,000 and 21,000 stars at month 6, indicating that institutional origin provides a significant, though not dominant, advantage at launch. Whether this earl...

    work page 2019

  9. [9]

    and Google ADK's design around enterprise software engineering principles [15], including modularity, testability, and deployment reliability, make both frameworks particularly attractive to the kind of experienced contributors whose elevated account ages and PR merge rates were observed across cross-ecosystem pairings in this dataset. 4.3 Awareness and A...

    work page 2024

  10. [10]

    This indicates that although the project initially attracted considerable attention, such interest did not result in sustained user engagement. We argue that AutoGPT's low retention reflects its release prior to the widespread adoption of engineering safeguards that later became standard in agentic frameworks, including loop termination, recursive error h...

    work page 2025

  11. [11]

    Limitation and Future Study This analysis is bounded by its scope and observation window. The 15 repositories were selected to represent the AI agent framework landscape as of March 2026, but the selection favors English-language, GitHub-hosted, open-source projects and excludes commercial products, private repositories, and projects on non-GitHub platfor...

    work page 2026

  12. [12]

    A survey on large language model based autonomous agents.Frontiers Comput

    Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., Zhao, W. X., Wei, Z., & Wen, J. R. (2024). A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), 186345. https://doi.org/10.1007/s11704-024-40231-1

    work page doi:10.1007/s11704-024-40231-1 2024

  13. [13]

    OpenAI. (2022). ChatGPT: Optimizing language models for dialogue. OpenAI Blog. https://openai.com/blog/chatgpt

    work page 2022

  14. [14]

    Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In Proceedings of the International Conference on Learning Representations (ICLR 2023). https://arxiv.org/abs/2210.03629

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [15]

    Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., Zheng, R., Fan, X., Wang, X., Xiong, L., Zhou, Y., Wang, W., Jiang, C., Zou, Y., Liu, X., ... Gui, T. (2023). The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864. https://arxiv.org/abs/2309.07864

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  16. [16]

    Borges, H., Hora, A., & Valente, M. T. (2016). Understanding the factors that impact the popularity of GitHub repositories. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME) (pp. 334–344). IEEE. https://doi.org/10.1109/ICSME.2016.31

    work page doi:10.1109/icsme.2016.31 2016

  17. [17]

    Coelho, J., & Valente, M. T. (2017). Why modern open source projects fail. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE

    work page 2017

  18. [18]

    186–196)

    (pp. 186–196). ACM. https://doi.org/10.1145/3106237.3106246

    work page doi:10.1145/3106237.3106246

  19. [19]

    Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S. K. S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C., & Schmidhuber, J. (2024). MetaGPT: Meta programming for a multi-agent collaborative framework. In Proceedings of the International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2308.00352

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  20. [20]

    Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., & Wang, C. (2023). AutoGen: Enabling next-gen LLM applications via multi-agent conversation framework. arXiv preprint arXiv:2308.08155. https://arxiv.org/abs/2308.08155

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  21. [21]

    He, H., Yang, H., Burckhardt, P., Kapravelos, A., Vasilescu, B., & Kästner, C. (2026). Six million (suspected) fake stars on GitHub: A growing spiral of popularity contests, spam, and malware. In Proceedings of the 48th International Conference on Software Engineering (ICSE '26). https://arxiv.org/abs/2412.13459

    work page arXiv 2026

  22. [22]

    (2025, February 25)

    IBM. (2025, February 25). IBM to acquire DataStax, deepening watsonx capabilities and addressing generative AI data needs for the enterprise. IBM Newsroom. https://newsroom.ibm.com/2025-02-25-ibm-to-acquire-datastax,-deepening-watsonx-capabilities-and-addressing-generative-ai-data-needs-for-the-enterprise

    work page 2025

  23. [23]

    Bessemer Venture Partners. (2024). Measuring the engagement of an open source software community. https://www.bvp.com/atlas/measuring-the-engagement-of-an-open-source-software-community

    work page 2024

  24. [24]

    G., & Matsumoto, K

    Kannee, K., Wattanakriengkrai, S., Rojpaisarnkit, R., Kula, R. G., & Matsumoto, K. (2022). Intertwining ecosystems: A large scale empirical study of libraries that cross software ecosystems. https://arxiv.org/abs/2208.06655

    work page arXiv 2022

  25. [25]

    IBM. (2025). What is LangChain? https://www.ibm.com/think/topics/langchain

    work page 2025

  26. [26]

    Pydantic. (2025). Pydantic AI v1: A predictable & robust GenAI framework. https://pydantic.dev/articles/pydantic-ai-v1

    work page 2025

  27. [27]

    Google. (2025). Technical overview — Agent Development Kit (ADK). https://google.github.io/adk-docs/get-started/about/

    work page 2025

  28. [28]

    LangChain AI. (2025). LangGraph overview. LangGraph Documentation. https://docs.langchain.com/oss/python/langgraph/overview

    work page 2025

  29. [29]

    CrewAI Inc. (2025). CrewAI documentation. https://docs.crewai.com

    work page 2025

  30. [30]

    A., Gerosa, M

    Steinmacher, I., Graciotto Silva, M. A., Gerosa, M. A., & Redmiles, D. F. (2015). A systematic literature review on the barriers faced by newcomers to open source software projects. Information and Software Technology, 59, 67–85. https://doi.org/10.1016/j.infsof.2014.11.001 Appendix Appendix A: Individual Cumulative Star and Monthly Star Histories for All...

    work page doi:10.1016/j.infsof.2014.11.001 2015