Pith. sign in

REVIEW 3 major objections 4 minor 22 references

Massive Multi-Agent Data-Driven Simulations of the GitHub Ecosystem

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Simple per-user frequency models, built from one month of history, outperform more complex agents on most metrics in large-scale GitHub simulation.

desk verdict A useful engineering report that shows simple per-user frequency models beating fancier ones at GitHub scale; the evaluation is thin but the qualitative result is credible. read the letter →

arxiv 1908.05437 v1 pith:LEIHLJW4 submitted 2019-08-15 cs.MA cs.SI

Jim Blythe , John Bollenbacher , Di Huang , Pik-Mai Hui , Rachel Krohn , Diogo Pacheco , Goran Muric , Anna Sapienza
show 7 more authors
Alexey Tregubov Yong-Yeol Ahn Alessandro Flammini Kristina Lerman Filippo Menczer Tim Weninger Emilio Ferrara
This is my paper · ORCID
classification cs.MAcs.SI
keywords multi-agentsimulationGitHubtechno-socialsystemsstationaryprobabilisticmodelagent-basedlinkpredictionBayesianlarge-scale
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a large-scale prediction exercise in which six different agent models were built to simulate activity on GitHub and their outputs were scored against one month of ground-truth events. Its central finding is that the most consistently accurate agents were not the most complex ones: they sampled each user's next action and repository from a probability distribution built from that same user's recent past behavior. This matters because it suggests that for a platform whose users change behavior slowly, a simple per-user empirical frequency model can outperform graph-embedding, Bayesian, and preferential-attachment agents on the majority of the reported metrics. The paper also shows that the simulation can be scaled to roughly three million agents and thirty million actions on commodity hardware, making this an existence proof for data-driven, planetary-scale agent simulation.

What carries the argument

The key object is the per-user stationary probability distribution over event types and repository targets, estimated from one month of each user's history and sampled at a rate fixed by that user's observed event frequency. One variant, the ground-event model, samples the event and repository jointly rather than independently, and it tended to do slightly better than the baseline variant. This distribution is what separates the winning agents from the null, Bayesian, and embedding-based agents; the paper attributes its advantage to preserving individual user identity and to the relative slowness of behavioral change on GitHub. The supporting machinery is a distributed simulation architecture that partitions users and repositories across compute nodes to run the full-scale simulation in about twenty minutes.

What would settle it

A direct test would retrain all six models on a different training-to-test month pair, say January 2019 to February 2019, and check whether the stationary model still wins on a majority of metrics; if a month containing an unusual shock, such as a large influx of new users or a major platform change, erases or reverses the stationary model's advantage, the claim that slow change explains its success would be falsified. A complementary check would measure per-user divergence between the training and test event distributions and show that users with large divergence are predicted as well as users with small divergence, which would contradict the stationarity explanation.

Watch

Extended reading notes

Core claim

The central claim, stated in the abstract and Section 5, is that among the six agent models tested in the GitHub simulation challenge, the broadly most successful agents sampled from a stationary probability distribution of actions and repositories for each user. The paper gives two reasons for this success: each agent is characterized by its own empirical frequencies rather than by population-level generalizations, and GitHub users change their behavior relatively slowly. No single model won every metric; in particular, the Bayesian model performed well on user popularity, and a null model that simply shifted past data forward did well on repository popularity because the top repositories turn over slowly. But on the majority of the reported metrics, including repository event counts and contributor dynamics, the stationary probabilistic models outperformed the link-prediction and Bayesian agents.

Load-bearing premise

The central claim rests on the premise that a GitHub user's behavior in the month being predicted looks like that same user's behavior in the previous month; if users' event rates or repository preferences shift quickly, the stationary model is merely replaying memorized data.

Editorial extensions

If this is right

  • For any platform where behavior is slow-changing, a one-month per-user frequency model becomes a strong baseline that more complex agents should be required to beat.
  • Using all available training data can hurt prediction; the paper found that roughly one month of recent data was optimal across most of the agent models.
  • The same simulation framework and agent-comparison methodology can be applied to other techno-social systems, and the authors report they have begun doing so for Twitter and Reddit.
  • The main unresolved gap is cold start: stationary models cannot predict interactions between users and repositories they have not touched before, so they must be paired with feature-based or network-based models to cover new entities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the stationarity explanation is right, then on faster-paced platforms the lead of simple frequency models should shrink; measuring the model's margin as a function of per-user behavioral-change rate would give a quantitative test.
  • The ground-event model's slight edge over the baseline suggests the joint distribution of event and repository carries information that the marginals discard; a low-rank approximation of that joint distribution might preserve the advantage at much lower memory cost.
  • A natural next experiment is to combine the stationary model for existing users with the feature-based cold-start model for new users; the paper mentions intra-agent combination as a direction, so this is an explicit extension rather than a claim.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper reports the authors' entry in the DARPA SocialSim GitHub challenge, in which six agent models—null, probabilistic baseline, probabilistic ground-event, preferential attachment, link prediction via embedding (LPE), and Bayesian—are compared for their ability to predict one month of GitHub activity (February 2018) from historical metadata. The authors find that no single model dominates on all metrics, but claim that the broadly most successful agents are those that sample from a stationary per-user probability distribution over event types and repositories, attributing this to the individual characterization of each user and to the relatively slow change of GitHub user behavior. The paper also describes the FARM/DASH multi-agent simulation framework and scaling techniques that allow simulations of roughly 3 million agents and 6 million repositories on a single 64GB host in about 20 minutes.

Significance. If the central claim holds, the result is practically significant: a simple, per-user empirical frequency model can outperform more sophisticated graph-embedding and Bayesian generative models on a majority of challenge metrics in a planetary-scale techno-social simulation. The paper's strengths include a common simulation environment for all six models, a held-out test month, and concrete engineering contributions for large-scale agent-based simulation, such as graph-partition-based communication reduction and demand-driven shared state. The authors are also explicit about several limitations of the stationary models, including their inability to generate previously unobserved user-repository pairs. However, the evidence for the central ranking claim is currently too thin to be conclusive, because it rests on a single test month, a subset of metrics, and no uncertainty quantification.

major comments (3)
  1. [Section 5, Figure 3] The central claim that the stationary probabilistic agents are 'broadly the most successful' is supported only by five metrics selected from 'more than a dozen' challenge metrics, a single test month (February 2018), and no error bars or significance tests. The reported metric-level winners vary: the null model has the lowest RMSE on repository contributors and is competitive on popularity metrics, the Bayesian model is credited with strong user popularity, and the ground-event and baseline models lead on repository event count issues. Without a pre-specified aggregation rule over all challenge metrics, or confidence intervals from repeated evaluation, the qualitative ranking 'broadly most successful' is not uniquely determined by the evidence shown.
  2. [Section 4.1 and Section 7] The stationarity assumption is asserted rather than tested. Section 4.1 states that the approach 'is justified if users' future behavior tends to be similar to their past behavior,' while Section 7 acknowledges that 'overall behavior is constantly changing' and that using one month of data proved optimal. The single February 2018 test period cannot validate this assumption, and the null model—a two-week shift of past data—is competitive on several reported metrics. The paper should evaluate the models on multiple held-out months (e.g., a rolling-origin evaluation across several test months) or provide bootstrapped confidence intervals; otherwise the dominance of the stationary agents could be period-specific or within noise.
  3. [Section 2 and Section 7] The description of the training data is internally inconsistent and hampers reproducibility. Section 2 says the training set comprises events from 8/1/17–8/31/17 and 1/17/18–1/31/18 (two one-month windows separated by a gap), while Section 7 states that 'one month of data proved optimal across most of the agents.' It is unclear which of the two training windows, or which combination, was used for each of the six models, including the stationary agents whose per-user frequencies are central to the main claim. Please clarify the exact training input for each model.
minor comments (4)
  1. [Throughout] There are several typographical issues: 'T able 1' in Section 4.2, 'e.g,.' in Section 4.2, 'underling' in Section 4.3, and inconsistent spacing in 'F ARM' and 'DASH'.
  2. [Section 4.2, Table 1] Table 1 reports MAP values for link prediction methods without standard deviations or significance tests; the choice of Graph Factorization is motivated by scalability, but the reported differences (e.g., GF vs. HOPE on forks) may not be significant.
  3. [Section 4.3, Figure 1] The probabilities shown in Figure 1 (e.g., 1.5% new user, 84.8% one-time event) are not derived in the text; please state which data period and filtering criteria were used to compute these frequencies.
  4. [Section 5] The phrase 'All models except the null model had a high percentage (more than 0.75) of community contributing users' is imprecise because Figure 3 does not show numeric values; please report the exact values in the text or table.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the agent comparison is a genuine held-out prediction task; self-citations are to infrastructure and external rank model.

full rationale

The paper's central claim is that stationary per-user probabilistic agents were broadly the most successful at predicting GitHub activity in February 2018. This is a genuine predictive evaluation: agents are fit on historical data from August 2017 and January 2018, and scored against held-out ground truth from February 2018. The stationary models' probabilities are empirical frequencies computed from past user actions, but the reported metrics (top-500 popularity RBO, repository event count R2, repository contributors RMSE, community contributing users percentage) are all computed on the test period, so a high score is not equivalent to the training input by construction. No equation in the paper reduces a predicted metric to a fitted parameter or defines the input in terms of the output. The self-citations to FARM [5] and DASH [4] describe the simulation framework used to run all agents; they are implementation infrastructure and do not bear the weight of the comparative empirical result. The rank model [11], though co-authored by two of the present authors, is an external published mechanism used inside one of six agent variants, and it is not invoked to forbid alternatives or to force the stationary agents' success. Section 4.1 explicitly states the stationarity assumption as a justification ('this modeling approach is justified if users' future behavior tends to be similar to their past behavior'), not as a derived theorem. Section 7's observation that one month of training data was optimal is a post-hoc empirical finding about the challenge period, not a parameter fitted to the evaluation metric. The paper is self-contained against external benchmarks in the sense that the challenge ground truth is external to the model construction. No circular step meeting the quoted-evidence bar was found.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claims rest on a small number of domain assumptions about GitHub user behavior and on several fitted or empirical parameters. No new theoretical entities are introduced; FARM and DASH are software frameworks, not postulated entities.

free parameters (7)
  • Per-user event rate = estimated from one month of training history
    Each agent's overall rate of actions is fixed at the frequency observed in the prior month (Sections 4.1 and 7). The choice of a one-month window is stated post hoc to be optimal.
  • Per-user event-type probabilities = empirical frequencies of event types per user
    Baseline and related models select event types according to the user's own historical frequency distribution over event types (Section 4.1).
  • User-repository pair frequencies = empirical counts from one-month training window
    The ground-event model selects event and repository pairs with probabilities equal to historical co-occurrence frequencies (Section 4.1).
  • 30-day half-life decay for user rank = 30 days
    The Bayesian model weights past activity with a 30-day half-life decay when ranking users (Section 4.3).
  • Mean of geometric distribution for social walk = 2
    The Bayesian model chooses other users' repositories via a random walk whose length follows a geometric distribution with mean 2 (Section 4.3).
  • LPE tail threshold = 100 repositories per user
    Link prediction via embedding truncates repository-selection probability tails to 100 repositories per user to manage memory (Section 5).
  • One-month training window = single previous month
    Section 7 reports that one month of data outperformed longer histories for most agents; this is a data-dependent hyperparameter, not a first-principles choice.
assumptions (4)
  • domain assumption GitHub users' future behavior is statistically similar to their recent past behavior over a one-month horizon.
    The stationary probabilistic models assume each user's event rate and preferences remain constant across the test month; the paper supports this with the observation that GitHub users change behavior slowly (Sections 4.1 and 7).
  • domain assumption The provided metadata are a faithful sample of the GitHub events that the evaluation metrics are computed on.
    The models and evaluation both rely on the supplied training set and ground truth without independent validation of data completeness (Section 2).
  • domain assumption The PNNL evaluation metrics are accepted measures of simulation fidelity.
    The paper takes the DARPA-defined metrics as given and does not justify that they measure predictive quality (Section 5).
  • domain assumption New users and repositories can be characterized by 124 hand-compiled features even without observable interaction history.
    Section 4.4 constructs models for new user-repository pairs from 124 engineered features, implicitly assuming these features capture the relevant generating process.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Massive Multi-Agent Data-Driven Simulations of the GitHub Ecosystem." pith.science (2026). https://pith.science/paper/LEIHLJW4

@misc{pith2026190805437,
  author       = {Pith},
  title        = {Pith review of: Massive Multi-Agent Data-Driven Simulations of the GitHub Ecosystem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LEIHLJW4}},
  note         = {Machine review of arXiv:1908.05437}
}
read the original abstract

Simulating and predicting planetary-scale techno-social systems poses heavy computational and modeling challenges. The DARPA SocialSim program set the challenge to model the evolution of GitHub, a large collaborative software-development ecosystem, using massive multi-agent simulations. We describe our best performing models and our agent-based simulation framework, which we are currently extending to allow simulating other planetary-scale techno-social systems. The challenge problem measured participant's ability, given 30 months of meta-data on user activity on GitHub, to predict the next months' activity as measured by a broad range of metrics applied to ground truth, using agent-based simulation. The challenge required scaling to a simulation of roughly 3 million agents producing a combined 30 million actions, acting on 6 million repositories with commodity hardware. It was also important to use the data optimally to predict the agent's next moves. We describe the agent framework and the data analysis employed by one of the winning teams in the challenge. Six different agent models were tested based on a variety of machine learning and statistical methods. While no single method proved the most accurate on every metric, the broadly most successful sampled from a stationary probability distribution of actions and repositories for each agent. Two reasons for the success of these agents were their use of a distinct characterization of each agent, and that GitHub users change their behavior relatively slowly.

Figures

Figures reproduced from arXiv: 1908.05437 by the authors.

Figure 1
Figure 1. Bayesian model from data inferred frequencies. We investigated the trade-off between recency and history as driving forces to popularity [2]. The results showed that less is more in terms of the amount of data needed to predict users’ activity level. For instance, the top-200 most active user rank of a particular month will intersect less with the rank from previous months as we aggregate more data; e.g., while +75%… view at source ↗
Figure 2
Figure 2. On the left, the variation in the target variable #PushEvents, explained by each feature in each consecutive step of the S3D algorithm. On the right, the two most important features to predict the number of events for a new user-repository interaction. The R 2 column shows the total gain of the two features. The aim is to build a parsimonious model, able to predict the frequency of a particular event type e performe… view at source ↗
Figure 3
Figure 3. Left: popularity metrics, RBO, and community contributing users, higher is better. Center: Repository contributors, RMSE, lower is better and right: event issue count, R 2 , higher is better. old of 100 repositories per user. The ground-event model showed slightly better results on popularity metrics than the baseline model and the preferential attach￾ment model. All three models use the same approach to compute rat… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages

  1. [1]

    In: Mining Software Repositories

    Aggarwal, K., Hindle, A., Stroulia, E.: Co-evolution of project documentation and popularity within github. In: Mining Software Repositories. MSR (2014)

    work page 2014

  2. [2]

    EPJ Data Science 4(1), 1–14 (dec 2015)

    Barbosa, H., de Lima-Neto, F.B., Evsukoff, A., Menezes, R.: The effect of recency to human mobility. EPJ Data Science 4(1), 1–14 (dec 2015)

    work page 2015

  3. [3]

    In: 2013 IEEE 37th Annual Computer Software and Applications Conference (2013)

    Bissyand, T.F., Thung, F., Lo, D., Jiang, L., Rveillre, L.: Popularity, interoper- ability, and impact of programming languages in 100,000 open source projects. In: 2013 IEEE 37th Annual Computer Software and Applications Conference (2013)

    work page 2013

  4. [4]

    In: 5th International Symposium on Resilient Control Systems

    Blythe, J.: A dual-process cognitive model for testing resilient control systems. In: 5th International Symposium on Resilient Control Systems. pp. 8–12 (Aug 2012)

    work page 2012

  5. [5]

    In: IJCAI/AAMAS Workshop on Massively Multi-Agent Systems (2018)

    Blythe, J., Tregubov, A.: Farm: Architecture for distributedagent-based social sim- ulations. In: IJCAI/AAMAS Workshop on Massively Multi-Agent Systems (2018)

    work page 2018

  6. [6]

    In: PROMISE (2016)

    Borges, H., Hora, A.C., Valente, M.T.: Predicting the popularity of github reposi- tories. In: PROMISE (2016)

    work page 2016

  7. [7]

    Simulation 89(10), 1215–1235 (2013)

    Collier, N., North, M.: Parallel agent-based simulation with repast for high perfor- mance computing. Simulation 89(10), 1215–1235 (2013)

    work page 2013

  8. [8]

    In: Parallel, Distributed and Network-Based Processing (PDP), 2011 19th Euromicro International Conference on

    Cosenza, B., Cordasco, G., De Chiara, R., Scarano, V.: Distributed load balancing for parallel agent-based simulations. In: Parallel, Distributed and Network-Based Processing (PDP), 2011 19th Euromicro International Conference on. IEEE (2011)

    work page 2011

Show all 22 references
  1. [9]

    In: CSCW (2012)

    Dabbish, L.A., Stuart, H.C., Tsay, J., Herbsleb, J.D.: Social coding in github: transparency and collaboration in an open software repository. In: CSCW (2012)

    2012

  2. [10]

    Fennell, P., Zuo, Z., Lerman, K.: Predicting and Explaining Behavioral Data with Structured Feature Space Decomposition (2018), https://arxiv.org/abs/1810. 09841

    2018

  3. [11]

    Physical Review Letters 96(21), 218701 (may 2006)

    Fortunato, S., Flammini, A., Menczer, F.: Scale-free network growth by ranking. Physical Review Letters 96(21), 218701 (may 2006)

    2006

  4. [12]

    In: 2012 9th IEEE Working Conference on Mining Software Repositories (MSR) (June 2012)

    Gousios, G., Spinellis, D.: Ghtorrent: Github’s data from a firehose. In: 2012 9th IEEE Working Conference on Mining Software Repositories (MSR) (June 2012)

    2012

  5. [13]

    In: Proceedings of the 10th Working Conference on Mining Software Repositories

    Gousios, G.: The ghtorent dataset and tool suite. In: Proceedings of the 10th Working Conference on Mining Software Repositories. MSR ’13, IEEE Press (2013)

    2013

  6. [14]

    Royal Society open science 3(4), 160007 (2016)

    Klug, M., Bagrow, J.P.: Understanding the group dynamics and success of teams. Royal Society open science 3(4), 160007 (2016)

    2016

  7. [15]

    CoRR abs/1407.2535 (2014)

    Lima, A., Rossi, L., Musolesi, M.: Coding together at scale: Github as a collabo- rative social network. CoRR abs/1407.2535 (2014)

    2014 arXiv

  8. [16]

    In: IJCAI/AAMAS Workshop on Massively Multi-Agent Systems (2018)

    Noda, I.: Multi-agent social simulation for social service design. In: IJCAI/AAMAS Workshop on Massively Multi-Agent Systems (2018)

    2018

  9. [17]

    In: Agents for Games and Simulations

    ˇSiˇ sl´ ak, D., Volf, P., Jakob, M., Pˇ echouˇ cek, M.: Distributed platform for large-scale agent-based simulations. In: Agents for Games and Simulations. Springer (2009)

    2009

  10. [18]

    PLoS ONE 9(8), e103023 (aug 2014)

    Sornette, D., Maillart, T., Ghezzi, G.: How Much Is the Whole Really More than the Sum of Its Parts? 1 + 1 = 2.5: Superlinear Productivity in Collective Group Actions. PLoS ONE 9(8), e103023 (aug 2014)

    2014

  11. [19]

    In: Software Maintenance and Reengineering

    Thung, F., Bissyande, T.F., Lo, D., Jiang, L.: Network structure of social coding in github. In: Software Maintenance and Reengineering. CSMR (2013)

    2013

  12. [20]

    In: Autonomous Agents and Multiagent Systems

    Tumer, K., Agogino, A.: Distributed agent-based air traffic flow management. In: Autonomous Agents and Multiagent Systems. AAMAS ’07, ACM (2007)

    2007

  13. [21]

    ACM Transactions on Information Systems 28(4), 1–38 (nov 2010)

    Webber, W., Moffat, A., Zobel, J.: A similarity measure for indefinite rankings. ACM Transactions on Information Systems 28(4), 1–38 (nov 2010)

    2010

  14. [22]

    In: Proc

    Zhu, J., Zhou, M., Mockus, A.: Patterns of folder use and project popularity: A case study of github repositories. In: Proc. 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. ESEM ’14, ACM (2014)

    2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.