{"id":"c7a9ea8d-473b-484c-9a95-9ba11936d558","arxiv_id":"1908.05437","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"In the DARPA SocialSim GitHub challenge, per-user stationary probabilistic agents broadly outperformed graph-embedding and Bayesian agents, because GitHub users' behavior changes slowly.","lead":"This paper describes a simulation of about three million GitHub users built to predict their activity one month ahead, as part of the DARPA SocialSim challenge. It finds that the simplest model, replaying each user's recent habits from a one-month history, was more broadly accurate than fancier machine-learning agents.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stationary-agent dominance rests on one test month and five selected metrics, with no error bars or pre-specified aggregation; robustness is unestablished.","rationale":"The reader's conditional verdict is appropriate. The paper does make a genuine held-out prediction: agents are trained on August 2017 and January 2018 and tested on February 2018, so the comparison is predictive rather than circular. The simulation scale and the systematic comparison of six models are also real strengths. However, the central comparative claim is underdetermined by the reported evidence. The paper does not provide error bars, significance tests, or a pre-specified aggregation rule for the 'broad range' of metrics it invokes. The reader's weakest assumption, stationarity, is part of the problem, but the more precise issue is that the evaluation protocol itself cannot separate a true modeling advantage from metric selection and a single favorable test month. The null model's competitiveness on several reported metrics reinforces this concern: if a two-week replay of past data is nearly as good on some metrics, the stationary agents' success may largely reflect persistence in the data rather than the specific probabilistic machinery. A concrete robustness check, using an additional held-out month and bootstrap confidence intervals over all challenge metrics, would settle whether the headline claim generalizes or is an artifact of the chosen evaluation slice.","tokens_in":9522,"tokens_out":3734,"duration_ms":38395,"concrete_test":"Re-run the six agent models on a second held-out month (e.g., March 2018, with the same training-window design) and on bootstrap resamples of days/events within February 2018. Compute all 'more than a dozen' PNNL metrics, not just the five in Figure 3, and aggregate them by mean rank (or median normalized score) with 95% bootstrap confidence intervals. If the stationary agents do not rank first under this aggregation, or if their margin is inside the confidence interval, the 'broadly most successful' claim is not robust and should be stated with explicit metric and time-period caveats.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the stationary per-user probabilistic agents are 'broadly the most successful' among the six models (abstract and Section 5). The evidence is Figure 3, which covers only five metrics selected from 'more than a dozen' challenge metrics, one test month, and no error bars. The metric-level winners vary: the null model (a two-week shift of past data) has the lowest RMSE on repository contributors and is competitive on popularity; the Bayesian model is credited with strong user popularity; the ground-event and baseline models lead on repository event count issues. Thus 'broadly most successful' depends on an unspecified weighting or aggregation over metrics. Section 4.1 justifies stationarity by asserting that 'users' future behavior tends to be similar to their past behavior,' but the single February 2018 test period cannot validate this; Section 7 itself notes that 'overall behavior is constantly changing,' and the best training window is only one month. The absence of repeated test months or bootstrapped confidence intervals means the advantage of the stationary agents could be period-specific or within noise, especially since a pure replay baseline is competitive on several reported metrics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the authors' entry in the DARPA SocialSim GitHub challenge, in which six agent models—null, probabilistic baseline, probabilistic ground-event, preferential attachment, link prediction via embedding (LPE), and Bayesian—are compared for their ability to predict one month of GitHub activity (February 2018) from historical metadata. The authors find that no single model dominates on all metrics, but claim that the broadly most successful agents are those that sample from a stationary per-user probability distribution over event types and repositories, attributing this to the individual characterization of each user and to the relatively slow change of GitHub user behavior. The paper also describes the FARM/DASH multi-agent simulation framework and scaling techniques that allow simulations of roughly 3 million agents and 6 million repositories on a single 64GB host in about 20 minutes.","tokens_in":9688,"tokens_out":3316,"duration_ms":33245,"significance":"If the central claim holds, the result is practically significant: a simple, per-user empirical frequency model can outperform more sophisticated graph-embedding and Bayesian generative models on a majority of challenge metrics in a planetary-scale techno-social simulation. The paper's strengths include a common simulation environment for all six models, a held-out test month, and concrete engineering contributions for large-scale agent-based simulation, such as graph-partition-based communication reduction and demand-driven shared state. The authors are also explicit about several limitations of the stationary models, including their inability to generate previously unobserved user-repository pairs. However, the evidence for the central ranking claim is currently too thin to be conclusive, because it rests on a single test month, a subset of metrics, and no uncertainty quantification.","major_comments":[{"comment":"The central claim that the stationary probabilistic agents are 'broadly the most successful' is supported only by five metrics selected from 'more than a dozen' challenge metrics, a single test month (February 2018), and no error bars or significance tests. The reported metric-level winners vary: the null model has the lowest RMSE on repository contributors and is competitive on popularity metrics, the Bayesian model is credited with strong user popularity, and the ground-event and baseline models lead on repository event count issues. Without a pre-specified aggregation rule over all challenge metrics, or confidence intervals from repeated evaluation, the qualitative ranking 'broadly most successful' is not uniquely determined by the evidence shown.","section":"Section 5, Figure 3"},{"comment":"The stationarity assumption is asserted rather than tested. Section 4.1 states that the approach 'is justified if users' future behavior tends to be similar to their past behavior,' while Section 7 acknowledges that 'overall behavior is constantly changing' and that using one month of data proved optimal. The single February 2018 test period cannot validate this assumption, and the null model—a two-week shift of past data—is competitive on several reported metrics. The paper should evaluate the models on multiple held-out months (e.g., a rolling-origin evaluation across several test months) or provide bootstrapped confidence intervals; otherwise the dominance of the stationary agents could be period-specific or within noise.","section":"Section 4.1 and Section 7"},{"comment":"The description of the training data is internally inconsistent and hampers reproducibility. Section 2 says the training set comprises events from 8/1/17–8/31/17 and 1/17/18–1/31/18 (two one-month windows separated by a gap), while Section 7 states that 'one month of data proved optimal across most of the agents.' It is unclear which of the two training windows, or which combination, was used for each of the six models, including the stationary agents whose per-user frequencies are central to the main claim. Please clarify the exact training input for each model.","section":"Section 2 and Section 7"}],"minor_comments":[{"comment":"There are several typographical issues: 'T able 1' in Section 4.2, 'e.g,.' in Section 4.2, 'underling' in Section 4.3, and inconsistent spacing in 'F ARM' and 'DASH'.","section":"Throughout"},{"comment":"Table 1 reports MAP values for link prediction methods without standard deviations or significance tests; the choice of Graph Factorization is motivated by scalability, but the reported differences (e.g., GF vs. HOPE on forks) may not be significant.","section":"Section 4.2, Table 1"},{"comment":"The probabilities shown in Figure 1 (e.g., 1.5% new user, 84.8% one-time event) are not derived in the text; please state which data period and filtering criteria were used to compute these frequencies.","section":"Section 4.3, Figure 1"},{"comment":"The phrase 'All models except the null model had a high percentage (more than 0.75) of community contributing users' is imprecise because Figure 3 does not show numeric values; please report the exact values in the text or table.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful systems and challenge report, but the evaluation evidence is the main limitation. I would encourage the editor to require the authors to add multi-month or uncertainty-quantified evaluation, and to clarify the training-data description, before publication. The central claim is plausible but currently rests on a fragile empirical basis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nQuick take: this paper is a solid engineering report from the DARPA SocialSim GitHub challenge, and its main claim holds up in the limited sense that matters: a simple per-user stationary frequency model beats fancier graph-embedding and Bayesian agents on most of the metrics they report. That is a genuinely useful result for anyone building agent-based models of techno-social platforms. It deserves a serious referee, but the evaluation section needs work before the claim can be fully trusted.\n\nWhat's new: the scale (roughly 3 million agents, 30 million actions) and the direct comparison of six model families in a common simulation framework. The authors are honest that no single model wins everywhere, and they highlight where the null model (a two-week replay) is competitive. The simulation framework (FARM/DASH) is described in enough detail to be useful. The evaluation is genuinely predictive: agents are trained on historical data and tested on a held-out month, so there's no obvious circularity.\n\nSoft spots: The central ranking rests on five metrics from a larger set, with no error bars or significance tests, and a single test month (February 2018). The metric-level winners vary—the null model does best on repository contributors, the Bayesian model on user popularity—so \"broadly most successful\" depends on an aggregation that is never specified. The one-month training window is reported as optimal across models, but that finding appears to be post-hoc, not pre-specified. And no code or data are released, which limits reproducibility.\n\nNone of this is fatal. The qualitative conclusion—that simple per-user frequency models are strong baselines for short-horizon prediction on GitHub—is plausible and consistent with the numbers shown. But a referee should ask for robustness checks: multiple test months, bootstrapped confidence intervals, or at least a clear statement of the metric aggregation rule. The paper would be stronger if the authors released the simulation code and the metric definitions.\n\nWho this is for: computational social scientists and agent-based modelers, especially those working on large-scale simulation challenges. It's a useful case study in evaluation pitfalls. I'd take it to a reading group and I'd cite it for the scaling result and the stationary-model finding, but I wouldn't treat the model ranking as definitive.\n\nMy recommendation: send it to peer review, but with a request for a robustness section before acceptance.\n\nBest","headline":"A useful engineering report that shows simple per-user frequency models beating fancier ones at GitHub scale; the evaluation is thin but the qualitative result is credible.","tokens_in":10322,"tokens_out":3160,"would_cite":true,"duration_ms":29431,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Simple per-user frequency models, built from one month of history, outperform more complex agents on most metrics in large-scale GitHub simulation.","keywords":["multi-agent simulation","GitHub","techno-social systems","stationary probabilistic model","agent-based simulation","link prediction","Bayesian model","large-scale simulation"],"falsifier":"A direct test would retrain all six models on a different training-to-test month pair, say January 2019 to February 2019, and check whether the stationary model still wins on a majority of metrics; if a month containing an unusual shock, such as a large influx of new users or a major platform change, erases or reverses the stationary model's advantage, the claim that slow change explains its success would be falsified. A complementary check would measure per-user divergence between the training and test event distributions and show that users with large divergence are predicted as well as users with small divergence, which would contradict the stationarity explanation.","tokens_in":9276,"feed_emoji":"📊","tokens_out":9381,"duration_ms":79753,"temperature":0.7,"pith_summary":"The paper reports a large-scale prediction exercise in which six different agent models were built to simulate activity on GitHub and their outputs were scored against one month of ground-truth events. Its central finding is that the most consistently accurate agents were not the most complex ones: they sampled each user's next action and repository from a probability distribution built from that same user's recent past behavior. This matters because it suggests that for a platform whose users change behavior slowly, a simple per-user empirical frequency model can outperform graph-embedding, Bayesian, and preferential-attachment agents on the majority of the reported metrics. The paper also shows that the simulation can be scaled to roughly three million agents and thirty million actions on commodity hardware, making this an existence proof for data-driven, planetary-scale agent simulation.","feed_headline":"Stationary per-user models beat complex agents on GitHub","feed_subtitle":"A user's own past month of GitHub activity outperformed complex agents on most metrics.","key_machinery":"The key object is the per-user stationary probability distribution over event types and repository targets, estimated from one month of each user's history and sampled at a rate fixed by that user's observed event frequency. One variant, the ground-event model, samples the event and repository jointly rather than independently, and it tended to do slightly better than the baseline variant. This distribution is what separates the winning agents from the null, Bayesian, and embedding-based agents; the paper attributes its advantage to preserving individual user identity and to the relative slowness of behavioral change on GitHub. The supporting machinery is a distributed simulation architecture that partitions users and repositories across compute nodes to run the full-scale simulation in about twenty minutes.","core_discovery":"The central claim, stated in the abstract and Section 5, is that among the six agent models tested in the GitHub simulation challenge, the broadly most successful agents sampled from a stationary probability distribution of actions and repositories for each user. The paper gives two reasons for this success: each agent is characterized by its own empirical frequencies rather than by population-level generalizations, and GitHub users change their behavior relatively slowly. No single model won every metric; in particular, the Bayesian model performed well on user popularity, and a null model that simply shifted past data forward did well on repository popularity because the top repositories turn over slowly. But on the majority of the reported metrics, including repository event counts and contributor dynamics, the stationary probabilistic models outperformed the link-prediction and Bayesian agents.","pith_inferences":["If the stationarity explanation is right, then on faster-paced platforms the lead of simple frequency models should shrink; measuring the model's margin as a function of per-user behavioral-change rate would give a quantitative test.","The ground-event model's slight edge over the baseline suggests the joint distribution of event and repository carries information that the marginals discard; a low-rank approximation of that joint distribution might preserve the advantage at much lower memory cost.","A natural next experiment is to combine the stationary model for existing users with the feature-based cold-start model for new users; the paper mentions intra-agent combination as a direction, so this is an explicit extension rather than a claim."],"forward_implications":["For any platform where behavior is slow-changing, a one-month per-user frequency model becomes a strong baseline that more complex agents should be required to beat.","Using all available training data can hurt prediction; the paper found that roughly one month of recent data was optimal across most of the agent models.","The same simulation framework and agent-comparison methodology can be applied to other techno-social systems, and the authors report they have begun doing so for Twitter and Reddit.","The main unresolved gap is cold start: stationary models cannot predict interactions between users and repositories they have not touched before, so they must be paired with feature-based or network-based models to cover new entities."],"supporting_citations":[{"why":"supplies the distributed simulation architecture that makes full-scale runs of millions of agents feasible","marker":"[5]"},{"why":"provides the agent framework used to implement the GitHub user agents","marker":"[4]"},{"why":"defines the rank-biased overlap score used to measure popularity prediction accuracy","marker":"[21]"},{"why":"provides the rank-based selection model used inside the Bayesian and preferential-attachment agents","marker":"[11]"},{"why":"supports the finding that recent activity predicts future activity better than longer histories, motivating the one-month training window","marker":"[2]"},{"why":"supplies the feature-selection method used to predict new user-repository interactions","marker":"[10]"}],"fun_headline_variants":["Simple per-user models beat complex agents on GitHub","Stationary user profiles win GitHub simulation challenge","Massive GitHub simulation favors simple probabilistic agents","3M agents, 30M actions: simplicity wins in GitHub forecast","GitHub prediction: user's own history beats sophisticated AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the premise that a GitHub user's behavior in the month being predicted looks like that same user's behavior in the previous month; if users' event rates or repository preferences shift quickly, the stationary model is merely replaying memorized data.","fun_headline_variants_meta":{"raw":{"variants":["Simple per-user models beat complex agents on GitHub","Stationary user profiles win GitHub simulation challenge","Massive GitHub simulation favors simple probabilistic agents","3M agents, 30M actions: simplicity wins in GitHub forecast","GitHub prediction: user's own history beats sophisticated AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1359,"prompt_tokens":923,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":360}},"tokens_in":539,"tokens_out":436,"duration_ms":4266,"temperature":1.0,"reasoning_tokens":360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:13:30.895193+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would retrain all six models on a different training-to-test month pair, say January 2019 to February 2019, and check whether the stationary model still wins on a majority of metrics; if a month containing an unusual shock, such as a large influx of new users or a major platform change, erases or reverses the stationary model's advantage, the claim that slow change explains its success would be falsified. A complementary check would measure per-user divergence between the training and test event distributions and show that users with large divergence are predicted as well as users with small divergence, which would contradict the stationarity explanation.","supporting_citations":[{"cited_title":"In: IJCAI/AAMAS Workshop on Massively Multi-Agent Systems (2018)","cited_arxiv_id":null,"evidence_quote":"supplies the distributed simulation architecture that makes full-scale runs of millions of agents feasible"},{"cited_title":"In: 5th International Symposium on Resilient Control Systems","cited_arxiv_id":null,"evidence_quote":"provides the agent framework used to implement the GitHub user agents"},{"cited_title":"ACM Transactions on Information Systems 28(4), 1–38 (nov 2010)","cited_arxiv_id":null,"evidence_quote":"defines the rank-biased overlap score used to measure popularity prediction accuracy"},{"cited_title":"Physical Review Letters 96(21), 218701 (may 2006)","cited_arxiv_id":null,"evidence_quote":"provides the rank-based selection model used inside the Bayesian and preferential-attachment agents"},{"cited_title":"EPJ Data Science 4(1), 1–14 (dec 2015)","cited_arxiv_id":null,"evidence_quote":"supports the finding that recent activity predicts future activity better than longer histories, motivating the one-month training window"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the feature-selection method used to predict new user-repository interactions"}],"review_version":1}