{"id":"29cb5281-acad-4d59-a989-46e13f92379e","arxiv_id":"1908.05873","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HOPE, a missing-data analogue of cross-validation for networks, is introduced as a model selection tool for ERGMs and applied to two social networks.","lead":"This paper adapts held-out predictive evaluation, a cross-validation analogue for network models, to select among exponential-family random graph models. It demonstrates on two real social networks that predictive metrics can rank competing model specifications and reveal when extra terms do not improve prediction.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"HOPE's model-selection claim lacks evidence that its rankings are statistically reliable: the decisive comparisons (Model 4 vs 5) differ by roughly 0.001 in accuracy, and no Monte Carlo error or ground-truth simulation is reported.","rationale":"The mechanics of HOPE are coherent: equation (2.3) is a valid observed-data likelihood, conditional simulation from (2.4) is standard, and random holdout makes missingness ignorable. The concern is not internal inconsistency but inferential support for the central claim. The reader's weakest assumption identifies the same load-bearing premise: predictive performance on held-out edge states of the same network is the criterion for model selection, and the paper does not validate that this proxy selects models that generalize. I sharpen this into a concrete, testable deficiency: the reported empirical differences between the two best models are so small that, without uncertainty quantification or a known ground truth, the rankings cannot be taken as reliable. The proposed simulation study would settle whether HOPE rankings track true generating models and independent-network predictive performance. This does not require rejecting the method; it requires additional evidence, so conditional acceptance is the appropriate verdict.","tokens_in":19824,"tokens_out":5792,"duration_ms":64090,"concrete_test":"Run a simulation study with S=100 independent networks generated from each of several known ERGMs (Bernoulli; GWESP; covariate homophily; GWESP plus covariates) at sizes comparable to the case studies (n=36 and 50). For each generated network, apply HOPE with B=500 and the paper's two held-out strategies, using the same candidate model set, and record rankings by each metric. Then compute how often HOPE selects the true generating model, and evaluate the selected model's predictive performance on an independent network generated from the same process, comparing it to the best available model. Also repeat the full HOPE procedure on 20 independent random partitions for at least one network to quantify partition variability.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim (Section 1) is that HOPE provides a practical basis for ERGM model selection. For that claim to hold, HOPE metrics must rank models reliably enough to adjudicate between specifications. This is not established. The empirical support consists of two real networks with no known generating model, and the decisive comparisons are extremely close: in Table 3 (Lazega) the leave-M-out Overall ACC for Model 4 is 0.789 versus 0.789 for Model 5, and TSL differs by 0.43; in Table 5 (friendship) leave-1-out Overall ACC is 0.938 versus 0.937. These are point estimates from B=500 simulations and a single random partition; no Monte Carlo standard errors, repeated-partition variability, or sensitivity analysis is reported, so differences of this size cannot be distinguished from noise. Moreover, HOPE evaluates conditional imputation of held-out edge states on the same network; held-out edge states are dependent on the training edge states, so the procedure does not automatically correct for overfitting as claimed in Section 1, and success at this imputation task is not shown to track generalization to new networks. Without a simulation study in which the true model is known and HOPE's selected model is compared with the model that best predicts independent networks from the same process, the central claim remains an assertion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a held-out predictive evaluation (HOPE) strategy for exponential-family random graph model (ERGM) selection. Building on the missing-data likelihood of Handcock and Gile (2010) and the HOPE scheme of Wang et al. (2016), the authors hold out edge states (rather than edge variables), fit the ERGM to the remaining data, simulate from the conditional distribution of the held-out states, and evaluate predictions using dyad-, node-, and graph-level metrics. Two hold-out schemes are described (random edge removal and node removal), and the method is demonstrated on two empirical networks, Lazega's lawyers and a teenage friendship network, with model rankings compared to nominal AIC and BIC.","tokens_in":20139,"tokens_out":6429,"duration_ms":61462,"significance":"If the proposed method reliably ranks ERGMs by their predictive performance, it would be a practically valuable complement to AIC/BIC and Bayes factors, which have questionable theoretical foundations for single-network ERGMs. The methodological core in Section 2 is standard and correctly stated, and the procedure builds on existing, widely used software (the ergm package), which is a clear practical strength. However, the empirical support is currently too weak to establish the central claim: the decisive model comparisons in the two case studies are extremely close in value, no uncertainty quantification is reported, and there is no simulation study with a known data-generating process showing that HOPE rankings track generalization to independent networks rather than mere imputation accuracy.","major_comments":[{"comment":"The empirical comparisons that carry the central claim are not accompanied by any measure of uncertainty. In Table 3, the leave-M-out Overall ACC for Model 4 is 0.789 and for Model 5 is 0.789, with TSL values 67.412 and 67.842; in Table 5, the leave-1-out Overall ACC is 0.938 versus 0.937 for Models 4 and 5. These differences are far smaller than the Monte Carlo error that should be expected from B=500 conditional simulations from a single random partition, and no repeated-partition or simulation standard errors are reported. The conclusion in Section 4.1 that \"HOPE suggests that model 4 is the preferred model\" therefore does not follow from the reported numbers, and the same issue affects the recommendation of Model 4 in Section 4.2. At minimum, the authors should report Monte Carlo standard errors or confidence intervals for the metrics, and ideally repeat the hold-out partitioning to quantify partition variability.","section":"Tables 3 and 5; Section 4.1"},{"comment":"The paper's central claim is that HOPE provides a practical basis for ERGM model selection because predictive assessments \"automatically correct for overfitting\" (Section 1). This claim is not validated by the two case studies, because in both cases the data-generating model is unknown and the held-out edge states are imputed conditional on the observed edge states of the same single network. There is no simulation study in which networks are generated from known models and HOPE's selected model is compared with the model that best predicts independent networks from the same process. Without such a study, the assertion that HOPE's ranking tracks generalization rather than mere imputation accuracy remains unsupported. I recommend adding a simulation section (e.g., generating networks from each candidate specification and from alternative processes, then comparing HOPE's selection with the best predictor on independent network replicates, with selection frequencies across repeated partitions).","section":"Section 1; Section 4"}],"minor_comments":[{"comment":"The table caption contains a typo: \"Descriptive statsitics\" should read \"Descriptive statistics\".","section":"Table 1"},{"comment":"The phrase \"computational challanges\" on page 4 should read \"computational challenges\".","section":"Section 1"},{"comment":"The prior is referred to as \"Jeffrey's prior\" but should be \"Jeffreys prior\".","section":"Section 3.2.1"},{"comment":"The running-time formula is labeled as both (3.3) and (3.4); the second equation number appears to be an error and should be removed or corrected.","section":"Section 3.6"},{"comment":"The AIC/BIC values for the same model differ between the main estimation tables and the HOPE tables (e.g., Model 2 in the Lazega analysis has AIC 524.3 in Table 2 but 523.0 in Table 3). The text should state explicitly what the AIC/BIC values in the HOPE tables represent, such as averages over held-out fits, or correct the discrepancy.","section":"Tables 2 and 3 / Tables 4 and 5"},{"comment":"The statement \"we set M = 35 (the same number of edge variables as under node-held-out)\" appears to confuse batch size with number of folds; under node-held-out, each batch contains all n-1 = 35 edge variables incident to a vertex, whereas with M = 35 random batches on an undirected 36-vertex graph, each batch contains 630/35 = 18 edge variables. Please clarify the intended comparison.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The topic is appropriate for the journal and the methodological core is sound, but the current empirical support is too weak for the claims made. I would not recommend rejection if the authors add a simulation study with known generating models and provide uncertainty quantification for the HOPE metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on Yin/Phillips/Butts (arXiv:1908.05873). The paper takes Wang et al.'s HOPE scheme, originally built for imputation assessment, and repurposes it as a model selection tool for ERGMs. That is a legitimate and useful extension: it gives practitioners a cross-validation analogue that avoids the shaky asymptotic assumptions behind AIC/BIC for single networks, and it is implementable with existing software. The authors lay out several held-out strategies (leave-1-out, leave-M-out, node isolation) and a sensible suite of metrics at dyad, node, and graph levels. The equations for missing-data likelihood and conditional simulation are standard and correct. The two case studies are well-known datasets, and the exposition is clear.\n\nThe soft spot is the empirical support for the central claim. The decisive comparisons, Model 4 vs Model 5 in both datasets, are extremely close: Overall ACC 0.789 vs 0.789 and TSL 67.4 vs 67.8 in the lawyer network; Overall ACC 0.938 vs 0.937 and TSL 40.0 vs 40.7 in the friendship network. These are point estimates from B=500 simulations on a single random partition, and the paper reports no Monte Carlo standard errors, no repeated-partition variability, and no sensitivity analysis. At that resolution, the differences are indistinguishable from noise. The authors say they focus on rankings rather than magnitudes, but a ranking separated by 0.001 is not a ranking established by the evidence. This is fixable: add MC error bars or show that the ordering is stable across partitions.\n\nThe stress-test note also flags that HOPE measures conditional imputation accuracy rather than generalization to new networks. That is true, but the paper is careful to frame HOPE as interpolation of held-out data; it does not claim to select models that forecast new networks. The intro's statement that predictive assessments 'automatically correct for overfitting' is slightly overstated—it protects against overfitting to the held-out edge states, not necessarily against overfitting to the network's particular configuration—but that's a minor quibble.\n\nAll in all, this is a solid methods paper with a clear gap in its empirical evidence. The method is sensible, the writing is honest about limitations, and the citation pattern is proper. I'd send it to review, but I'd ask the authors to either add uncertainty quantification or soften the model-selection claims.\n\nMy take: deserved a serious referee; conditional acceptance.","headline":"A sensible extension of HOPE to ERGM model selection, but the empirical case is under-powered: the decisive model comparisons are too close to distinguish from noise without Monte Carlo error bars.","tokens_in":20648,"tokens_out":3927,"would_cite":false,"duration_ms":35344,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a cross-validation-style procedure, HOPE, that selects among competing exponential-family random graph models by how well each predicts edge states held out from the observed network, and shows it can outperform…","keywords":["exponential-family random graph models","model selection","held-out predictive evaluation","cross-validation for networks","missing edge data","conditional simulation","network predictive metrics","imputation assessment"],"falsifier":"Find or construct a pair of ERGMs on a single network where the model with better HOPE-held-out metrics is nonetheless worse at predicting the edges of a new or future network from the same population; if such a case is common, HOPE's suitability for model selection in practice would be in doubt. A concrete version: fit competing models to year-one ties, score them by HOPE, then see whether the HOPE-preferred model also has lower error on year-two ties.","tokens_in":19645,"feed_emoji":"🕸️","tokens_out":5873,"duration_ms":51015,"temperature":0.7,"pith_summary":"The paper argues that model selection for exponential-family random graph models (ERGMs) can be put on a predictive footing, using a procedure it calls Held-Out Predictive Evaluation (HOPE): hold out the states of some edge variables, estimate the model from the remaining network with those states treated as missing, and score how well the fitted model predicts the held-out states under conditional simulation. The point of doing this, rather than relying on AIC/BIC or Bayes factors, is that those criteria rest on independence or asymptotic assumptions that are questionable for dependent network data, and they answer “which model is likeliest true” rather than “which model predicts what we care about.” The paper claims HOPE is simple to implement with standard software, adaptable to different held-out patterns and predictive metrics, and in two case studies it selects a more parsimonious model than AIC while exposing nominally significant terms that do not improve prediction.","feed_headline":"HOPE: pick network models by predicting held-out ties","feed_subtitle":"Judges ERGMs by predictive accuracy on held-out ties, not nominal fit; significant terms can add zero predictive power.","key_machinery":"The central object is the held-out edge-state partition of the adjacency matrix: the index set of edge variables is split into folds, and for each fold the states in that fold are marked missing while their presence is retained. Estimation uses the face-value likelihood that marginalizes over missing states, and evaluation uses conditional ERGM simulation from $\\hat{\\theta}$ with observed edges fixed, plus metrics such as overall accuracy, total squared loss, a centrality reliability coefficient $\\rho_C$, and centralization RMSE. That machinery converts the paper's notion of predictive performance into quantities that can be compared across model specifications.","core_discovery":"The central claim is that the ability of an ERGM to predict held-out edge states, conditional on the rest of the observed network, is a usable and illuminating model-selection criterion. The paper's innovation is to treat held-out edges as missing rather than removed: the presence of every edge variable is kept, only its state is hidden, so the dependence structure that makes standard cross-validation impossible for ERGMs is preserved. Estimation proceeds by the missing-data likelihood of Handcock and Gile, prediction by simulation from the conditional distribution of held-out states given observed states, and performance is summarized by dyad-, node-, and graph-level metrics. Applied to a lawyers' collaboration network and a teenage friendship network, HOPE ranks the models, prefers the more parsimonious model that includes both covariates and dependence terms, and shows that additional significant homophily terms do not improve predictive performance.","pith_inferences":["An implication the paper leaves implicit is that HOPE measures imputation within one network, not generalization to new networks; a model chosen by HOPE may not be the best at forecasting a future network, and testing against genuinely held-out networks would be a natural next step.","The paper's results suggest a testable extension: use HOPE-type predictive scores as a term-selection criterion rather than significance tests, since several significant homophily terms added no predictive value in the case studies.","A further consequence is that the choice of held-out strategy encodes a substantive assumption about the prediction task; researchers who care about vertex-level structure should match the holdout pattern to that task, and the paper's guidelines could be turned into a diagnostic test.","The exact leave-one-out marginal probabilities could be computed via change scores, as the paper notes; a cheaper algorithmic variant might use that exact computation to reduce simulation burden at scale."],"forward_implications":["HOPE provides a model-selection criterion that does not depend on the nominal sample size or asymptotic assumptions underlying AIC and BIC, so it is applicable even when those criteria are not.","Because scoring is done out-of-sample on held-out edge states, HOPE automatically penalizes overfitting: a complex model that merely fits the observed network better will not necessarily win.","The dyad-, node-, and graph-level metrics give diagnostic information about where a model fails, such as poor prediction of betweenness centralization, pointing to missing covariates or dependence terms.","Different held-out strategies are not interchangeable: leave-one-out gives stable dyad-level scores, node-holdout stresses local structure, and leave-M-out balances the two; the paper suggests leave-M-out as a general default.","HOPE can be applied to any network model for which parameter estimation with missing edge data is feasible, including valued-edge ERGMs, not just binary networks."],"supporting_citations":[{"why":"Introduced the HOPE scheme for imputation assessment that this paper adapts to model selection.","marker":"Wang et al. (2016)"},{"why":"Supplies the missing-data likelihood and conditional simulation machinery used to estimate and predict with held-out edge states.","marker":"Handcock & Gile (2010)"},{"why":"The ergm software package used here to fit ERGMs with missing data and to run conditional simulations.","marker":"Hunter et al. (2008b)"},{"why":"Provides ERGM specifications such as GWESP and the change-score calculations underlying exact leave-one-out dyad metrics.","marker":"Snijders et al. (2006)"},{"why":"Node-removal holdout strategy for influential-node assessment, adapted here as one HOPE holdout scheme.","marker":"Koskinen et al. (2018)"},{"why":"Provides the adjusted-pseudolikelihood Bayesian model-selection approach and the model terms and decay parameters used in the friendship case study.","marker":"Bouranis et al. (2018)"},{"why":"Cross-validation for ERGMs when multiple networks are available, the contrasting setting HOPE addresses for single networks.","marker":"Stewart et al. (2019)"}],"fun_headline_variants":["HOPE: ERGM selection via held-out predictive accuracy","Predict held-out ties to choose network models","HOPE: model networks by imputing missing ties","HOPE: better ERGM choice via predictive validation","HOPE: assess ERGMs on held-out edges, not fit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that predictive accuracy on edge states held out from a single observed network is a trustworthy guide to which model is genuinely better; since the held-out states are not independent test data, this equates model quality with imputation ability within that one network.","fun_headline_variants_meta":{"raw":{"variants":["HOPE: ERGM selection via held-out predictive accuracy","Predict held-out ties to choose network models","HOPE: model networks by imputing missing ties","HOPE: better ERGM choice via predictive validation","HOPE: assess ERGMs on held-out edges, not fit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1250,"prompt_tokens":964,"completion_tokens":286,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":580,"tokens_out":286,"duration_ms":3128,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:02:07.620652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find or construct a pair of ERGMs on a single network where the model with better HOPE-held-out metrics is nonetheless worse at predicting the edges of a new or future network from the same population; if such a case is common, HOPE's suitability for model selection in practice would be in doubt. A concrete version: fit competing models to year-one ties, score them by HOPE, then see whether the HOPE-preferred model also has lower error on year-two ties.","supporting_citations":[{"cited_title":", author Butts, C","cited_arxiv_id":null,"evidence_quote":"Introduced the HOPE scheme for imputation assessment that this paper adapts to model selection."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the missing-data likelihood and conditional simulation machinery used to estimate and predict with held-out edge states."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ERGM specifications such as GWESP and the change-score calculations underlying exact leave-one-out dyad metrics."},{"cited_title":", author Wang, P","cited_arxiv_id":null,"evidence_quote":"Node-removal holdout strategy for influential-node assessment, adapted here as one HOPE holdout scheme."},{"cited_title":", author Friel, N","cited_arxiv_id":null,"evidence_quote":"Provides the adjusted-pseudolikelihood Bayesian model-selection approach and the model terms and decay parameters used in the friendship case study."},{"cited_title":", author Schweinberger, M","cited_arxiv_id":null,"evidence_quote":"Cross-validation for ERGMs when multiple networks are available, the contrasting setting HOPE addresses for single networks."}],"review_version":1}