{"id":"a52cf6d5-9d45-48cc-966d-b07435a210cb","arxiv_id":"1908.08429","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Across 120 social networks, calibrated 2K and stochastic block models best reproduce selected structural metrics, but all tested models fail to capture the joint diameter-clustering relationship.","lead":"The authors studied 120 real social networks from three domains and generated synthetic counterparts with four network models calibrated to match them. They find that the 2K and stochastic block models mimic the selected network metrics best across domains, while none of the models capture the joint relationship between diameter and clustering.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Model ranking in Fig. 5 is evaluated on the same metric set used for calibration, so the 2K/SBM advantage is partly built into the fitting procedure.","rationale":"The reader's verdict is CONDITIONAL and correctly identifies circular evaluation as a major issue, although the formal 'weakest assumption' field focuses on metric completeness and the 0.65 correlation threshold. My stress-test agrees with the circularity concern as the most load-bearing: the central ranking claim depends on treating the calibrated metrics as if they were independent test metrics, when they are the same metrics used to fit the models. I do not think this warrants rejection, because the paper is transparent about its methodology and because the diameter-clustering failure is a useful, visually supported finding that does not depend on the ranking. The concrete test I propose would settle whether the ranking concern actually lands by comparing models on genuinely held-out structural features. If the ranking flips, the 'efficient mimicry' conclusion should be weakened to 'models fit the calibration metrics well'; if it persists, the concern is resolved. Thus the reader's CONDITIONAL verdict remains appropriate, with the condition being the requested out-of-sample evaluation.","tokens_in":7657,"tokens_out":2698,"duration_ms":31725,"concrete_test":"Refit the four models exactly as in the paper (Eq. (1), using the Table II metrics), then compute Canberra distances on a disjoint validation metric set deliberately excluded from calibration: for example, the non-selected metrics listed in Section II (max eigenvector centrality, max edge/vertex betweenness, global clustering coefficient, density, unnormalized average path length) plus at least one structural feature not in the original 17, such as graphlet degree distributions or k-core sizes. Re-rank the models per domain with bootstrap confidence intervals on mean distance. If the 2K/SBM advantage persists on those held-out metrics, the conclusion stands; if it shrinks or reverses, the current Fig. 5 ranking is a fitting artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the circularity between calibration and evaluation. In Section II, Eq. (1) fits model parameters θ* by minimizing the Canberra distance d(G_M(θ), G_T), where d is defined over a vector of structural metrics f. Section IV then ranks the four models by computing mean Canberra distances between original and generated graphs 'through their graph metrics', and the metrics in Table II are exactly the ones used during calibration. Thus Fig. 5 does not measure how well a model generalizes to new structural properties; it measures how well each model can minimize the training objective. The SBM and 2K models are parameter-rich and can match degree-distribution-related statistics by construction, so their advantage in Fig. 5 is at least partly an artifact of the fitting procedure. This does not invalidate the more specific scatter-plot observation that no model generates large diameter and high clustering coefficient simultaneously, because that failure is visible in Fig. 3 independent of the ranking. However, the broader claim that 2K and SBM 'can be used to mimic social networks relatively efficiently' is insecure until evaluation uses metrics that were not part of the calibration objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes 120 real social networks from three domains (friendship, communication, collaboration). For each network the authors compute 17 structural metrics, examine Spearman rank correlations between metrics across domains, and select a non-redundant subset using a maximal independent set on a correlation network thresholded at 0.65. They then calibrate four generative models (clustering Barabasi-Albert, stochastic block model, forest-fire, and 2K) by minimizing the Canberra distance between the selected metric vectors of the real and generated graphs, and compare the models by computing mean Canberra distances domain by domain. The main findings are that correlation patterns differ across domains, that 2K and stochastic block models mimic the selected metrics most closely, and that none of the models can generate graphs with simultaneously high diameter and high clustering coefficient.","tokens_in":7839,"tokens_out":3696,"duration_ms":39801,"significance":"The paper offers a useful large-scale empirical benchmark: 120 real networks and 480 calibrated synthetic graphs, with data and code made publicly available in a supplementary repository. The domain-specific correlation analysis of graph metrics is a valuable descriptive contribution, and the scatter-plot observation in Fig. 3 that no model reproduces the simultaneous combination of large normalized diameter and high clustering coefficient is an interesting, calibration-free falsifiable finding. However, the central comparative claim that 2K and SBM 'can be used to mimic social networks relatively efficiently' is weakened because the evaluation uses the same metric set and the same Canberra distance that were optimized during calibration. The independent content of the paper, especially the capacity limitations shown in Fig. 3, remains credible, but the model ranking in Fig. 5 needs a stronger, non-circular evaluation before the main conclusion is fully supported.","major_comments":[{"comment":"The evaluation is circular with respect to the calibration objective. Equation (1) calibrates model parameters by minimizing the Canberra distance d(G_M(θ), G_T) over a vector of graph metrics f, and Section IV then ranks the models by computing the mean Canberra distance between original and generated graphs over exactly the same metric set listed in Table II. Consequently, Fig. 5 measures how well each model can minimize the training objective rather than how well it generalizes to structural properties not used in fitting. The authors should evaluate the models on held-out metrics that were not part of the calibration objective, or at least report per-metric distances to show which properties drive the ranking. This issue is load-bearing for the conclusion that 2K and SBM efficiently mimic social networks.","section":"Section IV, Fig. 5, Eq. (1)"},{"comment":"The threshold of 0.65 for the domain-averaged absolute Spearman correlation and the maximal-independent-set selection rule are not justified and no sensitivity analysis is provided. The selected metric set determines the calibration objective and the subsequent evaluation, so a different threshold could change the metric set and possibly the model ranking. The authors should test a range of thresholds (for example 0.5, 0.6, 0.7, 0.8) and report whether the selected non-redundant set and the main conclusions are stable.","section":"Section III, Fig. 2"},{"comment":"The domain-averaged Canberra distances are presented as point estimates without any uncertainty quantification. There are no error bars, standard deviations, or significance tests, and some of the reported differences are small (for example, in the communication domain the original-2K distance is 0.93 while the original-SBM distance is 1.08). Without knowing the distribution of distances across the 43 communication networks, the claim that SBM and 2K efficiently capture the structural properties cannot be assessed quantitatively. The authors should report per-network distances, confidence intervals, or a paired significance test between models.","section":"Fig. 5, Section IV"}],"minor_comments":[{"comment":"The statement that the 17 graph measurements are 'chosen in such a way that together they measure every aspect of networks' is too strong; the paper should say that the metrics cover commonly studied aspects of degree distribution, shortest paths, centralities, and clustering.","section":"Section II"},{"comment":"The caption of Fig. 2 explains that correlations are domain-averaged absolute Spearman correlations, but this averaging is not described in the main text; it should be stated where the figure is referenced.","section":"Section III, Fig. 2 caption"},{"comment":"Table II lists 'domain' and 'category' as nominal variables, but the distance function in Eq. (1) is defined over real-valued metric functions; the role of these nominal variables in calibration and evaluation should be clarified.","section":"Table II"},{"comment":"The text says that models 'could mimic the structural properties of real networks, especially SBM and 2K', and later states that SBM and 2K generate similar graphs, but no quantitative measure of similarity between the two models is given.","section":"Section IV"},{"comment":"The conclusion refers to 'degree corrected stochastic block models' while Section II and Fig. 5 use 'stochastic block model (SBM)'; the authors should specify which variant was actually fitted, since the nonparametric microcanonical SBM of Peixoto can include degree correction but this is not stated in the methodology.","section":"Conclusion"},{"comment":"The calibration procedure is described only as grid search with details deferred to reference [12]; the parameter spaces and grid ranges for each model should be summarized in the main text to make the experiments reproducible without consulting the earlier paper.","section":"Section II"},{"comment":"The scatter plots in Figs. 3 and 4 use overlapping dots of different sizes, which makes it difficult to see the density of points; transparency, jitter, or separate density panels would improve readability.","section":"Fig. 3 and Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The circularity between calibration and evaluation is a genuine concern and the paper's headline comparison should not be published in its current form. That said, the paper contains a valuable dataset release and an interesting calibration-free observation about the diameter-clustering tradeoff, so the issues are fixable within the scope of a revision. I would not reject the paper outright."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a competent, transparent empirical study that gives the community a useful dataset and a clean negative result: none of the four tested models can generate graphs with both large diameter and high clustering coefficient. Second, the model ranking in Fig. 5 is on shakier ground than the prose suggests, because the same metric set and Canberra distance used to calibrate the models are also used to score them.\n\nWhat is genuinely new: the paper collects 120 social networks across three domains, reports domain-specific Spearman correlation profiles, prunes to a non-redundant metric set, calibrates four generative models, and shows the goodness-of-fit depends on domain. The scatter-based finding that 2K and SBM reproduce degree-related statistics well while all models fail on the joint diameter-clustering relation is real and visually supported. The authors are explicit about this limitation, and they ship data on GitHub, which is reproducible evidence.\n\nThe circularity concern is load-bearing for the broad claim that 2K and SBM “can be used to mimic social networks relatively efficiently.” Equation (1) fits parameters by minimizing Canberra distance over the selected metrics; Fig. 5 reports mean Canberra distances over the same metrics. So Fig. 5 is partly measuring in-sample optimization, not generalization. SBM and 2K are parameter-rich and match degree-distribution-related metrics by construction, so their apparent superiority is unsurprising. This does not invalidate the diameter-clustering observation, since that failure is visible in the scatter plots regardless of the ranking. But it does mean the paper has not shown that these models generalize better on unseen structural properties. The 0.65 Spearman threshold for metric selection is arbitrary, with no sensitivity analysis. Fig. 5 reports means without variance or significance tests, so the ordering of domains by modelability may be noise. Minor: the claim that the 17 metrics “measure every aspect of networks” is overstated.\n\nFor practitioners who need a benchmark of synthetic counterparts to real social networks, this paper is valuable. For a general network science reader, the headline takeaway is the diameter-clustering impossibility, not the model ranking. It deserves a serious referee; I would ask for held-out metrics and uncertainty quantification before accepting. I would cite it for the dataset and the negative result, not for the ranking.","headline":"Honest, useful empirical benchmark of calibrated network models on 120 social networks, but the model ranking in Fig. 5 is partly circular because evaluation uses the same metric set and distance as calibration.","tokens_in":8397,"tokens_out":1835,"would_cite":true,"duration_ms":20379,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Calibrated 2K and stochastic block models reproduce the structural metrics of 120 real social networks more closely than clustering preferential-attachment and forest-fire models, but no tested model creates a network that is both…","keywords":["social network analysis","graph metrics","network model calibration","2K model","stochastic block model","clustering coefficient","diameter","correlation analysis"],"falsifier":"Re-run the pipeline with a different metric basis, such as adding graphlet or spectral descriptors, or with a different redundancy threshold; if mean Canberra distances no longer put 2K and stochastic block models ahead of clustering preferential-attachment and forest-fire models in all three domains, the paper's ordering is not robust. The cannot-simultaneously claim can be tested directly by generating a large sample of calibrated 2K and stochastic block graphs while scanning parameter space for any point with normalized diameter and average clustering both above the real-network medians; one such graph would break the claim.","tokens_in":7389,"feed_emoji":"🕸️","tokens_out":7961,"duration_ms":76964,"temperature":0.7,"pith_summary":"The paper studies 120 large social networks from three domains—friendship, communication, and collaboration—and asks how well four standard generative models can imitate them after calibration to measured graph metrics. It claims that the 2K and stochastic block models are the most faithful imitators across all three domains, while clustering preferential-attachment and forest-fire models are less accurate on average. It also claims that the correlation profiles among structural metrics differ by domain, so a single universal description of social network topology would be misleading. The most pointed negative result is structural: none of the four models can generate graphs that combine a large normalized diameter with a high clustering coefficient, even though real social networks often have both.","feed_headline":"2K and block models best mimic 120 real social networks","feed_subtitle":"Across 120 networks, 2K and stochastic block models fit best, but the diameter-clustering gap remains.","key_machinery":"The argument is carried by a measurement-calibration pipeline rather than by a single identity. The key pieces are a selected vector of structural metrics—assortativity, average clustering coefficient, average degree, normalized pseudo-diameter, interval degree probabilities, and normalized maximum degree—chosen as a maximal independent set in the metric-correlation network; the Canberra distance between metric vectors as the goodness-of-fit measure; and grid-search parameter tuning for each of the four models against each real network. This machinery converts the abstract question of which model is more realistic into a numerical comparison of calibrated synthetic graphs with real graphs, and it is what allows the paper to localize which metric relationships models can and cannot reproduce.","core_discovery":"The central discovery is an empirical comparison with a negative joint finding. Across the 120 networks, the authors compute 17 structural measurements, reduce them to a non-redundant, size-independent set using a Spearman-correlation network with a 0.65 threshold, and fit each model by grid search minimizing the Canberra distance between the metric vectors of the real and synthetic graphs. Domain-averaged Canberra distances show that the 2K model and the stochastic block model reproduce the selected structural metrics most closely in every domain, with communication networks the easiest and friendship networks the hardest to mimic. However, when the authors examine pairwise relationships, they find a structural limitation shared by all four models: they cannot produce graphs that simultaneously have a large diameter normalized by log size and a high average clustering coefficient, a combination that appears in all three real-world domains.","pith_inferences":["The paper does not vary the 0.65 Spearman threshold; a natural stress test would be to rerun the entire selection-and-ranking pipeline at thresholds such as 0.5 or 0.8 to see whether the 2K and stochastic block model ordering is an artifact of that cutoff.","The claimed diameter-clustering gap concerns the four generative mechanisms; one can test whether adding a path-lengthening or triangle-adding postprocessing step to a 2K or stochastic block model reaches the missing corner of the metric space.","The authors suggest embedding-based comparison as a next step; a direct comparison of embedding distances with the Canberra metric on the selected basis would show whether the reduced metric set spans the information that modern graph representations extract."],"forward_implications":["Privacy-preserving synthetic counterparts of social networks can be generated by calibrating a 2K or stochastic block model to the selected metrics, avoiding release of the original graph.","Simulations of processes that depend jointly on diameter and clustering should not rely on any of the four models, since that trait combination is outside their reach.","Model selection should be domain-aware: communication networks are the easiest to mimic and friendship networks the hardest.","Because 2K and stochastic block models match degree-related metrics exactly through their construction, synthetic graphs from these models can be trusted when degree-distribution fidelity is the only requirement."],"supporting_citations":[{"why":"Supplies the clustering preferential-attachment model that is one of the four benchmarked generators.","marker":"[8]"},{"why":"Supplies the stochastic block model generator used in the calibration comparison.","marker":"[9]"},{"why":"Supplies the forest-fire model benchmarked in the comparison.","marker":"[10]"},{"why":"Supplies the 2K model, the joint-degree-matrix generator that is the strongest contender.","marker":"[11]"},{"why":"Establishes the calibration methodology and normalization procedure that this study extends to social networks.","marker":"[12]"},{"why":"Introduces measurement-calibrated graph models via grid search, the calibration approach adopted here.","marker":"[14]"},{"why":"Gives prior evidence that 2K models can efficiently mimic social networks, which this study corroborates at scale.","marker":"[15]"},{"why":"Defines the interval degree probabilities used among the selected structural metrics.","marker":"[26]"},{"why":"Motivates the use of Canberra distance as the graph-comparison metric in the calibration objective.","marker":"[27]"}],"fun_headline_variants":["Across 120 nets, 2K and block models lead, but miss combo","All four models fail to pair high diameter with high clustering","Best-fit model still can't recreate diameter-clustering combo","Social net mimicry: 2K and block models win, with a trade-off","Domain matters: 2K and block models fit, but structural gap remains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ordering of models depends on the assumption that the 17 computed metrics, after reduction at a 0.65 correlation threshold, still cover the descriptive space of network topology; if important structural information is missing from this basis, a model that looks faithful on these metrics could still be misleading.","fun_headline_variants_meta":{"raw":{"variants":["Across 120 nets, 2K and block models lead, but miss combo","All four models fail to pair high diameter with high clustering","Best-fit model still can't recreate diameter-clustering combo","Social net mimicry: 2K and block models win, with a trade-off","Domain matters: 2K and block models fit, but structural gap remains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000889,"raw_usage":{"total_tokens":3792,"prompt_tokens":857,"completion_tokens":2935,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":2839}},"tokens_in":473,"tokens_out":2935,"duration_ms":20005,"temperature":1.0,"reasoning_tokens":2839,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:38:53.528356+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline with a different metric basis, such as adding graphlet or spectral descriptors, or with a different redundancy threshold; if mean Canberra distances no longer put 2K and stochastic block models ahead of clustering preferential-attachment and forest-fire models in all three domains, the paper's ordering is not robust. The cannot-simultaneously claim can be tested directly by generating a large sample of calibrated 2K and stochastic block graphs while scanning parameter space for any point with normalized diameter and average clustering both above the real-network medians; one such graph would break the claim.","supporting_citations":[{"cited_title":"Growing scale-free networks with tunable clustering,","cited_arxiv_id":null,"evidence_quote":"Supplies the clustering preferential-attachment model that is one of the four benchmarked generators."},{"cited_title":"Nonparametric bayesian inference of the microcanonical stochastic block model,","cited_arxiv_id":null,"evidence_quote":"Supplies the stochastic block model generator used in the calibration comparison."},{"cited_title":"Graphs over time: densi- fication laws, shrinking diameters and possible explanations,","cited_arxiv_id":null,"evidence_quote":"Supplies the forest-fire model benchmarked in the comparison."},{"cited_title":"Construction of simple graphs with a target joint degree matrix and beyond,","cited_arxiv_id":null,"evidence_quote":"Supplies the 2K model, the joint-degree-matrix generator that is the strongest contender."},{"cited_title":"Network Classification Based Structural Analysis of Real Networks and their Model-Generated Counterparts","cited_arxiv_id":"1810.08498","evidence_quote":"Establishes the calibration methodology and normalization procedure that this study extends to social networks."},{"cited_title":"Measurement-calibrated graph models for social network experiments,","cited_arxiv_id":null,"evidence_quote":"Introduces measurement-calibrated graph models via grid search, the calibration approach adopted here."},{"cited_title":"Model selection for social networks using graphlets,","cited_arxiv_id":null,"evidence_quote":"Gives prior evidence that 2K models can efficiently mimic social networks, which this study corroborates at scale."},{"cited_title":"Quantification and compar- ison of degree distributions in complex networks,","cited_arxiv_id":null,"evidence_quote":"Defines the interval degree probabilities used among the selected structural metrics."},{"cited_title":"Efficient comparison of massive graphs through the use of ’graph fingerprints’,","cited_arxiv_id":null,"evidence_quote":"Motivates the use of Canberra distance as the graph-comparison metric in the calibration objective."}],"review_version":1}